[SR-Users] Core fm_realloc function

Mon Jul 30 21:01:25 CEST 2012

Hi,

I compiled kamailio with MEMDBG = 0 because I didn't want the memory debug
in kamailio log, but I didn't know these turns on f_malloc and disabling
q_malloc...

Probably the size of shared memory that I start kamailio is 32MB because I
didn't gave a different -m parameter value, exist a command to verify this
information?

At moment that crash happened, there were few registered users agents and
were being made tests with register and calls with TLS protocol. I got only
the two situations that I showed at first email.

Best Regards

2012/7/30 Daniel-Constantin Mierla <miconda at gmail.com>

>  Hello,
>
> MEMDBG=0 turns on f_malloc, disabling q_malloc. The token F_MALLOC in the
> kamailio -V output shows f_malloc is in use.
>
> q_malloc is more suitable for debugging, as it keep trace of location in
> the source code that allocates/frees the memory chunks, at the expense of a
> bit more memory usage due to tracking overhead. q_malloc was the default
> for 3.1, that's why I asked because the trace showed lines from f_malloc c
> code.
>
> What is the size of shared memory you start kamailio? Is the default 32MB
> or you give a different -m parameter value? Was the serve very loaded? How
> many such situations did you get so far?
>
> Cheers,
> Daniel
>
>
>
> On 7/30/12 7:16 PM, Bruno Bresciani wrote:
>
> Hi,
>
> Bellow is output of 'kamailio -V'...
>
>
> ------------------------------------------------------------------------------------------------------------------
> version: kamailio 3.1.2 (i386/linux) 4d9f90
> flags: STATS: Off, USE_IPV6, USE_TCP, USE_TLS, TLS_HOOKS, USE_RAW_SOCKS,
> DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC,
> F_MALLOC, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE,
> USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES
> ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16,
> MAX_URI_SIZE 1024, BUF_SIZE 65535, PKG_SIZE 4MB
> poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
> id: 4d9f90
> compiled on 10:39:54 Apr 29 2011 with gcc 4.1.2
>
> -------------------------------------------------------------------------------------------------------------------
>
> Daniel, I don't remember to enable f_malloc... Actually I do not know the
> difference between f_malloc and g_malloc to choose compile kamailio with
> f_malloc instead of g_malloc.
> My commands to compile kamailio were:
> *
> $ make PREFIX="/home2/local/kamailio" FLAVOUR=kamailio
> include_modules="db_postgres tls" MEMDBG=0 cfg**
> $ make all
> $ make install*
>
> If you need more informations, I am available...
>
> Best Regards
>
> 2012/7/30 Daniel-Constantin Mierla <miconda at gmail.com>
>
>>  Hello,
>>
>> can you provide output of 'kamailio -V'? Did you enable f_malloc instead
>> of q_malloc which was default in 3.1?
>>
>> Cheers,
>> Daniel
>>
>>
>> On 7/30/12 1:44 PM, Bruno Bresciani wrote:
>>
>> Thank's for feedback Daniel,
>>
>> This core is add to the tracker with ID 247...
>>
>> Best Regards
>>
>>
>> 2012/7/30 Daniel-Constantin Mierla <miconda at gmail.com>
>>
>>>  Hello,
>>>
>>> it is in my to-do list to investigate, so far I was not able to
>>> reproduce.
>>>
>>> Vacation time together with other travelings makes a bit more slower
>>> process. Please add it also to the tracker:
>>>
>>> http://sip-router.org/tracker/
>>>
>>> A minor release for 3.3 branch is going to be soon, being there makes
>>> sure it will be reviewed to see if it applies.
>>>
>>> Cheers,
>>> Daniel
>>>
>>>
>>> On 7/27/12 3:00 PM, Bruno Bresciani wrote:
>>>
>>> Hi Users,
>>>
>>> This is my ultimate try to ask for some help to debug the two cores
>>> generated by fm_realloc() function. If somebody can give me a feedback
>>> about my questions or doubts I will be very grateful, this user list of
>>> kamailio always help me to solve my problems.
>>>
>>> Best Regards
>>>
>>>
>>> 2012/7/23 Bruno Bresciani <bruno.bresciani at gmail.com>
>>>
>>>> Hi All,
>>>>
>>>> I imagine that this core doesn't easy to replicate but somebody  can
>>>> help me to understand why it was generated? There is some way to  prevent
>>>> it?
>>>>
>>>> Best Regards
>>>>
>>>>
>>>>
>>>>  2012/7/18 Renan Capaverde <renan.capaverde at digitro.com.br>
>>>>
>>>>>  I am having the same problem on kamailio 3.1.6. Someone please help.
>>>>>
>>>>> Cheers,
>>>>> Renan
>>>>>
>>>>> Em 17/7/2012 14:26, Bruno Bresciani escreveu:
>>>>>
>>>>> Excuse-me Daniel, I forgot this information... Kamailio version 3.1.2
>>>>>
>>>>> Best Regards
>>>>>
>>>>>
>>>>>  2012/7/17 Daniel-Constantin Mierla <miconda at gmail.com>
>>>>>
>>>>>>  Hello,
>>>>>>
>>>>>> send also the version 'kamailio -V' to match the proper sources.
>>>>>>
>>>>>> Cheers,
>>>>>> Daniel
>>>>>>
>>>>>>
>>>>>> On 7/17/12 7:11 PM, Bruno Bresciani wrote:
>>>>>>
>>>>>>  Hi all,
>>>>>>
>>>>>> I observed two at 0x08173bf0 in fm_realloc (qm=0xb5f7d000,
>>>>>> p=0xb61f0bc0, size=0) at mem/f_malloc.c:536.
>>>>>>
>>>>>> I don't know exactly what may be caused this core because the debug
>>>>>> of kamailio is disabled, but looking at backtrace (gdb) and source code I
>>>>>> could notice a relationship with TLS connection establishment (handshake).
>>>>>> Other important fact is that core was generated by the kamalio itself with
>>>>>> signal 6 (abort) because it can't find the pointer to memory realloc.
>>>>>> I'd like to know if someone already observed this issue and if my
>>>>>> analyse above is correct?
>>>>>>
>>>>>> Bellow is backtrace full with gdb:
>>>>>>
>>>>>> Core was generated by `/home2/local/kamailio/sbin/kamailio -P
>>>>>> /var/run/kamailio.pid'.
>>>>>> Program terminated with signal 6, Aborted.
>>>>>> #0  0x00f30402 in __kernel_vsyscall ()
>>>>>> (gdb)
>>>>>> (gdb)
>>>>>> (gdb)
>>>>>> (gdb)
>>>>>> (gdb) bt full
>>>>>> #0  0x00f30402 in __kernel_vsyscall ()
>>>>>> No symbol table info available.
>>>>>> #1  0x009fec10 in raise () from /lib/libc.so.6
>>>>>> No symbol table info available.
>>>>>> #2  0x00a00521 in abort () from /lib/libc.so.6
>>>>>> No symbol table info available.
>>>>>> #3  0x08173bf0 in fm_realloc (qm=0xb5f7d000, p=0xb61f0bc0, size=0) at
>>>>>> mem/f_malloc.c:536
>>>>>>         f = (struct fm_frag *) 0xb61f0bb8
>>>>>>         pf = <value optimized out>
>>>>>>         orig_size = 18
>>>>>> L=ÃbÂ¬Ã2n = (struct fm_frag *) 0xb61f0bd2
>>>>>>         ptr = <value optimized out>
>>>>>>         hash = 23548
>>>>>> #4  0x0051a87f in ser_realloc (ptr=0xb61f0bc0, size=32) at
>>>>>> ../../mem/shm_mem.h:266
>>>>>> No locals.
>>>>>> #5  0x00460295 in OPENSSL_cleanse () from /lib/libcrypto.so.6
>>>>>> No symbol table info available.
>>>>>> #6  0x004609f7 in CRYPTO_realloc () from /lib/libcrypto.so.6
>>>>>> No symbol table info available.
>>>>>> #7  0x003efb22 in sk_insert () from /lib/libcrypto.so.6
>>>>>> No symbol table info available.
>>>>>> #8  0x003efbaa in sk_push () from /lib/libcrypto.so.6
>>>>>> No symbol table info available.
>>>>>> #9  0x007c2ac3 in ssl_bytes_to_cipher_list () from /lib/libssl.so.6
>>>>>> No symbol table info available.
>>>>>> #10 0x007aa9f3 in ssl3_get_client_hello () from /lib/libssl.so.6
>>>>>> No symbol table info available.
>>>>>> #11 0x007ab815 in ssl3_accept () from /lib/libssl.so.6
>>>>>> No symbol table info available.
>>>>>> #12 0x007c1c3a in SSL_accept () from /lib/libssl.so.6
>>>>>> No symbol table info available.
>>>>>> #13 0x005278e3 in tls_accept (c=0xb621af88, error=0xbf91dfc8) at
>>>>>> tls_server.c:346
>>>>>>         ret = <value optimized out>
>>>>>>         ssl = (SSL *) 0xb61c1878
>>>>>>         cert = <value optimized out>
>>>>>>         tls_c = (struct tls_extra_data *) 0xb6208e58
>>>>>>         tls_log = <value optimized out>
>>>>>> #14 0x0052da8e in tls_read_f (c=0xb621af88, flags=0xbf91e200) at
>>>>>> tls_server.c:1028
>>>>>>         r = (struct tcp_req *) 0xb621affc
>>>>>>         bytes_free = 4095
>>>>>>         bytes_read = 70
>>>>>>         read_size = 4095
>>>>>>         ssl_error = 0
>>>>>>         ssl_read = 0
>>>>>>         ssl = (SSL *) 0xb61c1878
>>>>>>         rd_buf =
>>>>>> "\026\003\001\000A\001\000\000=\003\001OÃ¿\vÂ¡ywP\fkÃDÃ
>>>>>>
>>>>>> 0/\000\a\000\005\001\000ÃÃÃÃ¯000\000\026\0009\0008\0005\000\026\000\023\000\n\0003\0002\00
>>>>>> 202i<Ã¤<Ã¥Ã¾\031Ã¬234Â \237Ãª030Y){lÑ¶"...06Ã'Ãr9Ã¬Ã«Â¸{[Â©\t\205\037\036Ã¿HPH5Âµ\t(Ã©32Ã027Ã¦Ã²036Â¾\"\rÔ¤YÃ215uÂº\027)Ã¶000Ã±215\vÃÃ Ã©Ea\023xÃ©J\206T\204CÃ
>>>>>> wr_buf =
>>>>>> "\027\003\001\003Â wÂ²'\016Ñ¼Ã033ÃLÃ021Ã)Ã\220hnLÂ¦\020XÃª\236\200\tIhÃÂ¸Sf\234Ã\rÃ
>>>>>>
>>>>>> ?Â¯Â¼\024FÃ\200Âª/\2264RÃaÃÂ¤IpXÂ»Â¨Ã035Ã¤Qh\211Ã026Ã½4ÃU+\asrÂ£/`Â¤OÃ»1AÊ¹qUÜ¿Â·vÂ±Ã¼\205"...
>>>>>>         rd = {buf = 0xbf90dfa4 "\026\003\001", pos = 70, used = 70,
>>>>>> size = 65536}
>>>>>>         wr = {
>>>>>>   buf = 0xbf8fdfa4
>>>>>> "\027\003\001\003Â wÂ²'\016Ñ¼Ã033ÃLÃ021Ã)Ã\220hnLÂ¦\020XÃª\236\200\tIhÃÂ¸Sf\234Ã\rÃ
>>>>>> ?Â¯Â¼\024FÃ\200Âª/\2264RÃaÃÂ¤IpXÂ»Â¨Ã035Ã¤Qh\211Ã026Ã½4ÃU+\asrÂ£/`Â¤OÃ»1AÊ¹qUÜ¿Â·vÂ±Ã¼\205"...,
>>>>>> pos = 0, used = 0,
>>>>>> size = 65536}
>>>>>>         tls_c = (struct tls_extra_data *) 0xb6208e58
>>>>>>         enc_rd_buf = (struct tls_rd_buf *) 0x0
>>>>>>         n = 0
>>>>>>         flush_flags = <value optimized out>
>>>>>>         err_src = 0x541a83 "TLS read:"
>>>>>>         x = <value optimized out>
>>>>>> #15 0x08158ade in tcp_read_headers (c=0xb621af88,
>>>>>> read_flags=0xbf91e200) at tcp_read.c:406
>>>>>>         bytes = -1
>>>>>>         remaining = <value optimized out>
>>>>>>         p = <value optimized out>
>>>>>>         r = (struct tcp_req *) 0xb621affc
>>>>>> #16 0x08158fd4 in tcp_read_req (con=0xb621af88,
>>>>>> bytes_read=0xbf91e204, read_flags=0xbf91e200) at tcp_read.c:871
>>>>>>         bytes = <value optimized out>
>>>>>>         total_bytes = 0
>>>>>>         resp = <value optimized out>
>>>>>>         size = <value optimized out>
>>>>>>         req = (struct tcp_req *) 0xb621affc
>>>>>>         dst = {send_sock = 0x1, to = {s = {sa_family = 57540, sa_data
>>>>>> = "\221Â¿\020\000\000\000\000\000\000\000\020\000\000"}, sin = {sin_family
>>>>>> = 57540,
>>>>>>       sin_port = 49041, sin_addr = {s_addr = 16}, sin_zero =
>>>>>> "\000\000\000\000\020\000\000"}, sin6 = {sin6_family = 57540, sin6_port =
>>>>>> 49041,
>>>>>>       sin6_flowinfo = 16, sin6_addr = {in6_u = {u6_addr8 =
>>>>>> "\000\000\000\000\020\000\000\000\001\000\000\000\001\000\000", u6_addr16 =
>>>>>> {0, 0, 16, 0, 1, 0,
>>>>>>             1, 0}, u6_addr32 = {0, 16, 1, 1}}}, sin6_scope_id = 12}},
>>>>>> id = -1080958468, proto = 4 '\004', send_flags = {f = 0 '\0', blst_imask =
>>>>>> 0 '\0'}}
>>>>>>         c = 10 '\n'
>>>>>> #17 0x08159a0f in handle_io (fm=0x83296b0, events=1, idx=-1) at
>>>>>> tcp_read.c:1150
>>>>>>         ret = 4
>>>>>>         n = 4
>>>>>>         read_flags = 1
>>>>>>         con = (struct tcp_connection *) 0xb621af88
>>>>>>         s = 12
>>>>>>         resp = <value optimized out>
>>>>>>         t = <value optimized out>
>>>>>> #18 0x0815d653 in tcp_receive_loop (unix_sock=35) at io_wait.h:1091
>>>>>> No locals.
>>>>>> ---Type <return> to continue, or q <return> to quit---
>>>>>> #19 0x0812885b in tcp_init_children () at tcp_main.c:4819
>>>>>>         r = 3
>>>>>>         reader_fd_1 = 35
>>>>>>         pid = <value optimized out>
>>>>>>         si = <value optimized out>
>>>>>> #20 0x080ad412 in main_loop () at main.c:1632
>>>>>>         i = 4
>>>>>>         pid = <value optimized out>
>>>>>>         si = (struct socket_info *) 0x0
>>>>>>         si_desc = "udp receiver child=3 sock=192.168.166.31:5060
>>>>>> \000\221Â¿\002\000\000\000/\200\220\000\001\000\000\000\000\000H\000Â¨Ã¤21Â¿\001\000\000\000Â°Â¶Ã¨026Â¶\000\000\000\000\000ÃÂµ\030s2\b\002\000\000\000Ã¢f\b\000ÃÂµ\001\000\000\000\000\000\000\000\001\000\000\000\000\000\000\000Ã¿Ã¿Ã¿Ã¿Â¸Ã¤21Â¿"
>>>>>> #21 0x080afbe7 in main (argc=3, argv=dwarf2_read_address: Corrupted
>>>>>> DWARF expression.
>>>>>> ) at main.c:2398
>>>>>>         cfg_stream = (FILE *) 0x8a2d008
>>>>>>         c = <value optimized out>
>>>>>>         r = 0
>>>>>>         tmp = 0xbf91e674
>>>>>> "\212Ã¾\221Â¿\233Ã¾\221Â¿Â°Ã¾\221Â¿Â»Ã¾\221Â¿Ã\221Â¿Ã¹1Â¿\020Ã¿\221Â¿DÃ¿\221Â¿LÃ¿\221Â¿WÃ¿\221Â¿]Ã¿\221Â¿oÃ¿\221Â¿{Ã¿\221Â¿\202Ã¿\221
>>>>>> tmp_len = 10492229
>>>>>>         port = 136192361
>>>>>>         proto = -1080957480
>>>>>>         ret = <value optimized out>
>>>>>>         seed = 21782655
>>>>>>         rfd = 4
>>>>>>         debug_save = <value optimized out>
>>>>>>         debug_flag = 0
>>>>>>         dont_fork_cnt = 0
>>>>>>         n_lst = <value optimized out>
>>>>>>         p = <value optimized out>
>>>>>>
>>>>>>
>>>>>> Best Regards
>>>>>>
>>>>>>
>>>>>>  _______________________________________________
>>>>>> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing listsr-users at lists.sip-router.orghttp://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
>>>>>> Kamailio Advanced Training, Seattle, USA, Sep 23-26, 2012 - http://asipto.com/u/katu
>>>>>> Kamailio Practical Workshop, Netherlands, Sep 10-12, 2012 - http://asipto.com/u/kpw
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing listsr-users at lists.sip-router.orghttp://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
>>>>> sr-users at lists.sip-router.org
>>>>> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
>>>>>
>>>>>
>>>>
>>>
>>>
>>> _______________________________________________
>>> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing listsr-users at lists.sip-router.orghttp://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
>>>
>>>
>>> --
>>> Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
>>> Kamailio Advanced Training, Seattle, USA, Sep 23-26, 2012 - http://asipto.com/u/katu
>>> Kamailio Practical Workshop, Netherlands, Sep 10-12, 2012 - http://asipto.com/u/kpw
>>>
>>>
>>
>> --
>> Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
>> Kamailio Advanced Training, Seattle, USA, Sep 23-26, 2012 - http://asipto.com/u/katu
>> Kamailio Practical Workshop, Netherlands, Sep 10-12, 2012 - http://asipto.com/u/kpw
>>
>>
>
> --
> Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
> Kamailio Advanced Training, Seattle, USA, Sep 23-26, 2012 - http://asipto.com/u/katu
> Kamailio Practical Workshop, Netherlands, Sep 10-12, 2012 - http://asipto.com/u/kpw
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.sip-router.org/pipermail/sr-users/attachments/20120730/addc8c3c/attachment-0001.htm>