[SR-Users] Core fm_realloc function
Daniel-Constantin Mierla
miconda at gmail.com
Tue Jul 31 08:24:05 CEST 2012
Hello,
On 7/30/12 9:01 PM, Bruno Bresciani wrote:
> Hi,
>
> I compiled kamailio with MEMDBG = 0 because I didn't want the memory
> debug in kamailio log, but I didn't know these turns on f_malloc and
> disabling q_malloc...
the log messages related to memory operations can be controlled by
global parameters memdbg and memlog.
>
> Probably the size of shared memory that I start kamailio is 32MB
> because I didn't gave a different -m parameter value, exist a command
> to verify this information?
kamctl fifo get_statistics all
and see the shared memory total value. It will be interesting to see
available shared memory as well.
>
> At moment that crash happened, there were few registered users agents
> and were being made tests with register and calls with TLS protocol. I
> got only the two situations that I showed at first email.
If you plan to start a new installation, I strongly recommend 3.3
branch, the code is more actual and easier to debug. 3.1 is no longer an
official maintained branch, those being now 3.3 and 3.2. I'm looking at
this issue to be sure it is no longer in latest stable.
Cheers,
Daniel
>
>
> Best Regards
>
> 2012/7/30 Daniel-Constantin Mierla <miconda at gmail.com
> <mailto:miconda at gmail.com>>
>
> Hello,
>
> MEMDBG=0 turns on f_malloc, disabling q_malloc. The token F_MALLOC
> in the kamailio -V output shows f_malloc is in use.
>
> q_malloc is more suitable for debugging, as it keep trace of
> location in the source code that allocates/frees the memory
> chunks, at the expense of a bit more memory usage due to tracking
> overhead. q_malloc was the default for 3.1, that's why I asked
> because the trace showed lines from f_malloc c code.
>
> What is the size of shared memory you start kamailio? Is the
> default 32MB or you give a different -m parameter value? Was the
> serve very loaded? How many such situations did you get so far?
>
> Cheers,
> Daniel
>
>
>
> On 7/30/12 7:16 PM, Bruno Bresciani wrote:
>> Hi,
>>
>> Bellow is output of 'kamailio -V'...
>>
>> ------------------------------------------------------------------------------------------------------------------
>> version: kamailio 3.1.2 (i386/linux) 4d9f90
>> flags: STATS: Off, USE_IPV6, USE_TCP, USE_TLS, TLS_HOOKS,
>> USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM,
>> SHM_MMAP, PKG_MALLOC, F_MALLOC, USE_FUTEX,
>> FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER,
>> USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES
>> ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN
>> 16, MAX_URI_SIZE 1024, BUF_SIZE 65535, PKG_SIZE 4MB
>> poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
>> id: 4d9f90
>> compiled on 10:39:54 Apr 29 2011 with gcc 4.1.2
>> -------------------------------------------------------------------------------------------------------------------
>>
>> Daniel, I don't remember to enable f_malloc... Actually I do not
>> know the difference between f_malloc and g_malloc to choose
>> compile kamailio with f_malloc instead of g_malloc.
>> My commands to compile kamailio were:
>> *
>> $ make PREFIX="/home2/local/kamailio" FLAVOUR=kamailio
>> include_modules="db_postgres tls" MEMDBG=0 cfg**
>> $ make all
>> $ make install*
>>
>> If you need more informations, I am available...
>>
>> Best Regards
>> 2012/7/30 Daniel-Constantin Mierla <miconda at gmail.com
>> <mailto:miconda at gmail.com>>
>>
>> Hello,
>>
>> can you provide output of 'kamailio -V'? Did you enable
>> f_malloc instead of q_malloc which was default in 3.1?
>>
>> Cheers,
>> Daniel
>>
>>
>> On 7/30/12 1:44 PM, Bruno Bresciani wrote:
>>> Thank's for feedback Daniel,
>>>
>>> This core is add to the tracker with ID 247...
>>>
>>> Best Regards
>>>
>>>
>>> 2012/7/30 Daniel-Constantin Mierla <miconda at gmail.com
>>> <mailto:miconda at gmail.com>>
>>>
>>> Hello,
>>>
>>> it is in my to-do list to investigate, so far I was not
>>> able to reproduce.
>>>
>>> Vacation time together with other travelings makes a bit
>>> more slower process. Please add it also to the tracker:
>>>
>>> http://sip-router.org/tracker/
>>>
>>> A minor release for 3.3 branch is going to be soon,
>>> being there makes sure it will be reviewed to see if it
>>> applies.
>>>
>>> Cheers,
>>> Daniel
>>>
>>>
>>> On 7/27/12 3:00 PM, Bruno Bresciani wrote:
>>>> Hi Users,
>>>>
>>>> This is my ultimate try to ask for some help to debug
>>>> the two cores generated by fm_realloc() function. If
>>>> somebody can give me a feedback about my questions or
>>>> doubts I will be very grateful, this user list of
>>>> kamailio always help me to solve my problems.
>>>>
>>>> Best Regards
>>>>
>>>>
>>>> 2012/7/23 Bruno Bresciani <bruno.bresciani at gmail.com
>>>> <mailto:bruno.bresciani at gmail.com>>
>>>>
>>>> Hi All,
>>>>
>>>> I imagine that this core doesn't easy to replicate
>>>> but somebody can help me to understand why it was
>>>> generated? There is some way to prevent it?
>>>>
>>>> Best Regards
>>>>
>>>>
>>>>
>>>> 2012/7/18 Renan Capaverde
>>>> <renan.capaverde at digitro.com.br
>>>> <mailto:renan.capaverde at digitro.com.br>>
>>>>
>>>> I am having the same problem on kamailio 3.1.6.
>>>> Someone please help.
>>>>
>>>> Cheers,
>>>> Renan
>>>>
>>>> Em 17/7/2012 14:26, Bruno Bresciani escreveu:
>>>>> Excuse-me Daniel, I forgot this information...
>>>>> Kamailio version 3.1.2
>>>>>
>>>>> Best Regards
>>>>>
>>>>>
>>>>> 2012/7/17 Daniel-Constantin Mierla
>>>>> <miconda at gmail.com <mailto:miconda at gmail.com>>
>>>>>
>>>>> Hello,
>>>>>
>>>>> send also the version 'kamailio -V' to
>>>>> match the proper sources.
>>>>>
>>>>> Cheers,
>>>>> Daniel
>>>>>
>>>>>
>>>>> On 7/17/12 7:11 PM, Bruno Bresciani wrote:
>>>>>> Hi all,
>>>>>>
>>>>>> I observed two at 0x08173bf0 in
>>>>>> fm_realloc (qm=0xb5f7d000, p=0xb61f0bc0,
>>>>>> size=0) at mem/f_malloc.c:536.
>>>>>>
>>>>>> I don't know exactly what may be caused
>>>>>> this core because the debug of kamailio
>>>>>> is disabled, but looking at backtrace
>>>>>> (gdb) and source code I could notice a
>>>>>> relationship with TLS connection
>>>>>> establishment (handshake). Other
>>>>>> important fact is that core was generated
>>>>>> by the kamalio itself with signal 6
>>>>>> (abort) because it can't find the pointer
>>>>>> to memory realloc.
>>>>>> I'd like to know if someone already
>>>>>> observed this issue and if my analyse
>>>>>> above is correct?
>>>>>>
>>>>>> Bellow is backtrace full with gdb:
>>>>>>
>>>>>> Core was generated by
>>>>>> `/home2/local/kamailio/sbin/kamailio -P
>>>>>> /var/run/kamailio.pid'.
>>>>>> Program terminated with signal 6, Aborted.
>>>>>> #0 0x00f30402 in __kernel_vsyscall ()
>>>>>> (gdb)
>>>>>> (gdb)
>>>>>> (gdb)
>>>>>> (gdb)
>>>>>> (gdb) bt full
>>>>>> #0 0x00f30402 in __kernel_vsyscall ()
>>>>>> No symbol table info available.
>>>>>> #1 0x009fec10 in raise () from
>>>>>> /lib/libc.so.6
>>>>>> No symbol table info available.
>>>>>> #2 0x00a00521 in abort () from
>>>>>> /lib/libc.so.6
>>>>>> No symbol table info available.
>>>>>> #3 0x08173bf0 in fm_realloc
>>>>>> (qm=0xb5f7d000, p=0xb61f0bc0, size=0) at
>>>>>> mem/f_malloc.c:536
>>>>>> f = (struct fm_frag *) 0xb61f0bb8
>>>>>> pf = <value optimized out>
>>>>>> orig_size = 18
>>>>>> L=Ãb¬Ã2n = (struct fm_frag *) 0xb61f0bd2
>>>>>> ptr = <value optimized out>
>>>>>> hash = 23548
>>>>>> #4 0x0051a87f in ser_realloc
>>>>>> (ptr=0xb61f0bc0, size=32) at
>>>>>> ../../mem/shm_mem.h:266
>>>>>> No locals.
>>>>>> #5 0x00460295 in OPENSSL_cleanse () from
>>>>>> /lib/libcrypto.so.6
>>>>>> No symbol table info available.
>>>>>> #6 0x004609f7 in CRYPTO_realloc () from
>>>>>> /lib/libcrypto.so.6
>>>>>> No symbol table info available.
>>>>>> #7 0x003efb22 in sk_insert () from
>>>>>> /lib/libcrypto.so.6
>>>>>> No symbol table info available.
>>>>>> #8 0x003efbaa in sk_push () from
>>>>>> /lib/libcrypto.so.6
>>>>>> No symbol table info available.
>>>>>> #9 0x007c2ac3 in
>>>>>> ssl_bytes_to_cipher_list () from
>>>>>> /lib/libssl.so.6
>>>>>> No symbol table info available.
>>>>>> #10 0x007aa9f3 in ssl3_get_client_hello
>>>>>> () from /lib/libssl.so.6
>>>>>> No symbol table info available.
>>>>>> #11 0x007ab815 in ssl3_accept () from
>>>>>> /lib/libssl.so.6
>>>>>> No symbol table info available.
>>>>>> #12 0x007c1c3a in SSL_accept () from
>>>>>> /lib/libssl.so.6
>>>>>> No symbol table info available.
>>>>>> #13 0x005278e3 in tls_accept
>>>>>> (c=0xb621af88, error=0xbf91dfc8) at
>>>>>> tls_server.c:346
>>>>>> ret = <value optimized out>
>>>>>> ssl = (SSL *) 0xb61c1878
>>>>>> cert = <value optimized out>
>>>>>> tls_c = (struct tls_extra_data *)
>>>>>> 0xb6208e58
>>>>>> tls_log = <value optimized out>
>>>>>> #14 0x0052da8e in tls_read_f
>>>>>> (c=0xb621af88, flags=0xbf91e200) at
>>>>>> tls_server.c:1028
>>>>>> r = (struct tcp_req *) 0xb621affc
>>>>>> bytes_free = 4095
>>>>>> bytes_read = 70
>>>>>> read_size = 4095
>>>>>> ssl_error = 0
>>>>>> ssl_read = 0
>>>>>> ssl = (SSL *) 0xb61c1878
>>>>>> rd_buf =
>>>>>> "\026\003\001\000A\001\000\000=\003\001Oÿ\v¡ywP\fkÃDÃ
>>>>>> 0/\000\a\000\005\001\000ÃÃÃï000\000\026\0009\0008\0005\000\026\000\023\000\n\0003\0002\00
>>>>>> 202i<ä<åþ\031ì234 \237ê030Y){lѶ"...06Ã'Ãr9ìë¸{[©\t\205\037\036ÿHPH5µ\t(é32Ã027æò036¾\"\rÔ¤YÃ215uº\027)ö000ñ215\vÃà éEa\023xéJ\206T\204CÃ
>>>>>> wr_buf =
>>>>>> "\027\003\001\003 w²'\016ѼÃ033ÃLÃ021Ã)Ã\220hnL¦\020Xê\236\200\tIhøSf\234Ã\rÃ
>>>>>> ?¯¼\024FÃ\200ª/\2264RÃaäIpXȬÃ035äQh\211Ã026ý4ÃU+\asr£/`¤Oû1AʹqUܿ·v±ü\205"...
>>>>>> rd = {buf = 0xbf90dfa4
>>>>>> "\026\003\001", pos = 70, used = 70, size
>>>>>> = 65536}
>>>>>> wr = {
>>>>>> buf = 0xbf8fdfa4
>>>>>> "\027\003\001\003 w²'\016ѼÃ033ÃLÃ021Ã)Ã\220hnL¦\020Xê\236\200\tIhøSf\234Ã\rÃ
>>>>>> ?¯¼\024FÃ\200ª/\2264RÃaäIpXȬÃ035äQh\211Ã026ý4ÃU+\asr£/`¤Oû1AʹqUܿ·v±ü\205"...,
>>>>>> pos = 0, used = 0,
>>>>>> size = 65536}
>>>>>> tls_c = (struct tls_extra_data *)
>>>>>> 0xb6208e58
>>>>>> enc_rd_buf = (struct tls_rd_buf *) 0x0
>>>>>> n = 0
>>>>>> flush_flags = <value optimized out>
>>>>>> err_src = 0x541a83 "TLS read:"
>>>>>> x = <value optimized out>
>>>>>> #15 0x08158ade in tcp_read_headers
>>>>>> (c=0xb621af88, read_flags=0xbf91e200) at
>>>>>> tcp_read.c:406
>>>>>> bytes = -1
>>>>>> remaining = <value optimized out>
>>>>>> p = <value optimized out>
>>>>>> r = (struct tcp_req *) 0xb621affc
>>>>>> #16 0x08158fd4 in tcp_read_req
>>>>>> (con=0xb621af88, bytes_read=0xbf91e204,
>>>>>> read_flags=0xbf91e200) at tcp_read.c:871
>>>>>> bytes = <value optimized out>
>>>>>> total_bytes = 0
>>>>>> resp = <value optimized out>
>>>>>> size = <value optimized out>
>>>>>> req = (struct tcp_req *) 0xb621affc
>>>>>> dst = {send_sock = 0x1, to = {s =
>>>>>> {sa_family = 57540, sa_data =
>>>>>> "\221¿\020\000\000\000\000\000\000\000\020\000\000"},
>>>>>> sin = {sin_family = 57540,
>>>>>> sin_port = 49041, sin_addr =
>>>>>> {s_addr = 16}, sin_zero =
>>>>>> "\000\000\000\000\020\000\000"}, sin6 =
>>>>>> {sin6_family = 57540, sin6_port = 49041,
>>>>>> sin6_flowinfo = 16, sin6_addr = {in6_u =
>>>>>> {u6_addr8 =
>>>>>> "\000\000\000\000\020\000\000\000\001\000\000\000\001\000\000",
>>>>>> u6_addr16 = {0, 0, 16, 0, 1, 0,
>>>>>> 1, 0}, u6_addr32 = {0, 16, 1,
>>>>>> 1}}}, sin6_scope_id = 12}}, id =
>>>>>> -1080958468, proto = 4 '\004', send_flags
>>>>>> = {f = 0 '\0', blst_imask = 0 '\0'}}
>>>>>> c = 10 '\n'
>>>>>> #17 0x08159a0f in handle_io
>>>>>> (fm=0x83296b0, events=1, idx=-1) at
>>>>>> tcp_read.c:1150
>>>>>> ret = 4
>>>>>> n = 4
>>>>>> read_flags = 1
>>>>>> con = (struct tcp_connection *)
>>>>>> 0xb621af88
>>>>>> s = 12
>>>>>> resp = <value optimized out>
>>>>>> t = <value optimized out>
>>>>>> #18 0x0815d653 in tcp_receive_loop
>>>>>> (unix_sock=35) at io_wait.h:1091
>>>>>> No locals.
>>>>>> ---Type <return> to continue, or q
>>>>>> <return> to quit---
>>>>>> #19 0x0812885b in tcp_init_children () at
>>>>>> tcp_main.c:4819
>>>>>> r = 3
>>>>>> reader_fd_1 = 35
>>>>>> pid = <value optimized out>
>>>>>> si = <value optimized out>
>>>>>> #20 0x080ad412 in main_loop () at main.c:1632
>>>>>> i = 4
>>>>>> pid = <value optimized out>
>>>>>> si = (struct socket_info *) 0x0
>>>>>> si_desc = "udp receiver child=3
>>>>>> sock=192.168.166.31:5060
>>>>>> <http://192.168.166.31:5060>\000\221¿\002\000\000\000/\200\220\000\001\000\000\000\000\000H\000¨ä21¿\001\000\000\000°¶è026¶\000\000\000\000\000õ\030s2\b\002\000\000\000âf\b\000õ\001\000\000\000\000\000\000\000\001\000\000\000\000\000\000\000ÿÿÿÿ¸ä21¿"
>>>>>> #21 0x080afbe7 in main (argc=3,
>>>>>> argv=dwarf2_read_address: Corrupted DWARF
>>>>>> expression.
>>>>>> ) at main.c:2398
>>>>>> cfg_stream = (FILE *) 0x8a2d008
>>>>>> c = <value optimized out>
>>>>>> r = 0
>>>>>> tmp = 0xbf91e674
>>>>>> "\212þ\221¿\233þ\221¿°þ\221¿»þ\221¿Ã\221¿ù1¿\020ÿ\221¿Dÿ\221¿Lÿ\221¿Wÿ\221¿]ÿ\221¿oÿ\221¿{ÿ\221¿\202ÿ\221
>>>>>> tmp_len = 10492229
>>>>>> port = 136192361
>>>>>> proto = -1080957480
>>>>>> ret = <value optimized out>
>>>>>> seed = 21782655
>>>>>> rfd = 4
>>>>>> debug_save = <value optimized out>
>>>>>> debug_flag = 0
>>>>>> dont_fork_cnt = 0
>>>>>> n_lst = <value optimized out>
>>>>>> p = <value optimized out>
>>>>>>
>>>>>>
>>>>>> Best Regards
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
>>>>>> sr-users at lists.sip-router.org <mailto:sr-users at lists.sip-router.org>
>>>>>> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
>>>>>
>>>>> --
>>>>> Daniel-Constantin Mierla -http://www.asipto.com
>>>>> http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> -http://www.linkedin.com/in/miconda
>>>>> Kamailio Advanced Training, Seattle, USA, Sep 23-26, 2012 -http://asipto.com/u/katu
>>>>> Kamailio Practical Workshop, Netherlands, Sep 10-12, 2012 -http://asipto.com/u/kpw
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
>>>>> sr-users at lists.sip-router.org <mailto:sr-users at lists.sip-router.org>
>>>>> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> SIP Express Router (SER) and Kamailio (OpenSER)
>>>> - sr-users mailing list
>>>> sr-users at lists.sip-router.org
>>>> <mailto:sr-users at lists.sip-router.org>
>>>> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
>>>> sr-users at lists.sip-router.org <mailto:sr-users at lists.sip-router.org>
>>>> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
>>>
>>> --
>>> Daniel-Constantin Mierla -http://www.asipto.com
>>> http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> -http://www.linkedin.com/in/miconda
>>> Kamailio Advanced Training, Seattle, USA, Sep 23-26, 2012 -http://asipto.com/u/katu
>>> Kamailio Practical Workshop, Netherlands, Sep 10-12, 2012 -http://asipto.com/u/kpw
>>>
>>>
>>
>> --
>> Daniel-Constantin Mierla -http://www.asipto.com
>> http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> -http://www.linkedin.com/in/miconda
>> Kamailio Advanced Training, Seattle, USA, Sep 23-26, 2012 -http://asipto.com/u/katu
>> Kamailio Practical Workshop, Netherlands, Sep 10-12, 2012 -http://asipto.com/u/kpw
>>
>>
>
> --
> Daniel-Constantin Mierla -http://www.asipto.com
> http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> -http://www.linkedin.com/in/miconda
> Kamailio Advanced Training, Seattle, USA, Sep 23-26, 2012 -http://asipto.com/u/katu
> Kamailio Practical Workshop, Netherlands, Sep 10-12, 2012 -http://asipto.com/u/kpw
>
>
--
Daniel-Constantin Mierla - http://www.asipto.com
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Kamailio Advanced Training, Seattle, USA, Sep 23-26, 2012 - http://asipto.com/u/katu
Kamailio Practical Workshop, Netherlands, Sep 10-12, 2012 - http://asipto.com/u/kpw
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.sip-router.org/pipermail/sr-users/attachments/20120731/04e46e7a/attachment-0001.htm>
More information about the sr-users
mailing list