[SR-Users] Core fm_realloc function
Bruno Bresciani
bruno.bresciani at gmail.com
Tue Jul 31 15:39:27 CEST 2012
Hello,
I didn't know that log messages related to memory operations can be
controlled by global parameter, but I like to know if is recommendable I
recompile kamailio using q_malloc (default) and not f_malloc... memory
operations using q_malloc is more reliable and avoid problems or crashes or
it is only more suitable for debugging? What do you suggest Daniel?
The 'get_statistics all' command is avaliable by a specific module? I run
'kamctl fifo get_statistics all' and return '500 command 'get_statistics'
not available'
For while is impossible to start a new installation, first because I don't
know how much time I will spend to port and second because I am involved
with other developments and I have no time to make this. I know that 3.1
is no longer a official branch but now start a new installation it's very
very difficult, my in intention is discover what caused the crash and if
exists a way to fix or prevent it.
Best Regards
2012/7/31 Daniel-Constantin Mierla <miconda at gmail.com>
> Hello,
>
>
> On 7/30/12 9:01 PM, Bruno Bresciani wrote:
>
> Hi,
>
> I compiled kamailio with MEMDBG = 0 because I didn't want the memory
> debug in kamailio log, but I didn't know these turns on f_malloc and
> disabling q_malloc...
>
> the log messages related to memory operations can be controlled by global
> parameters memdbg and memlog.
>
>
>
> Probably the size of shared memory that I start kamailio is 32MB because I
> didn't gave a different -m parameter value, exist a command to verify this
> information?
>
>
> kamctl fifo get_statistics all
>
> and see the shared memory total value. It will be interesting to see
> available shared memory as well.
>
>
>
> At moment that crash happened, there were few registered users agents and
> were being made tests with register and calls with TLS protocol. I got only
> the two situations that I showed at first email.
>
>
> If you plan to start a new installation, I strongly recommend 3.3 branch,
> the code is more actual and easier to debug. 3.1 is no longer an official
> maintained branch, those being now 3.3 and 3.2. I'm looking at this issue
> to be sure it is no longer in latest stable.
>
> Cheers,
> Daniel
>
>
>
>
> Best Regards
>
> 2012/7/30 Daniel-Constantin Mierla <miconda at gmail.com>
>
>> Hello,
>>
>> MEMDBG=0 turns on f_malloc, disabling q_malloc. The token F_MALLOC in the
>> kamailio -V output shows f_malloc is in use.
>>
>> q_malloc is more suitable for debugging, as it keep trace of location in
>> the source code that allocates/frees the memory chunks, at the expense of a
>> bit more memory usage due to tracking overhead. q_malloc was the default
>> for 3.1, that's why I asked because the trace showed lines from f_malloc c
>> code.
>>
>> What is the size of shared memory you start kamailio? Is the default 32MB
>> or you give a different -m parameter value? Was the serve very loaded? How
>> many such situations did you get so far?
>>
>> Cheers,
>> Daniel
>>
>>
>>
>> On 7/30/12 7:16 PM, Bruno Bresciani wrote:
>>
>> Hi,
>>
>> Bellow is output of 'kamailio -V'...
>>
>>
>> ------------------------------------------------------------------------------------------------------------------
>> version: kamailio 3.1.2 (i386/linux) 4d9f90
>> flags: STATS: Off, USE_IPV6, USE_TCP, USE_TLS, TLS_HOOKS, USE_RAW_SOCKS,
>> DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC,
>> F_MALLOC, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE,
>> USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES
>> ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16,
>> MAX_URI_SIZE 1024, BUF_SIZE 65535, PKG_SIZE 4MB
>> poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
>> id: 4d9f90
>> compiled on 10:39:54 Apr 29 2011 with gcc 4.1.2
>>
>> -------------------------------------------------------------------------------------------------------------------
>>
>> Daniel, I don't remember to enable f_malloc... Actually I do not know
>> the difference between f_malloc and g_malloc to choose compile kamailio
>> with f_malloc instead of g_malloc.
>> My commands to compile kamailio were:
>> *
>> $ make PREFIX="/home2/local/kamailio" FLAVOUR=kamailio
>> include_modules="db_postgres tls" MEMDBG=0 cfg**
>> $ make all
>> $ make install*
>>
>> If you need more informations, I am available...
>>
>> Best Regards
>>
>> 2012/7/30 Daniel-Constantin Mierla <miconda at gmail.com>
>>
>>> Hello,
>>>
>>> can you provide output of 'kamailio -V'? Did you enable f_malloc instead
>>> of q_malloc which was default in 3.1?
>>>
>>> Cheers,
>>> Daniel
>>>
>>>
>>> On 7/30/12 1:44 PM, Bruno Bresciani wrote:
>>>
>>> Thank's for feedback Daniel,
>>>
>>> This core is add to the tracker with ID 247...
>>>
>>> Best Regards
>>>
>>>
>>> 2012/7/30 Daniel-Constantin Mierla <miconda at gmail.com>
>>>
>>>> Hello,
>>>>
>>>> it is in my to-do list to investigate, so far I was not able to
>>>> reproduce.
>>>>
>>>> Vacation time together with other travelings makes a bit more slower
>>>> process. Please add it also to the tracker:
>>>>
>>>> http://sip-router.org/tracker/
>>>>
>>>> A minor release for 3.3 branch is going to be soon, being there makes
>>>> sure it will be reviewed to see if it applies.
>>>>
>>>> Cheers,
>>>> Daniel
>>>>
>>>>
>>>> On 7/27/12 3:00 PM, Bruno Bresciani wrote:
>>>>
>>>> Hi Users,
>>>>
>>>> This is my ultimate try to ask for some help to debug the two cores
>>>> generated by fm_realloc() function. If somebody can give me a feedback
>>>> about my questions or doubts I will be very grateful, this user list of
>>>> kamailio always help me to solve my problems.
>>>>
>>>> Best Regards
>>>>
>>>>
>>>> 2012/7/23 Bruno Bresciani <bruno.bresciani at gmail.com>
>>>>
>>>>> Hi All,
>>>>>
>>>>> I imagine that this core doesn't easy to replicate but somebody can
>>>>> help me to understand why it was generated? There is some way to prevent
>>>>> it?
>>>>>
>>>>> Best Regards
>>>>>
>>>>>
>>>>>
>>>>> 2012/7/18 Renan Capaverde <renan.capaverde at digitro.com.br>
>>>>>
>>>>>> I am having the same problem on kamailio 3.1.6. Someone please help.
>>>>>>
>>>>>> Cheers,
>>>>>> Renan
>>>>>>
>>>>>> Em 17/7/2012 14:26, Bruno Bresciani escreveu:
>>>>>>
>>>>>> Excuse-me Daniel, I forgot this information... Kamailio version 3.1.2
>>>>>>
>>>>>> Best Regards
>>>>>>
>>>>>>
>>>>>> 2012/7/17 Daniel-Constantin Mierla <miconda at gmail.com>
>>>>>>
>>>>>>> Hello,
>>>>>>>
>>>>>>> send also the version 'kamailio -V' to match the proper sources.
>>>>>>>
>>>>>>> Cheers,
>>>>>>> Daniel
>>>>>>>
>>>>>>>
>>>>>>> On 7/17/12 7:11 PM, Bruno Bresciani wrote:
>>>>>>>
>>>>>>> Hi all,
>>>>>>>
>>>>>>> I observed two at 0x08173bf0 in fm_realloc (qm=0xb5f7d000,
>>>>>>> p=0xb61f0bc0, size=0) at mem/f_malloc.c:536.
>>>>>>>
>>>>>>> I don't know exactly what may be caused this core because the debug
>>>>>>> of kamailio is disabled, but looking at backtrace (gdb) and source code I
>>>>>>> could notice a relationship with TLS connection establishment (handshake).
>>>>>>> Other important fact is that core was generated by the kamalio itself with
>>>>>>> signal 6 (abort) because it can't find the pointer to memory realloc.
>>>>>>> I'd like to know if someone already observed this issue and if my
>>>>>>> analyse above is correct?
>>>>>>>
>>>>>>> Bellow is backtrace full with gdb:
>>>>>>>
>>>>>>> Core was generated by `/home2/local/kamailio/sbin/kamailio -P
>>>>>>> /var/run/kamailio.pid'.
>>>>>>> Program terminated with signal 6, Aborted.
>>>>>>> #0 0x00f30402 in __kernel_vsyscall ()
>>>>>>> (gdb)
>>>>>>> (gdb)
>>>>>>> (gdb)
>>>>>>> (gdb)
>>>>>>> (gdb) bt full
>>>>>>> #0 0x00f30402 in __kernel_vsyscall ()
>>>>>>> No symbol table info available.
>>>>>>> #1 0x009fec10 in raise () from /lib/libc.so.6
>>>>>>> No symbol table info available.
>>>>>>> #2 0x00a00521 in abort () from /lib/libc.so.6
>>>>>>> No symbol table info available.
>>>>>>> #3 0x08173bf0 in fm_realloc (qm=0xb5f7d000, p=0xb61f0bc0, size=0)
>>>>>>> at mem/f_malloc.c:536
>>>>>>> f = (struct fm_frag *) 0xb61f0bb8
>>>>>>> pf = <value optimized out>
>>>>>>> orig_size = 18
>>>>>>> L=Ãb¬Ã2n = (struct fm_frag *) 0xb61f0bd2
>>>>>>> ptr = <value optimized out>
>>>>>>> hash = 23548
>>>>>>> #4 0x0051a87f in ser_realloc (ptr=0xb61f0bc0, size=32) at
>>>>>>> ../../mem/shm_mem.h:266
>>>>>>> No locals.
>>>>>>> #5 0x00460295 in OPENSSL_cleanse () from /lib/libcrypto.so.6
>>>>>>> No symbol table info available.
>>>>>>> #6 0x004609f7 in CRYPTO_realloc () from /lib/libcrypto.so.6
>>>>>>> No symbol table info available.
>>>>>>> #7 0x003efb22 in sk_insert () from /lib/libcrypto.so.6
>>>>>>> No symbol table info available.
>>>>>>> #8 0x003efbaa in sk_push () from /lib/libcrypto.so.6
>>>>>>> No symbol table info available.
>>>>>>> #9 0x007c2ac3 in ssl_bytes_to_cipher_list () from /lib/libssl.so.6
>>>>>>> No symbol table info available.
>>>>>>> #10 0x007aa9f3 in ssl3_get_client_hello () from /lib/libssl.so.6
>>>>>>> No symbol table info available.
>>>>>>> #11 0x007ab815 in ssl3_accept () from /lib/libssl.so.6
>>>>>>> No symbol table info available.
>>>>>>> #12 0x007c1c3a in SSL_accept () from /lib/libssl.so.6
>>>>>>> No symbol table info available.
>>>>>>> #13 0x005278e3 in tls_accept (c=0xb621af88, error=0xbf91dfc8) at
>>>>>>> tls_server.c:346
>>>>>>> ret = <value optimized out>
>>>>>>> ssl = (SSL *) 0xb61c1878
>>>>>>> cert = <value optimized out>
>>>>>>> tls_c = (struct tls_extra_data *) 0xb6208e58
>>>>>>> tls_log = <value optimized out>
>>>>>>> #14 0x0052da8e in tls_read_f (c=0xb621af88, flags=0xbf91e200) at
>>>>>>> tls_server.c:1028
>>>>>>> r = (struct tcp_req *) 0xb621affc
>>>>>>> bytes_free = 4095
>>>>>>> bytes_read = 70
>>>>>>> read_size = 4095
>>>>>>> ssl_error = 0
>>>>>>> ssl_read = 0
>>>>>>> ssl = (SSL *) 0xb61c1878
>>>>>>> rd_buf =
>>>>>>> "\026\003\001\000A\001\000\000=\003\001Oÿ\v¡ywP\fkÃDÃ
>>>>>>>
>>>>>>> 0/\000\a\000\005\001\000ÃÃÃï000\000\026\0009\0008\0005\000\026\000\023\000\n\0003\0002\00
>>>>>>> 202i<ä<åþ\031ì234 \237ê030Y){lѶ"...06Ã'Ãr9ìë¸{[©\t\205\037\036ÿHPH5µ\t(é32Ã027æò036¾\"\rÔ¤YÃ215uº\027)ö000ñ215\vÃà éEa\023xéJ\206T\204CÃ
>>>>>>> wr_buf =
>>>>>>> "\027\003\001\003 w²'\016ѼÃ033ÃLÃ021Ã)Ã\220hnL¦\020Xê\236\200\tIhøSf\234Ã\rÃ
>>>>>>>
>>>>>>> ?¯¼\024FÃ\200ª/\2264RÃaäIpXȬÃ035äQh\211Ã026ý4ÃU+\asr£/`¤Oû1AʹqUܿ·v±ü\205"...
>>>>>>> rd = {buf = 0xbf90dfa4 "\026\003\001", pos = 70, used = 70,
>>>>>>> size = 65536}
>>>>>>> wr = {
>>>>>>> buf = 0xbf8fdfa4
>>>>>>> "\027\003\001\003 w²'\016ѼÃ033ÃLÃ021Ã)Ã\220hnL¦\020Xê\236\200\tIhøSf\234Ã\rÃ
>>>>>>> ?¯¼\024FÃ\200ª/\2264RÃaäIpXȬÃ035äQh\211Ã026ý4ÃU+\asr£/`¤Oû1AʹqUܿ·v±ü\205"...,
>>>>>>> pos = 0, used = 0,
>>>>>>> size = 65536}
>>>>>>> tls_c = (struct tls_extra_data *) 0xb6208e58
>>>>>>> enc_rd_buf = (struct tls_rd_buf *) 0x0
>>>>>>> n = 0
>>>>>>> flush_flags = <value optimized out>
>>>>>>> err_src = 0x541a83 "TLS read:"
>>>>>>> x = <value optimized out>
>>>>>>> #15 0x08158ade in tcp_read_headers (c=0xb621af88,
>>>>>>> read_flags=0xbf91e200) at tcp_read.c:406
>>>>>>> bytes = -1
>>>>>>> remaining = <value optimized out>
>>>>>>> p = <value optimized out>
>>>>>>> r = (struct tcp_req *) 0xb621affc
>>>>>>> #16 0x08158fd4 in tcp_read_req (con=0xb621af88,
>>>>>>> bytes_read=0xbf91e204, read_flags=0xbf91e200) at tcp_read.c:871
>>>>>>> bytes = <value optimized out>
>>>>>>> total_bytes = 0
>>>>>>> resp = <value optimized out>
>>>>>>> size = <value optimized out>
>>>>>>> req = (struct tcp_req *) 0xb621affc
>>>>>>> dst = {send_sock = 0x1, to = {s = {sa_family = 57540,
>>>>>>> sa_data = "\221¿\020\000\000\000\000\000\000\000\020\000\000"}, sin =
>>>>>>> {sin_family = 57540,
>>>>>>> sin_port = 49041, sin_addr = {s_addr = 16}, sin_zero =
>>>>>>> "\000\000\000\000\020\000\000"}, sin6 = {sin6_family = 57540, sin6_port =
>>>>>>> 49041,
>>>>>>> sin6_flowinfo = 16, sin6_addr = {in6_u = {u6_addr8 =
>>>>>>> "\000\000\000\000\020\000\000\000\001\000\000\000\001\000\000", u6_addr16 =
>>>>>>> {0, 0, 16, 0, 1, 0,
>>>>>>> 1, 0}, u6_addr32 = {0, 16, 1, 1}}}, sin6_scope_id =
>>>>>>> 12}}, id = -1080958468, proto = 4 '\004', send_flags = {f = 0 '\0',
>>>>>>> blst_imask = 0 '\0'}}
>>>>>>> c = 10 '\n'
>>>>>>> #17 0x08159a0f in handle_io (fm=0x83296b0, events=1, idx=-1) at
>>>>>>> tcp_read.c:1150
>>>>>>> ret = 4
>>>>>>> n = 4
>>>>>>> read_flags = 1
>>>>>>> con = (struct tcp_connection *) 0xb621af88
>>>>>>> s = 12
>>>>>>> resp = <value optimized out>
>>>>>>> t = <value optimized out>
>>>>>>> #18 0x0815d653 in tcp_receive_loop (unix_sock=35) at io_wait.h:1091
>>>>>>> No locals.
>>>>>>> ---Type <return> to continue, or q <return> to quit---
>>>>>>> #19 0x0812885b in tcp_init_children () at tcp_main.c:4819
>>>>>>> r = 3
>>>>>>> reader_fd_1 = 35
>>>>>>> pid = <value optimized out>
>>>>>>> si = <value optimized out>
>>>>>>> #20 0x080ad412 in main_loop () at main.c:1632
>>>>>>> i = 4
>>>>>>> pid = <value optimized out>
>>>>>>> si = (struct socket_info *) 0x0
>>>>>>> si_desc = "udp receiver child=3 sock=192.168.166.31:5060
>>>>>>> \000\221¿\002\000\000\000/\200\220\000\001\000\000\000\000\000H\000¨ä21¿\001\000\000\000°¶è026¶\000\000\000\000\000õ\030s2\b\002\000\000\000âf\b\000õ\001\000\000\000\000\000\000\000\001\000\000\000\000\000\000\000ÿÿÿÿ¸ä21¿"
>>>>>>> #21 0x080afbe7 in main (argc=3, argv=dwarf2_read_address: Corrupted
>>>>>>> DWARF expression.
>>>>>>> ) at main.c:2398
>>>>>>> cfg_stream = (FILE *) 0x8a2d008
>>>>>>> c = <value optimized out>
>>>>>>> r = 0
>>>>>>> tmp = 0xbf91e674
>>>>>>> "\212þ\221¿\233þ\221¿°þ\221¿»þ\221¿Ã\221¿ù1¿\020ÿ\221¿Dÿ\221¿Lÿ\221¿Wÿ\221¿]ÿ\221¿oÿ\221¿{ÿ\221¿\202ÿ\221
>>>>>>> tmp_len = 10492229
>>>>>>> port = 136192361
>>>>>>> proto = -1080957480
>>>>>>> ret = <value optimized out>
>>>>>>> seed = 21782655
>>>>>>> rfd = 4
>>>>>>> debug_save = <value optimized out>
>>>>>>> debug_flag = 0
>>>>>>> dont_fork_cnt = 0
>>>>>>> n_lst = <value optimized out>
>>>>>>> p = <value optimized out>
>>>>>>>
>>>>>>>
>>>>>>> Best Regards
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing listsr-users at lists.sip-router.orghttp://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
>>>>>>> Kamailio Advanced Training, Seattle, USA, Sep 23-26, 2012 - http://asipto.com/u/katu
>>>>>>> Kamailio Practical Workshop, Netherlands, Sep 10-12, 2012 - http://asipto.com/u/kpw
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing listsr-users at lists.sip-router.orghttp://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing
>>>>>> list
>>>>>> sr-users at lists.sip-router.org
>>>>>> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing listsr-users at lists.sip-router.orghttp://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
>>>>
>>>>
>>>> --
>>>> Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
>>>> Kamailio Advanced Training, Seattle, USA, Sep 23-26, 2012 - http://asipto.com/u/katu
>>>> Kamailio Practical Workshop, Netherlands, Sep 10-12, 2012 - http://asipto.com/u/kpw
>>>>
>>>>
>>>
>>> --
>>> Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
>>> Kamailio Advanced Training, Seattle, USA, Sep 23-26, 2012 - http://asipto.com/u/katu
>>> Kamailio Practical Workshop, Netherlands, Sep 10-12, 2012 - http://asipto.com/u/kpw
>>>
>>>
>>
>> --
>> Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
>> Kamailio Advanced Training, Seattle, USA, Sep 23-26, 2012 - http://asipto.com/u/katu
>> Kamailio Practical Workshop, Netherlands, Sep 10-12, 2012 - http://asipto.com/u/kpw
>>
>>
>
> --
> Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
> Kamailio Advanced Training, Seattle, USA, Sep 23-26, 2012 - http://asipto.com/u/katu
> Kamailio Practical Workshop, Netherlands, Sep 10-12, 2012 - http://asipto.com/u/kpw
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.sip-router.org/pipermail/sr-users/attachments/20120731/1d57b750/attachment-0001.htm>
More information about the sr-users
mailing list