[SR-Users] Core fm_realloc function

Daniel-Constantin Mierla miconda at gmail.com
Mon Jul 30 09:51:04 CEST 2012


Hello,

it is in my to-do list to investigate, so far I was not able to reproduce.

Vacation time together with other travelings makes a bit more slower 
process. Please add it also to the tracker:

http://sip-router.org/tracker/

A minor release for 3.3 branch is going to be soon, being there makes 
sure it will be reviewed to see if it applies.

Cheers,
Daniel

On 7/27/12 3:00 PM, Bruno Bresciani wrote:
> Hi Users,
>
> This is my ultimate try to ask for some help to debug the two cores 
> generated by fm_realloc() function. If somebody can give me a feedback 
> about my questions or doubts I will be very grateful, this user list 
> of kamailio always help me to solve my problems.
>
> Best Regards
>
>
> 2012/7/23 Bruno Bresciani <bruno.bresciani at gmail.com 
> <mailto:bruno.bresciani at gmail.com>>
>
>     Hi All,
>
>     I imagine that this core doesn't easy to replicate but somebody 
>     can help me to understand why it was generated? There is some way
>     to  prevent it?
>
>     Best Regards
>
>
>
>     2012/7/18 Renan Capaverde <renan.capaverde at digitro.com.br
>     <mailto:renan.capaverde at digitro.com.br>>
>
>         I am having the same problem on kamailio 3.1.6. Someone please
>         help.
>
>         Cheers,
>         Renan
>
>         Em 17/7/2012 14:26, Bruno Bresciani escreveu:
>>         Excuse-me Daniel, I forgot this information... Kamailio
>>         version 3.1.2
>>
>>         Best Regards
>>
>>
>>         2012/7/17 Daniel-Constantin Mierla <miconda at gmail.com
>>         <mailto:miconda at gmail.com>>
>>
>>             Hello,
>>
>>             send also the version 'kamailio -V' to match the proper
>>             sources.
>>
>>             Cheers,
>>             Daniel
>>
>>
>>             On 7/17/12 7:11 PM, Bruno Bresciani wrote:
>>>             Hi all,
>>>
>>>             I observed two at 0x08173bf0 in fm_realloc
>>>             (qm=0xb5f7d000, p=0xb61f0bc0, size=0) at mem/f_malloc.c:536.
>>>
>>>             I don't know exactly what may be caused this core
>>>             because the debug of kamailio is disabled, but looking
>>>             at backtrace (gdb) and source code I could notice a
>>>             relationship with TLS connection establishment
>>>             (handshake). Other important fact is that core was
>>>             generated by the kamalio itself with signal 6 (abort)
>>>             because it can't find the pointer to memory realloc.
>>>             I'd like to know if someone already observed this issue
>>>             and if my analyse above is correct?
>>>
>>>             Bellow is backtrace full with gdb:
>>>
>>>             Core was generated by
>>>             `/home2/local/kamailio/sbin/kamailio -P
>>>             /var/run/kamailio.pid'.
>>>             Program terminated with signal 6, Aborted.
>>>             #0  0x00f30402 in __kernel_vsyscall ()
>>>             (gdb)
>>>             (gdb)
>>>             (gdb)
>>>             (gdb)
>>>             (gdb) bt full
>>>             #0  0x00f30402 in __kernel_vsyscall ()
>>>             No symbol table info available.
>>>             #1  0x009fec10 in raise () from /lib/libc.so.6
>>>             No symbol table info available.
>>>             #2  0x00a00521 in abort () from /lib/libc.so.6
>>>             No symbol table info available.
>>>             #3  0x08173bf0 in fm_realloc (qm=0xb5f7d000,
>>>             p=0xb61f0bc0, size=0) at mem/f_malloc.c:536
>>>                     f = (struct fm_frag *) 0xb61f0bb8
>>>                     pf = <value optimized out>
>>>                     orig_size = 18
>>>             L=Ãb¬Ã2n = (struct fm_frag *) 0xb61f0bd2
>>>                     ptr = <value optimized out>
>>>                     hash = 23548
>>>             #4  0x0051a87f in ser_realloc (ptr=0xb61f0bc0, size=32)
>>>             at ../../mem/shm_mem.h:266
>>>             No locals.
>>>             #5  0x00460295 in OPENSSL_cleanse () from
>>>             /lib/libcrypto.so.6
>>>             No symbol table info available.
>>>             #6  0x004609f7 in CRYPTO_realloc () from /lib/libcrypto.so.6
>>>             No symbol table info available.
>>>             #7  0x003efb22 in sk_insert () from /lib/libcrypto.so.6
>>>             No symbol table info available.
>>>             #8  0x003efbaa in sk_push () from /lib/libcrypto.so.6
>>>             No symbol table info available.
>>>             #9  0x007c2ac3 in ssl_bytes_to_cipher_list () from
>>>             /lib/libssl.so.6
>>>             No symbol table info available.
>>>             #10 0x007aa9f3 in ssl3_get_client_hello () from
>>>             /lib/libssl.so.6
>>>             No symbol table info available.
>>>             #11 0x007ab815 in ssl3_accept () from /lib/libssl.so.6
>>>             No symbol table info available.
>>>             #12 0x007c1c3a in SSL_accept () from /lib/libssl.so.6
>>>             No symbol table info available.
>>>             #13 0x005278e3 in tls_accept (c=0xb621af88,
>>>             error=0xbf91dfc8) at tls_server.c:346
>>>                     ret = <value optimized out>
>>>                     ssl = (SSL *) 0xb61c1878
>>>                     cert = <value optimized out>
>>>                     tls_c = (struct tls_extra_data *) 0xb6208e58
>>>                     tls_log = <value optimized out>
>>>             #14 0x0052da8e in tls_read_f (c=0xb621af88,
>>>             flags=0xbf91e200) at tls_server.c:1028
>>>                     r = (struct tcp_req *) 0xb621affc
>>>                     bytes_free = 4095
>>>                     bytes_read = 70
>>>                     read_size = 4095
>>>                     ssl_error = 0
>>>                     ssl_read = 0
>>>                     ssl = (SSL *) 0xb61c1878
>>>                     rd_buf =
>>>             "\026\003\001\000A\001\000\000=\003\001Oÿ\v¡ywP\fkÃDÃ
>>>             0/\000\a\000\005\001\000ÃÃÃï000\000\026\0009\0008\0005\000\026\000\023\000\n\0003\0002\00
>>>             202i<ä<åþ\031ì234 \237ê030Y){lѶ"...06Ã'Ãr9ìë¸{[©\t\205\037\036ÿHPH5µ\t(é32Ã027æò036¾\"\rÔ¤YÃ215uº\027)ö000ñ215\vÃà éEa\023xéJ\206T\204CÃ
>>>             wr_buf =
>>>             "\027\003\001\003 w²'\016ѼÃ033ÃLÃ021Ã)Ã\220hnL¦\020Xê\236\200\tIhøSf\234Ã\rÃ
>>>             ?¯¼\024FÃ\200ª/\2264RÃaäIpXȬÃ035äQh\211Ã026ý4ÃU+\asr£/`¤Oû1AʹqUܿ·v±ü\205"...
>>>                     rd = {buf = 0xbf90dfa4 "\026\003\001", pos = 70,
>>>             used = 70, size = 65536}
>>>                     wr = {
>>>               buf = 0xbf8fdfa4
>>>             "\027\003\001\003 w²'\016ѼÃ033ÃLÃ021Ã)Ã\220hnL¦\020Xê\236\200\tIhøSf\234Ã\rÃ
>>>             ?¯¼\024FÃ\200ª/\2264RÃaäIpXȬÃ035äQh\211Ã026ý4ÃU+\asr£/`¤Oû1AʹqUܿ·v±ü\205"...,
>>>             pos = 0, used = 0,
>>>             size = 65536}
>>>                     tls_c = (struct tls_extra_data *) 0xb6208e58
>>>                     enc_rd_buf = (struct tls_rd_buf *) 0x0
>>>                     n = 0
>>>                     flush_flags = <value optimized out>
>>>                     err_src = 0x541a83 "TLS read:"
>>>                     x = <value optimized out>
>>>             #15 0x08158ade in tcp_read_headers (c=0xb621af88,
>>>             read_flags=0xbf91e200) at tcp_read.c:406
>>>                     bytes = -1
>>>                     remaining = <value optimized out>
>>>                     p = <value optimized out>
>>>                     r = (struct tcp_req *) 0xb621affc
>>>             #16 0x08158fd4 in tcp_read_req (con=0xb621af88,
>>>             bytes_read=0xbf91e204, read_flags=0xbf91e200) at
>>>             tcp_read.c:871
>>>                     bytes = <value optimized out>
>>>                     total_bytes = 0
>>>                     resp = <value optimized out>
>>>                     size = <value optimized out>
>>>                     req = (struct tcp_req *) 0xb621affc
>>>                     dst = {send_sock = 0x1, to = {s = {sa_family =
>>>             57540, sa_data =
>>>             "\221¿\020\000\000\000\000\000\000\000\020\000\000"},
>>>             sin = {sin_family = 57540,
>>>                   sin_port = 49041, sin_addr = {s_addr = 16},
>>>             sin_zero = "\000\000\000\000\020\000\000"}, sin6 =
>>>             {sin6_family = 57540, sin6_port = 49041,
>>>                   sin6_flowinfo = 16, sin6_addr = {in6_u = {u6_addr8
>>>             =
>>>             "\000\000\000\000\020\000\000\000\001\000\000\000\001\000\000",
>>>             u6_addr16 = {0, 0, 16, 0, 1, 0,
>>>                         1, 0}, u6_addr32 = {0, 16, 1, 1}}},
>>>             sin6_scope_id = 12}}, id = -1080958468, proto = 4
>>>             '\004', send_flags = {f = 0 '\0', blst_imask = 0 '\0'}}
>>>                     c = 10 '\n'
>>>             #17 0x08159a0f in handle_io (fm=0x83296b0, events=1,
>>>             idx=-1) at tcp_read.c:1150
>>>                     ret = 4
>>>                     n = 4
>>>                     read_flags = 1
>>>                     con = (struct tcp_connection *) 0xb621af88
>>>                     s = 12
>>>                     resp = <value optimized out>
>>>                     t = <value optimized out>
>>>             #18 0x0815d653 in tcp_receive_loop (unix_sock=35) at
>>>             io_wait.h:1091
>>>             No locals.
>>>             ---Type <return> to continue, or q <return> to quit---
>>>             #19 0x0812885b in tcp_init_children () at tcp_main.c:4819
>>>                     r = 3
>>>                     reader_fd_1 = 35
>>>                     pid = <value optimized out>
>>>                     si = <value optimized out>
>>>             #20 0x080ad412 in main_loop () at main.c:1632
>>>                     i = 4
>>>                     pid = <value optimized out>
>>>                     si = (struct socket_info *) 0x0
>>>                     si_desc = "udp receiver child=3
>>>             sock=192.168.166.31:5060
>>>             <http://192.168.166.31:5060>\000\221¿\002\000\000\000/\200\220\000\001\000\000\000\000\000H\000¨ä21¿\001\000\000\000°¶è026¶\000\000\000\000\000õ\030s2\b\002\000\000\000âf\b\000õ\001\000\000\000\000\000\000\000\001\000\000\000\000\000\000\000ÿÿÿÿ¸ä21¿"
>>>             #21 0x080afbe7 in main (argc=3,
>>>             argv=dwarf2_read_address: Corrupted DWARF expression.
>>>             ) at main.c:2398
>>>                     cfg_stream = (FILE *) 0x8a2d008
>>>                     c = <value optimized out>
>>>                     r = 0
>>>                     tmp = 0xbf91e674
>>>             "\212þ\221¿\233þ\221¿°þ\221¿»þ\221¿Ã\221¿ù1¿\020ÿ\221¿Dÿ\221¿Lÿ\221¿Wÿ\221¿]ÿ\221¿oÿ\221¿{ÿ\221¿\202ÿ\221
>>>             tmp_len = 10492229
>>>                     port = 136192361
>>>                     proto = -1080957480
>>>                     ret = <value optimized out>
>>>                     seed = 21782655
>>>                     rfd = 4
>>>                     debug_save = <value optimized out>
>>>                     debug_flag = 0
>>>                     dont_fork_cnt = 0
>>>                     n_lst = <value optimized out>
>>>                     p = <value optimized out>
>>>
>>>
>>>             Best Regards
>>>
>>>
>>>             _______________________________________________
>>>             SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
>>>             sr-users at lists.sip-router.org  <mailto:sr-users at lists.sip-router.org>
>>>             http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
>>
>>             -- 
>>             Daniel-Constantin Mierla -http://www.asipto.com
>>             http://twitter.com/#!/miconda  <http://twitter.com/#%21/miconda>  -http://www.linkedin.com/in/miconda
>>             Kamailio Advanced Training, Seattle, USA, Sep 23-26, 2012 -http://asipto.com/u/katu
>>             Kamailio Practical Workshop, Netherlands, Sep 10-12, 2012 -http://asipto.com/u/kpw
>>
>>
>>
>>
>>         _______________________________________________
>>         SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
>>         sr-users at lists.sip-router.org  <mailto:sr-users at lists.sip-router.org>
>>         http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
>
>
>
>         _______________________________________________
>         SIP Express Router (SER) and Kamailio (OpenSER) - sr-users
>         mailing list
>         sr-users at lists.sip-router.org
>         <mailto:sr-users at lists.sip-router.org>
>         http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
>
>
>
>
>
> _______________________________________________
> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
> sr-users at lists.sip-router.org
> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users

-- 
Daniel-Constantin Mierla - http://www.asipto.com
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Kamailio Advanced Training, Seattle, USA, Sep 23-26, 2012 - http://asipto.com/u/katu
Kamailio Practical Workshop, Netherlands, Sep 10-12, 2012 - http://asipto.com/u/kpw

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.sip-router.org/pipermail/sr-users/attachments/20120730/b0299433/attachment-0001.htm>


More information about the sr-users mailing list