Excuse-me Daniel, I forgot this information... Kamailio version 3.1.2

Best Regards

2012/7/17 Daniel-Constantin Mierla miconda@gmail.com

Hello,

send also the version 'kamailio -V' to match the proper sources.

Cheers, Daniel

On 7/17/12 7:11 PM, Bruno Bresciani wrote:

Hi all,

I observed two at 0x08173bf0 in fm_realloc (qm=0xb5f7d000, p=0xb61f0bc0, size=0) at mem/f_malloc.c:536.

I don't know exactly what may be caused this core because the debug of kamailio is disabled, but looking at backtrace (gdb) and source code I could notice a relationship with TLS connection establishment (handshake). Other important fact is that core was generated by the kamalio itself with signal 6 (abort) because it can't find the pointer to memory realloc. I'd like to know if someone already observed this issue and if my analyse above is correct?

Bellow is backtrace full with gdb:

Core was generated by `/home2/local/kamailio/sbin/kamailio -P /var/run/kamailio.pid'. Program terminated with signal 6, Aborted. #0 0x00f30402 in __kernel_vsyscall () (gdb) (gdb) (gdb) (gdb) (gdb) bt full #0 0x00f30402 in __kernel_vsyscall () No symbol table info available. #1 0x009fec10 in raise () from /lib/libc.so.6 No symbol table info available. #2 0x00a00521 in abort () from /lib/libc.so.6 No symbol table info available. #3 0x08173bf0 in fm_realloc (qm=0xb5f7d000, p=0xb61f0bc0, size=0) at mem/f_malloc.c:536 f = (struct fm_frag *) 0xb61f0bb8 pf = <value optimized out> orig_size = 18 L=Ãb¬Ã2n = (struct fm_frag *) 0xb61f0bd2 ptr = <value optimized out> hash = 23548 #4 0x0051a87f in ser_realloc (ptr=0xb61f0bc0, size=32) at ../../mem/shm_mem.h:266 No locals. #5 0x00460295 in OPENSSL_cleanse () from /lib/libcrypto.so.6 No symbol table info available. #6 0x004609f7 in CRYPTO_realloc () from /lib/libcrypto.so.6 No symbol table info available. #7 0x003efb22 in sk_insert () from /lib/libcrypto.so.6 No symbol table info available. #8 0x003efbaa in sk_push () from /lib/libcrypto.so.6 No symbol table info available. #9 0x007c2ac3 in ssl_bytes_to_cipher_list () from /lib/libssl.so.6 No symbol table info available. #10 0x007aa9f3 in ssl3_get_client_hello () from /lib/libssl.so.6 No symbol table info available. #11 0x007ab815 in ssl3_accept () from /lib/libssl.so.6 No symbol table info available. #12 0x007c1c3a in SSL_accept () from /lib/libssl.so.6 No symbol table info available. #13 0x005278e3 in tls_accept (c=0xb621af88, error=0xbf91dfc8) at tls_server.c:346 ret = <value optimized out> ssl = (SSL *) 0xb61c1878 cert = <value optimized out> tls_c = (struct tls_extra_data *) 0xb6208e58 tls_log = <value optimized out> #14 0x0052da8e in tls_read_f (c=0xb621af88, flags=0xbf91e200) at tls_server.c:1028 r = (struct tcp_req *) 0xb621affc bytes_free = 4095 bytes_read = 70 read_size = 4095 ssl_error = 0 ssl_read = 0 ssl = (SSL *) 0xb61c1878 rd_buf = "\026\003\001\000A\001\000\000=\003\001Oÿ\v¡ywP\fkÃDÃ

0/\000\a\000\005\001\000ÃÃÃï000\000\026\0009\0008\0005\000\026\000\023\000\n\0003\0002\00 202i<ä<åþ\031ì234 \237ê030Y){lѶ"...06Ã'Ãr9ìë¸{[©\t\205\037\036ÿHPH5µ\t(é32Ã027æò036¾"\rÔ¤YÃ215uº\027)ö000ñ215\vÃà éEa\023xéJ\206T\204Cà wr_buf = "\027\003\001\003 w²'\016ѼÃ033ÃLÃ021Ã)Ã\220hnL¦\020Xê\236\200\tIhøSf\234Ã\rÃ

?¯¼\024FÃ\200ª/\2264RÃaäIpXȬÃ035äQh\211Ã026ý4ÃU+\asr£/`¤Oû1AʹqUܿ·v±ü\205"... rd = {buf = 0xbf90dfa4 "\026\003\001", pos = 70, used = 70, size = 65536} wr = { buf = 0xbf8fdfa4 "\027\003\001\003 w²'\016ѼÃ033ÃLÃ021Ã)Ã\220hnL¦\020Xê\236\200\tIhøSf\234Ã\rà ?¯¼\024FÃ\200ª/\2264RÃaäIpXȬÃ035äQh\211Ã026ý4ÃU+\asr£/`¤Oû1AʹqUܿ·v±ü\205"..., pos = 0, used = 0, size = 65536} tls_c = (struct tls_extra_data *) 0xb6208e58 enc_rd_buf = (struct tls_rd_buf *) 0x0 n = 0 flush_flags = <value optimized out> err_src = 0x541a83 "TLS read:" x = <value optimized out> #15 0x08158ade in tcp_read_headers (c=0xb621af88, read_flags=0xbf91e200) at tcp_read.c:406 bytes = -1 remaining = <value optimized out> p = <value optimized out> r = (struct tcp_req *) 0xb621affc #16 0x08158fd4 in tcp_read_req (con=0xb621af88, bytes_read=0xbf91e204, read_flags=0xbf91e200) at tcp_read.c:871 bytes = <value optimized out> total_bytes = 0 resp = <value optimized out> size = <value optimized out> req = (struct tcp_req *) 0xb621affc dst = {send_sock = 0x1, to = {s = {sa_family = 57540, sa_data = "\221¿\020\000\000\000\000\000\000\000\020\000\000"}, sin = {sin_family = 57540, sin_port = 49041, sin_addr = {s_addr = 16}, sin_zero = "\000\000\000\000\020\000\000"}, sin6 = {sin6_family = 57540, sin6_port = 49041, sin6_flowinfo = 16, sin6_addr = {in6_u = {u6_addr8 = "\000\000\000\000\020\000\000\000\001\000\000\000\001\000\000", u6_addr16 = {0, 0, 16, 0, 1, 0, 1, 0}, u6_addr32 = {0, 16, 1, 1}}}, sin6_scope_id = 12}}, id = -1080958468, proto = 4 '\004', send_flags = {f = 0 '\0', blst_imask = 0 '\0'}} c = 10 '\n' #17 0x08159a0f in handle_io (fm=0x83296b0, events=1, idx=-1) at tcp_read.c:1150 ret = 4 n = 4 read_flags = 1 con = (struct tcp_connection *) 0xb621af88 s = 12 resp = <value optimized out> t = <value optimized out> #18 0x0815d653 in tcp_receive_loop (unix_sock=35) at io_wait.h:1091 No locals. ---Type <return> to continue, or q <return> to quit--- #19 0x0812885b in tcp_init_children () at tcp_main.c:4819 r = 3 reader_fd_1 = 35 pid = <value optimized out> si = <value optimized out> #20 0x080ad412 in main_loop () at main.c:1632 i = 4 pid = <value optimized out> si = (struct socket_info *) 0x0 si_desc = "udp receiver child=3 sock=192.168.166.31:5060 \000\221¿\002\000\000\000/\200\220\000\001\000\000\000\000\000H\000¨ä21¿\001\000\000\000°¶è026¶\000\000\000\000\000õ\030s2\b\002\000\000\000âf\b\000õ\001\000\000\000\000\000\000\000\001\000\000\000\000\000\000\000ÿÿÿÿ¸ä21¿" #21 0x080afbe7 in main (argc=3, argv=dwarf2_read_address: Corrupted DWARF expression. ) at main.c:2398 cfg_stream = (FILE *) 0x8a2d008 c = <value optimized out> r = 0 tmp = 0xbf91e674 "\212þ\221¿\233þ\221¿°þ\221¿»þ\221¿Ã\221¿ù1¿\020ÿ\221¿Dÿ\221¿Lÿ\221¿Wÿ\221¿]ÿ\221¿oÿ\221¿{ÿ\221¿\202ÿ\221 tmp_len = 10492229 port = 136192361 proto = -1080957480 ret = <value optimized out> seed = 21782655 rfd = 4 debug_save = <value optimized out> debug_flag = 0 dont_fork_cnt = 0 n_lst = <value optimized out> p = <value optimized out>

Best Regards

---

SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing listsr-users@lists.sip-router.orghttp://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users

-- Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Kamailio Advanced Training, Seattle, USA, Sep 23-26, 2012 - http://asipto.com/u/katu Kamailio Practical Workshop, Netherlands, Sep 10-12, 2012 - http://asipto.com/u/kpw

Hi All,

I imagine that this core doesn't easy to replicate but somebody can help me to understand why it was generated? There is some way to prevent it?

Best Regards

2012/7/18 Renan Capaverde renan.capaverde@digitro.com.br

I am having the same problem on kamailio 3.1.6. Someone please help.

Cheers, Renan

Em 17/7/2012 14:26, Bruno Bresciani escreveu:

Excuse-me Daniel, I forgot this information... Kamailio version 3.1.2

Best Regards

2012/7/17 Daniel-Constantin Mierla miconda@gmail.com

...

Hello,

send also the version 'kamailio -V' to match the proper sources.

Cheers, Daniel

On 7/17/12 7:11 PM, Bruno Bresciani wrote:

Hi all,

I observed two at 0x08173bf0 in fm_realloc (qm=0xb5f7d000, p=0xb61f0bc0, size=0) at mem/f_malloc.c:536.

I don't know exactly what may be caused this core because the debug of kamailio is disabled, but looking at backtrace (gdb) and source code I could notice a relationship with TLS connection establishment (handshake). Other important fact is that core was generated by the kamalio itself with signal 6 (abort) because it can't find the pointer to memory realloc. I'd like to know if someone already observed this issue and if my analyse above is correct?

Bellow is backtrace full with gdb:

Core was generated by `/home2/local/kamailio/sbin/kamailio -P /var/run/kamailio.pid'. Program terminated with signal 6, Aborted. #0 0x00f30402 in __kernel_vsyscall () (gdb) (gdb) (gdb) (gdb) (gdb) bt full #0 0x00f30402 in __kernel_vsyscall () No symbol table info available. #1 0x009fec10 in raise () from /lib/libc.so.6 No symbol table info available. #2 0x00a00521 in abort () from /lib/libc.so.6 No symbol table info available. #3 0x08173bf0 in fm_realloc (qm=0xb5f7d000, p=0xb61f0bc0, size=0) at mem/f_malloc.c:536 f = (struct fm_frag *) 0xb61f0bb8 pf = <value optimized out> orig_size = 18 L=Ãb¬Ã2n = (struct fm_frag *) 0xb61f0bd2 ptr = <value optimized out> hash = 23548 #4 0x0051a87f in ser_realloc (ptr=0xb61f0bc0, size=32) at ../../mem/shm_mem.h:266 No locals. #5 0x00460295 in OPENSSL_cleanse () from /lib/libcrypto.so.6 No symbol table info available. #6 0x004609f7 in CRYPTO_realloc () from /lib/libcrypto.so.6 No symbol table info available. #7 0x003efb22 in sk_insert () from /lib/libcrypto.so.6 No symbol table info available. #8 0x003efbaa in sk_push () from /lib/libcrypto.so.6 No symbol table info available. #9 0x007c2ac3 in ssl_bytes_to_cipher_list () from /lib/libssl.so.6 No symbol table info available. #10 0x007aa9f3 in ssl3_get_client_hello () from /lib/libssl.so.6 No symbol table info available. #11 0x007ab815 in ssl3_accept () from /lib/libssl.so.6 No symbol table info available. #12 0x007c1c3a in SSL_accept () from /lib/libssl.so.6 No symbol table info available. #13 0x005278e3 in tls_accept (c=0xb621af88, error=0xbf91dfc8) at tls_server.c:346 ret = <value optimized out> ssl = (SSL *) 0xb61c1878 cert = <value optimized out> tls_c = (struct tls_extra_data *) 0xb6208e58 tls_log = <value optimized out> #14 0x0052da8e in tls_read_f (c=0xb621af88, flags=0xbf91e200) at tls_server.c:1028 r = (struct tcp_req *) 0xb621affc bytes_free = 4095 bytes_read = 70 read_size = 4095 ssl_error = 0 ssl_read = 0 ssl = (SSL *) 0xb61c1878 rd_buf = "\026\003\001\000A\001\000\000=\003\001Oÿ\v¡ywP\fkÃDÃ

0/\000\a\000\005\001\000ÃÃÃï000\000\026\0009\0008\0005\000\026\000\023\000\n\0003\0002\00 202i<ä<åþ\031ì234 \237ê030Y){lѶ"...06Ã'Ãr9ìë¸{[©\t\205\037\036ÿHPH5µ\t(é32Ã027æò036¾"\rÔ¤YÃ215uº\027)ö000ñ215\vÃà éEa\023xéJ\206T\204Cà wr_buf = "\027\003\001\003 w²'\016ѼÃ033ÃLÃ021Ã)Ã\220hnL¦\020Xê\236\200\tIhøSf\234Ã\rÃ

?¯¼\024FÃ\200ª/\2264RÃaäIpXȬÃ035äQh\211Ã026ý4ÃU+\asr£/`¤Oû1AʹqUܿ·v±ü\205"... rd = {buf = 0xbf90dfa4 "\026\003\001", pos = 70, used = 70, size = 65536} wr = { buf = 0xbf8fdfa4 "\027\003\001\003 w²'\016ѼÃ033ÃLÃ021Ã)Ã\220hnL¦\020Xê\236\200\tIhøSf\234Ã\rà ?¯¼\024FÃ\200ª/\2264RÃaäIpXȬÃ035äQh\211Ã026ý4ÃU+\asr£/`¤Oû1AʹqUܿ·v±ü\205"..., pos = 0, used = 0, size = 65536} tls_c = (struct tls_extra_data *) 0xb6208e58 enc_rd_buf = (struct tls_rd_buf *) 0x0 n = 0 flush_flags = <value optimized out> err_src = 0x541a83 "TLS read:" x = <value optimized out> #15 0x08158ade in tcp_read_headers (c=0xb621af88, read_flags=0xbf91e200) at tcp_read.c:406 bytes = -1 remaining = <value optimized out> p = <value optimized out> r = (struct tcp_req *) 0xb621affc #16 0x08158fd4 in tcp_read_req (con=0xb621af88, bytes_read=0xbf91e204, read_flags=0xbf91e200) at tcp_read.c:871 bytes = <value optimized out> total_bytes = 0 resp = <value optimized out> size = <value optimized out> req = (struct tcp_req *) 0xb621affc dst = {send_sock = 0x1, to = {s = {sa_family = 57540, sa_data = "\221¿\020\000\000\000\000\000\000\000\020\000\000"}, sin = {sin_family = 57540, sin_port = 49041, sin_addr = {s_addr = 16}, sin_zero = "\000\000\000\000\020\000\000"}, sin6 = {sin6_family = 57540, sin6_port = 49041, sin6_flowinfo = 16, sin6_addr = {in6_u = {u6_addr8 = "\000\000\000\000\020\000\000\000\001\000\000\000\001\000\000", u6_addr16 = {0, 0, 16, 0, 1, 0, 1, 0}, u6_addr32 = {0, 16, 1, 1}}}, sin6_scope_id = 12}}, id = -1080958468, proto = 4 '\004', send_flags = {f = 0 '\0', blst_imask = 0 '\0'}} c = 10 '\n' #17 0x08159a0f in handle_io (fm=0x83296b0, events=1, idx=-1) at tcp_read.c:1150 ret = 4 n = 4 read_flags = 1 con = (struct tcp_connection *) 0xb621af88 s = 12 resp = <value optimized out> t = <value optimized out> #18 0x0815d653 in tcp_receive_loop (unix_sock=35) at io_wait.h:1091 No locals. ---Type <return> to continue, or q <return> to quit--- #19 0x0812885b in tcp_init_children () at tcp_main.c:4819 r = 3 reader_fd_1 = 35 pid = <value optimized out> si = <value optimized out> #20 0x080ad412 in main_loop () at main.c:1632 i = 4 pid = <value optimized out> si = (struct socket_info *) 0x0 si_desc = "udp receiver child=3 sock=192.168.166.31:5060 \000\221¿\002\000\000\000/\200\220\000\001\000\000\000\000\000H\000¨ä21¿\001\000\000\000°¶è026¶\000\000\000\000\000õ\030s2\b\002\000\000\000âf\b\000õ\001\000\000\000\000\000\000\000\001\000\000\000\000\000\000\000ÿÿÿÿ¸ä21¿" #21 0x080afbe7 in main (argc=3, argv=dwarf2_read_address: Corrupted DWARF expression. ) at main.c:2398 cfg_stream = (FILE *) 0x8a2d008 c = <value optimized out> r = 0 tmp = 0xbf91e674 "\212þ\221¿\233þ\221¿°þ\221¿»þ\221¿Ã\221¿ù1¿\020ÿ\221¿Dÿ\221¿Lÿ\221¿Wÿ\221¿]ÿ\221¿oÿ\221¿{ÿ\221¿\202ÿ\221 tmp_len = 10492229 port = 136192361 proto = -1080957480 ret = <value optimized out> seed = 21782655 rfd = 4 debug_save = <value optimized out> debug_flag = 0 dont_fork_cnt = 0 n_lst = <value optimized out> p = <value optimized out>

Best Regards

---

SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing listsr-users@lists.sip-router.orghttp://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users

-- Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Kamailio Advanced Training, Seattle, USA, Sep 23-26, 2012 - http://asipto.com/u/katu Kamailio Practical Workshop, Netherlands, Sep 10-12, 2012 - http://asipto.com/u/kpw

---

SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing listsr-users@lists.sip-router.orghttp://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users

---

SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users

Hello,

it is in my to-do list to investigate, so far I was not able to reproduce.

Vacation time together with other travelings makes a bit more slower process. Please add it also to the tracker:

http://sip-router.org/tracker/

A minor release for 3.3 branch is going to be soon, being there makes sure it will be reviewed to see if it applies.

Cheers, Daniel

On 7/27/12 3:00 PM, Bruno Bresciani wrote:

Hi Users,

This is my ultimate try to ask for some help to debug the two cores generated by fm_realloc() function. If somebody can give me a feedback about my questions or doubts I will be very grateful, this user list of kamailio always help me to solve my problems.

Best Regards

2012/7/23 Bruno Bresciani <bruno.bresciani@gmail.com mailto:bruno.bresciani@gmail.com>

Hi All,

I imagine that this core doesn't easy to replicate but somebody 
can help me to understand why it was generated? There is some way
to  prevent it?

Best Regards



2012/7/18 Renan Capaverde <renan.capaverde@digitro.com.br
<mailto:renan.capaverde@digitro.com.br>>

    I am having the same problem on kamailio 3.1.6. Someone please
    help.

    Cheers,
    Renan

    Em 17/7/2012 14:26, Bruno Bresciani escreveu:

...

    Excuse-me Daniel, I forgot this information... Kamailio
    version 3.1.2

    Best Regards


    2012/7/17 Daniel-Constantin Mierla <miconda@gmail.com
    <mailto:miconda@gmail.com>>

        Hello,

        send also the version 'kamailio -V' to match the proper
        sources.

        Cheers,
        Daniel


        On 7/17/12 7:11 PM, Bruno Bresciani wrote:

...

        Hi all,

        I observed two at 0x08173bf0 in fm_realloc
        (qm=0xb5f7d000, p=0xb61f0bc0, size=0) at mem/f_malloc.c:536.

        I don't know exactly what may be caused this core
        because the debug of kamailio is disabled, but looking
        at backtrace (gdb) and source code I could notice a
        relationship with TLS connection establishment
        (handshake). Other important fact is that core was
        generated by the kamalio itself with signal 6 (abort)
        because it can't find the pointer to memory realloc.
        I'd like to know if someone already observed this issue
        and if my analyse above is correct?

        Bellow is backtrace full with gdb:

        Core was generated by
        `/home2/local/kamailio/sbin/kamailio -P
        /var/run/kamailio.pid'.
        Program terminated with signal 6, Aborted.
        #0  0x00f30402 in __kernel_vsyscall ()
        (gdb)
        (gdb)
        (gdb)
        (gdb)
        (gdb) bt full
        #0  0x00f30402 in __kernel_vsyscall ()
        No symbol table info available.
        #1  0x009fec10 in raise () from /lib/libc.so.6
        No symbol table info available.
        #2  0x00a00521 in abort () from /lib/libc.so.6
        No symbol table info available.
        #3  0x08173bf0 in fm_realloc (qm=0xb5f7d000,
        p=0xb61f0bc0, size=0) at mem/f_malloc.c:536
                f = (struct fm_frag *) 0xb61f0bb8
                pf = <value optimized out>
                orig_size = 18
        L=Ãb¬Ã2n = (struct fm_frag *) 0xb61f0bd2
                ptr = <value optimized out>
                hash = 23548
        #4  0x0051a87f in ser_realloc (ptr=0xb61f0bc0, size=32)
        at ../../mem/shm_mem.h:266
        No locals.
        #5  0x00460295 in OPENSSL_cleanse () from
        /lib/libcrypto.so.6
        No symbol table info available.
        #6  0x004609f7 in CRYPTO_realloc () from /lib/libcrypto.so.6
        No symbol table info available.
        #7  0x003efb22 in sk_insert () from /lib/libcrypto.so.6
        No symbol table info available.
        #8  0x003efbaa in sk_push () from /lib/libcrypto.so.6
        No symbol table info available.
        #9  0x007c2ac3 in ssl_bytes_to_cipher_list () from
        /lib/libssl.so.6
        No symbol table info available.
        #10 0x007aa9f3 in ssl3_get_client_hello () from
        /lib/libssl.so.6
        No symbol table info available.
        #11 0x007ab815 in ssl3_accept () from /lib/libssl.so.6
        No symbol table info available.
        #12 0x007c1c3a in SSL_accept () from /lib/libssl.so.6
        No symbol table info available.
        #13 0x005278e3 in tls_accept (c=0xb621af88,
        error=0xbf91dfc8) at tls_server.c:346
                ret = <value optimized out>
                ssl = (SSL *) 0xb61c1878
                cert = <value optimized out>
                tls_c = (struct tls_extra_data *) 0xb6208e58
                tls_log = <value optimized out>
        #14 0x0052da8e in tls_read_f (c=0xb621af88,
        flags=0xbf91e200) at tls_server.c:1028
                r = (struct tcp_req *) 0xb621affc
                bytes_free = 4095
                bytes_read = 70
                read_size = 4095
                ssl_error = 0
                ssl_read = 0
                ssl = (SSL *) 0xb61c1878
                rd_buf =
        "\026\003\001\000A\001\000\000=\003\001Oÿ\v¡ywP\fkÃDÃ
        0/\000\a\000\005\001\000ÃÃÃï000\000\026\0009\0008\0005\000\026\000\023\000\n\0003\0002\00
        202i<ä<åþ\031ì234 \237ê030Y){lѶ"...06Ã'Ãr9ìë¸{[©\t\205\037\036ÿHPH5µ\t(é32Ã027æò036¾\"\rÔ¤YÃ215uº\027)ö000ñ215\vÃà éEa\023xéJ\206T\204CÃ
        wr_buf =
        "\027\003\001\003 w²'\016ѼÃ033ÃLÃ021Ã)Ã\220hnL¦\020Xê\236\200\tIhøSf\234Ã\rÃ
        ?¯¼\024FÃ\200ª/\2264RÃaäIpXȬÃ035äQh\211Ã026ý4ÃU+\asr£/`¤Oû1AʹqUܿ·v±ü\205"...
                rd = {buf = 0xbf90dfa4 "\026\003\001", pos = 70,
        used = 70, size = 65536}
                wr = {
          buf = 0xbf8fdfa4
        "\027\003\001\003 w²'\016ѼÃ033ÃLÃ021Ã)Ã\220hnL¦\020Xê\236\200\tIhøSf\234Ã\rÃ
        ?¯¼\024FÃ\200ª/\2264RÃaäIpXȬÃ035äQh\211Ã026ý4ÃU+\asr£/`¤Oû1AʹqUܿ·v±ü\205"...,
        pos = 0, used = 0,
        size = 65536}
                tls_c = (struct tls_extra_data *) 0xb6208e58
                enc_rd_buf = (struct tls_rd_buf *) 0x0
                n = 0
                flush_flags = <value optimized out>
                err_src = 0x541a83 "TLS read:"
                x = <value optimized out>
        #15 0x08158ade in tcp_read_headers (c=0xb621af88,
        read_flags=0xbf91e200) at tcp_read.c:406
                bytes = -1
                remaining = <value optimized out>
                p = <value optimized out>
                r = (struct tcp_req *) 0xb621affc
        #16 0x08158fd4 in tcp_read_req (con=0xb621af88,
        bytes_read=0xbf91e204, read_flags=0xbf91e200) at
        tcp_read.c:871
                bytes = <value optimized out>
                total_bytes = 0
                resp = <value optimized out>
                size = <value optimized out>
                req = (struct tcp_req *) 0xb621affc
                dst = {send_sock = 0x1, to = {s = {sa_family =
        57540, sa_data =
        "\221¿\020\000\000\000\000\000\000\000\020\000\000"},
        sin = {sin_family = 57540,
              sin_port = 49041, sin_addr = {s_addr = 16},
        sin_zero = "\000\000\000\000\020\000\000"}, sin6 =
        {sin6_family = 57540, sin6_port = 49041,
              sin6_flowinfo = 16, sin6_addr = {in6_u = {u6_addr8
        =
        "\000\000\000\000\020\000\000\000\001\000\000\000\001\000\000",
        u6_addr16 = {0, 0, 16, 0, 1, 0,
                    1, 0}, u6_addr32 = {0, 16, 1, 1}}},
        sin6_scope_id = 12}}, id = -1080958468, proto = 4
        '\004', send_flags = {f = 0 '\0', blst_imask = 0 '\0'}}
                c = 10 '\n'
        #17 0x08159a0f in handle_io (fm=0x83296b0, events=1,
        idx=-1) at tcp_read.c:1150
                ret = 4
                n = 4
                read_flags = 1
                con = (struct tcp_connection *) 0xb621af88
                s = 12
                resp = <value optimized out>
                t = <value optimized out>
        #18 0x0815d653 in tcp_receive_loop (unix_sock=35) at
        io_wait.h:1091
        No locals.
        ---Type <return> to continue, or q <return> to quit---
        #19 0x0812885b in tcp_init_children () at tcp_main.c:4819
                r = 3
                reader_fd_1 = 35
                pid = <value optimized out>
                si = <value optimized out>
        #20 0x080ad412 in main_loop () at main.c:1632
                i = 4
                pid = <value optimized out>
                si = (struct socket_info *) 0x0
                si_desc = "udp receiver child=3
        sock=192.168.166.31:5060
        <http://192.168.166.31:5060>\000\221¿\002\000\000\000/\200\220\000\001\000\000\000\000\000H\000¨ä21¿\001\000\000\000°¶è026¶\000\000\000\000\000õ\030s2\b\002\000\000\000âf\b\000õ\001\000\000\000\000\000\000\000\001\000\000\000\000\000\000\000ÿÿÿÿ¸ä21¿"
        #21 0x080afbe7 in main (argc=3,
        argv=dwarf2_read_address: Corrupted DWARF expression.
        ) at main.c:2398
                cfg_stream = (FILE *) 0x8a2d008
                c = <value optimized out>
                r = 0
                tmp = 0xbf91e674
        "\212þ\221¿\233þ\221¿°þ\221¿»þ\221¿Ã\221¿ù1¿\020ÿ\221¿Dÿ\221¿Lÿ\221¿Wÿ\221¿]ÿ\221¿oÿ\221¿{ÿ\221¿\202ÿ\221
        tmp_len = 10492229
                port = 136192361
                proto = -1080957480
                ret = <value optimized out>
                seed = 21782655
                rfd = 4
                debug_save = <value optimized out>
                debug_flag = 0
                dont_fork_cnt = 0
                n_lst = <value optimized out>
                p = <value optimized out>


        Best Regards


        _______________________________________________
        SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
        sr-users@lists.sip-router.org  <mailto:sr-users@lists.sip-router.org>
        http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users

        -- 
        Daniel-Constantin Mierla -http://www.asipto.com
        http://twitter.com/#!/miconda  <http://twitter.com/#%21/miconda>  -http://www.linkedin.com/in/miconda
        Kamailio Advanced Training, Seattle, USA, Sep 23-26, 2012 -http://asipto.com/u/katu
        Kamailio Practical Workshop, Netherlands, Sep 10-12, 2012 -http://asipto.com/u/kpw




    _______________________________________________
    SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
    sr-users@lists.sip-router.org  <mailto:sr-users@lists.sip-router.org>
    http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users

    _______________________________________________
    SIP Express Router (SER) and Kamailio (OpenSER) - sr-users
    mailing list
    sr-users@lists.sip-router.org
    <mailto:sr-users@lists.sip-router.org>
    http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users

---

SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users

Hello,

can you provide output of 'kamailio -V'? Did you enable f_malloc instead of q_malloc which was default in 3.1?

Cheers, Daniel

On 7/30/12 1:44 PM, Bruno Bresciani wrote:

Thank's for feedback Daniel,

This core is add to the tracker with ID 247...

Best Regards

2012/7/30 Daniel-Constantin Mierla <miconda@gmail.com mailto:miconda@gmail.com>

Hello,

it is in my to-do list to investigate, so far I was not able to
reproduce.

Vacation time together with other travelings makes a bit more
slower process. Please add it also to the tracker:

http://sip-router.org/tracker/

A minor release for 3.3 branch is going to be soon, being there
makes sure it will be reviewed to see if it applies.

Cheers,
Daniel


On 7/27/12 3:00 PM, Bruno Bresciani wrote:

...

Hi Users,

This is my ultimate try to ask for some help to debug the two
cores generated by fm_realloc() function. If somebody can give me
a feedback about my questions or doubts I will be very grateful,
this user list of kamailio always help me to solve my problems.

Best Regards


2012/7/23 Bruno Bresciani <bruno.bresciani@gmail.com
<mailto:bruno.bresciani@gmail.com>>

    Hi All,

    I imagine that this core doesn't easy to replicate but
    somebody  can help me to understand why it was generated?
    There is some way to  prevent it?

    Best Regards



    2012/7/18 Renan Capaverde <renan.capaverde@digitro.com.br
    <mailto:renan.capaverde@digitro.com.br>>

        I am having the same problem on kamailio 3.1.6. Someone
        please help.

        Cheers,
        Renan

        Em 17/7/2012 14:26, Bruno Bresciani escreveu:

...

        Excuse-me Daniel, I forgot this information... Kamailio
        version 3.1.2

        Best Regards


        2012/7/17 Daniel-Constantin Mierla <miconda@gmail.com
        <mailto:miconda@gmail.com>>

            Hello,

            send also the version 'kamailio -V' to match the
            proper sources.

            Cheers,
            Daniel


            On 7/17/12 7:11 PM, Bruno Bresciani wrote:

...

            Hi all,

            I observed two at 0x08173bf0 in fm_realloc
            (qm=0xb5f7d000, p=0xb61f0bc0, size=0) at
            mem/f_malloc.c:536.

            I don't know exactly what may be caused this core
            because the debug of kamailio is disabled, but
            looking at backtrace (gdb) and source code I could
            notice a relationship with TLS connection
            establishment (handshake). Other important fact is
            that core was generated by the kamalio itself with
            signal 6 (abort) because it can't find the pointer
            to memory realloc.
            I'd like to know if someone already observed this
            issue and if my analyse above is correct?

            Bellow is backtrace full with gdb:

            Core was generated by
            `/home2/local/kamailio/sbin/kamailio -P
            /var/run/kamailio.pid'.
            Program terminated with signal 6, Aborted.
            #0  0x00f30402 in __kernel_vsyscall ()
            (gdb)
            (gdb)
            (gdb)
            (gdb)
            (gdb) bt full
            #0  0x00f30402 in __kernel_vsyscall ()
            No symbol table info available.
            #1  0x009fec10 in raise () from /lib/libc.so.6
            No symbol table info available.
            #2  0x00a00521 in abort () from /lib/libc.so.6
            No symbol table info available.
            #3  0x08173bf0 in fm_realloc (qm=0xb5f7d000,
            p=0xb61f0bc0, size=0) at mem/f_malloc.c:536
                    f = (struct fm_frag *) 0xb61f0bb8
                    pf = <value optimized out>
                    orig_size = 18
            L=Ãb¬Ã2n = (struct fm_frag *) 0xb61f0bd2
                    ptr = <value optimized out>
                    hash = 23548
            #4  0x0051a87f in ser_realloc (ptr=0xb61f0bc0,
            size=32) at ../../mem/shm_mem.h:266
            No locals.
            #5  0x00460295 in OPENSSL_cleanse () from
            /lib/libcrypto.so.6
            No symbol table info available.
            #6  0x004609f7 in CRYPTO_realloc () from
            /lib/libcrypto.so.6
            No symbol table info available.
            #7  0x003efb22 in sk_insert () from /lib/libcrypto.so.6
            No symbol table info available.
            #8  0x003efbaa in sk_push () from /lib/libcrypto.so.6
            No symbol table info available.
            #9  0x007c2ac3 in ssl_bytes_to_cipher_list () from
            /lib/libssl.so.6
            No symbol table info available.
            #10 0x007aa9f3 in ssl3_get_client_hello () from
            /lib/libssl.so.6
            No symbol table info available.
            #11 0x007ab815 in ssl3_accept () from /lib/libssl.so.6
            No symbol table info available.
            #12 0x007c1c3a in SSL_accept () from /lib/libssl.so.6
            No symbol table info available.
            #13 0x005278e3 in tls_accept (c=0xb621af88,
            error=0xbf91dfc8) at tls_server.c:346
                    ret = <value optimized out>
                    ssl = (SSL *) 0xb61c1878
                    cert = <value optimized out>
                    tls_c = (struct tls_extra_data *) 0xb6208e58
                    tls_log = <value optimized out>
            #14 0x0052da8e in tls_read_f (c=0xb621af88,
            flags=0xbf91e200) at tls_server.c:1028
                    r = (struct tcp_req *) 0xb621affc
                    bytes_free = 4095
                    bytes_read = 70
                    read_size = 4095
                    ssl_error = 0
                    ssl_read = 0
                    ssl = (SSL *) 0xb61c1878
                    rd_buf =
            "\026\003\001\000A\001\000\000=\003\001Oÿ\v¡ywP\fkÃDÃ
            0/\000\a\000\005\001\000ÃÃÃï000\000\026\0009\0008\0005\000\026\000\023\000\n\0003\0002\00
            202i<ä<åþ\031ì234 \237ê030Y){lѶ"...06Ã'Ãr9ìë¸{[©\t\205\037\036ÿHPH5µ\t(é32Ã027æò036¾\"\rÔ¤YÃ215uº\027)ö000ñ215\vÃà éEa\023xéJ\206T\204CÃ
            wr_buf =
            "\027\003\001\003 w²'\016ѼÃ033ÃLÃ021Ã)Ã\220hnL¦\020Xê\236\200\tIhøSf\234Ã\rÃ
            ?¯¼\024FÃ\200ª/\2264RÃaäIpXȬÃ035äQh\211Ã026ý4ÃU+\asr£/`¤Oû1AʹqUܿ·v±ü\205"...
                    rd = {buf = 0xbf90dfa4 "\026\003\001", pos
            = 70, used = 70, size = 65536}
                    wr = {
              buf = 0xbf8fdfa4
            "\027\003\001\003 w²'\016ѼÃ033ÃLÃ021Ã)Ã\220hnL¦\020Xê\236\200\tIhøSf\234Ã\rÃ
            ?¯¼\024FÃ\200ª/\2264RÃaäIpXȬÃ035äQh\211Ã026ý4ÃU+\asr£/`¤Oû1AʹqUܿ·v±ü\205"...,
            pos = 0, used = 0,
            size = 65536}
                    tls_c = (struct tls_extra_data *) 0xb6208e58
                    enc_rd_buf = (struct tls_rd_buf *) 0x0
                    n = 0
                    flush_flags = <value optimized out>
                    err_src = 0x541a83 "TLS read:"
                    x = <value optimized out>
            #15 0x08158ade in tcp_read_headers (c=0xb621af88,
            read_flags=0xbf91e200) at tcp_read.c:406
                    bytes = -1
                    remaining = <value optimized out>
                    p = <value optimized out>
                    r = (struct tcp_req *) 0xb621affc
            #16 0x08158fd4 in tcp_read_req (con=0xb621af88,
            bytes_read=0xbf91e204, read_flags=0xbf91e200) at
            tcp_read.c:871
                    bytes = <value optimized out>
                    total_bytes = 0
                    resp = <value optimized out>
                    size = <value optimized out>
                    req = (struct tcp_req *) 0xb621affc
                    dst = {send_sock = 0x1, to = {s =
            {sa_family = 57540, sa_data =
            "\221¿\020\000\000\000\000\000\000\000\020\000\000"},
            sin = {sin_family = 57540,
                  sin_port = 49041, sin_addr = {s_addr = 16},
            sin_zero = "\000\000\000\000\020\000\000"}, sin6 =
            {sin6_family = 57540, sin6_port = 49041,
                  sin6_flowinfo = 16, sin6_addr = {in6_u =
            {u6_addr8 =
            "\000\000\000\000\020\000\000\000\001\000\000\000\001\000\000",
            u6_addr16 = {0, 0, 16, 0, 1, 0,
                        1, 0}, u6_addr32 = {0, 16, 1, 1}}},
            sin6_scope_id = 12}}, id = -1080958468, proto = 4
            '\004', send_flags = {f = 0 '\0', blst_imask = 0 '\0'}}
                    c = 10 '\n'
            #17 0x08159a0f in handle_io (fm=0x83296b0,
            events=1, idx=-1) at tcp_read.c:1150
                    ret = 4
                    n = 4
                    read_flags = 1
                    con = (struct tcp_connection *) 0xb621af88
                    s = 12
                    resp = <value optimized out>
                    t = <value optimized out>
            #18 0x0815d653 in tcp_receive_loop (unix_sock=35)
            at io_wait.h:1091
            No locals.
            ---Type <return> to continue, or q <return> to quit---
            #19 0x0812885b in tcp_init_children () at
            tcp_main.c:4819
                    r = 3
                    reader_fd_1 = 35
                    pid = <value optimized out>
                    si = <value optimized out>
            #20 0x080ad412 in main_loop () at main.c:1632
                    i = 4
                    pid = <value optimized out>
                    si = (struct socket_info *) 0x0
                    si_desc = "udp receiver child=3
            sock=192.168.166.31:5060
            <http://192.168.166.31:5060>\000\221¿\002\000\000\000/\200\220\000\001\000\000\000\000\000H\000¨ä21¿\001\000\000\000°¶è026¶\000\000\000\000\000õ\030s2\b\002\000\000\000âf\b\000õ\001\000\000\000\000\000\000\000\001\000\000\000\000\000\000\000ÿÿÿÿ¸ä21¿"
            #21 0x080afbe7 in main (argc=3,
            argv=dwarf2_read_address: Corrupted DWARF expression.
            ) at main.c:2398
                    cfg_stream = (FILE *) 0x8a2d008
                    c = <value optimized out>
                    r = 0
                    tmp = 0xbf91e674
            "\212þ\221¿\233þ\221¿°þ\221¿»þ\221¿Ã\221¿ù1¿\020ÿ\221¿Dÿ\221¿Lÿ\221¿Wÿ\221¿]ÿ\221¿oÿ\221¿{ÿ\221¿\202ÿ\221
            tmp_len = 10492229
                    port = 136192361
                    proto = -1080957480
                    ret = <value optimized out>
                    seed = 21782655
                    rfd = 4
                    debug_save = <value optimized out>
                    debug_flag = 0
                    dont_fork_cnt = 0
                    n_lst = <value optimized out>
                    p = <value optimized out>


            Best Regards


            _______________________________________________
            SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
            sr-users@lists.sip-router.org  <mailto:sr-users@lists.sip-router.org>
            http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users

            -- 
            Daniel-Constantin Mierla -http://www.asipto.com
            http://twitter.com/#!/miconda  <http://twitter.com/#%21/miconda>  -http://www.linkedin.com/in/miconda
            Kamailio Advanced Training, Seattle, USA, Sep 23-26, 2012 -http://asipto.com/u/katu
            Kamailio Practical Workshop, Netherlands, Sep 10-12, 2012 -http://asipto.com/u/kpw




        _______________________________________________
        SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
        sr-users@lists.sip-router.org  <mailto:sr-users@lists.sip-router.org>
        http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users

        _______________________________________________
        SIP Express Router (SER) and Kamailio (OpenSER) -
        sr-users mailing list
        sr-users@lists.sip-router.org
        <mailto:sr-users@lists.sip-router.org>
        http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users





_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users@lists.sip-router.org  <mailto:sr-users@lists.sip-router.org>
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users

-- 
Daniel-Constantin Mierla -http://www.asipto.com
http://twitter.com/#!/miconda  <http://twitter.com/#%21/miconda>  -http://www.linkedin.com/in/miconda
Kamailio Advanced Training, Seattle, USA, Sep 23-26, 2012 -http://asipto.com/u/katu
Kamailio Practical Workshop, Netherlands, Sep 10-12, 2012 -http://asipto.com/u/kpw

Hello,

MEMDBG=0 turns on f_malloc, disabling q_malloc. The token F_MALLOC in the kamailio -V output shows f_malloc is in use.

q_malloc is more suitable for debugging, as it keep trace of location in the source code that allocates/frees the memory chunks, at the expense of a bit more memory usage due to tracking overhead. q_malloc was the default for 3.1, that's why I asked because the trace showed lines from f_malloc c code.

What is the size of shared memory you start kamailio? Is the default 32MB or you give a different -m parameter value? Was the serve very loaded? How many such situations did you get so far?

Cheers, Daniel

On 7/30/12 7:16 PM, Bruno Bresciani wrote:

Hi,

Bellow is output of 'kamailio -V'...

---

version: kamailio 3.1.2 (i386/linux) 4d9f90 flags: STATS: Off, USE_IPV6, USE_TCP, USE_TLS, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, F_MALLOC, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535, PKG_SIZE 4MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: 4d9f90 compiled on 10:39:54 Apr 29 2011 with gcc 4.1.2

---

Daniel, I don't remember to enable f_malloc... Actually I do not know the difference between f_malloc and g_malloc to choose compile kamailio with f_malloc instead of g_malloc. My commands to compile kamailio were:

$ make PREFIX="/home2/local/kamailio" FLAVOUR=kamailio include_modules="db_postgres tls" MEMDBG=0 cfg** $ make all $ make install*

If you need more informations, I am available...

Best Regards 2012/7/30 Daniel-Constantin Mierla <miconda@gmail.com mailto:miconda@gmail.com>

Hello,

can you provide output of 'kamailio -V'? Did you enable f_malloc
instead of q_malloc which was default in 3.1?

Cheers,
Daniel


On 7/30/12 1:44 PM, Bruno Bresciani wrote:

...

Thank's for feedback Daniel,

This core is add to the tracker with ID 247...

Best Regards


2012/7/30 Daniel-Constantin Mierla <miconda@gmail.com
<mailto:miconda@gmail.com>>

    Hello,

    it is in my to-do list to investigate, so far I was not able
    to reproduce.

    Vacation time together with other travelings makes a bit more
    slower process. Please add it also to the tracker:

    http://sip-router.org/tracker/

    A minor release for 3.3 branch is going to be soon, being
    there makes sure it will be reviewed to see if it applies.

    Cheers,
    Daniel


    On 7/27/12 3:00 PM, Bruno Bresciani wrote:

...

    Hi Users,

    This is my ultimate try to ask for some help to debug the
    two cores generated by fm_realloc() function. If somebody
    can give me a feedback about my questions or doubts I will
    be very grateful, this user list of kamailio always help me
    to solve my problems.

    Best Regards


    2012/7/23 Bruno Bresciani <bruno.bresciani@gmail.com
    <mailto:bruno.bresciani@gmail.com>>

        Hi All,

        I imagine that this core doesn't easy to replicate but
        somebody  can help me to understand why it was
        generated? There is some way to  prevent it?

        Best Regards



        2012/7/18 Renan Capaverde
        <renan.capaverde@digitro.com.br
        <mailto:renan.capaverde@digitro.com.br>>

            I am having the same problem on kamailio 3.1.6.
            Someone please help.

            Cheers,
            Renan

            Em 17/7/2012 14:26, Bruno Bresciani escreveu:

...

            Excuse-me Daniel, I forgot this information...
            Kamailio version 3.1.2

            Best Regards


            2012/7/17 Daniel-Constantin Mierla
            <miconda@gmail.com <mailto:miconda@gmail.com>>

                Hello,

                send also the version 'kamailio -V' to match
                the proper sources.

                Cheers,
                Daniel


                On 7/17/12 7:11 PM, Bruno Bresciani wrote:

...

                Hi all,

                I observed two at 0x08173bf0 in fm_realloc
                (qm=0xb5f7d000, p=0xb61f0bc0, size=0) at
                mem/f_malloc.c:536.

                I don't know exactly what may be caused this
                core because the debug of kamailio is
                disabled, but looking at backtrace (gdb) and
                source code I could notice a relationship with
                TLS connection establishment (handshake).
                Other important fact is that core was
                generated by the kamalio itself with signal 6
                (abort) because it can't find the pointer to
                memory realloc.
                I'd like to know if someone already observed
                this issue and if my analyse above is correct?

                Bellow is backtrace full with gdb:

                Core was generated by
                `/home2/local/kamailio/sbin/kamailio -P
                /var/run/kamailio.pid'.
                Program terminated with signal 6, Aborted.
                #0  0x00f30402 in __kernel_vsyscall ()
                (gdb)
                (gdb)
                (gdb)
                (gdb)
                (gdb) bt full
                #0  0x00f30402 in __kernel_vsyscall ()
                No symbol table info available.
                #1  0x009fec10 in raise () from /lib/libc.so.6
                No symbol table info available.
                #2  0x00a00521 in abort () from /lib/libc.so.6
                No symbol table info available.
                #3  0x08173bf0 in fm_realloc (qm=0xb5f7d000,
                p=0xb61f0bc0, size=0) at mem/f_malloc.c:536
                        f = (struct fm_frag *) 0xb61f0bb8
                        pf = <value optimized out>
                orig_size = 18
                L=Ãb¬Ã2n = (struct fm_frag *) 0xb61f0bd2
                        ptr = <value optimized out>
                        hash = 23548
                #4  0x0051a87f in ser_realloc (ptr=0xb61f0bc0,
                size=32) at ../../mem/shm_mem.h:266
                No locals.
                #5  0x00460295 in OPENSSL_cleanse () from
                /lib/libcrypto.so.6
                No symbol table info available.
                #6  0x004609f7 in CRYPTO_realloc () from
                /lib/libcrypto.so.6
                No symbol table info available.
                #7  0x003efb22 in sk_insert () from
                /lib/libcrypto.so.6
                No symbol table info available.
                #8  0x003efbaa in sk_push () from
                /lib/libcrypto.so.6
                No symbol table info available.
                #9  0x007c2ac3 in ssl_bytes_to_cipher_list ()
                from /lib/libssl.so.6
                No symbol table info available.
                #10 0x007aa9f3 in ssl3_get_client_hello ()
                from /lib/libssl.so.6
                No symbol table info available.
                #11 0x007ab815 in ssl3_accept () from
                /lib/libssl.so.6
                No symbol table info available.
                #12 0x007c1c3a in SSL_accept () from
                /lib/libssl.so.6
                No symbol table info available.
                #13 0x005278e3 in tls_accept (c=0xb621af88,
                error=0xbf91dfc8) at tls_server.c:346
                        ret = <value optimized out>
                        ssl = (SSL *) 0xb61c1878
                        cert = <value optimized out>
                        tls_c = (struct tls_extra_data *)
                0xb6208e58
                tls_log = <value optimized out>
                #14 0x0052da8e in tls_read_f (c=0xb621af88,
                flags=0xbf91e200) at tls_server.c:1028
                        r = (struct tcp_req *) 0xb621affc
                bytes_free = 4095
                bytes_read = 70
                read_size = 4095
                ssl_error = 0
                ssl_read = 0
                        ssl = (SSL *) 0xb61c1878
                        rd_buf =
                "\026\003\001\000A\001\000\000=\003\001Oÿ\v¡ywP\fkÃDÃ
                0/\000\a\000\005\001\000ÃÃÃï000\000\026\0009\0008\0005\000\026\000\023\000\n\0003\0002\00
                202i<ä<åþ\031ì234 \237ê030Y){lѶ"...06Ã'Ãr9ìë¸{[©\t\205\037\036ÿHPH5µ\t(é32Ã027æò036¾\"\rÔ¤YÃ215uº\027)ö000ñ215\vÃà éEa\023xéJ\206T\204CÃ
                wr_buf =
                "\027\003\001\003 w²'\016ѼÃ033ÃLÃ021Ã)Ã\220hnL¦\020Xê\236\200\tIhøSf\234Ã\rÃ
                ?¯¼\024FÃ\200ª/\2264RÃaäIpXȬÃ035äQh\211Ã026ý4ÃU+\asr£/`¤Oû1AʹqUܿ·v±ü\205"...
                        rd = {buf = 0xbf90dfa4 "\026\003\001",
                pos = 70, used = 70, size = 65536}
                        wr = {
                  buf = 0xbf8fdfa4
                "\027\003\001\003 w²'\016ѼÃ033ÃLÃ021Ã)Ã\220hnL¦\020Xê\236\200\tIhøSf\234Ã\rÃ
                ?¯¼\024FÃ\200ª/\2264RÃaäIpXȬÃ035äQh\211Ã026ý4ÃU+\asr£/`¤Oû1AʹqUܿ·v±ü\205"...,
                pos = 0, used = 0,
                size = 65536}
                        tls_c = (struct tls_extra_data *)
                0xb6208e58
                enc_rd_buf = (struct tls_rd_buf *) 0x0
                        n = 0
                flush_flags = <value optimized out>
                err_src = 0x541a83 "TLS read:"
                        x = <value optimized out>
                #15 0x08158ade in tcp_read_headers
                (c=0xb621af88, read_flags=0xbf91e200) at
                tcp_read.c:406
                        bytes = -1
                remaining = <value optimized out>
                        p = <value optimized out>
                        r = (struct tcp_req *) 0xb621affc
                #16 0x08158fd4 in tcp_read_req
                (con=0xb621af88, bytes_read=0xbf91e204,
                read_flags=0xbf91e200) at tcp_read.c:871
                        bytes = <value optimized out>
                total_bytes = 0
                        resp = <value optimized out>
                        size = <value optimized out>
                        req = (struct tcp_req *) 0xb621affc
                        dst = {send_sock = 0x1, to = {s =
                {sa_family = 57540, sa_data =
                "\221¿\020\000\000\000\000\000\000\000\020\000\000"},
                sin = {sin_family = 57540,
                      sin_port = 49041, sin_addr = {s_addr =
                16}, sin_zero =
                "\000\000\000\000\020\000\000"}, sin6 =
                {sin6_family = 57540, sin6_port = 49041,
                sin6_flowinfo = 16, sin6_addr = {in6_u =
                {u6_addr8 =
                "\000\000\000\000\020\000\000\000\001\000\000\000\001\000\000",
                u6_addr16 = {0, 0, 16, 0, 1, 0,
                            1, 0}, u6_addr32 = {0, 16, 1,
                1}}}, sin6_scope_id = 12}}, id = -1080958468,
                proto = 4 '\004', send_flags = {f = 0 '\0',
                blst_imask = 0 '\0'}}
                        c = 10 '\n'
                #17 0x08159a0f in handle_io (fm=0x83296b0,
                events=1, idx=-1) at tcp_read.c:1150
                        ret = 4
                        n = 4
                read_flags = 1
                        con = (struct tcp_connection *) 0xb621af88
                        s = 12
                        resp = <value optimized out>
                        t = <value optimized out>
                #18 0x0815d653 in tcp_receive_loop
                (unix_sock=35) at io_wait.h:1091
                No locals.
                ---Type <return> to continue, or q <return> to
                quit---
                #19 0x0812885b in tcp_init_children () at
                tcp_main.c:4819
                        r = 3
                reader_fd_1 = 35
                        pid = <value optimized out>
                        si = <value optimized out>
                #20 0x080ad412 in main_loop () at main.c:1632
                        i = 4
                        pid = <value optimized out>
                        si = (struct socket_info *) 0x0
                si_desc = "udp receiver child=3
                sock=192.168.166.31:5060
                <http://192.168.166.31:5060>\000\221¿\002\000\000\000/\200\220\000\001\000\000\000\000\000H\000¨ä21¿\001\000\000\000°¶è026¶\000\000\000\000\000õ\030s2\b\002\000\000\000âf\b\000õ\001\000\000\000\000\000\000\000\001\000\000\000\000\000\000\000ÿÿÿÿ¸ä21¿"
                #21 0x080afbe7 in main (argc=3,
                argv=dwarf2_read_address: Corrupted DWARF
                expression.
                ) at main.c:2398
                cfg_stream = (FILE *) 0x8a2d008
                        c = <value optimized out>
                        r = 0
                        tmp = 0xbf91e674
                "\212þ\221¿\233þ\221¿°þ\221¿»þ\221¿Ã\221¿ù1¿\020ÿ\221¿Dÿ\221¿Lÿ\221¿Wÿ\221¿]ÿ\221¿oÿ\221¿{ÿ\221¿\202ÿ\221
                tmp_len = 10492229
                        port = 136192361
                        proto = -1080957480
                        ret = <value optimized out>
                        seed = 21782655
                        rfd = 4
                debug_save = <value optimized out>
                debug_flag = 0
                dont_fork_cnt = 0
                        n_lst = <value optimized out>
                        p = <value optimized out>


                Best Regards


                _______________________________________________
                SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
                sr-users@lists.sip-router.org  <mailto:sr-users@lists.sip-router.org>
                http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users

                -- 
                Daniel-Constantin Mierla -http://www.asipto.com
                http://twitter.com/#!/miconda  <http://twitter.com/#%21/miconda>  -http://www.linkedin.com/in/miconda
                Kamailio Advanced Training, Seattle, USA, Sep 23-26, 2012 -http://asipto.com/u/katu
                Kamailio Practical Workshop, Netherlands, Sep 10-12, 2012 -http://asipto.com/u/kpw




            _______________________________________________
            SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
            sr-users@lists.sip-router.org  <mailto:sr-users@lists.sip-router.org>
            http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users

            _______________________________________________
            SIP Express Router (SER) and Kamailio (OpenSER) -
            sr-users mailing list
            sr-users@lists.sip-router.org
            <mailto:sr-users@lists.sip-router.org>
            http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users





    _______________________________________________
    SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
    sr-users@lists.sip-router.org  <mailto:sr-users@lists.sip-router.org>
    http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users

    -- 
    Daniel-Constantin Mierla -http://www.asipto.com
    http://twitter.com/#!/miconda  <http://twitter.com/#%21/miconda>  -http://www.linkedin.com/in/miconda
    Kamailio Advanced Training, Seattle, USA, Sep 23-26, 2012 -http://asipto.com/u/katu
    Kamailio Practical Workshop, Netherlands, Sep 10-12, 2012 -http://asipto.com/u/kpw

-- 
Daniel-Constantin Mierla -http://www.asipto.com
http://twitter.com/#!/miconda  <http://twitter.com/#%21/miconda>  -http://www.linkedin.com/in/miconda
Kamailio Advanced Training, Seattle, USA, Sep 23-26, 2012 -http://asipto.com/u/katu
Kamailio Practical Workshop, Netherlands, Sep 10-12, 2012 -http://asipto.com/u/kpw

Hello,

On 7/30/12 9:01 PM, Bruno Bresciani wrote:

Hi,

I compiled kamailio with MEMDBG = 0 because I didn't want the memory debug in kamailio log, but I didn't know these turns on f_malloc and disabling q_malloc...

the log messages related to memory operations can be controlled by global parameters memdbg and memlog.

Probably the size of shared memory that I start kamailio is 32MB because I didn't gave a different -m parameter value, exist a command to verify this information?

kamctl fifo get_statistics all

and see the shared memory total value. It will be interesting to see available shared memory as well.

At moment that crash happened, there were few registered users agents and were being made tests with register and calls with TLS protocol. I got only the two situations that I showed at first email.

If you plan to start a new installation, I strongly recommend 3.3 branch, the code is more actual and easier to debug. 3.1 is no longer an official maintained branch, those being now 3.3 and 3.2. I'm looking at this issue to be sure it is no longer in latest stable.

Cheers, Daniel

Best Regards

2012/7/30 Daniel-Constantin Mierla <miconda@gmail.com mailto:miconda@gmail.com>

Hello,

MEMDBG=0 turns on f_malloc, disabling q_malloc. The token F_MALLOC
in the kamailio -V output shows f_malloc is in use.

q_malloc is more suitable for debugging, as it keep trace of
location in the source code that allocates/frees the memory
chunks, at the expense of a bit more memory usage due to tracking
overhead. q_malloc was the default for 3.1, that's why I asked
because the trace showed lines from f_malloc c code.

What is the size of shared memory you start kamailio? Is the
default 32MB or you give a different -m parameter value? Was the
serve very loaded? How many such situations did you get so far?

Cheers,
Daniel



On 7/30/12 7:16 PM, Bruno Bresciani wrote:

...

Hi,

Bellow is output of 'kamailio -V'...

------------------------------------------------------------------------------------------------------------------
version: kamailio 3.1.2 (i386/linux) 4d9f90
flags: STATS: Off, USE_IPV6, USE_TCP, USE_TLS, TLS_HOOKS,
USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM,
SHM_MMAP, PKG_MALLOC, F_MALLOC, USE_FUTEX,
FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER,
USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN
16, MAX_URI_SIZE 1024, BUF_SIZE 65535, PKG_SIZE 4MB
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
id: 4d9f90
compiled on 10:39:54 Apr 29 2011 with gcc 4.1.2
-------------------------------------------------------------------------------------------------------------------

Daniel, I don't remember to enable f_malloc... Actually I do not
know the difference between f_malloc and g_malloc to choose
compile kamailio with f_malloc instead of g_malloc.
My commands to compile kamailio were:
*
$ make PREFIX="/home2/local/kamailio" FLAVOUR=kamailio
include_modules="db_postgres tls" MEMDBG=0 cfg**
$ make all
$ make install*

If you need more informations, I am available...

Best Regards
2012/7/30 Daniel-Constantin Mierla <miconda@gmail.com
<mailto:miconda@gmail.com>>

    Hello,

    can you provide output of 'kamailio -V'? Did you enable
    f_malloc instead of q_malloc which was default in 3.1?

    Cheers,
    Daniel


    On 7/30/12 1:44 PM, Bruno Bresciani wrote:

...

    Thank's for feedback Daniel,

    This core is add to the tracker with ID 247...

    Best Regards


    2012/7/30 Daniel-Constantin Mierla <miconda@gmail.com
    <mailto:miconda@gmail.com>>

        Hello,

        it is in my to-do list to investigate, so far I was not
        able to reproduce.

        Vacation time together with other travelings makes a bit
        more slower process. Please add it also to the tracker:

        http://sip-router.org/tracker/

        A minor release for 3.3 branch is going to be soon,
        being there makes sure it will be reviewed to see if it
        applies.

        Cheers,
        Daniel


        On 7/27/12 3:00 PM, Bruno Bresciani wrote:

...

        Hi Users,

        This is my ultimate try to ask for some help to debug
        the two cores generated by fm_realloc() function. If
        somebody can give me a feedback about my questions or
        doubts I will be very grateful, this user list of
        kamailio always help me to solve my problems.

        Best Regards


        2012/7/23 Bruno Bresciani <bruno.bresciani@gmail.com
        <mailto:bruno.bresciani@gmail.com>>

            Hi All,

            I imagine that this core doesn't easy to replicate
            but somebody  can help me to understand why it was
            generated? There is some way to  prevent it?

            Best Regards



            2012/7/18 Renan Capaverde
            <renan.capaverde@digitro.com.br
            <mailto:renan.capaverde@digitro.com.br>>

                I am having the same problem on kamailio 3.1.6.
                Someone please help.

                Cheers,
                Renan

                Em 17/7/2012 14:26, Bruno Bresciani escreveu:

...

                Excuse-me Daniel, I forgot this information...
                Kamailio version 3.1.2

                Best Regards


                2012/7/17 Daniel-Constantin Mierla
                <miconda@gmail.com <mailto:miconda@gmail.com>>

                    Hello,

                    send also the version 'kamailio -V' to
                    match the proper sources.

                    Cheers,
                    Daniel


                    On 7/17/12 7:11 PM, Bruno Bresciani wrote:

...

                    Hi all,

                    I observed two at 0x08173bf0 in
                    fm_realloc (qm=0xb5f7d000, p=0xb61f0bc0,
                    size=0) at mem/f_malloc.c:536.

                    I don't know exactly what may be caused
                    this core because the debug of kamailio
                    is disabled, but looking at backtrace
                    (gdb) and source code I could notice a
                    relationship with TLS connection
                    establishment (handshake). Other
                    important fact is that core was generated
                    by the kamalio itself with signal 6
                    (abort) because it can't find the pointer
                    to memory realloc.
                    I'd like to know if someone already
                    observed this issue and if my analyse
                    above is correct?

                    Bellow is backtrace full with gdb:

                    Core was generated by
                    `/home2/local/kamailio/sbin/kamailio -P
                    /var/run/kamailio.pid'.
                    Program terminated with signal 6, Aborted.
                    #0  0x00f30402 in __kernel_vsyscall ()
                    (gdb)
                    (gdb)
                    (gdb)
                    (gdb)
                    (gdb) bt full
                    #0  0x00f30402 in __kernel_vsyscall ()
                    No symbol table info available.
                    #1  0x009fec10 in raise () from
                    /lib/libc.so.6
                    No symbol table info available.
                    #2  0x00a00521 in abort () from
                    /lib/libc.so.6
                    No symbol table info available.
                    #3  0x08173bf0 in fm_realloc
                    (qm=0xb5f7d000, p=0xb61f0bc0, size=0) at
                    mem/f_malloc.c:536
                            f = (struct fm_frag *) 0xb61f0bb8
                            pf = <value optimized out>
                    orig_size = 18
                    L=Ãb¬Ã2n = (struct fm_frag *) 0xb61f0bd2
                            ptr = <value optimized out>
                            hash = 23548
                    #4  0x0051a87f in ser_realloc
                    (ptr=0xb61f0bc0, size=32) at
                    ../../mem/shm_mem.h:266
                    No locals.
                    #5  0x00460295 in OPENSSL_cleanse () from
                    /lib/libcrypto.so.6
                    No symbol table info available.
                    #6  0x004609f7 in CRYPTO_realloc () from
                    /lib/libcrypto.so.6
                    No symbol table info available.
                    #7  0x003efb22 in sk_insert () from
                    /lib/libcrypto.so.6
                    No symbol table info available.
                    #8  0x003efbaa in sk_push () from
                    /lib/libcrypto.so.6
                    No symbol table info available.
                    #9  0x007c2ac3 in
                    ssl_bytes_to_cipher_list () from
                    /lib/libssl.so.6
                    No symbol table info available.
                    #10 0x007aa9f3 in ssl3_get_client_hello
                    () from /lib/libssl.so.6
                    No symbol table info available.
                    #11 0x007ab815 in ssl3_accept () from
                    /lib/libssl.so.6
                    No symbol table info available.
                    #12 0x007c1c3a in SSL_accept () from
                    /lib/libssl.so.6
                    No symbol table info available.
                    #13 0x005278e3 in tls_accept
                    (c=0xb621af88, error=0xbf91dfc8) at
                    tls_server.c:346
                            ret = <value optimized out>
                            ssl = (SSL *) 0xb61c1878
                            cert = <value optimized out>
                            tls_c = (struct tls_extra_data *)
                    0xb6208e58
                    tls_log = <value optimized out>
                    #14 0x0052da8e in tls_read_f
                    (c=0xb621af88, flags=0xbf91e200) at
                    tls_server.c:1028
                            r = (struct tcp_req *) 0xb621affc
                    bytes_free = 4095
                    bytes_read = 70
                    read_size = 4095
                    ssl_error = 0
                    ssl_read = 0
                            ssl = (SSL *) 0xb61c1878
                            rd_buf =
                    "\026\003\001\000A\001\000\000=\003\001Oÿ\v¡ywP\fkÃDÃ
                    0/\000\a\000\005\001\000ÃÃÃï000\000\026\0009\0008\0005\000\026\000\023\000\n\0003\0002\00
                    202i<ä<åþ\031ì234 \237ê030Y){lѶ"...06Ã'Ãr9ìë¸{[©\t\205\037\036ÿHPH5µ\t(é32Ã027æò036¾\"\rÔ¤YÃ215uº\027)ö000ñ215\vÃà éEa\023xéJ\206T\204CÃ
                    wr_buf =
                    "\027\003\001\003 w²'\016ѼÃ033ÃLÃ021Ã)Ã\220hnL¦\020Xê\236\200\tIhøSf\234Ã\rÃ
                    ?¯¼\024FÃ\200ª/\2264RÃaäIpXȬÃ035äQh\211Ã026ý4ÃU+\asr£/`¤Oû1AʹqUܿ·v±ü\205"...
                            rd = {buf = 0xbf90dfa4
                    "\026\003\001", pos = 70, used = 70, size
                    = 65536}
                            wr = {
                      buf = 0xbf8fdfa4
                    "\027\003\001\003 w²'\016ѼÃ033ÃLÃ021Ã)Ã\220hnL¦\020Xê\236\200\tIhøSf\234Ã\rÃ
                    ?¯¼\024FÃ\200ª/\2264RÃaäIpXȬÃ035äQh\211Ã026ý4ÃU+\asr£/`¤Oû1AʹqUܿ·v±ü\205"...,
                    pos = 0, used = 0,
                    size = 65536}
                            tls_c = (struct tls_extra_data *)
                    0xb6208e58
                    enc_rd_buf = (struct tls_rd_buf *) 0x0
                            n = 0
                    flush_flags = <value optimized out>
                    err_src = 0x541a83 "TLS read:"
                            x = <value optimized out>
                    #15 0x08158ade in tcp_read_headers
                    (c=0xb621af88, read_flags=0xbf91e200) at
                    tcp_read.c:406
                            bytes = -1
                    remaining = <value optimized out>
                            p = <value optimized out>
                            r = (struct tcp_req *) 0xb621affc
                    #16 0x08158fd4 in tcp_read_req
                    (con=0xb621af88, bytes_read=0xbf91e204,
                    read_flags=0xbf91e200) at tcp_read.c:871
                            bytes = <value optimized out>
                    total_bytes = 0
                            resp = <value optimized out>
                            size = <value optimized out>
                            req = (struct tcp_req *) 0xb621affc
                            dst = {send_sock = 0x1, to = {s =
                    {sa_family = 57540, sa_data =
                    "\221¿\020\000\000\000\000\000\000\000\020\000\000"},
                    sin = {sin_family = 57540,
                          sin_port = 49041, sin_addr =
                    {s_addr = 16}, sin_zero =
                    "\000\000\000\000\020\000\000"}, sin6 =
                    {sin6_family = 57540, sin6_port = 49041,
                    sin6_flowinfo = 16, sin6_addr = {in6_u =
                    {u6_addr8 =
                    "\000\000\000\000\020\000\000\000\001\000\000\000\001\000\000",
                    u6_addr16 = {0, 0, 16, 0, 1, 0,
                                1, 0}, u6_addr32 = {0, 16, 1,
                    1}}}, sin6_scope_id = 12}}, id =
                    -1080958468, proto = 4 '\004', send_flags
                    = {f = 0 '\0', blst_imask = 0 '\0'}}
                            c = 10 '\n'
                    #17 0x08159a0f in handle_io
                    (fm=0x83296b0, events=1, idx=-1) at
                    tcp_read.c:1150
                            ret = 4
                            n = 4
                    read_flags = 1
                            con = (struct tcp_connection *)
                    0xb621af88
                            s = 12
                            resp = <value optimized out>
                            t = <value optimized out>
                    #18 0x0815d653 in tcp_receive_loop
                    (unix_sock=35) at io_wait.h:1091
                    No locals.
                    ---Type <return> to continue, or q
                    <return> to quit---
                    #19 0x0812885b in tcp_init_children () at
                    tcp_main.c:4819
                            r = 3
                    reader_fd_1 = 35
                            pid = <value optimized out>
                            si = <value optimized out>
                    #20 0x080ad412 in main_loop () at main.c:1632
                            i = 4
                            pid = <value optimized out>
                            si = (struct socket_info *) 0x0
                    si_desc = "udp receiver child=3
                    sock=192.168.166.31:5060
                    <http://192.168.166.31:5060>\000\221¿\002\000\000\000/\200\220\000\001\000\000\000\000\000H\000¨ä21¿\001\000\000\000°¶è026¶\000\000\000\000\000õ\030s2\b\002\000\000\000âf\b\000õ\001\000\000\000\000\000\000\000\001\000\000\000\000\000\000\000ÿÿÿÿ¸ä21¿"
                    #21 0x080afbe7 in main (argc=3,
                    argv=dwarf2_read_address: Corrupted DWARF
                    expression.
                    ) at main.c:2398
                    cfg_stream = (FILE *) 0x8a2d008
                            c = <value optimized out>
                            r = 0
                            tmp = 0xbf91e674
                    "\212þ\221¿\233þ\221¿°þ\221¿»þ\221¿Ã\221¿ù1¿\020ÿ\221¿Dÿ\221¿Lÿ\221¿Wÿ\221¿]ÿ\221¿oÿ\221¿{ÿ\221¿\202ÿ\221
                    tmp_len = 10492229
                            port = 136192361
                            proto = -1080957480
                            ret = <value optimized out>
                            seed = 21782655
                            rfd = 4
                    debug_save = <value optimized out>
                    debug_flag = 0
                    dont_fork_cnt = 0
                            n_lst = <value optimized out>
                            p = <value optimized out>


                    Best Regards


                    _______________________________________________
                    SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
                    sr-users@lists.sip-router.org  <mailto:sr-users@lists.sip-router.org>
                    http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users

                    -- 
                    Daniel-Constantin Mierla -http://www.asipto.com
                    http://twitter.com/#!/miconda  <http://twitter.com/#%21/miconda>  -http://www.linkedin.com/in/miconda
                    Kamailio Advanced Training, Seattle, USA, Sep 23-26, 2012 -http://asipto.com/u/katu
                    Kamailio Practical Workshop, Netherlands, Sep 10-12, 2012 -http://asipto.com/u/kpw




                _______________________________________________
                SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
                sr-users@lists.sip-router.org  <mailto:sr-users@lists.sip-router.org>
                http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users

                _______________________________________________
                SIP Express Router (SER) and Kamailio (OpenSER)
                - sr-users mailing list
                sr-users@lists.sip-router.org
                <mailto:sr-users@lists.sip-router.org>
                http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users





        _______________________________________________
        SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
        sr-users@lists.sip-router.org  <mailto:sr-users@lists.sip-router.org>
        http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users

        -- 
        Daniel-Constantin Mierla -http://www.asipto.com
        http://twitter.com/#!/miconda  <http://twitter.com/#%21/miconda>  -http://www.linkedin.com/in/miconda
        Kamailio Advanced Training, Seattle, USA, Sep 23-26, 2012 -http://asipto.com/u/katu
        Kamailio Practical Workshop, Netherlands, Sep 10-12, 2012 -http://asipto.com/u/kpw

    -- 
    Daniel-Constantin Mierla -http://www.asipto.com
    http://twitter.com/#!/miconda  <http://twitter.com/#%21/miconda>  -http://www.linkedin.com/in/miconda
    Kamailio Advanced Training, Seattle, USA, Sep 23-26, 2012 -http://asipto.com/u/katu
    Kamailio Practical Workshop, Netherlands, Sep 10-12, 2012 -http://asipto.com/u/kpw

-- 
Daniel-Constantin Mierla -http://www.asipto.com
http://twitter.com/#!/miconda  <http://twitter.com/#%21/miconda>  -http://www.linkedin.com/in/miconda
Kamailio Advanced Training, Seattle, USA, Sep 23-26, 2012 -http://asipto.com/u/katu
Kamailio Practical Workshop, Netherlands, Sep 10-12, 2012 -http://asipto.com/u/kpw

Hello,

On 7/31/12 3:39 PM, Bruno Bresciani wrote:

Hello,

I didn't know that log messages related to memory operations can be controlled by global parameter, but I like to know if is recommendable I recompile kamailio using q_malloc (default) and not f_malloc... memory operations using q_malloc is more reliable and avoid problems or crashes or it is only more suitable for debugging? What do you suggest Daniel?

q_malloc is more suitable for debugging.

The 'get_statistics all' command is avaliable by a specific module? I run 'kamctl fifo get_statistics all' and return '500 command 'get_statistics' not available'

Do you have kex module loaded? What is the output of 'kamctl fifo which'?

For while is impossible to start a new installation, first because I don't know how much time I will spend to port and second because I am involved with other developments and I have no time to make this. I know that 3.1 is no longer a official branch but now start a new installation it's very very difficult, my in intention is discover what caused the crash and if exists a way to fix or prevent it.

Discovering may require additional patches, like more debug messages in the C code, that's why is better to start with the latest stable.

Cheers, Daniel

Best Regards

2012/7/31 Daniel-Constantin Mierla <miconda@gmail.com mailto:miconda@gmail.com>

Hello,


On 7/30/12 9:01 PM, Bruno Bresciani wrote:

...

Hi,

I compiled kamailio with MEMDBG = 0 because I didn't want the
memory debug  in kamailio log, but I didn't know these turns on
f_malloc and disabling q_malloc...

the log messages related to memory operations can be controlled by
global parameters memdbg and memlog.

...

Probably the size of shared memory that I start kamailio is 32MB
because I didn't gave a different -m parameter value, exist a
command to verify this information?

kamctl fifo get_statistics all

and see the shared memory total value. It will be interesting to
see available shared memory as well.

...

At moment that crash happened, there were few registered users
agents and were being made tests with register and calls with TLS
protocol. I got only the two situations that I showed at first email.

If you plan to start a new installation, I strongly recommend 3.3
branch, the code is more actual and easier to debug. 3.1 is no
longer an official maintained branch, those being now 3.3 and 3.2.
I'm looking at this issue to be sure it is no longer in latest stable.

Cheers,
Daniel

Hello,

if you refer to free(0) issue with libssl, I am not sure it was ever put in 3.1 branch, as the last release was before reporting the issue.

Cheers, Daniel

On 12/20/12 6:01 PM, Bruno Bresciani wrote:

Hello Daniel,

What patch this bug was fixed? This bug was open in 30 July 2012 and ultimate Release to 3.1.X (3.1.6) was released at 14 July 2012...

Best Regards

2012/8/1 Daniel-Constantin Mierla <miconda@gmail.com mailto:miconda@gmail.com>

Hello,


On 7/31/12 3:39 PM, Bruno Bresciani wrote:

...

Hello,

I didn't know that log messages related to memory operations can
be controlled by global parameter, but I like to know if is
recommendable I recompile kamailio using q_malloc (default) and
not f_malloc... memory operations using q_malloc is more reliable
and avoid problems or crashes or it is only more suitable for
debugging? What do you suggest Daniel?

q_malloc is more suitable for debugging.

...

The 'get_statistics all' command is avaliable by a specific
module? I run 'kamctl fifo get_statistics all' and return '500
command 'get_statistics' not available'

Do you have kex module loaded? What is the output of 'kamctl fifo
which'?

...

For while is impossible to start a new installation, first
because I don't know how much time I will spend to port and
second because I am involved with other developments and I have
no time  to make this. I know that 3.1 is no longer a official
branch but now start a new installation it's very very difficult,
my in intention is discover what caused the crash and if exists a
way to fix or prevent it.

Discovering may require additional patches, like more debug
messages in the C code, that's why is better to start with the
latest stable.


Cheers,
Daniel

...

Best Regards

2012/7/31 Daniel-Constantin Mierla <miconda@gmail.com
<mailto:miconda@gmail.com>>

    Hello,


    On 7/30/12 9:01 PM, Bruno Bresciani wrote:

...

    Hi,

    I compiled kamailio with MEMDBG = 0 because I didn't want
    the memory debug  in kamailio log, but I didn't know these
    turns on f_malloc and disabling q_malloc...

    the log messages related to memory operations can be
    controlled by global parameters memdbg and memlog.

...

    Probably the size of shared memory that I start kamailio is
    32MB because I didn't gave a different -m parameter value,
    exist a command to verify this information?

    kamctl fifo get_statistics all

    and see the shared memory total value. It will be interesting
    to see available shared memory as well.

...

    At moment that crash happened, there were few registered
    users agents and were being made tests with register and
    calls with TLS protocol. I got only the two situations that
    I showed at first email.

    If you plan to start a new installation, I strongly recommend
    3.3 branch, the code is more actual and easier to debug. 3.1
    is no longer an official maintained branch, those being now
    3.3 and 3.2. I'm looking at this issue to be sure it is no
    longer in latest stable.

    Cheers,
    Daniel

-- 
Daniel-Constantin Mierla -http://www.asipto.com
http://twitter.com/#!/miconda  <http://twitter.com/#%21/miconda>  -http://www.linkedin.com/in/miconda
Kamailio Advanced Training, Seattle, USA, Sep 23-26, 2012 -http://asipto.com/u/katu
Kamailio Practical Workshop, Netherlands, Sep 10-12, 2012 -http://asipto.com/u/kpw

I think Daniel is referring to this patch: http://git.sip-router.org/cgi-bin/gitweb.cgi/sip-router/?a=commit;h=020acff3...

This fix was done to devel and 3.3 branch and is included in latest 3.3.3 release.

regards Klaus

Am 20.12.2012 19:29, schrieb Bruno Bresciani:

Hi,

I refer the bug fix FS#247 that I open in 30 July 2012 and you closed in 25 october 2012 with the information: "Reopen if the patch for free(0) didn't work.".

What Release this fix was included?

Best Regards

2012/12/20 Daniel-Constantin Mierla <miconda@gmail.com mailto:miconda@gmail.com>

Hello,

if you refer to free(0) issue with libssl, I am not sure it was
ever put in 3.1 branch, as the last release was before reporting
the issue.

Cheers,
Daniel


On 12/20/12 6:01 PM, Bruno Bresciani wrote:

...

Hello Daniel,

What patch this bug was fixed?  This bug was open in 30 July 2012
and ultimate Release to 3.1.X (3.1.6) was released at  14 July
2012...

Best Regards

2012/8/1 Daniel-Constantin Mierla <miconda@gmail.com
<mailto:miconda@gmail.com>>

    Hello,


    On 7/31/12 3:39 PM, Bruno Bresciani wrote:

...

    Hello,

    I didn't know that log messages related to memory operations
    can be controlled by global parameter, but I like to know if
    is recommendable I recompile kamailio using q_malloc
    (default) and not f_malloc... memory operations using
    q_malloc is more reliable and avoid problems or crashes or
    it is only more suitable for debugging? What do you suggest
    Daniel?

    q_malloc is more suitable for debugging.

...

    The 'get_statistics all' command is avaliable by a specific
    module? I run 'kamctl fifo get_statistics all' and return
    '500 command 'get_statistics' not available'

    Do you have kex module loaded? What is the output of 'kamctl
    fifo which'?

...

    For while is impossible to start a new installation, first
    because I don't know how much time I will spend to port and
    second because I am involved with other developments and I
    have no time  to make this. I know that 3.1 is no longer a
    official branch but now start a new installation it's very
    very difficult, my in intention is discover what caused the
    crash and if exists a way to fix or prevent it.

    Discovering may require additional patches, like more debug
    messages in the C code, that's why is better to start with
    the latest stable.


    Cheers,
    Daniel

...

    Best Regards

    2012/7/31 Daniel-Constantin Mierla <miconda@gmail.com
    <mailto:miconda@gmail.com>>

        Hello,


        On 7/30/12 9:01 PM, Bruno Bresciani wrote:

...

        Hi,

        I compiled kamailio with MEMDBG = 0 because I didn't
        want the memory debug  in kamailio log, but I didn't
        know these turns on f_malloc and disabling q_malloc...

        the log messages related to memory operations can be
        controlled by global parameters memdbg and memlog.

...

        Probably the size of shared memory that I start
        kamailio is 32MB because I didn't gave a different -m
        parameter value, exist a command to verify this
        information?

        kamctl fifo get_statistics all

        and see the shared memory total value. It will be
        interesting to see available shared memory as well.

...

        At moment that crash happened, there were few
        registered users agents and were being made tests with
        register and calls with TLS protocol. I got only the
        two situations that I showed at first email.

        If you plan to start a new installation, I strongly
        recommend 3.3 branch, the code is more actual and easier
        to debug. 3.1 is no longer an official maintained
        branch, those being now 3.3 and 3.2. I'm looking at this
        issue to be sure it is no longer in latest stable.

        Cheers,
        Daniel

    -- 
    Daniel-Constantin Mierla -http://www.asipto.com
    http://twitter.com/#!/miconda  <http://twitter.com/#%21/miconda>  -http://www.linkedin.com/in/miconda
    Kamailio Advanced Training, Seattle, USA, Sep 23-26, 2012 -http://asipto.com/u/katu
    Kamailio Practical Workshop, Netherlands, Sep 10-12, 2012 -http://asipto.com/u/kpw

-- 
Daniel-Constantin Mierla -http://www.asipto.com
http://twitter.com/#!/miconda  <http://twitter.com/#%21/miconda>  -http://www.linkedin.com/in/miconda

---

SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users

You can also check which openSSL version you are using. According to http://openssl.6102.n7.nabble.com/Custom-free-routine-is-invoked-with-NULL-a... it happened somewhere between 1.0.0. and 1.0.1c

regards klaus

On 03.01.2013 14:56, Bruno Bresciani wrote:

Thank's for your reply Klaus,

but I don't know if is exactly this patch that Daniel is referring, because the backtrace full with gdb showed that problem happened at ser_realloc function. I will insert this check to NULL pointer at ser_free and ser_realloc function and recompile kamailio 3.1.2 because I can't update to version 3.3.3 for while.

Best Regards

2013/1/2 Klaus Darilion <klaus.mailinglists@pernau.at mailto:klaus.mailinglists@pernau.at>

I think Daniel is referring to this patch:
http://git.sip-router.org/cgi-bin/gitweb.cgi/sip-router/?a=commit;h=020acff35f8e9dfa62aba8678a781a0f7bbb110b

This fix was done to devel and 3.3 branch and is included in latest
3.3.3 release.

regards
Klaus

Am 20.12.2012 19 <tel:20.12.2012%2019>:29, schrieb Bruno Bresciani:

...

Hi,

I refer the bug fix FS#247  that I open in 30 July 2012 and you
closed in 25 october 2012 with the information: "Reopen if the
patch for free(0) didn't work.".

What Release this fix was included?


Best Regards

2012/12/20 Daniel-Constantin Mierla <miconda@gmail.com
<mailto:miconda@gmail.com>>

    Hello,

    if you refer to free(0) issue with libssl, I am not sure it
    was ever put in 3.1 branch, as the last release was before
    reporting the issue.

    Cheers,
    Daniel


    On 12/20/12 6:01 PM, Bruno Bresciani wrote:

...

    Hello Daniel,

    What patch this bug was fixed?  This bug was open in 30 July
    2012 and ultimate Release to 3.1.X (3.1.6) was released at
    14 July 2012...

    Best Regards

    2012/8/1 Daniel-Constantin Mierla <miconda@gmail.com
    <mailto:miconda@gmail.com>>

        Hello,


        On 7/31/12 3:39 PM, Bruno Bresciani wrote:

...

        Hello,

        I didn't know that log messages related to memory
        operations can be controlled by global parameter, but I
        like to know if is recommendable I recompile kamailio
        using q_malloc (default) and not f_malloc... memory
        operations using q_malloc is more reliable and avoid
        problems or crashes or it is only more suitable for
        debugging? What do you suggest Daniel?

        q_malloc is more suitable for debugging.

...

        The 'get_statistics all' command is avaliable by a
        specific module? I run 'kamctl fifo get_statistics all'
        and return '500 command 'get_statistics' not available'

        Do you have kex module loaded? What is the output of
        'kamctl fifo which'?

...

        For while is impossible to start a new installation,
        first because I don't know how much time I will spend to
        port and second because I am involved with other
        developments and I have no time  to make this. I know
        that 3.1 is no longer a official branch but now start a
        new installation it's very very difficult, my in
        intention is discover what caused the crash and if
        exists a way to fix or prevent it.

        Discovering may require additional patches, like more
        debug messages in the C code, that's why is better to
        start with the latest stable.


        Cheers,
        Daniel

...

        Best Regards

        2012/7/31 Daniel-Constantin Mierla <miconda@gmail.com
        <mailto:miconda@gmail.com>>

            Hello,


            On 7/30/12 9:01 PM, Bruno Bresciani wrote:

...

            Hi,

            I compiled kamailio with MEMDBG = 0 because I
            didn't want the memory debug  in kamailio log, but
            I didn't know these turns on f_malloc and disabling
            q_malloc...

            the log messages related to memory operations can be
            controlled by global parameters memdbg and memlog.

...

            Probably the size of shared memory that I start
            kamailio is 32MB because I didn't gave a different
            -m parameter value, exist a command to verify this
            information?

            kamctl fifo get_statistics all

            and see the shared memory total value. It will be
            interesting to see available shared memory as well.

...

            At moment that crash happened, there were few
            registered users agents and were being made tests
            with register and calls with TLS protocol. I got
            only the two situations that I showed at first email.

            If you plan to start a new installation, I strongly
            recommend 3.3 branch, the code is more actual and
            easier to debug. 3.1 is no longer an official
            maintained branch, those being now 3.3 and 3.2. I'm
            looking at this issue to be sure it is no longer in
            latest stable.

            Cheers,
            Daniel

        --
        Daniel-Constantin Mierla -http://www.asipto.com
        http://twitter.com/#!/miconda  <http://twitter.com/#%21/miconda>  -http://www.linkedin.com/in/miconda
        Kamailio Advanced Training, Seattle, USA, Sep 23-26, 2012 -http://asipto.com/u/katu
        Kamailio Practical Workshop, Netherlands, Sep 10-12, 2012 -http://asipto.com/u/kpw

    --
    Daniel-Constantin Mierla -http://www.asipto.com
    http://twitter.com/#!/miconda  <http://twitter.com/#%21/miconda>  -http://www.linkedin.com/in/miconda




_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users@lists.sip-router.org  <mailto:sr-users@lists.sip-router.org>
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users

_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users@lists.sip-router.org <mailto:sr-users@lists.sip-router.org>
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users

---

SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users

Then it seems that this fix and is not related to your bug FS247. You should re-open the bug report.

regards Klaus

On 03.01.2013 17:21, Bruno Bresciani wrote:

My openSSL version is 0.9.8... According to http://openssl.6102.n7.nabble.com/Custom-free-routine-is-invoked-with-NULL-a... this was working fine in 0.9.8.

I was analysing the backtrace full with gdb and the line below shows that ptr was not a NULL parameter.

#4 0x0051a87f in ser_realloc (ptr=0xb61f0bc0, size=32) at ../../mem/shm_mem.h:266

ptr=0xb61f0bc0 size=32

Now, I'm a little confused...the changes to allow freeing of NULL pointer to behave like standard free () function will take effect to prevent the bug FS#247?

Best Regards

2013/1/3 Klaus Darilion <klaus.mailinglists@pernau.at mailto:klaus.mailinglists@pernau.at>

You can also check which openSSL version you are using. According to
http://openssl.6102.n7.nabble.__com/Custom-free-routine-is-__invoked-with-NULL-argument-in-__openssl-1-0-1-td25937.html
<http://openssl.6102.n7.nabble.com/Custom-free-routine-is-invoked-with-NULL-argument-in-openssl-1-0-1-td25937.html>
it happened somewhere between 1.0.0. and 1.0.1c

regards
klaus


On 03.01.2013 14:56, Bruno Bresciani wrote:

    Thank's for your reply Klaus,

    but I don't know if is exactly this patch that Daniel is referring,
    because the backtrace full with gdb showed that problem happened at
    ser_realloc function. I will insert this check to NULL pointer at
    ser_free and ser_realloc function and recompile kamailio 3.1.2
    because I
    can't update to version 3.3.3 for while.

    Best Regards

    2013/1/2 Klaus Darilion <klaus.mailinglists@pernau.at
    <mailto:klaus.mailinglists@pernau.at>
    <mailto:klaus.mailinglists@__pernau.at
    <mailto:klaus.mailinglists@pernau.at>>>


         I think Daniel is referring to this patch:
    http://git.sip-router.org/cgi-__bin/gitweb.cgi/sip-router/?a=__commit;h=__020acff35f8e9dfa62aba8678a781a__0f7bbb110b
    <http://git.sip-router.org/cgi-bin/gitweb.cgi/sip-router/?a=commit;h=020acff35f8e9dfa62aba8678a781a0f7bbb110b>

         This fix was done to devel and 3.3 branch and is included
    in latest
         3.3.3 release.

         regards
         Klaus

         Am 20.12.2012 19 <tel:20.12.2012%2019>
    <tel:20.12.2012%2019>:29, schrieb Bruno Bresciani:

             Hi,

             I refer the bug fix FS#247  that I open in 30 July 2012
        and you
             closed in 25 october 2012 with the information: "Reopen
        if the
             patch for free(0) didn't work.".

             What Release this fix was included?


             Best Regards

             2012/12/20 Daniel-Constantin Mierla <miconda@gmail.com
        <mailto:miconda@gmail.com>
             <mailto:miconda@gmail.com <mailto:miconda@gmail.com>>>


                 Hello,

                 if you refer to free(0) issue with libssl, I am not
        sure it
                 was ever put in 3.1 branch, as the last release was
        before
                 reporting the issue.

                 Cheers,
                 Daniel


                 On 12/20/12 6:01 PM, Bruno Bresciani wrote:

                     Hello Daniel,

                     What patch this bug was fixed?  This bug was
            open in 30 July
                     2012 and ultimate Release to 3.1.X (3.1.6) was
            released at
                     14 July 2012...

                     Best Regards

                     2012/8/1 Daniel-Constantin Mierla
            <miconda@gmail.com <mailto:miconda@gmail.com>
                     <mailto:miconda@gmail.com
            <mailto:miconda@gmail.com>>>


                         Hello,


                         On 7/31/12 3:39 PM, Bruno Bresciani wrote:

                             Hello,

                             I didn't know that log messages related
                to memory
                             operations can be controlled by global
                parameter, but I
                             like to know if is recommendable I
                recompile kamailio
                             using q_malloc (default) and not
                f_malloc... memory
                             operations using q_malloc is more
                reliable and avoid
                             problems or crashes or it is only more
                suitable for
                             debugging? What do you suggest Daniel?


                         q_malloc is more suitable for debugging.



                             The 'get_statistics all' command is
                avaliable by a
                             specific module? I run 'kamctl fifo
                get_statistics all'
                             and return '500 command
                'get_statistics' not available'


                         Do you have kex module loaded? What is the
            output of
                         'kamctl fifo which'?



                             For while is impossible to start a new
                installation,
                             first because I don't know how much
                time I will spend to
                             port and second because I am involved
                with other
                             developments and I have no time  to
                make this. I know
                             that 3.1 is no longer a official branch
                but now start a
                             new installation it's very very
                difficult, my in
                             intention is discover what caused the
                crash and if
                             exists a way to fix or prevent it.

                         Discovering may require additional patches,
            like more
                         debug messages in the C code, that's why is
            better to
                         start with the latest stable.


                         Cheers,
                         Daniel


                             Best Regards

                             2012/7/31 Daniel-Constantin Mierla
                <miconda@gmail.com <mailto:miconda@gmail.com>
                             <mailto:miconda@gmail.com
                <mailto:miconda@gmail.com>>>


                                 Hello,


                                 On 7/30/12 9:01 PM, Bruno Bresciani
                wrote:

                                     Hi,

                                     I compiled kamailio with MEMDBG
                    = 0 because I
                                     didn't want the memory debug
                      in kamailio log, but
                                     I didn't know these turns on
                    f_malloc and disabling
                                     q_malloc...

                                 the log messages related to memory
                operations can be
                                 controlled by global parameters
                memdbg and memlog.



                                     Probably the size of shared
                    memory that I start
                                     kamailio is 32MB because I
                    didn't gave a different
                                     -m parameter value, exist a
                    command to verify this
                                     information?


                                 kamctl fifo get_statistics all

                                 and see the shared memory total
                value. It will be
                                 interesting to see available shared
                memory as well.



                                     At moment that crash happened,
                    there were few
                                     registered users agents and
                    were being made tests
                                     with register and calls with
                    TLS protocol. I got
                                     only the two situations that I
                    showed at first email.


                                 If you plan to start a new
                installation, I strongly
                                 recommend 3.3 branch, the code is
                more actual and
                                 easier to debug. 3.1 is no longer
                an official
                                 maintained branch, those being now
                3.3 and 3.2. I'm
                                 looking at this issue to be sure it
                is no longer in
                                 latest stable.

                                 Cheers,
                                 Daniel



                         --
                         Daniel-Constantin Mierla -http://www.asipto.com
            http://twitter.com/#!/miconda
              <http://twitter.com/#%21/__miconda
            <http://twitter.com/#%21/miconda>>
              -http://www.linkedin.com/in/__miconda
            <http://www.linkedin.com/in/miconda>
                         Kamailio Advanced Training, Seattle, USA,
            Sep 23-26, 2012 -http://asipto.com/u/katu

                         Kamailio Practical Workshop, Netherlands,
            Sep 10-12, 2012 -http://asipto.com/u/kpw



                 --
                 Daniel-Constantin Mierla -http://www.asipto.com
        http://twitter.com/#!/miconda
          <http://twitter.com/#%21/__miconda
        <http://twitter.com/#%21/miconda>>
          -http://www.linkedin.com/in/__miconda
        <http://www.linkedin.com/in/miconda>





             _________________________________________________
             SIP Express Router (SER) and Kamailio (OpenSER) -
        sr-users mailing list
        sr-users@lists.sip-router.org
        <mailto:sr-users@lists.sip-router.org>
          <mailto:sr-users@lists.sip-__router.org
        <mailto:sr-users@lists.sip-router.org>>
        http://lists.sip-router.org/__cgi-bin/mailman/listinfo/sr-__users
        <http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users>



         _________________________________________________
         SIP Express Router (SER) and Kamailio (OpenSER) - sr-users
    mailing list
    sr-users@lists.sip-router.org
    <mailto:sr-users@lists.sip-router.org>
    <mailto:sr-users@lists.sip-__router.org
    <mailto:sr-users@lists.sip-router.org>>

    http://lists.sip-router.org/__cgi-bin/mailman/listinfo/sr-__users <http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users>




    _________________________________________________
    SIP Express Router (SER) and Kamailio (OpenSER) - sr-users
    mailing list
    sr-users@lists.sip-router.org <mailto:sr-users@lists.sip-router.org>
    http://lists.sip-router.org/__cgi-bin/mailman/listinfo/sr-__users <http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users>

Hello,

do you have a backtrace that you can send just to compare with the one from Bruno?

Also, send the output of 'kamailio -V' to look at compile flags. Bruno had a custom compile time flag, a matter of that a different memory manager was used comparing with the default one for 3.1 series. If you have the default one, it may include more info that can give some hints.

Cheers, Daniel

On 7/18/12 9:03 PM, Renan Capaverde wrote:

I am having the same problem on kamailio 3.1.6. Someone please help.

Cheers, Renan

Em 17/7/2012 14:26, Bruno Bresciani escreveu:

...

Excuse-me Daniel, I forgot this information... Kamailio version 3.1.2

Best Regards

2012/7/17 Daniel-Constantin Mierla <miconda@gmail.com mailto:miconda@gmail.com>

Hello,

send also the version 'kamailio -V' to match the proper sources.

Cheers,
Daniel


On 7/17/12 7:11 PM, Bruno Bresciani wrote:

...

Hi all,

I observed two at 0x08173bf0 in fm_realloc (qm=0xb5f7d000,
p=0xb61f0bc0, size=0) at mem/f_malloc.c:536.

I don't know exactly what may be caused this core because the
debug of kamailio is disabled, but looking at backtrace (gdb)
and source code I could notice a relationship with TLS
connection establishment (handshake). Other important fact is
that core was generated by the kamalio itself with signal 6
(abort) because it can't find the pointer to memory realloc.
I'd like to know if someone already observed this issue and if
my analyse above is correct?

Bellow is backtrace full with gdb:

Core was generated by `/home2/local/kamailio/sbin/kamailio -P
/var/run/kamailio.pid'.
Program terminated with signal 6, Aborted.
#0  0x00f30402 in __kernel_vsyscall ()
(gdb)
(gdb)
(gdb)
(gdb)
(gdb) bt full
#0  0x00f30402 in __kernel_vsyscall ()
No symbol table info available.
#1  0x009fec10 in raise () from /lib/libc.so.6
No symbol table info available.
#2  0x00a00521 in abort () from /lib/libc.so.6
No symbol table info available.
#3  0x08173bf0 in fm_realloc (qm=0xb5f7d000, p=0xb61f0bc0,
size=0) at mem/f_malloc.c:536
        f = (struct fm_frag *) 0xb61f0bb8
        pf = <value optimized out>
        orig_size = 18
L=Ãb¬Ã2n = (struct fm_frag *) 0xb61f0bd2
        ptr = <value optimized out>
        hash = 23548
#4  0x0051a87f in ser_realloc (ptr=0xb61f0bc0, size=32) at
../../mem/shm_mem.h:266
No locals.
#5  0x00460295 in OPENSSL_cleanse () from /lib/libcrypto.so.6
No symbol table info available.
#6  0x004609f7 in CRYPTO_realloc () from /lib/libcrypto.so.6
No symbol table info available.
#7  0x003efb22 in sk_insert () from /lib/libcrypto.so.6
No symbol table info available.
#8  0x003efbaa in sk_push () from /lib/libcrypto.so.6
No symbol table info available.
#9  0x007c2ac3 in ssl_bytes_to_cipher_list () from /lib/libssl.so.6
No symbol table info available.
#10 0x007aa9f3 in ssl3_get_client_hello () from /lib/libssl.so.6
No symbol table info available.
#11 0x007ab815 in ssl3_accept () from /lib/libssl.so.6
No symbol table info available.
#12 0x007c1c3a in SSL_accept () from /lib/libssl.so.6
No symbol table info available.
#13 0x005278e3 in tls_accept (c=0xb621af88, error=0xbf91dfc8) at
tls_server.c:346
        ret = <value optimized out>
        ssl = (SSL *) 0xb61c1878
        cert = <value optimized out>
        tls_c = (struct tls_extra_data *) 0xb6208e58
        tls_log = <value optimized out>
#14 0x0052da8e in tls_read_f (c=0xb621af88, flags=0xbf91e200) at
tls_server.c:1028
        r = (struct tcp_req *) 0xb621affc
        bytes_free = 4095
        bytes_read = 70
        read_size = 4095
        ssl_error = 0
        ssl_read = 0
        ssl = (SSL *) 0xb61c1878
        rd_buf =
"\026\003\001\000A\001\000\000=\003\001Oÿ\v¡ywP\fkÃDÃ
0/\000\a\000\005\001\000ÃÃÃï000\000\026\0009\0008\0005\000\026\000\023\000\n\0003\0002\00
202i<ä<åþ\031ì234 \237ê030Y){lѶ"...06Ã'Ãr9ìë¸{[©\t\205\037\036ÿHPH5µ\t(é32Ã027æò036¾\"\rÔ¤YÃ215uº\027)ö000ñ215\vÃà éEa\023xéJ\206T\204CÃ
wr_buf =
"\027\003\001\003 w²'\016ѼÃ033ÃLÃ021Ã)Ã\220hnL¦\020Xê\236\200\tIhøSf\234Ã\rÃ
?¯¼\024FÃ\200ª/\2264RÃaäIpXȬÃ035äQh\211Ã026ý4ÃU+\asr£/`¤Oû1AʹqUܿ·v±ü\205"...
        rd = {buf = 0xbf90dfa4 "\026\003\001", pos = 70, used =
70, size = 65536}
        wr = {
  buf = 0xbf8fdfa4
"\027\003\001\003 w²'\016ѼÃ033ÃLÃ021Ã)Ã\220hnL¦\020Xê\236\200\tIhøSf\234Ã\rÃ
?¯¼\024FÃ\200ª/\2264RÃaäIpXȬÃ035äQh\211Ã026ý4ÃU+\asr£/`¤Oû1AʹqUܿ·v±ü\205"...,
pos = 0, used = 0,
size = 65536}
        tls_c = (struct tls_extra_data *) 0xb6208e58
        enc_rd_buf = (struct tls_rd_buf *) 0x0
        n = 0
        flush_flags = <value optimized out>
        err_src = 0x541a83 "TLS read:"
        x = <value optimized out>
#15 0x08158ade in tcp_read_headers (c=0xb621af88,
read_flags=0xbf91e200) at tcp_read.c:406
        bytes = -1
        remaining = <value optimized out>
        p = <value optimized out>
        r = (struct tcp_req *) 0xb621affc
#16 0x08158fd4 in tcp_read_req (con=0xb621af88,
bytes_read=0xbf91e204, read_flags=0xbf91e200) at tcp_read.c:871
        bytes = <value optimized out>
        total_bytes = 0
        resp = <value optimized out>
        size = <value optimized out>
        req = (struct tcp_req *) 0xb621affc
        dst = {send_sock = 0x1, to = {s = {sa_family = 57540,
sa_data = "\221¿\020\000\000\000\000\000\000\000\020\000\000"},
sin = {sin_family = 57540,
      sin_port = 49041, sin_addr = {s_addr = 16}, sin_zero =
"\000\000\000\000\020\000\000"}, sin6 = {sin6_family = 57540,
sin6_port = 49041,
      sin6_flowinfo = 16, sin6_addr = {in6_u = {u6_addr8 =
"\000\000\000\000\020\000\000\000\001\000\000\000\001\000\000",
u6_addr16 = {0, 0, 16, 0, 1, 0,
            1, 0}, u6_addr32 = {0, 16, 1, 1}}}, sin6_scope_id =
12}}, id = -1080958468, proto = 4 '\004', send_flags = {f = 0
'\0', blst_imask = 0 '\0'}}
        c = 10 '\n'
#17 0x08159a0f in handle_io (fm=0x83296b0, events=1, idx=-1) at
tcp_read.c:1150
        ret = 4
        n = 4
        read_flags = 1
        con = (struct tcp_connection *) 0xb621af88
        s = 12
        resp = <value optimized out>
        t = <value optimized out>
#18 0x0815d653 in tcp_receive_loop (unix_sock=35) at io_wait.h:1091
No locals.
---Type <return> to continue, or q <return> to quit---
#19 0x0812885b in tcp_init_children () at tcp_main.c:4819
        r = 3
        reader_fd_1 = 35
        pid = <value optimized out>
        si = <value optimized out>
#20 0x080ad412 in main_loop () at main.c:1632
        i = 4
        pid = <value optimized out>
        si = (struct socket_info *) 0x0
        si_desc = "udp receiver child=3 sock=192.168.166.31:5060
<http://192.168.166.31:5060>\000\221¿\002\000\000\000/\200\220\000\001\000\000\000\000\000H\000¨ä21¿\001\000\000\000°¶è026¶\000\000\000\000\000õ\030s2\b\002\000\000\000âf\b\000õ\001\000\000\000\000\000\000\000\001\000\000\000\000\000\000\000ÿÿÿÿ¸ä21¿"
#21 0x080afbe7 in main (argc=3, argv=dwarf2_read_address:
Corrupted DWARF expression.
) at main.c:2398
        cfg_stream = (FILE *) 0x8a2d008
        c = <value optimized out>
        r = 0
        tmp = 0xbf91e674
"\212þ\221¿\233þ\221¿°þ\221¿»þ\221¿Ã\221¿ù1¿\020ÿ\221¿Dÿ\221¿Lÿ\221¿Wÿ\221¿]ÿ\221¿oÿ\221¿{ÿ\221¿\202ÿ\221
tmp_len = 10492229
        port = 136192361
        proto = -1080957480
        ret = <value optimized out>
        seed = 21782655
        rfd = 4
        debug_save = <value optimized out>
        debug_flag = 0
        dont_fork_cnt = 0
        n_lst = <value optimized out>
        p = <value optimized out>


Best Regards


_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users@lists.sip-router.org  <mailto:sr-users@lists.sip-router.org>
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users

-- 
Daniel-Constantin Mierla -http://www.asipto.com
http://twitter.com/#!/miconda  <http://twitter.com/#%21/miconda>  -http://www.linkedin.com/in/miconda
Kamailio Advanced Training, Seattle, USA, Sep 23-26, 2012 -http://asipto.com/u/katu
Kamailio Practical Workshop, Netherlands, Sep 10-12, 2012 -http://asipto.com/u/kpw

---

SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users

---

SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users