Re: [SR-Users] Crash Kamailio 4.1.4

Hello,

not really troubleshooting, as I got traveling, but I thought a bit of it, also from performances point of view.

As it was related to parallel processing of 1xx/2xx sip responses that you are accounting, I think the safest solutions would be:

- clone the message locally to acc if it is the tm (share memory) - do synchronized access to tm data inside the acc (use locks there)

Instead of using locks or other workarounds in tm (lower performances in all cases), better do it on the part doing parallel access out of tm. Acc is doing some particular processing -- it uses a callback for a reply to access the response, so it is quite specific to it.

If you want to be safe side immediately, store the PAI in an avp inside request_route and use that avp for accounting.

A solution will be there before 4.2 is out.

Cheers, Daniel

On 02/10/14 09:50, Igor Potjevlesch wrote:

Hello Daniel,

Have you had the opportunity to look at this?

Thank you for your feedback.

Regards,

Igor.

*De :*Daniel-Constantin Mierla [mailto:miconda@gmail.com] *Envoyé :* mercredi 24 septembre 2014 09:00 *À :* Igor Potjevlesch *Cc :* 'Kamailio (SER) - Users Mailing List' *Objet :* Re: [SR-Users] Crash Kamailio 4.1.4

Hello,

so it is still related to PAI header -- I will have more time to look at it by end of the week.

Cheers, Daniel

On 23/09/14 19:16, Igor Potjevlesch wrote:

Hello Daniel,

Patching has been done in the same time than the upgrade to 4.1.5.

A new crash occurred in pvapi.c (in addition of my other recent
post “Crash Kamailio 4.1.5”).

Here is the result of a “bt full”:

#0  0x000000000049580e in pv_get_strval (msg=0x7f14efe24ea8,
param=0x7f14fb65d058, res=0x7fff6c6b7430, sval=0x22) at pvapi.c:521

No locals.

#1  0x00007f14f9677f2b in pv_get_pai (msg=0x7f14efe24ea8,
param=0x7f14fb65d058, res=0x7fff6c6b7430) at pv_core.c:1026

        idxf = 0

        idx = 0

        pai_body = 0x7f14fb679d38

        pai_uri = 0xa

        i = 0

        cur_id = 0

        __FUNCTION__ = "pv_get_pai"

#2  0x0000000000499594 in pv_get_spec_value (msg=0x7f14efe24ea8,
sp=0x7f14fb65d040, value=0x7fff6c6b7430) at pvapi.c:1266

        ret = 0

        __FUNCTION__ = "pv_get_spec_value"

#3  0x00007f14f7d3481d in extra2strar (extra=0x7f14fb65d030,
rq=0x7f14efe24ea8, val_arr=0x7f14f7f41e30, int_arr=0x7f14f7f4237c,
type_arr=0x7f14f7f424e7 "\002\002\002\002") at acc_extra.c:261

        value = {rs = {s = 0x0, len = 0}, ri = 0, flags = 0}

        n = 4

        r = 0

        __FUNCTION__ = "extra2strar"

#4  0x00007f14f7d2c3e3 in acc_db_request (rq=0x7f14efe24ea8) at
acc.c:474

        m = 7

        n = -270381400

        i = 6

        t = 0x414cc0

        __FUNCTION__ = "acc_db_request"

#5  0x00007f14f7d36bc8 in acc_onreply (t=0x7f14efe525b8,
req=0x7f14efe24ea8, reply=0x7f14fb670c48, code=200) at acc_logic.c:471

        new_uri_bk = {s = 0x7f14efe25590
"sip:ABCDEFGHIJ@<IP_GW>oTE sINVITE
sip:ABCDEFGHIJ@sip.fqdn.tldSIP/2.0\r\nVia: SIP/2.0/UDP
<IP_UAC>:5060;branch=z9hG4bK5f32deec\r\nMax-Forwards: 69\r\nFrom:
\"KLMNOPQRST\" <sip:KLMNOPQRST@sip.fqdn."..., len = 19}

        br = 0

        hdr = 0x7f14f7d3dc20

        __FUNCTION__ = "acc_onreply"

#6  0x00007f14f7d3730a in tmcb_func (t=0x7f14efe525b8, type=512,
ps=0x7fff6c6b76b0) at acc_logic.c:573

        __FUNCTION__ = "tmcb_func"

#7  0x00007f14f9f3146c in run_trans_callbacks_internal
(cb_lst=0x7f14efe52628, type=512, trans=0x7f14efe525b8,
params=0x7fff6c6b76b0) at t_hooks.c:290

        cbp = 0x7f14ee4c81b0

        backup_from = 0x934630

        backup_to = 0x934638

        backup_dom_from = 0x934640

        backup_dom_to = 0x934648

        backup_uri_from = 0x934620

        backup_uri_to = 0x934628

        backup_xavps = 0x934760

        __FUNCTION__ = "run_trans_callbacks_internal"

#8  0x00007f14f9f3167e in run_trans_callbacks_with_buf (type=512,
rbuf=0x7f14efe52678, req=0x7f14efe24ea8, repl=0x7f14fb670c48,
flags=183) at t_hooks.c:336

        params = {req = 0x7f14efe24ea8, rpl = 0x7f14fb670c48,
param = 0x7f14ee4c81c0, code = 200, flags = 183, branch = 0,
t_rbuf = 0x7f14efe52678, dst = 0x7f14efe526c8, send_buf = {

            s = 0x7f14efd7c408 "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP
<IP_UAC>:5060;rport=5060;branch=z9hG4bK5f32deec\r\nCall-ID:
7846f7332ce6a7db4484c3d06ce1c387@sip.fqdn.tld\r\nFrom
<mailto:7846f7332ce6a7db4484c3d06ce1c387@sip.fqdn.tld%5Cr%5CnFrom>: \"KLMNOPQRST\"
<sip:KLMNOPQRST@sip.fqdn.tld> <sip:KLMNOPQRST@sip.fqdn.tld>"...,
len = 980}}

        trans = 0x7f14efe525b8

#9  0x00007f14f9f63bfa in relay_reply (t=0x7f14efe525b8,
p_msg=0x7f14fb670c48, branch=0, msg_status=183,
cancel_data=0x7fff6c6b7a10, do_put_on_wait=1) at t_reply.c:2001

        relay = 0

        save_clone = 0

        buf = 0x7f14fb67e740 "SIP/2.0 183 Session Progress\r\nVia:
SIP/2.0/UDP
<IP_UAC>:5060;rport=5060;branch=z9hG4bK5f32deec\r\nCall-ID:
7846f7332ce6a7db4484c3d06ce1c387@sip.fqdn.tld\r\nFrom
<mailto:7846f7332ce6a7db4484c3d06ce1c387@sip.fqdn.tld%5Cr%5CnFrom>: \"KLMNOPQRST\"
<sip:KLMNOPQRST@tru"...

        res_len = 777

        relayed_code = 183

        relayed_msg = 0x7f14fb670c48

        reply_bak = 0x414cc0

        bm = {to_tag_val = {s = 0x7f14efe53b50 "", len = -79437808}}

        totag_retr = 0

        reply_status = RPS_PROVISIONAL

        uas_rb = 0x7f14efe52678

        to_tag = 0x0

        reason = {s = 0x800000001 <Address 0x800000001 out of
bounds>, len = 1}

        onsend_params = {req = 0x200924970, rpl = 0x7f14f9f83f90,
param = 0x414cc0, code = 1818984640, flags = 3, branch = 0, t_rbuf
= 0x0, dst = 0x7f14fb670e40, send_buf = {s = 0x7fff6c6b7830
"`xkl\377\177", len = -101469275}}

        __FUNCTION__ = "relay_reply"

#10 0x00007f14f9f660ab in reply_received (p_msg=0x7f14fb670c48) at
t_reply.c:2499

        msg_status = 183

        last_uac_status = 183

        ack = 0x40 <Address 0x40 out of bounds>

        ack_len = 0

        branch = 0

        reply_status = -77092928

        onreply_route = 1

        cancel_data = {cancel_bitmap = 0, reason = {cause = 0, u =
{text = {s = 0x0, len = 9586205}, e2e_cancel = 0x0, packed_hdrs =
{s = 0x0, len = 9586205}}}}

        uac = 0x7f14efe52720

        t = 0x7f14efe525b8

        lack_dst = {send_sock = 0x7f14fb584420, to = {s =
{sa_family = 57360, sa_data =
"C\373\024\177\000\000\000F#\000\000\000\000"}, sin = {sin_family
= 57360, sin_port = 64323, sin_addr = {s_addr = 32532}, sin_zero =
"\000F#\000\000\000\000"}, sin6 = {

              sin6_family = 57360, sin6_port = 64323,
sin6_flowinfo = 32532, sin6_addr = {__in6_u = {__u6_addr8 =
"\000F#\000\000\000\000\000\020\341C\373\024\177\000", __u6_addr16
= {17920, 35, 0, 0, 57616, 64323, 32532, 0}, __u6_addr32 =
{2311680, 0, 4215529744,

                    32532}}}, sin6_scope_id = 4215529744}}, id =
32532, proto = 96 '`', send_flags = {f = 64 '@', blst_imask = 103
'g'}}

        backup_user_from = 0x934630

        backup_user_to = 0x934638

        backup_domain_from = 0x934640

       backup_domain_to = 0x934648

        backup_uri_from = 0x934620

        backup_uri_to = 0x934628

        backup_xavps = 0x934760

        replies_locked = 1

        branch_ret = 0

        prev_branch = 1818983120

        blst_503_timeout = 32767

        hf = 0x7f14fb670c68

        onsend_params = {req = 0x7fff6c6b7a90, rpl = 0x550bb0,
param = 0x234540, code = 0, flags = 3, branch = 0, t_rbuf =
0x7f14fb43e380, dst = 0x7f14fb674030, send_buf = {s =
0x7fff6c6b7a90 "`G\223", len = 5538065}}

        ctx = {rec_lev = 0, run_flags = 0, last_retcode = 0,
jmp_env = {{__jmpbuf = {139728093908544, 1898006658898931560,
4279488, 140735012372672, 0, 0, 1898006658936680296,
-1897762211976106136}, __mask_was_saved = 0, __saved_mask = {__val
= {9586373,

                  1365809186688, 124554051613, 9586450,
139728093947840, 9587056, 9586211, 361695345073193192, 9586309,
9586288, 4217874320, 139728093947840, 139728093942016,
139728093908544, 4279488, 140735012372672}}}}}

        __FUNCTION__ = "reply_received"

#11 0x000000000045d853 in do_forward_reply (msg=0x7f14fb670c48,
mode=0) at forward.c:777

        new_buf = 0x0

        dst = {send_sock = 0x0, to = {s = {sa_family = 0, sa_data
= '\000' <repeats 13 times>}, sin = {sin_family = 0, sin_port = 0,
sin_addr = {s_addr = 0}, sin_zero =
"\000\000\000\000\000\000\000"}, sin6 = {sin6_family = 0,
sin6_port = 0, sin6_flowinfo = 0,

              sin6_addr = {__in6_u = {__u6_addr8 = '\000' <repeats
15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 =
{0, 0, 0, 0}}}, sin6_scope_id = 0}}, id = 0, proto = 0 '\000',
send_flags = {f = 0 '\000', blst_imask = 0 '\000'}}

        new_len = 32532

        r = 1

        s = 0x370fb670c50 <Address 0x370fb670c50 out of bounds>

        len = 0

        __FUNCTION__ = "do_forward_reply"

#12 0x000000000045e114 in forward_reply (msg=0x7f14fb670c48) at
forward.c:860

No locals.

#13 0x00000000004a5903 in receive_msg (buf=0x924600 "SIP/2.0 183
Session Progress\r\nVia: SIP/2.0/UDP
<IP_KAMAILIO>;branch=z9hG4bKca48.51c2c569361ea0fedd9c6c70c21b5eed.0;received=<IP_KAMAILIO>\r\nVia:
SIP/2.0/UDP <IP_UAC>:5060;rport=5060;branch=z9hG4bK5f32d"...,

    len=880, rcv_info=0x7fff6c6b7d90) at receive.c:273

        msg = 0x7f14fb670c48

        ctx = {rec_lev = 8868984, run_flags = 0, last_retcode = 0,
jmp_env = {{__jmpbuf = {0, 0, 0, 263853236176, 1, 0, 171719254808,
9586112}, __mask_was_saved = 1818983832, __saved_mask = {__val =
{139728091862768, 12884901899, 139728091862768, 4279488,

                  140735012372672, 140735012371728, 5477982, 0,
139727728366976, 50195, 171356018048, 9586112, 140735012371856,
140735012371776, 5474817, 4279488}}}}}

        ret = 32532

        inb = {s = 0x924600 "SIP/2.0 183 Session Progress\r\nVia:
SIP/2.0/UDP
<IP_KAMAILIO>;branch=z9hG4bKca48.51c2c569361ea0fedd9c6c70c21b5eed.0;received=<IP_KAMAILIO>\r\nVia:
SIP/2.0/UDP <IP_UAC>:5060;rport=5060;branch=z9hG4bK5f32d"..., len
= 880}

        __FUNCTION__ = "receive_msg"

#14 0x000000000053c9c4 in udp_rcv_loop () at udp_server.c:536

        len = 880

        buf = "SIP/2.0 183 Session Progress\r\nVia: SIP/2.0/UDP
<IP_KAMAILIO>;branch=z9hG4bKca48.51c2c569361ea0fedd9c6c70c21b5eed.0;received=<IP_KAMAILIO>\r\nVia:
SIP/2.0/UDP <IP_UAC>:5060;rport=5060;branch=z9hG4bK5f32d"...

        tmp = 0x9245c0 "10.143.1.10"

        from = 0x7f14fb5add70

        fromlen = 16

        ri = {src_ip = {af = 2, len = 4, u = {addrl = {403182777,
139728091862768}, addr32 = {403182777, 0, 4215789296, 32532},
addr16 = {5305, 6152, 0, 0, 55024, 64327, 32532, 0}, addr =
"\271\024\b\030\000\000\000\000\360\326G\373\024\177\000"}},
dst_ip = {af = 2,

            len = 4, u = {addrl = {67638457, 0}, addr32 =
{67638457, 0, 0, 0}, addr16 = {5305, 1032, 0, 0, 0, 0, 0, 0}, addr
= "\271\024\b\004", '\000' <repeats 11 times>}}, src_port = 5060,
dst_port = 5060, proto_reserved1 = 0, proto_reserved2 = 0, src_su
= {s = {

              sa_family = 2, sa_data =
"\023Ĺ\024\b\030\000\000\000\000\000\000\000"}, sin = {sin_family
= 2, sin_port = 50195, sin_addr = {s_addr = 403182777}, sin_zero =
"\000\000\000\000\000\000\000"}, sin6 = {sin6_family = 2,
sin6_port = 50195,

              sin6_flowinfo = 403182777, sin6_addr = {__in6_u =
{__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0,
0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 0}},
bind_address = 0x7f14fb47d588, proto = 1 '\001'}

        __FUNCTION__ = "udp_rcv_loop"

#15 0x000000000046d447 in main_loop () at main.c:1617

        i = 13

        pid = 0

        si = 0x7f14fb47d588

        si_desc = "udp receiver child=13
sock=<IP_KAMAILIO>:5060\000\373\024\177\000\000\b$P\373\024\177\000\000\036\205^\000\000\000\000\000\000w^\000\000\000\000\000\000\244\303v\000\000\000\000\300LA\000\000\000\000\000\300\200kl\377\177",
'\000' <repeats 19 times>,
"\177kl\377\177\000\000\020\245K\000\000\000\000"

        nrprocs = 15

        __FUNCTION__ = "main_loop"

#16 0x000000000047054f in main (argc=7, argv=0x7fff6c6b80c8) at
main.c:2545

        cfg_stream = 0xf42010

        c = -1

        r = 0

tmp = 0x7fff6c6b8f70 ""

tmp_len = 0

port = 0

proto = 0

options = 0x5e0a58
":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:"

ret = -1

        seed = 3572644655

        rfd = 4

        debug_save = 0

        debug_flag = 0

        dont_fork_cnt = 0

        n_lst = 0x3d6f60fb88

        p = 0x5caba0 "H\211l$\330L\211d$\340H\215-O\244*"

        __FUNCTION__ = "main"

Let me know if you need further information.

Regards,

Igor.

*De :*Daniel-Constantin Mierla [mailto:miconda@gmail.com]
*Envoyé :* mercredi 6 août 2014 16:39
*À :* Igor Potjevlesch
*Cc :* 'Kamailio \(SER\) - Users Mailing List'
*Objet :* Re: [SR-Users] Crash Kamailio 4.1.4

Hello,

it is not in my plans for 4.1.5, because I didn't get any feedback
on testing and its results, whether it fixes or not the issue.

Cheers,
Daniel

On 06/08/14 16:07, Igor Potjevlesch wrote:

    Hello Daniel,

    Thank you for this exhaustive feedback.

    Do you include the patch to 4.1.5?

    Regards,

    Igor.

    *De :*Daniel-Constantin Mierla [mailto:miconda@gmail.com]
    *Envoyé :* lundi 4 août 2014 16:24
    *À :* Igor Potjevlesch
    *Cc :* Kamailio \(SER\) - Users Mailing List
    *Objet :* Re: [SR-Users] Crash Kamailio 4.1.4

    Hello,

    the problem was that a structure in shared memory (the request
    cloned in tm) could have been used in parallel by different
    kamailio processes.

    If there were two processes at the same time, parsing PAI
    resulted in setting the header pointer to a private memory.
    The other process could overtake in processing, using the same
    cloned request, and this time the pai pointer is set, but to
    another private memory zone. I added the locks for calling the
    callbacks, so the process that parse the PAI is the one
    cleaning it.

    Performances should not be impacted that much, the transaction
    lock is used and will add sequential processing when there are
    two replies at the same time, which is not the common.

    Cheers,
    Daniel

    On 07/07/14 12:40, Igor Potjevlesch wrote:

        Hello,

        Can you explain the modification and the impact on our
        plateform?

        Is it for the pai problem?

        Do you have explanation for the km_val.c problem wich
        cause crash for Kamailio too?

        Regards,

        Igor

        2014-07-01 16:40 GMT+02:00 Daniel-Constantin Mierla
        <miconda@gmail.com <mailto:miconda@gmail.com>>:

            Hello,

            can you give it a try with the patch from next commit?

            -
            http://git.sip-router.org/cgi-bin/gitweb.cgi/sip-router/?a=commit;h=da9d56be28e050dd0cb4aed50efcbda043a3e5cf

            If all goes fine while testing, I will backport.

            Cheers,
            Daniel

            On 26/06/14 12:58, Igor Potjevlesch wrote:

                Hello,

                Here the result :

                (gdb) frame 6


                #6  0x00007f127cb6dde6 in acc_onreply
                (t=0x7f1274c157f0, req=0x7f1274c3ac08,
                    reply=0x7f12804a6d70, code=200) at acc_logic.c:501

                501 clean_hdr_field(hdr);
                (gdb) print hdr
                $1 = (hdr_field_t *) 0x7f1274c3c238
                (gdb) print *hdr
                $2 = {type = HDR_PAI_T, name = {
                    s = 0x7f1274c3b6cd "P-Asserted-Identity:
                <sip:0123456789@domain;user=phone>\r\nP-Sig-Options:
                Sending-Complete\r\n\r\nv=0\r\no=- 111851 1 IN IP4
                A.B.C.D\r\ns=-\r\nt=0 0\r\nm=audio 21336 RTP/AVP 8
                101 13\r\nc=IN IP4 A.B.C"..., len = 19}, body = {
                    s = 0x7f1274c3b6e2
                "<sip:0123456789@domain;user=phone>\r\nP-Sig-Options:
                Sending-Complete\r\n\r\nv=0\r\no=- 111851 1 IN IP4
                A.B.C.D\r\ns=-\r\nt=0 0\r\nm=audio 21336 RTP/AVP 8
                101 13\r\nc=IN IP4 A.B.C.D\r\na=rtpmap:101
                tele"..., len = 44}, len = 67, parsed = 0x0, next
                = 0x7f1274c3c278}

                (gdb) frame 4


                #4  0x000000000056e5e6 in free_pai_ppi_body
                (pid_b=0x7f12803cb480)
                    at parser/parse_ppi_pai.c:102

                102 pkg_free(pid_b);
                (gdb) print *pid_b
                $3 = {id = 0x0, num_ids = 0, next = 0x1d0}

                This is the bt full :



                #0  0x0000003d6f6328a5 in raise () from
                /lib64/libc.so.6
                No symbol table info available.
                #1  0x0000003d6f634085 in abort () from
                /lib64/libc.so.6
                No symbol table info available.
                #2  0x0000000000546d3c in qm_debug_frag
                (qm=0x7f1280275010, f=0x7f12803cb450) at
                mem/q_malloc.c:142

                        __FUNCTION__ = "qm_debug_frag"


                #3  0x0000000000548b26 in qm_free
                (qm=0x7f1280275010, p=0x7f12803cb480,
                file=0x6276a0 "<core>: parser/parse_ppi_pai.c",
                func=0x627a00 "free_pai_ppi_body", line=102) at
                mem/q_malloc.c:464

                        f = 0x7f12803cb450
                        size = 139717434027144
                        next = 0xf00000000
                        prev = 0x7f127cd79e00
                        __FUNCTION__ = "qm_free"


                #4  0x000000000056e5e6 in free_pai_ppi_body
                (pid_b=0x7f12803cb480) at parser/parse_ppi_pai.c:102

                        __FUNCTION__ = "free_pai_ppi_body"


                #5  0x000000000054fee0 in clean_hdr_field
                (hf=0x7f1274c3c238) at parser/hf.c:126

                        h_parsed = 0x7f1274c3c268
                        __FUNCTION__ = "clean_hdr_field"


                #6  0x00007f127cb6dde6 in acc_onreply
                (t=0x7f1274c157f0, req=0x7f1274c3ac08,
                reply=0x7f12804a6d70, code=200) at acc_logic.c:501

                        new_uri_bk = {s = 0x7f1274b53cdf
                "sip:0987654321@GW SIP/2.0\r\nRecord-Route:
                <sip:A.B.C.D;lr=on>\r\nVia: SIP/2.0/UDP
                A.B.C.D;branch=z9hG4bK512b.82b197888826f6b60c0c63b79801294d.0\r\nVia:
                SIP/2.0/UDP A.B.C.D:2057;branch=z9hG4bK-12
                <sip:0987654321@GW%20SIP/2.0%5Cr%5CnRecord-Route:%20%3csip:A.B.C.D;lr=on%3e%5Cr%5CnVia:%20SIP/2.0/UDP%20A.B.C.D;branch=z9hG4bK512b.82b197888826f6b60c0c63b79801294d.0%5Cr%5CnVia:%20SIP/2.0/UDP%20A.B.C.D:2057;branch=z9hG4bK-12>"...,
                len = 19}
                        br = 0
                        hdr = 0x7f1274c3c238
                        __FUNCTION__ = "acc_onreply"


                #7  0x00007f127cb6e30a in tmcb_func
                (t=0x7f1274c157f0, type=512, ps=0x7fff0b015580) at
                acc_logic.c:573

                        __FUNCTION__ = "tmcb_func"


                #8  0x00007f127ed68478 in
                run_trans_callbacks_internal
                (cb_lst=0x7f1274c15860, type=512,
                trans=0x7f1274c157f0, params=0x7fff0b015580) at
                t_hooks.c:290

                        cbp = 0x7f1274ac0e90
                        backup_from = 0x934630
                        backup_to = 0x934638
                        backup_dom_from = 0x934640
                        backup_dom_to = 0x934648
                        backup_uri_from = 0x934620
                        backup_uri_to = 0x934628
                        backup_xavps = 0x934760
                        __FUNCTION__ = "run_trans_callbacks_internal"


                #9  0x00007f127ed6868a in
                run_trans_callbacks_with_buf (type=512,
                rbuf=0x7f1274c158b0, req=0x7f1274c3ac08,
                repl=0x7f12804a6d70, flags=200) at t_hooks.c:336

                        params = {req = 0x7f1274c3ac08, rpl =
                0x7f12804a6d70, param = 0x7f1274ac0ea0, code =
                200, flags = 200, branch = 0, t_rbuf =
                0x7f1274c158b0, dst = 0x7f1274c15900, send_buf = {
                            s = 0x7f1274c27620 "SIP/2.0 200
                OK\r\nVia: SIP/2.0/UDP
                A.B.C.D:2057;branch=z9hG4bK-129F259C;rport=2057\r\nCall-ID:
                cb03dc02e909d3118f86009033290024@A.B.C.D\r\nFrom
                <mailto:cb03dc02e909d3118f86009033290024@A.B.C.D%5Cr%5CnFrom>:
                <sip:0123456789@domain;user=phone>;epid=00903"...,
                len = 1021}}
                        trans = 0x7f1274c157f0


                #10 0x00007f127ed9ac06 in relay_reply
                (t=0x7f1274c157f0, p_msg=0x7f12804a6d70, branch=0,
                msg_status=200, cancel_data=0x7fff0b0158e0,
                do_put_on_wait=1) at t_reply.c:2001

                        relay = 0
                        save_clone = 0
                        buf = 0x7f12804a7cc0 "SIP/2.0 200
                OK\r\nVia: SIP/2.0/UDP
                A.B.C.D:2057;branch=z9hG4bK-129F259C;rport=2057\r\nCall-ID:
                cb03dc02e909d3118f86009033290024@A.B.C.D\r\nFrom
                <mailto:cb03dc02e909d3118f86009033290024@A.B.C.D%5Cr%5CnFrom>:
                <sip:0123456789@domain;user=phone>;epid=00903"...
                        res_len = 1021
                        relayed_code = 200
                        relayed_msg = 0x7f12804a6d70
                        reply_bak = 0x7fff0b015730
                        bm = {to_tag_val = {s = 0x7f1274c16d88 "",
                len = 5449343}}
                        totag_retr = 0
                        reply_status = RPS_COMPLETED
                        uas_rb = 0x7f1274c158b0
                        to_tag = 0x0
                        reason = {s = 0x10b0156e0 <Address
                0x10b0156e0 out of bounds>, len = 1}
                        onsend_params = {req = 0x200924a64, rpl =
                0x7f127edbaf90, param = 0x414cc0, code = 1, flags
                = 0, branch = 0, t_rbuf = 0x7f126a80c828, dst =
                0x7f12804a6f68, send_buf = {s = 0xb015700 <Address
                0xb015700 out of bounds>, len = 1024}}
                        __FUNCTION__ = "relay_reply"


                #11 0x00007f127ed9d0b7 in reply_received
                (p_msg=0x7f12804a6d70) at t_reply.c:2499

                        msg_status = 200
                        last_uac_status = 183
                        ack = 0x40 <Address 0x40 out of bounds>
                        ack_len = 0
                        branch = 0
                        reply_status = -2143420688 <tel:2143420688>
                        onreply_route = 1
                        cancel_data = {cancel_bitmap = 0, reason =
                {cause = 200, u = {text = {s = 0x0, len =
                9586191}, e2e_cancel = 0x0, packed_hdrs = {s =
                0x0, len = 9586191}}}}
                        uac = 0x7f1274c15958
                        t = 0x7f1274c157f0
                        lack_dst = {send_sock = 0x7f12803e4110, to
                = {s = {sa_family = 20496, sa_data =
                "'\200\022\177\000\000\310\036#\000\000\000\000"},
                sin = {sin_family = 20496, sin_port = 32807,
                sin_addr = {s_addr = 32530}, sin_zero =
                "\310\036#\000\000\000\000"}, sin6 = {
                              sin6_family = 20496, sin6_port =
                32807, sin6_flowinfo = 32530, sin6_addr = {__in6_u
                = {__u6_addr8 =
                "\310\036#\000\000\000\000\000\360\247=\200\022\177\000",
                __u6_addr16 = {7880, 35, 0, 0, 42992, 32829,
                32530, 0}, __u6_addr32 = {2301640, 0, 2151524336,
                                    32530}}}, sin6_scope_id =
                2150060928}}, id = 32530, proto = 72 'H',
                send_flags = {f = 228 '\344', blst_imask = 61 '='}}
                        backup_user_from = 0x934630
                        backup_user_to = 0x934638
                        backup_domain_from = 0x934640
                        backup_domain_to = 0x934648
                        backup_uri_from = 0x934620
                        backup_uri_to = 0x934628
                        backup_xavps = 0x934760
                        replies_locked = 1
                        branch_ret = 0
                        prev_branch = 184637856
                        blst_503_timeout = 32767
                        hf = 0x7f12804a6d90
                        onsend_params = {req = 0x7fff0b015960, rpl
                = 0x550b94, param = 0x231dc8, code = 0, flags = 3,
                branch = 0, t_rbuf = 0x7f1280275380, dst =
                0x7f12803de418, send_buf = {s = 0x7fff0b015960
                "`G\223", len = 5538037}}
                        ctx = {rec_lev = 0, run_flags = 0,
                last_retcode = 0, jmp_env = {{__jmpbuf =
                {139717438500712, 3644308075193502665, 4279488,
                140733378027408, 0, 0, 3644308075281583049,
                -3644194520509117495}, __mask_was_saved = 0,
                __saved_mask = {__val = {9586395,
                                  1065161476041, 124554051613,
                9586471, 139717437685488, 9587300, 9586197,
                361695345073193192, 9586295, 9586274, 2151546560,
                139717437685488, 139717437615640, 139717438500712,
                4279488, 140733378027408}}}}}
                        __FUNCTION__ = "reply_received"


                #12 0x000000000045d837 in do_forward_reply
                (msg=0x7f12804a6d70, mode=0) at forward.c:777

                        new_buf = 0x0
                        dst = {send_sock = 0x0, to = {s =
                {sa_family = 0, sa_data = '\000' <repeats 13
                times>}, sin = {sin_family = 0, sin_port = 0,
                sin_addr = {s_addr = 0}, sin_zero =
                "\000\000\000\000\000\000\000"}, sin6 =
                {sin6_family = 0, sin6_port = 0, sin6_flowinfo = 0,
                              sin6_addr = {__in6_u = {__u6_addr8 =
                '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0,
                0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}},
                sin6_scope_id = 0}}, id = 0, proto = 0 '\000',
                send_flags = {f = 0 '\000', blst_imask = 0 '\000'}}
                        new_len = 32530
                        r = 1
                        s = 0x464804a6d78 <Address 0x464804a6d78
                out of bounds>
                        len = 0
                        __FUNCTION__ = "do_forward_reply"


                #13 0x000000000045e0f8 in forward_reply
                (msg=0x7f12804a6d70) at forward.c:860

                No locals.
                #14 0x00000000004a58e7 in receive_msg
                (buf=0x924600 "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP
                185.20.8.4;branch=z9hG4bK512b.82b197888826f6b60c0c63b79801294d.0;received=185.20.8.4\r\nVia:
                SIP/2.0/UDP
                10.143.1.2:2057;branch=z9hG4bK-129F259C;rport=2057\r\nCall-ID:
                cb0"...,

                    len=1124, rcv_info=0x7fff0b015c60) at
                receive.c:273

                         msg = 0x7f12804a6d70
                        ctx = {rec_lev = 8868984, run_flags = 0,
                last_retcode = 0, jmp_env = {{__jmpbuf = {0, 0, 0,
                263853236176, 1, 0, 169653785368, 9586112},
                __mask_was_saved = 184638568, __saved_mask =
                {__val = {139717436454816, 12884901899,
                139717436454816, 4279488,
                                  140733378027408,
                140733378026464, 5477954, 0, 139717072962944,
                50195, 169290548608, 9586112, 140733378026592,
                140733378026512, 5474789, 4279488}}}}}
                        ret = 32530
                        inb = {s = 0x924600 "SIP/2.0 200
                OK\r\nVia: SIP/2.0/UDP
                A.B.C.D;branch=z9hG4bK512b.82b197888826f6b60c0c63b79801294d.0;received=A.B.C.D\r\nVia:
                SIP/2.0/UDP
                A.B.C.D:2057;branch=z9hG4bK-129F259C;rport=2057\r\nCall-ID:
                cb0"..., len = 1124}
                        __FUNCTION__ = "receive_msg"


                #15 0x000000000053c9a8 in udp_rcv_loop () at
                udp_server.c:536

                        len = 1124
                        buf = "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP
                A.B.C.D;branch=z9hG4bK512b.82b197888826f6b60c0c63b79801294d.0;received=A.B.C.D\r\nVia:
                SIP/2.0/UDP
                A.B.C.D:2057;branch=z9hG4bK-129F259C;rport=2057\r\nCall-ID:
                cb0"...
                        tmp = 0x9245c0 "10.143.1.10"
                        from = 0x7f12803e3f68
                        fromlen = 16
                        ri = {src_ip = {af = 2, len = 4, u =
                {addrl = {403182777, 139717436454816}, addr32 =
                {403182777, 0, 2150315936, 32530}, addr16 = {5305,
                6152, 0, 0, 14240, 32811, 32530, 0}, addr =
                "\271\024\b\030\000\000\000\000\240\067+\200\022\177\000"}},
                dst_ip = {af = 2,
                            len = 4, u = {addrl = {67638457, 0},
                addr32 = {67638457, 0, 0, 0}, addr16 = {5305,
                1032, 0, 0, 0, 0, 0, 0}, addr = "\271\024\b\004",
                '\000' <repeats 11 times>}}, src_port = 5060,
                dst_port = 5060, proto_reserved1 = 0,
                proto_reserved2 = 0, src_su = {s = {
                              sa_family = 2, sa_data =
                "\023Ĺ\024\b\030\000\000\000\000\000\000\000"},
                sin = {sin_family = 2, sin_port = 50195, sin_addr
                = {s_addr = 403182777}, sin_zero =
                "\000\000\000\000\000\000\000"}, sin6 =
                {sin6_family = 2, sin6_port = 50195,
                              sin6_flowinfo = 403182777, sin6_addr
                = {__in6_u = {__u6_addr8 = '\000' <repeats 15
                times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0},
                __u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 0}},
                bind_address = 0x7f12802b3638, proto = 1 '\001'}
                        __FUNCTION__ = "udp_rcv_loop"


                #16 0x000000000046d42b in main_loop () at main.c:1617

                        i = 1
                        pid = 0
                        si = 0x7f12802b3638
                        si_desc = "udp receiver child=1
                sock=A.B.C.D:5060\000D\200\022\177\000\000\000\206\063\200\022\177\000\000.\205^\000\000\000\000\000\020w^\000\000\000\000\000\275\005r/\000\000\000\000\300LA\000\000\000\000\000\220_\001\v\377\177",
                '\000' <repeats 18 times>"\320,
                ]\001\v\377\177\000\000\364\244K\000\000\000\000"
                        nrprocs = 15
                        __FUNCTION__ = "main_loop"


                #17 0x0000000000470533 in main (argc=7,
                argv=0x7fff0b015f98) at main.c:2545

                        cfg_stream = 0xe20010
                        c = -1
                        r = 0
                        tmp = 0x7fff0b017f70 ""
                        tmp_len = 0
                        port = 0
                        proto = 0
                        options = 0x5e0a68
                ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:"
                        ret = -1
                        seed = 1972285608
                        rfd = 4
                debug_save = 0
                        debug_flag = 0
                        dont_fork_cnt = 0
                        n_lst = 0x3d6f60fb88
                        p = 0x5cab80
                "H\211l$\330L\211d$\340H\215-o\244*"
                        __FUNCTION__ = "main"

                In a next mail you will find a new bt full of
                Kamailio 's crash but about  km_val.c :
                db_mysql_val2str

                2014-06-25 18:26 GMT+02:00 Daniel-Constantin
                Mierla <miconda@gmail.com <mailto:miconda@gmail.com>>:

                2014-06-25 18:26 GMT+02:00 Daniel-Constantin
                Mierla <miconda@gmail.com <mailto:miconda@gmail.com>>:

                    Hello,

                    can you give the output of:

                    frame 6
                    print hdr
                    print *hdr

                    frame 4
                    print *pid_b

                    Also, it would be good to have full trace for
                    other details:

                    bt full

                    Cheers,
                    Daniel



                    On 25/06/14 14:49, Igor Potjevlesch wrote:

                        Hello,

                        We updated this morning Kamailio in 4.1.4
                        with your patch.

                        [...]


                    -- 
                    Daniel-Constantin Mierla - http://www.asipto.com
                    http://twitter.com/#!/miconda
                    <http://twitter.com/#%21/miconda> -
                    http://www.linkedin.com/in/miconda





            -- 

            Daniel-Constantin Mierla -http://www.asipto.com

            http://twitter.com/#!/miconda  <http://twitter.com/#%21/miconda>  -http://www.linkedin.com/in/miconda





    -- 

    Daniel-Constantin Mierla -http://www.asipto.com

    http://twitter.com/#!/miconda  <http://twitter.com/#%21/miconda>  -http://www.linkedin.com/in/miconda




-- 

Daniel-Constantin Mierla

http://twitter.com/#!/miconda  <http://twitter.com/#%21/miconda>  -http://www.linkedin.com/in/miconda

Next Kamailio Advanced Trainings 2014 -http://www.asipto.com

Sep 22-25, Berlin, Germany ::: Oct 15-17, San Francisco, USA

-- Daniel-Constantin Mierla http://twitter.com/#!/miconda http://twitter.com/#%21/miconda -http://www.linkedin.com/in/miconda Next Kamailio Advanced Trainings 2014 -http://www.asipto.com Sep 22-25, Berlin, Germany