Re: [SR-Users] Crash Kamailio 4.1.4

Hello,

I pushed a fix to kamailio master branch that should solve it -- last commits to acc module. If you can give it a try and report results, would be appreciated.

Cheers, Daniel

On 03/10/14 17:00, Igor Potjevlesch wrote:

Hello Daniel,

I’m just seeing that 4.2 is scheduled for 15th October.

I can wait until this date.

Regards,

Igor.

*De :*Igor Potjevlesch [mailto:igor.potjevlesch@gmail.com] *Envoyé :* jeudi 2 octobre 2014 18:43 *À :* miconda@gmail.com *Cc :* 'Kamailio (SER) - Users Mailing List' *Objet :* RE: [SR-Users] Crash Kamailio 4.1.4

Hello,

Thank you for your time on this issue.

I understand that I can define “db_extra” like:

modparam("acc", "db_extra",

"src_user=$fU;username=$Au;src_domain=$fd;src_ip=$si;src_pai=*$avp(s:pai)*;"

"dst_ouser=$tU;dst_user=$rU;dst_domain=$rd")

Instead of:

modparam("acc", "db_extra",

"src_user=$fU;username=$Au;src_domain=$fd;src_ip=$si;*src_pai=$ai*;"

"dst_ouser=$tU;dst_user=$rU;dst_domain=$rd")

And, in the request_route

request_route {

•    #Store PAI in AVP for accounting*
  
  

•    $avp(s:pai)=$ai;*
  
  

# per request initial checks

route(REQINIT);

[…]

}

I can put this fix while waiting for 4.2.

Regards,

Igor.

*De :*Daniel-Constantin Mierla [mailto:miconda@gmail.com] *Envoyé :* jeudi 2 octobre 2014 10:17 *À :* Igor Potjevlesch *Cc :* 'Kamailio (SER) - Users Mailing List' *Objet :* Re: [SR-Users] Crash Kamailio 4.1.4

Hello,

not really troubleshooting, as I got traveling, but I thought a bit of it, also from performances point of view.

As it was related to parallel processing of 1xx/2xx sip responses that you are accounting, I think the safest solutions would be:

• clone the message locally to acc if it is the tm (share memory)
• do synchronized access to tm data inside the acc (use locks there)

Instead of using locks or other workarounds in tm (lower performances in all cases), better do it on the part doing parallel access out of tm. Acc is doing some particular processing -- it uses a callback for a reply to access the response, so it is quite specific to it.

If you want to be safe side immediately, store the PAI in an avp inside request_route and use that avp for accounting.

A solution will be there before 4.2 is out.

Cheers, Daniel

On 02/10/14 09:50, Igor Potjevlesch wrote:

Hello Daniel,

Have you had the opportunity to look at this?

Thank you for your feedback.

Regards,

Igor.

*De :*Daniel-Constantin Mierla [mailto:miconda@gmail.com]
*Envoyé :* mercredi 24 septembre 2014 09:00
*À :* Igor Potjevlesch
*Cc :* 'Kamailio \(SER\) - Users Mailing List'
*Objet :* Re: [SR-Users] Crash Kamailio 4.1.4

Hello,

so it is still related to PAI header -- I will have more time to
look at it by end of the week.

Cheers,
Daniel

On 23/09/14 19:16, Igor Potjevlesch wrote:

    Hello Daniel,

    Patching has been done in the same time than the upgrade to 4.1.5.

    A new crash occurred in pvapi.c (in addition of my other
    recent post “Crash Kamailio 4.1.5”).

    Here is the result of a “bt full”:

    #0  0x000000000049580e in pv_get_strval (msg=0x7f14efe24ea8,
    param=0x7f14fb65d058, res=0x7fff6c6b7430, sval=0x22) at
    pvapi.c:521

    No locals.

    #1  0x00007f14f9677f2b in pv_get_pai (msg=0x7f14efe24ea8,
    param=0x7f14fb65d058, res=0x7fff6c6b7430) at pv_core.c:1026

            idxf = 0

            idx = 0

            pai_body = 0x7f14fb679d38

            pai_uri = 0xa

            i = 0

            cur_id = 0

            __FUNCTION__ = "pv_get_pai"

    #2  0x0000000000499594 in pv_get_spec_value
    (msg=0x7f14efe24ea8, sp=0x7f14fb65d040, value=0x7fff6c6b7430)
    at pvapi.c:1266

            ret = 0

            __FUNCTION__ = "pv_get_spec_value"

    #3  0x00007f14f7d3481d in extra2strar (extra=0x7f14fb65d030,
    rq=0x7f14efe24ea8, val_arr=0x7f14f7f41e30,
    int_arr=0x7f14f7f4237c, type_arr=0x7f14f7f424e7
    "\002\002\002\002") at acc_extra.c:261

            value = {rs = {s = 0x0, len = 0}, ri = 0, flags = 0}

            n = 4

            r = 0

            __FUNCTION__ = "extra2strar"

    #4  0x00007f14f7d2c3e3 in acc_db_request (rq=0x7f14efe24ea8)
    at acc.c:474

            m = 7

            n = -270381400

            i = 6

            t = 0x414cc0

            __FUNCTION__ = "acc_db_request"

    #5  0x00007f14f7d36bc8 in acc_onreply (t=0x7f14efe525b8,
    req=0x7f14efe24ea8, reply=0x7f14fb670c48, code=200) at
    acc_logic.c:471

            new_uri_bk = {s = 0x7f14efe25590
    "sip:ABCDEFGHIJ@<IP_GW>oTE sINVITE
    sip:ABCDEFGHIJ@sip.fqdn.tldSIP/2.0\r\nVia: SIP/2.0/UDP
    <IP_UAC>:5060;branch=z9hG4bK5f32deec\r\nMax-Forwards:
    69\r\nFrom: \"KLMNOPQRST\" <sip:KLMNOPQRST@sip.fqdn."..., len
    = 19}

            br = 0

            hdr = 0x7f14f7d3dc20

            __FUNCTION__ = "acc_onreply"

    #6  0x00007f14f7d3730a in tmcb_func (t=0x7f14efe525b8,
    type=512, ps=0x7fff6c6b76b0) at acc_logic.c:573

            __FUNCTION__ = "tmcb_func"

    #7  0x00007f14f9f3146c in run_trans_callbacks_internal
    (cb_lst=0x7f14efe52628, type=512, trans=0x7f14efe525b8,
    params=0x7fff6c6b76b0) at t_hooks.c:290

            cbp = 0x7f14ee4c81b0

            backup_from = 0x934630

            backup_to = 0x934638

            backup_dom_from = 0x934640

            backup_dom_to = 0x934648

            backup_uri_from = 0x934620

            backup_uri_to = 0x934628

            backup_xavps = 0x934760

            __FUNCTION__ = "run_trans_callbacks_internal"

    #8  0x00007f14f9f3167e in run_trans_callbacks_with_buf
    (type=512, rbuf=0x7f14efe52678, req=0x7f14efe24ea8,
    repl=0x7f14fb670c48, flags=183) at t_hooks.c:336

            params = {req = 0x7f14efe24ea8, rpl = 0x7f14fb670c48,
    param = 0x7f14ee4c81c0, code = 200, flags = 183, branch = 0,
    t_rbuf = 0x7f14efe52678, dst = 0x7f14efe526c8, send_buf = {

                s = 0x7f14efd7c408 "SIP/2.0 200 OK\r\nVia:
    SIP/2.0/UDP
    <IP_UAC>:5060;rport=5060;branch=z9hG4bK5f32deec\r\nCall-ID:
    7846f7332ce6a7db4484c3d06ce1c387@sip.fqdn.tld\r\nFrom
    <mailto:7846f7332ce6a7db4484c3d06ce1c387@sip.fqdn.tld%5Cr%5CnFrom>:
    \"KLMNOPQRST\" <sip:KLMNOPQRST@sip.fqdn.tld>
    <sip:KLMNOPQRST@sip.fqdn.tld>"..., len = 980}}

            trans = 0x7f14efe525b8

    #9  0x00007f14f9f63bfa in relay_reply (t=0x7f14efe525b8,
    p_msg=0x7f14fb670c48, branch=0, msg_status=183,
    cancel_data=0x7fff6c6b7a10, do_put_on_wait=1) at t_reply.c:2001

            relay = 0

            save_clone = 0

            buf = 0x7f14fb67e740 "SIP/2.0 183 Session
    Progress\r\nVia: SIP/2.0/UDP
    <IP_UAC>:5060;rport=5060;branch=z9hG4bK5f32deec\r\nCall-ID:
    7846f7332ce6a7db4484c3d06ce1c387@sip.fqdn.tld\r\nFrom
    <mailto:7846f7332ce6a7db4484c3d06ce1c387@sip.fqdn.tld%5Cr%5CnFrom>:
    \"KLMNOPQRST\" <sip:KLMNOPQRST@tru"...

            res_len = 777

            relayed_code = 183

            relayed_msg = 0x7f14fb670c48

            reply_bak = 0x414cc0

            bm = {to_tag_val = {s = 0x7f14efe53b50 "", len =
    -79437808}}

            totag_retr = 0

            reply_status = RPS_PROVISIONAL

            uas_rb = 0x7f14efe52678

            to_tag = 0x0

            reason = {s = 0x800000001 <Address 0x800000001 out of
    bounds>, len = 1}

            onsend_params = {req = 0x200924970, rpl =
    0x7f14f9f83f90, param = 0x414cc0, code = 1818984640, flags =
    3, branch = 0, t_rbuf = 0x0, dst = 0x7f14fb670e40, send_buf =
    {s = 0x7fff6c6b7830 "`xkl\377\177", len = -101469275}}

            __FUNCTION__ = "relay_reply"

    #10 0x00007f14f9f660ab in reply_received
    (p_msg=0x7f14fb670c48) at t_reply.c:2499

            msg_status = 183

            last_uac_status = 183

            ack = 0x40 <Address 0x40 out of bounds>

            ack_len = 0

            branch = 0

            reply_status = -77092928

            onreply_route = 1

            cancel_data = {cancel_bitmap = 0, reason = {cause = 0,
    u = {text = {s = 0x0, len = 9586205}, e2e_cancel = 0x0,
    packed_hdrs = {s = 0x0, len = 9586205}}}}

            uac = 0x7f14efe52720

            t = 0x7f14efe525b8

            lack_dst = {send_sock = 0x7f14fb584420, to = {s =
    {sa_family = 57360, sa_data =
    "C\373\024\177\000\000\000F#\000\000\000\000"}, sin =
    {sin_family = 57360, sin_port = 64323, sin_addr = {s_addr =
    32532}, sin_zero = "\000F#\000\000\000\000"}, sin6 = {

                  sin6_family = 57360, sin6_port = 64323,
    sin6_flowinfo = 32532, sin6_addr = {__in6_u = {__u6_addr8 =
    "\000F#\000\000\000\000\000\020\341C\373\024\177\000",
    __u6_addr16 = {17920, 35, 0, 0, 57616, 64323, 32532, 0},
    __u6_addr32 = {2311680, 0, 4215529744,

                        32532}}}, sin6_scope_id = 4215529744}}, id
    = 32532, proto = 96 '`', send_flags = {f = 64 '@', blst_imask
    = 103 'g'}}

            backup_user_from = 0x934630

            backup_user_to = 0x934638

            backup_domain_from = 0x934640

           backup_domain_to = 0x934648

            backup_uri_from = 0x934620

            backup_uri_to = 0x934628

            backup_xavps = 0x934760

            replies_locked = 1

            branch_ret = 0

            prev_branch = 1818983120

            blst_503_timeout = 32767

            hf = 0x7f14fb670c68

            onsend_params = {req = 0x7fff6c6b7a90, rpl = 0x550bb0,
    param = 0x234540, code = 0, flags = 3, branch = 0, t_rbuf =
    0x7f14fb43e380, dst = 0x7f14fb674030, send_buf = {s =
    0x7fff6c6b7a90 "`G\223", len = 5538065}}

            ctx = {rec_lev = 0, run_flags = 0, last_retcode = 0,
    jmp_env = {{__jmpbuf = {139728093908544, 1898006658898931560,
    4279488, 140735012372672, 0, 0, 1898006658936680296,
    -1897762211976106136}, __mask_was_saved = 0, __saved_mask =
    {__val = {9586373,

                      1365809186688, 124554051613, 9586450,
    139728093947840, 9587056, 9586211, 361695345073193192,
    9586309, 9586288, 4217874320, 139728093947840,
    139728093942016, 139728093908544, 4279488, 140735012372672}}}}}

            __FUNCTION__ = "reply_received"

    #11 0x000000000045d853 in do_forward_reply
    (msg=0x7f14fb670c48, mode=0) at forward.c:777

            new_buf = 0x0

            dst = {send_sock = 0x0, to = {s = {sa_family = 0,
    sa_data = '\000' <repeats 13 times>}, sin = {sin_family = 0,
    sin_port = 0, sin_addr = {s_addr = 0}, sin_zero =
    "\000\000\000\000\000\000\000"}, sin6 = {sin6_family = 0,
    sin6_port = 0, sin6_flowinfo = 0,

                  sin6_addr = {__in6_u = {__u6_addr8 = '\000'
    <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0},
    __u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 0}}, id = 0,
    proto = 0 '\000', send_flags = {f = 0 '\000', blst_imask = 0
    '\000'}}

            new_len = 32532

            r = 1

            s = 0x370fb670c50 <Address 0x370fb670c50 out of bounds>

            len = 0

            __FUNCTION__ = "do_forward_reply"

    #12 0x000000000045e114 in forward_reply (msg=0x7f14fb670c48)
    at forward.c:860

    No locals.

    #13 0x00000000004a5903 in receive_msg (buf=0x924600 "SIP/2.0
    183 Session Progress\r\nVia: SIP/2.0/UDP
    <IP_KAMAILIO>;branch=z9hG4bKca48.51c2c569361ea0fedd9c6c70c21b5eed.0;received=<IP_KAMAILIO>\r\nVia:
    SIP/2.0/UDP <IP_UAC>:5060;rport=5060;branch=z9hG4bK5f32d"...,

        len=880, rcv_info=0x7fff6c6b7d90) at receive.c:273

            msg = 0x7f14fb670c48

            ctx = {rec_lev = 8868984, run_flags = 0, last_retcode
    = 0, jmp_env = {{__jmpbuf = {0, 0, 0, 263853236176, 1, 0,
    171719254808, 9586112}, __mask_was_saved = 1818983832,
    __saved_mask = {__val = {139728091862768, 12884901899,
    139728091862768, 4279488,

                      140735012372672, 140735012371728, 5477982,
    0, 139727728366976, 50195, 171356018048, 9586112,
    140735012371856, 140735012371776, 5474817, 4279488}}}}}

            ret = 32532

            inb = {s = 0x924600 "SIP/2.0 183 Session
    Progress\r\nVia: SIP/2.0/UDP
    <IP_KAMAILIO>;branch=z9hG4bKca48.51c2c569361ea0fedd9c6c70c21b5eed.0;received=<IP_KAMAILIO>\r\nVia:
    SIP/2.0/UDP <IP_UAC>:5060;rport=5060;branch=z9hG4bK5f32d"...,
    len = 880}

            __FUNCTION__ = "receive_msg"

    #14 0x000000000053c9c4 in udp_rcv_loop () at udp_server.c:536

            len = 880

            buf = "SIP/2.0 183 Session Progress\r\nVia:
    SIP/2.0/UDP
    <IP_KAMAILIO>;branch=z9hG4bKca48.51c2c569361ea0fedd9c6c70c21b5eed.0;received=<IP_KAMAILIO>\r\nVia:
    SIP/2.0/UDP <IP_UAC>:5060;rport=5060;branch=z9hG4bK5f32d"...

            tmp = 0x9245c0 "10.143.1.10"

            from = 0x7f14fb5add70

            fromlen = 16

            ri = {src_ip = {af = 2, len = 4, u = {addrl =
    {403182777, 139728091862768}, addr32 = {403182777, 0,
    4215789296, 32532}, addr16 = {5305, 6152, 0, 0, 55024, 64327,
    32532, 0}, addr =
    "\271\024\b\030\000\000\000\000\360\326G\373\024\177\000"}},
    dst_ip = {af = 2,

                len = 4, u = {addrl = {67638457, 0}, addr32 =
    {67638457, 0, 0, 0}, addr16 = {5305, 1032, 0, 0, 0, 0, 0, 0},
    addr = "\271\024\b\004", '\000' <repeats 11 times>}}, src_port
    = 5060, dst_port = 5060, proto_reserved1 = 0, proto_reserved2
    = 0, src_su = {s = {

                  sa_family = 2, sa_data =
    "\023Ĺ\024\b\030\000\000\000\000\000\000\000"}, sin =
    {sin_family = 2, sin_port = 50195, sin_addr = {s_addr =
    403182777}, sin_zero = "\000\000\000\000\000\000\000"}, sin6 =
    {sin6_family = 2, sin6_port = 50195,

                  sin6_flowinfo = 403182777, sin6_addr = {__in6_u
    = {__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0,
    0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}},
    sin6_scope_id = 0}}, bind_address = 0x7f14fb47d588, proto = 1
    '\001'}

            __FUNCTION__ = "udp_rcv_loop"

    #15 0x000000000046d447 in main_loop () at main.c:1617

            i = 13

            pid = 0

            si = 0x7f14fb47d588

            si_desc = "udp receiver child=13
    sock=<IP_KAMAILIO>:5060\000\373\024\177\000\000\b$P\373\024\177\000\000\036\205^\000\000\000\000\000\000w^\000\000\000\000\000\000\244\303v\000\000\000\000\300LA\000\000\000\000\000\300\200kl\377\177",
    '\000' <repeats 19 times>,
    "\177kl\377\177\000\000\020\245K\000\000\000\000"

            nrprocs = 15

            __FUNCTION__ = "main_loop"

    #16 0x000000000047054f in main (argc=7, argv=0x7fff6c6b80c8)
    at main.c:2545

            cfg_stream = 0xf42010

            c = -1

            r = 0

    tmp = 0x7fff6c6b8f70 ""

    tmp_len = 0

    port = 0

    proto = 0

    options = 0x5e0a58
    ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:"

    ret = -1

            seed = 3572644655

            rfd = 4

            debug_save = 0

            debug_flag = 0

            dont_fork_cnt = 0

            n_lst = 0x3d6f60fb88

            p = 0x5caba0 "H\211l$\330L\211d$\340H\215-O\244*"

            __FUNCTION__ = "main"

    Let me know if you need further information.

    Regards,

    Igor.

http://www.linkedin.com/in/miconda