[SR-Users] Crash Kamailio 4.1.4

Daniel-Constantin Mierla miconda at gmail.com
Wed Aug 6 16:38:38 CEST 2014


Hello,

it is not in my plans for 4.1.5, because I didn't get any feedback on 
testing and its results, whether it fixes or not the issue.

Cheers,
Daniel

On 06/08/14 16:07, Igor Potjevlesch wrote:
>
> Hello Daniel,
>
> Thank you for this exhaustive feedback.
>
> Do you include the patch to 4.1.5?
>
> Regards,
>
> Igor.
>
> *De :*Daniel-Constantin Mierla [mailto:miconda at gmail.com]
> *Envoyé :* lundi 4 août 2014 16:24
> *À :* Igor Potjevlesch
> *Cc :* Kamailio \(SER\) - Users Mailing List
> *Objet :* Re: [SR-Users] Crash Kamailio 4.1.4
>
> Hello,
>
> the problem was that a structure in shared memory (the request cloned 
> in tm) could have been used in parallel by different kamailio processes.
>
> If there were two processes at the same time, parsing PAI resulted in 
> setting the header pointer to a private memory. The other process 
> could overtake in processing, using the same cloned request, and this 
> time the pai pointer is set, but to another private memory zone. I 
> added the locks for calling the callbacks, so the process that parse 
> the PAI is the one cleaning it.
>
> Performances should not be impacted that much, the transaction lock is 
> used and will add sequential processing when there are two replies at 
> the same time, which is not the common.
>
> Cheers,
> Daniel
>
> On 07/07/14 12:40, Igor Potjevlesch wrote:
>
>     Hello,
>
>     Can you explain the modification and the impact on our plateform?
>
>     Is it for the pai problem?
>
>     Do you have explanation for the km_val.c problem wich cause crash
>     for Kamailio too?
>
>     Regards,
>
>     Igor
>
>     2014-07-01 16:40 GMT+02:00 Daniel-Constantin Mierla
>     <miconda at gmail.com <mailto:miconda at gmail.com>>:
>
>         Hello,
>
>         can you give it a try with the patch from next commit?
>
>         -
>         http://git.sip-router.org/cgi-bin/gitweb.cgi/sip-router/?a=commit;h=da9d56be28e050dd0cb4aed50efcbda043a3e5cf
>
>         If all goes fine while testing, I will backport.
>
>         Cheers,
>         Daniel
>
>         On 26/06/14 12:58, Igor Potjevlesch wrote:
>
>             Hello,
>
>             Here the result :
>
>             (gdb) frame 6
>
>
>             #6  0x00007f127cb6dde6 in acc_onreply (t=0x7f1274c157f0,
>             req=0x7f1274c3ac08,
>                 reply=0x7f12804a6d70, code=200) at acc_logic.c:501
>
>             501 clean_hdr_field(hdr);
>             (gdb) print hdr
>             $1 = (hdr_field_t *) 0x7f1274c3c238
>             (gdb) print *hdr
>             $2 = {type = HDR_PAI_T, name = {
>                 s = 0x7f1274c3b6cd "P-Asserted-Identity:
>             <sip:0123456789 at domain;user=phone>\r\nP-Sig-Options:
>             Sending-Complete\r\n\r\nv=0\r\no=- 111851 1 IN IP4
>             A.B.C.D\r\ns=-\r\nt=0 0\r\nm=audio 21336 RTP/AVP 8 101
>             13\r\nc=IN IP4 A.B.C"..., len = 19}, body = {
>                 s = 0x7f1274c3b6e2
>             "<sip:0123456789 at domain;user=phone>\r\nP-Sig-Options:
>             Sending-Complete\r\n\r\nv=0\r\no=- 111851 1 IN IP4
>             A.B.C.D\r\ns=-\r\nt=0 0\r\nm=audio 21336 RTP/AVP 8 101
>             13\r\nc=IN IP4 A.B.C.D\r\na=rtpmap:101 tele"..., len =
>             44}, len = 67, parsed = 0x0, next = 0x7f1274c3c278}
>
>             (gdb) frame 4
>
>
>             #4  0x000000000056e5e6 in free_pai_ppi_body
>             (pid_b=0x7f12803cb480)
>                 at parser/parse_ppi_pai.c:102
>
>             102 pkg_free(pid_b);
>             (gdb) print *pid_b
>             $3 = {id = 0x0, num_ids = 0, next = 0x1d0}
>
>             This is the bt full :
>
>
>
>             #0  0x0000003d6f6328a5 in raise () from /lib64/libc.so.6
>             No symbol table info available.
>             #1  0x0000003d6f634085 in abort () from /lib64/libc.so.6
>             No symbol table info available.
>             #2  0x0000000000546d3c in qm_debug_frag
>             (qm=0x7f1280275010, f=0x7f12803cb450) at mem/q_malloc.c:142
>
>                     __FUNCTION__ = "qm_debug_frag"
>
>
>             #3  0x0000000000548b26 in qm_free (qm=0x7f1280275010,
>             p=0x7f12803cb480, file=0x6276a0 "<core>:
>             parser/parse_ppi_pai.c", func=0x627a00
>             "free_pai_ppi_body", line=102) at mem/q_malloc.c:464
>
>                     f = 0x7f12803cb450
>                     size = 139717434027144
>                     next = 0xf00000000
>                     prev = 0x7f127cd79e00
>                     __FUNCTION__ = "qm_free"
>
>
>             #4  0x000000000056e5e6 in free_pai_ppi_body
>             (pid_b=0x7f12803cb480) at parser/parse_ppi_pai.c:102
>
>                     __FUNCTION__ = "free_pai_ppi_body"
>
>
>             #5  0x000000000054fee0 in clean_hdr_field
>             (hf=0x7f1274c3c238) at parser/hf.c:126
>
>                     h_parsed = 0x7f1274c3c268
>                     __FUNCTION__ = "clean_hdr_field"
>
>
>             #6  0x00007f127cb6dde6 in acc_onreply (t=0x7f1274c157f0,
>             req=0x7f1274c3ac08, reply=0x7f12804a6d70, code=200) at
>             acc_logic.c:501
>
>                     new_uri_bk = {s = 0x7f1274b53cdf
>             "sip:0987654321 at GW SIP/2.0\r\nRecord-Route:
>             <sip:A.B.C.D;lr=on>\r\nVia: SIP/2.0/UDP
>             A.B.C.D;branch=z9hG4bK512b.82b197888826f6b60c0c63b79801294d.0\r\nVia:
>             SIP/2.0/UDP A.B.C.D:2057;branch=z9hG4bK-12
>             <sip:0987654321 at GW%20SIP/2.0%5Cr%5CnRecord-Route:%20%3csip:A.B.C.D;lr=on%3e%5Cr%5CnVia:%20SIP/2.0/UDP%20A.B.C.D;branch=z9hG4bK512b.82b197888826f6b60c0c63b79801294d.0%5Cr%5CnVia:%20SIP/2.0/UDP%20A.B.C.D:2057;branch=z9hG4bK-12>"...,
>             len = 19}
>                     br = 0
>                     hdr = 0x7f1274c3c238
>                     __FUNCTION__ = "acc_onreply"
>
>
>             #7  0x00007f127cb6e30a in tmcb_func (t=0x7f1274c157f0,
>             type=512, ps=0x7fff0b015580) at acc_logic.c:573
>
>                     __FUNCTION__ = "tmcb_func"
>
>
>             #8  0x00007f127ed68478 in run_trans_callbacks_internal
>             (cb_lst=0x7f1274c15860, type=512, trans=0x7f1274c157f0,
>             params=0x7fff0b015580) at t_hooks.c:290
>
>                     cbp = 0x7f1274ac0e90
>                     backup_from = 0x934630
>                     backup_to = 0x934638
>                     backup_dom_from = 0x934640
>                     backup_dom_to = 0x934648
>                     backup_uri_from = 0x934620
>                     backup_uri_to = 0x934628
>                     backup_xavps = 0x934760
>                     __FUNCTION__ = "run_trans_callbacks_internal"
>
>
>             #9  0x00007f127ed6868a in run_trans_callbacks_with_buf
>             (type=512, rbuf=0x7f1274c158b0, req=0x7f1274c3ac08,
>             repl=0x7f12804a6d70, flags=200) at t_hooks.c:336
>
>                     params = {req = 0x7f1274c3ac08, rpl =
>             0x7f12804a6d70, param = 0x7f1274ac0ea0, code = 200, flags
>             = 200, branch = 0, t_rbuf = 0x7f1274c158b0, dst =
>             0x7f1274c15900, send_buf = {
>                         s = 0x7f1274c27620 "SIP/2.0 200 OK\r\nVia:
>             SIP/2.0/UDP
>             A.B.C.D:2057;branch=z9hG4bK-129F259C;rport=2057\r\nCall-ID: cb03dc02e909d3118f86009033290024 at A.B.C.D\r\nFrom
>             <mailto:cb03dc02e909d3118f86009033290024 at A.B.C.D%5Cr%5CnFrom>:
>             <sip:0123456789 at domain;user=phone>;epid=00903"..., len =
>             1021}}
>                     trans = 0x7f1274c157f0
>
>
>             #10 0x00007f127ed9ac06 in relay_reply (t=0x7f1274c157f0,
>             p_msg=0x7f12804a6d70, branch=0, msg_status=200,
>             cancel_data=0x7fff0b0158e0, do_put_on_wait=1) at
>             t_reply.c:2001
>
>                     relay = 0
>                     save_clone = 0
>                     buf = 0x7f12804a7cc0 "SIP/2.0 200 OK\r\nVia:
>             SIP/2.0/UDP
>             A.B.C.D:2057;branch=z9hG4bK-129F259C;rport=2057\r\nCall-ID: cb03dc02e909d3118f86009033290024 at A.B.C.D\r\nFrom
>             <mailto:cb03dc02e909d3118f86009033290024 at A.B.C.D%5Cr%5CnFrom>:
>             <sip:0123456789 at domain;user=phone>;epid=00903"...
>                     res_len = 1021
>                     relayed_code = 200
>                     relayed_msg = 0x7f12804a6d70
>                     reply_bak = 0x7fff0b015730
>                     bm = {to_tag_val = {s = 0x7f1274c16d88 "", len =
>             5449343}}
>                     totag_retr = 0
>                     reply_status = RPS_COMPLETED
>                     uas_rb = 0x7f1274c158b0
>                     to_tag = 0x0
>                     reason = {s = 0x10b0156e0 <Address 0x10b0156e0 out
>             of bounds>, len = 1}
>                     onsend_params = {req = 0x200924a64, rpl =
>             0x7f127edbaf90, param = 0x414cc0, code = 1, flags = 0,
>             branch = 0, t_rbuf = 0x7f126a80c828, dst = 0x7f12804a6f68,
>             send_buf = {s = 0xb015700 <Address 0xb015700 out of
>             bounds>, len = 1024}}
>                     __FUNCTION__ = "relay_reply"
>
>
>             #11 0x00007f127ed9d0b7 in reply_received
>             (p_msg=0x7f12804a6d70) at t_reply.c:2499
>
>                     msg_status = 200
>                     last_uac_status = 183
>                     ack = 0x40 <Address 0x40 out of bounds>
>                     ack_len = 0
>                     branch = 0
>                     reply_status = -2143420688 <tel:2143420688>
>                     onreply_route = 1
>                     cancel_data = {cancel_bitmap = 0, reason = {cause
>             = 200, u = {text = {s = 0x0, len = 9586191}, e2e_cancel =
>             0x0, packed_hdrs = {s = 0x0, len = 9586191}}}}
>                     uac = 0x7f1274c15958
>                     t = 0x7f1274c157f0
>                     lack_dst = {send_sock = 0x7f12803e4110, to = {s =
>             {sa_family = 20496, sa_data =
>             "'\200\022\177\000\000\310\036#\000\000\000\000"}, sin =
>             {sin_family = 20496, sin_port = 32807, sin_addr = {s_addr
>             = 32530}, sin_zero = "\310\036#\000\000\000\000"}, sin6 = {
>                           sin6_family = 20496, sin6_port = 32807,
>             sin6_flowinfo = 32530, sin6_addr = {__in6_u = {__u6_addr8
>             =
>             "\310\036#\000\000\000\000\000\360\247=\200\022\177\000",
>             __u6_addr16 = {7880, 35, 0, 0, 42992, 32829, 32530, 0},
>             __u6_addr32 = {2301640, 0, 2151524336,
>                                 32530}}}, sin6_scope_id =
>             2150060928}}, id = 32530, proto = 72 'H', send_flags = {f
>             = 228 '\344', blst_imask = 61 '='}}
>                     backup_user_from = 0x934630
>                     backup_user_to = 0x934638
>                     backup_domain_from = 0x934640
>                     backup_domain_to = 0x934648
>                     backup_uri_from = 0x934620
>                     backup_uri_to = 0x934628
>                     backup_xavps = 0x934760
>                     replies_locked = 1
>                     branch_ret = 0
>                     prev_branch = 184637856
>                     blst_503_timeout = 32767
>                     hf = 0x7f12804a6d90
>                     onsend_params = {req = 0x7fff0b015960, rpl =
>             0x550b94, param = 0x231dc8, code = 0, flags = 3, branch =
>             0, t_rbuf = 0x7f1280275380, dst = 0x7f12803de418, send_buf
>             = {s = 0x7fff0b015960 "`G\223", len = 5538037}}
>                     ctx = {rec_lev = 0, run_flags = 0, last_retcode =
>             0, jmp_env = {{__jmpbuf = {139717438500712,
>             3644308075193502665, 4279488, 140733378027408, 0, 0,
>             3644308075281583049, -3644194520509117495},
>             __mask_was_saved = 0, __saved_mask = {__val = {9586395,
>                               1065161476041, 124554051613, 9586471,
>             139717437685488, 9587300, 9586197, 361695345073193192,
>             9586295, 9586274, 2151546560, 139717437685488,
>             139717437615640, 139717438500712, 4279488,
>             140733378027408}}}}}
>                     __FUNCTION__ = "reply_received"
>
>
>             #12 0x000000000045d837 in do_forward_reply
>             (msg=0x7f12804a6d70, mode=0) at forward.c:777
>
>                     new_buf = 0x0
>                     dst = {send_sock = 0x0, to = {s = {sa_family = 0,
>             sa_data = '\000' <repeats 13 times>}, sin = {sin_family =
>             0, sin_port = 0, sin_addr = {s_addr = 0}, sin_zero =
>             "\000\000\000\000\000\000\000"}, sin6 = {sin6_family = 0,
>             sin6_port = 0, sin6_flowinfo = 0,
>                           sin6_addr = {__in6_u = {__u6_addr8 = '\000'
>             <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0,
>             0}, __u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 0}}, id
>             = 0, proto = 0 '\000', send_flags = {f = 0 '\000',
>             blst_imask = 0 '\000'}}
>                     new_len = 32530
>                     r = 1
>                     s = 0x464804a6d78 <Address 0x464804a6d78 out of
>             bounds>
>                     len = 0
>                     __FUNCTION__ = "do_forward_reply"
>
>
>             #13 0x000000000045e0f8 in forward_reply
>             (msg=0x7f12804a6d70) at forward.c:860
>
>             No locals.
>             #14 0x00000000004a58e7 in receive_msg (buf=0x924600
>             "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP
>             185.20.8.4;branch=z9hG4bK512b.82b197888826f6b60c0c63b79801294d.0;received=185.20.8.4\r\nVia:
>             SIP/2.0/UDP
>             10.143.1.2:2057;branch=z9hG4bK-129F259C;rport=2057\r\nCall-ID:
>             cb0"...,
>
>                 len=1124, rcv_info=0x7fff0b015c60) at receive.c:273
>
>                      msg = 0x7f12804a6d70
>                     ctx = {rec_lev = 8868984, run_flags = 0,
>             last_retcode = 0, jmp_env = {{__jmpbuf = {0, 0, 0,
>             263853236176, 1, 0, 169653785368, 9586112},
>             __mask_was_saved = 184638568, __saved_mask = {__val =
>             {139717436454816, 12884901899, 139717436454816, 4279488,
>                               140733378027408, 140733378026464,
>             5477954, 0, 139717072962944, 50195, 169290548608, 9586112,
>             140733378026592, 140733378026512, 5474789, 4279488}}}}}
>                     ret = 32530
>                     inb = {s = 0x924600 "SIP/2.0 200 OK\r\nVia:
>             SIP/2.0/UDP
>             A.B.C.D;branch=z9hG4bK512b.82b197888826f6b60c0c63b79801294d.0;received=A.B.C.D\r\nVia:
>             SIP/2.0/UDP
>             A.B.C.D:2057;branch=z9hG4bK-129F259C;rport=2057\r\nCall-ID: cb0"...,
>             len = 1124}
>                     __FUNCTION__ = "receive_msg"
>
>
>             #15 0x000000000053c9a8 in udp_rcv_loop () at udp_server.c:536
>
>                     len = 1124
>                     buf = "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP
>             A.B.C.D;branch=z9hG4bK512b.82b197888826f6b60c0c63b79801294d.0;received=A.B.C.D\r\nVia:
>             SIP/2.0/UDP
>             A.B.C.D:2057;branch=z9hG4bK-129F259C;rport=2057\r\nCall-ID: cb0"...
>                     tmp = 0x9245c0 "10.143.1.10"
>                     from = 0x7f12803e3f68
>                     fromlen = 16
>                     ri = {src_ip = {af = 2, len = 4, u = {addrl =
>             {403182777, 139717436454816}, addr32 = {403182777, 0,
>             2150315936, 32530}, addr16 = {5305, 6152, 0, 0, 14240,
>             32811, 32530, 0}, addr =
>             "\271\024\b\030\000\000\000\000\240\067+\200\022\177\000"}},
>             dst_ip = {af = 2,
>                         len = 4, u = {addrl = {67638457, 0}, addr32 =
>             {67638457, 0, 0, 0}, addr16 = {5305, 1032, 0, 0, 0, 0, 0,
>             0}, addr = "\271\024\b\004", '\000' <repeats 11 times>}},
>             src_port = 5060, dst_port = 5060, proto_reserved1 = 0,
>             proto_reserved2 = 0, src_su = {s = {
>                           sa_family = 2, sa_data =
>             "\023Ĺ\024\b\030\000\000\000\000\000\000\000"}, sin =
>             {sin_family = 2, sin_port = 50195, sin_addr = {s_addr =
>             403182777}, sin_zero = "\000\000\000\000\000\000\000"},
>             sin6 = {sin6_family = 2, sin6_port = 50195,
>                           sin6_flowinfo = 403182777, sin6_addr =
>             {__in6_u = {__u6_addr8 = '\000' <repeats 15 times>,
>             __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0,
>             0, 0, 0}}}, sin6_scope_id = 0}}, bind_address =
>             0x7f12802b3638, proto = 1 '\001'}
>                     __FUNCTION__ = "udp_rcv_loop"
>
>
>             #16 0x000000000046d42b in main_loop () at main.c:1617
>
>                     i = 1
>                     pid = 0
>                     si = 0x7f12802b3638
>                     si_desc = "udp receiver child=1
>             sock=A.B.C.D:5060\000D\200\022\177\000\000\000\206\063\200\022\177\000\000.\205^\000\000\000\000\000\020w^\000\000\000\000\000\275\005r/\000\000\000\000\300LA\000\000\000\000\000\220_\001\v\377\177",
>             '\000' <repeats 18 times>"\320,
>             ]\001\v\377\177\000\000\364\244K\000\000\000\000"
>                     nrprocs = 15
>                     __FUNCTION__ = "main_loop"
>
>
>             #17 0x0000000000470533 in main (argc=7,
>             argv=0x7fff0b015f98) at main.c:2545
>
>                     cfg_stream = 0xe20010
>                     c = -1
>                     r = 0
>                     tmp = 0x7fff0b017f70 ""
>                     tmp_len = 0
>                     port = 0
>                     proto = 0
>                     options = 0x5e0a68
>             ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:"
>                     ret = -1
>                     seed = 1972285608
>                     rfd = 4
>             debug_save = 0
>                     debug_flag = 0
>                     dont_fork_cnt = 0
>                     n_lst = 0x3d6f60fb88
>                     p = 0x5cab80 "H\211l$\330L\211d$\340H\215-o\244*"
>                     __FUNCTION__ = "main"
>
>             In a next mail you will find a new bt full of Kamailio 's
>             crash but about  km_val.c : db_mysql_val2str
>
>             2014-06-25 18:26 GMT+02:00 Daniel-Constantin Mierla
>             <miconda at gmail.com <mailto:miconda at gmail.com>>:
>
>             2014-06-25 18:26 GMT+02:00 Daniel-Constantin Mierla
>             <miconda at gmail.com <mailto:miconda at gmail.com>>:
>
>                 Hello,
>
>                 can you give the output of:
>
>                 frame 6
>                 print hdr
>                 print *hdr
>
>                 frame 4
>                 print *pid_b
>
>                 Also, it would be good to have full trace for other
>                 details:
>
>                 bt full
>
>                 Cheers,
>                 Daniel
>
>
>
>                 On 25/06/14 14:49, Igor Potjevlesch wrote:
>
>                     Hello,
>
>                     We updated this morning Kamailio in 4.1.4 with
>                     your patch.
>
>                     [...]
>
>
>                 -- 
>                 Daniel-Constantin Mierla - http://www.asipto.com
>                 http://twitter.com/#!/miconda
>                 <http://twitter.com/#%21/miconda> -
>                 http://www.linkedin.com/in/miconda
>
>
>
>         -- 
>
>         Daniel-Constantin Mierla -http://www.asipto.com
>
>         http://twitter.com/#!/miconda  <http://twitter.com/#%21/miconda>  -http://www.linkedin.com/in/miconda
>
>
>
> -- 
> Daniel-Constantin Mierla -http://www.asipto.com
> http://twitter.com/#!/miconda  <http://twitter.com/#%21/miconda>  -http://www.linkedin.com/in/miconda

-- 
Daniel-Constantin Mierla
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Next Kamailio Advanced Trainings 2014 - http://www.asipto.com
Sep 22-25, Berlin, Germany ::: Oct 15-17, San Francisco, USA

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.sip-router.org/pipermail/sr-users/attachments/20140806/c138041e/attachment.html>


More information about the sr-users mailing list