Hi Daniel, Here is the output of `bt full`: http://pastebin.com/tnrWML3L Let me know if I can be of any assistance further. Regards, Dragos ------------------------------ *From:* Daniel-Constantin Mierla &lt;miconda(a)gmail.com&gt; *To:* Dragos Oancea &lt;droancea(a)yahoo.com&gt;om>; Kamailio (SER) - Development Mailing List &lt;sr-dev(a)lists.sip-router.org&gt; *Sent:* Thursday, April 4, 2013 7:28 PM *Subject:* Re: [sr-dev] segfault with t_suspend() / t_continue() Hello, could be related, but the issue seems to be while processing a SIP reply, not with suspend/continue of a request. Can you send the output of 'bt full'? Cheers, Daniel On 4/4/13 6:56 PM, Dragos Oancea wrote: Hello, I got the following coredump while doing some testing with t_suspend() and t_continue() on the first INVITE: (gdb) bt #0 qm_detach_free (qm=0x7fe2dae65000, size=16) at mem/q_malloc.c:269 #1 qm_malloc (qm=0x7fe2dae65000, size=16) at mem/q_malloc.c:386 #2 0x00007fe2e5e6d541 in update_totag_set (t=0x7fe2db0f42d0, ok=<value optimized out>) at t_reply.c:372 #3 0x00007fe2e5e72387 in relay_reply (t=<value optimized out>, p_msg=<value optimized out>, branch=2, msg_status=200, cancel_data=0x7fff63fdf9b0, do_put_on_wait=1) at t_reply.c:1854 #4 0x00007fe2e5e733e6 in reply_received (p_msg=0x7fe2e6a55ba0) at t_reply.c:2370 #5 0x00000000004547f5 in do_forward_reply (msg=0x7fe2e6a55ba0, mode=<value optimized out>) at forward.c:799 #6 0x000000000049a5ba in receive_msg (buf=<value optimized out>, len=961, rcv_info=0x7fff63fdfc40) at receive.c:270 #7 0x00000000005281a6 in udp_rcv_loop () at udp_server.c:544 #8 0x0000000000464d3a in main_loop () at main.c:1638 #9 0x0000000000467a6c in main (argc=<value optimized out>, argv=<value optimized out>) at main.c:2566 I suspend the transaction in one route and then I continue it in another route, similar to below: route[INVITE] # Suspend and put to memcache: t_suspend(); xlog("proceesing suspended in transaction [$T(id_index):$T(id_label)]\n"); $var(transaction_index)=$T(id_index); $var(transaction_label)=$T(id_label); $mct(wait_$tU)= "" + $var(transaction_index) + ":" + $var(transaction_label); $mctex(wait_$tU) = 100; } route[INVITE_resume] { # get the transaction index and lable from memcache $avp(check) = $mct(wait_$tU) ; # I should have the transaction index and the transaction label here $var(my_transaction_index) = $(avp(check){s.select,0,:}); $var(my_transaction_label) = $(avp(check){s.select,1,:}); $var(a)= $(var(my_transaction_index){s.int}); $var(b) = $(var(my_transaction_label){s.int}); t_continue( "$var(a)", "$var(b)" , "my_relay_route"); } route[INVITE_resume] executes only on an asynchronous event (a certain SIP message that might come or not) . I use kamailio 4.0.0/Centos Linux. I suspect it happened when I sent CANCEL, but I could not reproduce. I still have the core file. Regards, Dragos _______________________________________________ sr-dev mailing listsr-dev@lists.sip-router.orghttp://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-dev -- Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Kamailio World Conference, April 16-17, 2013, Berlin - http://conference.kamailio.com - _______________________________________________ sr-dev mailing list sr-dev(a)lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-dev

Hi Daniel, Here is the output of `bt full`: http://pastebin.com/tnrWML3L Let me know if I can be of any assistance further. Regards, Dragos ------------------------------------------------------------------------ *From:* Daniel-Constantin Mierla &lt;miconda(a)gmail.com&gt; *To:* Dragos Oancea &lt;droancea(a)yahoo.com&gt;om>; Kamailio (SER) - Development Mailing List &lt;sr-dev(a)lists.sip-router.org&gt; *Sent:* Thursday, April 4, 2013 7:28 PM *Subject:* Re: [sr-dev] segfault with t_suspend() / t_continue() Hello, could be related, but the issue seems to be while processing a SIP reply, not with suspend/continue of a request. Can you send the output of 'bt full'? Cheers, Daniel On 4/4/13 6:56 PM, Dragos Oancea wrote: -- Daniel-Constantin Mierla - http://www.asipto.com http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Kamailio World Conference, April 16-17, 2013, Berlin - http://conference.kamailio.com -

Hi Daniel, hi Guys, I did not stress test it, I only had one registered device (which sent the INVITE) , and another one was supposed to register after some event triggered by kamailio via a http query (using the utils module) - that's the reason I parked the transaction with "t_suspend()" . When the REGISTER for the second device comes I send the INVITE to the it .( lookup() , t_relay() ) . This is a scenario that works for me. I could say I tested 10 times and 1 time crashed. I plan to stress test these days, let me know if I should get some information for you while doing it. Regards, Dragos ------------------------------ *From:* Daniel-Constantin Mierla &lt;miconda(a)gmail.com&gt; *To:* Dragos Oancea &lt;droancea(a)yahoo.com&gt;om>; Kamailio (SER) - Development Mailing List &lt;sr-dev(a)lists.sip-router.org&gt; *Sent:* Thursday, April 4, 2013 8:53 PM *Subject:* Re: [sr-dev] segfault with t_suspend() / t_continue() Hello, at first sight looks like a missing safety check, but I may be wrong after a long day. I need to look a bit deeper at it. Did you stress test it or was just casual trying? You said it happened one, couldn't be reproduced, right? Cheers, Daniel On 4/4/13 8:28 PM, Dragos Oancea wrote: Hi Daniel, Here is the output of `bt full`: http://pastebin.com/tnrWML3L Let me know if I can be of any assistance further. Regards, Dragos ------------------------------ *From:* Daniel-Constantin Mierla &lt;miconda(a)gmail.com&gt; &lt;miconda(a)gmail.com&gt; *To:* Dragos Oancea &lt;droancea(a)yahoo.com&gt; &lt;droancea(a)yahoo.com&gt;om>; Kamailio (SER) - Development Mailing List <sr-dev@lists.sip-router.org><sr-dev@lists.sip-router.org> *Sent:* Thursday, April 4, 2013 7:28 PM *Subject:* Re: [sr-dev] segfault with t_suspend() / t_continue() Hello, could be related, but the issue seems to be while processing a SIP reply, not with suspend/continue of a request. Can you send the output of 'bt full'? Cheers, Daniel On 4/4/13 6:56 PM, Dragos Oancea wrote: Hello, I got the following coredump while doing some testing with t_suspend() and t_continue() on the first INVITE: (gdb) bt #0 qm_detach_free (qm=0x7fe2dae65000, size=16) at mem/q_malloc.c:269 #1 qm_malloc (qm=0x7fe2dae65000, size=16) at mem/q_malloc.c:386 #2 0x00007fe2e5e6d541 in update_totag_set (t=0x7fe2db0f42d0, ok=<value optimized out>) at t_reply.c:372 #3 0x00007fe2e5e72387 in relay_reply (t=<value optimized out>, p_msg=<value optimized out>, branch=2, msg_status=200, cancel_data=0x7fff63fdf9b0, do_put_on_wait=1) at t_reply.c:1854 #4 0x00007fe2e5e733e6 in reply_received (p_msg=0x7fe2e6a55ba0) at t_reply.c:2370 #5 0x00000000004547f5 in do_forward_reply (msg=0x7fe2e6a55ba0, mode=<value optimized out>) at forward.c:799 #6 0x000000000049a5ba in receive_msg (buf=<value optimized out>, len=961, rcv_info=0x7fff63fdfc40) at receive.c:270 #7 0x00000000005281a6 in udp_rcv_loop () at udp_server.c:544 #8 0x0000000000464d3a in main_loop () at main.c:1638 #9 0x0000000000467a6c in main (argc=<value optimized out>, argv=<value optimized out>) at main.c:2566 I suspend the transaction in one route and then I continue it in another route, similar to below: route[INVITE] # Suspend and put to memcache: t_suspend(); xlog("proceesing suspended in transaction [$T(id_index):$T(id_label)]\n"); $var(transaction_index)=$T(id_index); $var(transaction_label)=$T(id_label); $mct(wait_$tU)= "" + $var(transaction_index) + ":" + $var(transaction_label); $mctex(wait_$tU) = 100; } route[INVITE_resume] { # get the transaction index and lable from memcache $avp(check) = $mct(wait_$tU) ; # I should have the transaction index and the transaction label here $var(my_transaction_index) = $(avp(check){s.select,0,:}); $var(my_transaction_label) = $(avp(check){s.select,1,:}); $var(a)= $(var(my_transaction_index){s.int}); $var(b) = $(var(my_transaction_label){s.int}); t_continue( "$var(a)", "$var(b)" , "my_relay_route"); } route[INVITE_resume] executes only on an asynchronous event (a certain SIP message that might come or not) . I use kamailio 4.0.0/Centos Linux. I suspect it happened when I sent CANCEL, but I could not reproduce. I still have the core file. Regards, Dragos _______________________________________________ sr-dev mailing listsr-dev@lists.sip-router.orghttp://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-dev -- Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Kamailio World Conference, April 16-17, 2013, Berlin - http://conference.kamailio.com - -- Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Kamailio World Conference, April 16-17, 2013, Berlin - http://conference.kamailio.com - _______________________________________________ sr-dev mailing list sr-dev(a)lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-dev

Hi Daniel, hi Guys, I did not stress test it, I only had one registered device (which sent the INVITE) , and another one was supposed to register after some event triggered by kamailio via a http query (using the utils module) - that's the reason I parked the transaction with "t_suspend()" . When the REGISTER for the second device comes I send the INVITE to the it .( lookup() , t_relay() ) . This is a scenario that works for me. I could say I tested 10 times and 1 time crashed. I plan to stress test these days, let me know if I should get some information for you while doing it.

Regards, Dragos ------------------------------------------------------------------------ *From:* Daniel-Constantin Mierla &lt;miconda(a)gmail.com&gt; *To:* Dragos Oancea &lt;droancea(a)yahoo.com&gt;om>; Kamailio (SER) - Development Mailing List &lt;sr-dev(a)lists.sip-router.org&gt; *Sent:* Thursday, April 4, 2013 8:53 PM *Subject:* Re: [sr-dev] segfault with t_suspend() / t_continue() Hello, at first sight looks like a missing safety check, but I may be wrong after a long day. I need to look a bit deeper at it. Did you stress test it or was just casual trying? You said it happened one, couldn't be reproduced, right? Cheers, Daniel On 4/4/13 8:28 PM, Dragos Oancea wrote: -- Daniel-Constantin Mierla -http://www.asipto.com <http://www.asipto.com/> http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> -http://www.linkedin.com/in/miconda Kamailio World Conference, April 16-17, 2013, Berlin -http://conference.kamailio.com <http://conference.kamailio.com/> -

Hi, (gdb) p *(struct qm_frag_end*)(((char*)qm) - sizeof(struct qm_frag_end)) $1 = {size = 0, prev_free = 0x0} I also have more information (I found meanwhile the kamailio debug log ): - i am using async_sleep() somewhere in the script - it indeed crashes when it receives a reply from a voicemail server (the call should go to voicemail if the second user - callee - does not REGISTER ) 0(21290) DEBUG: tm [t_lookup.c:1666]: DEBUG: t_lookup_ident: transaction not found 0(21290) ERROR: tm [t_suspend.c:128]: ERROR: t_continue: transaction not found 0(21290) ERROR: tmx [tmx_mod.c:567]: resuming the processing of transaction [44781:1223434987] failed [...] 0(21290) ERROR: <script>: route[REGISTER]:transaction 15704:1312607975 should continue 0(21290) ERROR: *** cfgtrace: c=[suspend.cfg] l=668 a=27 n=t_continue 0(21290) DEBUG: tm [t_lookup.c:1657]: DEBUG: t_lookup_ident: transaction found 0(21290) ERROR: *** cfgtrace: c=[suspend.cfg] l=678 a=63 n=assign Daniel, let me know if I may send the log to you in private, it contains alot of private information. Regards, Dragos ------------------------------------------------------------------------ *From:* Daniel-Constantin Mierla &lt;miconda(a)gmail.com&gt; *To:* Dragos Oancea &lt;droancea(a)yahoo.com&gt;om>; Kamailio (SER) - Development Mailing List &lt;sr-dev(a)lists.sip-router.org&gt; *Sent:* Friday, April 5, 2013 2:28 PM *Subject:* Re: [sr-dev] segfault with t_suspend() / t_continue() Hello, On 4/4/13 9:26 PM, Dragos Oancea wrote: can you spot if there are any error messages in the syslog just before the crash? It can be a buffer overflow done on the previous packet. If you are doing some tests, maybe is better to re-compile with 'MEMDBG=1', to have details about what functions allocated the chunks. For now, try to get the following from gdb: p *(struct qm_frag_end*)(((char*)qm) - sizeof(struct qm_frag_end)) Cheers, Daniel -- Daniel-Constantin Mierla -http://www.asipto.com <http://www.asipto.com/> http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> -http://www.linkedin.com/in/miconda Kamailio World Conference, April 16-17, 2013, Berlin -http://conference.kamailio.com <http://conference.kamailio.com/> -