From the second crash, can you get:

frame 1

p *dlg

So far it looks like either to a double free or some buffer overflow...

Cheers, Daniel

On 15/07/16 10:51, Dirk Teurlings - Signet B.V. wrote:

Just got another segfault.

Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". Core was generated by `/usr/sbin/kamailio -f /etc/kamailio/kamailio.cfg -P /var/run/kamailio/kamailio.'. Program terminated with signal 11, Segmentation fault. #0 atomic_get (v=0x7f6264d11378) at ../../mem/../atomic/atomic_common.h:74 74 return atomic_get_int(&(v->val)); (gdb) bt #0 atomic_get (v=0x7f6264d11378) at ../../mem/../atomic/atomic_common.h:74 #1 dlg_unref (dlg=dlg@entry=0x7f585c494b40, cnt=cnt@entry=1) at dlg_hash.c:921 #2 0x00007f5855912802 in dlg_run_event_route (dlg=dlg@entry=0x7f585c494b40, msg=msg@entry=0x7f587d4be8e8, ostate=<optimized out>, nstate=<optimized out>) at dlg_handlers.c:1630 #3 0x00007f585591416a in dlg_onroute (req=0x7f587d4be8e8, route_params=<optimized out>, param=<optimized out>) at dlg_handlers.c:1307 #4 0x00007f585965b0e2 in run_rr_callbacks (req=req@entry=0x7f587d4be8e8, rr_param=rr_param@entry=0x7f58598677a0) at rr_cb.c:96 #5 0x00007f58596452c5 in after_loose (_m=0x7f587d4be8e8, preloaded=0) at loose.c:919 #6 0x000000000042b618 in do_action (h=h@entry=0x7ffd6e277fd0, a=a@entry=0x7f587d264338, msg=msg@entry=0x7f587d4be8e8) at action.c:1060 #7 0x000000000042a10a in run_actions (h=h@entry=0x7ffd6e277fd0, a=0x7f587d264338, msg=0x7f587d4be8e8) at action.c:1549 #8 0x0000000000437544 in run_actions_safe (h=h@entry=0x7ffd6e279500, a=<optimized out>, msg=<optimized out>) at action.c:1614 #9 0x000000000053b2e8 in rval_get_int (h=0x7ffd6e279500, msg=<optimized out>, i=0x7ffd6e278430, rv=rv@entry=0x7f587d264d58, cache=cache@entry=0x0) at rvalue.c:912 #10 0x000000000054261c in rval_expr_eval_int (h=h@entry=0x7ffd6e279500, msg=msg@entry=0x7f587d4be8e8, res=res@entry=0x7ffd6e278430, rve=rve@entry=0x7f587d264d50) at rvalue.c:1910 #11 0x000000000042bc91 in do_action (h=h@entry=0x7ffd6e279500, a=a@entry=0x7f587d268f88, msg=msg@entry=0x7f587d4be8e8) at action.c:1030 #12 0x000000000042a10a in run_actions (h=h@entry=0x7ffd6e279500, a=0x7f587d268f88, msg=msg@entry=0x7f587d4be8e8) at action.c:1549 #13 0x000000000042bcf2 in do_action (h=h@entry=0x7ffd6e279500, a=a@entry=0x7f587d2691e8, msg=msg@entry=0x7f587d4be8e8) at action.c:1049 #14 0x000000000042a10a in run_actions (h=h@entry=0x7ffd6e279500, a=0x7f587d263f48, msg=msg@entry=0x7f587d4be8e8) at action.c:1549 #15 0x000000000042bde0 in do_action (h=h@entry=0x7ffd6e279500, a=a@entry=0x7f587d073d70, msg=msg@entry=0x7f587d4be8e8) at action.c:678 #16 0x000000000042a10a in run_actions (h=h@entry=0x7ffd6e279500, a=a@entry=0x7f587d071698, msg=msg@entry=0x7f587d4be8e8) at action.c:1549 #17 0x00000000004375d0 in run_top_route (a=0x7f587d071698, msg=msg@entry=0x7f587d4be8e8, c=c@entry=0x0) at action.c:1635 #18 0x0000000000504386 in receive_msg (buf=<optimized out>, len=<optimized out>, rcv_info=<optimized out>) at receive.c:240 #19 0x00000000005f5bd4 in udp_rcv_loop () at udp_server.c:495 #20 0x00000000004b2625 in main_loop () at main.c:1600 #21 0x0000000000427e2b in main (argc=<optimized out>, argv=<optimized out>) at main.c:2616

Relevant logmessages before crash: Jul 15 10:37:55 server /usr/sbin/kamailio[12426]: NOTICE: dialog [dlg_hash.c:245]: dlg_clean_run(): dialog in delete state is too old (0x7f585c4a6820 ref 4) Jul 15 10:37:55 server /usr/sbin/kamailio[12397]: WARNING: dialog [dlg_handlers.c:1219]: dlg_onroute(): unable to find dialog for BYE with route param '70f.b9d1' [3847:7579] Jul 15 10:37:55 server /usr/sbin/kamailio[12395]: WARNING: dialog [dlg_handlers.c:1348]: dlg_onroute(): inconsitent dlg timer data on dlg 0x7f585c4a6820 [3847:7579] with clid '4c41f08d317ecb9342b93f22738003f3@server' and tags 'as5f3a16b4' 'as71cb6036' Jul 15 10:40:13 server /usr/sbin/kamailio[12378]: WARNING: dialog [dlg_handlers.c:1219]: dlg_onroute(): unable to find dialog for BYE with route param 'eb6.1e21' [1726:4833] Jul 15 10:40:13 server /usr/sbin/kamailio[12376]: WARNING: dialog [dlg_handlers.c:1219]: dlg_onroute(): unable to find dialog for BYE with route param 'eb6.1e21' [1726:4833] Jul 15 10:40:14 server /usr/sbin/kamailio[12377]: WARNING: dialog [dlg_handlers.c:1219]: dlg_onroute(): unable to find dialog for BYE with route param 'eb6.1e21' [1726:4833] Jul 15 10:40:16 server /usr/sbin/kamailio[12377]: WARNING: dialog [dlg_handlers.c:1219]: dlg_onroute(): unable to find dialog for BYE with route param 'eb6.1e21' [1726:4833] Jul 15 10:40:16 server /usr/sbin/kamailio[12396]: WARNING: dialog [dlg_handlers.c:1219]: dlg_onroute(): unable to find dialog for BYE with route param 'eb6.1e21' [1726:4833] Jul 15 10:41:34 server /usr/sbin/kamailio[12396]: ERROR: sl [sl_funcs.c:363]: sl_reply_error(): ERROR: sl_reply_error used: I'm terribly sorry, server error occurred (1/SL) Jul 15 10:41:34 server /usr/sbin/kamailio[12396]: ERROR: tm [t_reply.c:533]: _reply_light(): ERROR: _reply_light: can't generate 487 reply when a final 487 was sent out Jul 15 10:41:34 server /usr/sbin/kamailio[12396]: ERROR: tm [t_lookup.c:1471]: t_unref(): ERROR: t_unref: generation of a delayed stateful reply failed Jul 15 10:42:25 server /usr/sbin/kamailio[12426]: NOTICE: dialog [dlg_hash.c:245]: dlg_clean_run(): dialog in delete state is too old (0x7f585c49d5b0 ref 4) Jul 15 10:42:25 server /usr/sbin/kamailio[12426]: NOTICE: dialog [dlg_hash.c:245]: dlg_clean_run(): dialog in delete state is too old (0x7f585c604f18 ref 4) Jul 15 10:42:25 server /usr/sbin/kamailio[12426]: NOTICE: dialog [dlg_hash.c:245]: dlg_clean_run(): dialog in delete state is too old (0x7f585c494b40 ref 4) Jul 15 10:42:25 server /usr/sbin/kamailio[12383]: WARNING: dialog [dlg_handlers.c:1348]: dlg_onroute(): inconsitent dlg timer data on dlg 0x7f585c604f18 [2396:9046] with clid '1b3ff5f0246fb7e82ed949544bcccbba@192.168.10.233:5060' and tags 'as4d83d6f8' '5788A162-2557E04D-3E86ED15' Jul 15 10:42:25 server /usr/sbin/kamailio[12395]: WARNING: dialog [dlg_handlers.c:1219]: dlg_onroute(): unable to find dialog for BYE with route param '6b3.c6b' [950:2924] Jul 15 10:42:25 server kernel: [209851.262461] kamailio[12376]: segfault at 7f6264d11378 ip 00007f585592a908 sp 00007ffd6e277330 error 4 in dialog.so[7f58558e0000+88000] Jul 15 10:42:25 server /usr/sbin/kamailio[12394]: WARNING: dialog [dlg_handlers.c:1348]: dlg_onroute(): inconsitent dlg timer data on dlg 0x7f585c49d5b0 [950:2924] with clid '45fe86ce065f5543342e51ad355d1b75@server' and tags 'as152f7465' 'as4d03f77d' Jul 15 10:42:26 server /usr/sbin/kamailio[12431]: CRITICAL: <core> [pass_fd.c:275]: receive_fd(): EOF on 32 Jul 15 10:42:26 server /usr/sbin/kamailio[12370]: ALERT: <core> [main.c:739]: handle_sigs(): child process 12376 exited by a signal 11 Jul 15 10:42:26 server /usr/sbin/kamailio[12370]: ALERT: <core> [main.c:742]: handle_sigs(): core was generated Jul 15 10:42:26 server /usr/sbin/kamailio[12370]: INFO: <core> [main.c:754]: handle_sigs(): terminating due to SIGCHLD

Cheers, Dirk

On 07/15/2016 10:06 AM, Dirk Teurlings - Signet B.V. wrote:

...

Hi,

Running Kamailio on Debian from the Kamailio repository with 4.4.2 stable (unpatched). Getting some random segfaults with it now, here's the relevant backtrace from the generated core.

Core was generated by `/usr/sbin/kamailio -f /etc/kamailio/kamailio.cfg -P /var/run/kamailio/kamailio.'. Program terminated with signal 11, Segmentation fault. #0 run_dlg_callbacks (type=type@entry=64, dlg=dlg@entry=0x7fceb400e2f0, req=req@entry=0x7fced4f093c8, rpl=rpl@entry=0x0, dir=<optimized out>, dlg_data=dlg_data@entry=0x0) at dlg_cb.c:253 253 if ( (cb->types)&type ) { (gdb) bt #0 run_dlg_callbacks (type=type@entry=64, dlg=dlg@entry=0x7fceb400e2f0, req=req@entry=0x7fced4f093c8, rpl=rpl@entry=0x0, dir=<optimized out>, dlg_data=dlg_data@entry=0x0) at dlg_cb.c:253 #1 0x00007fcead3648f9 in dlg_terminated (dir=<optimized out>, dlg=0x7fceb400e2f0, req=0x7fced4f093c8) at dlg_handlers.c:368 #2 dlg_onroute (req=0x7fced4f093c8, route_params=<optimized out>, param=<optimized out>) at dlg_handlers.c:1354 #3 0x00007fceb10ab0e2 in run_rr_callbacks (req=req@entry=0x7fced4f093c8, rr_param=rr_param@entry=0x7fceb12b77a0) at rr_cb.c:96 #4 0x00007fceb10952c5 in after_loose (_m=0x7fced4f093c8, preloaded=0) at loose.c:919 #5 0x000000000042b618 in do_action (h=h@entry=0x7ffeb0b3ed80, a=a@entry=0x7fced4cb4338, msg=msg@entry=0x7fced4f093c8) at action.c:1060 #6 0x000000000042a10a in run_actions (h=h@entry=0x7ffeb0b3ed80, a=0x7fced4cb4338, msg=0x7fced4f093c8) at action.c:1549 #7 0x0000000000437544 in run_actions_safe (h=h@entry=0x7ffeb0b402b0, a=<optimized out>, msg=<optimized out>) at action.c:1614 #8 0x000000000053b2e8 in rval_get_int (h=0x7ffeb0b402b0, msg=<optimized out>, i=0x7ffeb0b3f1e0, rv=rv@entry=0x7fced4cb4d58, cache=cache@entry=0x0) at rvalue.c:912 #9 0x000000000054261c in rval_expr_eval_int (h=h@entry=0x7ffeb0b402b0, msg=msg@entry=0x7fced4f093c8, res=res@entry=0x7ffeb0b3f1e0, rve=rve@entry=0x7fced4cb4d50) at rvalue.c:1910 #10 0x000000000042bc91 in do_action (h=h@entry=0x7ffeb0b402b0, a=a@entry=0x7fced4cb8f88, msg=msg@entry=0x7fced4f093c8) at action.c:1030 #11 0x000000000042a10a in run_actions (h=h@entry=0x7ffeb0b402b0, a=0x7fced4cb8f88, msg=msg@entry=0x7fced4f093c8) at action.c:1549 #12 0x000000000042bcf2 in do_action (h=h@entry=0x7ffeb0b402b0, a=a@entry=0x7fced4cb91e8, msg=msg@entry=0x7fced4f093c8) at action.c:1049 #13 0x000000000042a10a in run_actions (h=h@entry=0x7ffeb0b402b0, a=0x7fced4cb3f48, msg=msg@entry=0x7fced4f093c8) at action.c:1549 #14 0x000000000042bde0 in do_action (h=h@entry=0x7ffeb0b402b0, a=a@entry=0x7fced4ac3d70, msg=msg@entry=0x7fced4f093c8) at action.c:678 #15 0x000000000042a10a in run_actions (h=h@entry=0x7ffeb0b402b0, a=a@entry=0x7fced4ac1698, msg=msg@entry=0x7fced4f093c8) at action.c:1549 #16 0x00000000004375d0 in run_top_route (a=0x7fced4ac1698, msg=msg@entry=0x7fced4f093c8, c=c@entry=0x0) at action.c:1635 #17 0x0000000000504386 in receive_msg (buf=<optimized out>, len=<optimized out>, rcv_info=<optimized out>) at receive.c:240 #18 0x00000000005f5bd4 in udp_rcv_loop () at udp_server.c:495 #19 0x00000000004b2625 in main_loop () at main.c:1600 #20 0x0000000000427e2b in main (argc=<optimized out>, argv=<optimized out>) at main.c:2616

And from syslog the relevant messages before this dump:

Jul 15 08:55:03 server /usr/sbin/kamailio[16470]: WARNING: dialog [dlg_handlers.c:1219]: dlg_onroute(): unable to find dialog for BYE with route param 'd4c.26d1' [3149:7522] Jul 15 08:56:01 server /usr/sbin/kamailio[16481]: WARNING: dialog [dlg_handlers.c:1219]: dlg_onroute(): unable to find dialog for BYE with route param 'fc.99f1' [207:8089] Jul 15 08:56:27 server /usr/sbin/kamailio[16470]: CRITICAL: dialog [dlg_timer.c:200]: update_dlg_timer(): Trying to update a bogus dlg tl=0x7fceb3f7d920 tl->next=(nil) tl->prev=(nil) Jul 15 08:56:27 server /usr/sbin/kamailio[16470]: ERROR: dialog [dlg_handlers.c:1377]: dlg_onroute(): failed to update dialog lifetime Jul 15 08:57:01 server /usr/sbin/kamailio[16482]: ERROR: db_mysql [km_dbase.c:128]: db_mysql_submit_query(): driver error on query: Duplicate entry '9584-3854-435' for key 'hash_index' (1062) Jul 15 08:57:01 server /usr/sbin/kamailio[16482]: ERROR: <core> [db_query.c:181]: db_do_raw_query(): error while submitting query Jul 15 08:57:01 server /usr/sbin/kamailio[16482]: ERROR: sqlops [sql_api.c:265]: sql_do_query(): cannot do the query [INSERT INTO `dialog_extra` (`h_i] Jul 15 08:57:01 server /usr/sbin/kamailio[16482]: ERROR: auth [api.c:119]: auth_check_hdr_md5(): auth:pre_auth: Credentials are not filled properly Jul 15 08:57:01 server /usr/sbin/kamailio[16483]: ERROR: auth [api.c:119]: auth_check_hdr_md5(): auth:pre_auth: Credentials are not filled properly Jul 15 08:57:54 server /usr/sbin/kamailio[16506]: NOTICE: dialog [dlg_hash.c:245]: dlg_clean_run(): dialog in delete state is too old (0x7fceb3f64470 ref 4) Jul 15 08:57:54 server /usr/sbin/kamailio[16473]: WARNING: dialog [dlg_handlers.c:1348]: dlg_onroute(): inconsitent dlg timer data on dlg 0x7fceb3f64470 [1182:5803] with clid '09ad128753e2535d24bde58e3d7eda04@192.168.10.232:5060' and tags 'as1b497b34' '5788890C-EC6F55F-3E86ED0C' Jul 15 08:57:54 server /usr/sbin/kamailio[16469]: ERROR: dialog [dlg_handlers.c:334]: dlg_terminated_confirmed(): failed to get dialog from params! Jul 15 08:58:49 server /usr/sbin/kamailio[16467]: WARNING: dialog [dlg_handlers.c:1219]: dlg_onroute(): unable to find dialog for BYE with route param '6d2.2581' [726:6226] Jul 15 08:59:24 server /usr/sbin/kamailio[16506]: NOTICE: dialog [dlg_hash.c:245]: dlg_clean_run(): dialog in delete state is too old (0x7fceb400e2f0 ref 4) Jul 15 08:59:25 server /usr/sbin/kamailio[16464]: WARNING: dialog [dlg_handlers.c:1219]: dlg_onroute(): unable to find dialog for BYE with route param '3e4.b5c1' [1251:7259] Jul 15 08:59:25 server /usr/sbin/kamailio[16465]: WARNING: dialog [dlg_handlers.c:1348]: dlg_onroute(): inconsitent dlg timer data on dlg 0x7fceb400e2f0 [1251:7259] with clid '87791a#015#012Call-ID: 25750e286a5654361ef9405d72edbc' and tags '' 'as148f41b1' Jul 15 08:59:25 server kernel: [203670.830521] kamailio[16465] general protection ip:7fcead34b3a5 sp:7ffeb0b3e220 error:0 in dialog.so[7fcead330000+88000] Jul 15 08:59:26 server /usr/sbin/kamailio[16511]: CRITICAL: <core> [pass_fd.c:275]: receive_fd(): EOF on 33 Jul 15 08:59:26 server /usr/sbin/kamailio[16458]: ALERT: <core> [main.c:739]: handle_sigs(): child process 16465 exited by a signal 11 Jul 15 08:59:26 server /usr/sbin/kamailio[16458]: ALERT: <core> [main.c:742]: handle_sigs(): core was generated Jul 15 08:59:26 server /usr/sbin/kamailio[16458]: INFO: <core> [main.c:754]: handle_sigs(): terminating due to SIGCHLD

Any insight would be appreciated!

Cheers, Dirk

---

SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users

---

SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users

The content of dlg is not valid, likely freed. Can you run with -x qm and see if you get new error messages?

Also, what modules are you using, specially interested in those using dialog module, such as cnxcc or presence dialog info?!?!

Cheers, Daniel

On 15/07/16 13:06, Dirk Teurlings - Signet B.V. wrote:

(gdb) frame 1 #1 dlg_unref (dlg=dlg@entry=0x7f585c494b40, cnt=cnt@entry=1) at dlg_hash.c:921 921 dlg_lock( d_table, d_entry); (gdb) p *dlg $1 = {ref = 793790803, next = 0xa0d4b4f20303032, prev = 0x504953203a616956, h_id = 808333871, h_entry = 1346655535, state = 774976288, lifetime = 775107122, init_ts = 775435825, start_ts = 976303410, end_ts = 808857653, dflags = 1667592763, iflags = 1702259045, sflags = 825441636, toroute = 858927662, toroute_name = { s = 0x6172623b3135322e <Address 0x6172623b3135322e out of bounds>, len = 1030251374}, from_rr_nb = 894132788, tl = {next = 0x726f70723b646262, prev = 0xa0d303630353d74, timeout = 1836020294}, callid = {s = 0x20226e776f6e6b6e <Address 0x20226e776f6e6b6e out of bounds>, len = 1885958972}, from_uri = { s = 0x7340444c4f74656e <Address 0x7340444c4f74656e out of bounds>, len = 1999532137}, to_uri = {s = 0x743b3e74656e2e70 <Address 0x743b3e74656e2e70 out of bounds>, len = 1631414113}, req_uri = {s = 0x540a0d3536343766 <Address 0x540a0d3536343766 out of bounds>, len = 1008745071}, tag = {{ s = 0x363233313431332b <Address 0x363233313431332b out of bounds>, len = 892614711}, {s = 0x2e3836312e333232 <Address 0x2e3836312e333232 out of bounds>, len = 1043608370}}, cseq = {{s = 0x663330643473613d <Address 0x663330643473613d out of bounds>, len = 224671543}, {s = 0x3534203a44492d6c <Address 0x3534203a44492d6c out of bounds>, len = 909665638}}, route_set = {{s = 0x3433333435356635 <Address 0x3433333435356635 out of bounds>, len = 825582898}, { s = 0x7340353762316435 <Address 0x7340353762316435 out of bounds>, len = 1999532137}}, contact = {{s = 0x430a0d74656e2e70 <Address 0x430a0d74656e2e70 out of bounds>, len = 980510035}, {s = 0x65530a0d45594220 <Address 0x65530a0d45594220 out of bounds>, len = 1919252082}}, bind_addr = {0x70696f766c772e70, 0x6c410a0d74656e2e}, cbs = { first = 0x564e49203a776f6c, types = 742741065}, profile_links = 0x4c45434e4143202c, vars = 0x4e4f4954504f202c}

On 07/15/2016 01:00 PM, Daniel-Constantin Mierla wrote:

...

From the second crash, can you get:

frame 1

p *dlg

So far it looks like either to a double free or some buffer overflow...

Cheers, Daniel

On 15/07/16 10:51, Dirk Teurlings - Signet B.V. wrote:

...

Just got another segfault.

Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". Core was generated by `/usr/sbin/kamailio -f /etc/kamailio/kamailio.cfg -P /var/run/kamailio/kamailio.'. Program terminated with signal 11, Segmentation fault. #0 atomic_get (v=0x7f6264d11378) at ../../mem/../atomic/atomic_common.h:74 74 return atomic_get_int(&(v->val)); (gdb) bt #0 atomic_get (v=0x7f6264d11378) at ../../mem/../atomic/atomic_common.h:74 #1 dlg_unref (dlg=dlg@entry=0x7f585c494b40, cnt=cnt@entry=1) at dlg_hash.c:921 #2 0x00007f5855912802 in dlg_run_event_route (dlg=dlg@entry=0x7f585c494b40, msg=msg@entry=0x7f587d4be8e8, ostate=<optimized out>, nstate=<optimized out>) at dlg_handlers.c:1630 #3 0x00007f585591416a in dlg_onroute (req=0x7f587d4be8e8, route_params=<optimized out>, param=<optimized out>) at dlg_handlers.c:1307 #4 0x00007f585965b0e2 in run_rr_callbacks (req=req@entry=0x7f587d4be8e8, rr_param=rr_param@entry=0x7f58598677a0) at rr_cb.c:96 #5 0x00007f58596452c5 in after_loose (_m=0x7f587d4be8e8, preloaded=0) at loose.c:919 #6 0x000000000042b618 in do_action (h=h@entry=0x7ffd6e277fd0, a=a@entry=0x7f587d264338, msg=msg@entry=0x7f587d4be8e8) at action.c:1060 #7 0x000000000042a10a in run_actions (h=h@entry=0x7ffd6e277fd0, a=0x7f587d264338, msg=0x7f587d4be8e8) at action.c:1549 #8 0x0000000000437544 in run_actions_safe (h=h@entry=0x7ffd6e279500, a=<optimized out>, msg=<optimized out>) at action.c:1614 #9 0x000000000053b2e8 in rval_get_int (h=0x7ffd6e279500, msg=<optimized out>, i=0x7ffd6e278430, rv=rv@entry=0x7f587d264d58, cache=cache@entry=0x0) at rvalue.c:912 #10 0x000000000054261c in rval_expr_eval_int (h=h@entry=0x7ffd6e279500, msg=msg@entry=0x7f587d4be8e8, res=res@entry=0x7ffd6e278430, rve=rve@entry=0x7f587d264d50) at rvalue.c:1910 #11 0x000000000042bc91 in do_action (h=h@entry=0x7ffd6e279500, a=a@entry=0x7f587d268f88, msg=msg@entry=0x7f587d4be8e8) at action.c:1030 #12 0x000000000042a10a in run_actions (h=h@entry=0x7ffd6e279500, a=0x7f587d268f88, msg=msg@entry=0x7f587d4be8e8) at action.c:1549 #13 0x000000000042bcf2 in do_action (h=h@entry=0x7ffd6e279500, a=a@entry=0x7f587d2691e8, msg=msg@entry=0x7f587d4be8e8) at action.c:1049 #14 0x000000000042a10a in run_actions (h=h@entry=0x7ffd6e279500, a=0x7f587d263f48, msg=msg@entry=0x7f587d4be8e8) at action.c:1549 #15 0x000000000042bde0 in do_action (h=h@entry=0x7ffd6e279500, a=a@entry=0x7f587d073d70, msg=msg@entry=0x7f587d4be8e8) at action.c:678 #16 0x000000000042a10a in run_actions (h=h@entry=0x7ffd6e279500, a=a@entry=0x7f587d071698, msg=msg@entry=0x7f587d4be8e8) at action.c:1549 #17 0x00000000004375d0 in run_top_route (a=0x7f587d071698, msg=msg@entry=0x7f587d4be8e8, c=c@entry=0x0) at action.c:1635 #18 0x0000000000504386 in receive_msg (buf=<optimized out>, len=<optimized out>, rcv_info=<optimized out>) at receive.c:240 #19 0x00000000005f5bd4 in udp_rcv_loop () at udp_server.c:495 #20 0x00000000004b2625 in main_loop () at main.c:1600 #21 0x0000000000427e2b in main (argc=<optimized out>, argv=<optimized out>) at main.c:2616

Relevant logmessages before crash: Jul 15 10:37:55 server /usr/sbin/kamailio[12426]: NOTICE: dialog [dlg_hash.c:245]: dlg_clean_run(): dialog in delete state is too old (0x7f585c4a6820 ref 4) Jul 15 10:37:55 server /usr/sbin/kamailio[12397]: WARNING: dialog [dlg_handlers.c:1219]: dlg_onroute(): unable to find dialog for BYE with route param '70f.b9d1' [3847:7579] Jul 15 10:37:55 server /usr/sbin/kamailio[12395]: WARNING: dialog [dlg_handlers.c:1348]: dlg_onroute(): inconsitent dlg timer data on dlg 0x7f585c4a6820 [3847:7579] with clid '4c41f08d317ecb9342b93f22738003f3@server' and tags 'as5f3a16b4' 'as71cb6036' Jul 15 10:40:13 server /usr/sbin/kamailio[12378]: WARNING: dialog [dlg_handlers.c:1219]: dlg_onroute(): unable to find dialog for BYE with route param 'eb6.1e21' [1726:4833] Jul 15 10:40:13 server /usr/sbin/kamailio[12376]: WARNING: dialog [dlg_handlers.c:1219]: dlg_onroute(): unable to find dialog for BYE with route param 'eb6.1e21' [1726:4833] Jul 15 10:40:14 server /usr/sbin/kamailio[12377]: WARNING: dialog [dlg_handlers.c:1219]: dlg_onroute(): unable to find dialog for BYE with route param 'eb6.1e21' [1726:4833] Jul 15 10:40:16 server /usr/sbin/kamailio[12377]: WARNING: dialog [dlg_handlers.c:1219]: dlg_onroute(): unable to find dialog for BYE with route param 'eb6.1e21' [1726:4833] Jul 15 10:40:16 server /usr/sbin/kamailio[12396]: WARNING: dialog [dlg_handlers.c:1219]: dlg_onroute(): unable to find dialog for BYE with route param 'eb6.1e21' [1726:4833] Jul 15 10:41:34 server /usr/sbin/kamailio[12396]: ERROR: sl [sl_funcs.c:363]: sl_reply_error(): ERROR: sl_reply_error used: I'm terribly sorry, server error occurred (1/SL) Jul 15 10:41:34 server /usr/sbin/kamailio[12396]: ERROR: tm [t_reply.c:533]: _reply_light(): ERROR: _reply_light: can't generate 487 reply when a final 487 was sent out Jul 15 10:41:34 server /usr/sbin/kamailio[12396]: ERROR: tm [t_lookup.c:1471]: t_unref(): ERROR: t_unref: generation of a delayed stateful reply failed Jul 15 10:42:25 server /usr/sbin/kamailio[12426]: NOTICE: dialog [dlg_hash.c:245]: dlg_clean_run(): dialog in delete state is too old (0x7f585c49d5b0 ref 4) Jul 15 10:42:25 server /usr/sbin/kamailio[12426]: NOTICE: dialog [dlg_hash.c:245]: dlg_clean_run(): dialog in delete state is too old (0x7f585c604f18 ref 4) Jul 15 10:42:25 server /usr/sbin/kamailio[12426]: NOTICE: dialog [dlg_hash.c:245]: dlg_clean_run(): dialog in delete state is too old (0x7f585c494b40 ref 4) Jul 15 10:42:25 server /usr/sbin/kamailio[12383]: WARNING: dialog [dlg_handlers.c:1348]: dlg_onroute(): inconsitent dlg timer data on dlg 0x7f585c604f18 [2396:9046] with clid '1b3ff5f0246fb7e82ed949544bcccbba@192.168.10.233:5060' and tags 'as4d83d6f8' '5788A162-2557E04D-3E86ED15' Jul 15 10:42:25 server /usr/sbin/kamailio[12395]: WARNING: dialog [dlg_handlers.c:1219]: dlg_onroute(): unable to find dialog for BYE with route param '6b3.c6b' [950:2924] Jul 15 10:42:25 server kernel: [209851.262461] kamailio[12376]: segfault at 7f6264d11378 ip 00007f585592a908 sp 00007ffd6e277330 error 4 in dialog.so[7f58558e0000+88000] Jul 15 10:42:25 server /usr/sbin/kamailio[12394]: WARNING: dialog [dlg_handlers.c:1348]: dlg_onroute(): inconsitent dlg timer data on dlg 0x7f585c49d5b0 [950:2924] with clid '45fe86ce065f5543342e51ad355d1b75@server' and tags 'as152f7465' 'as4d03f77d' Jul 15 10:42:26 server /usr/sbin/kamailio[12431]: CRITICAL: <core> [pass_fd.c:275]: receive_fd(): EOF on 32 Jul 15 10:42:26 server /usr/sbin/kamailio[12370]: ALERT: <core> [main.c:739]: handle_sigs(): child process 12376 exited by a signal 11 Jul 15 10:42:26 server /usr/sbin/kamailio[12370]: ALERT: <core> [main.c:742]: handle_sigs(): core was generated Jul 15 10:42:26 server /usr/sbin/kamailio[12370]: INFO: <core> [main.c:754]: handle_sigs(): terminating due to SIGCHLD

Cheers, Dirk

Hello,

have you run with -x qm before reverting to 4.2?

Is the same config you run with 4.2 and 4.4?

The version 4.2.8 has qm as default memory manager. In 4.4, fm is the default one but qm can be selected at startup with -x. The main difference is that qm has more safety checks for detecting double free and buffer overflows...

Cheers, Daniel

On 18/07/16 08:54, Dirk Teurlings - Signet B.V. wrote:

Hi Daniel,

Had to revert back to our old 4.2.5 for now, we can't cope with these crashes. Anyway, here are all the modules currently loaded by our config.

sqlops db_mysql mi_fifo.so kex.so corex.so tm.so tmx.so sl.so rr.so pv.so maxfwd.so usrloc.so registrar.so textops.so siputils.so xlog.so sanity.so ctl.so cfg_rpc.so mi_rpc.so dispatcher.so regex.so lcr.so avpops.so uac.so uac_redirect.so ratelimit.so ipops.so

And about the shared memory, is there any explenation available as to what the different options do? And what is the default?

Cheers, Dirk

On 07/15/2016 02:08 PM, Daniel-Constantin Mierla wrote:

...

The content of dlg is not valid, likely freed. Can you run with -x qm and see if you get new error messages?

Also, what modules are you using, specially interested in those using dialog module, such as cnxcc or presence dialog info?!?!

Cheers, Daniel

On 15/07/16 13:06, Dirk Teurlings - Signet B.V. wrote:

...

(gdb) frame 1 #1 dlg_unref (dlg=dlg@entry=0x7f585c494b40, cnt=cnt@entry=1) at dlg_hash.c:921 921 dlg_lock( d_table, d_entry); (gdb) p *dlg $1 = {ref = 793790803, next = 0xa0d4b4f20303032, prev = 0x504953203a616956, h_id = 808333871, h_entry = 1346655535, state = 774976288, lifetime = 775107122, init_ts = 775435825, start_ts = 976303410, end_ts = 808857653, dflags = 1667592763, iflags = 1702259045, sflags = 825441636, toroute = 858927662, toroute_name = { s = 0x6172623b3135322e <Address 0x6172623b3135322e out of bounds>, len = 1030251374}, from_rr_nb = 894132788, tl = {next = 0x726f70723b646262, prev = 0xa0d303630353d74, timeout = 1836020294}, callid = {s = 0x20226e776f6e6b6e <Address 0x20226e776f6e6b6e out of bounds>, len = 1885958972}, from_uri = { s = 0x7340444c4f74656e <Address 0x7340444c4f74656e out of bounds>, len = 1999532137}, to_uri = {s = 0x743b3e74656e2e70 <Address 0x743b3e74656e2e70 out of bounds>, len = 1631414113}, req_uri = {s = 0x540a0d3536343766 <Address 0x540a0d3536343766 out of bounds>, len = 1008745071}, tag = {{ s = 0x363233313431332b <Address 0x363233313431332b out of bounds>, len = 892614711}, {s = 0x2e3836312e333232 <Address 0x2e3836312e333232 out of bounds>, len = 1043608370}}, cseq = {{s = 0x663330643473613d <Address 0x663330643473613d out of bounds>, len = 224671543}, {s = 0x3534203a44492d6c <Address 0x3534203a44492d6c out of bounds>, len = 909665638}}, route_set = {{s = 0x3433333435356635 <Address 0x3433333435356635 out of bounds>, len = 825582898}, { s = 0x7340353762316435 <Address 0x7340353762316435 out of bounds>, len = 1999532137}}, contact = {{s = 0x430a0d74656e2e70 <Address 0x430a0d74656e2e70 out of bounds>, len = 980510035}, {s = 0x65530a0d45594220 <Address 0x65530a0d45594220 out of bounds>, len = 1919252082}}, bind_addr = {0x70696f766c772e70, 0x6c410a0d74656e2e}, cbs = { first = 0x564e49203a776f6c, types = 742741065}, profile_links = 0x4c45434e4143202c, vars = 0x4e4f4954504f202c}

On 07/15/2016 01:00 PM, Daniel-Constantin Mierla wrote:

...

From the second crash, can you get:

frame 1

p *dlg

So far it looks like either to a double free or some buffer overflow...

Cheers, Daniel

On 15/07/16 10:51, Dirk Teurlings - Signet B.V. wrote:

...

Just got another segfault.

Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". Core was generated by `/usr/sbin/kamailio -f /etc/kamailio/kamailio.cfg -P /var/run/kamailio/kamailio.'. Program terminated with signal 11, Segmentation fault. #0 atomic_get (v=0x7f6264d11378) at ../../mem/../atomic/atomic_common.h:74 74 return atomic_get_int(&(v->val)); (gdb) bt #0 atomic_get (v=0x7f6264d11378) at ../../mem/../atomic/atomic_common.h:74 #1 dlg_unref (dlg=dlg@entry=0x7f585c494b40, cnt=cnt@entry=1) at dlg_hash.c:921 #2 0x00007f5855912802 in dlg_run_event_route (dlg=dlg@entry=0x7f585c494b40, msg=msg@entry=0x7f587d4be8e8, ostate=<optimized out>, nstate=<optimized out>) at dlg_handlers.c:1630 #3 0x00007f585591416a in dlg_onroute (req=0x7f587d4be8e8, route_params=<optimized out>, param=<optimized out>) at dlg_handlers.c:1307 #4 0x00007f585965b0e2 in run_rr_callbacks (req=req@entry=0x7f587d4be8e8, rr_param=rr_param@entry=0x7f58598677a0) at rr_cb.c:96 #5 0x00007f58596452c5 in after_loose (_m=0x7f587d4be8e8, preloaded=0) at loose.c:919 #6 0x000000000042b618 in do_action (h=h@entry=0x7ffd6e277fd0, a=a@entry=0x7f587d264338, msg=msg@entry=0x7f587d4be8e8) at action.c:1060 #7 0x000000000042a10a in run_actions (h=h@entry=0x7ffd6e277fd0, a=0x7f587d264338, msg=0x7f587d4be8e8) at action.c:1549 #8 0x0000000000437544 in run_actions_safe (h=h@entry=0x7ffd6e279500, a=<optimized out>, msg=<optimized out>) at action.c:1614 #9 0x000000000053b2e8 in rval_get_int (h=0x7ffd6e279500, msg=<optimized out>, i=0x7ffd6e278430, rv=rv@entry=0x7f587d264d58, cache=cache@entry=0x0) at rvalue.c:912 #10 0x000000000054261c in rval_expr_eval_int (h=h@entry=0x7ffd6e279500, msg=msg@entry=0x7f587d4be8e8, res=res@entry=0x7ffd6e278430, rve=rve@entry=0x7f587d264d50) at rvalue.c:1910 #11 0x000000000042bc91 in do_action (h=h@entry=0x7ffd6e279500, a=a@entry=0x7f587d268f88, msg=msg@entry=0x7f587d4be8e8) at action.c:1030 #12 0x000000000042a10a in run_actions (h=h@entry=0x7ffd6e279500, a=0x7f587d268f88, msg=msg@entry=0x7f587d4be8e8) at action.c:1549 #13 0x000000000042bcf2 in do_action (h=h@entry=0x7ffd6e279500, a=a@entry=0x7f587d2691e8, msg=msg@entry=0x7f587d4be8e8) at action.c:1049 #14 0x000000000042a10a in run_actions (h=h@entry=0x7ffd6e279500, a=0x7f587d263f48, msg=msg@entry=0x7f587d4be8e8) at action.c:1549 #15 0x000000000042bde0 in do_action (h=h@entry=0x7ffd6e279500, a=a@entry=0x7f587d073d70, msg=msg@entry=0x7f587d4be8e8) at action.c:678 #16 0x000000000042a10a in run_actions (h=h@entry=0x7ffd6e279500, a=a@entry=0x7f587d071698, msg=msg@entry=0x7f587d4be8e8) at action.c:1549 #17 0x00000000004375d0 in run_top_route (a=0x7f587d071698, msg=msg@entry=0x7f587d4be8e8, c=c@entry=0x0) at action.c:1635 #18 0x0000000000504386 in receive_msg (buf=<optimized out>, len=<optimized out>, rcv_info=<optimized out>) at receive.c:240 #19 0x00000000005f5bd4 in udp_rcv_loop () at udp_server.c:495 #20 0x00000000004b2625 in main_loop () at main.c:1600 #21 0x0000000000427e2b in main (argc=<optimized out>, argv=<optimized out>) at main.c:2616

Relevant logmessages before crash: Jul 15 10:37:55 server /usr/sbin/kamailio[12426]: NOTICE: dialog [dlg_hash.c:245]: dlg_clean_run(): dialog in delete state is too old (0x7f585c4a6820 ref 4) Jul 15 10:37:55 server /usr/sbin/kamailio[12397]: WARNING: dialog [dlg_handlers.c:1219]: dlg_onroute(): unable to find dialog for BYE with route param '70f.b9d1' [3847:7579] Jul 15 10:37:55 server /usr/sbin/kamailio[12395]: WARNING: dialog [dlg_handlers.c:1348]: dlg_onroute(): inconsitent dlg timer data on dlg 0x7f585c4a6820 [3847:7579] with clid '4c41f08d317ecb9342b93f22738003f3@server' and tags 'as5f3a16b4' 'as71cb6036' Jul 15 10:40:13 server /usr/sbin/kamailio[12378]: WARNING: dialog [dlg_handlers.c:1219]: dlg_onroute(): unable to find dialog for BYE with route param 'eb6.1e21' [1726:4833] Jul 15 10:40:13 server /usr/sbin/kamailio[12376]: WARNING: dialog [dlg_handlers.c:1219]: dlg_onroute(): unable to find dialog for BYE with route param 'eb6.1e21' [1726:4833] Jul 15 10:40:14 server /usr/sbin/kamailio[12377]: WARNING: dialog [dlg_handlers.c:1219]: dlg_onroute(): unable to find dialog for BYE with route param 'eb6.1e21' [1726:4833] Jul 15 10:40:16 server /usr/sbin/kamailio[12377]: WARNING: dialog [dlg_handlers.c:1219]: dlg_onroute(): unable to find dialog for BYE with route param 'eb6.1e21' [1726:4833] Jul 15 10:40:16 server /usr/sbin/kamailio[12396]: WARNING: dialog [dlg_handlers.c:1219]: dlg_onroute(): unable to find dialog for BYE with route param 'eb6.1e21' [1726:4833] Jul 15 10:41:34 server /usr/sbin/kamailio[12396]: ERROR: sl [sl_funcs.c:363]: sl_reply_error(): ERROR: sl_reply_error used: I'm terribly sorry, server error occurred (1/SL) Jul 15 10:41:34 server /usr/sbin/kamailio[12396]: ERROR: tm [t_reply.c:533]: _reply_light(): ERROR: _reply_light: can't generate 487 reply when a final 487 was sent out Jul 15 10:41:34 server /usr/sbin/kamailio[12396]: ERROR: tm [t_lookup.c:1471]: t_unref(): ERROR: t_unref: generation of a delayed stateful reply failed Jul 15 10:42:25 server /usr/sbin/kamailio[12426]: NOTICE: dialog [dlg_hash.c:245]: dlg_clean_run(): dialog in delete state is too old (0x7f585c49d5b0 ref 4) Jul 15 10:42:25 server /usr/sbin/kamailio[12426]: NOTICE: dialog [dlg_hash.c:245]: dlg_clean_run(): dialog in delete state is too old (0x7f585c604f18 ref 4) Jul 15 10:42:25 server /usr/sbin/kamailio[12426]: NOTICE: dialog [dlg_hash.c:245]: dlg_clean_run(): dialog in delete state is too old (0x7f585c494b40 ref 4) Jul 15 10:42:25 server /usr/sbin/kamailio[12383]: WARNING: dialog [dlg_handlers.c:1348]: dlg_onroute(): inconsitent dlg timer data on dlg 0x7f585c604f18 [2396:9046] with clid '1b3ff5f0246fb7e82ed949544bcccbba@192.168.10.233:5060' and tags 'as4d83d6f8' '5788A162-2557E04D-3E86ED15' Jul 15 10:42:25 server /usr/sbin/kamailio[12395]: WARNING: dialog [dlg_handlers.c:1219]: dlg_onroute(): unable to find dialog for BYE with route param '6b3.c6b' [950:2924] Jul 15 10:42:25 server kernel: [209851.262461] kamailio[12376]: segfault at 7f6264d11378 ip 00007f585592a908 sp 00007ffd6e277330 error 4 in dialog.so[7f58558e0000+88000] Jul 15 10:42:25 server /usr/sbin/kamailio[12394]: WARNING: dialog [dlg_handlers.c:1348]: dlg_onroute(): inconsitent dlg timer data on dlg 0x7f585c49d5b0 [950:2924] with clid '45fe86ce065f5543342e51ad355d1b75@server' and tags 'as152f7465' 'as4d03f77d' Jul 15 10:42:26 server /usr/sbin/kamailio[12431]: CRITICAL: <core> [pass_fd.c:275]: receive_fd(): EOF on 32 Jul 15 10:42:26 server /usr/sbin/kamailio[12370]: ALERT: <core> [main.c:739]: handle_sigs(): child process 12376 exited by a signal 11 Jul 15 10:42:26 server /usr/sbin/kamailio[12370]: ALERT: <core> [main.c:742]: handle_sigs(): core was generated Jul 15 10:42:26 server /usr/sbin/kamailio[12370]: INFO: <core> [main.c:754]: handle_sigs(): terminating due to SIGCHLD

Cheers, Dirk

On 18/07/16 11:23, Dirk Teurlings - Signet B.V. wrote:

On 07/18/2016 10:49 AM, Daniel-Constantin Mierla wrote:

...

have you run with -x qm before reverting to 4.2?

No we didn't, as I didn't know what the effects would be, it's a live server so we don't want to try changes before testing.

OK -- just to have full picture -- did you experienced more crashes than the two ones you reported?

...

Is the same config you run with 4.2 and 4.4?

Not at the moment, but we developed the new routes in 4.2 first, then upgraded the testplatform to 4.4 without issues. Though the testplatform is not identical in hardware to the live system, this looked promising.

I see -- can you describe a bit the changes you did from old config to the new one (e.g., new modules and routing actions)? If you want to keep that private, email those details directly to me.

...

The version 4.2.8 has qm as default memory manager. In 4.4, fm is the default one but qm can be selected at startup with -x. The main difference is that qm has more safety checks for detecting double free and buffer overflows...

Right, that sounds very plausible then... Upgrading "back" to 4.4 now will have to wait a bit, we don't want to experience the same issues again. That being said, it's difficult to track what causes this exactly if we can't reproduce it on the testplatform. Unfortunately I'm not at liberty to try these things on the running servers.

Anyway, can I get any more information out of the backtraces to help resolve this issue with the new memory manager as well? Or do you need more info for that?

If you didn't get other crashes, I can't think of anything else that can help from the backtrace.

Cheers, Daniel

Hi Daniel,

Did you manage to get a chance to look at all the dumps I sent?

Cheers, Dirk

On 18-07-16 11:58, Daniel-Constantin Mierla wrote:

If you didn't get other crashes, I can't think of anything else that can help from the backtrace.

Cheers, Daniel

Hi, first crash:

(gdb) info locals cb = <optimized out> __FUNCTION__ = "run_dlg_callbacks" (gdb) bt full #0 run_dlg_callbacks (type=type@entry=64, dlg=dlg@entry=0x7fceb400e2f0, req=req@entry=0x7fced4f093c8, rpl=rpl@entry=0x0, dir=<optimized out>, dlg_data=dlg_data@entry=0x0) at dlg_cb.c:253 cb = <optimized out> __FUNCTION__ = "run_dlg_callbacks" #1 0x00007fcead3648f9 in dlg_terminated (dir=<optimized out>, dlg=0x7fceb400e2f0, req=0x7fced4f093c8) at dlg_handlers.c:368 iuid = 0x0 #2 dlg_onroute (req=0x7fced4f093c8, route_params=<optimized out>, param=<optimized out>) at dlg_handlers.c:1354 dlg = 0x7fceb400e2f0 iuid = <optimized out> val = { s = 0xa36706 "3e4.b5c1;nat=yes>,sip:31.223.168.234;r2=on;lr;ftag=as0b87791a;did=3e4.b5c1;nat=yes\r\nMax-Forwards: 69\r\nFrom: sip:+31737470617@31.223.168.234;tag=as148f41b1\r\nTo: +33612818032 <sip:signetOLD@sip.wlvo"..., len = 8} callid = { s = 0xa367ef "25750e286a5654361ef9405d72edbc54@sip.wlvoip.net\r\nCSeq: 102 BYE\r\nUser-Agent: sip.wlvoip.net\r\nX-Asterisk-HangupCause: Normal Clearing\r\nX-Asterisk-HangupCauseCode: 16\r\nContent-Length: 0\r\n\r\n", len = 47} ftag = { s = 0xa3679a "as148f41b1\r\nTo: +33612818032 sip:signetOLD@sip.wlvoip.net;tag=as0b87791a\r\nCall-ID: 25750e286a5654361ef9405d72edbc54@sip.wlvoip.net\r\nCSeq: 102 BYE\r\nUser-Agent: sip.wlvoip.net\r\nX-Asterisk-HangupCause:"..., len = 10} ttag = { s = 0xa367da "as0b87791a\r\nCall-ID: 25750e286a5654361ef9405d72edbc54@sip.wlvoip.net\r\nCSeq: 102 BYE\r\nUser-Agent: sip.wlvoip.net\r\nX-Asterisk-HangupCause: Normal Clearing\r\nX-Asterisk-HangupCauseCode: 16\r\nContent-Length"..., len = 10} h_entry = 1251 h_id = 7259 new_state = 5 old_state = 4 unref = 1 event = <optimized out> timeout = -722431032 reset = <optimized out> dir = 2 ret = <optimized out> __FUNCTION__ = "dlg_onroute" #3 0x00007fceb10ab0e2 in run_rr_callbacks (req=req@entry=0x7fced4f093c8, rr_param=rr_param@entry=0x7fceb12b77a0) at rr_cb.c:96 l_param = { s = 0xa366e9 "r2=on;lr;ftag=as0b87791a;did=3e4.b5c1;nat=yes>,sip:31.223.168.234;r2=on;lr;ftag=as0b87791a;did=3e4.b5c1;nat=yes\r\nMax-Forwards: 69\r\nFrom: sip:+31737470617@31.223.168.234;tag=as148f41b1\r\nTo: +336128"..., len = 45} cbp = 0x7fced4abc0a0 __FUNCTION__ = "run_rr_callbacks" #4 0x00007fceb10952c5 in after_loose (_m=0x7fced4f093c8, preloaded=0) at loose.c:919 hdr = <optimized out> puri = {user = {s = 0x0, len = 0}, passwd = {s = 0x0, len = 0}, host = { s = 0xa3671d "31.223.168.234;r2=on;lr;ftag=as0b87791a;did=3e4.b5c1;nat=yes>\r\nMax-Forwards: 69\r\nFrom: sip:+31737470617@31.223.168.234;tag=as148f41b1\r\nTo: +33612818032 sip:signetOLD@sip.wlvoip.net;tag=as0b87791a\r"..., len = 14}, port = {s = 0x0, len = 0}, params = { s = 0xa3672c "r2=on;lr;ftag=as0b87791a;did=3e4.b5c1;nat=yes>\r\nMax-Forwards: 69\r\nFrom: sip:+31737470617@31.223.168.234;tag=as148f41b1\r\nTo: +33612818032 sip:signetOLD@sip.wlvoip.net;tag=as0b87791a\r\nCall-ID: 25750"..., len = 45}, sip_params = { s = 0xa3672c "r2=on;lr;ftag=as0b87791a;did=3e4.b5c1;nat=yes>\r\nMax-Forwards: 69\r\nFrom: sip:+31737470617@31.223.168.234;tag=as148f41b1\r\nTo: +33612818032 sip:signetOLD@sip.wlvoip.net;tag=as0b87791a\r\nCall-ID: 25750"..., len = 45}, headers = {s = 0x0, len = 0}, port_no = 0, proto = 0, type = SIP_URI_T, flags = 0, transport = {s = 0x0, len = 0}, ttl = {s = 0x0, len = 0}, user_param = {s = 0x0, len = 0}, maddr = {s = 0x0, len = 0}, method = {s = 0x0, len = 0}, lr = { s = 0xa36732 "lr;ftag=as0b87791a;did=3e4.b5c1;nat=yes>\r\nMax-Forwards: 69\r\nFrom: sip:+31737470617@31.223.168.234;tag=as148f41b1\r\nTo: +33612818032 sip:signetOLD@sip.wlvoip.net;tag=as0b87791a\r\nCall-ID: 25750e286a5"..., len = 2}, r2 = { ---Type <return> to continue, or q <return> to quit--- s = 0xa3672c "r2=on;lr;ftag=as0b87791a;did=3e4.b5c1;nat=yes>\r\nMax-Forwards: 69\r\nFrom: sip:+31737470617@31.223.168.234;tag=as148f41b1\r\nTo: +33612818032 sip:signetOLD@sip.wlvoip.net;tag=as0b87791a\r\nCall-ID: 25750"..., len = 5}, gr = {s = 0x0, len = 0}, transport_val = {s = 0x0, len = 0}, ttl_val = {s = 0x0, len = 0}, user_param_val = { s = 0x0, len = 0}, maddr_val = {s = 0x0, len = 0}, method_val = {s = 0x0, len = 0}, lr_val = {s = 0x0, len = 0}, r2_val = { s = 0xa3672f "on;lr;ftag=as0b87791a;did=3e4.b5c1;nat=yes>\r\nMax-Forwards: 69\r\nFrom: sip:+31737470617@31.223.168.234;tag=as148f41b1\r\nTo: +33612818032 sip:signetOLD@sip.wlvoip.net;tag=as0b87791a\r\nCall-ID: 25750e28"..., len = 2}, gr_val = {s = 0x0, len = 0}} rt = <optimized out> status = <optimized out> uri = { s = 0xa366d6 "sip:192.168.10.230;r2=on;lr;ftag=as0b87791a;did=3e4.b5c1;nat=yes>,sip:31.223.168.234;r2=on;lr;ftag=as0b87791a;did=3e4.b5c1;nat=yes\r\nMax-Forwards: 69\r\nFrom: sip:+31737470617@31.223.168.234;tag=as14"..., len = 64} si = <optimized out> uri_is_myself = 25841 use_ob = 0 __FUNCTION__ = "after_loose" #5 0x000000000042b618 in do_action (h=h@entry=0x7ffeb0b3ed80, a=a@entry=0x7fced4cb4338, msg=msg@entry=0x7fced4f093c8) at action.c:1060 ret = -5 v = <optimized out> dst = {send_sock = 0xb0b3f770, to = {s = {sa_family = 61456, sa_data = "\263\260\376\177\000\000\377\377\377\377\316\177\000"}, sin = {sin_family = 61456, sin_port = 45235, sin_addr = {s_addr = 32766}, sin_zero = "\377\377\377\377\316\177\000"}, sin6 = {sin6_family = 61456, sin6_port = 45235, sin6_flowinfo = 32766, sin6_addr = {__in6_u = { __u6_addr8 = "\377\377\377\377\316\177\000\000DuC\000\000\000\000", __u6_addr16 = {65535, 65535, 32718, 0, 30020, 67, 0, 0}, __u6_addr32 = {4294967295, 32718, 4420932, 0}}}, sin6_scope_id = 0}}, id = 0, proto = -20 '\354', send_flags = {f = 123 '{', blst_imask = 172 '\254'}} tmp = <optimized out> new_uri = <optimized out> end = <optimized out> crt = <optimized out> cmd = 0x7fced4aa75a8 len = <optimized out> user = <optimized out> uri = {user = {s = 0x7fced4dd1a90 "\001", len = -1343281044}, passwd = {s = 0x48a0f8 "\207\231", len = 8}, host = {s = 0x14b <Address 0x14b out of bounds>, len = -723700576}, port = {s = 0x7fced4dd34a0 "6", len = 6334740}, params = {s = 0x40 <Address 0x40 out of bounds>, len = 6344353}, sip_params = { s = 0x7fced465a845 "%-d", len = -723706432}, headers = { s = 0xa367bc "signetOLD@sip.wlvoip.net>;tag=as0b87791a\r\nCall-ID: 25750e286a5654361ef9405d72edbc54@sip.wlvoip.net\r\nCSeq: 102 BYE\r\nUser-Agent: sip.wlvoip.net\r\nX-Asterisk-HangupCause: Normal Clearing\r\nX-Asterisk-Hangu"..., len = 9}, port_no = 0, proto = 0, type = ERROR_URI_T, flags = 4294967295, transport = {s = 0x7fced4c75270 "\001", len = -722431032}, ttl = {s = 0x7ffeb0b402b0 "\003", len = -1330385904}, user_param = {s = 0x7fced4f093c8 "Z\025", len = 5514780}, maddr = {s = 0x0, len = 0}, method = { s = 0x0, len = 0}, lr = {s = 0x100000000 <Address 0x100000000 out of bounds>, len = 0}, r2 = {s = 0x0, len = 0}, gr = {s = 0x0, len = 0}, transport_val = {s = 0x0, len = 0}, ttl_val = {s = 0x0, len = 0}, user_param_val = {s = 0x0, len = 0}, maddr_val = {s = 0x0, len = 0}, method_val = {s = 0x0, len = 0}, lr_val = {s = 0x0, len = -1011016704}, r2_val = {s = 0x0, len = -725128304}, gr_val = {s = 0x7fced4f093c8 "Z\025", len = 0}} next_hop = {user = {s = 0xfffffffb <Address 0xfffffffb out of bounds>, len = -1330380112}, passwd = {s = 0x7fced4f093c8 "Z\025", len = 4366602}, host = {s = 0x0, len = 0}, port = {s = 0x0, len = 0}, params = {s = 0x0, len = 0}, sip_params = {s = 0x0, len = 0}, headers = {s = 0x0, len = 1511}, port_no = 37832, proto = 54512, type = 32718, flags = 7, transport = {s = 0xfffffffb <Address 0xfffffffb out of bounds>, len = -726987760}, ttl = {s = 0x7fced4f093c8 "Z\025", len = -1343315701}, user_param = { s = 0x7fced4c98fd0 "\n", len = -722431032}, maddr = {s = 0x7ffeb0b402b0 "\003", len = -725128304}, method = {s = 0x7fced4f093c8 "Z\025", len = -1329584299}, lr = { s = 0x0, len = 4}, r2 = {s = 0x7fce00000000 <Address 0x7fce00000000 out of bounds>, len = -1011016704}, gr = {s = 0xa <Address 0xa out of bounds>, len = -725128000}, transport_val = {s = 0x7fced4f093c8 "Z\025", len = -1330380112}, ttl_val = {s = 0x1 <Address 0x1 out of bounds>, len = -1330380112}, user_param_val = { s = 0x7fced4f093c8 "Z\025", len = 4373746}, maddr_val = {s = 0x7fced4f093c8 "Z\025", len = 4371992}, method_val = {s = 0x70 <Address 0x70 out of bounds>, len = -1330385016}, lr_val = {s = 0x7fced4658a7d "%s%.*s%s ", len = 0}, r2_val = {s = 0x7ffe00000000 <Address 0x7ffe00000000 out of bounds>, len = -1330382992}, gr_val = {s = 0x0, len = -722431032}} u = <optimized out> port = <optimized out> dst_host = <optimized out> i = <optimized out> ---Type <return> to continue, or q <return> to quit--- flags = <optimized out> avp = <optimized out> st = {flags = 2964587184, id = 32766, name = {n = 5485288, s = { s = 0x53b2e8 "\205\300A\017\237\300\061\300E\017\266\310D\211\v\203e\004\371\351 \377\377\377H\211\372H\215q\bH\215|$P\350?\244\001", len = -723724760}, re = 0x53b2e8}, avp = 0x7fced4dd3858} sct = <optimized out> sjt = <optimized out> rve = <optimized out> mct = <optimized out> rv = <optimized out> rv1 = <optimized out> c1 = {cache_type = 3563424653, val_type = 32718, c = {avp_val = {n = 1034164339, s = {s = 0x1dec91573da41873 <Address 0x1dec91573da41873 out of bounds>, len = 1}, re = 0x1dec91573da41873}, pval = {rs = {s = 0x1dec91573da41873 <Address 0x1dec91573da41873 out of bounds>, len = 1}, ri = -1330384888, flags = 32766}}, i2s = "\001\000\000\000\316\177\000\000\000\000\000\000\000\000\000\000\232\213e\324\316\177"} s = {s = 0x0, len = 789} srevp = {0x3b00000000, 0x7ffeb0b3ec10} mod_f_params = {{type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = { number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, { type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}} __FUNCTION__ = "do_action" #6 0x000000000042a10a in run_actions (h=h@entry=0x7ffeb0b3ed80, a=0x7fced4cb4338, msg=0x7fced4f093c8) at action.c:1549 t = <optimized out> ret = <optimized out> mod = <optimized out> ms = 0 __FUNCTION__ = "run_actions" #7 0x0000000000437544 in run_actions_safe (h=h@entry=0x7ffeb0b402b0, a=<optimized out>, msg=<optimized out>) at action.c:1614 ctx = {rec_lev = 1, run_flags = 0, last_retcode = 0, jmp_env = {{__jmpbuf = {140731863007920, 2156258125485971644, 140526312526792, 140731863007920, 140731863003616, 140526312526792, -2156678653738215236, 2156258654672656572}, __mask_was_saved = 0, __saved_mask = {__val = {39367408, 140731862994288, 140731862990144, 140728898420737, 140731863002752, 39289776, 18446744073709551615, 4294967291, 140525646082718, 140526445694632, 2, 4, 3760787172421009408, 3689070625499067697, 0, 0}}}}} ret = 25841 ign_on_break = 0 #8 0x000000000053b2e8 in rval_get_int (h=0x7ffeb0b402b0, msg=<optimized out>, i=0x7ffeb0b3f1e0, rv=rv@entry=0x7fced4cb4d58, cache=cache@entry=0x0) at rvalue.c:912 r_avp = <optimized out> avp_val = {n = -723402960, s = {s = 0x7fced4e1bf30 "\001", len = -1388967936}, re = 0x7fced4e1bf30} pval = {rs = {s = 0x3000000018 <Address 0x3000000018 out of bounds>, len = -1330384320}, ri = -1330384512, flags = 32766} tmp = {s = 0x0, len = -592155317} s = 0x0 r = <optimized out> ret = 0 destroy_pval = 0 __FUNCTION__ = "rval_get_int" #9 0x000000000054261c in rval_expr_eval_int (h=h@entry=0x7ffeb0b402b0, msg=msg@entry=0x7fced4f093c8, res=res@entry=0x7ffeb0b3f1e0, rve=rve@entry=0x7fced4cb4d50) at rvalue.c:1910 i1 = <optimized out> i2 = <optimized out> ret = -1 ---Type <return> to continue, or q <return> to quit--- c1 = {cache_type = RV_CACHE_EMPTY, val_type = RV_NONE, c = {avp_val = {n = 0, s = {s = 0x0, len = 38525735}, re = 0x0}, pval = {rs = {s = 0x0, len = 38525735}, ri = -1330384448, flags = 32766}}, i2s = "\001", '\000' <repeats 15 times>"\327, \376\000\000\000"} c2 = {cache_type = 2964587184, val_type = 32766, c = {avp_val = {n = -725128000, s = {s = 0x7fced4c76cc0 "%", len = -722431032}, re = 0x7fced4c76cc0}, pval = {rs = { s = 0x7fced4c76cc0 "%", len = -722431032}, ri = -724987952, flags = 32718}}, i2s = "ȓ\360\324\316\177\000\000\003\000\000\000\316\177\000\000\026\000\000\000\000"} rv1 = <optimized out> rv2 = <optimized out> __FUNCTION__ = "rval_expr_eval_int" #10 0x000000000042bc91 in do_action (h=h@entry=0x7ffeb0b402b0, a=a@entry=0x7fced4cb8f88, msg=msg@entry=0x7fced4f093c8) at action.c:1030 ret = -5 v = <optimized out> dst = {send_sock = 0x7fcead5b9990, to = {s = {sa_family = 39280, sa_data = "[\255\316\177\000\000P\231[\255\316\177\000"}, sin = {sin_family = 39280, sin_port = 44379, sin_addr = {s_addr = 32718}, sin_zero = "P\231[\255\316\177\000"}, sin6 = {sin6_family = 39280, sin6_port = 44379, sin6_flowinfo = 32718, sin6_addr = {__in6_u = { __u6_addr8 = "P\231[\255\316\177\000\000`\231[\255\316\177\000", __u6_addr16 = {39248, 44379, 32718, 0, 39264, 44379, 32718, 0}, __u6_addr32 = {2908461392, 32718, 2908461408, 32718}}}, sin6_scope_id = 2908461328}}, id = 32718, proto = 32 ' ', send_flags = {f = 153 '\231', blst_imask = 91 '['}} tmp = <optimized out> new_uri = <optimized out> end = <optimized out> crt = <optimized out> cmd = <optimized out> len = <optimized out> user = <optimized out> uri = {user = {s = 0x7fced485e210 "", len = -1279372408}, passwd = {s = 0x0, len = 0}, host = { s = 0x24bda00 "insert into `dialog` (`hash_entry`,`hash_id`,`callid`,`from_uri`,`from_tag`,`to_uri`,`to_tag`,`caller_sock`,`callee_sock`,`start_time`,`state`,`timeout`,`caller_cseq`,`callee_cseq`,`caller_contact`,`c"..., len = -723684376}, port = {s = 0x7fced4dd76e8 "", len = -1286996357}, params = {s = 0x7fced485e210 "", len = -723684376}, sip_params = {s = 0x7fced485e210 "", len = -1286991478}, headers = { s = 0x24bda00 "insert into `dialog` (`hash_entry`,`hash_id`,`callid`,`from_uri`,`from_tag`,`to_uri`,`to_tag`,`caller_sock`,`callee_sock`,`start_time`,`state`,`timeout`,`caller_cseq`,`callee_cseq`,`caller_contact`,`c"..., len = 39331786}, port_no = 29672, proto = 54493, type = 32718, flags = 3565543952, transport = {s = 0x7ffeb0b3f350 "", len = -724876992}, ttl = {s = 0x7fced4f093c8 "Z\025", len = -722431032}, user_param = {s = 0x7ffeb0b402b0 "\003", len = 0}, maddr = {s = 0x7fced4f093c8 "Z\025", len = 5514780}, method = {s = 0x111 <Address 0x111 out of bounds>, len = -1330380112}, lr = {s = 0x0, len = -722431032}, r2 = {s = 0x7ffeb0b402b0 "\003", len = -1330382928}, gr = {s = 0x7fced4f093c8 "Z\025", len = 4420932}, transport_val = {s = 0x0, len = 1}, ttl_val = {s = 0x7ffeb0b402b0 "\003", len = 1034164412}, user_param_val = {s = 0x7fced4f093c8 "Z\025", len = -1330380112}, maddr_val = {s = 0x7ffeb0b3f7b0 "\001", len = -722431032}, method_val = { s = 0xe211f030d92418bc <Address 0xe211f030d92418bc out of bounds>, len = 1939871932}, lr_val = {s = 0x0, len = -1388646483}, r2_val = {s = 0x7fceb39dcf46 "`", len = 5060}, gr_val = {s = 0x2 <Address 0x2 out of bounds>, len = 0}} next_hop = {user = {s = 0x7fceb3fb1540 "\002", len = 0}, passwd = {s = 0x7ffefffffffb <Address 0x7ffefffffffb out of bounds>, len = 10042548}, host = { s = 0x7fce00000016 <Address 0x7fce00000016 out of bounds>, len = -1389034191}, port = {s = 0x7fceb392f640 "\002", len = -1011016704}, params = { s = 0x7fceb3fb1540 "\002", len = -1330382928}, sip_params = {s = 0x7ffeb0b402b0 "\003", len = -722431032}, headers = {s = 0x7ffeb0b402b0 "\003", len = -1330382928}, port_no = 688, proto = 45236, type = 32766, flags = 5485288, transport = {s = 0x3 <Address 0x3 out of bounds>, len = -1389001313}, ttl = { s = 0xa3664c "etOLD@31.223.168.251:5060;alias=31.223.168.251~5060~1 SIP/2.0\r\nVia: SIP/2.0/UDP 192.168.10.233:5060;branch=z9hG4bK1fbee766;rport\r\nRoute: <sip:192.168.10.230;r2=on;lr;ftag=as0b87791a;did=3e4.b5c1;nat=y"..., len = -731535288}, user_param = {s = 0x0, len = 1}, maddr = {s = 0x13c4 <Address 0x13c4 out of bounds>, len = 0}, method = { s = 0x0, len = 5060}, lr = {s = 0x7fce00000000 <Address 0x7fce00000000 out of bounds>, len = -1310504815}, r2 = {s = 0x8000000 <Address 0x8000000 out of bounds>, len = 129936856}, gr = {s = 0x370690 <Address 0x370690 out of bounds>, len = -1011016704}, transport_val = {s = 0x0, len = -1}, ttl_val = {s = 0x7fced4cb4598 "\001", len = -722431032}, user_param_val = {s = 0x7ffeb0b402b0 "\003", len = -1330382928}, maddr_val = {s = 0x7fced4f093c8 "Z\025", len = 5514780}, method_val = { s = 0x7fcead5b9a00 "mu:\255\316\177", len = -1386505744}, lr_val = {s = 0x7fcead5b99e0 "du:\255\316\177", len = -1386505776}, r2_val = { s = 0x7fcead5b99c0 "]u:\255\316\177", len = -1386506000}, gr_val = {s = 0x7fcead5b9900 "\356t:\255\316\177", len = -1386505856}} u = <optimized out> port = <optimized out> dst_host = <optimized out> i = <optimized out> flags = <optimized out> avp = <optimized out> ---Type <return> to continue, or q <return> to quit--- st = {flags = 3572536264, id = 32718, name = {n = 4366602, s = {s = 0x42a10a "H\213\r\177\201V", len = -1330380112}, re = 0x42a10a}, avp = 0x53b2e8} sct = <optimized out> sjt = <optimized out> rve = 0x7fced4cb4d50 mct = <optimized out> rv = <optimized out> rv1 = <optimized out> c1 = {cache_type = RV_CACHE_PVAR, val_type = 32718, c = {avp_val = {n = -1386505920, s = {s = 0x7fcead5b9940 ")u:\255\316\177", len = -1386506016}, re = 0x7fcead5b9940}, pval = {rs = {s = 0x7fcead5b9940 ")u:\255\316\177", len = -1386506016}, ri = -1386506048, flags = 32718}}, i2s = "\260\230[\255\316\177\000\000\240\230[\255\316\177\000\000И[\255\316\177"} s = {s = 0x0, len = 0} srevp = {0xfffffffb, 0x7ffeb0b3f350} mod_f_params = {{type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = { number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, { type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}} __FUNCTION__ = "do_action" #11 0x000000000042a10a in run_actions (h=h@entry=0x7ffeb0b402b0, a=0x7fced4cb8f88, msg=msg@entry=0x7fced4f093c8) at action.c:1549 t = <optimized out> ret = <optimized out> mod = <optimized out> ms = 0 __FUNCTION__ = "run_actions" #12 0x000000000042bcf2 in do_action (h=h@entry=0x7ffeb0b402b0, a=a@entry=0x7fced4cb91e8, msg=msg@entry=0x7fced4f093c8) at action.c:1049 ret = 1 v = 1 dst = {send_sock = 0x0, to = {s = {sa_family = 57696, sa_data = "\244\000\000\000\000\000ȓ\360\324\316\177\000"}, sin = {sin_family = 57696, sin_port = 164, sin_addr = { s_addr = 0}, sin_zero = "ȓ\360\324\316\177\000"}, sin6 = {sin6_family = 57696, sin6_port = 164, sin6_flowinfo = 0, sin6_addr = {__in6_u = { __u6_addr8 = "ȓ\360\324\316\177\000\000^\337r\261\316\177\000", __u6_addr16 = {37832, 54512, 32718, 0, 57182, 45426, 32718, 0}, __u6_addr32 = {3572536264, 32718, 2977095518, 32718}}}, sin6_scope_id = 3019872104}}, id = 32718, proto = -111 '\221', send_flags = {f = 72 'H', blst_imask = 227 '\343'}} tmp = <optimized out> new_uri = <optimized out> end = <optimized out> crt = <optimized out> cmd = <optimized out> len = <optimized out> user = <optimized out> uri = {user = {s = 0x100000004 <Address 0x100000004 out of bounds>, len = 131072}, passwd = {s = 0x0, len = 0}, host = {s = 0x4 <Address 0x4 out of bounds>, len = -1275390656}, port = {s = 0x7fceb3fb1715 "sip:+31203128011@sip.wlvoip.net5060et60DP 1\030", len = 31}, params = { s = 0x100000004 <Address 0x100000004 out of bounds>, len = 10046212}, sip_params = {s = 0x0, len = -1388862181}, headers = {s = 0x0, len = -1330381968}, port_no = 3, proto = 0, type = ERROR_URI_T, flags = 2905997787, transport = {s = 0x7ffeb0b3f920 "", len = -726918520}, ttl = {s = 0x7fced4f093c8 "Z\025", len = 16465}, user_param = { s = 0x7fceb3ec2bf4 "", len = 0}, maddr = {s = 0x7fced4dd40e8 "\001", len = -1388997077}, method = {s = 0x7fced4dcddc0 "", len = -1330380112}, lr = {s = 0x0, len = -722431032}, r2 = {s = 0x7ffeb0b402b0 "\003", len = -1330381440}, gr = {s = 0x7fced4f093c8 "Z\025", len = 4420932}, transport_val = {s = 0x0, len = -1}, ttl_val = { s = 0x7ffeb0b402b0 "\003", len = 1034164412}, user_param_val = {s = 0x7fced4f093c8 "Z\025", len = -1330380112}, maddr_val = {s = 0x7ffeb0b3fd80 "", len = -722431032}, method_val = {s = 0xe211f030cd8418bc <Address 0xe211f030cd8418bc out of bounds>, len = 1939871932}, lr_val = {s = 0x0, len = 56}, r2_val = { s = 0x38 <Address 0x38 out of bounds>, len = 6344353}, gr_val = {s = 0x7fced4b5f6e0 "\024", len = -1315021152}} next_hop = {user = {s = 0x7fceb19e5cc0 "\275p\002", len = 16482}, passwd = {s = 0x7fceb3fa4c88 "\001", len = 160167}, host = {s = 0x7fceb3fa8da0 "p\202\344\263\316\177", len = -1330381632}, port = {s = 0x2 <Address 0x2 out of bounds>, len = -1388814079}, params = {s = 0x8000000 <Address 0x8000000 out of bounds>, len = 129937120}, ---Type <return> to continue, or q <return> to quit--- sip_params = {s = 0x370588 <Address 0x370588 out of bounds>, len = 4280608}, headers = {s = 0x3 <Address 0x3 out of bounds>, len = -1330381440}, port_no = 688, proto = 45236, type = 32766, flags = 5485288, transport = {s = 0x2 <Address 0x2 out of bounds>, len = -1275095000}, ttl = { s = 0x7fced4dcbc00 "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP 176.74.252.249:5060;received=176.74.252.249;branch=z9hG4bK6e6c5f53;rport=35255\r\nRecord-Route: sip:192.168.10.230;r2=on;lr;ftag=as77d9913b;did=3f5.3321;nat=yes\r\nRecor"..., len = -1275442040}, user_param = {s = 0x7fced4f093c8 "Z\025", len = -1275199272}, maddr = { s = 0x7fceb3fa8da0 "p\202\344\263\316\177", len = -1330381632}, method = {s = 0x2 <Address 0x2 out of bounds>, len = -1388928332}, lr = { s = 0x8000000 <Address 0x8000000 out of bounds>, len = 129941872}, r2 = {s = 0x36f2f8 <Address 0x36f2f8 out of bounds>, len = 4275856}, gr = {s = 0x48a0f8 "\207\231", len = 8}, transport_val = {s = 0x154 <Address 0x154 out of bounds>, len = -1}, ttl_val = {s = 0x7fced4ac1c80 "\001", len = -722431032}, user_param_val = { s = 0x7ffeb0b402b0 "\003", len = -1330381440}, maddr_val = {s = 0x7fced4f093c8 "Z\025", len = 5514780}, method_val = { s = 0x7fced4dcbc00 "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP 176.74.252.249:5060;received=176.74.252.249;branch=z9hG4bK6e6c5f53;rport=35255\r\nRecord-Route: sip:192.168.10.230;r2=on;lr;ftag=as77d9913b;did=3f5.3321;nat=yes\r\nRecor"..., len = 973}, lr_val = {s = 0x7fceb3ff9878 "\230\273\356\324\316\177", len = 0}, r2_val = {s = 0xa466b0 "", len = -1275095192}, gr_val = {s = 0x30 <Address 0x30 out of bounds>, len = -723715568}} u = <optimized out> port = <optimized out> dst_host = <optimized out> i = <optimized out> flags = <optimized out> avp = <optimized out> st = {flags = 3572536264, id = 32718, name = {n = 4366602, s = {s = 0x42a10a "H\213\r\177\201V", len = 0}, re = 0x42a10a}, avp = 0x7fceb3f6b8a0} sct = <optimized out> sjt = <optimized out> rve = <optimized out> mct = <optimized out> rv = <optimized out> rv1 = <optimized out> c1 = {cache_type = 134217728, val_type = RV_NONE, c = {avp_val = {n = 129941688, s = {s = 0x7bec0b8 <Address 0x7bec0b8 out of bounds>, len = 3601328}, re = 0x7bec0b8}, pval = {rs = {s = 0x7bec0b8 <Address 0x7bec0b8 out of bounds>, len = 3601328}, ri = 4276040, flags = 0}}, i2s = "\370\240H\000\000\000\000\000\b", '\000' <repeats 12 times>} s = {s = 0x0, len = 0} srevp = {0xfffffffb, 0x7ffeb0b3f920} mod_f_params = {{type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = { number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, { type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}} __FUNCTION__ = "do_action" #13 0x000000000042a10a in run_actions (h=h@entry=0x7ffeb0b402b0, a=0x7fced4cb3f48, msg=msg@entry=0x7fced4f093c8) at action.c:1549 t = <optimized out> ret = <optimized out> mod = <optimized out> ms = 0 __FUNCTION__ = "run_actions" #14 0x000000000042bde0 in do_action (h=h@entry=0x7ffeb0b402b0, a=a@entry=0x7fced4ac3d70, msg=msg@entry=0x7fced4f093c8) at action.c:678 ret = -5 v = <optimized out> dst = {send_sock = 0x1, to = {s = {sa_family = 14152, sa_data = "\335\324\316\177\000\000\200f\243\000\000\000\000"}, sin = {sin_family = 14152, sin_port = 54493, sin_addr = {s_addr = 32718}, sin_zero = "\200f\243\000\000\000\000"}, sin6 = {sin6_family = 14152, sin6_port = 54493, sin6_flowinfo = 32718, sin6_addr = {__in6_u = { __u6_addr8 = "\200f\243", '\000' <repeats 12 times>, __u6_addr16 = {26240, 163, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {10708608, 0, 0, 0}}}, sin6_scope_id = 3571266416}}, id = 32718, proto = -57 '\307', send_flags = {f = 102 'f', blst_imask = 163 '\243'}} tmp = <optimized out> ---Type <return> to continue, or q <return> to quit--- new_uri = <optimized out> end = <optimized out> crt = <optimized out> cmd = <optimized out> len = <optimized out> user = <optimized out> uri = {user = {s = 0x7fced4dd40e8 "\001", len = -1315036816}, passwd = {s = 0x7fced4ef53b8 "\020\325\177\255\316\177", len = -1317733006}, host = { s = 0x7fced4f093c8 "Z\025", len = 359}, port = {s = 0x7fceb3f7b208 "\230\273\356\324\316\177", len = 0}, params = {s = 0x7fced485e210 "", len = -1275084800}, sip_params = {s = 0x7fceb3ffc208 "", len = -1317725785}, headers = {s = 0x7fceb3ffc000 "\260<\343\263\316\177", len = 404}, port_no = 0, proto = 0, type = ERROR_URI_T, flags = 2977095518, transport = {s = 0x7fceb17cfc2e "tm: t_reply.c", len = -1317206832}, ttl = {s = 0x8a6 <Address 0x8a6 out of bounds>, len = -1317574291}, user_param = { s = 0x0, len = 0}, maddr = {s = 0x0, len = 0}, method = {s = 0x7fced4f093c8 "Z\025", len = -723697432}, lr = {s = 0x7fceb3fa8fa8 "", len = -722431032}, r2 = { s = 0x7fceb3fa8da0 "p\202\344\263\316\177", len = 200}, gr = {s = 0x1e7 <Address 0x1e7 out of bounds>, len = -1317536758}, transport_val = {s = 0x0, len = 1468565862}, ttl_val = {s = 0x7fced4dd3c68 "H.Y\002", len = -1281634096}, user_param_val = {s = 0x992290 "\314\060\356\263\316\177", len = 69}, maddr_val = {s = 0x993cb4 "\377\377", len = 0}, method_val = { s = 0x24bda00 "insert into `dialog` (`hash_entry`,`hash_id`,`callid`,`from_uri`,`from_tag`,`to_uri`,`to_tag`,`caller_sock`,`callee_sock`,`start_time`,`state`,`timeout`,`caller_cseq`,`callee_cseq`,`caller_contact`,`c"..., len = -731621092}, lr_val = {s = 0x2 <Address 0x2 out of bounds>, len = -1310504815}, r2_val = { s = 0x8000000 <Address 0x8000000 out of bounds>, len = 129942584}, gr_val = {s = 0x36f030 <Address 0x36f030 out of bounds>, len = 4275144}} next_hop = {user = {s = 0x48a0f8 "\207\231", len = 8}, passwd = {s = 0x159 <Address 0x159 out of bounds>, len = 64}, host = {s = 0x40 <Address 0x40 out of bounds>, len = 6344353}, port = {s = 0xa466c8 "", len = 10774544}, params = {s = 0x167 <Address 0x167 out of bounds>, len = 6333667}, sip_params = {s = 0x0, len = -1276371820}, headers = { s = 0xa36778 ":+31737470617@31.223.168.234>;tag=as148f41b1\r\nTo: +33612818032 sip:signetOLD@sip.wlvoip.net;tag=as0b87791a\r\nCall-ID: 25750e286a5654361ef9405d72edbc54@sip.wlvoip.net\r\nCSeq: 102 BYE\r\nUser-Agent: sip.w"..., len = 52}, port_no = 0, proto = 0, type = 32718, flags = 3018588160, transport = {s = 0x40 <Address 0x40 out of bounds>, len = -723699896}, ttl = {s = 0x30 <Address 0x30 out of bounds>, len = -723715568}, user_param = {s = 0x0, len = -591593503}, maddr = {s = 0x258ac6c "", len = 2510}, method = {s = 0x38 <Address 0x38 out of bounds>, len = 10708679}, lr = {s = 0x6 <Address 0x6 out of bounds>, len = -723700880}, r2 = {s = 0xa368a9 "", len = 100}, gr = { s = 0xa36690 "SIP/2.0/UDP 192.168.10.233:5060;branch=z9hG4bK1fbee766;rport\r\nRoute: sip:192.168.10.230;r2=on;lr;ftag=as0b87791a;did=3e4.b5c1;nat=yes,<sip:31.223.168.234;r2=on;lr;ftag=as0b87791a;did=3e4.b5c1;nat=ye"..., len = 6727432}, transport_val = {s = 0x8000000 <Address 0x8000000 out of bounds>, len = 129942712}, ttl_val = { s = 0x36efb0 <Address 0x36efb0 out of bounds>, len = 4275016}, user_param_val = {s = 0x48a0f8 "\207\231", len = 8}, maddr_val = {s = 0x15b <Address 0x15b out of bounds>, len = 232}, method_val = {s = 0xe8 <Address 0xe8 out of bounds>, len = 6344353}, lr_val = {s = 0x5007fcedce08e40 <Address 0x5007fcedce08e40 out of bounds>, len = -1275010320}, r2_val = {s = 0x7fcead5b9bc0 "\001", len = 1}, gr_val = {s = 0x0, len = -1276371820}} u = <optimized out> port = <optimized out> dst_host = <optimized out> i = 21 flags = <optimized out> avp = <optimized out> st = {flags = 180, id = 0, name = {n = -722431032, s = {s = 0x7fced4f093c8 "Z\025", len = -1314950320}, re = 0x7fced4f093c8}, avp = 0x7ffeb0b3ffc0} sct = <optimized out> sjt = <optimized out> rve = <optimized out> mct = <optimized out> rv = 0xa36781 rv1 = <optimized out> c1 = {cache_type = RV_CACHE_EMPTY, val_type = RV_NONE, c = {avp_val = {n = 10708645, s = { s = 0xa366a5 "0.233:5060;branch=z9hG4bK1fbee766;rport\r\nRoute: sip:192.168.10.230;r2=on;lr;ftag=as0b87791a;did=3e4.b5c1;nat=yes,sip:31.223.168.234;r2=on;lr;ftag=as0b87791a;did=3e4.b5c1;nat=yes\r\nMax-Forwards: 69\r"..., len = 0}, re = 0xa366a5}, pval = {rs = { s = 0xa366a5 "0.233:5060;branch=z9hG4bK1fbee766;rport\r\nRoute: sip:192.168.10.230;r2=on;lr;ftag=as0b87791a;did=3e4.b5c1;nat=yes,sip:31.223.168.234;r2=on;lr;ftag=as0b87791a;did=3e4.b5c1;nat=yes\r\nMax-Forwards: 69\r"..., len = 0}, ri = 3, flags = 29}}, i2s = "\000\000\000\000\316\177\000\000t\000\000\000\316\177\000\000\000\000\000\000\000"} s = {s = 0x7ffeb0b3ffc0 "", len = -726278104} srevp = {0x7fced4f093c8, 0xc8} mod_f_params = {{type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, ---Type <return> to continue, or q <return> to quit--- string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = { number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, { type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}} __FUNCTION__ = "do_action" #15 0x000000000042a10a in run_actions (h=h@entry=0x7ffeb0b402b0, a=a@entry=0x7fced4ac1698, msg=msg@entry=0x7fced4f093c8) at action.c:1549 t = <optimized out> ret = <optimized out> mod = <optimized out> ms = 0 __FUNCTION__ = "run_actions" #16 0x00000000004375d0 in run_top_route (a=0x7fced4ac1698, msg=msg@entry=0x7fced4f093c8, c=c@entry=0x0) at action.c:1635 ctx = {rec_lev = 3, run_flags = 0, last_retcode = 1, jmp_env = {{__jmpbuf = {140731863007920, 2156258125485971644, 140526308038296, 140526312526792, 0, 0, -2156678592105015108, 2156258654672656572}, __mask_was_saved = 0, __saved_mask = {__val = {10708544, 617, 0, 0, 6452880, 140526311250576, 7138637, 3200, 140526312526792, 0, 140526311260392, 140731863008128, 140526312441400, 1073741825, 140525724418962, 140526312427128}}}}} p = 0x7ffeb0b402b0 ret = <optimized out> sfbk = 0 #17 0x0000000000504386 in receive_msg (buf=<optimized out>, len=<optimized out>, rcv_info=<optimized out>) at receive.c:240 msg = 0x7fced4f093c8 ctx = {rec_lev = 134217728, run_flags = 0, last_retcode = 129951416, jmp_env = {{__jmpbuf = {3595920, 4266312, 140526442034080, 140526452292352, 297, 4294967295, 32, 5}, __mask_was_saved = 10031968, __saved_mask = {__val = {0, 0, 0, 0, 140526450104770, 140522739990533, 0, 140525758673168, 140526442034080, 128, 140526311259528, 140526312404216, 140526452242088, 140731863008806, 140526450129445, 140526452242088}}}}} ret = <optimized out> inb = { s = 0xa36640 "BYE sip:signetOLD@31.223.168.251:5060;alias=31.223.168.251~5060~1 SIP/2.0\r\nVia: SIP/2.0/UDP 192.168.10.233:5060;branch=z9hG4bK1fbee766;rport\r\nRoute: <sip:192.168.10.230;r2=on;lr;ftag=as0b87791a;did=3e"..., len = 617} netinfo = {data = {s = 0x1 <Address 0x1 out of bounds>, len = -1330379296}, dst = 0x7f, rcv = 0x7fceb1e34891} __FUNCTION__ = "receive_msg" #18 0x00000000005f5bd4 in udp_rcv_loop () at udp_server.c:495 len = 617 buf = "BYE sip:signetOLD@31.223.168.251:5060;alias=31.223.168.251~5060~1 SIP/2.0\r\nVia: SIP/2.0/UDP 192.168.10.233:5060;branch=z9hG4bK1fbee766;rport\r\nRoute: <sip:192.168.10.230;r2=on;lr;ftag=as0b87791a;did=3e"... from = 0x7fced4dd3d88 fromlen = 16 ri = {src_ip = {af = 2, len = 4, u = {addrl = {140526649780416, 5}, addr32 = {3909789888, 32718, 5, 0}, addr16 = {43200, 59658, 32718, 0, 5, 0, 0, 0}, addr = "\300\250\n\351\316\177\000\000\005\000\000\000\000\000\000"}}, dst_ip = {af = 2, len = 4, u = {addrl = {3859458240, 0}, addr32 = {3859458240, 0, 0, 0}, addr16 = {43200, 58890, 0, 0, 0, 0, 0, 0}, addr = "\300\250\n\346", '\000' <repeats 11 times>}}, src_port = 5060, dst_port = 5060, proto_reserved1 = 0, proto_reserved2 = 0, src_su = {s = {sa_family = 2, sa_data = "\023\304\300\250\n\351\000\000\000\000\000\000\000"}, sin = {sin_family = 2, sin_port = 50195, sin_addr = { s_addr = 3909789888}, sin_zero = "\000\000\000\000\000\000\000"}, sin6 = {sin6_family = 2, sin6_port = 50195, sin6_flowinfo = 3909789888, sin6_addr = {__in6_u = { __u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 0}}, bind_address = 0x7fced4eeb4f8, proto = 1 '\001'} __FUNCTION__ = "udp_rcv_loop" #19 0x00000000004b2625 in main_loop () at main.c:1600 i = <optimized out> pid = <optimized out> si = <optimized out> si_desc = "udp receiver child=1 sock=192.168.10.230:5060", '\000' <repeats 11 times>, "\003", '\000' <repeats 15 times>, "\004", '\000' <repeats 16 times>, "\034\275\303\035\202\340Y\000\000\000\000\000\000\000\000\000\034\275\303\035\202\340Y\350\341\244\000\000\000\000\000\004\000\000\000\000\000\000" ---Type <return> to continue, or q <return> to quit--- nrprocs = <optimized out> woneinit = <optimized out> __FUNCTION__ = "main_loop" #20 0x0000000000427e2b in main (argc=<optimized out>, argv=<optimized out>) at main.c:2616 cfg_stream = <optimized out> c = <optimized out> r = <optimized out> tmp = 0x7ffeb0b41d7b "" tmp_len = 0 port = 1 proto = 1 options = 0x6e8c70 ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:x:X:" ret = -1 seed = 690501560 rfd = 4 debug_save = <optimized out> debug_flag = <optimized out> dont_fork_cnt = <optimized out> n_lst = 0x4 p = <optimized out> st = {st_dev = 14, st_ino = 10524, st_nlink = 2, st_mode = 16877, st_uid = 107, st_gid = 112, __pad0 = 0, st_rdev = 0, st_size = 60, st_blksize = 4096, st_blocks = 0, st_atim = {tv_sec = 1468460663, tv_nsec = 740929866}, st_mtim = {tv_sec = 1468531732, tv_nsec = 898509779}, st_ctim = {tv_sec = 1468531739, tv_nsec = 982096020}, __unused = {0, 0, 0}} __FUNCTION__ = "main"

Second crash:

(gdb) info locals No locals. (gdb) bt full #0 atomic_get (v=0x7f6264d11378) at ../../mem/../atomic/atomic_common.h:74 No locals. #1 dlg_unref (dlg=dlg@entry=0x7f585c494b40, cnt=cnt@entry=1) at dlg_hash.c:921 mypid = 12376 d_entry = 0x7f6264d11360 __FUNCTION__ = "dlg_unref" #2 0x00007f5855912802 in dlg_run_event_route (dlg=dlg@entry=0x7f585c494b40, msg=msg@entry=0x7f587d4be8e8, ostate=<optimized out>, nstate=<optimized out>) at dlg_handlers.c:1630 fmsg = 0x7f587d4be8e8 rt = 3 bkroute = 1 __FUNCTION__ = "dlg_run_event_route" #3 0x00007f585591416a in dlg_onroute (req=0x7f587d4be8e8, route_params=<optimized out>, param=<optimized out>) at dlg_handlers.c:1307 dlg = 0x7f585c494b40 iuid = <optimized out> val = { s = 0xa36706 "6ad.54d;nat=yes>,sip:31.223.168.234;r2=on;lr;ftag=as5d3a5a72;did=6ad.54d;nat=yes\r\nMax-Forwards: 69\r\nFrom: sip:+31492382120@31.223.168.234;tag=as6035a2af\r\nTo: +31499847821 <sip:signetOLD@sip.wlvoip"..., len = 7} callid = { s = 0xa367ed "1201b1b5692a911a1aa7e060399ad825@sip.wlvoip.net\r\nCSeq: 102 BYE\r\nUser-Agent: sip.wlvoip.net\r\nX-Asterisk-HangupCause: Normal Clearing\r\nX-Asterisk-HangupCauseCode: 16\r\nContent-Length: 0\r\n\r\n", len = 47} ftag = { s = 0xa36798 "as6035a2af\r\nTo: +31499847821 sip:signetOLD@sip.wlvoip.net;tag=as5d3a5a72\r\nCall-ID: 1201b1b5692a911a1aa7e060399ad825@sip.wlvoip.net\r\nCSeq: 102 BYE\r\nUser-Agent: sip.wlvoip.net\r\nX-Asterisk-HangupCause:"..., len = 10} ttag = { s = 0xa367d8 "as5d3a5a72\r\nCall-ID: 1201b1b5692a911a1aa7e060399ad825@sip.wlvoip.net\r\nCSeq: 102 BYE\r\nUser-Agent: sip.wlvoip.net\r\nX-Asterisk-HangupCause: Normal Clearing\r\nX-Asterisk-HangupCauseCode: 16\r\nContent-Length"..., len = 10} h_entry = 3494 h_id = 3397 new_state = 5 old_state = 4 unref = <optimized out> event = 7 timeout = 5 reset = <optimized out> dir = 2 ret = 0 __FUNCTION__ = "dlg_onroute" #4 0x00007f585965b0e2 in run_rr_callbacks (req=req@entry=0x7f587d4be8e8, rr_param=rr_param@entry=0x7f58598677a0) at rr_cb.c:96 l_param = { s = 0xa366e9 "r2=on;lr;ftag=as5d3a5a72;did=6ad.54d;nat=yes>,sip:31.223.168.234;r2=on;lr;ftag=as5d3a5a72;did=6ad.54d;nat=yes\r\nMax-Forwards: 69\r\nFrom: sip:+31492382120@31.223.168.234;tag=as6035a2af\r\nTo: +31499847"..., len = 44} cbp = 0x7f587d06c0a0 __FUNCTION__ = "run_rr_callbacks" #5 0x00007f58596452c5 in after_loose (_m=0x7f587d4be8e8, preloaded=0) at loose.c:919 hdr = <optimized out> puri = {user = {s = 0x0, len = 0}, passwd = {s = 0x0, len = 0}, host = { s = 0xa3671c "31.223.168.234;r2=on;lr;ftag=as5d3a5a72;did=6ad.54d;nat=yes>\r\nMax-Forwards: 69\r\nFrom: sip:+31492382120@31.223.168.234;tag=as6035a2af\r\nTo: +31499847821 sip:signetOLD@sip.wlvoip.net;tag=as5d3a5a72\r\n"..., len = 14}, port = {s = 0x0, len = 0}, params = { s = 0xa3672b "r2=on;lr;ftag=as5d3a5a72;did=6ad.54d;nat=yes>\r\nMax-Forwards: 69\r\nFrom: sip:+31492382120@31.223.168.234;tag=as6035a2af\r\nTo: +31499847821 sip:signetOLD@sip.wlvoip.net;tag=as5d3a5a72\r\nCall-ID: 1201b1"..., len = 44}, sip_params = { ---Type <return> to continue, or q <return> to quit--- s = 0xa3672b "r2=on;lr;ftag=as5d3a5a72;did=6ad.54d;nat=yes>\r\nMax-Forwards: 69\r\nFrom: sip:+31492382120@31.223.168.234;tag=as6035a2af\r\nTo: +31499847821 sip:signetOLD@sip.wlvoip.net;tag=as5d3a5a72\r\nCall-ID: 1201b1"..., len = 44}, headers = {s = 0x0, len = 0}, port_no = 0, proto = 0, type = SIP_URI_T, flags = 0, transport = {s = 0x0, len = 0}, ttl = {s = 0x0, len = 0}, user_param = {s = 0x0, len = 0}, maddr = {s = 0x0, len = 0}, method = {s = 0x0, len = 0}, lr = { s = 0xa36731 "lr;ftag=as5d3a5a72;did=6ad.54d;nat=yes>\r\nMax-Forwards: 69\r\nFrom: sip:+31492382120@31.223.168.234;tag=as6035a2af\r\nTo: +31499847821 sip:signetOLD@sip.wlvoip.net;tag=as5d3a5a72\r\nCall-ID: 1201b1b5692a"..., len = 2}, r2 = { s = 0xa3672b "r2=on;lr;ftag=as5d3a5a72;did=6ad.54d;nat=yes>\r\nMax-Forwards: 69\r\nFrom: sip:+31492382120@31.223.168.234;tag=as6035a2af\r\nTo: +31499847821 sip:signetOLD@sip.wlvoip.net;tag=as5d3a5a72\r\nCall-ID: 1201b1"..., len = 5}, gr = {s = 0x0, len = 0}, transport_val = {s = 0x0, len = 0}, ttl_val = {s = 0x0, len = 0}, user_param_val = { s = 0x0, len = 0}, maddr_val = {s = 0x0, len = 0}, method_val = {s = 0x0, len = 0}, lr_val = {s = 0x0, len = 0}, r2_val = { s = 0xa3672e "on;lr;ftag=as5d3a5a72;did=6ad.54d;nat=yes>\r\nMax-Forwards: 69\r\nFrom: sip:+31492382120@31.223.168.234;tag=as6035a2af\r\nTo: +31499847821 sip:signetOLD@sip.wlvoip.net;tag=as5d3a5a72\r\nCall-ID: 1201b1b56"..., len = 2}, gr_val = {s = 0x0, len = 0}} rt = <optimized out> status = <optimized out> uri = { s = 0xa366d6 "sip:192.168.10.230;r2=on;lr;ftag=as5d3a5a72;did=6ad.54d;nat=yes>,sip:31.223.168.234;r2=on;lr;ftag=as5d3a5a72;did=6ad.54d;nat=yes\r\nMax-Forwards: 69\r\nFrom: sip:+31492382120@31.223.168.234;tag=as6035"..., len = 63} si = <optimized out> uri_is_myself = 12376 use_ob = 0 __FUNCTION__ = "after_loose" #6 0x000000000042b618 in do_action (h=h@entry=0x7ffd6e277fd0, a=a@entry=0x7f587d264338, msg=msg@entry=0x7f587d4be8e8) at action.c:1060 ret = -5 v = <optimized out> dst = {send_sock = 0x6e2789c0, to = {s = {sa_family = 33376, sa_data = "'n\375\177\000\000\377\377\377\377X\177\000"}, sin = {sin_family = 33376, sin_port = 28199, sin_addr = {s_addr = 32765}, sin_zero = "\377\377\377\377X\177\000"}, sin6 = {sin6_family = 33376, sin6_port = 28199, sin6_flowinfo = 32765, sin6_addr = {__in6_u = { __u6_addr8 = "\377\377\377\377X\177\000\000DuC\000\000\000\000", __u6_addr16 = {65535, 65535, 32600, 0, 30020, 67, 0, 0}, __u6_addr32 = {4294967295, 32600, 4420932, 0}}}, sin6_scope_id = 0}}, id = 0, proto = -20 '\354', send_flags = {f = 123 '{', blst_imask = 7 '\a'}} tmp = <optimized out> new_uri = <optimized out> end = <optimized out> crt = <optimized out> cmd = 0x7f587d0575a8 len = <optimized out> user = <optimized out> uri = {user = {s = 0x7f587d4d41e0 "\001", len = 1481255020}, passwd = {s = 0x4ac0a0 "", len = 8}, host = {s = 0x1d6 <Address 0x1d6 out of bounds>, len = 2102216368}, port = { s = 0x7f587d4d46b0 "6", len = 6334740}, params = {s = 0x40 <Address 0x40 out of bounds>, len = 6344353}, sip_params = {s = 0x7f587cc0a845 "%-d", len = 2102218424}, headers = { s = 0xa367ba "signetOLD@sip.wlvoip.net>;tag=as5d3a5a72\r\nCall-ID: 1201b1b5692a911a1aa7e060399ad825@sip.wlvoip.net\r\nCSeq: 102 BYE\r\nUser-Agent: sip.wlvoip.net\r\nX-Asterisk-HangupCause: Normal Clearing\r\nX-Asterisk-Hangu"..., len = 9}, port_no = 0, proto = 0, type = ERROR_URI_T, flags = 4294967295, transport = {s = 0x7f587d225270 "\001", len = 2102126824}, ttl = {s = 0x7ffd6e279500 "\003", len = 1848082016}, user_param = {s = 0x7f587d4be8e8 "\353\022", len = 5514780}, maddr = {s = 0x0, len = 0}, method = {s = 0x0, len = 0}, lr = {s = 0x100000000 <Address 0x100000000 out of bounds>, len = 0}, r2 = {s = 0x0, len = 0}, gr = {s = 0x0, len = 0}, transport_val = { s = 0x0, len = 0}, ttl_val = {s = 0x0, len = 0}, user_param_val = {s = 0x0, len = 0}, maddr_val = {s = 0x0, len = 0}, method_val = {s = 0x0, len = 0}, lr_val = {s = 0x0, len = -2027077120}, r2_val = {s = 0x0, len = 2099407760}, gr_val = {s = 0x7f587d4be8e8 "\353\022", len = 0}} next_hop = {user = {s = 0xfffffffb <Address 0xfffffffb out of bounds>, len = 1848087808}, passwd = {s = 0x7f587d4be8e8 "\353\022", len = 4366602}, host = {s = 0x0, len = 0}, port = {s = 0x0, len = 0}, params = {s = 0x0, len = 0}, sip_params = {s = 0x0, len = 0}, headers = {s = 0x0, len = 1511}, port_no = 59624, proto = 32075, type = 32600, flags = 7, transport = {s = 0xfffffffb <Address 0xfffffffb out of bounds>, len = 2097548304}, ttl = {s = 0x7f587d4be8e8 "\353\022", len = 1481220363}, user_param = { s = 0x7f587d248fd0 "\n", len = 2102126824}, maddr = {s = 0x7ffd6e279500 "\003", len = 2099407760}, method = {s = 0x7f587d4be8e8 "\353\022", len = 1494951765}, lr = { s = 0x0, len = 4}, r2 = {s = 0x7f5800000000 <Address 0x7f5800000000 out of bounds>, len = -2027077120}, gr = {s = 0x7f580000000a <Address 0x7f580000000a out of bounds>, len = 2099408064}, transport_val = {s = 0x7f587d4be8e8 "\353\022", len = 1848087808}, ttl_val = {s = 0x1 <Address 0x1 out of bounds>, len = 1848087808}, user_param_val = {s = 0x7f587d4be8e8 "\353\022", len = 4373746}, maddr_val = {s = 0x7f587d4be8e8 "\353\022", len = 4371992}, method_val = { s = 0x70 <Address 0x70 out of bounds>, len = 1848082904}, lr_val = {s = 0x7f587cc08a7d "%s%.*s%s ", len = 0}, r2_val = { ---Type <return> to continue, or q <return> to quit--- s = 0x7ffd00000000 <Address 0x7ffd00000000 out of bounds>, len = 1848084928}, gr_val = {s = 0x0, len = 2102126824}} u = <optimized out> port = <optimized out> dst_host = <optimized out> i = <optimized out> flags = <optimized out> avp = <optimized out> st = {flags = 1848087808, id = 32765, name = {n = 5485288, s = { s = 0x53b2e8 "\205\300A\017\237\300\061\300E\017\266\310D\211\v\203e\004\371\351 \377\377\377H\211\372H\215q\bH\215|$P\350?\244\001", len = 2102206808}, re = 0x53b2e8}, avp = 0x7f587d4d4410} sct = <optimized out> sjt = <optimized out> rve = <optimized out> mct = <optimized out> rv = <optimized out> rv1 = <optimized out> c1 = {cache_type = 1848087184, val_type = 32765, c = {avp_val = {n = 2099481144, s = {s = 0x7f587d238a38 "T", len = 1}, re = 0x7f587d238a38}, pval = {rs = { s = 0x7f587d238a38 "T", len = 1}, ri = 1848083032, flags = 32765}}, i2s = "\001\000\000\000X\177\000\000\003\000\000\000\000\000\000\000\232\213\300|X\177"} s = {s = 0x0, len = 789} srevp = {0x7ffd00000000, 0x7ffd6e277e60} mod_f_params = {{type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = { number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, { type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}} __FUNCTION__ = "do_action" #7 0x000000000042a10a in run_actions (h=h@entry=0x7ffd6e277fd0, a=0x7f587d264338, msg=0x7f587d4be8e8) at action.c:1549 t = <optimized out> ret = <optimized out> mod = <optimized out> ms = 0 __FUNCTION__ = "run_actions" #8 0x0000000000437544 in run_actions_safe (h=h@entry=0x7ffd6e279500, a=<optimized out>, msg=<optimized out>) at action.c:1614 ctx = {rec_lev = 1, run_flags = 0, last_retcode = 0, jmp_env = {{__jmpbuf = {140726451541248, 140440723803439876, 140018035976424, 140726451541248, 140726451536944, 140018035976424, -141812437538209020, 140441295872688900}, __mask_was_saved = 0, __saved_mask = {__val = {40558768, 140726451527616, 140726451523472, 140724603453441, 140726451536080, 40465328, 18446744073709551615, 4294967291, 140017369510558, 140018169122472, 2, 4, 8589934592, 3690471437605877047, 0, 0}}}}} ret = 12376 ign_on_break = 0 #9 0x000000000053b2e8 in rval_get_int (h=0x7ffd6e279500, msg=<optimized out>, i=0x7ffd6e278430, rv=rv@entry=0x7f587d264d58, cache=cache@entry=0x0) at rvalue.c:912 r_avp = <optimized out> avp_val = {n = 40561024, s = {s = 0x26ae980 "\004", len = -2062898852}, re = 0x26ae980} pval = {rs = {s = 0x3000000018 <Address 0x3000000018 out of bounds>, len = 0}, ri = 1848082944, flags = 32765} tmp = {s = 0x0, len = -2062586549} s = 0x0 r = <optimized out> ret = 0 destroy_pval = 0 __FUNCTION__ = "rval_get_int" #10 0x000000000054261c in rval_expr_eval_int (h=h@entry=0x7ffd6e279500, msg=msg@entry=0x7f587d4be8e8, res=res@entry=0x7ffd6e278430, rve=rve@entry=0x7f587d264d50) at rvalue.c:1910 ---Type <return> to continue, or q <return> to quit--- i1 = <optimized out> i2 = <optimized out> ret = -1 c1 = {cache_type = RV_CACHE_EMPTY, val_type = RV_NONE, c = {avp_val = {n = 0, s = {s = 0x0, len = 39701287}, re = 0x0}, pval = {rs = {s = 0x0, len = 39701287}, ri = 1848083472, flags = 32765}}, i2s = "\001", '\000' <repeats 15 times>"\327, \376\000\000\000"} c2 = {cache_type = 1848087808, val_type = 32765, c = {avp_val = {n = 2099408064, s = {s = 0x7f587d226cc0 "%", len = 2102126824}, re = 0x7f587d226cc0}, pval = {rs = { s = 0x7f587d226cc0 "%", len = 2102126824}, ri = 2099548112, flags = 32600}}, i2s = "\350\350K}X\177\000\000\003\000\000\000:\a==\000\000\000\000\000"} rv1 = <optimized out> rv2 = <optimized out> __FUNCTION__ = "rval_expr_eval_int" #11 0x000000000042bc91 in do_action (h=h@entry=0x7ffd6e279500, a=a@entry=0x7f587d268f88, msg=msg@entry=0x7f587d4be8e8) at action.c:1030 ret = -5 v = <optimized out> dst = {send_sock = 0x7f5855b69990, to = {s = {sa_family = 39280, sa_data = "\266UX\177\000\000P\231\266UX\177\000"}, sin = {sin_family = 39280, sin_port = 21942, sin_addr = { s_addr = 32600}, sin_zero = "P\231\266UX\177\000"}, sin6 = {sin6_family = 39280, sin6_port = 21942, sin6_flowinfo = 32600, sin6_addr = {__in6_u = { __u6_addr8 = "P\231\266UX\177\000\000`\231\266UX\177\000", __u6_addr16 = {39248, 21942, 32600, 0, 39264, 21942, 32600, 0}, __u6_addr32 = {1438030160, 32600, 1438030176, 32600}}}, sin6_scope_id = 1438030096}}, id = 32600, proto = 32 ' ', send_flags = {f = 153 '\231', blst_imask = 182 '\266'}} tmp = <optimized out> new_uri = <optimized out> end = <optimized out> crt = <optimized out> cmd = <optimized out> len = <optimized out> user = <optimized out> uri = {user = {s = 0x7f587d4be8e8 "\353\022", len = 5354102}, passwd = {s = 0x0, len = 0}, host = { s = 0x25dca00 "update `dialog` set `state`=4,`timeout`=1468582942,`caller_cseq`='3',`callee_cseq`='0',`caller_contact`='sip:dehaan01@176.74.254.179;alias=176.74.254.179~5060~1',`callee_contact`='sip:+31641157337@192"..., len = 2101683296}, port = {s = 0x7f587d452760 "", len = 1537539707}, params = {s = 0x7f587ce0e210 "", len = 2101683296}, sip_params = {s = 0x7f587ce0e210 "", len = 1537544586}, headers = { s = 0x25dca00 "update `dialog` set `state`=4,`timeout`=1468582942,`caller_cseq`='3',`callee_cseq`='0',`caller_contact`='sip:dehaan01@176.74.254.179;alias=176.74.254.179~5060~1',`callee_contact`='sip:+31641157337@192"..., len = 40507338}, port_no = 9312, proto = 32069, type = 32600, flags = 2095112720, transport = {s = 0x7ffd6e2785a0 "", len = 2099659072}, ttl = {s = 0x7f587d4be8e8 "\353\022", len = 2102126824}, user_param = {s = 0x7ffd6e279500 "\003", len = 0}, maddr = {s = 0x7f587d4be8e8 "\353\022", len = 5514780}, method = {s = 0x111 <Address 0x111 out of bounds>, len = 1848087808}, lr = {s = 0x0, len = 2102126824}, r2 = {s = 0x7ffd6e279500 "\003", len = 1848084992}, gr = {s = 0x7f587d4be8e8 "\353\022", len = 4420932}, transport_val = {s = 0x0, len = 1}, ttl_val = {s = 0x7ffd6e279500 "\003", len = 508396292}, user_param_val = {s = 0x7f587d4be8e8 "\353\022", len = 1848087808}, maddr_val = {s = 0x7ffd6e278a00 "\001", len = 2102126824}, method_val = { s = 0xfe082e57176d8304 <Address 0xfe082e57176d8304 out of bounds>, len = 1346994948}, lr_val = {s = 0x0, len = 1435889581}, r2_val = {s = 0x7f585bf8cf46 "`", len = 5060}, gr_val = {s = 0x2 <Address 0x2 out of bounds>, len = 0}} next_hop = {user = {s = 0x7f585c490ad0 "\002", len = 0}, passwd = {s = 0x7ffdfffffffb <Address 0x7ffdfffffffb out of bounds>, len = 10042548}, host = { s = 0x7f5800000016 <Address 0x7f5800000016 out of bounds>, len = 1435501873}, port = {s = 0x7f585bedf640 "\002", len = -2027077120}, params = {s = 0x7f585c490ad0 "\002", len = 1848084992}, sip_params = {s = 0x7ffd6e279500 "\003", len = 2102126824}, headers = {s = 0x7ffd6e279500 "\003", len = 1848084992}, port_no = 38144, proto = 28199, type = 32765, flags = 5485288, transport = {s = 0x3 <Address 0x3 out of bounds>, len = 1435534751}, ttl = { s = 0xa3664c "etOLD@31.223.168.251:5060;alias=31.223.168.251~5060~1 SIP/2.0\r\nVia: SIP/2.0/UDP 192.168.10.232:5060;branch=z9hG4bK7b052368;rport\r\nRoute: <sip:192.168.10.230;r2=on;lr;ftag=as5d3a5a72;did=6ad.54d;nat=ye"..., len = 2093000776}, user_param = {s = 0x0, len = 1}, maddr = {s = 0x13c4 <Address 0x13c4 out of bounds>, len = 0}, method = { s = 0x0, len = 5060}, lr = {s = 0x7f5800000000 <Address 0x7f5800000000 out of bounds>, len = 1514031249}, r2 = {s = 0x8000000 <Address 0x8000000 out of bounds>, len = 129902608}, gr = {s = 0x375da8 <Address 0x375da8 out of bounds>, len = -2027077120}, transport_val = {s = 0x0, len = -1}, ttl_val = {s = 0x7f587d264598 "\001", len = 2102126824}, user_param_val = {s = 0x7ffd6e279500 "\003", len = 1848084992}, maddr_val = {s = 0x7f587d4be8e8 "\353\022", len = 5514780}, method_val = { s = 0x7f5855b69a00 "mu\225UX\177", len = 1848085248}, lr_val = {s = 0x7ffd6e279290 "\006", len = 2102126824}, r2_val = {s = 0x7ffd6e279290 "\006", len = 1848085248}, gr_val = {s = 0x7f587d4be8e8 "\353\022", len = 5485239}} u = <optimized out> port = <optimized out> dst_host = <optimized out> ---Type <return> to continue, or q <return> to quit--- i = <optimized out> flags = <optimized out> avp = <optimized out> st = {flags = 2102126824, id = 32600, name = {n = 4366602, s = {s = 0x42a10a "H\213\r\177\201V", len = 1848087808}, re = 0x42a10a}, avp = 0x53b2e8} sct = <optimized out> sjt = <optimized out> rve = 0x7f587d264d50 mct = <optimized out> rv = <optimized out> rv1 = <optimized out> c1 = {cache_type = RV_CACHE_PVAR, val_type = 32600, c = {avp_val = {n = 1438030144, s = {s = 0x7f5855b69940 ")u\225UX\177", len = 1438030048}, re = 0x7f5855b69940}, pval = { rs = {s = 0x7f5855b69940 ")u\225UX\177", len = 1438030048}, ri = 1438030016, flags = 32600}}, i2s = "\260\230\266UX\177\000\000\240\230\266UX\177\000\000И\266UX\177"} s = {s = 0x0, len = 0} srevp = {0xfffffffb, 0x7ffd6e2785a0} mod_f_params = {{type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = { number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, { type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}} __FUNCTION__ = "do_action" #12 0x000000000042a10a in run_actions (h=h@entry=0x7ffd6e279500, a=0x7f587d268f88, msg=msg@entry=0x7f587d4be8e8) at action.c:1549 t = <optimized out> ret = <optimized out> mod = <optimized out> ms = 0 __FUNCTION__ = "run_actions" #13 0x000000000042bcf2 in do_action (h=h@entry=0x7ffd6e279500, a=a@entry=0x7f587d2691e8, msg=msg@entry=0x7f587d4be8e8) at action.c:1049 ret = 1 v = 1 dst = {send_sock = 0x6, to = {s = {sa_family = 12576, sa_data = "M}X\177\000\000\254h\243\000\000\000\000"}, sin = {sin_family = 12576, sin_port = 32077, sin_addr = { s_addr = 32600}, sin_zero = "\254h\243\000\000\000\000"}, sin6 = {sin6_family = 12576, sin6_port = 32077, sin6_flowinfo = 32600, sin6_addr = {__in6_u = { __u6_addr8 = "\254h\243\000\000\000\000\000d\000\000\000\000\000\000", __u6_addr16 = {26796, 163, 0, 0, 100, 0, 0, 0}, __u6_addr32 = {10709164, 0, 100, 0}}}, sin6_scope_id = 10708691}}, id = 0, proto = -111 '\221', send_flags = {f = 72 'H', blst_imask = 62 '>'}} tmp = <optimized out> new_uri = <optimized out> end = <optimized out> crt = <optimized out> cmd = <optimized out> len = <optimized out> user = <optimized out> uri = {user = {s = 0x100000004 <Address 0x100000004 out of bounds>, len = 131072}, passwd = {s = 0x0, len = 0}, host = {s = 0x4 <Address 0x4 out of bounds>, len = 1548290768}, port = {s = 0x7f585c490ca2 "sip:+31263250863@31.223.168.2340\r\n\r\n60\020", len = 31}, params = {s = 0x100000004 <Address 0x100000004 out of bounds>, len = 1435660958}, sip_params = {s = 0x0, len = 1435673883}, headers = {s = 0x0, len = 25}, port_no = 3, proto = 0, type = 32765, flags = URI_USER_NORMALIZE, transport = {s = 0x7ffd6e278b70 "", len = 2097617544}, ttl = {s = 0x7f587d4be8e8 "\353\022", len = 12376}, user_param = {s = 0x7f585c47e774 "", len = 0}, maddr = { s = 0x7f587d4d4410 "\003", len = 1435538987}, method = {s = 0x0, len = 1848087808}, lr = {s = 0x0, len = 2102126824}, r2 = {s = 0x7ffd6e279500 "\003", len = 1848086480}, gr = {s = 0x7f587d4be8e8 "\353\022", len = 4420932}, transport_val = {s = 0x0, len = -1}, ttl_val = {s = 0x7ffd6e279500 "\003", len = 508396292}, user_param_val = { s = 0x7f587d4be8e8 "\353\022", len = 1848087808}, maddr_val = {s = 0x7ffd6e278fd0 "", len = 2102126824}, method_val = { s = 0xfe082e570a8d8304 <Address 0xfe082e570a8d8304 out of bounds>, len = 1346994948}, lr_val = {s = 0x0, len = 56}, r2_val = {s = 0x38 <Address 0x38 out of bounds>, len = 6344353}, gr_val = {s = 0x7f5800000000 <Address 0x7f5800000000 out of bounds>, len = 1509514912}} ---Type <return> to continue, or q <return> to quit--- next_hop = {user = {s = 0x7f5859f95cc0 "\020\002", len = 12396}, passwd = {s = 0x7f585c496fb8 "\002", len = 32860}, host = {s = 0x7f585c693640 "\220\232>\X\177", len = 1848086288}, port = {s = 0x2 <Address 0x2 out of bounds>, len = 1435721985}, params = {s = 0x8000000 <Address 0x8000000 out of bounds>, len = 129905640}, sip_params = {s = 0x3751d0 <Address 0x3751d0 out of bounds>, len = 4312088}, headers = {s = 0x4ac0a0 "", len = 1848086480}, port_no = 38144, proto = 28199, type = 32765, flags = 5485288, transport = {s = 0x1c8 <Address 0x1c8 out of bounds>, len = 1514031249}, ttl = {s = 0x8000000 <Address 0x8000000 out of bounds>, len = 1548316600}, user_param = {s = 0x7f587d4be8e8 "\353\022", len = 1548965320}, maddr = {s = 0x7f585c693640 "\220\232>\X\177", len = 1848086288}, method = { s = 0x2 <Address 0x2 out of bounds>, len = 1435607732}, lr = {s = 0x8000000 <Address 0x8000000 out of bounds>, len = 129906608}, r2 = { s = 0x374e08 <Address 0x374e08 out of bounds>, len = 4311120}, gr = {s = 0x4ac0a0 "", len = 8}, transport_val = {s = 0x1da <Address 0x1da out of bounds>, len = -1}, ttl_val = {s = 0x7f587d071c80 "\001", len = 2102126824}, user_param_val = {s = 0x7ffd6e279500 "\003", len = 1848086480}, maddr_val = {s = 0x7f587d4be8e8 "\353\022", len = 5514780}, method_val = {s = 0x7f587d4d2158 "\001", len = 449}, lr_val = {s = 0x7f585c5914c8 "\270\020J}X\177", len = 1506954664}, r2_val = { s = 0x1 <Address 0x1 out of bounds>, len = 2510}, gr_val = {s = 0x30 <Address 0x30 out of bounds>, len = 2102216912}} u = <optimized out> port = <optimized out> dst_host = <optimized out> i = <optimized out> flags = <optimized out> avp = <optimized out> st = {flags = 2102126824, id = 32600, name = {n = 4366602, s = {s = 0x42a10a "H\213\r\177\201V", len = 0}, re = 0x42a10a}, avp = 0x5aaa8a} sct = <optimized out> sjt = <optimized out> rve = <optimized out> mct = <optimized out> rv = <optimized out> rv1 = <optimized out> c1 = {cache_type = 134217728, val_type = RV_NONE, c = {avp_val = {n = 129906432, s = {s = 0x7be3700 <Address 0x7be3700 out of bounds>, len = 0}, re = 0x7be3700}, pval = { rs = {s = 0x7be3700 <Address 0x7be3700 out of bounds>, len = 0}, ri = 4311296, flags = 0}}, i2s = "\240\300J\000\000\000\000\000\b", '\000' <repeats 12 times>} s = {s = 0x0, len = 0} srevp = {0xfffffffb, 0x7ffd6e278b70} mod_f_params = {{type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = { number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, { type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}} __FUNCTION__ = "do_action" #14 0x000000000042a10a in run_actions (h=h@entry=0x7ffd6e279500, a=0x7f587d263f48, msg=msg@entry=0x7f587d4be8e8) at action.c:1549 t = <optimized out> ret = <optimized out> mod = <optimized out> ms = 0 __FUNCTION__ = "run_actions" #15 0x000000000042bde0 in do_action (h=h@entry=0x7ffd6e279500, a=a@entry=0x7f587d073d70, msg=msg@entry=0x7f587d4be8e8) at action.c:678 ret = -5 v = <optimized out> dst = {send_sock = 0x1, to = {s = {sa_family = 20288, sa_data = "M}X\177\000\000\200f\243\000\000\000\000"}, sin = {sin_family = 20288, sin_port = 32077, sin_addr = { s_addr = 32600}, sin_zero = "\200f\243\000\000\000\000"}, sin6 = {sin6_family = 20288, sin6_port = 32077, sin6_flowinfo = 32600, sin6_addr = {__in6_u = { __u6_addr8 = "\200f\243", '\000' <repeats 12 times>, __u6_addr16 = {26240, 163, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {10708608, 0, 0, 0}}}, sin6_scope_id = 2102217448}}, id = 32600, proto = -57 '\307', send_flags = {f = 102 'f', blst_imask = 163 '\243'}} tmp = <optimized out> new_uri = <optimized out> end = <optimized out> ---Type <return> to continue, or q <return> to quit--- crt = <optimized out> cmd = <optimized out> len = <optimized out> user = <optimized out> uri = {user = {s = 0x7f587d4d4628 "\004", len = 1509499248}, passwd = {s = 0x7f587d4aa8d8 "\020\325\332UX\177", len = 1506803058}, host = {s = 0x7f587d4be8e8 "\353\022", len = 2101683296}, port = {s = 0x7f587d452760 "", len = 1537539707}, params = {s = 0x7f587ce0e210 "", len = 1549665520}, sip_params = {s = 0x7f585c5e06f8 "", len = 1506810279}, headers = {s = 0x7f585c5e04f0 "\320m?\X\177", len = 404}, port_no = 0, proto = 0, type = ERROR_URI_T, flags = 1506664286, transport = { s = 0x7f585c195388 "\001", len = 1537548094}, ttl = {s = 0x7f587d4d54b8 "\220\230\266UX\177", len = 1506961773}, user_param = {s = 0x0, len = 2102219960}, maddr = { s = 0x0, len = -2027077120}, method = {s = 0x7f587d4be8e8 "\353\022", len = 1}, lr = {s = 0x7f585c693848 "", len = 2102126824}, r2 = { s = 0x780 <Address 0x780 out of bounds>, len = 1550399040}, gr = {s = 0x58554fe0 <Address 0x58554fe0 out of bounds>, len = 1506999306}, transport_val = {s = 0x0, len = 1468572138}, ttl_val = { s = 0x7f5885032978 "\227\201B\250|`\234|\374Qcq\zy\360\003\346`\033|\231/\310Km\301\243\353\372\266=/\222\210\353", <incomplete sequence \352>, len = 1542901968}, user_param_val = {s = 0x992290 "\314JO\X\177", len = 68}, maddr_val = {s = 0x993cb4 "\377\377", len = 0}, method_val = { s = 0x25dca00 "update `dialog` set `state`=4,`timeout`=1468582942,`caller_cseq`='3',`callee_cseq`='0',`caller_contact`='sip:dehaan01@176.74.254.179;alias=176.74.254.179~5060~1',`callee_contact`='sip:+31641157337@192"..., len = 2092914972}, lr_val = {s = 0x2 <Address 0x2 out of bounds>, len = 1514031249}, r2_val = { s = 0x8000000 <Address 0x8000000 out of bounds>, len = 129907920}, gr_val = {s = 0x3748e8 <Address 0x3748e8 out of bounds>, len = 4309808}} next_hop = {user = {s = 0x4ac0a0 "", len = 8}, passwd = {s = 0x1e4 <Address 0x1e4 out of bounds>, len = 64}, host = {s = 0x40 <Address 0x40 out of bounds>, len = 6344353}, port = {s = 0xa466c8 "", len = 10774544}, params = {s = 0x4ac0a0 "", len = 8}, sip_params = {s = 0x0, len = 48}, headers = {s = 0x30 <Address 0x30 out of bounds>, len = 6344353}, port_no = 0, proto = 0, type = 32600, flags = 1548156928, transport = {s = 0x40 <Address 0x40 out of bounds>, len = 2102218560}, ttl = { s = 0x30 <Address 0x30 out of bounds>, len = 2102214888}, user_param = {s = 0xe6a8df1faced0002 <Address 0xe6a8df1faced0002 out of bounds>, len = 2102206808}, maddr = { s = 0x7f587d4d2158 "\001", len = 2510}, method = {s = 0x30 <Address 0x30 out of bounds>, len = 10708679}, lr = {s = 0x6 <Address 0x6 out of bounds>, len = 2102217448}, r2 = {s = 0xa368a7 "", len = 100}, gr = { s = 0xa36690 "SIP/2.0/UDP 192.168.10.232:5060;branch=z9hG4bK7b052368;rport\r\nRoute: sip:192.168.10.230;r2=on;lr;ftag=as5d3a5a72;did=6ad.54d;nat=yes,sip:31.223.168.234;r2=on;lr;ftag=as5d3a5a72;did=6ad.54d;nat=yes"..., len = 6727432}, transport_val = {s = 0x8000000 <Address 0x8000000 out of bounds>, len = 129908048}, ttl_val = { s = 0x374868 <Address 0x374868 out of bounds>, len = 4309680}, user_param_val = {s = 0x4ac0a0 "", len = 8}, maddr_val = {s = 0x1e6 <Address 0x1e6 out of bounds>, len = 232}, method_val = {s = 0xe8 <Address 0xe8 out of bounds>, len = 6344353}, lr_val = {s = 0x5007f587d4d2158 <Address 0x5007f587d4d2158 out of bounds>, len = 6594876}, r2_val = {s = 0x7f5855b69bc0 "\001", len = 1}, gr_val = {s = 0x0, len = 1548202964}} u = <optimized out> port = <optimized out> dst_host = <optimized out> i = 21 flags = <optimized out> avp = <optimized out> st = {flags = 0, id = 0, name = {n = 4421082, s = {s = 0x4375da "D\211\340H\213\234", <incomplete sequence \350>, len = 1509585744}, re = 0x4375da}, avp = 0x7ffd6e279210} sct = <optimized out> sjt = <optimized out> rve = <optimized out> mct = <optimized out> rv = 0xa367d7 rv1 = <optimized out> c1 = {cache_type = RV_CACHE_EMPTY, val_type = RV_NONE, c = {avp_val = {n = 10708645, s = { s = 0xa366a5 "0.232:5060;branch=z9hG4bK7b052368;rport\r\nRoute: sip:192.168.10.230;r2=on;lr;ftag=as5d3a5a72;did=6ad.54d;nat=yes,sip:31.223.168.234;r2=on;lr;ftag=as5d3a5a72;did=6ad.54d;nat=yes\r\nMax-Forwards: 69\r\nF"..., len = 0}, re = 0xa366a5}, pval = {rs = { s = 0xa366a5 "0.232:5060;branch=z9hG4bK7b052368;rport\r\nRoute: sip:192.168.10.230;r2=on;lr;ftag=as5d3a5a72;did=6ad.54d;nat=yes,sip:31.223.168.234;r2=on;lr;ftag=as5d3a5a72;did=6ad.54d;nat=yes\r\nMax-Forwards: 69\r\nF"..., len = 0}, ri = 3, flags = 29}}, i2s = "\000\000\000\000X\177\000\000t\000\000\000X\177\000\000\000\000\000\000\000"} s = {s = 0x0, len = 2098257960} srevp = {0x7f587d4be8e8, 0x64} mod_f_params = {{type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = { ---Type <return> to continue, or q <return> to quit--- number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, { type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}} __FUNCTION__ = "do_action" #16 0x000000000042a10a in run_actions (h=h@entry=0x7ffd6e279500, a=a@entry=0x7f587d071698, msg=msg@entry=0x7f587d4be8e8) at action.c:1549 t = <optimized out> ret = <optimized out> mod = <optimized out> ms = 0 __FUNCTION__ = "run_actions" #17 0x00000000004375d0 in run_top_route (a=0x7f587d071698, msg=msg@entry=0x7f587d4be8e8, c=c@entry=0x0) at action.c:1635 ctx = {rec_lev = 3, run_flags = 0, last_retcode = 1, jmp_env = {{__jmpbuf = {140726451541248, 140440723803439876, 140018031466136, 140018035976424, 0, 0, -141812436097465596, 140441295872688900}, __mask_was_saved = 0, __saved_mask = {__val = {10708544, 615, 0, 0, 6452880, 140018036064736, 7138637, 3152, 140018035976424, 0, 140018036061392, 140726451541456, 140018035891032, 1073741825, 140017447846802, 140018035876760}}}}} p = 0x7ffd6e279500 ret = <optimized out> sfbk = 0 #18 0x0000000000504386 in receive_msg (buf=<optimized out>, len=<optimized out>, rcv_info=<optimized out>) at receive.c:240 msg = 0x7f587d4be8e8 ctx = {rec_lev = 134217728, run_flags = 0, last_retcode = 129916136, jmp_env = {{__jmpbuf = {3618528, 4301592, 140018165461920, 140018175721472, 446, 4294967295, 32, 5}, __mask_was_saved = 10031968, __saved_mask = {__val = {0, 1, 0, 0, 140018173532610, 140015933849605, 0, 140017482500740, 140018165461920, 128, 140018036068808, 140018035853848, 140018175719080, 140726451542134, 140018173557285, 140018175719080}}}}} ret = <optimized out> inb = { s = 0xa36640 "BYE sip:signetOLD@31.223.168.251:5060;alias=31.223.168.251~5060~1 SIP/2.0\r\nVia: SIP/2.0/UDP 192.168.10.232:5060;branch=z9hG4bK7b052368;rport\r\nRoute: <sip:192.168.10.230;r2=on;lr;ftag=as5d3a5a72;did=6a"..., len = 615} netinfo = {data = {s = 0x1 <Address 0x1 out of bounds>, len = 1848088624}, dst = 0x7f, rcv = 0x7f585a3e4891} __FUNCTION__ = "receive_msg" #19 0x00000000005f5bd4 in udp_rcv_loop () at udp_server.c:495 len = 615 buf = "BYE sip:signetOLD@31.223.168.251:5060;alias=31.223.168.251~5060~1 SIP/2.0\r\nVia: SIP/2.0/UDP 192.168.10.232:5060;branch=z9hG4bK7b052368;rport\r\nRoute: <sip:192.168.10.230;r2=on;lr;ftag=as5d3a5a72;did=6a"... from = 0x7f587d4d51c8 fromlen = 16 ri = {src_ip = {af = 2, len = 4, u = {addrl = {140019826862272, 5}, addr32 = {3893012672, 32600, 5, 0}, addr16 = {43200, 59402, 32600, 0, 5, 0, 0, 0}, addr = "\300\250\n\350X\177\000\000\005\000\000\000\000\000\000"}}, dst_ip = {af = 2, len = 4, u = {addrl = {3859458240, 0}, addr32 = {3859458240, 0, 0, 0}, addr16 = { 43200, 58890, 0, 0, 0, 0, 0, 0}, addr = "\300\250\n\346", '\000' <repeats 11 times>}}, src_port = 5060, dst_port = 5060, proto_reserved1 = 0, proto_reserved2 = 0, src_su = {s = {sa_family = 2, sa_data = "\023\304\300\250\n\350\000\000\000\000\000\000\000"}, sin = {sin_family = 2, sin_port = 50195, sin_addr = {s_addr = 3893012672}, sin_zero = "\000\000\000\000\000\000\000"}, sin6 = {sin6_family = 2, sin6_port = 50195, sin6_flowinfo = 3893012672, sin6_addr = {__in6_u = { __u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 0}}, bind_address = 0x7f587d4a0a18, proto = 1 '\001'} __FUNCTION__ = "udp_rcv_loop" #20 0x00000000004b2625 in main_loop () at main.c:1600 i = <optimized out> pid = <optimized out> si = <optimized out> si_desc = "udp receiver child=0 sock=192.168.10.230:5060", '\000' <repeats 11 times>, "\003", '\000' <repeats 15 times>, "\004", '\000' <repeats 16 times>, "B-\207:\a==\000\000\000\000\000\000\000\000\000B-\207:\a==\350\341\244\000\000\000\000\000\004\000\000\000\000\000\000" nrprocs = <optimized out> woneinit = <optimized out> ---Type <return> to continue, or q <return> to quit--- __FUNCTION__ = "main_loop" #21 0x0000000000427e2b in main (argc=<optimized out>, argv=<optimized out>) at main.c:2616 cfg_stream = <optimized out> c = <optimized out> r = <optimized out> tmp = 0x7ffd6e279f80 "" tmp_len = 0 port = 1 proto = 1 options = 0x6e8c70 ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:x:X:" ret = -1 seed = 1372741379 rfd = 4 debug_save = <optimized out> debug_flag = <optimized out> dont_fork_cnt = <optimized out> n_lst = 0x4 p = <optimized out> st = {st_dev = 14, st_ino = 10524, st_nlink = 2, st_mode = 16877, st_uid = 107, st_gid = 112, __pad0 = 0, st_rdev = 0, st_size = 100, st_blksize = 4096, st_blocks = 0, st_atim = {tv_sec = 1468547329, tv_nsec = 31440594}, st_mtim = {tv_sec = 1468566559, tv_nsec = 440763330}, st_ctim = {tv_sec = 1468566560, tv_nsec = 524700180}, __unused = {0, 0, 0}} __FUNCTION__ = "main" (gdb)

Cheers, Dirk

On 07/15/2016 12:40 PM, Daniel-Constantin Mierla wrote:

Hello,

can you send the output of gdb commands:

info locals

bt full

Cheers, Daniel

On 15/07/16 10:06, Dirk Teurlings - Signet B.V. wrote:

...

Hi,

Running Kamailio on Debian from the Kamailio repository with 4.4.2 stable (unpatched). Getting some random segfaults with it now, here's the relevant backtrace from the generated core.

Core was generated by `/usr/sbin/kamailio -f /etc/kamailio/kamailio.cfg -P /var/run/kamailio/kamailio.'. Program terminated with signal 11, Segmentation fault. #0 run_dlg_callbacks (type=type@entry=64, dlg=dlg@entry=0x7fceb400e2f0, req=req@entry=0x7fced4f093c8, rpl=rpl@entry=0x0, dir=<optimized out>, dlg_data=dlg_data@entry=0x0) at dlg_cb.c:253 253 if ( (cb->types)&type ) { (gdb) bt #0 run_dlg_callbacks (type=type@entry=64, dlg=dlg@entry=0x7fceb400e2f0, req=req@entry=0x7fced4f093c8, rpl=rpl@entry=0x0, dir=<optimized out>, dlg_data=dlg_data@entry=0x0) at dlg_cb.c:253 #1 0x00007fcead3648f9 in dlg_terminated (dir=<optimized out>, dlg=0x7fceb400e2f0, req=0x7fced4f093c8) at dlg_handlers.c:368 #2 dlg_onroute (req=0x7fced4f093c8, route_params=<optimized out>, param=<optimized out>) at dlg_handlers.c:1354 #3 0x00007fceb10ab0e2 in run_rr_callbacks (req=req@entry=0x7fced4f093c8, rr_param=rr_param@entry=0x7fceb12b77a0) at rr_cb.c:96 #4 0x00007fceb10952c5 in after_loose (_m=0x7fced4f093c8, preloaded=0) at loose.c:919 #5 0x000000000042b618 in do_action (h=h@entry=0x7ffeb0b3ed80, a=a@entry=0x7fced4cb4338, msg=msg@entry=0x7fced4f093c8) at action.c:1060 #6 0x000000000042a10a in run_actions (h=h@entry=0x7ffeb0b3ed80, a=0x7fced4cb4338, msg=0x7fced4f093c8) at action.c:1549 #7 0x0000000000437544 in run_actions_safe (h=h@entry=0x7ffeb0b402b0, a=<optimized out>, msg=<optimized out>) at action.c:1614 #8 0x000000000053b2e8 in rval_get_int (h=0x7ffeb0b402b0, msg=<optimized out>, i=0x7ffeb0b3f1e0, rv=rv@entry=0x7fced4cb4d58, cache=cache@entry=0x0) at rvalue.c:912 #9 0x000000000054261c in rval_expr_eval_int (h=h@entry=0x7ffeb0b402b0, msg=msg@entry=0x7fced4f093c8, res=res@entry=0x7ffeb0b3f1e0, rve=rve@entry=0x7fced4cb4d50) at rvalue.c:1910 #10 0x000000000042bc91 in do_action (h=h@entry=0x7ffeb0b402b0, a=a@entry=0x7fced4cb8f88, msg=msg@entry=0x7fced4f093c8) at action.c:1030 #11 0x000000000042a10a in run_actions (h=h@entry=0x7ffeb0b402b0, a=0x7fced4cb8f88, msg=msg@entry=0x7fced4f093c8) at action.c:1549 #12 0x000000000042bcf2 in do_action (h=h@entry=0x7ffeb0b402b0, a=a@entry=0x7fced4cb91e8, msg=msg@entry=0x7fced4f093c8) at action.c:1049 #13 0x000000000042a10a in run_actions (h=h@entry=0x7ffeb0b402b0, a=0x7fced4cb3f48, msg=msg@entry=0x7fced4f093c8) at action.c:1549 #14 0x000000000042bde0 in do_action (h=h@entry=0x7ffeb0b402b0, a=a@entry=0x7fced4ac3d70, msg=msg@entry=0x7fced4f093c8) at action.c:678 #15 0x000000000042a10a in run_actions (h=h@entry=0x7ffeb0b402b0, a=a@entry=0x7fced4ac1698, msg=msg@entry=0x7fced4f093c8) at action.c:1549 #16 0x00000000004375d0 in run_top_route (a=0x7fced4ac1698, msg=msg@entry=0x7fced4f093c8, c=c@entry=0x0) at action.c:1635 #17 0x0000000000504386 in receive_msg (buf=<optimized out>, len=<optimized out>, rcv_info=<optimized out>) at receive.c:240 #18 0x00000000005f5bd4 in udp_rcv_loop () at udp_server.c:495 #19 0x00000000004b2625 in main_loop () at main.c:1600 #20 0x0000000000427e2b in main (argc=<optimized out>, argv=<optimized out>) at main.c:2616

And from syslog the relevant messages before this dump:

Jul 15 08:55:03 server /usr/sbin/kamailio[16470]: WARNING: dialog [dlg_handlers.c:1219]: dlg_onroute(): unable to find dialog for BYE with route param 'd4c.26d1' [3149:7522] Jul 15 08:56:01 server /usr/sbin/kamailio[16481]: WARNING: dialog [dlg_handlers.c:1219]: dlg_onroute(): unable to find dialog for BYE with route param 'fc.99f1' [207:8089] Jul 15 08:56:27 server /usr/sbin/kamailio[16470]: CRITICAL: dialog [dlg_timer.c:200]: update_dlg_timer(): Trying to update a bogus dlg tl=0x7fceb3f7d920 tl->next=(nil) tl->prev=(nil) Jul 15 08:56:27 server /usr/sbin/kamailio[16470]: ERROR: dialog [dlg_handlers.c:1377]: dlg_onroute(): failed to update dialog lifetime Jul 15 08:57:01 server /usr/sbin/kamailio[16482]: ERROR: db_mysql [km_dbase.c:128]: db_mysql_submit_query(): driver error on query: Duplicate entry '9584-3854-435' for key 'hash_index' (1062) Jul 15 08:57:01 server /usr/sbin/kamailio[16482]: ERROR: <core> [db_query.c:181]: db_do_raw_query(): error while submitting query Jul 15 08:57:01 server /usr/sbin/kamailio[16482]: ERROR: sqlops [sql_api.c:265]: sql_do_query(): cannot do the query [INSERT INTO `dialog_extra` (`h_i] Jul 15 08:57:01 server /usr/sbin/kamailio[16482]: ERROR: auth [api.c:119]: auth_check_hdr_md5(): auth:pre_auth: Credentials are not filled properly Jul 15 08:57:01 server /usr/sbin/kamailio[16483]: ERROR: auth [api.c:119]: auth_check_hdr_md5(): auth:pre_auth: Credentials are not filled properly Jul 15 08:57:54 server /usr/sbin/kamailio[16506]: NOTICE: dialog [dlg_hash.c:245]: dlg_clean_run(): dialog in delete state is too old (0x7fceb3f64470 ref 4) Jul 15 08:57:54 server /usr/sbin/kamailio[16473]: WARNING: dialog [dlg_handlers.c:1348]: dlg_onroute(): inconsitent dlg timer data on dlg 0x7fceb3f64470 [1182:5803] with clid '09ad128753e2535d24bde58e3d7eda04@192.168.10.232:5060' and tags 'as1b497b34' '5788890C-EC6F55F-3E86ED0C' Jul 15 08:57:54 server /usr/sbin/kamailio[16469]: ERROR: dialog [dlg_handlers.c:334]: dlg_terminated_confirmed(): failed to get dialog from params! Jul 15 08:58:49 server /usr/sbin/kamailio[16467]: WARNING: dialog [dlg_handlers.c:1219]: dlg_onroute(): unable to find dialog for BYE with route param '6d2.2581' [726:6226] Jul 15 08:59:24 server /usr/sbin/kamailio[16506]: NOTICE: dialog [dlg_hash.c:245]: dlg_clean_run(): dialog in delete state is too old (0x7fceb400e2f0 ref 4) Jul 15 08:59:25 server /usr/sbin/kamailio[16464]: WARNING: dialog [dlg_handlers.c:1219]: dlg_onroute(): unable to find dialog for BYE with route param '3e4.b5c1' [1251:7259] Jul 15 08:59:25 server /usr/sbin/kamailio[16465]: WARNING: dialog [dlg_handlers.c:1348]: dlg_onroute(): inconsitent dlg timer data on dlg 0x7fceb400e2f0 [1251:7259] with clid '87791a#015#012Call-ID: 25750e286a5654361ef9405d72edbc' and tags '' 'as148f41b1' Jul 15 08:59:25 server kernel: [203670.830521] kamailio[16465] general protection ip:7fcead34b3a5 sp:7ffeb0b3e220 error:0 in dialog.so[7fcead330000+88000] Jul 15 08:59:26 server /usr/sbin/kamailio[16511]: CRITICAL: <core> [pass_fd.c:275]: receive_fd(): EOF on 33 Jul 15 08:59:26 server /usr/sbin/kamailio[16458]: ALERT: <core> [main.c:739]: handle_sigs(): child process 16465 exited by a signal 11 Jul 15 08:59:26 server /usr/sbin/kamailio[16458]: ALERT: <core> [main.c:742]: handle_sigs(): core was generated Jul 15 08:59:26 server /usr/sbin/kamailio[16458]: INFO: <core> [main.c:754]: handle_sigs(): terminating due to SIGCHLD

Any insight would be appreciated!

Cheers, Dirk

---

SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users