15 Jul
2016
15 Jul
'16
10:06 a.m.
Hi,
Running Kamailio on Debian from the Kamailio repository with 4.4.2
stable (unpatched). Getting some random segfaults with it now, here's
the relevant backtrace from the generated core.
Core was generated by `/usr/sbin/kamailio -f /etc/kamailio/kamailio.cfg
-P /var/run/kamailio/kamailio.'.
Program terminated with signal 11, Segmentation fault.
#0 run_dlg_callbacks (type=type@entry=64, dlg=dlg@entry=0x7fceb400e2f0,
req=req@entry=0x7fced4f093c8, rpl=rpl@entry=0x0, dir=<optimized out>,
dlg_data=dlg_data@entry=0x0) at dlg_cb.c:253
253 if ( (cb->types)&type ) {
(gdb) bt
#0 run_dlg_callbacks (type=type@entry=64, dlg=dlg@entry=0x7fceb400e2f0,
req=req@entry=0x7fced4f093c8, rpl=rpl@entry=0x0, dir=<optimized out>,
dlg_data=dlg_data@entry=0x0) at dlg_cb.c:253
#1 0x00007fcead3648f9 in dlg_terminated (dir=<optimized out>,
dlg=0x7fceb400e2f0, req=0x7fced4f093c8) at dlg_handlers.c:368
#2 dlg_onroute (req=0x7fced4f093c8, route_params=<optimized out>,
param=<optimized out>) at dlg_handlers.c:1354
#3 0x00007fceb10ab0e2 in run_rr_callbacks
(req=req@entry=0x7fced4f093c8, rr_param=rr_param@entry=0x7fceb12b77a0)
at rr_cb.c:96
#4 0x00007fceb10952c5 in after_loose (_m=0x7fced4f093c8, preloaded=0)
at loose.c:919
#5 0x000000000042b618 in do_action (h=h@entry=0x7ffeb0b3ed80,
a=a@entry=0x7fced4cb4338, msg=msg@entry=0x7fced4f093c8) at action.c:1060
#6 0x000000000042a10a in run_actions (h=h@entry=0x7ffeb0b3ed80,
a=0x7fced4cb4338, msg=0x7fced4f093c8) at action.c:1549
#7 0x0000000000437544 in run_actions_safe (h=h@entry=0x7ffeb0b402b0,
a=<optimized out>, msg=<optimized out>) at action.c:1614
#8 0x000000000053b2e8 in rval_get_int (h=0x7ffeb0b402b0, msg=<optimized
out>, i=0x7ffeb0b3f1e0, rv=rv@entry=0x7fced4cb4d58,
cache=cache@entry=0x0) at rvalue.c:912
#9 0x000000000054261c in rval_expr_eval_int (h=h@entry=0x7ffeb0b402b0,
msg=msg@entry=0x7fced4f093c8, res=res@entry=0x7ffeb0b3f1e0,
rve=rve@entry=0x7fced4cb4d50) at rvalue.c:1910
#10 0x000000000042bc91 in do_action (h=h@entry=0x7ffeb0b402b0,
a=a@entry=0x7fced4cb8f88, msg=msg@entry=0x7fced4f093c8) at action.c:1030
#11 0x000000000042a10a in run_actions (h=h@entry=0x7ffeb0b402b0,
a=0x7fced4cb8f88, msg=msg@entry=0x7fced4f093c8) at action.c:1549
#12 0x000000000042bcf2 in do_action (h=h@entry=0x7ffeb0b402b0,
a=a@entry=0x7fced4cb91e8, msg=msg@entry=0x7fced4f093c8) at action.c:1049
#13 0x000000000042a10a in run_actions (h=h@entry=0x7ffeb0b402b0,
a=0x7fced4cb3f48, msg=msg@entry=0x7fced4f093c8) at action.c:1549
#14 0x000000000042bde0 in do_action (h=h@entry=0x7ffeb0b402b0,
a=a@entry=0x7fced4ac3d70, msg=msg@entry=0x7fced4f093c8) at action.c:678
#15 0x000000000042a10a in run_actions (h=h@entry=0x7ffeb0b402b0,
a=a@entry=0x7fced4ac1698, msg=msg@entry=0x7fced4f093c8) at action.c:1549
#16 0x00000000004375d0 in run_top_route (a=0x7fced4ac1698,
msg=msg@entry=0x7fced4f093c8, c=c@entry=0x0) at action.c:1635
#17 0x0000000000504386 in receive_msg (buf=<optimized out>,
len=<optimized out>, rcv_info=<optimized out>) at receive.c:240
#18 0x00000000005f5bd4 in udp_rcv_loop () at udp_server.c:495
#19 0x00000000004b2625 in main_loop () at main.c:1600
#20 0x0000000000427e2b in main (argc=<optimized out>, argv=<optimized
out>) at main.c:2616
And from syslog the relevant messages before this dump:
Jul 15 08:55:03 server /usr/sbin/kamailio[16470]: WARNING: dialog
[dlg_handlers.c:1219]: dlg_onroute(): unable to find dialog for BYE with
route param 'd4c.26d1' [3149:7522]
Jul 15 08:56:01 server /usr/sbin/kamailio[16481]: WARNING: dialog
[dlg_handlers.c:1219]: dlg_onroute(): unable to find dialog for BYE with
route param 'fc.99f1' [207:8089]
Jul 15 08:56:27 server /usr/sbin/kamailio[16470]: CRITICAL: dialog
[dlg_timer.c:200]: update_dlg_timer(): Trying to update a bogus dlg
tl=0x7fceb3f7d920 tl->next=(nil) tl->prev=(nil)
Jul 15 08:56:27 server /usr/sbin/kamailio[16470]: ERROR: dialog
[dlg_handlers.c:1377]: dlg_onroute(): failed to update dialog lifetime
Jul 15 08:57:01 server /usr/sbin/kamailio[16482]: ERROR: db_mysql
[km_dbase.c:128]: db_mysql_submit_query(): driver error on query:
Duplicate entry '9584-3854-435' for key 'hash_index' (1062)
Jul 15 08:57:01 server /usr/sbin/kamailio[16482]: ERROR: <core>
[db_query.c:181]: db_do_raw_query(): error while submitting query
Jul 15 08:57:01 server /usr/sbin/kamailio[16482]: ERROR: sqlops
[sql_api.c:265]: sql_do_query(): cannot do the query [INSERT INTO
`dialog_extra` (`h_i]
Jul 15 08:57:01 server /usr/sbin/kamailio[16482]: ERROR: auth
[api.c:119]: auth_check_hdr_md5(): auth:pre_auth: Credentials are not
filled properly
Jul 15 08:57:01 server /usr/sbin/kamailio[16483]: ERROR: auth
[api.c:119]: auth_check_hdr_md5(): auth:pre_auth: Credentials are not
filled properly
Jul 15 08:57:54 server /usr/sbin/kamailio[16506]: NOTICE: dialog
[dlg_hash.c:245]: dlg_clean_run(): dialog in delete state is too old
(0x7fceb3f64470 ref 4)
Jul 15 08:57:54 server /usr/sbin/kamailio[16473]: WARNING: dialog
[dlg_handlers.c:1348]: dlg_onroute(): inconsitent dlg timer data on dlg
0x7fceb3f64470 [1182:5803] with clid
'09ad128753e2535d24bde58e3d7eda04@192.168.10.232:5060' and tags
'as1b497b34' '5788890C-EC6F55F-3E86ED0C'
Jul 15 08:57:54 server /usr/sbin/kamailio[16469]: ERROR: dialog
[dlg_handlers.c:334]: dlg_terminated_confirmed(): failed to get dialog
from params!
Jul 15 08:58:49 server /usr/sbin/kamailio[16467]: WARNING: dialog
[dlg_handlers.c:1219]: dlg_onroute(): unable to find dialog for BYE with
route param '6d2.2581' [726:6226]
Jul 15 08:59:24 server /usr/sbin/kamailio[16506]: NOTICE: dialog
[dlg_hash.c:245]: dlg_clean_run(): dialog in delete state is too old
(0x7fceb400e2f0 ref 4)
Jul 15 08:59:25 server /usr/sbin/kamailio[16464]: WARNING: dialog
[dlg_handlers.c:1219]: dlg_onroute(): unable to find dialog for BYE with
route param '3e4.b5c1' [1251:7259]
Jul 15 08:59:25 server /usr/sbin/kamailio[16465]: WARNING: dialog
[dlg_handlers.c:1348]: dlg_onroute(): inconsitent dlg timer data on dlg
0x7fceb400e2f0 [1251:7259] with clid '87791a#015#012Call-ID:
25750e286a5654361ef9405d72edbc' and tags '' 'as148f41b1'
Jul 15 08:59:25 server kernel: [203670.830521] kamailio[16465] general
protection ip:7fcead34b3a5 sp:7ffeb0b3e220 error:0 in
dialog.so[7fcead330000+88000]
Jul 15 08:59:26 server /usr/sbin/kamailio[16511]: CRITICAL: <core>
[pass_fd.c:275]: receive_fd(): EOF on 33
Jul 15 08:59:26 server /usr/sbin/kamailio[16458]: ALERT: <core>
[main.c:739]: handle_sigs(): child process 16465 exited by a signal 11
Jul 15 08:59:26 server /usr/sbin/kamailio[16458]: ALERT: <core>
[main.c:742]: handle_sigs(): core was generated
Jul 15 08:59:26 server /usr/sbin/kamailio[16458]: INFO: <core>
[main.c:754]: handle_sigs(): terminating due to SIGCHLD
Any insight would be appreciated!
Cheers,
Dirk
15 Jul
15 Jul
10:51 a.m.
Just got another segfault.
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/sbin/kamailio -f /etc/kamailio/kamailio.cfg
-P /var/run/kamailio/kamailio.'.
Program terminated with signal 11, Segmentation fault.
#0 atomic_get (v=0x7f6264d11378) at ../../mem/../atomic/atomic_common.h:74
74 return atomic_get_int(&(v->val));
(gdb) bt
#0 atomic_get (v=0x7f6264d11378) at ../../mem/../atomic/atomic_common.h:74
#1 dlg_unref (dlg=dlg@entry=0x7f585c494b40, cnt=cnt@entry=1) at
dlg_hash.c:921
#2 0x00007f5855912802 in dlg_run_event_route
(dlg=dlg@entry=0x7f585c494b40, msg=msg@entry=0x7f587d4be8e8,
ostate=<optimized out>, nstate=<optimized out>) at dlg_handlers.c:1630
#3 0x00007f585591416a in dlg_onroute (req=0x7f587d4be8e8,
route_params=<optimized out>, param=<optimized out>) at dlg_handlers.c:1307
#4 0x00007f585965b0e2 in run_rr_callbacks
(req=req@entry=0x7f587d4be8e8, rr_param=rr_param@entry=0x7f58598677a0)
at rr_cb.c:96
#5 0x00007f58596452c5 in after_loose (_m=0x7f587d4be8e8, preloaded=0)
at loose.c:919
#6 0x000000000042b618 in do_action (h=h@entry=0x7ffd6e277fd0,
a=a@entry=0x7f587d264338, msg=msg@entry=0x7f587d4be8e8) at action.c:1060
#7 0x000000000042a10a in run_actions (h=h@entry=0x7ffd6e277fd0,
a=0x7f587d264338, msg=0x7f587d4be8e8) at action.c:1549
#8 0x0000000000437544 in run_actions_safe (h=h@entry=0x7ffd6e279500,
a=<optimized out>, msg=<optimized out>) at action.c:1614
#9 0x000000000053b2e8 in rval_get_int (h=0x7ffd6e279500, msg=<optimized
out>, i=0x7ffd6e278430, rv=rv@entry=0x7f587d264d58,
cache=cache@entry=0x0) at rvalue.c:912
#10 0x000000000054261c in rval_expr_eval_int (h=h@entry=0x7ffd6e279500,
msg=msg@entry=0x7f587d4be8e8, res=res@entry=0x7ffd6e278430,
rve=rve@entry=0x7f587d264d50) at rvalue.c:1910
#11 0x000000000042bc91 in do_action (h=h@entry=0x7ffd6e279500,
a=a@entry=0x7f587d268f88, msg=msg@entry=0x7f587d4be8e8) at action.c:1030
#12 0x000000000042a10a in run_actions (h=h@entry=0x7ffd6e279500,
a=0x7f587d268f88, msg=msg@entry=0x7f587d4be8e8) at action.c:1549
#13 0x000000000042bcf2 in do_action (h=h@entry=0x7ffd6e279500,
a=a@entry=0x7f587d2691e8, msg=msg@entry=0x7f587d4be8e8) at action.c:1049
#14 0x000000000042a10a in run_actions (h=h@entry=0x7ffd6e279500,
a=0x7f587d263f48, msg=msg@entry=0x7f587d4be8e8) at action.c:1549
#15 0x000000000042bde0 in do_action (h=h@entry=0x7ffd6e279500,
a=a@entry=0x7f587d073d70, msg=msg@entry=0x7f587d4be8e8) at action.c:678
#16 0x000000000042a10a in run_actions (h=h@entry=0x7ffd6e279500,
a=a@entry=0x7f587d071698, msg=msg@entry=0x7f587d4be8e8) at action.c:1549
#17 0x00000000004375d0 in run_top_route (a=0x7f587d071698,
msg=msg@entry=0x7f587d4be8e8, c=c@entry=0x0) at action.c:1635
#18 0x0000000000504386 in receive_msg (buf=<optimized out>,
len=<optimized out>, rcv_info=<optimized out>) at receive.c:240
#19 0x00000000005f5bd4 in udp_rcv_loop () at udp_server.c:495
#20 0x00000000004b2625 in main_loop () at main.c:1600
#21 0x0000000000427e2b in main (argc=<optimized out>, argv=<optimized
out>) at main.c:2616
Relevant logmessages before crash:
Jul 15 10:37:55 server /usr/sbin/kamailio[12426]: NOTICE: dialog
[dlg_hash.c:245]: dlg_clean_run(): dialog in delete state is too old
(0x7f585c4a6820 ref 4)
Jul 15 10:37:55 server /usr/sbin/kamailio[12397]: WARNING: dialog
[dlg_handlers.c:1219]: dlg_onroute(): unable to find dialog for BYE with
route param '70f.b9d1' [3847:7579]
Jul 15 10:37:55 server /usr/sbin/kamailio[12395]: WARNING: dialog
[dlg_handlers.c:1348]: dlg_onroute(): inconsitent dlg timer data on dlg
0x7f585c4a6820 [3847:7579] with clid
'4c41f08d317ecb9342b93f22738003f3@server' and tags 'as5f3a16b4'
'as71cb6036'
Jul 15 10:40:13 server /usr/sbin/kamailio[12378]: WARNING: dialog
[dlg_handlers.c:1219]: dlg_onroute(): unable to find dialog for BYE with
route param 'eb6.1e21' [1726:4833]
Jul 15 10:40:13 server /usr/sbin/kamailio[12376]: WARNING: dialog
[dlg_handlers.c:1219]: dlg_onroute(): unable to find dialog for BYE with
route param 'eb6.1e21' [1726:4833]
Jul 15 10:40:14 server /usr/sbin/kamailio[12377]: WARNING: dialog
[dlg_handlers.c:1219]: dlg_onroute(): unable to find dialog for BYE with
route param 'eb6.1e21' [1726:4833]
Jul 15 10:40:16 server /usr/sbin/kamailio[12377]: WARNING: dialog
[dlg_handlers.c:1219]: dlg_onroute(): unable to find dialog for BYE with
route param 'eb6.1e21' [1726:4833]
Jul 15 10:40:16 server /usr/sbin/kamailio[12396]: WARNING: dialog
[dlg_handlers.c:1219]: dlg_onroute(): unable to find dialog for BYE with
route param 'eb6.1e21' [1726:4833]
Jul 15 10:41:34 server /usr/sbin/kamailio[12396]: ERROR: sl
[sl_funcs.c:363]: sl_reply_error(): ERROR: sl_reply_error used: I'm
terribly sorry, server error occurred (1/SL)
Jul 15 10:41:34 server /usr/sbin/kamailio[12396]: ERROR: tm
[t_reply.c:533]: _reply_light(): ERROR: _reply_light: can't generate 487
reply when a final 487 was sent out
Jul 15 10:41:34 server /usr/sbin/kamailio[12396]: ERROR: tm
[t_lookup.c:1471]: t_unref(): ERROR: t_unref: generation of a delayed
stateful reply failed
Jul 15 10:42:25 server /usr/sbin/kamailio[12426]: NOTICE: dialog
[dlg_hash.c:245]: dlg_clean_run(): dialog in delete state is too old
(0x7f585c49d5b0 ref 4)
Jul 15 10:42:25 server /usr/sbin/kamailio[12426]: NOTICE: dialog
[dlg_hash.c:245]: dlg_clean_run(): dialog in delete state is too old
(0x7f585c604f18 ref 4)
Jul 15 10:42:25 server /usr/sbin/kamailio[12426]: NOTICE: dialog
[dlg_hash.c:245]: dlg_clean_run(): dialog in delete state is too old
(0x7f585c494b40 ref 4)
Jul 15 10:42:25 server /usr/sbin/kamailio[12383]: WARNING: dialog
[dlg_handlers.c:1348]: dlg_onroute(): inconsitent dlg timer data on dlg
0x7f585c604f18 [2396:9046] with clid
'1b3ff5f0246fb7e82ed949544bcccbba@192.168.10.233:5060' and tags
'as4d83d6f8' '5788A162-2557E04D-3E86ED15'
Jul 15 10:42:25 server /usr/sbin/kamailio[12395]: WARNING: dialog
[dlg_handlers.c:1219]: dlg_onroute(): unable to find dialog for BYE with
route param '6b3.c6b' [950:2924]
Jul 15 10:42:25 server kernel: [209851.262461] kamailio[12376]: segfault
at 7f6264d11378 ip 00007f585592a908 sp 00007ffd6e277330 error 4 in
dialog.so[7f58558e0000+88000]
Jul 15 10:42:25 server /usr/sbin/kamailio[12394]: WARNING: dialog
[dlg_handlers.c:1348]: dlg_onroute(): inconsitent dlg timer data on dlg
0x7f585c49d5b0 [950:2924] with clid
'45fe86ce065f5543342e51ad355d1b75@server' and tags 'as152f7465'
'as4d03f77d'
Jul 15 10:42:26 server /usr/sbin/kamailio[12431]: CRITICAL: <core>
[pass_fd.c:275]: receive_fd(): EOF on 32
Jul 15 10:42:26 server /usr/sbin/kamailio[12370]: ALERT: <core>
[main.c:739]: handle_sigs(): child process 12376 exited by a signal 11
Jul 15 10:42:26 server /usr/sbin/kamailio[12370]: ALERT: <core>
[main.c:742]: handle_sigs(): core was generated
Jul 15 10:42:26 server /usr/sbin/kamailio[12370]: INFO: <core>
[main.c:754]: handle_sigs(): terminating due to SIGCHLD
Cheers,
Dirk
On 07/15/2016 10:06 AM, Dirk Teurlings - Signet B.V. wrote:
Hi,
Running Kamailio on Debian from the Kamailio repository with 4.4.2
stable (unpatched). Getting some random segfaults with it now, here's
the relevant backtrace from the generated core.
Core was generated by `/usr/sbin/kamailio -f /etc/kamailio/kamailio.cfg
-P /var/run/kamailio/kamailio.'.
Program terminated with signal 11, Segmentation fault.
#0 run_dlg_callbacks (type=type@entry=64, dlg=dlg@entry=0x7fceb400e2f0,
req=req@entry=0x7fced4f093c8, rpl=rpl@entry=0x0, dir=<optimized out>,
dlg_data=dlg_data@entry=0x0) at dlg_cb.c:253
253 if ( (cb->types)&type ) {
(gdb) bt
#0 run_dlg_callbacks (type=type@entry=64, dlg=dlg@entry=0x7fceb400e2f0,
req=req@entry=0x7fced4f093c8, rpl=rpl@entry=0x0, dir=<optimized out>,
dlg_data=dlg_data@entry=0x0) at dlg_cb.c:253
#1 0x00007fcead3648f9 in dlg_terminated (dir=<optimized out>,
dlg=0x7fceb400e2f0, req=0x7fced4f093c8) at dlg_handlers.c:368
#2 dlg_onroute (req=0x7fced4f093c8, route_params=<optimized out>,
param=<optimized out>) at dlg_handlers.c:1354
#3 0x00007fceb10ab0e2 in run_rr_callbacks
(req=req@entry=0x7fced4f093c8, rr_param=rr_param@entry=0x7fceb12b77a0)
at rr_cb.c:96
#4 0x00007fceb10952c5 in after_loose (_m=0x7fced4f093c8, preloaded=0)
at loose.c:919
#5 0x000000000042b618 in do_action (h=h@entry=0x7ffeb0b3ed80,
a=a@entry=0x7fced4cb4338, msg=msg@entry=0x7fced4f093c8) at action.c:1060
#6 0x000000000042a10a in run_actions (h=h@entry=0x7ffeb0b3ed80,
a=0x7fced4cb4338, msg=0x7fced4f093c8) at action.c:1549
#7 0x0000000000437544 in run_actions_safe (h=h@entry=0x7ffeb0b402b0,
a=<optimized out>, msg=<optimized out>) at action.c:1614
#8 0x000000000053b2e8 in rval_get_int (h=0x7ffeb0b402b0, msg=<optimized
out>, i=0x7ffeb0b3f1e0, rv=rv@entry=0x7fced4cb4d58,
cache=cache@entry=0x0) at rvalue.c:912
#9 0x000000000054261c in rval_expr_eval_int (h=h@entry=0x7ffeb0b402b0,
msg=msg@entry=0x7fced4f093c8, res=res@entry=0x7ffeb0b3f1e0,
rve=rve@entry=0x7fced4cb4d50) at rvalue.c:1910
#10 0x000000000042bc91 in do_action (h=h@entry=0x7ffeb0b402b0,
a=a@entry=0x7fced4cb8f88, msg=msg@entry=0x7fced4f093c8) at action.c:1030
#11 0x000000000042a10a in run_actions (h=h@entry=0x7ffeb0b402b0,
a=0x7fced4cb8f88, msg=msg@entry=0x7fced4f093c8) at action.c:1549
#12 0x000000000042bcf2 in do_action (h=h@entry=0x7ffeb0b402b0,
a=a@entry=0x7fced4cb91e8, msg=msg@entry=0x7fced4f093c8) at action.c:1049
#13 0x000000000042a10a in run_actions (h=h@entry=0x7ffeb0b402b0,
a=0x7fced4cb3f48, msg=msg@entry=0x7fced4f093c8) at action.c:1549
#14 0x000000000042bde0 in do_action (h=h@entry=0x7ffeb0b402b0,
a=a@entry=0x7fced4ac3d70, msg=msg@entry=0x7fced4f093c8) at action.c:678
#15 0x000000000042a10a in run_actions (h=h@entry=0x7ffeb0b402b0,
a=a@entry=0x7fced4ac1698, msg=msg@entry=0x7fced4f093c8) at action.c:1549
#16 0x00000000004375d0 in run_top_route (a=0x7fced4ac1698,
msg=msg@entry=0x7fced4f093c8, c=c@entry=0x0) at action.c:1635
#17 0x0000000000504386 in receive_msg (buf=<optimized out>,
len=<optimized out>, rcv_info=<optimized out>) at receive.c:240
#18 0x00000000005f5bd4 in udp_rcv_loop () at udp_server.c:495
#19 0x00000000004b2625 in main_loop () at main.c:1600
#20 0x0000000000427e2b in main (argc=<optimized out>, argv=<optimized
out>) at main.c:2616
And from syslog the relevant messages before this dump:
Jul 15 08:55:03 server /usr/sbin/kamailio[16470]: WARNING: dialog
[dlg_handlers.c:1219]: dlg_onroute(): unable to find dialog for BYE with
route param 'd4c.26d1' [3149:7522]
Jul 15 08:56:01 server /usr/sbin/kamailio[16481]: WARNING: dialog
[dlg_handlers.c:1219]: dlg_onroute(): unable to find dialog for BYE with
route param 'fc.99f1' [207:8089]
Jul 15 08:56:27 server /usr/sbin/kamailio[16470]: CRITICAL: dialog
[dlg_timer.c:200]: update_dlg_timer(): Trying to update a bogus dlg
tl=0x7fceb3f7d920 tl->next=(nil) tl->prev=(nil)
Jul 15 08:56:27 server /usr/sbin/kamailio[16470]: ERROR: dialog
[dlg_handlers.c:1377]: dlg_onroute(): failed to update dialog lifetime
Jul 15 08:57:01 server /usr/sbin/kamailio[16482]: ERROR: db_mysql
[km_dbase.c:128]: db_mysql_submit_query(): driver error on query:
Duplicate entry '9584-3854-435' for key 'hash_index' (1062)
Jul 15 08:57:01 server /usr/sbin/kamailio[16482]: ERROR: <core>
[db_query.c:181]: db_do_raw_query(): error while submitting query
Jul 15 08:57:01 server /usr/sbin/kamailio[16482]: ERROR: sqlops
[sql_api.c:265]: sql_do_query(): cannot do the query [INSERT INTO
`dialog_extra` (`h_i]
Jul 15 08:57:01 server /usr/sbin/kamailio[16482]: ERROR: auth
[api.c:119]: auth_check_hdr_md5(): auth:pre_auth: Credentials are not
filled properly
Jul 15 08:57:01 server /usr/sbin/kamailio[16483]: ERROR: auth
[api.c:119]: auth_check_hdr_md5(): auth:pre_auth: Credentials are not
filled properly
Jul 15 08:57:54 server /usr/sbin/kamailio[16506]: NOTICE: dialog
[dlg_hash.c:245]: dlg_clean_run(): dialog in delete state is too old
(0x7fceb3f64470 ref 4)
Jul 15 08:57:54 server /usr/sbin/kamailio[16473]: WARNING: dialog
[dlg_handlers.c:1348]: dlg_onroute(): inconsitent dlg timer data on dlg
0x7fceb3f64470 [1182:5803] with clid
'09ad128753e2535d24bde58e3d7eda04@192.168.10.232:5060' and tags
'as1b497b34' '5788890C-EC6F55F-3E86ED0C'
Jul 15 08:57:54 server /usr/sbin/kamailio[16469]: ERROR: dialog
[dlg_handlers.c:334]: dlg_terminated_confirmed(): failed to get dialog
from params!
Jul 15 08:58:49 server /usr/sbin/kamailio[16467]: WARNING: dialog
[dlg_handlers.c:1219]: dlg_onroute(): unable to find dialog for BYE with
route param '6d2.2581' [726:6226]
Jul 15 08:59:24 server /usr/sbin/kamailio[16506]: NOTICE: dialog
[dlg_hash.c:245]: dlg_clean_run(): dialog in delete state is too old
(0x7fceb400e2f0 ref 4)
Jul 15 08:59:25 server /usr/sbin/kamailio[16464]: WARNING: dialog
[dlg_handlers.c:1219]: dlg_onroute(): unable to find dialog for BYE with
route param '3e4.b5c1' [1251:7259]
Jul 15 08:59:25 server /usr/sbin/kamailio[16465]: WARNING: dialog
[dlg_handlers.c:1348]: dlg_onroute(): inconsitent dlg timer data on dlg
0x7fceb400e2f0 [1251:7259] with clid '87791a#015#012Call-ID:
25750e286a5654361ef9405d72edbc' and tags '' 'as148f41b1'
Jul 15 08:59:25 server kernel: [203670.830521] kamailio[16465] general
protection ip:7fcead34b3a5 sp:7ffeb0b3e220 error:0 in
dialog.so[7fcead330000+88000]
Jul 15 08:59:26 server /usr/sbin/kamailio[16511]: CRITICAL: <core>
[pass_fd.c:275]: receive_fd(): EOF on 33
Jul 15 08:59:26 server /usr/sbin/kamailio[16458]: ALERT: <core>
[main.c:739]: handle_sigs(): child process 16465 exited by a signal 11
Jul 15 08:59:26 server /usr/sbin/kamailio[16458]: ALERT: <core>
[main.c:742]: handle_sigs(): core was generated
Jul 15 08:59:26 server /usr/sbin/kamailio[16458]: INFO: <core>
[main.c:754]: handle_sigs(): terminating due to SIGCHLD
Any insight would be appreciated!
Cheers,
Dirk
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users(a)lists.sip-router.org
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
12:40 p.m.
Hello,
can you send the output of gdb commands:
info locals
bt full
Cheers,
Daniel
On 15/07/16 10:06, Dirk Teurlings - Signet B.V. wrote:
Hi,
Running Kamailio on Debian from the Kamailio repository with 4.4.2
stable (unpatched). Getting some random segfaults with it now, here's
the relevant backtrace from the generated core.
Core was generated by `/usr/sbin/kamailio -f /etc/kamailio/kamailio.cfg
-P /var/run/kamailio/kamailio.'.
Program terminated with signal 11, Segmentation fault.
#0 run_dlg_callbacks (type=type@entry=64, dlg=dlg@entry=0x7fceb400e2f0,
req=req@entry=0x7fced4f093c8, rpl=rpl@entry=0x0, dir=<optimized out>,
dlg_data=dlg_data@entry=0x0) at dlg_cb.c:253
253 if ( (cb->types)&type ) {
(gdb) bt
#0 run_dlg_callbacks (type=type@entry=64, dlg=dlg@entry=0x7fceb400e2f0,
req=req@entry=0x7fced4f093c8, rpl=rpl@entry=0x0, dir=<optimized out>,
dlg_data=dlg_data@entry=0x0) at dlg_cb.c:253
#1 0x00007fcead3648f9 in dlg_terminated (dir=<optimized out>,
dlg=0x7fceb400e2f0, req=0x7fced4f093c8) at dlg_handlers.c:368
#2 dlg_onroute (req=0x7fced4f093c8, route_params=<optimized out>,
param=<optimized out>) at dlg_handlers.c:1354
#3 0x00007fceb10ab0e2 in run_rr_callbacks
(req=req@entry=0x7fced4f093c8, rr_param=rr_param@entry=0x7fceb12b77a0)
at rr_cb.c:96
#4 0x00007fceb10952c5 in after_loose (_m=0x7fced4f093c8, preloaded=0)
at loose.c:919
#5 0x000000000042b618 in do_action (h=h@entry=0x7ffeb0b3ed80,
a=a@entry=0x7fced4cb4338, msg=msg@entry=0x7fced4f093c8) at action.c:1060
#6 0x000000000042a10a in run_actions (h=h@entry=0x7ffeb0b3ed80,
a=0x7fced4cb4338, msg=0x7fced4f093c8) at action.c:1549
#7 0x0000000000437544 in run_actions_safe (h=h@entry=0x7ffeb0b402b0,
a=<optimized out>, msg=<optimized out>) at action.c:1614
#8 0x000000000053b2e8 in rval_get_int (h=0x7ffeb0b402b0, msg=<optimized
out>, i=0x7ffeb0b3f1e0, rv=rv@entry=0x7fced4cb4d58,
cache=cache@entry=0x0) at rvalue.c:912
#9 0x000000000054261c in rval_expr_eval_int (h=h@entry=0x7ffeb0b402b0,
msg=msg@entry=0x7fced4f093c8, res=res@entry=0x7ffeb0b3f1e0,
rve=rve@entry=0x7fced4cb4d50) at rvalue.c:1910
#10 0x000000000042bc91 in do_action (h=h@entry=0x7ffeb0b402b0,
a=a@entry=0x7fced4cb8f88, msg=msg@entry=0x7fced4f093c8) at action.c:1030
#11 0x000000000042a10a in run_actions (h=h@entry=0x7ffeb0b402b0,
a=0x7fced4cb8f88, msg=msg@entry=0x7fced4f093c8) at action.c:1549
#12 0x000000000042bcf2 in do_action (h=h@entry=0x7ffeb0b402b0,
a=a@entry=0x7fced4cb91e8, msg=msg@entry=0x7fced4f093c8) at action.c:1049
#13 0x000000000042a10a in run_actions (h=h@entry=0x7ffeb0b402b0,
a=0x7fced4cb3f48, msg=msg@entry=0x7fced4f093c8) at action.c:1549
#14 0x000000000042bde0 in do_action (h=h@entry=0x7ffeb0b402b0,
a=a@entry=0x7fced4ac3d70, msg=msg@entry=0x7fced4f093c8) at action.c:678
#15 0x000000000042a10a in run_actions (h=h@entry=0x7ffeb0b402b0,
a=a@entry=0x7fced4ac1698, msg=msg@entry=0x7fced4f093c8) at action.c:1549
#16 0x00000000004375d0 in run_top_route (a=0x7fced4ac1698,
msg=msg@entry=0x7fced4f093c8, c=c@entry=0x0) at action.c:1635
#17 0x0000000000504386 in receive_msg (buf=<optimized out>,
len=<optimized out>, rcv_info=<optimized out>) at receive.c:240
#18 0x00000000005f5bd4 in udp_rcv_loop () at udp_server.c:495
#19 0x00000000004b2625 in main_loop () at main.c:1600
#20 0x0000000000427e2b in main (argc=<optimized out>, argv=<optimized
out>) at main.c:2616
And from syslog the relevant messages before this dump:
Jul 15 08:55:03 server /usr/sbin/kamailio[16470]: WARNING: dialog
[dlg_handlers.c:1219]: dlg_onroute(): unable to find dialog for BYE with
route param 'd4c.26d1' [3149:7522]
Jul 15 08:56:01 server /usr/sbin/kamailio[16481]: WARNING: dialog
[dlg_handlers.c:1219]: dlg_onroute(): unable to find dialog for BYE with
route param 'fc.99f1' [207:8089]
Jul 15 08:56:27 server /usr/sbin/kamailio[16470]: CRITICAL: dialog
[dlg_timer.c:200]: update_dlg_timer(): Trying to update a bogus dlg
tl=0x7fceb3f7d920 tl->next=(nil) tl->prev=(nil)
Jul 15 08:56:27 server /usr/sbin/kamailio[16470]: ERROR: dialog
[dlg_handlers.c:1377]: dlg_onroute(): failed to update dialog lifetime
Jul 15 08:57:01 server /usr/sbin/kamailio[16482]: ERROR: db_mysql
[km_dbase.c:128]: db_mysql_submit_query(): driver error on query:
Duplicate entry '9584-3854-435' for key 'hash_index' (1062)
Jul 15 08:57:01 server /usr/sbin/kamailio[16482]: ERROR: <core>
[db_query.c:181]: db_do_raw_query(): error while submitting query
Jul 15 08:57:01 server /usr/sbin/kamailio[16482]: ERROR: sqlops
[sql_api.c:265]: sql_do_query(): cannot do the query [INSERT INTO
`dialog_extra` (`h_i]
Jul 15 08:57:01 server /usr/sbin/kamailio[16482]: ERROR: auth
[api.c:119]: auth_check_hdr_md5(): auth:pre_auth: Credentials are not
filled properly
Jul 15 08:57:01 server /usr/sbin/kamailio[16483]: ERROR: auth
[api.c:119]: auth_check_hdr_md5(): auth:pre_auth: Credentials are not
filled properly
Jul 15 08:57:54 server /usr/sbin/kamailio[16506]: NOTICE: dialog
[dlg_hash.c:245]: dlg_clean_run(): dialog in delete state is too old
(0x7fceb3f64470 ref 4)
Jul 15 08:57:54 server /usr/sbin/kamailio[16473]: WARNING: dialog
[dlg_handlers.c:1348]: dlg_onroute(): inconsitent dlg timer data on dlg
0x7fceb3f64470 [1182:5803] with clid
'09ad128753e2535d24bde58e3d7eda04@192.168.10.232:5060' and tags
'as1b497b34' '5788890C-EC6F55F-3E86ED0C'
Jul 15 08:57:54 server /usr/sbin/kamailio[16469]: ERROR: dialog
[dlg_handlers.c:334]: dlg_terminated_confirmed(): failed to get dialog
from params!
Jul 15 08:58:49 server /usr/sbin/kamailio[16467]: WARNING: dialog
[dlg_handlers.c:1219]: dlg_onroute(): unable to find dialog for BYE with
route param '6d2.2581' [726:6226]
Jul 15 08:59:24 server /usr/sbin/kamailio[16506]: NOTICE: dialog
[dlg_hash.c:245]: dlg_clean_run(): dialog in delete state is too old
(0x7fceb400e2f0 ref 4)
Jul 15 08:59:25 server /usr/sbin/kamailio[16464]: WARNING: dialog
[dlg_handlers.c:1219]: dlg_onroute(): unable to find dialog for BYE with
route param '3e4.b5c1' [1251:7259]
Jul 15 08:59:25 server /usr/sbin/kamailio[16465]: WARNING: dialog
[dlg_handlers.c:1348]: dlg_onroute(): inconsitent dlg timer data on dlg
0x7fceb400e2f0 [1251:7259] with clid '87791a#015#012Call-ID:
25750e286a5654361ef9405d72edbc' and tags '' 'as148f41b1'
Jul 15 08:59:25 server kernel: [203670.830521] kamailio[16465] general
protection ip:7fcead34b3a5 sp:7ffeb0b3e220 error:0 in
dialog.so[7fcead330000+88000]
Jul 15 08:59:26 server /usr/sbin/kamailio[16511]: CRITICAL: <core>
[pass_fd.c:275]: receive_fd(): EOF on 33
Jul 15 08:59:26 server /usr/sbin/kamailio[16458]: ALERT: <core>
[main.c:739]: handle_sigs(): child process 16465 exited by a signal 11
Jul 15 08:59:26 server /usr/sbin/kamailio[16458]: ALERT: <core>
[main.c:742]: handle_sigs(): core was generated
Jul 15 08:59:26 server /usr/sbin/kamailio[16458]: INFO: <core>
[main.c:754]: handle_sigs(): terminating due to SIGCHLD
Any insight would be appreciated!
Cheers,
Dirk
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users(a)lists.sip-router.org
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
--
Daniel-Constantin Mierla
http://www.asipto.com - http://www.kamailio.org
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
12:48 p.m.
Hi, first crash:
(gdb) info locals
cb = <optimized out>
__FUNCTION__ = "run_dlg_callbacks"
(gdb) bt full
#0 run_dlg_callbacks (type=type@entry=64, dlg=dlg@entry=0x7fceb400e2f0,
req=req@entry=0x7fced4f093c8, rpl=rpl@entry=0x0, dir=<optimized out>,
dlg_data=dlg_data@entry=0x0)
at dlg_cb.c:253
cb = <optimized out>
__FUNCTION__ = "run_dlg_callbacks"
#1 0x00007fcead3648f9 in dlg_terminated (dir=<optimized out>,
dlg=0x7fceb400e2f0, req=0x7fced4f093c8) at dlg_handlers.c:368
iuid = 0x0
#2 dlg_onroute (req=0x7fced4f093c8, route_params=<optimized out>,
param=<optimized out>) at dlg_handlers.c:1354
dlg = 0x7fceb400e2f0
iuid = <optimized out>
val = {
s = 0xa36706
"3e4.b5c1;nat=yes>,<sip:31.223.168.234;r2=on;lr;ftag=as0b87791a;did=3e4.b5c1;nat=yes>\r\nMax-Forwards:
69\r\nFrom: <sip:+31737470617@31.223.168.234>;tag=as148f41b1\r\nTo:
+33612818032 <sip:signetOLD@sip.wlvo"..., len = 8}
callid = {
s = 0xa367ef
"25750e286a5654361ef9405d72edbc54(a)sip.wlvoip.net\r\nCSeqnCSeq: 102
BYE\r\nUser-Agent: sip.wlvoip.net\r\nX-Asterisk-HangupCause: Normal
Clearing\r\nX-Asterisk-HangupCauseCode: 16\r\nContent-Length:
0\r\n\r\n", len = 47}
ftag = {
s = 0xa3679a "as148f41b1\r\nTo: +33612818032
<sip:signetOLD@sip.wlvoip.net>;tag=as0b87791a\r\nCall-ID:
25750e286a5654361ef9405d72edbc54(a)sip.wlvoip.net\r\nCSeq: 102
BYE\r\nUser-Agent: sip.wlvoip.net\r\nX-Asterisk-HangupCause:"..., len = 10}
ttag = {
s = 0xa367da "as0b87791a\r\nCall-ID:
25750e286a5654361ef9405d72edbc54(a)sip.wlvoip.net\r\nCSeq: 102
BYE\r\nUser-Agent: sip.wlvoip.net\r\nX-Asterisk-HangupCause: Normal
Clearing\r\nX-Asterisk-HangupCauseCode: 16\r\nContent-Length"..., len = 10}
h_entry = 1251
h_id = 7259
new_state = 5
old_state = 4
unref = 1
event = <optimized out>
timeout = -722431032
reset = <optimized out>
dir = 2
ret = <optimized out>
__FUNCTION__ = "dlg_onroute"
#3 0x00007fceb10ab0e2 in run_rr_callbacks
(req=req@entry=0x7fced4f093c8, rr_param=rr_param@entry=0x7fceb12b77a0)
at rr_cb.c:96
l_param = {
s = 0xa366e9
"r2=on;lr;ftag=as0b87791a;did=3e4.b5c1;nat=yes>,<sip:31.223.168.234;r2=on;lr;ftag=as0b87791a;did=3e4.b5c1;nat=yes>\r\nMax-Forwards:
69\r\nFrom: <sip:+31737470617@31.223.168.234>;tag=as148f41b1\r\nTo:
+336128"..., len = 45}
cbp = 0x7fced4abc0a0
__FUNCTION__ = "run_rr_callbacks"
#4 0x00007fceb10952c5 in after_loose (_m=0x7fced4f093c8, preloaded=0)
at loose.c:919
hdr = <optimized out>
puri = {user = {s = 0x0, len = 0}, passwd = {s = 0x0, len = 0},
host = {
s = 0xa3671d
"31.223.168.234;r2=on;lr;ftag=as0b87791a;did=3e4.b5c1;nat=yes>\r\nMax-Forwards:
69\r\nFrom: <sip:+31737470617@31.223.168.234>;tag=as148f41b1\r\nTo:
+33612818032 <sip:signetOLD@sip.wlvoip.net>;tag=as0b87791a\r"..., len =
14}, port = {s = 0x0, len = 0}, params = {
s = 0xa3672c
"r2=on;lr;ftag=as0b87791a;did=3e4.b5c1;nat=yes>\r\nMax-Forwards:
69\r\nFrom: <sip:+31737470617@31.223.168.234>;tag=as148f41b1\r\nTo:
+33612818032 <sip:signetOLD@sip.wlvoip.net>;tag=as0b87791a\r\nCall-ID:
25750"..., len = 45}, sip_params = {
s = 0xa3672c
"r2=on;lr;ftag=as0b87791a;did=3e4.b5c1;nat=yes>\r\nMax-Forwards:
69\r\nFrom: <sip:+31737470617@31.223.168.234>;tag=as148f41b1\r\nTo:
+33612818032 <sip:signetOLD@sip.wlvoip.net>;tag=as0b87791a\r\nCall-ID:
25750"..., len = 45}, headers = {s = 0x0, len = 0}, port_no = 0, proto =
0, type = SIP_URI_T, flags = 0, transport = {s = 0x0,
len = 0}, ttl = {s = 0x0, len = 0}, user_param = {s = 0x0,
len = 0}, maddr = {s = 0x0, len = 0}, method = {s = 0x0, len = 0}, lr = {
s = 0xa36732
"lr;ftag=as0b87791a;did=3e4.b5c1;nat=yes>\r\nMax-Forwards: 69\r\nFrom:
<sip:+31737470617@31.223.168.234>;tag=as148f41b1\r\nTo: +33612818032
<sip:signetOLD@sip.wlvoip.net>;tag=as0b87791a\r\nCall-ID:
25750e286a5"..., len = 2}, r2 = {
---Type <return> to continue, or q <return> to quit---
s = 0xa3672c
"r2=on;lr;ftag=as0b87791a;did=3e4.b5c1;nat=yes>\r\nMax-Forwards:
69\r\nFrom: <sip:+31737470617@31.223.168.234>;tag=as148f41b1\r\nTo:
+33612818032 <sip:signetOLD@sip.wlvoip.net>;tag=as0b87791a\r\nCall-ID:
25750"..., len = 5}, gr = {s = 0x0, len = 0}, transport_val = {s = 0x0,
len = 0}, ttl_val = {s = 0x0, len = 0}, user_param_val = {
s = 0x0, len = 0}, maddr_val = {s = 0x0, len = 0},
method_val = {s = 0x0, len = 0}, lr_val = {s = 0x0, len = 0}, r2_val = {
s = 0xa3672f
"on;lr;ftag=as0b87791a;did=3e4.b5c1;nat=yes>\r\nMax-Forwards:
69\r\nFrom: <sip:+31737470617@31.223.168.234>;tag=as148f41b1\r\nTo:
+33612818032 <sip:signetOLD@sip.wlvoip.net>;tag=as0b87791a\r\nCall-ID:
25750e28"..., len = 2}, gr_val = {s = 0x0, len = 0}}
rt = <optimized out>
status = <optimized out>
uri = {
s = 0xa366d6
"sip:192.168.10.230;r2=on;lr;ftag=as0b87791a;did=3e4.b5c1;nat=yes>,<sip:31.223.168.234;r2=on;lr;ftag=as0b87791a;did=3e4.b5c1;nat=yes>\r\nMax-Forwards:
69\r\nFrom: <sip:+31737470617@31.223.168.234>;tag=as14"..., len = 64}
si = <optimized out>
uri_is_myself = 25841
use_ob = 0
__FUNCTION__ = "after_loose"
#5 0x000000000042b618 in do_action (h=h@entry=0x7ffeb0b3ed80,
a=a@entry=0x7fced4cb4338, msg=msg@entry=0x7fced4f093c8) at action.c:1060
ret = -5
v = <optimized out>
dst = {send_sock = 0xb0b3f770, to = {s = {sa_family = 61456,
sa_data = "\263\260\376\177\000\000\377\377\377\377\316\177\000"}, sin =
{sin_family = 61456, sin_port = 45235,
sin_addr = {s_addr = 32766}, sin_zero =
"\377\377\377\377\316\177\000"}, sin6 = {sin6_family = 61456, sin6_port
= 45235, sin6_flowinfo = 32766, sin6_addr = {__in6_u = {
__u6_addr8 =
"\377\377\377\377\316\177\000\000DuC\000\000\000\000", __u6_addr16 =
{65535, 65535, 32718, 0, 30020, 67, 0, 0}, __u6_addr32 = {4294967295,
32718,
4420932, 0}}}, sin6_scope_id = 0}}, id = 0, proto =
-20 '\354', send_flags = {f = 123 '{', blst_imask = 172 '\254'}}
tmp = <optimized out>
new_uri = <optimized out>
end = <optimized out>
crt = <optimized out>
cmd = 0x7fced4aa75a8
len = <optimized out>
user = <optimized out>
uri = {user = {s = 0x7fced4dd1a90 "\001", len = -1343281044},
passwd = {s = 0x48a0f8 "\207\231", len = 8}, host = {s = 0x14b <Address
0x14b out of bounds>,
len = -723700576}, port = {s = 0x7fced4dd34a0 "6", len =
6334740}, params = {s = 0x40 <Address 0x40 out of bounds>, len =
6344353}, sip_params = {
s = 0x7fced465a845 "%-d", len = -723706432}, headers = {
s = 0xa367bc
"signetOLD(a)sip.wlvoip.net>;tag=as0b87791a\r\nCall-IDnCall-ID:
25750e286a5654361ef9405d72edbc54(a)sip.wlvoip.net\r\nCSeq: 102
BYE\r\nUser-Agent: sip.wlvoip.net\r\nX-Asterisk-HangupCause: Normal
Clearing\r\nX-Asterisk-Hangu"..., len = 9}, port_no = 0, proto = 0, type
= ERROR_URI_T, flags = 4294967295, transport = {s = 0x7fced4c75270 "\001",
len = -722431032}, ttl = {s = 0x7ffeb0b402b0 "\003", len =
-1330385904}, user_param = {s = 0x7fced4f093c8 "Z\025", len = 5514780},
maddr = {s = 0x0, len = 0}, method = {
s = 0x0, len = 0}, lr = {s = 0x100000000 <Address
0x100000000 out of bounds>, len = 0}, r2 = {s = 0x0, len = 0}, gr = {s =
0x0, len = 0}, transport_val = {s = 0x0,
len = 0}, ttl_val = {s = 0x0, len = 0}, user_param_val = {s
= 0x0, len = 0}, maddr_val = {s = 0x0, len = 0}, method_val = {s = 0x0,
len = 0}, lr_val = {s = 0x0,
len = -1011016704}, r2_val = {s = 0x0, len = -725128304},
gr_val = {s = 0x7fced4f093c8 "Z\025", len = 0}}
next_hop = {user = {s = 0xfffffffb <Address 0xfffffffb out of
bounds>, len = -1330380112}, passwd = {s = 0x7fced4f093c8 "Z\025", len =
4366602}, host = {s = 0x0, len = 0},
port = {s = 0x0, len = 0}, params = {s = 0x0, len = 0},
sip_params = {s = 0x0, len = 0}, headers = {s = 0x0, len = 1511},
port_no = 37832, proto = 54512, type = 32718,
flags = 7, transport = {s = 0xfffffffb <Address 0xfffffffb out
of bounds>, len = -726987760}, ttl = {s = 0x7fced4f093c8 "Z\025", len =
-1343315701}, user_param = {
s = 0x7fced4c98fd0 "\n", len = -722431032}, maddr = {s =
0x7ffeb0b402b0 "\003", len = -725128304}, method = {s = 0x7fced4f093c8
"Z\025", len = -1329584299}, lr = {
s = 0x0, len = 4}, r2 = {s = 0x7fce00000000 <Address
0x7fce00000000 out of bounds>, len = -1011016704}, gr = {s = 0xa
<Address 0xa out of bounds>, len = -725128000},
transport_val = {s = 0x7fced4f093c8 "Z\025", len =
-1330380112}, ttl_val = {s = 0x1 <Address 0x1 out of bounds>, len =
-1330380112}, user_param_val = {
s = 0x7fced4f093c8 "Z\025", len = 4373746}, maddr_val = {s =
0x7fced4f093c8 "Z\025", len = 4371992}, method_val = {s = 0x70 <Address
0x70 out of bounds>,
len = -1330385016}, lr_val = {s = 0x7fced4658a7d "%s%.*s%s
", len = 0}, r2_val = {s = 0x7ffe00000000 <Address 0x7ffe00000000 out of
bounds>, len = -1330382992},
gr_val = {s = 0x0, len = -722431032}}
u = <optimized out>
port = <optimized out>
dst_host = <optimized out>
i = <optimized out>
---Type <return> to continue, or q <return> to quit---
flags = <optimized out>
avp = <optimized out>
st = {flags = 2964587184, id = 32766, name = {n = 5485288, s = {
s = 0x53b2e8
"\205\300A\017\237\300\061\300E\017\266\310D\211\v\203e\004\371\351
\377\377\377H\211\372H\215q\bH\215|$P\350?\244\001", len = -723724760},
re = 0x53b2e8}, avp = 0x7fced4dd3858}
sct = <optimized out>
sjt = <optimized out>
rve = <optimized out>
mct = <optimized out>
rv = <optimized out>
rv1 = <optimized out>
c1 = {cache_type = 3563424653, val_type = 32718, c = {avp_val =
{n = 1034164339, s = {s = 0x1dec91573da41873 <Address 0x1dec91573da41873
out of bounds>, len = 1},
re = 0x1dec91573da41873}, pval = {rs = {s =
0x1dec91573da41873 <Address 0x1dec91573da41873 out of bounds>, len = 1},
ri = -1330384888, flags = 32766}},
i2s =
"\001\000\000\000\316\177\000\000\000\000\000\000\000\000\000\000\232\213e\324\316\177"}
s = {s = 0x0, len = 789}
srevp = {0x3b00000000, 0x7ffeb0b3ec10}
mod_f_params = {{type = NOSUBTYPE, u = {number = 0, string =
0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}},
{type = NOSUBTYPE, u = {number = 0,
string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr =
0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0,
str = {s = 0x0, len = 0}, data = 0x0,
attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number
= 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0,
select = 0x0}}, {type = NOSUBTYPE, u = {
number = 0, string = 0x0, str = {s = 0x0, len = 0}, data =
0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0,
string = 0x0, str = {s = 0x0, len = 0},
data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE,
u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0,
attr = 0x0, select = 0x0}}, {
type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s =
0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}}
__FUNCTION__ = "do_action"
#6 0x000000000042a10a in run_actions (h=h@entry=0x7ffeb0b3ed80,
a=0x7fced4cb4338, msg=0x7fced4f093c8) at action.c:1549
t = <optimized out>
ret = <optimized out>
mod = <optimized out>
ms = 0
__FUNCTION__ = "run_actions"
#7 0x0000000000437544 in run_actions_safe (h=h@entry=0x7ffeb0b402b0,
a=<optimized out>, msg=<optimized out>) at action.c:1614
ctx = {rec_lev = 1, run_flags = 0, last_retcode = 0, jmp_env =
{{__jmpbuf = {140731863007920, 2156258125485971644, 140526312526792,
140731863007920, 140731863003616,
140526312526792, -2156678653738215236,
2156258654672656572}, __mask_was_saved = 0, __saved_mask = {__val =
{39367408, 140731862994288, 140731862990144,
140728898420737, 140731863002752, 39289776,
18446744073709551615, 4294967291, 140525646082718, 140526445694632, 2,
4, 3760787172421009408, 3689070625499067697, 0,
0}}}}}
ret = 25841
ign_on_break = 0
#8 0x000000000053b2e8 in rval_get_int (h=0x7ffeb0b402b0, msg=<optimized
out>, i=0x7ffeb0b3f1e0, rv=rv@entry=0x7fced4cb4d58,
cache=cache@entry=0x0) at rvalue.c:912
r_avp = <optimized out>
avp_val = {n = -723402960, s = {s = 0x7fced4e1bf30 "\001", len =
-1388967936}, re = 0x7fced4e1bf30}
pval = {rs = {s = 0x3000000018 <Address 0x3000000018 out of
bounds>, len = -1330384320}, ri = -1330384512, flags = 32766}
tmp = {s = 0x0, len = -592155317}
s = 0x0
r = <optimized out>
ret = 0
destroy_pval = 0
__FUNCTION__ = "rval_get_int"
#9 0x000000000054261c in rval_expr_eval_int (h=h@entry=0x7ffeb0b402b0,
msg=msg@entry=0x7fced4f093c8, res=res@entry=0x7ffeb0b3f1e0,
rve=rve@entry=0x7fced4cb4d50) at rvalue.c:1910
i1 = <optimized out>
i2 = <optimized out>
ret = -1
---Type <return> to continue, or q <return> to quit---
c1 = {cache_type = RV_CACHE_EMPTY, val_type = RV_NONE, c =
{avp_val = {n = 0, s = {s = 0x0, len = 38525735}, re = 0x0}, pval = {rs
= {s = 0x0, len = 38525735},
ri = -1330384448, flags = 32766}}, i2s = "\001", '\000'
<repeats 15 times>"\327, \376\000\000\000"}
c2 = {cache_type = 2964587184, val_type = 32766, c = {avp_val =
{n = -725128000, s = {s = 0x7fced4c76cc0 "%", len = -722431032}, re =
0x7fced4c76cc0}, pval = {rs = {
s = 0x7fced4c76cc0 "%", len = -722431032}, ri =
-724987952, flags = 32718}}, i2s =
"È“\360\324\316\177\000\000\003\000\000\000\316\177\000\000\026\000\000\000\000"}
rv1 = <optimized out>
rv2 = <optimized out>
__FUNCTION__ = "rval_expr_eval_int"
#10 0x000000000042bc91 in do_action (h=h@entry=0x7ffeb0b402b0,
a=a@entry=0x7fced4cb8f88, msg=msg@entry=0x7fced4f093c8) at action.c:1030
ret = -5
v = <optimized out>
dst = {send_sock = 0x7fcead5b9990, to = {s = {sa_family = 39280,
sa_data = "[\255\316\177\000\000P\231[\255\316\177\000"}, sin =
{sin_family = 39280, sin_port = 44379,
sin_addr = {s_addr = 32718}, sin_zero =
"P\231[\255\316\177\000"}, sin6 = {sin6_family = 39280, sin6_port =
44379, sin6_flowinfo = 32718, sin6_addr = {__in6_u = {
__u6_addr8 =
"P\231[\255\316\177\000\000`\231[\255\316\177\000", __u6_addr16 =
{39248, 44379, 32718, 0, 39264, 44379, 32718, 0}, __u6_addr32 =
{2908461392, 32718,
2908461408, 32718}}}, sin6_scope_id = 2908461328}},
id = 32718, proto = 32 ' ', send_flags = {f = 153 '\231', blst_imask =
91 '['}}
tmp = <optimized out>
new_uri = <optimized out>
end = <optimized out>
crt = <optimized out>
cmd = <optimized out>
len = <optimized out>
user = <optimized out>
uri = {user = {s = 0x7fced485e210 "", len = -1279372408}, passwd
= {s = 0x0, len = 0}, host = {
s = 0x24bda00 "insert into `dialog`
(`hash_entry`,`hash_id`,`callid`,`from_uri`,`from_tag`,`to_uri`,`to_tag`,`caller_sock`,`callee_sock`,`start_time`,`state`,`timeout`,`caller_cseq`,`callee_cseq`,`caller_contact`,`c"...,
len = -723684376}, port = {s = 0x7fced4dd76e8 "", len = -1286996357},
params = {s = 0x7fced485e210 "", len = -723684376},
sip_params = {s = 0x7fced485e210 "", len = -1286991478},
headers = {
s = 0x24bda00 "insert into `dialog`
(`hash_entry`,`hash_id`,`callid`,`from_uri`,`from_tag`,`to_uri`,`to_tag`,`caller_sock`,`callee_sock`,`start_time`,`state`,`timeout`,`caller_cseq`,`callee_cseq`,`caller_contact`,`c"...,
len = 39331786}, port_no = 29672, proto = 54493, type = 32718, flags =
3565543952, transport = {s = 0x7ffeb0b3f350 "",
len = -724876992}, ttl = {s = 0x7fced4f093c8 "Z\025", len =
-722431032}, user_param = {s = 0x7ffeb0b402b0 "\003", len = 0}, maddr =
{s = 0x7fced4f093c8 "Z\025",
len = 5514780}, method = {s = 0x111 <Address 0x111 out of
bounds>, len = -1330380112}, lr = {s = 0x0, len = -722431032}, r2 = {s =
0x7ffeb0b402b0 "\003",
len = -1330382928}, gr = {s = 0x7fced4f093c8 "Z\025", len =
4420932}, transport_val = {s = 0x0, len = 1}, ttl_val = {s =
0x7ffeb0b402b0 "\003", len = 1034164412},
user_param_val = {s = 0x7fced4f093c8 "Z\025", len =
-1330380112}, maddr_val = {s = 0x7ffeb0b3f7b0 "\001", len = -722431032},
method_val = {
s = 0xe211f030d92418bc <Address 0xe211f030d92418bc out of
bounds>, len = 1939871932}, lr_val = {s = 0x0, len = -1388646483},
r2_val = {s = 0x7fceb39dcf46 "`",
len = 5060}, gr_val = {s = 0x2 <Address 0x2 out of bounds>,
len = 0}}
next_hop = {user = {s = 0x7fceb3fb1540 "\002", len = 0}, passwd
= {s = 0x7ffefffffffb <Address 0x7ffefffffffb out of bounds>, len =
10042548}, host = {
s = 0x7fce00000016 <Address 0x7fce00000016 out of bounds>,
len = -1389034191}, port = {s = 0x7fceb392f640 "\002", len =
-1011016704}, params = {
s = 0x7fceb3fb1540 "\002", len = -1330382928}, sip_params =
{s = 0x7ffeb0b402b0 "\003", len = -722431032}, headers = {s =
0x7ffeb0b402b0 "\003", len = -1330382928},
port_no = 688, proto = 45236, type = 32766, flags = 5485288,
transport = {s = 0x3 <Address 0x3 out of bounds>, len = -1389001313},
ttl = {
s = 0xa3664c
"etOLD@31.223.168.251:5060;alias=31.223.168.251~5060~1 SIP/2.0\r\nVia:
SIP/2.0/UDP 192.168.10.233:5060;branch=z9hG4bK1fbee766;rport\r\nRoute:
<sip:192.168.10.230;r2=on;lr;ftag=as0b87791a;did=3e4.b5c1;nat=y"..., len
= -731535288}, user_param = {s = 0x0, len = 1}, maddr = {s = 0x13c4
<Address 0x13c4 out of bounds>, len = 0}, method = {
s = 0x0, len = 5060}, lr = {s = 0x7fce00000000 <Address
0x7fce00000000 out of bounds>, len = -1310504815}, r2 = {s = 0x8000000
<Address 0x8000000 out of bounds>,
len = 129936856}, gr = {s = 0x370690 <Address 0x370690 out
of bounds>, len = -1011016704}, transport_val = {s = 0x0, len = -1},
ttl_val = {s = 0x7fced4cb4598 "\001",
len = -722431032}, user_param_val = {s = 0x7ffeb0b402b0
"\003", len = -1330382928}, maddr_val = {s = 0x7fced4f093c8 "Z\025",
len
= 5514780}, method_val = {
s = 0x7fcead5b9a00 "mu:\255\316\177", len = -1386505744},
lr_val = {s = 0x7fcead5b99e0 "du:\255\316\177", len = -1386505776},
r2_val = {
s = 0x7fcead5b99c0 "]u:\255\316\177", len = -1386506000},
gr_val = {s = 0x7fcead5b9900 "\356t:\255\316\177", len = -1386505856}}
u = <optimized out>
port = <optimized out>
dst_host = <optimized out>
i = <optimized out>
flags = <optimized out>
avp = <optimized out>
---Type <return> to continue, or q <return> to quit---
st = {flags = 3572536264, id = 32718, name = {n = 4366602, s =
{s = 0x42a10a "H\213\r\177\201V", len = -1330380112}, re = 0x42a10a},
avp = 0x53b2e8}
sct = <optimized out>
sjt = <optimized out>
rve = 0x7fced4cb4d50
mct = <optimized out>
rv = <optimized out>
rv1 = <optimized out>
c1 = {cache_type = RV_CACHE_PVAR, val_type = 32718, c = {avp_val
= {n = -1386505920, s = {s = 0x7fcead5b9940 ")u:\255\316\177", len =
-1386506016}, re = 0x7fcead5b9940},
pval = {rs = {s = 0x7fcead5b9940 ")u:\255\316\177", len =
-1386506016}, ri = -1386506048, flags = 32718}},
i2s =
"\260\230[\255\316\177\000\000\240\230[\255\316\177\000\000И[\255\316\177"}
s = {s = 0x0, len = 0}
srevp = {0xfffffffb, 0x7ffeb0b3f350}
mod_f_params = {{type = NOSUBTYPE, u = {number = 0, string =
0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}},
{type = NOSUBTYPE, u = {number = 0,
string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr =
0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0,
str = {s = 0x0, len = 0}, data = 0x0,
attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number
= 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0,
select = 0x0}}, {type = NOSUBTYPE, u = {
number = 0, string = 0x0, str = {s = 0x0, len = 0}, data =
0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0,
string = 0x0, str = {s = 0x0, len = 0},
data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE,
u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0,
attr = 0x0, select = 0x0}}, {
type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s =
0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}}
__FUNCTION__ = "do_action"
#11 0x000000000042a10a in run_actions (h=h@entry=0x7ffeb0b402b0,
a=0x7fced4cb8f88, msg=msg@entry=0x7fced4f093c8) at action.c:1549
t = <optimized out>
ret = <optimized out>
mod = <optimized out>
ms = 0
__FUNCTION__ = "run_actions"
#12 0x000000000042bcf2 in do_action (h=h@entry=0x7ffeb0b402b0,
a=a@entry=0x7fced4cb91e8, msg=msg@entry=0x7fced4f093c8) at action.c:1049
ret = 1
v = 1
dst = {send_sock = 0x0, to = {s = {sa_family = 57696, sa_data =
"\244\000\000\000\000\000È“\360\324\316\177\000"}, sin = {sin_family =
57696, sin_port = 164, sin_addr = {
s_addr = 0}, sin_zero = "È“\360\324\316\177\000"}, sin6 =
{sin6_family = 57696, sin6_port = 164, sin6_flowinfo = 0, sin6_addr =
{__in6_u = {
__u6_addr8 =
"È“\360\324\316\177\000\000^\337r\261\316\177\000", __u6_addr16 = {37832,
54512, 32718, 0, 57182, 45426, 32718, 0}, __u6_addr32 = {3572536264, 32718,
2977095518, 32718}}}, sin6_scope_id = 3019872104}},
id = 32718, proto = -111 '\221', send_flags = {f = 72 'H', blst_imask =
227 '\343'}}
tmp = <optimized out>
new_uri = <optimized out>
end = <optimized out>
crt = <optimized out>
cmd = <optimized out>
len = <optimized out>
user = <optimized out>
uri = {user = {s = 0x100000004 <Address 0x100000004 out of
bounds>, len = 131072}, passwd = {s = 0x0, len = 0}, host = {s = 0x4
<Address 0x4 out of bounds>,
len = -1275390656}, port = {s = 0x7fceb3fb1715
"sip:+31203128011@sip.wlvoip.net5060et60DP 1\030", len = 31}, params = {
s = 0x100000004 <Address 0x100000004 out of bounds>, len =
10046212}, sip_params = {s = 0x0, len = -1388862181}, headers = {s =
0x0, len = -1330381968}, port_no = 3,
proto = 0, type = ERROR_URI_T, flags = 2905997787, transport =
{s = 0x7ffeb0b3f920 "", len = -726918520}, ttl = {s = 0x7fced4f093c8
"Z\025", len = 16465}, user_param = {
s = 0x7fceb3ec2bf4 "", len = 0}, maddr = {s = 0x7fced4dd40e8
"\001", len = -1388997077}, method = {s = 0x7fced4dcddc0 "", len =
-1330380112}, lr = {s = 0x0,
len = -722431032}, r2 = {s = 0x7ffeb0b402b0 "\003", len =
-1330381440}, gr = {s = 0x7fced4f093c8 "Z\025", len = 4420932},
transport_val = {s = 0x0, len = -1}, ttl_val = {
s = 0x7ffeb0b402b0 "\003", len = 1034164412}, user_param_val
= {s = 0x7fced4f093c8 "Z\025", len = -1330380112}, maddr_val = {s =
0x7ffeb0b3fd80 "", len = -722431032},
method_val = {s = 0xe211f030cd8418bc <Address
0xe211f030cd8418bc out of bounds>, len = 1939871932}, lr_val = {s = 0x0,
len = 56}, r2_val = {
s = 0x38 <Address 0x38 out of bounds>, len = 6344353},
gr_val = {s = 0x7fced4b5f6e0 "\024", len = -1315021152}}
next_hop = {user = {s = 0x7fceb19e5cc0 "\275p\002", len =
16482}, passwd = {s = 0x7fceb3fa4c88 "\001", len = 160167}, host = {s =
0x7fceb3fa8da0 "p\202\344\263\316\177",
len = -1330381632}, port = {s = 0x2 <Address 0x2 out of
bounds>, len = -1388814079}, params = {s = 0x8000000 <Address 0x8000000
out of bounds>, len = 129937120},
---Type <return> to continue, or q <return> to quit---
sip_params = {s = 0x370588 <Address 0x370588 out of bounds>,
len = 4280608}, headers = {s = 0x3 <Address 0x3 out of bounds>, len =
-1330381440}, port_no = 688,
proto = 45236, type = 32766, flags = 5485288, transport = {s =
0x2 <Address 0x2 out of bounds>, len = -1275095000}, ttl = {
s = 0x7fced4dcbc00 "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP
176.74.252.249:5060;received=176.74.252.249;branch=z9hG4bK6e6c5f53;rport=35255\r\nRecord-Route:
<sip:192.168.10.230;r2=on;lr;ftag=as77d9913b;did=3f5.3321;nat=yes>\r\nRecor"...,
len = -1275442040}, user_param = {s = 0x7fced4f093c8 "Z\025", len =
-1275199272}, maddr = {
s = 0x7fceb3fa8da0 "p\202\344\263\316\177", len =
-1330381632}, method = {s = 0x2 <Address 0x2 out of bounds>, len =
-1388928332}, lr = {
s = 0x8000000 <Address 0x8000000 out of bounds>, len =
129941872}, r2 = {s = 0x36f2f8 <Address 0x36f2f8 out of bounds>, len =
4275856}, gr = {s = 0x48a0f8 "\207\231",
len = 8}, transport_val = {s = 0x154 <Address 0x154 out of
bounds>, len = -1}, ttl_val = {s = 0x7fced4ac1c80 "\001", len =
-722431032}, user_param_val = {
s = 0x7ffeb0b402b0 "\003", len = -1330381440}, maddr_val =
{s = 0x7fced4f093c8 "Z\025", len = 5514780}, method_val = {
s = 0x7fced4dcbc00 "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP
176.74.252.249:5060;received=176.74.252.249;branch=z9hG4bK6e6c5f53;rport=35255\r\nRecord-Route:
<sip:192.168.10.230;r2=on;lr;ftag=as77d9913b;did=3f5.3321;nat=yes>\r\nRecor"...,
len = 973}, lr_val = {s = 0x7fceb3ff9878 "\230\273\356\324\316\177", len
= 0}, r2_val = {s = 0xa466b0 "",
len = -1275095192}, gr_val = {s = 0x30 <Address 0x30 out of
bounds>, len = -723715568}}
u = <optimized out>
port = <optimized out>
dst_host = <optimized out>
i = <optimized out>
flags = <optimized out>
avp = <optimized out>
st = {flags = 3572536264, id = 32718, name = {n = 4366602, s =
{s = 0x42a10a "H\213\r\177\201V", len = 0}, re = 0x42a10a}, avp =
0x7fceb3f6b8a0}
sct = <optimized out>
sjt = <optimized out>
rve = <optimized out>
mct = <optimized out>
rv = <optimized out>
rv1 = <optimized out>
c1 = {cache_type = 134217728, val_type = RV_NONE, c = {avp_val =
{n = 129941688, s = {s = 0x7bec0b8 <Address 0x7bec0b8 out of bounds>,
len = 3601328}, re = 0x7bec0b8},
pval = {rs = {s = 0x7bec0b8 <Address 0x7bec0b8 out of
bounds>, len = 3601328}, ri = 4276040, flags = 0}},
i2s = "\370\240H\000\000\000\000\000\b", '\000' <repeats
12
times>}
s = {s = 0x0, len = 0}
srevp = {0xfffffffb, 0x7ffeb0b3f920}
mod_f_params = {{type = NOSUBTYPE, u = {number = 0, string =
0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}},
{type = NOSUBTYPE, u = {number = 0,
string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr =
0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0,
str = {s = 0x0, len = 0}, data = 0x0,
attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number
= 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0,
select = 0x0}}, {type = NOSUBTYPE, u = {
number = 0, string = 0x0, str = {s = 0x0, len = 0}, data =
0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0,
string = 0x0, str = {s = 0x0, len = 0},
data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE,
u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0,
attr = 0x0, select = 0x0}}, {
type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s =
0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}}
__FUNCTION__ = "do_action"
#13 0x000000000042a10a in run_actions (h=h@entry=0x7ffeb0b402b0,
a=0x7fced4cb3f48, msg=msg@entry=0x7fced4f093c8) at action.c:1549
t = <optimized out>
ret = <optimized out>
mod = <optimized out>
ms = 0
__FUNCTION__ = "run_actions"
#14 0x000000000042bde0 in do_action (h=h@entry=0x7ffeb0b402b0,
a=a@entry=0x7fced4ac3d70, msg=msg@entry=0x7fced4f093c8) at action.c:678
ret = -5
v = <optimized out>
dst = {send_sock = 0x1, to = {s = {sa_family = 14152, sa_data =
"\335\324\316\177\000\000\200f\243\000\000\000\000"}, sin = {sin_family
= 14152, sin_port = 54493,
sin_addr = {s_addr = 32718}, sin_zero =
"\200f\243\000\000\000\000"}, sin6 = {sin6_family = 14152, sin6_port =
54493, sin6_flowinfo = 32718, sin6_addr = {__in6_u = {
__u6_addr8 = "\200f\243", '\000' <repeats 12
times>,
__u6_addr16 = {26240, 163, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {10708608,
0, 0, 0}}},
sin6_scope_id = 3571266416}}, id = 32718, proto = -57
'\307', send_flags = {f = 102 'f', blst_imask = 163 '\243'}}
tmp = <optimized out>
---Type <return> to continue, or q <return> to quit---
new_uri = <optimized out>
end = <optimized out>
crt = <optimized out>
cmd = <optimized out>
len = <optimized out>
user = <optimized out>
uri = {user = {s = 0x7fced4dd40e8 "\001", len = -1315036816},
passwd = {s = 0x7fced4ef53b8 "\020\325\177\255\316\177", len =
-1317733006}, host = {
s = 0x7fced4f093c8 "Z\025", len = 359}, port = {s =
0x7fceb3f7b208 "\230\273\356\324\316\177", len = 0}, params = {s =
0x7fced485e210 "", len = -1275084800},
sip_params = {s = 0x7fceb3ffc208 "", len = -1317725785},
headers = {s = 0x7fceb3ffc000 "\260<\343\263\316\177", len = 404},
port_no = 0, proto = 0, type = ERROR_URI_T,
flags = 2977095518, transport = {s = 0x7fceb17cfc2e "tm:
t_reply.c", len = -1317206832}, ttl = {s = 0x8a6 <Address 0x8a6 out of
bounds>, len = -1317574291}, user_param = {
s = 0x0, len = 0}, maddr = {s = 0x0, len = 0}, method = {s =
0x7fced4f093c8 "Z\025", len = -723697432}, lr = {s = 0x7fceb3fa8fa8
"",
len = -722431032}, r2 = {
s = 0x7fceb3fa8da0 "p\202\344\263\316\177", len = 200}, gr =
{s = 0x1e7 <Address 0x1e7 out of bounds>, len = -1317536758},
transport_val = {s = 0x0, len = 1468565862},
ttl_val = {s = 0x7fced4dd3c68 "H.Y\002", len = -1281634096},
user_param_val = {s = 0x992290 "\314\060\356\263\316\177", len = 69},
maddr_val = {s = 0x993cb4 "\377\377",
len = 0}, method_val = {
s = 0x24bda00 "insert into `dialog`
(`hash_entry`,`hash_id`,`callid`,`from_uri`,`from_tag`,`to_uri`,`to_tag`,`caller_sock`,`callee_sock`,`start_time`,`state`,`timeout`,`caller_cseq`,`callee_cseq`,`caller_contact`,`c"...,
len = -731621092}, lr_val = {s = 0x2 <Address 0x2 out of bounds>, len =
-1310504815}, r2_val = {
s = 0x8000000 <Address 0x8000000 out of bounds>, len =
129942584}, gr_val = {s = 0x36f030 <Address 0x36f030 out of bounds>, len
= 4275144}}
next_hop = {user = {s = 0x48a0f8 "\207\231", len = 8}, passwd =
{s = 0x159 <Address 0x159 out of bounds>, len = 64}, host = {s = 0x40
<Address 0x40 out of bounds>,
len = 6344353}, port = {s = 0xa466c8 "", len = 10774544},
params = {s = 0x167 <Address 0x167 out of bounds>, len = 6333667},
sip_params = {s = 0x0, len = -1276371820},
headers = {
s = 0xa36778
":+31737470617@31.223.168.234>;tag=as148f41b1\r\nTo: +33612818032
<sip:signetOLD@sip.wlvoip.net>;tag=as0b87791a\r\nCall-ID:
25750e286a5654361ef9405d72edbc54(a)sip.wlvoip.net\r\nCSeq: 102
BYE\r\nUser-Agent: sip.w"..., len = 52}, port_no = 0, proto = 0, type =
32718, flags = 3018588160, transport = {s = 0x40 <Address 0x40 out of
bounds>,
len = -723699896}, ttl = {s = 0x30 <Address 0x30 out of
bounds>, len = -723715568}, user_param = {s = 0x0, len = -591593503},
maddr = {s = 0x258ac6c "", len = 2510},
method = {s = 0x38 <Address 0x38 out of bounds>, len =
10708679}, lr = {s = 0x6 <Address 0x6 out of bounds>, len = -723700880},
r2 = {s = 0xa368a9 "", len = 100}, gr = {
s = 0xa36690 "SIP/2.0/UDP
192.168.10.233:5060;branch=z9hG4bK1fbee766;rport\r\nRoute:
<sip:192.168.10.230;r2=on;lr;ftag=as0b87791a;did=3e4.b5c1;nat=yes>,<sip:31.223.168.234;r2=on;lr;ftag=as0b87791a;did=3e4.b5c1;nat=ye"...,
len = 6727432}, transport_val = {s = 0x8000000 <Address 0x8000000 out of
bounds>, len = 129942712}, ttl_val = {
s = 0x36efb0 <Address 0x36efb0 out of bounds>, len =
4275016}, user_param_val = {s = 0x48a0f8 "\207\231", len = 8}, maddr_val
= {s = 0x15b <Address 0x15b out of bounds>,
len = 232}, method_val = {s = 0xe8 <Address 0xe8 out of
bounds>, len = 6344353}, lr_val = {s = 0x5007fcedce08e40 <Address
0x5007fcedce08e40 out of bounds>,
len = -1275010320}, r2_val = {s = 0x7fcead5b9bc0 "\001", len
= 1}, gr_val = {s = 0x0, len = -1276371820}}
u = <optimized out>
port = <optimized out>
dst_host = <optimized out>
i = 21
flags = <optimized out>
avp = <optimized out>
st = {flags = 180, id = 0, name = {n = -722431032, s = {s =
0x7fced4f093c8 "Z\025", len = -1314950320}, re = 0x7fced4f093c8}, avp =
0x7ffeb0b3ffc0}
sct = <optimized out>
sjt = <optimized out>
rve = <optimized out>
mct = <optimized out>
rv = 0xa36781
rv1 = <optimized out>
c1 = {cache_type = RV_CACHE_EMPTY, val_type = RV_NONE, c =
{avp_val = {n = 10708645, s = {
s = 0xa366a5
"0.233:5060;branch=z9hG4bK1fbee766;rport\r\nRoute:
<sip:192.168.10.230;r2=on;lr;ftag=as0b87791a;did=3e4.b5c1;nat=yes>,<sip:31.223.168.234;r2=on;lr;ftag=as0b87791a;did=3e4.b5c1;nat=yes>\r\nMax-Forwards:
69\r"..., len = 0}, re = 0xa366a5}, pval = {rs = {
s = 0xa366a5
"0.233:5060;branch=z9hG4bK1fbee766;rport\r\nRoute:
<sip:192.168.10.230;r2=on;lr;ftag=as0b87791a;did=3e4.b5c1;nat=yes>,<sip:31.223.168.234;r2=on;lr;ftag=as0b87791a;did=3e4.b5c1;nat=yes>\r\nMax-Forwards:
69\r"..., len = 0}, ri = 3, flags = 29}}, i2s =
"\000\000\000\000\316\177\000\000t\000\000\000\316\177\000\000\000\000\000\000\000"}
s = {s = 0x7ffeb0b3ffc0 "", len = -726278104}
srevp = {0x7fced4f093c8, 0xc8}
mod_f_params = {{type = NOSUBTYPE, u = {number = 0, string =
0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}},
{type = NOSUBTYPE, u = {number = 0,
---Type <return> to continue, or q <return> to quit---
string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr =
0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0,
str = {s = 0x0, len = 0}, data = 0x0,
attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number
= 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0,
select = 0x0}}, {type = NOSUBTYPE, u = {
number = 0, string = 0x0, str = {s = 0x0, len = 0}, data =
0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0,
string = 0x0, str = {s = 0x0, len = 0},
data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE,
u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0,
attr = 0x0, select = 0x0}}, {
type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s =
0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}}
__FUNCTION__ = "do_action"
#15 0x000000000042a10a in run_actions (h=h@entry=0x7ffeb0b402b0,
a=a@entry=0x7fced4ac1698, msg=msg@entry=0x7fced4f093c8) at action.c:1549
t = <optimized out>
ret = <optimized out>
mod = <optimized out>
ms = 0
__FUNCTION__ = "run_actions"
#16 0x00000000004375d0 in run_top_route (a=0x7fced4ac1698,
msg=msg@entry=0x7fced4f093c8, c=c@entry=0x0) at action.c:1635
ctx = {rec_lev = 3, run_flags = 0, last_retcode = 1, jmp_env =
{{__jmpbuf = {140731863007920, 2156258125485971644, 140526308038296,
140526312526792, 0, 0,
-2156678592105015108, 2156258654672656572},
__mask_was_saved = 0, __saved_mask = {__val = {10708544, 617, 0, 0,
6452880, 140526311250576, 7138637, 3200,
140526312526792, 0, 140526311260392, 140731863008128,
140526312441400, 1073741825, 140525724418962, 140526312427128}}}}}
p = 0x7ffeb0b402b0
ret = <optimized out>
sfbk = 0
#17 0x0000000000504386 in receive_msg (buf=<optimized out>,
len=<optimized out>, rcv_info=<optimized out>) at receive.c:240
msg = 0x7fced4f093c8
ctx = {rec_lev = 134217728, run_flags = 0, last_retcode =
129951416, jmp_env = {{__jmpbuf = {3595920, 4266312, 140526442034080,
140526452292352, 297, 4294967295, 32, 5},
__mask_was_saved = 10031968, __saved_mask = {__val = {0,
0, 0, 0, 140526450104770, 140522739990533, 0, 140525758673168,
140526442034080, 128, 140526311259528,
140526312404216, 140526452242088, 140731863008806,
140526450129445, 140526452242088}}}}}
ret = <optimized out>
inb = {
s = 0xa36640 "BYE
sip:signetOLD@31.223.168.251:5060;alias=31.223.168.251~5060~1
SIP/2.0\r\nVia: SIP/2.0/UDP
192.168.10.233:5060;branch=z9hG4bK1fbee766;rport\r\nRoute:
<sip:192.168.10.230;r2=on;lr;ftag=as0b87791a;did=3e"..., len = 617}
netinfo = {data = {s = 0x1 <Address 0x1 out of bounds>, len =
-1330379296}, dst = 0x7f, rcv = 0x7fceb1e34891}
__FUNCTION__ = "receive_msg"
#18 0x00000000005f5bd4 in udp_rcv_loop () at udp_server.c:495
len = 617
buf = "BYE
sip:signetOLD@31.223.168.251:5060;alias=31.223.168.251~5060~1
SIP/2.0\r\nVia: SIP/2.0/UDP
192.168.10.233:5060;branch=z9hG4bK1fbee766;rport\r\nRoute:
<sip:192.168.10.230;r2=on;lr;ftag=as0b87791a;did=3e"...
from = 0x7fced4dd3d88
fromlen = 16
ri = {src_ip = {af = 2, len = 4, u = {addrl = {140526649780416,
5}, addr32 = {3909789888, 32718, 5, 0}, addr16 = {43200, 59658, 32718,
0, 5, 0, 0, 0},
addr =
"\300\250\n\351\316\177\000\000\005\000\000\000\000\000\000"}}, dst_ip =
{af = 2, len = 4, u = {addrl = {3859458240, 0}, addr32 = {3859458240, 0,
0, 0},
addr16 = {43200, 58890, 0, 0, 0, 0, 0, 0}, addr =
"\300\250\n\346", '\000' <repeats 11 times>}}, src_port = 5060,
dst_port
= 5060, proto_reserved1 = 0,
proto_reserved2 = 0, src_su = {s = {sa_family = 2, sa_data =
"\023\304\300\250\n\351\000\000\000\000\000\000\000"}, sin = {sin_family
= 2, sin_port = 50195, sin_addr = {
s_addr = 3909789888}, sin_zero =
"\000\000\000\000\000\000\000"}, sin6 = {sin6_family = 2, sin6_port =
50195, sin6_flowinfo = 3909789888, sin6_addr = {__in6_u = {
__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 =
{0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id =
0}}, bind_address = 0x7fced4eeb4f8,
proto = 1 '\001'}
__FUNCTION__ = "udp_rcv_loop"
#19 0x00000000004b2625 in main_loop () at main.c:1600
i = <optimized out>
pid = <optimized out>
si = <optimized out>
si_desc = "udp receiver child=1 sock=192.168.10.230:5060",
'\000' <repeats 11 times>, "\003", '\000' <repeats 15
times>, "\004",
'\000' <repeats 16 times>,
"\034\275\303\035\202\340Y\000\000\000\000\000\000\000\000\000\034\275\303\035\202\340Y\350\341\244\000\000\000\000\000\004\000\000\000\000\000\000"
---Type <return> to continue, or q <return> to quit---
nrprocs = <optimized out>
woneinit = <optimized out>
__FUNCTION__ = "main_loop"
#20 0x0000000000427e2b in main (argc=<optimized out>, argv=<optimized
out>) at main.c:2616
cfg_stream = <optimized out>
c = <optimized out>
r = <optimized out>
tmp = 0x7ffeb0b41d7b ""
tmp_len = 0
port = 1
proto = 1
options = 0x6e8c70
":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:x:X:"
ret = -1
seed = 690501560
rfd = 4
debug_save = <optimized out>
debug_flag = <optimized out>
dont_fork_cnt = <optimized out>
n_lst = 0x4
p = <optimized out>
st = {st_dev = 14, st_ino = 10524, st_nlink = 2, st_mode =
16877, st_uid = 107, st_gid = 112, __pad0 = 0, st_rdev = 0, st_size =
60, st_blksize = 4096, st_blocks = 0,
st_atim = {tv_sec = 1468460663, tv_nsec = 740929866}, st_mtim
= {tv_sec = 1468531732, tv_nsec = 898509779}, st_ctim = {tv_sec =
1468531739, tv_nsec = 982096020},
__unused = {0, 0, 0}}
__FUNCTION__ = "main"
Second crash:
(gdb) info locals
No locals.
(gdb) bt full
#0 atomic_get (v=0x7f6264d11378) at ../../mem/../atomic/atomic_common.h:74
No locals.
#1 dlg_unref (dlg=dlg@entry=0x7f585c494b40, cnt=cnt@entry=1) at
dlg_hash.c:921
mypid = 12376
d_entry = 0x7f6264d11360
__FUNCTION__ = "dlg_unref"
#2 0x00007f5855912802 in dlg_run_event_route
(dlg=dlg@entry=0x7f585c494b40, msg=msg@entry=0x7f587d4be8e8,
ostate=<optimized out>, nstate=<optimized out>) at dlg_handlers.c:1630
fmsg = 0x7f587d4be8e8
rt = 3
bkroute = 1
__FUNCTION__ = "dlg_run_event_route"
#3 0x00007f585591416a in dlg_onroute (req=0x7f587d4be8e8,
route_params=<optimized out>, param=<optimized out>) at dlg_handlers.c:1307
dlg = 0x7f585c494b40
iuid = <optimized out>
val = {
s = 0xa36706
"6ad.54d;nat=yes>,<sip:31.223.168.234;r2=on;lr;ftag=as5d3a5a72;did=6ad.54d;nat=yes>\r\nMax-Forwards:
69\r\nFrom: <sip:+31492382120@31.223.168.234>;tag=as6035a2af\r\nTo:
+31499847821 <sip:signetOLD@sip.wlvoip"..., len = 7}
callid = {
s = 0xa367ed
"1201b1b5692a911a1aa7e060399ad825(a)sip.wlvoip.net\r\nCSeqnCSeq: 102
BYE\r\nUser-Agent: sip.wlvoip.net\r\nX-Asterisk-HangupCause: Normal
Clearing\r\nX-Asterisk-HangupCauseCode: 16\r\nContent-Length:
0\r\n\r\n", len = 47}
ftag = {
s = 0xa36798 "as6035a2af\r\nTo: +31499847821
<sip:signetOLD@sip.wlvoip.net>;tag=as5d3a5a72\r\nCall-ID:
1201b1b5692a911a1aa7e060399ad825(a)sip.wlvoip.net\r\nCSeq: 102
BYE\r\nUser-Agent: sip.wlvoip.net\r\nX-Asterisk-HangupCause:"..., len = 10}
ttag = {
s = 0xa367d8 "as5d3a5a72\r\nCall-ID:
1201b1b5692a911a1aa7e060399ad825(a)sip.wlvoip.net\r\nCSeq: 102
BYE\r\nUser-Agent: sip.wlvoip.net\r\nX-Asterisk-HangupCause: Normal
Clearing\r\nX-Asterisk-HangupCauseCode: 16\r\nContent-Length"..., len = 10}
h_entry = 3494
h_id = 3397
new_state = 5
old_state = 4
unref = <optimized out>
event = 7
timeout = 5
reset = <optimized out>
dir = 2
ret = 0
__FUNCTION__ = "dlg_onroute"
#4 0x00007f585965b0e2 in run_rr_callbacks
(req=req@entry=0x7f587d4be8e8, rr_param=rr_param@entry=0x7f58598677a0)
at rr_cb.c:96
l_param = {
s = 0xa366e9
"r2=on;lr;ftag=as5d3a5a72;did=6ad.54d;nat=yes>,<sip:31.223.168.234;r2=on;lr;ftag=as5d3a5a72;did=6ad.54d;nat=yes>\r\nMax-Forwards:
69\r\nFrom: <sip:+31492382120@31.223.168.234>;tag=as6035a2af\r\nTo:
+31499847"..., len = 44}
cbp = 0x7f587d06c0a0
__FUNCTION__ = "run_rr_callbacks"
#5 0x00007f58596452c5 in after_loose (_m=0x7f587d4be8e8, preloaded=0)
at loose.c:919
hdr = <optimized out>
puri = {user = {s = 0x0, len = 0}, passwd = {s = 0x0, len = 0},
host = {
s = 0xa3671c
"31.223.168.234;r2=on;lr;ftag=as5d3a5a72;did=6ad.54d;nat=yes>\r\nMax-Forwards:
69\r\nFrom: <sip:+31492382120@31.223.168.234>;tag=as6035a2af\r\nTo:
+31499847821 <sip:signetOLD@sip.wlvoip.net>;tag=as5d3a5a72\r\n"..., len
= 14}, port = {s = 0x0, len = 0}, params = {
s = 0xa3672b
"r2=on;lr;ftag=as5d3a5a72;did=6ad.54d;nat=yes>\r\nMax-Forwards:
69\r\nFrom: <sip:+31492382120@31.223.168.234>;tag=as6035a2af\r\nTo:
+31499847821 <sip:signetOLD@sip.wlvoip.net>;tag=as5d3a5a72\r\nCall-ID:
1201b1"..., len = 44}, sip_params = {
---Type <return> to continue, or q <return> to quit---
s = 0xa3672b
"r2=on;lr;ftag=as5d3a5a72;did=6ad.54d;nat=yes>\r\nMax-Forwards:
69\r\nFrom: <sip:+31492382120@31.223.168.234>;tag=as6035a2af\r\nTo:
+31499847821 <sip:signetOLD@sip.wlvoip.net>;tag=as5d3a5a72\r\nCall-ID:
1201b1"..., len = 44}, headers = {s = 0x0, len = 0}, port_no = 0, proto
= 0, type = SIP_URI_T, flags = 0, transport = {s = 0x0,
len = 0}, ttl = {s = 0x0, len = 0}, user_param = {s = 0x0,
len = 0}, maddr = {s = 0x0, len = 0}, method = {s = 0x0, len = 0}, lr = {
s = 0xa36731
"lr;ftag=as5d3a5a72;did=6ad.54d;nat=yes>\r\nMax-Forwards: 69\r\nFrom:
<sip:+31492382120@31.223.168.234>;tag=as6035a2af\r\nTo: +31499847821
<sip:signetOLD@sip.wlvoip.net>;tag=as5d3a5a72\r\nCall-ID:
1201b1b5692a"..., len = 2}, r2 = {
s = 0xa3672b
"r2=on;lr;ftag=as5d3a5a72;did=6ad.54d;nat=yes>\r\nMax-Forwards:
69\r\nFrom: <sip:+31492382120@31.223.168.234>;tag=as6035a2af\r\nTo:
+31499847821 <sip:signetOLD@sip.wlvoip.net>;tag=as5d3a5a72\r\nCall-ID:
1201b1"..., len = 5}, gr = {s = 0x0, len = 0}, transport_val = {s = 0x0,
len = 0}, ttl_val = {s = 0x0, len = 0}, user_param_val = {
s = 0x0, len = 0}, maddr_val = {s = 0x0, len = 0},
method_val = {s = 0x0, len = 0}, lr_val = {s = 0x0, len = 0}, r2_val = {
s = 0xa3672e
"on;lr;ftag=as5d3a5a72;did=6ad.54d;nat=yes>\r\nMax-Forwards: 69\r\nFrom:
<sip:+31492382120@31.223.168.234>;tag=as6035a2af\r\nTo: +31499847821
<sip:signetOLD@sip.wlvoip.net>;tag=as5d3a5a72\r\nCall-ID: 1201b1b56"...,
len = 2}, gr_val = {s = 0x0, len = 0}}
rt = <optimized out>
status = <optimized out>
uri = {
s = 0xa366d6
"sip:192.168.10.230;r2=on;lr;ftag=as5d3a5a72;did=6ad.54d;nat=yes>,<sip:31.223.168.234;r2=on;lr;ftag=as5d3a5a72;did=6ad.54d;nat=yes>\r\nMax-Forwards:
69\r\nFrom: <sip:+31492382120@31.223.168.234>;tag=as6035"..., len = 63}
si = <optimized out>
uri_is_myself = 12376
use_ob = 0
__FUNCTION__ = "after_loose"
#6 0x000000000042b618 in do_action (h=h@entry=0x7ffd6e277fd0,
a=a@entry=0x7f587d264338, msg=msg@entry=0x7f587d4be8e8) at action.c:1060
ret = -5
v = <optimized out>
dst = {send_sock = 0x6e2789c0, to = {s = {sa_family = 33376,
sa_data = "'n\375\177\000\000\377\377\377\377X\177\000"}, sin =
{sin_family = 33376, sin_port = 28199,
sin_addr = {s_addr = 32765}, sin_zero =
"\377\377\377\377X\177\000"}, sin6 = {sin6_family = 33376, sin6_port =
28199, sin6_flowinfo = 32765, sin6_addr = {__in6_u = {
__u6_addr8 =
"\377\377\377\377X\177\000\000DuC\000\000\000\000", __u6_addr16 =
{65535, 65535, 32600, 0, 30020, 67, 0, 0}, __u6_addr32 = {4294967295,
32600,
4420932, 0}}}, sin6_scope_id = 0}}, id = 0, proto =
-20 '\354', send_flags = {f = 123 '{', blst_imask = 7 '\a'}}
tmp = <optimized out>
new_uri = <optimized out>
end = <optimized out>
crt = <optimized out>
cmd = 0x7f587d0575a8
len = <optimized out>
user = <optimized out>
uri = {user = {s = 0x7f587d4d41e0 "\001", len = 1481255020},
passwd = {s = 0x4ac0a0 "", len = 8}, host = {s = 0x1d6 <Address 0x1d6
out of bounds>, len = 2102216368}, port = {
s = 0x7f587d4d46b0 "6", len = 6334740}, params = {s = 0x40
<Address 0x40 out of bounds>, len = 6344353}, sip_params = {s =
0x7f587cc0a845 "%-d", len = 2102218424},
headers = {
s = 0xa367ba
"signetOLD(a)sip.wlvoip.net>;tag=as5d3a5a72\r\nCall-IDnCall-ID:
1201b1b5692a911a1aa7e060399ad825(a)sip.wlvoip.net\r\nCSeq: 102
BYE\r\nUser-Agent: sip.wlvoip.net\r\nX-Asterisk-HangupCause: Normal
Clearing\r\nX-Asterisk-Hangu"..., len = 9}, port_no = 0, proto = 0, type
= ERROR_URI_T, flags = 4294967295, transport = {s = 0x7f587d225270 "\001",
len = 2102126824}, ttl = {s = 0x7ffd6e279500 "\003", len =
1848082016}, user_param = {s = 0x7f587d4be8e8 "\353\022", len =
5514780}, maddr = {s = 0x0, len = 0},
method = {s = 0x0, len = 0}, lr = {s = 0x100000000 <Address
0x100000000 out of bounds>, len = 0}, r2 = {s = 0x0, len = 0}, gr = {s =
0x0, len = 0}, transport_val = {
s = 0x0, len = 0}, ttl_val = {s = 0x0, len = 0},
user_param_val = {s = 0x0, len = 0}, maddr_val = {s = 0x0, len = 0},
method_val = {s = 0x0, len = 0}, lr_val = {s = 0x0,
len = -2027077120}, r2_val = {s = 0x0, len = 2099407760},
gr_val = {s = 0x7f587d4be8e8 "\353\022", len = 0}}
next_hop = {user = {s = 0xfffffffb <Address 0xfffffffb out of
bounds>, len = 1848087808}, passwd = {s = 0x7f587d4be8e8 "\353\022", len
= 4366602}, host = {s = 0x0, len = 0},
port = {s = 0x0, len = 0}, params = {s = 0x0, len = 0},
sip_params = {s = 0x0, len = 0}, headers = {s = 0x0, len = 1511},
port_no = 59624, proto = 32075, type = 32600,
flags = 7, transport = {s = 0xfffffffb <Address 0xfffffffb out
of bounds>, len = 2097548304}, ttl = {s = 0x7f587d4be8e8 "\353\022", len
= 1481220363}, user_param = {
s = 0x7f587d248fd0 "\n", len = 2102126824}, maddr = {s =
0x7ffd6e279500 "\003", len = 2099407760}, method = {s = 0x7f587d4be8e8
"\353\022", len = 1494951765}, lr = {
s = 0x0, len = 4}, r2 = {s = 0x7f5800000000 <Address
0x7f5800000000 out of bounds>, len = -2027077120}, gr = {s =
0x7f580000000a <Address 0x7f580000000a out of bounds>,
len = 2099408064}, transport_val = {s = 0x7f587d4be8e8
"\353\022", len = 1848087808}, ttl_val = {s = 0x1 <Address 0x1 out of
bounds>, len = 1848087808},
user_param_val = {s = 0x7f587d4be8e8 "\353\022", len =
4373746}, maddr_val = {s = 0x7f587d4be8e8 "\353\022", len = 4371992},
method_val = {
s = 0x70 <Address 0x70 out of bounds>, len = 1848082904},
lr_val = {s = 0x7f587cc08a7d "%s%.*s%s ", len = 0}, r2_val = {
---Type <return> to continue, or q <return> to quit---
s = 0x7ffd00000000 <Address 0x7ffd00000000 out of bounds>,
len = 1848084928}, gr_val = {s = 0x0, len = 2102126824}}
u = <optimized out>
port = <optimized out>
dst_host = <optimized out>
i = <optimized out>
flags = <optimized out>
avp = <optimized out>
st = {flags = 1848087808, id = 32765, name = {n = 5485288, s = {
s = 0x53b2e8
"\205\300A\017\237\300\061\300E\017\266\310D\211\v\203e\004\371\351
\377\377\377H\211\372H\215q\bH\215|$P\350?\244\001", len = 2102206808},
re = 0x53b2e8}, avp = 0x7f587d4d4410}
sct = <optimized out>
sjt = <optimized out>
rve = <optimized out>
mct = <optimized out>
rv = <optimized out>
rv1 = <optimized out>
c1 = {cache_type = 1848087184, val_type = 32765, c = {avp_val =
{n = 2099481144, s = {s = 0x7f587d238a38 "T", len = 1}, re =
0x7f587d238a38}, pval = {rs = {
s = 0x7f587d238a38 "T", len = 1}, ri = 1848083032, flags
= 32765}}, i2s =
"\001\000\000\000X\177\000\000\003\000\000\000\000\000\000\000\232\213\300|X\177"}
s = {s = 0x0, len = 789}
srevp = {0x7ffd00000000, 0x7ffd6e277e60}
mod_f_params = {{type = NOSUBTYPE, u = {number = 0, string =
0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}},
{type = NOSUBTYPE, u = {number = 0,
string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr =
0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0,
str = {s = 0x0, len = 0}, data = 0x0,
attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number
= 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0,
select = 0x0}}, {type = NOSUBTYPE, u = {
number = 0, string = 0x0, str = {s = 0x0, len = 0}, data =
0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0,
string = 0x0, str = {s = 0x0, len = 0},
data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE,
u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0,
attr = 0x0, select = 0x0}}, {
type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s =
0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}}
__FUNCTION__ = "do_action"
#7 0x000000000042a10a in run_actions (h=h@entry=0x7ffd6e277fd0,
a=0x7f587d264338, msg=0x7f587d4be8e8) at action.c:1549
t = <optimized out>
ret = <optimized out>
mod = <optimized out>
ms = 0
__FUNCTION__ = "run_actions"
#8 0x0000000000437544 in run_actions_safe (h=h@entry=0x7ffd6e279500,
a=<optimized out>, msg=<optimized out>) at action.c:1614
ctx = {rec_lev = 1, run_flags = 0, last_retcode = 0, jmp_env =
{{__jmpbuf = {140726451541248, 140440723803439876, 140018035976424,
140726451541248, 140726451536944,
140018035976424, -141812437538209020,
140441295872688900}, __mask_was_saved = 0, __saved_mask = {__val =
{40558768, 140726451527616, 140726451523472,
140724603453441, 140726451536080, 40465328,
18446744073709551615, 4294967291, 140017369510558, 140018169122472, 2,
4, 8589934592, 3690471437605877047, 0, 0}}}}}
ret = 12376
ign_on_break = 0
#9 0x000000000053b2e8 in rval_get_int (h=0x7ffd6e279500, msg=<optimized
out>, i=0x7ffd6e278430, rv=rv@entry=0x7f587d264d58,
cache=cache@entry=0x0) at rvalue.c:912
r_avp = <optimized out>
avp_val = {n = 40561024, s = {s = 0x26ae980 "\004", len =
-2062898852}, re = 0x26ae980}
pval = {rs = {s = 0x3000000018 <Address 0x3000000018 out of
bounds>, len = 0}, ri = 1848082944, flags = 32765}
tmp = {s = 0x0, len = -2062586549}
s = 0x0
r = <optimized out>
ret = 0
destroy_pval = 0
__FUNCTION__ = "rval_get_int"
#10 0x000000000054261c in rval_expr_eval_int (h=h@entry=0x7ffd6e279500,
msg=msg@entry=0x7f587d4be8e8, res=res@entry=0x7ffd6e278430,
rve=rve@entry=0x7f587d264d50) at rvalue.c:1910
---Type <return> to continue, or q <return> to quit---
i1 = <optimized out>
i2 = <optimized out>
ret = -1
c1 = {cache_type = RV_CACHE_EMPTY, val_type = RV_NONE, c =
{avp_val = {n = 0, s = {s = 0x0, len = 39701287}, re = 0x0}, pval = {rs
= {s = 0x0, len = 39701287},
ri = 1848083472, flags = 32765}}, i2s = "\001", '\000'
<repeats 15 times>"\327, \376\000\000\000"}
c2 = {cache_type = 1848087808, val_type = 32765, c = {avp_val =
{n = 2099408064, s = {s = 0x7f587d226cc0 "%", len = 2102126824}, re =
0x7f587d226cc0}, pval = {rs = {
s = 0x7f587d226cc0 "%", len = 2102126824}, ri =
2099548112, flags = 32600}}, i2s =
"\350\350K}X\177\000\000\003\000\000\000:\a==\000\000\000\000\000"}
rv1 = <optimized out>
rv2 = <optimized out>
__FUNCTION__ = "rval_expr_eval_int"
#11 0x000000000042bc91 in do_action (h=h@entry=0x7ffd6e279500,
a=a@entry=0x7f587d268f88, msg=msg@entry=0x7f587d4be8e8) at action.c:1030
ret = -5
v = <optimized out>
dst = {send_sock = 0x7f5855b69990, to = {s = {sa_family = 39280,
sa_data = "\266UX\177\000\000P\231\266UX\177\000"}, sin = {sin_family =
39280, sin_port = 21942, sin_addr = {
s_addr = 32600}, sin_zero = "P\231\266UX\177\000"}, sin6
= {sin6_family = 39280, sin6_port = 21942, sin6_flowinfo = 32600,
sin6_addr = {__in6_u = {
__u6_addr8 =
"P\231\266UX\177\000\000`\231\266UX\177\000", __u6_addr16 = {39248,
21942, 32600, 0, 39264, 21942, 32600, 0}, __u6_addr32 = {1438030160, 32600,
1438030176, 32600}}}, sin6_scope_id = 1438030096}},
id = 32600, proto = 32 ' ', send_flags = {f = 153 '\231', blst_imask =
182 '\266'}}
tmp = <optimized out>
new_uri = <optimized out>
end = <optimized out>
crt = <optimized out>
cmd = <optimized out>
len = <optimized out>
user = <optimized out>
uri = {user = {s = 0x7f587d4be8e8 "\353\022", len = 5354102},
passwd = {s = 0x0, len = 0}, host = {
s = 0x25dca00 "update `dialog` set
`state`=4,`timeout`=1468582942,`caller_cseq`='3',`callee_cseq`='0',`caller_contact`='sip:dehaan01@176.74.254.179;alias=176.74.254.179~5060~1',`callee_contact`='sip:+31641157337@192"...,
len = 2101683296}, port = {s = 0x7f587d452760 "", len = 1537539707},
params = {s = 0x7f587ce0e210 "", len = 2101683296},
sip_params = {s = 0x7f587ce0e210 "", len = 1537544586},
headers = {
s = 0x25dca00 "update `dialog` set
`state`=4,`timeout`=1468582942,`caller_cseq`='3',`callee_cseq`='0',`caller_contact`='sip:dehaan01@176.74.254.179;alias=176.74.254.179~5060~1',`callee_contact`='sip:+31641157337@192"...,
len = 40507338}, port_no = 9312, proto = 32069, type = 32600, flags =
2095112720, transport = {s = 0x7ffd6e2785a0 "",
len = 2099659072}, ttl = {s = 0x7f587d4be8e8 "\353\022", len
= 2102126824}, user_param = {s = 0x7ffd6e279500 "\003", len = 0}, maddr
= {s = 0x7f587d4be8e8 "\353\022",
len = 5514780}, method = {s = 0x111 <Address 0x111 out of
bounds>, len = 1848087808}, lr = {s = 0x0, len = 2102126824}, r2 = {s =
0x7ffd6e279500 "\003",
len = 1848084992}, gr = {s = 0x7f587d4be8e8 "\353\022", len
= 4420932}, transport_val = {s = 0x0, len = 1}, ttl_val = {s =
0x7ffd6e279500 "\003", len = 508396292},
user_param_val = {s = 0x7f587d4be8e8 "\353\022", len =
1848087808}, maddr_val = {s = 0x7ffd6e278a00 "\001", len = 2102126824},
method_val = {
s = 0xfe082e57176d8304 <Address 0xfe082e57176d8304 out of
bounds>, len = 1346994948}, lr_val = {s = 0x0, len = 1435889581}, r2_val
= {s = 0x7f585bf8cf46 "`",
len = 5060}, gr_val = {s = 0x2 <Address 0x2 out of bounds>,
len = 0}}
next_hop = {user = {s = 0x7f585c490ad0 "\002", len = 0}, passwd
= {s = 0x7ffdfffffffb <Address 0x7ffdfffffffb out of bounds>, len =
10042548}, host = {
s = 0x7f5800000016 <Address 0x7f5800000016 out of bounds>,
len = 1435501873}, port = {s = 0x7f585bedf640 "\002", len =
-2027077120}, params = {s = 0x7f585c490ad0 "\002",
len = 1848084992}, sip_params = {s = 0x7ffd6e279500 "\003",
len = 2102126824}, headers = {s = 0x7ffd6e279500 "\003", len =
1848084992}, port_no = 38144, proto = 28199,
type = 32765, flags = 5485288, transport = {s = 0x3 <Address
0x3 out of bounds>, len = 1435534751}, ttl = {
s = 0xa3664c
"etOLD@31.223.168.251:5060;alias=31.223.168.251~5060~1 SIP/2.0\r\nVia:
SIP/2.0/UDP 192.168.10.232:5060;branch=z9hG4bK7b052368;rport\r\nRoute:
<sip:192.168.10.230;r2=on;lr;ftag=as5d3a5a72;did=6ad.54d;nat=ye"..., len
= 2093000776}, user_param = {s = 0x0, len = 1}, maddr = {s = 0x13c4
<Address 0x13c4 out of bounds>, len = 0}, method = {
s = 0x0, len = 5060}, lr = {s = 0x7f5800000000 <Address
0x7f5800000000 out of bounds>, len = 1514031249}, r2 = {s = 0x8000000
<Address 0x8000000 out of bounds>,
len = 129902608}, gr = {s = 0x375da8 <Address 0x375da8 out
of bounds>, len = -2027077120}, transport_val = {s = 0x0, len = -1},
ttl_val = {s = 0x7f587d264598 "\001",
len = 2102126824}, user_param_val = {s = 0x7ffd6e279500
"\003", len = 1848084992}, maddr_val = {s = 0x7f587d4be8e8
"\353\022",
len = 5514780}, method_val = {
s = 0x7f5855b69a00 "mu\225UX\177", len = 1848085248}, lr_val
= {s = 0x7ffd6e279290 "\006", len = 2102126824}, r2_val = {s =
0x7ffd6e279290 "\006", len = 1848085248},
gr_val = {s = 0x7f587d4be8e8 "\353\022", len = 5485239}}
u = <optimized out>
port = <optimized out>
dst_host = <optimized out>
---Type <return> to continue, or q <return> to quit---
i = <optimized out>
flags = <optimized out>
avp = <optimized out>
st = {flags = 2102126824, id = 32600, name = {n = 4366602, s =
{s = 0x42a10a "H\213\r\177\201V", len = 1848087808}, re = 0x42a10a}, avp
= 0x53b2e8}
sct = <optimized out>
sjt = <optimized out>
rve = 0x7f587d264d50
mct = <optimized out>
rv = <optimized out>
rv1 = <optimized out>
c1 = {cache_type = RV_CACHE_PVAR, val_type = 32600, c = {avp_val
= {n = 1438030144, s = {s = 0x7f5855b69940 ")u\225UX\177", len =
1438030048}, re = 0x7f5855b69940}, pval = {
rs = {s = 0x7f5855b69940 ")u\225UX\177", len =
1438030048}, ri = 1438030016, flags = 32600}}, i2s =
"\260\230\266UX\177\000\000\240\230\266UX\177\000\000И\266UX\177"}
s = {s = 0x0, len = 0}
srevp = {0xfffffffb, 0x7ffd6e2785a0}
mod_f_params = {{type = NOSUBTYPE, u = {number = 0, string =
0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}},
{type = NOSUBTYPE, u = {number = 0,
string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr =
0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0,
str = {s = 0x0, len = 0}, data = 0x0,
attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number
= 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0,
select = 0x0}}, {type = NOSUBTYPE, u = {
number = 0, string = 0x0, str = {s = 0x0, len = 0}, data =
0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0,
string = 0x0, str = {s = 0x0, len = 0},
data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE,
u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0,
attr = 0x0, select = 0x0}}, {
type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s =
0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}}
__FUNCTION__ = "do_action"
#12 0x000000000042a10a in run_actions (h=h@entry=0x7ffd6e279500,
a=0x7f587d268f88, msg=msg@entry=0x7f587d4be8e8) at action.c:1549
t = <optimized out>
ret = <optimized out>
mod = <optimized out>
ms = 0
__FUNCTION__ = "run_actions"
#13 0x000000000042bcf2 in do_action (h=h@entry=0x7ffd6e279500,
a=a@entry=0x7f587d2691e8, msg=msg@entry=0x7f587d4be8e8) at action.c:1049
ret = 1
v = 1
dst = {send_sock = 0x6, to = {s = {sa_family = 12576, sa_data =
"M}X\177\000\000\254h\243\000\000\000\000"}, sin = {sin_family = 12576,
sin_port = 32077, sin_addr = {
s_addr = 32600}, sin_zero =
"\254h\243\000\000\000\000"}, sin6 = {sin6_family = 12576, sin6_port =
32077, sin6_flowinfo = 32600, sin6_addr = {__in6_u = {
__u6_addr8 =
"\254h\243\000\000\000\000\000d\000\000\000\000\000\000", __u6_addr16 =
{26796, 163, 0, 0, 100, 0, 0, 0}, __u6_addr32 = {10709164, 0, 100, 0}}},
sin6_scope_id = 10708691}}, id = 0, proto = -111 '\221',
send_flags = {f = 72 'H', blst_imask = 62 '>'}}
tmp = <optimized out>
new_uri = <optimized out>
end = <optimized out>
crt = <optimized out>
cmd = <optimized out>
len = <optimized out>
user = <optimized out>
uri = {user = {s = 0x100000004 <Address 0x100000004 out of
bounds>, len = 131072}, passwd = {s = 0x0, len = 0}, host = {s = 0x4
<Address 0x4 out of bounds>,
len = 1548290768}, port = {s = 0x7f585c490ca2
"sip:+31263250863@31.223.168.2340\r\n\r\n60\020", len = 31}, params = {s
= 0x100000004 <Address 0x100000004 out of bounds>,
len = 1435660958}, sip_params = {s = 0x0, len = 1435673883},
headers = {s = 0x0, len = 25}, port_no = 3, proto = 0, type = 32765,
flags = URI_USER_NORMALIZE,
transport = {s = 0x7ffd6e278b70 "", len = 2097617544}, ttl =
{s = 0x7f587d4be8e8 "\353\022", len = 12376}, user_param = {s =
0x7f585c47e774 "", len = 0}, maddr = {
s = 0x7f587d4d4410 "\003", len = 1435538987}, method = {s =
0x0, len = 1848087808}, lr = {s = 0x0, len = 2102126824}, r2 = {s =
0x7ffd6e279500 "\003", len = 1848086480},
gr = {s = 0x7f587d4be8e8 "\353\022", len = 4420932},
transport_val = {s = 0x0, len = -1}, ttl_val = {s = 0x7ffd6e279500
"\003", len = 508396292}, user_param_val = {
s = 0x7f587d4be8e8 "\353\022", len = 1848087808}, maddr_val
= {s = 0x7ffd6e278fd0 "", len = 2102126824}, method_val = {
s = 0xfe082e570a8d8304 <Address 0xfe082e570a8d8304 out of
bounds>, len = 1346994948}, lr_val = {s = 0x0, len = 56}, r2_val = {s =
0x38 <Address 0x38 out of bounds>,
len = 6344353}, gr_val = {s = 0x7f5800000000 <Address
0x7f5800000000 out of bounds>, len = 1509514912}}
---Type <return> to continue, or q <return> to quit---
next_hop = {user = {s = 0x7f5859f95cc0 "\020\002", len = 12396},
passwd = {s = 0x7f585c496fb8 "\002", len = 32860}, host = {s =
0x7f585c693640 "\220\232>\\X\177",
len = 1848086288}, port = {s = 0x2 <Address 0x2 out of
bounds>, len = 1435721985}, params = {s = 0x8000000 <Address 0x8000000
out of bounds>, len = 129905640},
sip_params = {s = 0x3751d0 <Address 0x3751d0 out of bounds>,
len = 4312088}, headers = {s = 0x4ac0a0 "", len = 1848086480}, port_no =
38144, proto = 28199, type = 32765,
flags = 5485288, transport = {s = 0x1c8 <Address 0x1c8 out of
bounds>, len = 1514031249}, ttl = {s = 0x8000000 <Address 0x8000000 out
of bounds>, len = 1548316600},
user_param = {s = 0x7f587d4be8e8 "\353\022", len =
1548965320}, maddr = {s = 0x7f585c693640 "\220\232>\\X\177", len =
1848086288}, method = {
s = 0x2 <Address 0x2 out of bounds>, len = 1435607732}, lr =
{s = 0x8000000 <Address 0x8000000 out of bounds>, len = 129906608}, r2 = {
s = 0x374e08 <Address 0x374e08 out of bounds>, len =
4311120}, gr = {s = 0x4ac0a0 "", len = 8}, transport_val = {s = 0x1da
<Address 0x1da out of bounds>, len = -1},
ttl_val = {s = 0x7f587d071c80 "\001", len = 2102126824},
user_param_val = {s = 0x7ffd6e279500 "\003", len = 1848086480},
maddr_val = {s = 0x7f587d4be8e8 "\353\022",
len = 5514780}, method_val = {s = 0x7f587d4d2158 "\001", len
= 449}, lr_val = {s = 0x7f585c5914c8 "\270\020J}X\177", len =
1506954664}, r2_val = {
s = 0x1 <Address 0x1 out of bounds>, len = 2510}, gr_val =
{s = 0x30 <Address 0x30 out of bounds>, len = 2102216912}}
u = <optimized out>
port = <optimized out>
dst_host = <optimized out>
i = <optimized out>
flags = <optimized out>
avp = <optimized out>
st = {flags = 2102126824, id = 32600, name = {n = 4366602, s =
{s = 0x42a10a "H\213\r\177\201V", len = 0}, re = 0x42a10a}, avp = 0x5aaa8a}
sct = <optimized out>
sjt = <optimized out>
rve = <optimized out>
mct = <optimized out>
rv = <optimized out>
rv1 = <optimized out>
c1 = {cache_type = 134217728, val_type = RV_NONE, c = {avp_val =
{n = 129906432, s = {s = 0x7be3700 <Address 0x7be3700 out of bounds>,
len = 0}, re = 0x7be3700}, pval = {
rs = {s = 0x7be3700 <Address 0x7be3700 out of bounds>, len
= 0}, ri = 4311296, flags = 0}}, i2s =
"\240\300J\000\000\000\000\000\b", '\000' <repeats 12 times>}
s = {s = 0x0, len = 0}
srevp = {0xfffffffb, 0x7ffd6e278b70}
mod_f_params = {{type = NOSUBTYPE, u = {number = 0, string =
0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}},
{type = NOSUBTYPE, u = {number = 0,
string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr =
0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0,
str = {s = 0x0, len = 0}, data = 0x0,
attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number
= 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0,
select = 0x0}}, {type = NOSUBTYPE, u = {
number = 0, string = 0x0, str = {s = 0x0, len = 0}, data =
0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0,
string = 0x0, str = {s = 0x0, len = 0},
data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE,
u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0,
attr = 0x0, select = 0x0}}, {
type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s =
0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}}
__FUNCTION__ = "do_action"
#14 0x000000000042a10a in run_actions (h=h@entry=0x7ffd6e279500,
a=0x7f587d263f48, msg=msg@entry=0x7f587d4be8e8) at action.c:1549
t = <optimized out>
ret = <optimized out>
mod = <optimized out>
ms = 0
__FUNCTION__ = "run_actions"
#15 0x000000000042bde0 in do_action (h=h@entry=0x7ffd6e279500,
a=a@entry=0x7f587d073d70, msg=msg@entry=0x7f587d4be8e8) at action.c:678
ret = -5
v = <optimized out>
dst = {send_sock = 0x1, to = {s = {sa_family = 20288, sa_data =
"M}X\177\000\000\200f\243\000\000\000\000"}, sin = {sin_family = 20288,
sin_port = 32077, sin_addr = {
s_addr = 32600}, sin_zero =
"\200f\243\000\000\000\000"}, sin6 = {sin6_family = 20288, sin6_port =
32077, sin6_flowinfo = 32600, sin6_addr = {__in6_u = {
__u6_addr8 = "\200f\243", '\000' <repeats 12
times>,
__u6_addr16 = {26240, 163, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {10708608,
0, 0, 0}}},
sin6_scope_id = 2102217448}}, id = 32600, proto = -57
'\307', send_flags = {f = 102 'f', blst_imask = 163 '\243'}}
tmp = <optimized out>
new_uri = <optimized out>
end = <optimized out>
---Type <return> to continue, or q <return> to quit---
crt = <optimized out>
cmd = <optimized out>
len = <optimized out>
user = <optimized out>
uri = {user = {s = 0x7f587d4d4628 "\004", len = 1509499248},
passwd = {s = 0x7f587d4aa8d8 "\020\325\332UX\177", len = 1506803058},
host = {s = 0x7f587d4be8e8 "\353\022",
len = 2101683296}, port = {s = 0x7f587d452760 "", len =
1537539707}, params = {s = 0x7f587ce0e210 "", len = 1549665520},
sip_params = {s = 0x7f585c5e06f8 "",
len = 1506810279}, headers = {s = 0x7f585c5e04f0
"\320m?\\X\177", len = 404}, port_no = 0, proto = 0, type = ERROR_URI_T,
flags = 1506664286, transport = {
s = 0x7f585c195388 "\001", len = 1537548094}, ttl = {s =
0x7f587d4d54b8 "\220\230\266UX\177", len = 1506961773}, user_param = {s
= 0x0, len = 2102219960}, maddr = {
s = 0x0, len = -2027077120}, method = {s = 0x7f587d4be8e8
"\353\022", len = 1}, lr = {s = 0x7f585c693848 "", len = 2102126824},
r2 = {
s = 0x780 <Address 0x780 out of bounds>, len = 1550399040},
gr = {s = 0x58554fe0 <Address 0x58554fe0 out of bounds>, len =
1506999306}, transport_val = {s = 0x0,
len = 1468572138}, ttl_val = {
s = 0x7f5885032978
"\227\201B\250|`\234|\374Qcq\\zy\360\003\346`\033|\231/\310Km\301\243\353\372\266=/\222\210\353",
<incomplete sequence \352>, len = 1542901968},
user_param_val = {s = 0x992290 "\314JO\\X\177", len = 68},
maddr_val = {s = 0x993cb4 "\377\377", len = 0}, method_val = {
s = 0x25dca00 "update `dialog` set
`state`=4,`timeout`=1468582942,`caller_cseq`='3',`callee_cseq`='0',`caller_contact`='sip:dehaan01@176.74.254.179;alias=176.74.254.179~5060~1',`callee_contact`='sip:+31641157337@192"...,
len = 2092914972}, lr_val = {s = 0x2 <Address 0x2 out of bounds>, len =
1514031249}, r2_val = {
s = 0x8000000 <Address 0x8000000 out of bounds>, len =
129907920}, gr_val = {s = 0x3748e8 <Address 0x3748e8 out of bounds>, len
= 4309808}}
next_hop = {user = {s = 0x4ac0a0 "", len = 8}, passwd = {s =
0x1e4 <Address 0x1e4 out of bounds>, len = 64}, host = {s = 0x40
<Address 0x40 out of bounds>, len = 6344353},
port = {s = 0xa466c8 "", len = 10774544}, params = {s =
0x4ac0a0 "", len = 8}, sip_params = {s = 0x0, len = 48}, headers = {s =
0x30 <Address 0x30 out of bounds>,
len = 6344353}, port_no = 0, proto = 0, type = 32600, flags
= 1548156928, transport = {s = 0x40 <Address 0x40 out of bounds>, len =
2102218560}, ttl = {
s = 0x30 <Address 0x30 out of bounds>, len = 2102214888},
user_param = {s = 0xe6a8df1faced0002 <Address 0xe6a8df1faced0002 out of
bounds>, len = 2102206808}, maddr = {
s = 0x7f587d4d2158 "\001", len = 2510}, method = {s = 0x30
<Address 0x30 out of bounds>, len = 10708679}, lr = {s = 0x6 <Address
0x6 out of bounds>, len = 2102217448},
r2 = {s = 0xa368a7 "", len = 100}, gr = {
s = 0xa36690 "SIP/2.0/UDP
192.168.10.232:5060;branch=z9hG4bK7b052368;rport\r\nRoute:
<sip:192.168.10.230;r2=on;lr;ftag=as5d3a5a72;did=6ad.54d;nat=yes>,<sip:31.223.168.234;r2=on;lr;ftag=as5d3a5a72;did=6ad.54d;nat=yes>"...,
len = 6727432}, transport_val = {s = 0x8000000 <Address 0x8000000 out of
bounds>, len = 129908048}, ttl_val = {
s = 0x374868 <Address 0x374868 out of bounds>, len =
4309680}, user_param_val = {s = 0x4ac0a0 "", len = 8}, maddr_val = {s =
0x1e6 <Address 0x1e6 out of bounds>,
len = 232}, method_val = {s = 0xe8 <Address 0xe8 out of
bounds>, len = 6344353}, lr_val = {s = 0x5007f587d4d2158 <Address
0x5007f587d4d2158 out of bounds>,
len = 6594876}, r2_val = {s = 0x7f5855b69bc0 "\001", len =
1}, gr_val = {s = 0x0, len = 1548202964}}
u = <optimized out>
port = <optimized out>
dst_host = <optimized out>
i = 21
flags = <optimized out>
avp = <optimized out>
st = {flags = 0, id = 0, name = {n = 4421082, s = {s = 0x4375da
"D\211\340H\213\234", <incomplete sequence \350>, len = 1509585744}, re
= 0x4375da}, avp = 0x7ffd6e279210}
sct = <optimized out>
sjt = <optimized out>
rve = <optimized out>
mct = <optimized out>
rv = 0xa367d7
rv1 = <optimized out>
c1 = {cache_type = RV_CACHE_EMPTY, val_type = RV_NONE, c =
{avp_val = {n = 10708645, s = {
s = 0xa366a5
"0.232:5060;branch=z9hG4bK7b052368;rport\r\nRoute:
<sip:192.168.10.230;r2=on;lr;ftag=as5d3a5a72;did=6ad.54d;nat=yes>,<sip:31.223.168.234;r2=on;lr;ftag=as5d3a5a72;did=6ad.54d;nat=yes>\r\nMax-Forwards:
69\r\nF"..., len = 0}, re = 0xa366a5}, pval = {rs = {
s = 0xa366a5
"0.232:5060;branch=z9hG4bK7b052368;rport\r\nRoute:
<sip:192.168.10.230;r2=on;lr;ftag=as5d3a5a72;did=6ad.54d;nat=yes>,<sip:31.223.168.234;r2=on;lr;ftag=as5d3a5a72;did=6ad.54d;nat=yes>\r\nMax-Forwards:
69\r\nF"..., len = 0}, ri = 3, flags = 29}}, i2s =
"\000\000\000\000X\177\000\000t\000\000\000X\177\000\000\000\000\000\000\000"}
s = {s = 0x0, len = 2098257960}
srevp = {0x7f587d4be8e8, 0x64}
mod_f_params = {{type = NOSUBTYPE, u = {number = 0, string =
0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}},
{type = NOSUBTYPE, u = {number = 0,
string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr =
0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0,
str = {s = 0x0, len = 0}, data = 0x0,
attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number
= 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0,
select = 0x0}}, {type = NOSUBTYPE, u = {
---Type <return> to continue, or q <return> to quit---
number = 0, string = 0x0, str = {s = 0x0, len = 0}, data =
0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0,
string = 0x0, str = {s = 0x0, len = 0},
data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE,
u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0,
attr = 0x0, select = 0x0}}, {
type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s =
0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}}
__FUNCTION__ = "do_action"
#16 0x000000000042a10a in run_actions (h=h@entry=0x7ffd6e279500,
a=a@entry=0x7f587d071698, msg=msg@entry=0x7f587d4be8e8) at action.c:1549
t = <optimized out>
ret = <optimized out>
mod = <optimized out>
ms = 0
__FUNCTION__ = "run_actions"
#17 0x00000000004375d0 in run_top_route (a=0x7f587d071698,
msg=msg@entry=0x7f587d4be8e8, c=c@entry=0x0) at action.c:1635
ctx = {rec_lev = 3, run_flags = 0, last_retcode = 1, jmp_env =
{{__jmpbuf = {140726451541248, 140440723803439876, 140018031466136,
140018035976424, 0, 0,
-141812436097465596, 140441295872688900},
__mask_was_saved = 0, __saved_mask = {__val = {10708544, 615, 0, 0,
6452880, 140018036064736, 7138637, 3152,
140018035976424, 0, 140018036061392, 140726451541456,
140018035891032, 1073741825, 140017447846802, 140018035876760}}}}}
p = 0x7ffd6e279500
ret = <optimized out>
sfbk = 0
#18 0x0000000000504386 in receive_msg (buf=<optimized out>,
len=<optimized out>, rcv_info=<optimized out>) at receive.c:240
msg = 0x7f587d4be8e8
ctx = {rec_lev = 134217728, run_flags = 0, last_retcode =
129916136, jmp_env = {{__jmpbuf = {3618528, 4301592, 140018165461920,
140018175721472, 446, 4294967295, 32, 5},
__mask_was_saved = 10031968, __saved_mask = {__val = {0,
1, 0, 0, 140018173532610, 140015933849605, 0, 140017482500740,
140018165461920, 128, 140018036068808,
140018035853848, 140018175719080, 140726451542134,
140018173557285, 140018175719080}}}}}
ret = <optimized out>
inb = {
s = 0xa36640 "BYE
sip:signetOLD@31.223.168.251:5060;alias=31.223.168.251~5060~1
SIP/2.0\r\nVia: SIP/2.0/UDP
192.168.10.232:5060;branch=z9hG4bK7b052368;rport\r\nRoute:
<sip:192.168.10.230;r2=on;lr;ftag=as5d3a5a72;did=6a"..., len = 615}
netinfo = {data = {s = 0x1 <Address 0x1 out of bounds>, len =
1848088624}, dst = 0x7f, rcv = 0x7f585a3e4891}
__FUNCTION__ = "receive_msg"
#19 0x00000000005f5bd4 in udp_rcv_loop () at udp_server.c:495
len = 615
buf = "BYE
sip:signetOLD@31.223.168.251:5060;alias=31.223.168.251~5060~1
SIP/2.0\r\nVia: SIP/2.0/UDP
192.168.10.232:5060;branch=z9hG4bK7b052368;rport\r\nRoute:
<sip:192.168.10.230;r2=on;lr;ftag=as5d3a5a72;did=6a"...
from = 0x7f587d4d51c8
fromlen = 16
ri = {src_ip = {af = 2, len = 4, u = {addrl = {140019826862272,
5}, addr32 = {3893012672, 32600, 5, 0}, addr16 = {43200, 59402, 32600,
0, 5, 0, 0, 0},
addr =
"\300\250\n\350X\177\000\000\005\000\000\000\000\000\000"}}, dst_ip =
{af = 2, len = 4, u = {addrl = {3859458240, 0}, addr32 = {3859458240, 0,
0, 0}, addr16 = {
43200, 58890, 0, 0, 0, 0, 0, 0}, addr =
"\300\250\n\346", '\000' <repeats 11 times>}}, src_port = 5060,
dst_port
= 5060, proto_reserved1 = 0, proto_reserved2 = 0,
src_su = {s = {sa_family = 2, sa_data =
"\023\304\300\250\n\350\000\000\000\000\000\000\000"}, sin = {sin_family
= 2, sin_port = 50195, sin_addr = {s_addr = 3893012672},
sin_zero = "\000\000\000\000\000\000\000"}, sin6 =
{sin6_family = 2, sin6_port = 50195, sin6_flowinfo = 3893012672,
sin6_addr = {__in6_u = {
__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 =
{0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id =
0}}, bind_address = 0x7f587d4a0a18,
proto = 1 '\001'}
__FUNCTION__ = "udp_rcv_loop"
#20 0x00000000004b2625 in main_loop () at main.c:1600
i = <optimized out>
pid = <optimized out>
si = <optimized out>
si_desc = "udp receiver child=0 sock=192.168.10.230:5060",
'\000' <repeats 11 times>, "\003", '\000' <repeats 15
times>, "\004",
'\000' <repeats 16 times>,
"B-\207:\a==\000\000\000\000\000\000\000\000\000B-\207:\a==\350\341\244\000\000\000\000\000\004\000\000\000\000\000\000"
nrprocs = <optimized out>
woneinit = <optimized out>
---Type <return> to continue, or q <return> to quit---
__FUNCTION__ = "main_loop"
#21 0x0000000000427e2b in main (argc=<optimized out>, argv=<optimized
out>) at main.c:2616
cfg_stream = <optimized out>
c = <optimized out>
r = <optimized out>
tmp = 0x7ffd6e279f80 ""
tmp_len = 0
port = 1
proto = 1
options = 0x6e8c70
":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:x:X:"
ret = -1
seed = 1372741379
rfd = 4
debug_save = <optimized out>
debug_flag = <optimized out>
dont_fork_cnt = <optimized out>
n_lst = 0x4
p = <optimized out>
st = {st_dev = 14, st_ino = 10524, st_nlink = 2, st_mode =
16877, st_uid = 107, st_gid = 112, __pad0 = 0, st_rdev = 0, st_size =
100, st_blksize = 4096, st_blocks = 0,
st_atim = {tv_sec = 1468547329, tv_nsec = 31440594}, st_mtim =
{tv_sec = 1468566559, tv_nsec = 440763330}, st_ctim = {tv_sec =
1468566560, tv_nsec = 524700180},
__unused = {0, 0, 0}}
__FUNCTION__ = "main"
(gdb)
Cheers,
Dirk
On 07/15/2016 12:40 PM, Daniel-Constantin Mierla wrote:
Hello,
can you send the output of gdb commands:
info locals
bt full
Cheers,
Daniel
On 15/07/16 10:06, Dirk Teurlings - Signet B.V. wrote:
Hi,
Running Kamailio on Debian from the Kamailio repository with 4.4.2
stable (unpatched). Getting some random segfaults with it now, here's
the relevant backtrace from the generated core.
Core was generated by `/usr/sbin/kamailio -f /etc/kamailio/kamailio.cfg
-P /var/run/kamailio/kamailio.'.
Program terminated with signal 11, Segmentation fault.
#0 run_dlg_callbacks (type=type@entry=64, dlg=dlg@entry=0x7fceb400e2f0,
req=req@entry=0x7fced4f093c8, rpl=rpl@entry=0x0, dir=<optimized out>,
dlg_data=dlg_data@entry=0x0) at dlg_cb.c:253
253 if ( (cb->types)&type ) {
(gdb) bt
#0 run_dlg_callbacks (type=type@entry=64, dlg=dlg@entry=0x7fceb400e2f0,
req=req@entry=0x7fced4f093c8, rpl=rpl@entry=0x0, dir=<optimized out>,
dlg_data=dlg_data@entry=0x0) at dlg_cb.c:253
#1 0x00007fcead3648f9 in dlg_terminated (dir=<optimized out>,
dlg=0x7fceb400e2f0, req=0x7fced4f093c8) at dlg_handlers.c:368
#2 dlg_onroute (req=0x7fced4f093c8, route_params=<optimized out>,
param=<optimized out>) at dlg_handlers.c:1354
#3 0x00007fceb10ab0e2 in run_rr_callbacks
(req=req@entry=0x7fced4f093c8, rr_param=rr_param@entry=0x7fceb12b77a0)
at rr_cb.c:96
#4 0x00007fceb10952c5 in after_loose (_m=0x7fced4f093c8, preloaded=0)
at loose.c:919
#5 0x000000000042b618 in do_action (h=h@entry=0x7ffeb0b3ed80,
a=a@entry=0x7fced4cb4338, msg=msg@entry=0x7fced4f093c8) at action.c:1060
#6 0x000000000042a10a in run_actions (h=h@entry=0x7ffeb0b3ed80,
a=0x7fced4cb4338, msg=0x7fced4f093c8) at action.c:1549
#7 0x0000000000437544 in run_actions_safe (h=h@entry=0x7ffeb0b402b0,
a=<optimized out>, msg=<optimized out>) at action.c:1614
#8 0x000000000053b2e8 in rval_get_int (h=0x7ffeb0b402b0, msg=<optimized
out>, i=0x7ffeb0b3f1e0, rv=rv@entry=0x7fced4cb4d58,
cache=cache@entry=0x0) at rvalue.c:912
#9 0x000000000054261c in rval_expr_eval_int (h=h@entry=0x7ffeb0b402b0,
msg=msg@entry=0x7fced4f093c8, res=res@entry=0x7ffeb0b3f1e0,
rve=rve@entry=0x7fced4cb4d50) at rvalue.c:1910
#10 0x000000000042bc91 in do_action (h=h@entry=0x7ffeb0b402b0,
a=a@entry=0x7fced4cb8f88, msg=msg@entry=0x7fced4f093c8) at action.c:1030
#11 0x000000000042a10a in run_actions (h=h@entry=0x7ffeb0b402b0,
a=0x7fced4cb8f88, msg=msg@entry=0x7fced4f093c8) at action.c:1549
#12 0x000000000042bcf2 in do_action (h=h@entry=0x7ffeb0b402b0,
a=a@entry=0x7fced4cb91e8, msg=msg@entry=0x7fced4f093c8) at action.c:1049
#13 0x000000000042a10a in run_actions (h=h@entry=0x7ffeb0b402b0,
a=0x7fced4cb3f48, msg=msg@entry=0x7fced4f093c8) at action.c:1549
#14 0x000000000042bde0 in do_action (h=h@entry=0x7ffeb0b402b0,
a=a@entry=0x7fced4ac3d70, msg=msg@entry=0x7fced4f093c8) at action.c:678
#15 0x000000000042a10a in run_actions (h=h@entry=0x7ffeb0b402b0,
a=a@entry=0x7fced4ac1698, msg=msg@entry=0x7fced4f093c8) at action.c:1549
#16 0x00000000004375d0 in run_top_route (a=0x7fced4ac1698,
msg=msg@entry=0x7fced4f093c8, c=c@entry=0x0) at action.c:1635
#17 0x0000000000504386 in receive_msg (buf=<optimized out>,
len=<optimized out>, rcv_info=<optimized out>) at receive.c:240
#18 0x00000000005f5bd4 in udp_rcv_loop () at udp_server.c:495
#19 0x00000000004b2625 in main_loop () at main.c:1600
#20 0x0000000000427e2b in main (argc=<optimized out>, argv=<optimized
out>) at main.c:2616
And from syslog the relevant messages before this dump:
Jul 15 08:55:03 server /usr/sbin/kamailio[16470]: WARNING: dialog
[dlg_handlers.c:1219]: dlg_onroute(): unable to find dialog for BYE with
route param 'd4c.26d1' [3149:7522]
Jul 15 08:56:01 server /usr/sbin/kamailio[16481]: WARNING: dialog
[dlg_handlers.c:1219]: dlg_onroute(): unable to find dialog for BYE with
route param 'fc.99f1' [207:8089]
Jul 15 08:56:27 server /usr/sbin/kamailio[16470]: CRITICAL: dialog
[dlg_timer.c:200]: update_dlg_timer(): Trying to update a bogus dlg
tl=0x7fceb3f7d920 tl->next=(nil) tl->prev=(nil)
Jul 15 08:56:27 server /usr/sbin/kamailio[16470]: ERROR: dialog
[dlg_handlers.c:1377]: dlg_onroute(): failed to update dialog lifetime
Jul 15 08:57:01 server /usr/sbin/kamailio[16482]: ERROR: db_mysql
[km_dbase.c:128]: db_mysql_submit_query(): driver error on query:
Duplicate entry '9584-3854-435' for key 'hash_index' (1062)
Jul 15 08:57:01 server /usr/sbin/kamailio[16482]: ERROR: <core>
[db_query.c:181]: db_do_raw_query(): error while submitting query
Jul 15 08:57:01 server /usr/sbin/kamailio[16482]: ERROR: sqlops
[sql_api.c:265]: sql_do_query(): cannot do the query [INSERT INTO
`dialog_extra` (`h_i]
Jul 15 08:57:01 server /usr/sbin/kamailio[16482]: ERROR: auth
[api.c:119]: auth_check_hdr_md5(): auth:pre_auth: Credentials are not
filled properly
Jul 15 08:57:01 server /usr/sbin/kamailio[16483]: ERROR: auth
[api.c:119]: auth_check_hdr_md5(): auth:pre_auth: Credentials are not
filled properly
Jul 15 08:57:54 server /usr/sbin/kamailio[16506]: NOTICE: dialog
[dlg_hash.c:245]: dlg_clean_run(): dialog in delete state is too old
(0x7fceb3f64470 ref 4)
Jul 15 08:57:54 server /usr/sbin/kamailio[16473]: WARNING: dialog
[dlg_handlers.c:1348]: dlg_onroute(): inconsitent dlg timer data on dlg
0x7fceb3f64470 [1182:5803] with clid
'09ad128753e2535d24bde58e3d7eda04@192.168.10.232:5060' and tags
'as1b497b34' '5788890C-EC6F55F-3E86ED0C'
Jul 15 08:57:54 server /usr/sbin/kamailio[16469]: ERROR: dialog
[dlg_handlers.c:334]: dlg_terminated_confirmed(): failed to get dialog
from params!
Jul 15 08:58:49 server /usr/sbin/kamailio[16467]: WARNING: dialog
[dlg_handlers.c:1219]: dlg_onroute(): unable to find dialog for BYE with
route param '6d2.2581' [726:6226]
Jul 15 08:59:24 server /usr/sbin/kamailio[16506]: NOTICE: dialog
[dlg_hash.c:245]: dlg_clean_run(): dialog in delete state is too old
(0x7fceb400e2f0 ref 4)
Jul 15 08:59:25 server /usr/sbin/kamailio[16464]: WARNING: dialog
[dlg_handlers.c:1219]: dlg_onroute(): unable to find dialog for BYE with
route param '3e4.b5c1' [1251:7259]
Jul 15 08:59:25 server /usr/sbin/kamailio[16465]: WARNING: dialog
[dlg_handlers.c:1348]: dlg_onroute(): inconsitent dlg timer data on dlg
0x7fceb400e2f0 [1251:7259] with clid '87791a#015#012Call-ID:
25750e286a5654361ef9405d72edbc' and tags '' 'as148f41b1'
Jul 15 08:59:25 server kernel: [203670.830521] kamailio[16465] general
protection ip:7fcead34b3a5 sp:7ffeb0b3e220 error:0 in
dialog.so[7fcead330000+88000]
Jul 15 08:59:26 server /usr/sbin/kamailio[16511]: CRITICAL: <core>
[pass_fd.c:275]: receive_fd(): EOF on 33
Jul 15 08:59:26 server /usr/sbin/kamailio[16458]: ALERT: <core>
[main.c:739]: handle_sigs(): child process 16465 exited by a signal 11
Jul 15 08:59:26 server /usr/sbin/kamailio[16458]: ALERT: <core>
[main.c:742]: handle_sigs(): core was generated
Jul 15 08:59:26 server /usr/sbin/kamailio[16458]: INFO: <core>
[main.c:754]: handle_sigs(): terminating due to SIGCHLD
Any insight would be appreciated!
Cheers,
Dirk
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users(a)lists.sip-router.org
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
1 p.m.
From the second crash, can you get:
frame 1
p *dlg
So far it looks like either to a double free or some buffer overflow...
Cheers,
Daniel
On 15/07/16 10:51, Dirk Teurlings - Signet B.V. wrote:
Just got another segfault.
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/sbin/kamailio -f /etc/kamailio/kamailio.cfg
-P /var/run/kamailio/kamailio.'.
Program terminated with signal 11, Segmentation fault.
#0 atomic_get (v=0x7f6264d11378) at ../../mem/../atomic/atomic_common.h:74
74 return atomic_get_int(&(v->val));
(gdb) bt
#0 atomic_get (v=0x7f6264d11378) at ../../mem/../atomic/atomic_common.h:74
#1 dlg_unref (dlg=dlg@entry=0x7f585c494b40, cnt=cnt@entry=1) at
dlg_hash.c:921
#2 0x00007f5855912802 in dlg_run_event_route
(dlg=dlg@entry=0x7f585c494b40, msg=msg@entry=0x7f587d4be8e8,
ostate=<optimized out>, nstate=<optimized out>) at dlg_handlers.c:1630
#3 0x00007f585591416a in dlg_onroute (req=0x7f587d4be8e8,
route_params=<optimized out>, param=<optimized out>) at dlg_handlers.c:1307
#4 0x00007f585965b0e2 in run_rr_callbacks
(req=req@entry=0x7f587d4be8e8, rr_param=rr_param@entry=0x7f58598677a0)
at rr_cb.c:96
#5 0x00007f58596452c5 in after_loose (_m=0x7f587d4be8e8, preloaded=0)
at loose.c:919
#6 0x000000000042b618 in do_action (h=h@entry=0x7ffd6e277fd0,
a=a@entry=0x7f587d264338, msg=msg@entry=0x7f587d4be8e8) at action.c:1060
#7 0x000000000042a10a in run_actions (h=h@entry=0x7ffd6e277fd0,
a=0x7f587d264338, msg=0x7f587d4be8e8) at action.c:1549
#8 0x0000000000437544 in run_actions_safe (h=h@entry=0x7ffd6e279500,
a=<optimized out>, msg=<optimized out>) at action.c:1614
#9 0x000000000053b2e8 in rval_get_int (h=0x7ffd6e279500, msg=<optimized
out>, i=0x7ffd6e278430, rv=rv@entry=0x7f587d264d58,
cache=cache@entry=0x0) at rvalue.c:912
#10 0x000000000054261c in rval_expr_eval_int (h=h@entry=0x7ffd6e279500,
msg=msg@entry=0x7f587d4be8e8, res=res@entry=0x7ffd6e278430,
rve=rve@entry=0x7f587d264d50) at rvalue.c:1910
#11 0x000000000042bc91 in do_action (h=h@entry=0x7ffd6e279500,
a=a@entry=0x7f587d268f88, msg=msg@entry=0x7f587d4be8e8) at action.c:1030
#12 0x000000000042a10a in run_actions (h=h@entry=0x7ffd6e279500,
a=0x7f587d268f88, msg=msg@entry=0x7f587d4be8e8) at action.c:1549
#13 0x000000000042bcf2 in do_action (h=h@entry=0x7ffd6e279500,
a=a@entry=0x7f587d2691e8, msg=msg@entry=0x7f587d4be8e8) at action.c:1049
#14 0x000000000042a10a in run_actions (h=h@entry=0x7ffd6e279500,
a=0x7f587d263f48, msg=msg@entry=0x7f587d4be8e8) at action.c:1549
#15 0x000000000042bde0 in do_action (h=h@entry=0x7ffd6e279500,
a=a@entry=0x7f587d073d70, msg=msg@entry=0x7f587d4be8e8) at action.c:678
#16 0x000000000042a10a in run_actions (h=h@entry=0x7ffd6e279500,
a=a@entry=0x7f587d071698, msg=msg@entry=0x7f587d4be8e8) at action.c:1549
#17 0x00000000004375d0 in run_top_route (a=0x7f587d071698,
msg=msg@entry=0x7f587d4be8e8, c=c@entry=0x0) at action.c:1635
#18 0x0000000000504386 in receive_msg (buf=<optimized out>,
len=<optimized out>, rcv_info=<optimized out>) at receive.c:240
#19 0x00000000005f5bd4 in udp_rcv_loop () at udp_server.c:495
#20 0x00000000004b2625 in main_loop () at main.c:1600
#21 0x0000000000427e2b in main (argc=<optimized out>, argv=<optimized
out>) at main.c:2616
Relevant logmessages before crash:
Jul 15 10:37:55 server /usr/sbin/kamailio[12426]: NOTICE: dialog
[dlg_hash.c:245]: dlg_clean_run(): dialog in delete state is too old
(0x7f585c4a6820 ref 4)
Jul 15 10:37:55 server /usr/sbin/kamailio[12397]: WARNING: dialog
[dlg_handlers.c:1219]: dlg_onroute(): unable to find dialog for BYE with
route param '70f.b9d1' [3847:7579]
Jul 15 10:37:55 server /usr/sbin/kamailio[12395]: WARNING: dialog
[dlg_handlers.c:1348]: dlg_onroute(): inconsitent dlg timer data on dlg
0x7f585c4a6820 [3847:7579] with clid
'4c41f08d317ecb9342b93f22738003f3@server' and tags 'as5f3a16b4'
'as71cb6036'
Jul 15 10:40:13 server /usr/sbin/kamailio[12378]: WARNING: dialog
[dlg_handlers.c:1219]: dlg_onroute(): unable to find dialog for BYE with
route param 'eb6.1e21' [1726:4833]
Jul 15 10:40:13 server /usr/sbin/kamailio[12376]: WARNING: dialog
[dlg_handlers.c:1219]: dlg_onroute(): unable to find dialog for BYE with
route param 'eb6.1e21' [1726:4833]
Jul 15 10:40:14 server /usr/sbin/kamailio[12377]: WARNING: dialog
[dlg_handlers.c:1219]: dlg_onroute(): unable to find dialog for BYE with
route param 'eb6.1e21' [1726:4833]
Jul 15 10:40:16 server /usr/sbin/kamailio[12377]: WARNING: dialog
[dlg_handlers.c:1219]: dlg_onroute(): unable to find dialog for BYE with
route param 'eb6.1e21' [1726:4833]
Jul 15 10:40:16 server /usr/sbin/kamailio[12396]: WARNING: dialog
[dlg_handlers.c:1219]: dlg_onroute(): unable to find dialog for BYE with
route param 'eb6.1e21' [1726:4833]
Jul 15 10:41:34 server /usr/sbin/kamailio[12396]: ERROR: sl
[sl_funcs.c:363]: sl_reply_error(): ERROR: sl_reply_error used: I'm
terribly sorry, server error occurred (1/SL)
Jul 15 10:41:34 server /usr/sbin/kamailio[12396]: ERROR: tm
[t_reply.c:533]: _reply_light(): ERROR: _reply_light: can't generate 487
reply when a final 487 was sent out
Jul 15 10:41:34 server /usr/sbin/kamailio[12396]: ERROR: tm
[t_lookup.c:1471]: t_unref(): ERROR: t_unref: generation of a delayed
stateful reply failed
Jul 15 10:42:25 server /usr/sbin/kamailio[12426]: NOTICE: dialog
[dlg_hash.c:245]: dlg_clean_run(): dialog in delete state is too old
(0x7f585c49d5b0 ref 4)
Jul 15 10:42:25 server /usr/sbin/kamailio[12426]: NOTICE: dialog
[dlg_hash.c:245]: dlg_clean_run(): dialog in delete state is too old
(0x7f585c604f18 ref 4)
Jul 15 10:42:25 server /usr/sbin/kamailio[12426]: NOTICE: dialog
[dlg_hash.c:245]: dlg_clean_run(): dialog in delete state is too old
(0x7f585c494b40 ref 4)
Jul 15 10:42:25 server /usr/sbin/kamailio[12383]: WARNING: dialog
[dlg_handlers.c:1348]: dlg_onroute(): inconsitent dlg timer data on dlg
0x7f585c604f18 [2396:9046] with clid
'1b3ff5f0246fb7e82ed949544bcccbba@192.168.10.233:5060' and tags
'as4d83d6f8' '5788A162-2557E04D-3E86ED15'
Jul 15 10:42:25 server /usr/sbin/kamailio[12395]: WARNING: dialog
[dlg_handlers.c:1219]: dlg_onroute(): unable to find dialog for BYE with
route param '6b3.c6b' [950:2924]
Jul 15 10:42:25 server kernel: [209851.262461] kamailio[12376]: segfault
at 7f6264d11378 ip 00007f585592a908 sp 00007ffd6e277330 error 4 in
dialog.so[7f58558e0000+88000]
Jul 15 10:42:25 server /usr/sbin/kamailio[12394]: WARNING: dialog
[dlg_handlers.c:1348]: dlg_onroute(): inconsitent dlg timer data on dlg
0x7f585c49d5b0 [950:2924] with clid
'45fe86ce065f5543342e51ad355d1b75@server' and tags 'as152f7465'
'as4d03f77d'
Jul 15 10:42:26 server /usr/sbin/kamailio[12431]: CRITICAL: <core>
[pass_fd.c:275]: receive_fd(): EOF on 32
Jul 15 10:42:26 server /usr/sbin/kamailio[12370]: ALERT: <core>
[main.c:739]: handle_sigs(): child process 12376 exited by a signal 11
Jul 15 10:42:26 server /usr/sbin/kamailio[12370]: ALERT: <core>
[main.c:742]: handle_sigs(): core was generated
Jul 15 10:42:26 server /usr/sbin/kamailio[12370]: INFO: <core>
[main.c:754]: handle_sigs(): terminating due to SIGCHLD
Cheers,
Dirk
On 07/15/2016 10:06 AM, Dirk Teurlings - Signet B.V. wrote:
--
Daniel-Constantin Mierla
http://www.asipto.com - http://www.kamailio.org
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Hi,
Running Kamailio on Debian from the Kamailio repository with 4.4.2
stable (unpatched). Getting some random segfaults with it now, here's
the relevant backtrace from the generated core.
Core was generated by `/usr/sbin/kamailio -f /etc/kamailio/kamailio.cfg
-P /var/run/kamailio/kamailio.'.
Program terminated with signal 11, Segmentation fault.
#0 run_dlg_callbacks (type=type@entry=64, dlg=dlg@entry=0x7fceb400e2f0,
req=req@entry=0x7fced4f093c8, rpl=rpl@entry=0x0, dir=<optimized out>,
dlg_data=dlg_data@entry=0x0) at dlg_cb.c:253
253 if ( (cb->types)&type ) {
(gdb) bt
#0 run_dlg_callbacks (type=type@entry=64, dlg=dlg@entry=0x7fceb400e2f0,
req=req@entry=0x7fced4f093c8, rpl=rpl@entry=0x0, dir=<optimized out>,
dlg_data=dlg_data@entry=0x0) at dlg_cb.c:253
#1 0x00007fcead3648f9 in dlg_terminated (dir=<optimized out>,
dlg=0x7fceb400e2f0, req=0x7fced4f093c8) at dlg_handlers.c:368
#2 dlg_onroute (req=0x7fced4f093c8, route_params=<optimized out>,
param=<optimized out>) at dlg_handlers.c:1354
#3 0x00007fceb10ab0e2 in run_rr_callbacks
(req=req@entry=0x7fced4f093c8, rr_param=rr_param@entry=0x7fceb12b77a0)
at rr_cb.c:96
#4 0x00007fceb10952c5 in after_loose (_m=0x7fced4f093c8, preloaded=0)
at loose.c:919
#5 0x000000000042b618 in do_action (h=h@entry=0x7ffeb0b3ed80,
a=a@entry=0x7fced4cb4338, msg=msg@entry=0x7fced4f093c8) at action.c:1060
#6 0x000000000042a10a in run_actions (h=h@entry=0x7ffeb0b3ed80,
a=0x7fced4cb4338, msg=0x7fced4f093c8) at action.c:1549
#7 0x0000000000437544 in run_actions_safe (h=h@entry=0x7ffeb0b402b0,
a=<optimized out>, msg=<optimized out>) at action.c:1614
#8 0x000000000053b2e8 in rval_get_int (h=0x7ffeb0b402b0, msg=<optimized
out>, i=0x7ffeb0b3f1e0, rv=rv@entry=0x7fced4cb4d58,
cache=cache@entry=0x0) at rvalue.c:912
#9 0x000000000054261c in rval_expr_eval_int (h=h@entry=0x7ffeb0b402b0,
msg=msg@entry=0x7fced4f093c8, res=res@entry=0x7ffeb0b3f1e0,
rve=rve@entry=0x7fced4cb4d50) at rvalue.c:1910
#10 0x000000000042bc91 in do_action (h=h@entry=0x7ffeb0b402b0,
a=a@entry=0x7fced4cb8f88, msg=msg@entry=0x7fced4f093c8) at action.c:1030
#11 0x000000000042a10a in run_actions (h=h@entry=0x7ffeb0b402b0,
a=0x7fced4cb8f88, msg=msg@entry=0x7fced4f093c8) at action.c:1549
#12 0x000000000042bcf2 in do_action (h=h@entry=0x7ffeb0b402b0,
a=a@entry=0x7fced4cb91e8, msg=msg@entry=0x7fced4f093c8) at action.c:1049
#13 0x000000000042a10a in run_actions (h=h@entry=0x7ffeb0b402b0,
a=0x7fced4cb3f48, msg=msg@entry=0x7fced4f093c8) at action.c:1549
#14 0x000000000042bde0 in do_action (h=h@entry=0x7ffeb0b402b0,
a=a@entry=0x7fced4ac3d70, msg=msg@entry=0x7fced4f093c8) at action.c:678
#15 0x000000000042a10a in run_actions (h=h@entry=0x7ffeb0b402b0,
a=a@entry=0x7fced4ac1698, msg=msg@entry=0x7fced4f093c8) at action.c:1549
#16 0x00000000004375d0 in run_top_route (a=0x7fced4ac1698,
msg=msg@entry=0x7fced4f093c8, c=c@entry=0x0) at action.c:1635
#17 0x0000000000504386 in receive_msg (buf=<optimized out>,
len=<optimized out>, rcv_info=<optimized out>) at receive.c:240
#18 0x00000000005f5bd4 in udp_rcv_loop () at udp_server.c:495
#19 0x00000000004b2625 in main_loop () at main.c:1600
#20 0x0000000000427e2b in main (argc=<optimized out>, argv=<optimized
out>) at main.c:2616
And from syslog the relevant messages before this dump:
Jul 15 08:55:03 server /usr/sbin/kamailio[16470]: WARNING: dialog
[dlg_handlers.c:1219]: dlg_onroute(): unable to find dialog for BYE with
route param 'd4c.26d1' [3149:7522]
Jul 15 08:56:01 server /usr/sbin/kamailio[16481]: WARNING: dialog
[dlg_handlers.c:1219]: dlg_onroute(): unable to find dialog for BYE with
route param 'fc.99f1' [207:8089]
Jul 15 08:56:27 server /usr/sbin/kamailio[16470]: CRITICAL: dialog
[dlg_timer.c:200]: update_dlg_timer(): Trying to update a bogus dlg
tl=0x7fceb3f7d920 tl->next=(nil) tl->prev=(nil)
Jul 15 08:56:27 server /usr/sbin/kamailio[16470]: ERROR: dialog
[dlg_handlers.c:1377]: dlg_onroute(): failed to update dialog lifetime
Jul 15 08:57:01 server /usr/sbin/kamailio[16482]: ERROR: db_mysql
[km_dbase.c:128]: db_mysql_submit_query(): driver error on query:
Duplicate entry '9584-3854-435' for key 'hash_index' (1062)
Jul 15 08:57:01 server /usr/sbin/kamailio[16482]: ERROR: <core>
[db_query.c:181]: db_do_raw_query(): error while submitting query
Jul 15 08:57:01 server /usr/sbin/kamailio[16482]: ERROR: sqlops
[sql_api.c:265]: sql_do_query(): cannot do the query [INSERT INTO
`dialog_extra` (`h_i]
Jul 15 08:57:01 server /usr/sbin/kamailio[16482]: ERROR: auth
[api.c:119]: auth_check_hdr_md5(): auth:pre_auth: Credentials are not
filled properly
Jul 15 08:57:01 server /usr/sbin/kamailio[16483]: ERROR: auth
[api.c:119]: auth_check_hdr_md5(): auth:pre_auth: Credentials are not
filled properly
Jul 15 08:57:54 server /usr/sbin/kamailio[16506]: NOTICE: dialog
[dlg_hash.c:245]: dlg_clean_run(): dialog in delete state is too old
(0x7fceb3f64470 ref 4)
Jul 15 08:57:54 server /usr/sbin/kamailio[16473]: WARNING: dialog
[dlg_handlers.c:1348]: dlg_onroute(): inconsitent dlg timer data on dlg
0x7fceb3f64470 [1182:5803] with clid
'09ad128753e2535d24bde58e3d7eda04@192.168.10.232:5060' and tags
'as1b497b34' '5788890C-EC6F55F-3E86ED0C'
Jul 15 08:57:54 server /usr/sbin/kamailio[16469]: ERROR: dialog
[dlg_handlers.c:334]: dlg_terminated_confirmed(): failed to get dialog
from params!
Jul 15 08:58:49 server /usr/sbin/kamailio[16467]: WARNING: dialog
[dlg_handlers.c:1219]: dlg_onroute(): unable to find dialog for BYE with
route param '6d2.2581' [726:6226]
Jul 15 08:59:24 server /usr/sbin/kamailio[16506]: NOTICE: dialog
[dlg_hash.c:245]: dlg_clean_run(): dialog in delete state is too old
(0x7fceb400e2f0 ref 4)
Jul 15 08:59:25 server /usr/sbin/kamailio[16464]: WARNING: dialog
[dlg_handlers.c:1219]: dlg_onroute(): unable to find dialog for BYE with
route param '3e4.b5c1' [1251:7259]
Jul 15 08:59:25 server /usr/sbin/kamailio[16465]: WARNING: dialog
[dlg_handlers.c:1348]: dlg_onroute(): inconsitent dlg timer data on dlg
0x7fceb400e2f0 [1251:7259] with clid '87791a#015#012Call-ID:
25750e286a5654361ef9405d72edbc' and tags '' 'as148f41b1'
Jul 15 08:59:25 server kernel: [203670.830521] kamailio[16465] general
protection ip:7fcead34b3a5 sp:7ffeb0b3e220 error:0 in
dialog.so[7fcead330000+88000]
Jul 15 08:59:26 server /usr/sbin/kamailio[16511]: CRITICAL: <core>
[pass_fd.c:275]: receive_fd(): EOF on 33
Jul 15 08:59:26 server /usr/sbin/kamailio[16458]: ALERT: <core>
[main.c:739]: handle_sigs(): child process 16465 exited by a signal 11
Jul 15 08:59:26 server /usr/sbin/kamailio[16458]: ALERT: <core>
[main.c:742]: handle_sigs(): core was generated
Jul 15 08:59:26 server /usr/sbin/kamailio[16458]: INFO: <core>
[main.c:754]: handle_sigs(): terminating due to SIGCHLD
Any insight would be appreciated!
Cheers,
Dirk
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users(a)lists.sip-router.org
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users(a)lists.sip-router.org
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
1:06 p.m.
(gdb) frame 1
#1 dlg_unref (dlg=dlg@entry=0x7f585c494b40, cnt=cnt@entry=1) at
dlg_hash.c:921
921 dlg_lock( d_table, d_entry);
(gdb) p *dlg
$1 = {ref = 793790803, next = 0xa0d4b4f20303032, prev =
0x504953203a616956, h_id = 808333871, h_entry = 1346655535, state =
774976288, lifetime = 775107122, init_ts = 775435825,
start_ts = 976303410, end_ts = 808857653, dflags = 1667592763, iflags
= 1702259045, sflags = 825441636, toroute = 858927662, toroute_name = {
s = 0x6172623b3135322e <Address 0x6172623b3135322e out of bounds>,
len = 1030251374}, from_rr_nb = 894132788, tl = {next =
0x726f70723b646262, prev = 0xa0d303630353d74,
timeout = 1836020294}, callid = {s = 0x20226e776f6e6b6e <Address
0x20226e776f6e6b6e out of bounds>, len = 1885958972}, from_uri = {
s = 0x7340444c4f74656e <Address 0x7340444c4f74656e out of bounds>,
len = 1999532137}, to_uri = {s = 0x743b3e74656e2e70 <Address
0x743b3e74656e2e70 out of bounds>,
len = 1631414113}, req_uri = {s = 0x540a0d3536343766 <Address
0x540a0d3536343766 out of bounds>, len = 1008745071}, tag = {{
s = 0x363233313431332b <Address 0x363233313431332b out of bounds>,
len = 892614711}, {s = 0x2e3836312e333232 <Address 0x2e3836312e333232
out of bounds>, len = 1043608370}},
cseq = {{s = 0x663330643473613d <Address 0x663330643473613d out of
bounds>, len = 224671543}, {s = 0x3534203a44492d6c <Address
0x3534203a44492d6c out of bounds>,
len = 909665638}}, route_set = {{s = 0x3433333435356635 <Address
0x3433333435356635 out of bounds>, len = 825582898}, {
s = 0x7340353762316435 <Address 0x7340353762316435 out of bounds>,
len = 1999532137}}, contact = {{s = 0x430a0d74656e2e70 <Address
0x430a0d74656e2e70 out of bounds>,
len = 980510035}, {s = 0x65530a0d45594220 <Address
0x65530a0d45594220 out of bounds>, len = 1919252082}}, bind_addr =
{0x70696f766c772e70, 0x6c410a0d74656e2e}, cbs = {
first = 0x564e49203a776f6c, types = 742741065}, profile_links =
0x4c45434e4143202c, vars = 0x4e4f4954504f202c}
On 07/15/2016 01:00 PM, Daniel-Constantin Mierla wrote:
From the second crash, can you get:
frame 1
p *dlg
So far it looks like either to a double free or some buffer overflow...
Cheers,
Daniel
On 15/07/16 10:51, Dirk Teurlings - Signet B.V. wrote:
Just got another segfault.
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/sbin/kamailio -f /etc/kamailio/kamailio.cfg
-P /var/run/kamailio/kamailio.'.
Program terminated with signal 11, Segmentation fault.
#0 atomic_get (v=0x7f6264d11378) at ../../mem/../atomic/atomic_common.h:74
74 return atomic_get_int(&(v->val));
(gdb) bt
#0 atomic_get (v=0x7f6264d11378) at ../../mem/../atomic/atomic_common.h:74
#1 dlg_unref (dlg=dlg@entry=0x7f585c494b40, cnt=cnt@entry=1) at
dlg_hash.c:921
#2 0x00007f5855912802 in dlg_run_event_route
(dlg=dlg@entry=0x7f585c494b40, msg=msg@entry=0x7f587d4be8e8,
ostate=<optimized out>, nstate=<optimized out>) at dlg_handlers.c:1630
#3 0x00007f585591416a in dlg_onroute (req=0x7f587d4be8e8,
route_params=<optimized out>, param=<optimized out>) at dlg_handlers.c:1307
#4 0x00007f585965b0e2 in run_rr_callbacks
(req=req@entry=0x7f587d4be8e8, rr_param=rr_param@entry=0x7f58598677a0)
at rr_cb.c:96
#5 0x00007f58596452c5 in after_loose (_m=0x7f587d4be8e8, preloaded=0)
at loose.c:919
#6 0x000000000042b618 in do_action (h=h@entry=0x7ffd6e277fd0,
a=a@entry=0x7f587d264338, msg=msg@entry=0x7f587d4be8e8) at action.c:1060
#7 0x000000000042a10a in run_actions (h=h@entry=0x7ffd6e277fd0,
a=0x7f587d264338, msg=0x7f587d4be8e8) at action.c:1549
#8 0x0000000000437544 in run_actions_safe (h=h@entry=0x7ffd6e279500,
a=<optimized out>, msg=<optimized out>) at action.c:1614
#9 0x000000000053b2e8 in rval_get_int (h=0x7ffd6e279500, msg=<optimized
out>, i=0x7ffd6e278430, rv=rv@entry=0x7f587d264d58,
cache=cache@entry=0x0) at rvalue.c:912
#10 0x000000000054261c in rval_expr_eval_int (h=h@entry=0x7ffd6e279500,
msg=msg@entry=0x7f587d4be8e8, res=res@entry=0x7ffd6e278430,
rve=rve@entry=0x7f587d264d50) at rvalue.c:1910
#11 0x000000000042bc91 in do_action (h=h@entry=0x7ffd6e279500,
a=a@entry=0x7f587d268f88, msg=msg@entry=0x7f587d4be8e8) at action.c:1030
#12 0x000000000042a10a in run_actions (h=h@entry=0x7ffd6e279500,
a=0x7f587d268f88, msg=msg@entry=0x7f587d4be8e8) at action.c:1549
#13 0x000000000042bcf2 in do_action (h=h@entry=0x7ffd6e279500,
a=a@entry=0x7f587d2691e8, msg=msg@entry=0x7f587d4be8e8) at action.c:1049
#14 0x000000000042a10a in run_actions (h=h@entry=0x7ffd6e279500,
a=0x7f587d263f48, msg=msg@entry=0x7f587d4be8e8) at action.c:1549
#15 0x000000000042bde0 in do_action (h=h@entry=0x7ffd6e279500,
a=a@entry=0x7f587d073d70, msg=msg@entry=0x7f587d4be8e8) at action.c:678
#16 0x000000000042a10a in run_actions (h=h@entry=0x7ffd6e279500,
a=a@entry=0x7f587d071698, msg=msg@entry=0x7f587d4be8e8) at action.c:1549
#17 0x00000000004375d0 in run_top_route (a=0x7f587d071698,
msg=msg@entry=0x7f587d4be8e8, c=c@entry=0x0) at action.c:1635
#18 0x0000000000504386 in receive_msg (buf=<optimized out>,
len=<optimized out>, rcv_info=<optimized out>) at receive.c:240
#19 0x00000000005f5bd4 in udp_rcv_loop () at udp_server.c:495
#20 0x00000000004b2625 in main_loop () at main.c:1600
#21 0x0000000000427e2b in main (argc=<optimized out>, argv=<optimized
out>) at main.c:2616
Relevant logmessages before crash:
Jul 15 10:37:55 server /usr/sbin/kamailio[12426]: NOTICE: dialog
[dlg_hash.c:245]: dlg_clean_run(): dialog in delete state is too old
(0x7f585c4a6820 ref 4)
Jul 15 10:37:55 server /usr/sbin/kamailio[12397]: WARNING: dialog
[dlg_handlers.c:1219]: dlg_onroute(): unable to find dialog for BYE with
route param '70f.b9d1' [3847:7579]
Jul 15 10:37:55 server /usr/sbin/kamailio[12395]: WARNING: dialog
[dlg_handlers.c:1348]: dlg_onroute(): inconsitent dlg timer data on dlg
0x7f585c4a6820 [3847:7579] with clid
'4c41f08d317ecb9342b93f22738003f3@server' and tags 'as5f3a16b4'
'as71cb6036'
Jul 15 10:40:13 server /usr/sbin/kamailio[12378]: WARNING: dialog
[dlg_handlers.c:1219]: dlg_onroute(): unable to find dialog for BYE with
route param 'eb6.1e21' [1726:4833]
Jul 15 10:40:13 server /usr/sbin/kamailio[12376]: WARNING: dialog
[dlg_handlers.c:1219]: dlg_onroute(): unable to find dialog for BYE with
route param 'eb6.1e21' [1726:4833]
Jul 15 10:40:14 server /usr/sbin/kamailio[12377]: WARNING: dialog
[dlg_handlers.c:1219]: dlg_onroute(): unable to find dialog for BYE with
route param 'eb6.1e21' [1726:4833]
Jul 15 10:40:16 server /usr/sbin/kamailio[12377]: WARNING: dialog
[dlg_handlers.c:1219]: dlg_onroute(): unable to find dialog for BYE with
route param 'eb6.1e21' [1726:4833]
Jul 15 10:40:16 server /usr/sbin/kamailio[12396]: WARNING: dialog
[dlg_handlers.c:1219]: dlg_onroute(): unable to find dialog for BYE with
route param 'eb6.1e21' [1726:4833]
Jul 15 10:41:34 server /usr/sbin/kamailio[12396]: ERROR: sl
[sl_funcs.c:363]: sl_reply_error(): ERROR: sl_reply_error used: I'm
terribly sorry, server error occurred (1/SL)
Jul 15 10:41:34 server /usr/sbin/kamailio[12396]: ERROR: tm
[t_reply.c:533]: _reply_light(): ERROR: _reply_light: can't generate 487
reply when a final 487 was sent out
Jul 15 10:41:34 server /usr/sbin/kamailio[12396]: ERROR: tm
[t_lookup.c:1471]: t_unref(): ERROR: t_unref: generation of a delayed
stateful reply failed
Jul 15 10:42:25 server /usr/sbin/kamailio[12426]: NOTICE: dialog
[dlg_hash.c:245]: dlg_clean_run(): dialog in delete state is too old
(0x7f585c49d5b0 ref 4)
Jul 15 10:42:25 server /usr/sbin/kamailio[12426]: NOTICE: dialog
[dlg_hash.c:245]: dlg_clean_run(): dialog in delete state is too old
(0x7f585c604f18 ref 4)
Jul 15 10:42:25 server /usr/sbin/kamailio[12426]: NOTICE: dialog
[dlg_hash.c:245]: dlg_clean_run(): dialog in delete state is too old
(0x7f585c494b40 ref 4)
Jul 15 10:42:25 server /usr/sbin/kamailio[12383]: WARNING: dialog
[dlg_handlers.c:1348]: dlg_onroute(): inconsitent dlg timer data on dlg
0x7f585c604f18 [2396:9046] with clid
'1b3ff5f0246fb7e82ed949544bcccbba@192.168.10.233:5060' and tags
'as4d83d6f8' '5788A162-2557E04D-3E86ED15'
Jul 15 10:42:25 server /usr/sbin/kamailio[12395]: WARNING: dialog
[dlg_handlers.c:1219]: dlg_onroute(): unable to find dialog for BYE with
route param '6b3.c6b' [950:2924]
Jul 15 10:42:25 server kernel: [209851.262461] kamailio[12376]: segfault
at 7f6264d11378 ip 00007f585592a908 sp 00007ffd6e277330 error 4 in
dialog.so[7f58558e0000+88000]
Jul 15 10:42:25 server /usr/sbin/kamailio[12394]: WARNING: dialog
[dlg_handlers.c:1348]: dlg_onroute(): inconsitent dlg timer data on dlg
0x7f585c49d5b0 [950:2924] with clid
'45fe86ce065f5543342e51ad355d1b75@server' and tags 'as152f7465'
'as4d03f77d'
Jul 15 10:42:26 server /usr/sbin/kamailio[12431]: CRITICAL: <core>
[pass_fd.c:275]: receive_fd(): EOF on 32
Jul 15 10:42:26 server /usr/sbin/kamailio[12370]: ALERT: <core>
[main.c:739]: handle_sigs(): child process 12376 exited by a signal 11
Jul 15 10:42:26 server /usr/sbin/kamailio[12370]: ALERT: <core>
[main.c:742]: handle_sigs(): core was generated
Jul 15 10:42:26 server /usr/sbin/kamailio[12370]: INFO: <core>
[main.c:754]: handle_sigs(): terminating due to SIGCHLD
Cheers,
Dirk
On 07/15/2016 10:06 AM, Dirk Teurlings - Signet B.V. wrote:
Hi,
Running Kamailio on Debian from the Kamailio repository with 4.4.2
stable (unpatched). Getting some random segfaults with it now, here's
the relevant backtrace from the generated core.
Core was generated by `/usr/sbin/kamailio -f /etc/kamailio/kamailio.cfg
-P /var/run/kamailio/kamailio.'.
Program terminated with signal 11, Segmentation fault.
#0 run_dlg_callbacks (type=type@entry=64, dlg=dlg@entry=0x7fceb400e2f0,
req=req@entry=0x7fced4f093c8, rpl=rpl@entry=0x0, dir=<optimized out>,
dlg_data=dlg_data@entry=0x0) at dlg_cb.c:253
253 if ( (cb->types)&type ) {
(gdb) bt
#0 run_dlg_callbacks (type=type@entry=64, dlg=dlg@entry=0x7fceb400e2f0,
req=req@entry=0x7fced4f093c8, rpl=rpl@entry=0x0, dir=<optimized out>,
dlg_data=dlg_data@entry=0x0) at dlg_cb.c:253
#1 0x00007fcead3648f9 in dlg_terminated (dir=<optimized out>,
dlg=0x7fceb400e2f0, req=0x7fced4f093c8) at dlg_handlers.c:368
#2 dlg_onroute (req=0x7fced4f093c8, route_params=<optimized out>,
param=<optimized out>) at dlg_handlers.c:1354
#3 0x00007fceb10ab0e2 in run_rr_callbacks
(req=req@entry=0x7fced4f093c8, rr_param=rr_param@entry=0x7fceb12b77a0)
at rr_cb.c:96
#4 0x00007fceb10952c5 in after_loose (_m=0x7fced4f093c8, preloaded=0)
at loose.c:919
#5 0x000000000042b618 in do_action (h=h@entry=0x7ffeb0b3ed80,
a=a@entry=0x7fced4cb4338, msg=msg@entry=0x7fced4f093c8) at action.c:1060
#6 0x000000000042a10a in run_actions (h=h@entry=0x7ffeb0b3ed80,
a=0x7fced4cb4338, msg=0x7fced4f093c8) at action.c:1549
#7 0x0000000000437544 in run_actions_safe (h=h@entry=0x7ffeb0b402b0,
a=<optimized out>, msg=<optimized out>) at action.c:1614
#8 0x000000000053b2e8 in rval_get_int (h=0x7ffeb0b402b0, msg=<optimized
out>, i=0x7ffeb0b3f1e0, rv=rv@entry=0x7fced4cb4d58,
cache=cache@entry=0x0) at rvalue.c:912
#9 0x000000000054261c in rval_expr_eval_int (h=h@entry=0x7ffeb0b402b0,
msg=msg@entry=0x7fced4f093c8, res=res@entry=0x7ffeb0b3f1e0,
rve=rve@entry=0x7fced4cb4d50) at rvalue.c:1910
#10 0x000000000042bc91 in do_action (h=h@entry=0x7ffeb0b402b0,
a=a@entry=0x7fced4cb8f88, msg=msg@entry=0x7fced4f093c8) at action.c:1030
#11 0x000000000042a10a in run_actions (h=h@entry=0x7ffeb0b402b0,
a=0x7fced4cb8f88, msg=msg@entry=0x7fced4f093c8) at action.c:1549
#12 0x000000000042bcf2 in do_action (h=h@entry=0x7ffeb0b402b0,
a=a@entry=0x7fced4cb91e8, msg=msg@entry=0x7fced4f093c8) at action.c:1049
#13 0x000000000042a10a in run_actions (h=h@entry=0x7ffeb0b402b0,
a=0x7fced4cb3f48, msg=msg@entry=0x7fced4f093c8) at action.c:1549
#14 0x000000000042bde0 in do_action (h=h@entry=0x7ffeb0b402b0,
a=a@entry=0x7fced4ac3d70, msg=msg@entry=0x7fced4f093c8) at action.c:678
#15 0x000000000042a10a in run_actions (h=h@entry=0x7ffeb0b402b0,
a=a@entry=0x7fced4ac1698, msg=msg@entry=0x7fced4f093c8) at action.c:1549
#16 0x00000000004375d0 in run_top_route (a=0x7fced4ac1698,
msg=msg@entry=0x7fced4f093c8, c=c@entry=0x0) at action.c:1635
#17 0x0000000000504386 in receive_msg (buf=<optimized out>,
len=<optimized out>, rcv_info=<optimized out>) at receive.c:240
#18 0x00000000005f5bd4 in udp_rcv_loop () at udp_server.c:495
#19 0x00000000004b2625 in main_loop () at main.c:1600
#20 0x0000000000427e2b in main (argc=<optimized out>, argv=<optimized
out>) at main.c:2616
And from syslog the relevant messages before this dump:
Jul 15 08:55:03 server /usr/sbin/kamailio[16470]: WARNING: dialog
[dlg_handlers.c:1219]: dlg_onroute(): unable to find dialog for BYE with
route param 'd4c.26d1' [3149:7522]
Jul 15 08:56:01 server /usr/sbin/kamailio[16481]: WARNING: dialog
[dlg_handlers.c:1219]: dlg_onroute(): unable to find dialog for BYE with
route param 'fc.99f1' [207:8089]
Jul 15 08:56:27 server /usr/sbin/kamailio[16470]: CRITICAL: dialog
[dlg_timer.c:200]: update_dlg_timer(): Trying to update a bogus dlg
tl=0x7fceb3f7d920 tl->next=(nil) tl->prev=(nil)
Jul 15 08:56:27 server /usr/sbin/kamailio[16470]: ERROR: dialog
[dlg_handlers.c:1377]: dlg_onroute(): failed to update dialog lifetime
Jul 15 08:57:01 server /usr/sbin/kamailio[16482]: ERROR: db_mysql
[km_dbase.c:128]: db_mysql_submit_query(): driver error on query:
Duplicate entry '9584-3854-435' for key 'hash_index' (1062)
Jul 15 08:57:01 server /usr/sbin/kamailio[16482]: ERROR: <core>
[db_query.c:181]: db_do_raw_query(): error while submitting query
Jul 15 08:57:01 server /usr/sbin/kamailio[16482]: ERROR: sqlops
[sql_api.c:265]: sql_do_query(): cannot do the query [INSERT INTO
`dialog_extra` (`h_i]
Jul 15 08:57:01 server /usr/sbin/kamailio[16482]: ERROR: auth
[api.c:119]: auth_check_hdr_md5(): auth:pre_auth: Credentials are not
filled properly
Jul 15 08:57:01 server /usr/sbin/kamailio[16483]: ERROR: auth
[api.c:119]: auth_check_hdr_md5(): auth:pre_auth: Credentials are not
filled properly
Jul 15 08:57:54 server /usr/sbin/kamailio[16506]: NOTICE: dialog
[dlg_hash.c:245]: dlg_clean_run(): dialog in delete state is too old
(0x7fceb3f64470 ref 4)
Jul 15 08:57:54 server /usr/sbin/kamailio[16473]: WARNING: dialog
[dlg_handlers.c:1348]: dlg_onroute(): inconsitent dlg timer data on dlg
0x7fceb3f64470 [1182:5803] with clid
'09ad128753e2535d24bde58e3d7eda04@192.168.10.232:5060' and tags
'as1b497b34' '5788890C-EC6F55F-3E86ED0C'
Jul 15 08:57:54 server /usr/sbin/kamailio[16469]: ERROR: dialog
[dlg_handlers.c:334]: dlg_terminated_confirmed(): failed to get dialog
from params!
Jul 15 08:58:49 server /usr/sbin/kamailio[16467]: WARNING: dialog
[dlg_handlers.c:1219]: dlg_onroute(): unable to find dialog for BYE with
route param '6d2.2581' [726:6226]
Jul 15 08:59:24 server /usr/sbin/kamailio[16506]: NOTICE: dialog
[dlg_hash.c:245]: dlg_clean_run(): dialog in delete state is too old
(0x7fceb400e2f0 ref 4)
Jul 15 08:59:25 server /usr/sbin/kamailio[16464]: WARNING: dialog
[dlg_handlers.c:1219]: dlg_onroute(): unable to find dialog for BYE with
route param '3e4.b5c1' [1251:7259]
Jul 15 08:59:25 server /usr/sbin/kamailio[16465]: WARNING: dialog
[dlg_handlers.c:1348]: dlg_onroute(): inconsitent dlg timer data on dlg
0x7fceb400e2f0 [1251:7259] with clid '87791a#015#012Call-ID:
25750e286a5654361ef9405d72edbc' and tags '' 'as148f41b1'
Jul 15 08:59:25 server kernel: [203670.830521] kamailio[16465] general
protection ip:7fcead34b3a5 sp:7ffeb0b3e220 error:0 in
dialog.so[7fcead330000+88000]
Jul 15 08:59:26 server /usr/sbin/kamailio[16511]: CRITICAL: <core>
[pass_fd.c:275]: receive_fd(): EOF on 33
Jul 15 08:59:26 server /usr/sbin/kamailio[16458]: ALERT: <core>
[main.c:739]: handle_sigs(): child process 16465 exited by a signal 11
Jul 15 08:59:26 server /usr/sbin/kamailio[16458]: ALERT: <core>
[main.c:742]: handle_sigs(): core was generated
Jul 15 08:59:26 server /usr/sbin/kamailio[16458]: INFO: <core>
[main.c:754]: handle_sigs(): terminating due to SIGCHLD
Any insight would be appreciated!
Cheers,
Dirk
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users(a)lists.sip-router.org
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users(a)lists.sip-router.org
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
2:08 p.m.
The content of dlg is not valid, likely freed. Can you run with -x qm
and see if you get new error messages?
Also, what modules are you using, specially interested in those using
dialog module, such as cnxcc or presence dialog info?!?!
Cheers,
Daniel
On 15/07/16 13:06, Dirk Teurlings - Signet B.V. wrote:
(gdb) frame 1
#1 dlg_unref (dlg=dlg@entry=0x7f585c494b40, cnt=cnt@entry=1) at
dlg_hash.c:921
921 dlg_lock( d_table, d_entry);
(gdb) p *dlg
$1 = {ref = 793790803, next = 0xa0d4b4f20303032, prev =
0x504953203a616956, h_id = 808333871, h_entry = 1346655535, state =
774976288, lifetime = 775107122, init_ts = 775435825,
start_ts = 976303410, end_ts = 808857653, dflags = 1667592763, iflags
= 1702259045, sflags = 825441636, toroute = 858927662, toroute_name = {
s = 0x6172623b3135322e <Address 0x6172623b3135322e out of bounds>,
len = 1030251374}, from_rr_nb = 894132788, tl = {next =
0x726f70723b646262, prev = 0xa0d303630353d74,
timeout = 1836020294}, callid = {s = 0x20226e776f6e6b6e <Address
0x20226e776f6e6b6e out of bounds>, len = 1885958972}, from_uri = {
s = 0x7340444c4f74656e <Address 0x7340444c4f74656e out of bounds>,
len = 1999532137}, to_uri = {s = 0x743b3e74656e2e70 <Address
0x743b3e74656e2e70 out of bounds>,
len = 1631414113}, req_uri = {s = 0x540a0d3536343766 <Address
0x540a0d3536343766 out of bounds>, len = 1008745071}, tag = {{
s = 0x363233313431332b <Address 0x363233313431332b out of bounds>,
len = 892614711}, {s = 0x2e3836312e333232 <Address 0x2e3836312e333232
out of bounds>, len = 1043608370}},
cseq = {{s = 0x663330643473613d <Address 0x663330643473613d out of
bounds>, len = 224671543}, {s = 0x3534203a44492d6c <Address
0x3534203a44492d6c out of bounds>,
len = 909665638}}, route_set = {{s = 0x3433333435356635 <Address
0x3433333435356635 out of bounds>, len = 825582898}, {
s = 0x7340353762316435 <Address 0x7340353762316435 out of bounds>,
len = 1999532137}}, contact = {{s = 0x430a0d74656e2e70 <Address
0x430a0d74656e2e70 out of bounds>,
len = 980510035}, {s = 0x65530a0d45594220 <Address
0x65530a0d45594220 out of bounds>, len = 1919252082}}, bind_addr =
{0x70696f766c772e70, 0x6c410a0d74656e2e}, cbs = {
first = 0x564e49203a776f6c, types = 742741065}, profile_links =
0x4c45434e4143202c, vars = 0x4e4f4954504f202c}
On 07/15/2016 01:00 PM, Daniel-Constantin Mierla wrote:
> From the second crash, can you get:
>
> frame 1
>
> p *dlg
>
> So far it looks like either to a double free or some buffer overflow...
>
> Cheers,
> Daniel
>
>
> On 15/07/16 10:51, Dirk Teurlings - Signet B.V. wrote:
>> Just got another segfault.
>>
>> Using host libthread_db library
"/lib/x86_64-linux-gnu/libthread_db.so.1".
>> Core was generated by `/usr/sbin/kamailio -f /etc/kamailio/kamailio.cfg
>> -P /var/run/kamailio/kamailio.'.
>> Program terminated with signal 11, Segmentation fault.
>> #0 atomic_get (v=0x7f6264d11378) at ../../mem/../atomic/atomic_common.h:74
>> 74 return atomic_get_int(&(v->val));
>> (gdb) bt
>> #0 atomic_get (v=0x7f6264d11378) at ../../mem/../atomic/atomic_common.h:74
>> #1 dlg_unref (dlg=dlg@entry=0x7f585c494b40, cnt=cnt@entry=1) at
>> dlg_hash.c:921
>> #2 0x00007f5855912802 in dlg_run_event_route
>> (dlg=dlg@entry=0x7f585c494b40, msg=msg@entry=0x7f587d4be8e8,
>> ostate=<optimized out>, nstate=<optimized out>) at
dlg_handlers.c:1630
>> #3 0x00007f585591416a in dlg_onroute (req=0x7f587d4be8e8,
>> route_params=<optimized out>, param=<optimized out>) at
dlg_handlers.c:1307
>> #4 0x00007f585965b0e2 in run_rr_callbacks
>> (req=req@entry=0x7f587d4be8e8, rr_param=rr_param@entry=0x7f58598677a0)
>> at rr_cb.c:96
>> #5 0x00007f58596452c5 in after_loose (_m=0x7f587d4be8e8, preloaded=0)
>> at loose.c:919
>> #6 0x000000000042b618 in do_action (h=h@entry=0x7ffd6e277fd0,
>> a=a@entry=0x7f587d264338, msg=msg@entry=0x7f587d4be8e8) at action.c:1060
>> #7 0x000000000042a10a in run_actions (h=h@entry=0x7ffd6e277fd0,
>> a=0x7f587d264338, msg=0x7f587d4be8e8) at action.c:1549
>> #8 0x0000000000437544 in run_actions_safe (h=h@entry=0x7ffd6e279500,
>> a=<optimized out>, msg=<optimized out>) at action.c:1614
>> #9 0x000000000053b2e8 in rval_get_int (h=0x7ffd6e279500, msg=<optimized
>> out>, i=0x7ffd6e278430, rv=rv@entry=0x7f587d264d58,
>> cache=cache@entry=0x0) at rvalue.c:912
>> #10 0x000000000054261c in rval_expr_eval_int (h=h@entry=0x7ffd6e279500,
>> msg=msg@entry=0x7f587d4be8e8, res=res@entry=0x7ffd6e278430,
>> rve=rve@entry=0x7f587d264d50) at rvalue.c:1910
>> #11 0x000000000042bc91 in do_action (h=h@entry=0x7ffd6e279500,
>> a=a@entry=0x7f587d268f88, msg=msg@entry=0x7f587d4be8e8) at action.c:1030
>> #12 0x000000000042a10a in run_actions (h=h@entry=0x7ffd6e279500,
>> a=0x7f587d268f88, msg=msg@entry=0x7f587d4be8e8) at action.c:1549
>> #13 0x000000000042bcf2 in do_action (h=h@entry=0x7ffd6e279500,
>> a=a@entry=0x7f587d2691e8, msg=msg@entry=0x7f587d4be8e8) at action.c:1049
>> #14 0x000000000042a10a in run_actions (h=h@entry=0x7ffd6e279500,
>> a=0x7f587d263f48, msg=msg@entry=0x7f587d4be8e8) at action.c:1549
>> #15 0x000000000042bde0 in do_action (h=h@entry=0x7ffd6e279500,
>> a=a@entry=0x7f587d073d70, msg=msg@entry=0x7f587d4be8e8) at action.c:678
>> #16 0x000000000042a10a in run_actions (h=h@entry=0x7ffd6e279500,
>> a=a@entry=0x7f587d071698, msg=msg@entry=0x7f587d4be8e8) at action.c:1549
>> #17 0x00000000004375d0 in run_top_route (a=0x7f587d071698,
>> msg=msg@entry=0x7f587d4be8e8, c=c@entry=0x0) at action.c:1635
>> #18 0x0000000000504386 in receive_msg (buf=<optimized out>,
>> len=<optimized out>, rcv_info=<optimized out>) at receive.c:240
>> #19 0x00000000005f5bd4 in udp_rcv_loop () at udp_server.c:495
>> #20 0x00000000004b2625 in main_loop () at main.c:1600
>> #21 0x0000000000427e2b in main (argc=<optimized out>, argv=<optimized
>> out>) at main.c:2616
>>
>>
>> Relevant logmessages before crash:
>> Jul 15 10:37:55 server /usr/sbin/kamailio[12426]: NOTICE: dialog
>> [dlg_hash.c:245]: dlg_clean_run(): dialog in delete state is too old
>> (0x7f585c4a6820 ref 4)
>> Jul 15 10:37:55 server /usr/sbin/kamailio[12397]: WARNING: dialog
>> [dlg_handlers.c:1219]: dlg_onroute(): unable to find dialog for BYE with
>> route param '70f.b9d1' [3847:7579]
>> Jul 15 10:37:55 server /usr/sbin/kamailio[12395]: WARNING: dialog
>> [dlg_handlers.c:1348]: dlg_onroute(): inconsitent dlg timer data on dlg
>> 0x7f585c4a6820 [3847:7579] with clid
>> '4c41f08d317ecb9342b93f22738003f3@server' and tags 'as5f3a16b4'
'as71cb6036'
>> Jul 15 10:40:13 server /usr/sbin/kamailio[12378]: WARNING: dialog
>> [dlg_handlers.c:1219]: dlg_onroute(): unable to find dialog for BYE with
>> route param 'eb6.1e21' [1726:4833]
>> Jul 15 10:40:13 server /usr/sbin/kamailio[12376]: WARNING: dialog
>> [dlg_handlers.c:1219]: dlg_onroute(): unable to find dialog for BYE with
>> route param 'eb6.1e21' [1726:4833]
>> Jul 15 10:40:14 server /usr/sbin/kamailio[12377]: WARNING: dialog
>> [dlg_handlers.c:1219]: dlg_onroute(): unable to find dialog for BYE with
>> route param 'eb6.1e21' [1726:4833]
>> Jul 15 10:40:16 server /usr/sbin/kamailio[12377]: WARNING: dialog
>> [dlg_handlers.c:1219]: dlg_onroute(): unable to find dialog for BYE with
>> route param 'eb6.1e21' [1726:4833]
>> Jul 15 10:40:16 server /usr/sbin/kamailio[12396]: WARNING: dialog
>> [dlg_handlers.c:1219]: dlg_onroute(): unable to find dialog for BYE with
>> route param 'eb6.1e21' [1726:4833]
>> Jul 15 10:41:34 server /usr/sbin/kamailio[12396]: ERROR: sl
>> [sl_funcs.c:363]: sl_reply_error(): ERROR: sl_reply_error used: I'm
>> terribly sorry, server error occurred (1/SL)
>> Jul 15 10:41:34 server /usr/sbin/kamailio[12396]: ERROR: tm
>> [t_reply.c:533]: _reply_light(): ERROR: _reply_light: can't generate 487
>> reply when a final 487 was sent out
>> Jul 15 10:41:34 server /usr/sbin/kamailio[12396]: ERROR: tm
>> [t_lookup.c:1471]: t_unref(): ERROR: t_unref: generation of a delayed
>> stateful reply failed
>> Jul 15 10:42:25 server /usr/sbin/kamailio[12426]: NOTICE: dialog
>> [dlg_hash.c:245]: dlg_clean_run(): dialog in delete state is too old
>> (0x7f585c49d5b0 ref 4)
>> Jul 15 10:42:25 server /usr/sbin/kamailio[12426]: NOTICE: dialog
>> [dlg_hash.c:245]: dlg_clean_run(): dialog in delete state is too old
>> (0x7f585c604f18 ref 4)
>> Jul 15 10:42:25 server /usr/sbin/kamailio[12426]: NOTICE: dialog
>> [dlg_hash.c:245]: dlg_clean_run(): dialog in delete state is too old
>> (0x7f585c494b40 ref 4)
>> Jul 15 10:42:25 server /usr/sbin/kamailio[12383]: WARNING: dialog
>> [dlg_handlers.c:1348]: dlg_onroute(): inconsitent dlg timer data on dlg
>> 0x7f585c604f18 [2396:9046] with clid
>> '1b3ff5f0246fb7e82ed949544bcccbba@192.168.10.233:5060' and tags
>> 'as4d83d6f8' '5788A162-2557E04D-3E86ED15'
>> Jul 15 10:42:25 server /usr/sbin/kamailio[12395]: WARNING: dialog
>> [dlg_handlers.c:1219]: dlg_onroute(): unable to find dialog for BYE with
>> route param '6b3.c6b' [950:2924]
>> Jul 15 10:42:25 server kernel: [209851.262461] kamailio[12376]: segfault
>> at 7f6264d11378 ip 00007f585592a908 sp 00007ffd6e277330 error 4 in
>> dialog.so[7f58558e0000+88000]
>> Jul 15 10:42:25 server /usr/sbin/kamailio[12394]: WARNING: dialog
>> [dlg_handlers.c:1348]: dlg_onroute(): inconsitent dlg timer data on dlg
>> 0x7f585c49d5b0 [950:2924] with clid
>> '45fe86ce065f5543342e51ad355d1b75@server' and tags 'as152f7465'
'as4d03f77d'
>> Jul 15 10:42:26 server /usr/sbin/kamailio[12431]: CRITICAL: <core>
>> [pass_fd.c:275]: receive_fd(): EOF on 32
>> Jul 15 10:42:26 server /usr/sbin/kamailio[12370]: ALERT: <core>
>> [main.c:739]: handle_sigs(): child process 12376 exited by a signal 11
>> Jul 15 10:42:26 server /usr/sbin/kamailio[12370]: ALERT: <core>
>> [main.c:742]: handle_sigs(): core was generated
>> Jul 15 10:42:26 server /usr/sbin/kamailio[12370]: INFO: <core>
>> [main.c:754]: handle_sigs(): terminating due to SIGCHLD
>>
>>
>> Cheers,
>> Dirk
--
Daniel-Constantin Mierla
http://www.asipto.com - http://www.kamailio.org
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
18 Jul
18 Jul
8:54 a.m.
Hi Daniel,
Had to revert back to our old 4.2.5 for now, we can't cope with these
crashes. Anyway, here are all the modules currently loaded by our config.
sqlops
db_mysql
mi_fifo.so
kex.so
corex.so
tm.so
tmx.so
sl.so
rr.so
pv.so
maxfwd.so
usrloc.so
registrar.so
textops.so
siputils.so
xlog.so
sanity.so
ctl.so
cfg_rpc.so
mi_rpc.so
dispatcher.so
regex.so
lcr.so
avpops.so
uac.so
uac_redirect.so
ratelimit.so
ipops.so
And about the shared memory, is there any explenation available as to
what the different options do? And what is the default?
Cheers,
Dirk
On 07/15/2016 02:08 PM, Daniel-Constantin Mierla wrote:
The content of dlg is not valid, likely freed. Can you
run with -x qm
and see if you get new error messages?
Also, what modules are you using, specially interested in those using
dialog module, such as cnxcc or presence dialog info?!?!
Cheers,
Daniel
On 15/07/16 13:06, Dirk Teurlings - Signet B.V. wrote:
(gdb) frame 1
#1 dlg_unref (dlg=dlg@entry=0x7f585c494b40, cnt=cnt@entry=1) at
dlg_hash.c:921
921 dlg_lock( d_table, d_entry);
(gdb) p *dlg
$1 = {ref = 793790803, next = 0xa0d4b4f20303032, prev =
0x504953203a616956, h_id = 808333871, h_entry = 1346655535, state =
774976288, lifetime = 775107122, init_ts = 775435825,
start_ts = 976303410, end_ts = 808857653, dflags = 1667592763, iflags
= 1702259045, sflags = 825441636, toroute = 858927662, toroute_name = {
s = 0x6172623b3135322e <Address 0x6172623b3135322e out of bounds>,
len = 1030251374}, from_rr_nb = 894132788, tl = {next =
0x726f70723b646262, prev = 0xa0d303630353d74,
timeout = 1836020294}, callid = {s = 0x20226e776f6e6b6e <Address
0x20226e776f6e6b6e out of bounds>, len = 1885958972}, from_uri = {
s = 0x7340444c4f74656e <Address 0x7340444c4f74656e out of bounds>,
len = 1999532137}, to_uri = {s = 0x743b3e74656e2e70 <Address
0x743b3e74656e2e70 out of bounds>,
len = 1631414113}, req_uri = {s = 0x540a0d3536343766 <Address
0x540a0d3536343766 out of bounds>, len = 1008745071}, tag = {{
s = 0x363233313431332b <Address 0x363233313431332b out of bounds>,
len = 892614711}, {s = 0x2e3836312e333232 <Address 0x2e3836312e333232
out of bounds>, len = 1043608370}},
cseq = {{s = 0x663330643473613d <Address 0x663330643473613d out of
bounds>, len = 224671543}, {s = 0x3534203a44492d6c <Address
0x3534203a44492d6c out of bounds>,
len = 909665638}}, route_set = {{s = 0x3433333435356635 <Address
0x3433333435356635 out of bounds>, len = 825582898}, {
s = 0x7340353762316435 <Address 0x7340353762316435 out of bounds>,
len = 1999532137}}, contact = {{s = 0x430a0d74656e2e70 <Address
0x430a0d74656e2e70 out of bounds>,
len = 980510035}, {s = 0x65530a0d45594220 <Address
0x65530a0d45594220 out of bounds>, len = 1919252082}}, bind_addr =
{0x70696f766c772e70, 0x6c410a0d74656e2e}, cbs = {
first = 0x564e49203a776f6c, types = 742741065}, profile_links =
0x4c45434e4143202c, vars = 0x4e4f4954504f202c}
On 07/15/2016 01:00 PM, Daniel-Constantin Mierla wrote:
> From the second crash, can you get:
>
> frame 1
>
> p *dlg
>
> So far it looks like either to a double free or some buffer overflow...
>
> Cheers,
> Daniel
>
>
> On 15/07/16 10:51, Dirk Teurlings - Signet B.V. wrote:
>> Just got another segfault.
>>
>> Using host libthread_db library
"/lib/x86_64-linux-gnu/libthread_db.so.1".
>> Core was generated by `/usr/sbin/kamailio -f /etc/kamailio/kamailio.cfg
>> -P /var/run/kamailio/kamailio.'.
>> Program terminated with signal 11, Segmentation fault.
>> #0 atomic_get (v=0x7f6264d11378) at ../../mem/../atomic/atomic_common.h:74
>> 74 return atomic_get_int(&(v->val));
>> (gdb) bt
>> #0 atomic_get (v=0x7f6264d11378) at ../../mem/../atomic/atomic_common.h:74
>> #1 dlg_unref (dlg=dlg@entry=0x7f585c494b40, cnt=cnt@entry=1) at
>> dlg_hash.c:921
>> #2 0x00007f5855912802 in dlg_run_event_route
>> (dlg=dlg@entry=0x7f585c494b40, msg=msg@entry=0x7f587d4be8e8,
>> ostate=<optimized out>, nstate=<optimized out>) at
dlg_handlers.c:1630
>> #3 0x00007f585591416a in dlg_onroute (req=0x7f587d4be8e8,
>> route_params=<optimized out>, param=<optimized out>) at
dlg_handlers.c:1307
>> #4 0x00007f585965b0e2 in run_rr_callbacks
>> (req=req@entry=0x7f587d4be8e8, rr_param=rr_param@entry=0x7f58598677a0)
>> at rr_cb.c:96
>> #5 0x00007f58596452c5 in after_loose (_m=0x7f587d4be8e8, preloaded=0)
>> at loose.c:919
>> #6 0x000000000042b618 in do_action (h=h@entry=0x7ffd6e277fd0,
>> a=a@entry=0x7f587d264338, msg=msg@entry=0x7f587d4be8e8) at action.c:1060
>> #7 0x000000000042a10a in run_actions (h=h@entry=0x7ffd6e277fd0,
>> a=0x7f587d264338, msg=0x7f587d4be8e8) at action.c:1549
>> #8 0x0000000000437544 in run_actions_safe (h=h@entry=0x7ffd6e279500,
>> a=<optimized out>, msg=<optimized out>) at action.c:1614
>> #9 0x000000000053b2e8 in rval_get_int (h=0x7ffd6e279500, msg=<optimized
>> out>, i=0x7ffd6e278430, rv=rv@entry=0x7f587d264d58,
>> cache=cache@entry=0x0) at rvalue.c:912
>> #10 0x000000000054261c in rval_expr_eval_int (h=h@entry=0x7ffd6e279500,
>> msg=msg@entry=0x7f587d4be8e8, res=res@entry=0x7ffd6e278430,
>> rve=rve@entry=0x7f587d264d50) at rvalue.c:1910
>> #11 0x000000000042bc91 in do_action (h=h@entry=0x7ffd6e279500,
>> a=a@entry=0x7f587d268f88, msg=msg@entry=0x7f587d4be8e8) at action.c:1030
>> #12 0x000000000042a10a in run_actions (h=h@entry=0x7ffd6e279500,
>> a=0x7f587d268f88, msg=msg@entry=0x7f587d4be8e8) at action.c:1549
>> #13 0x000000000042bcf2 in do_action (h=h@entry=0x7ffd6e279500,
>> a=a@entry=0x7f587d2691e8, msg=msg@entry=0x7f587d4be8e8) at action.c:1049
>> #14 0x000000000042a10a in run_actions (h=h@entry=0x7ffd6e279500,
>> a=0x7f587d263f48, msg=msg@entry=0x7f587d4be8e8) at action.c:1549
>> #15 0x000000000042bde0 in do_action (h=h@entry=0x7ffd6e279500,
>> a=a@entry=0x7f587d073d70, msg=msg@entry=0x7f587d4be8e8) at action.c:678
>> #16 0x000000000042a10a in run_actions (h=h@entry=0x7ffd6e279500,
>> a=a@entry=0x7f587d071698, msg=msg@entry=0x7f587d4be8e8) at action.c:1549
>> #17 0x00000000004375d0 in run_top_route (a=0x7f587d071698,
>> msg=msg@entry=0x7f587d4be8e8, c=c@entry=0x0) at action.c:1635
>> #18 0x0000000000504386 in receive_msg (buf=<optimized out>,
>> len=<optimized out>, rcv_info=<optimized out>) at receive.c:240
>> #19 0x00000000005f5bd4 in udp_rcv_loop () at udp_server.c:495
>> #20 0x00000000004b2625 in main_loop () at main.c:1600
>> #21 0x0000000000427e2b in main (argc=<optimized out>, argv=<optimized
>> out>) at main.c:2616
>>
>>
>> Relevant logmessages before crash:
>> Jul 15 10:37:55 server /usr/sbin/kamailio[12426]: NOTICE: dialog
>> [dlg_hash.c:245]: dlg_clean_run(): dialog in delete state is too old
>> (0x7f585c4a6820 ref 4)
>> Jul 15 10:37:55 server /usr/sbin/kamailio[12397]: WARNING: dialog
>> [dlg_handlers.c:1219]: dlg_onroute(): unable to find dialog for BYE with
>> route param '70f.b9d1' [3847:7579]
>> Jul 15 10:37:55 server /usr/sbin/kamailio[12395]: WARNING: dialog
>> [dlg_handlers.c:1348]: dlg_onroute(): inconsitent dlg timer data on dlg
>> 0x7f585c4a6820 [3847:7579] with clid
>> '4c41f08d317ecb9342b93f22738003f3@server' and tags 'as5f3a16b4'
'as71cb6036'
>> Jul 15 10:40:13 server /usr/sbin/kamailio[12378]: WARNING: dialog
>> [dlg_handlers.c:1219]: dlg_onroute(): unable to find dialog for BYE with
>> route param 'eb6.1e21' [1726:4833]
>> Jul 15 10:40:13 server /usr/sbin/kamailio[12376]: WARNING: dialog
>> [dlg_handlers.c:1219]: dlg_onroute(): unable to find dialog for BYE with
>> route param 'eb6.1e21' [1726:4833]
>> Jul 15 10:40:14 server /usr/sbin/kamailio[12377]: WARNING: dialog
>> [dlg_handlers.c:1219]: dlg_onroute(): unable to find dialog for BYE with
>> route param 'eb6.1e21' [1726:4833]
>> Jul 15 10:40:16 server /usr/sbin/kamailio[12377]: WARNING: dialog
>> [dlg_handlers.c:1219]: dlg_onroute(): unable to find dialog for BYE with
>> route param 'eb6.1e21' [1726:4833]
>> Jul 15 10:40:16 server /usr/sbin/kamailio[12396]: WARNING: dialog
>> [dlg_handlers.c:1219]: dlg_onroute(): unable to find dialog for BYE with
>> route param 'eb6.1e21' [1726:4833]
>> Jul 15 10:41:34 server /usr/sbin/kamailio[12396]: ERROR: sl
>> [sl_funcs.c:363]: sl_reply_error(): ERROR: sl_reply_error used: I'm
>> terribly sorry, server error occurred (1/SL)
>> Jul 15 10:41:34 server /usr/sbin/kamailio[12396]: ERROR: tm
>> [t_reply.c:533]: _reply_light(): ERROR: _reply_light: can't generate 487
>> reply when a final 487 was sent out
>> Jul 15 10:41:34 server /usr/sbin/kamailio[12396]: ERROR: tm
>> [t_lookup.c:1471]: t_unref(): ERROR: t_unref: generation of a delayed
>> stateful reply failed
>> Jul 15 10:42:25 server /usr/sbin/kamailio[12426]: NOTICE: dialog
>> [dlg_hash.c:245]: dlg_clean_run(): dialog in delete state is too old
>> (0x7f585c49d5b0 ref 4)
>> Jul 15 10:42:25 server /usr/sbin/kamailio[12426]: NOTICE: dialog
>> [dlg_hash.c:245]: dlg_clean_run(): dialog in delete state is too old
>> (0x7f585c604f18 ref 4)
>> Jul 15 10:42:25 server /usr/sbin/kamailio[12426]: NOTICE: dialog
>> [dlg_hash.c:245]: dlg_clean_run(): dialog in delete state is too old
>> (0x7f585c494b40 ref 4)
>> Jul 15 10:42:25 server /usr/sbin/kamailio[12383]: WARNING: dialog
>> [dlg_handlers.c:1348]: dlg_onroute(): inconsitent dlg timer data on dlg
>> 0x7f585c604f18 [2396:9046] with clid
>> '1b3ff5f0246fb7e82ed949544bcccbba@192.168.10.233:5060' and tags
>> 'as4d83d6f8' '5788A162-2557E04D-3E86ED15'
>> Jul 15 10:42:25 server /usr/sbin/kamailio[12395]: WARNING: dialog
>> [dlg_handlers.c:1219]: dlg_onroute(): unable to find dialog for BYE with
>> route param '6b3.c6b' [950:2924]
>> Jul 15 10:42:25 server kernel: [209851.262461] kamailio[12376]: segfault
>> at 7f6264d11378 ip 00007f585592a908 sp 00007ffd6e277330 error 4 in
>> dialog.so[7f58558e0000+88000]
>> Jul 15 10:42:25 server /usr/sbin/kamailio[12394]: WARNING: dialog
>> [dlg_handlers.c:1348]: dlg_onroute(): inconsitent dlg timer data on dlg
>> 0x7f585c49d5b0 [950:2924] with clid
>> '45fe86ce065f5543342e51ad355d1b75@server' and tags 'as152f7465'
'as4d03f77d'
>> Jul 15 10:42:26 server /usr/sbin/kamailio[12431]: CRITICAL: <core>
>> [pass_fd.c:275]: receive_fd(): EOF on 32
>> Jul 15 10:42:26 server /usr/sbin/kamailio[12370]: ALERT: <core>
>> [main.c:739]: handle_sigs(): child process 12376 exited by a signal 11
>> Jul 15 10:42:26 server /usr/sbin/kamailio[12370]: ALERT: <core>
>> [main.c:742]: handle_sigs(): core was generated
>> Jul 15 10:42:26 server /usr/sbin/kamailio[12370]: INFO: <core>
>> [main.c:754]: handle_sigs(): terminating due to SIGCHLD
>>
>>
>> Cheers,
>> Dirk
10:49 a.m.
Hello,
have you run with -x qm before reverting to 4.2?
Is the same config you run with 4.2 and 4.4?
The version 4.2.8 has qm as default memory manager. In 4.4, fm is the
default one but qm can be selected at startup with -x. The main
difference is that qm has more safety checks for detecting double free
and buffer overflows...
Cheers,
Daniel
On 18/07/16 08:54, Dirk Teurlings - Signet B.V. wrote:
Hi Daniel,
Had to revert back to our old 4.2.5 for now, we can't cope with these
crashes. Anyway, here are all the modules currently loaded by our config.
sqlops
db_mysql
mi_fifo.so
kex.so
corex.so
tm.so
tmx.so
sl.so
rr.so
pv.so
maxfwd.so
usrloc.so
registrar.so
textops.so
siputils.so
xlog.so
sanity.so
ctl.so
cfg_rpc.so
mi_rpc.so
dispatcher.so
regex.so
lcr.so
avpops.so
uac.so
uac_redirect.so
ratelimit.so
ipops.so
And about the shared memory, is there any explenation available as to
what the different options do? And what is the default?
Cheers,
Dirk
On 07/15/2016 02:08 PM, Daniel-Constantin Mierla wrote:
> The content of dlg is not valid, likely freed. Can you run with -x qm
> and see if you get new error messages?
>
> Also, what modules are you using, specially interested in those using
> dialog module, such as cnxcc or presence dialog info?!?!
>
> Cheers,
> Daniel
>
>
> On 15/07/16 13:06, Dirk Teurlings - Signet B.V. wrote:
>> (gdb) frame 1
>> #1 dlg_unref (dlg=dlg@entry=0x7f585c494b40, cnt=cnt@entry=1) at
>> dlg_hash.c:921
>> 921 dlg_lock( d_table, d_entry);
>> (gdb) p *dlg
>> $1 = {ref = 793790803, next = 0xa0d4b4f20303032, prev =
>> 0x504953203a616956, h_id = 808333871, h_entry = 1346655535, state =
>> 774976288, lifetime = 775107122, init_ts = 775435825,
>> start_ts = 976303410, end_ts = 808857653, dflags = 1667592763, iflags
>> = 1702259045, sflags = 825441636, toroute = 858927662, toroute_name = {
>> s = 0x6172623b3135322e <Address 0x6172623b3135322e out of bounds>,
>> len = 1030251374}, from_rr_nb = 894132788, tl = {next =
>> 0x726f70723b646262, prev = 0xa0d303630353d74,
>> timeout = 1836020294}, callid = {s = 0x20226e776f6e6b6e <Address
>> 0x20226e776f6e6b6e out of bounds>, len = 1885958972}, from_uri = {
>> s = 0x7340444c4f74656e <Address 0x7340444c4f74656e out of bounds>,
>> len = 1999532137}, to_uri = {s = 0x743b3e74656e2e70 <Address
>> 0x743b3e74656e2e70 out of bounds>,
>> len = 1631414113}, req_uri = {s = 0x540a0d3536343766 <Address
>> 0x540a0d3536343766 out of bounds>, len = 1008745071}, tag = {{
>> s = 0x363233313431332b <Address 0x363233313431332b out of bounds>,
>> len = 892614711}, {s = 0x2e3836312e333232 <Address 0x2e3836312e333232
>> out of bounds>, len = 1043608370}},
>> cseq = {{s = 0x663330643473613d <Address 0x663330643473613d out of
>> bounds>, len = 224671543}, {s = 0x3534203a44492d6c <Address
>> 0x3534203a44492d6c out of bounds>,
>> len = 909665638}}, route_set = {{s = 0x3433333435356635 <Address
>> 0x3433333435356635 out of bounds>, len = 825582898}, {
>> s = 0x7340353762316435 <Address 0x7340353762316435 out of bounds>,
>> len = 1999532137}}, contact = {{s = 0x430a0d74656e2e70 <Address
>> 0x430a0d74656e2e70 out of bounds>,
>> len = 980510035}, {s = 0x65530a0d45594220 <Address
>> 0x65530a0d45594220 out of bounds>, len = 1919252082}}, bind_addr =
>> {0x70696f766c772e70, 0x6c410a0d74656e2e}, cbs = {
>> first = 0x564e49203a776f6c, types = 742741065}, profile_links =
>> 0x4c45434e4143202c, vars = 0x4e4f4954504f202c}
>>
>>
>>
>>
>> On 07/15/2016 01:00 PM, Daniel-Constantin Mierla wrote:
>>> From the second crash, can you get:
>>>
>>> frame 1
>>>
>>> p *dlg
>>>
>>> So far it looks like either to a double free or some buffer overflow...
>>>
>>> Cheers,
>>> Daniel
>>>
>>>
>>> On 15/07/16 10:51, Dirk Teurlings - Signet B.V. wrote:
>>>> Just got another segfault.
>>>>
>>>> Using host libthread_db library
"/lib/x86_64-linux-gnu/libthread_db.so.1".
>>>> Core was generated by `/usr/sbin/kamailio -f /etc/kamailio/kamailio.cfg
>>>> -P /var/run/kamailio/kamailio.'.
>>>> Program terminated with signal 11, Segmentation fault.
>>>> #0 atomic_get (v=0x7f6264d11378) at
../../mem/../atomic/atomic_common.h:74
>>>> 74 return atomic_get_int(&(v->val));
>>>> (gdb) bt
>>>> #0 atomic_get (v=0x7f6264d11378) at
../../mem/../atomic/atomic_common.h:74
>>>> #1 dlg_unref (dlg=dlg@entry=0x7f585c494b40, cnt=cnt@entry=1) at
>>>> dlg_hash.c:921
>>>> #2 0x00007f5855912802 in dlg_run_event_route
>>>> (dlg=dlg@entry=0x7f585c494b40, msg=msg@entry=0x7f587d4be8e8,
>>>> ostate=<optimized out>, nstate=<optimized out>) at
dlg_handlers.c:1630
>>>> #3 0x00007f585591416a in dlg_onroute (req=0x7f587d4be8e8,
>>>> route_params=<optimized out>, param=<optimized out>) at
dlg_handlers.c:1307
>>>> #4 0x00007f585965b0e2 in run_rr_callbacks
>>>> (req=req@entry=0x7f587d4be8e8, rr_param=rr_param@entry=0x7f58598677a0)
>>>> at rr_cb.c:96
>>>> #5 0x00007f58596452c5 in after_loose (_m=0x7f587d4be8e8, preloaded=0)
>>>> at loose.c:919
>>>> #6 0x000000000042b618 in do_action (h=h@entry=0x7ffd6e277fd0,
>>>> a=a@entry=0x7f587d264338, msg=msg@entry=0x7f587d4be8e8) at action.c:1060
>>>> #7 0x000000000042a10a in run_actions (h=h@entry=0x7ffd6e277fd0,
>>>> a=0x7f587d264338, msg=0x7f587d4be8e8) at action.c:1549
>>>> #8 0x0000000000437544 in run_actions_safe (h=h@entry=0x7ffd6e279500,
>>>> a=<optimized out>, msg=<optimized out>) at action.c:1614
>>>> #9 0x000000000053b2e8 in rval_get_int (h=0x7ffd6e279500,
msg=<optimized
>>>> out>, i=0x7ffd6e278430, rv=rv@entry=0x7f587d264d58,
>>>> cache=cache@entry=0x0) at rvalue.c:912
>>>> #10 0x000000000054261c in rval_expr_eval_int (h=h@entry=0x7ffd6e279500,
>>>> msg=msg@entry=0x7f587d4be8e8, res=res@entry=0x7ffd6e278430,
>>>> rve=rve@entry=0x7f587d264d50) at rvalue.c:1910
>>>> #11 0x000000000042bc91 in do_action (h=h@entry=0x7ffd6e279500,
>>>> a=a@entry=0x7f587d268f88, msg=msg@entry=0x7f587d4be8e8) at action.c:1030
>>>> #12 0x000000000042a10a in run_actions (h=h@entry=0x7ffd6e279500,
>>>> a=0x7f587d268f88, msg=msg@entry=0x7f587d4be8e8) at action.c:1549
>>>> #13 0x000000000042bcf2 in do_action (h=h@entry=0x7ffd6e279500,
>>>> a=a@entry=0x7f587d2691e8, msg=msg@entry=0x7f587d4be8e8) at action.c:1049
>>>> #14 0x000000000042a10a in run_actions (h=h@entry=0x7ffd6e279500,
>>>> a=0x7f587d263f48, msg=msg@entry=0x7f587d4be8e8) at action.c:1549
>>>> #15 0x000000000042bde0 in do_action (h=h@entry=0x7ffd6e279500,
>>>> a=a@entry=0x7f587d073d70, msg=msg@entry=0x7f587d4be8e8) at action.c:678
>>>> #16 0x000000000042a10a in run_actions (h=h@entry=0x7ffd6e279500,
>>>> a=a@entry=0x7f587d071698, msg=msg@entry=0x7f587d4be8e8) at action.c:1549
>>>> #17 0x00000000004375d0 in run_top_route (a=0x7f587d071698,
>>>> msg=msg@entry=0x7f587d4be8e8, c=c@entry=0x0) at action.c:1635
>>>> #18 0x0000000000504386 in receive_msg (buf=<optimized out>,
>>>> len=<optimized out>, rcv_info=<optimized out>) at
receive.c:240
>>>> #19 0x00000000005f5bd4 in udp_rcv_loop () at udp_server.c:495
>>>> #20 0x00000000004b2625 in main_loop () at main.c:1600
>>>> #21 0x0000000000427e2b in main (argc=<optimized out>,
argv=<optimized
>>>> out>) at main.c:2616
>>>>
>>>>
>>>> Relevant logmessages before crash:
>>>> Jul 15 10:37:55 server /usr/sbin/kamailio[12426]: NOTICE: dialog
>>>> [dlg_hash.c:245]: dlg_clean_run(): dialog in delete state is too old
>>>> (0x7f585c4a6820 ref 4)
>>>> Jul 15 10:37:55 server /usr/sbin/kamailio[12397]: WARNING: dialog
>>>> [dlg_handlers.c:1219]: dlg_onroute(): unable to find dialog for BYE with
>>>> route param '70f.b9d1' [3847:7579]
>>>> Jul 15 10:37:55 server /usr/sbin/kamailio[12395]: WARNING: dialog
>>>> [dlg_handlers.c:1348]: dlg_onroute(): inconsitent dlg timer data on dlg
>>>> 0x7f585c4a6820 [3847:7579] with clid
>>>> '4c41f08d317ecb9342b93f22738003f3@server' and tags
'as5f3a16b4' 'as71cb6036'
>>>> Jul 15 10:40:13 server /usr/sbin/kamailio[12378]: WARNING: dialog
>>>> [dlg_handlers.c:1219]: dlg_onroute(): unable to find dialog for BYE with
>>>> route param 'eb6.1e21' [1726:4833]
>>>> Jul 15 10:40:13 server /usr/sbin/kamailio[12376]: WARNING: dialog
>>>> [dlg_handlers.c:1219]: dlg_onroute(): unable to find dialog for BYE with
>>>> route param 'eb6.1e21' [1726:4833]
>>>> Jul 15 10:40:14 server /usr/sbin/kamailio[12377]: WARNING: dialog
>>>> [dlg_handlers.c:1219]: dlg_onroute(): unable to find dialog for BYE with
>>>> route param 'eb6.1e21' [1726:4833]
>>>> Jul 15 10:40:16 server /usr/sbin/kamailio[12377]: WARNING: dialog
>>>> [dlg_handlers.c:1219]: dlg_onroute(): unable to find dialog for BYE with
>>>> route param 'eb6.1e21' [1726:4833]
>>>> Jul 15 10:40:16 server /usr/sbin/kamailio[12396]: WARNING: dialog
>>>> [dlg_handlers.c:1219]: dlg_onroute(): unable to find dialog for BYE with
>>>> route param 'eb6.1e21' [1726:4833]
>>>> Jul 15 10:41:34 server /usr/sbin/kamailio[12396]: ERROR: sl
>>>> [sl_funcs.c:363]: sl_reply_error(): ERROR: sl_reply_error used: I'm
>>>> terribly sorry, server error occurred (1/SL)
>>>> Jul 15 10:41:34 server /usr/sbin/kamailio[12396]: ERROR: tm
>>>> [t_reply.c:533]: _reply_light(): ERROR: _reply_light: can't generate
487
>>>> reply when a final 487 was sent out
>>>> Jul 15 10:41:34 server /usr/sbin/kamailio[12396]: ERROR: tm
>>>> [t_lookup.c:1471]: t_unref(): ERROR: t_unref: generation of a delayed
>>>> stateful reply failed
>>>> Jul 15 10:42:25 server /usr/sbin/kamailio[12426]: NOTICE: dialog
>>>> [dlg_hash.c:245]: dlg_clean_run(): dialog in delete state is too old
>>>> (0x7f585c49d5b0 ref 4)
>>>> Jul 15 10:42:25 server /usr/sbin/kamailio[12426]: NOTICE: dialog
>>>> [dlg_hash.c:245]: dlg_clean_run(): dialog in delete state is too old
>>>> (0x7f585c604f18 ref 4)
>>>> Jul 15 10:42:25 server /usr/sbin/kamailio[12426]: NOTICE: dialog
>>>> [dlg_hash.c:245]: dlg_clean_run(): dialog in delete state is too old
>>>> (0x7f585c494b40 ref 4)
>>>> Jul 15 10:42:25 server /usr/sbin/kamailio[12383]: WARNING: dialog
>>>> [dlg_handlers.c:1348]: dlg_onroute(): inconsitent dlg timer data on dlg
>>>> 0x7f585c604f18 [2396:9046] with clid
>>>> '1b3ff5f0246fb7e82ed949544bcccbba@192.168.10.233:5060' and tags
>>>> 'as4d83d6f8' '5788A162-2557E04D-3E86ED15'
>>>> Jul 15 10:42:25 server /usr/sbin/kamailio[12395]: WARNING: dialog
>>>> [dlg_handlers.c:1219]: dlg_onroute(): unable to find dialog for BYE with
>>>> route param '6b3.c6b' [950:2924]
>>>> Jul 15 10:42:25 server kernel: [209851.262461] kamailio[12376]: segfault
>>>> at 7f6264d11378 ip 00007f585592a908 sp 00007ffd6e277330 error 4 in
>>>> dialog.so[7f58558e0000+88000]
>>>> Jul 15 10:42:25 server /usr/sbin/kamailio[12394]: WARNING: dialog
>>>> [dlg_handlers.c:1348]: dlg_onroute(): inconsitent dlg timer data on dlg
>>>> 0x7f585c49d5b0 [950:2924] with clid
>>>> '45fe86ce065f5543342e51ad355d1b75@server' and tags
'as152f7465' 'as4d03f77d'
>>>> Jul 15 10:42:26 server /usr/sbin/kamailio[12431]: CRITICAL: <core>
>>>> [pass_fd.c:275]: receive_fd(): EOF on 32
>>>> Jul 15 10:42:26 server /usr/sbin/kamailio[12370]: ALERT: <core>
>>>> [main.c:739]: handle_sigs(): child process 12376 exited by a signal 11
>>>> Jul 15 10:42:26 server /usr/sbin/kamailio[12370]: ALERT: <core>
>>>> [main.c:742]: handle_sigs(): core was generated
>>>> Jul 15 10:42:26 server /usr/sbin/kamailio[12370]: INFO: <core>
>>>> [main.c:754]: handle_sigs(): terminating due to SIGCHLD
>>>>
>>>>
>>>> Cheers,
>>>> Dirk
--
Daniel-Constantin Mierla
http://www.asipto.com - http://www.kamailio.org
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
11:23 a.m.
On 07/18/2016 10:49 AM, Daniel-Constantin Mierla wrote:
have you run with -x qm before reverting to 4.2?
No we didn't, as I didn't know what the effects would be, it's a live
server so we don't want to try changes before testing.
Is the same config you run with 4.2 and 4.4?
Not
at the moment, but we developed the new routes in 4.2 first, then
upgraded the testplatform to 4.4 without issues. Though the testplatform
is not identical in hardware to the live system, this looked promising.
The version 4.2.8 has qm as default memory manager. In
4.4, fm is the
default one but qm can be selected at startup with -x. The main
difference is that qm has more safety checks for detecting double free
and buffer overflows...
Right, that sounds very plausible then... Upgrading
"back" to 4.4 now
will have to wait a bit, we don't want to experience the same issues
again. That being said, it's difficult to track what causes this exactly
if we can't reproduce it on the testplatform. Unfortunately I'm not at
liberty to try these things on the running servers.
Anyway, can I get any more information out of the backtraces to help
resolve this issue with the new memory manager as well? Or do you need
more info for that?
Cheers,
Dirk
11:58 a.m.
On 18/07/16 11:23, Dirk Teurlings - Signet B.V. wrote:
I see -- can you describe a bit the changes you did from old config to
the new one (e.g., new modules and routing actions)? If you want to keep
that private, email those details directly to me.
If you didn't get other crashes, I can't think of anything else that can
help from the backtrace.
Cheers,
Daniel
--
Daniel-Constantin Mierla
http://www.asipto.com - http://www.kamailio.org
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
On 07/18/2016 10:49 AM, Daniel-Constantin Mierla
wrote:
OK -- just to have full picture -- did you experienced more crashes than
the two ones you reported?
have you run with -x qm before reverting to 4.2?
No we didn't, as I didn't know what the effects would be, it's a live
server so we don't want to try changes before testing. Is the same config you run with 4.2 and 4.4?
Not at the moment, but we developed the new routes in 4.2 first, then
upgraded the testplatform to 4.4 without issues. Though the testplatform
is not identical in hardware to the live system, this looked promising. The version 4.2.8 has qm as default memory
manager. In 4.4, fm is the
default one but qm can be selected at startup with -x. The main
difference is that qm has more safety checks for detecting double free
and buffer overflows...
Right, that sounds very plausible then... Upgrading
"back" to 4.4 now
will have to wait a bit, we don't want to experience the same issues
again. That being said, it's difficult to track what causes this exactly
if we can't reproduce it on the testplatform. Unfortunately I'm not at
liberty to try these things on the running servers.
Anyway, can I get any more information out of the backtraces to help
resolve this issue with the new memory manager as well? Or do you need
more info for that?
1:38 p.m.
On 07/18/2016 11:58 AM, Daniel-Constantin Mierla wrote:
OK -- just to have full picture -- did you experienced more crashes than
the two ones you reported?
Yes, we experienced three more crashes
I will send you these gdb bt full crashes directly.
I see -- can you describe a bit the changes you did
from old config to
the new one (e.g., new modules and routing actions)? If you want to keep
that private, email those details directly to me.
Main change is that CNXCC forked
modules are no longer used. Terminating
calls is now dealt with via dlg.end_dlg h_entry h_id RPC. So in theory,
it's all a lot simpler routing wise now.
By the way, which memory manager is default for 4.3? Same as 4.2 or 4.4?
Hope to find the culprit for this somehow!
Cheers,
Dirk
2851
days inactive
2862
days old
12 comments
2 participants
participants (2)
-
Daniel-Constantin Mierla -
Dirk Teurlings - Signet B.V.