[SR-Users] Segfault in dialog.so with 4.4.2 stable

Fri Jul 15 13:00:06 CEST 2016

>From the second crash, can you get:

frame 1

p *dlg

So far it looks like either to a double free or some buffer overflow...

Cheers,
Daniel

On 15/07/16 10:51, Dirk Teurlings - Signet B.V. wrote:
> Just got another segfault.
>
> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
> Core was generated by `/usr/sbin/kamailio -f /etc/kamailio/kamailio.cfg
> -P /var/run/kamailio/kamailio.'.
> Program terminated with signal 11, Segmentation fault.
> #0  atomic_get (v=0x7f6264d11378) at ../../mem/../atomic/atomic_common.h:74
> 74		return atomic_get_int(&(v->val));
> (gdb) bt
> #0  atomic_get (v=0x7f6264d11378) at ../../mem/../atomic/atomic_common.h:74
> #1  dlg_unref (dlg=dlg at entry=0x7f585c494b40, cnt=cnt at entry=1) at
> dlg_hash.c:921
> #2  0x00007f5855912802 in dlg_run_event_route
> (dlg=dlg at entry=0x7f585c494b40, msg=msg at entry=0x7f587d4be8e8,
> ostate=<optimized out>, nstate=<optimized out>) at dlg_handlers.c:1630
> #3  0x00007f585591416a in dlg_onroute (req=0x7f587d4be8e8,
> route_params=<optimized out>, param=<optimized out>) at dlg_handlers.c:1307
> #4  0x00007f585965b0e2 in run_rr_callbacks
> (req=req at entry=0x7f587d4be8e8, rr_param=rr_param at entry=0x7f58598677a0)
> at rr_cb.c:96
> #5  0x00007f58596452c5 in after_loose (_m=0x7f587d4be8e8, preloaded=0)
> at loose.c:919
> #6  0x000000000042b618 in do_action (h=h at entry=0x7ffd6e277fd0,
> a=a at entry=0x7f587d264338, msg=msg at entry=0x7f587d4be8e8) at action.c:1060
> #7  0x000000000042a10a in run_actions (h=h at entry=0x7ffd6e277fd0,
> a=0x7f587d264338, msg=0x7f587d4be8e8) at action.c:1549
> #8  0x0000000000437544 in run_actions_safe (h=h at entry=0x7ffd6e279500,
> a=<optimized out>, msg=<optimized out>) at action.c:1614
> #9  0x000000000053b2e8 in rval_get_int (h=0x7ffd6e279500, msg=<optimized
> out>, i=0x7ffd6e278430, rv=rv at entry=0x7f587d264d58,
> cache=cache at entry=0x0) at rvalue.c:912
> #10 0x000000000054261c in rval_expr_eval_int (h=h at entry=0x7ffd6e279500,
> msg=msg at entry=0x7f587d4be8e8, res=res at entry=0x7ffd6e278430,
> rve=rve at entry=0x7f587d264d50) at rvalue.c:1910
> #11 0x000000000042bc91 in do_action (h=h at entry=0x7ffd6e279500,
> a=a at entry=0x7f587d268f88, msg=msg at entry=0x7f587d4be8e8) at action.c:1030
> #12 0x000000000042a10a in run_actions (h=h at entry=0x7ffd6e279500,
> a=0x7f587d268f88, msg=msg at entry=0x7f587d4be8e8) at action.c:1549
> #13 0x000000000042bcf2 in do_action (h=h at entry=0x7ffd6e279500,
> a=a at entry=0x7f587d2691e8, msg=msg at entry=0x7f587d4be8e8) at action.c:1049
> #14 0x000000000042a10a in run_actions (h=h at entry=0x7ffd6e279500,
> a=0x7f587d263f48, msg=msg at entry=0x7f587d4be8e8) at action.c:1549
> #15 0x000000000042bde0 in do_action (h=h at entry=0x7ffd6e279500,
> a=a at entry=0x7f587d073d70, msg=msg at entry=0x7f587d4be8e8) at action.c:678
> #16 0x000000000042a10a in run_actions (h=h at entry=0x7ffd6e279500,
> a=a at entry=0x7f587d071698, msg=msg at entry=0x7f587d4be8e8) at action.c:1549
> #17 0x00000000004375d0 in run_top_route (a=0x7f587d071698,
> msg=msg at entry=0x7f587d4be8e8, c=c at entry=0x0) at action.c:1635
> #18 0x0000000000504386 in receive_msg (buf=<optimized out>,
> len=<optimized out>, rcv_info=<optimized out>) at receive.c:240
> #19 0x00000000005f5bd4 in udp_rcv_loop () at udp_server.c:495
> #20 0x00000000004b2625 in main_loop () at main.c:1600
> #21 0x0000000000427e2b in main (argc=<optimized out>, argv=<optimized
> out>) at main.c:2616
>
>
> Relevant logmessages before crash:
> Jul 15 10:37:55 server /usr/sbin/kamailio[12426]: NOTICE: dialog
> [dlg_hash.c:245]: dlg_clean_run(): dialog in delete state is too old
> (0x7f585c4a6820 ref 4)
> Jul 15 10:37:55 server /usr/sbin/kamailio[12397]: WARNING: dialog
> [dlg_handlers.c:1219]: dlg_onroute(): unable to find dialog for BYE with
> route param '70f.b9d1' [3847:7579]
> Jul 15 10:37:55 server /usr/sbin/kamailio[12395]: WARNING: dialog
> [dlg_handlers.c:1348]: dlg_onroute(): inconsitent dlg timer data on dlg
> 0x7f585c4a6820 [3847:7579] with clid
> '4c41f08d317ecb9342b93f22738003f3 at server' and tags 'as5f3a16b4' 'as71cb6036'
> Jul 15 10:40:13 server /usr/sbin/kamailio[12378]: WARNING: dialog
> [dlg_handlers.c:1219]: dlg_onroute(): unable to find dialog for BYE with
> route param 'eb6.1e21' [1726:4833]
> Jul 15 10:40:13 server /usr/sbin/kamailio[12376]: WARNING: dialog
> [dlg_handlers.c:1219]: dlg_onroute(): unable to find dialog for BYE with
> route param 'eb6.1e21' [1726:4833]
> Jul 15 10:40:14 server /usr/sbin/kamailio[12377]: WARNING: dialog
> [dlg_handlers.c:1219]: dlg_onroute(): unable to find dialog for BYE with
> route param 'eb6.1e21' [1726:4833]
> Jul 15 10:40:16 server /usr/sbin/kamailio[12377]: WARNING: dialog
> [dlg_handlers.c:1219]: dlg_onroute(): unable to find dialog for BYE with
> route param 'eb6.1e21' [1726:4833]
> Jul 15 10:40:16 server /usr/sbin/kamailio[12396]: WARNING: dialog
> [dlg_handlers.c:1219]: dlg_onroute(): unable to find dialog for BYE with
> route param 'eb6.1e21' [1726:4833]
> Jul 15 10:41:34 server /usr/sbin/kamailio[12396]: ERROR: sl
> [sl_funcs.c:363]: sl_reply_error(): ERROR: sl_reply_error used: I'm
> terribly sorry, server error occurred (1/SL)
> Jul 15 10:41:34 server /usr/sbin/kamailio[12396]: ERROR: tm
> [t_reply.c:533]: _reply_light(): ERROR: _reply_light: can't generate 487
> reply when a final 487 was sent out
> Jul 15 10:41:34 server /usr/sbin/kamailio[12396]: ERROR: tm
> [t_lookup.c:1471]: t_unref(): ERROR: t_unref: generation of a delayed
> stateful reply failed
> Jul 15 10:42:25 server /usr/sbin/kamailio[12426]: NOTICE: dialog
> [dlg_hash.c:245]: dlg_clean_run(): dialog in delete state is too old
> (0x7f585c49d5b0 ref 4)
> Jul 15 10:42:25 server /usr/sbin/kamailio[12426]: NOTICE: dialog
> [dlg_hash.c:245]: dlg_clean_run(): dialog in delete state is too old
> (0x7f585c604f18 ref 4)
> Jul 15 10:42:25 server /usr/sbin/kamailio[12426]: NOTICE: dialog
> [dlg_hash.c:245]: dlg_clean_run(): dialog in delete state is too old
> (0x7f585c494b40 ref 4)
> Jul 15 10:42:25 server /usr/sbin/kamailio[12383]: WARNING: dialog
> [dlg_handlers.c:1348]: dlg_onroute(): inconsitent dlg timer data on dlg
> 0x7f585c604f18 [2396:9046] with clid
> '1b3ff5f0246fb7e82ed949544bcccbba at 192.168.10.233:5060' and tags
> 'as4d83d6f8' '5788A162-2557E04D-3E86ED15'
> Jul 15 10:42:25 server /usr/sbin/kamailio[12395]: WARNING: dialog
> [dlg_handlers.c:1219]: dlg_onroute(): unable to find dialog for BYE with
> route param '6b3.c6b' [950:2924]
> Jul 15 10:42:25 server kernel: [209851.262461] kamailio[12376]: segfault
> at 7f6264d11378 ip 00007f585592a908 sp 00007ffd6e277330 error 4 in
> dialog.so[7f58558e0000+88000]
> Jul 15 10:42:25 server /usr/sbin/kamailio[12394]: WARNING: dialog
> [dlg_handlers.c:1348]: dlg_onroute(): inconsitent dlg timer data on dlg
> 0x7f585c49d5b0 [950:2924] with clid
> '45fe86ce065f5543342e51ad355d1b75 at server' and tags 'as152f7465' 'as4d03f77d'
> Jul 15 10:42:26 server /usr/sbin/kamailio[12431]: CRITICAL: <core>
> [pass_fd.c:275]: receive_fd(): EOF on 32
> Jul 15 10:42:26 server /usr/sbin/kamailio[12370]: ALERT: <core>
> [main.c:739]: handle_sigs(): child process 12376 exited by a signal 11
> Jul 15 10:42:26 server /usr/sbin/kamailio[12370]: ALERT: <core>
> [main.c:742]: handle_sigs(): core was generated
> Jul 15 10:42:26 server /usr/sbin/kamailio[12370]: INFO: <core>
> [main.c:754]: handle_sigs(): terminating due to SIGCHLD
>
>
> Cheers,
> Dirk
>
>
> On 07/15/2016 10:06 AM, Dirk Teurlings - Signet B.V. wrote:
>> Hi,
>>
>> Running Kamailio on Debian from the Kamailio repository with 4.4.2
>> stable (unpatched). Getting some random segfaults with it now, here's
>> the relevant backtrace from the generated core.
>>
>> Core was generated by `/usr/sbin/kamailio -f /etc/kamailio/kamailio.cfg
>> -P /var/run/kamailio/kamailio.'.
>> Program terminated with signal 11, Segmentation fault.
>> #0  run_dlg_callbacks (type=type at entry=64, dlg=dlg at entry=0x7fceb400e2f0,
>> req=req at entry=0x7fced4f093c8, rpl=rpl at entry=0x0, dir=<optimized out>,
>> dlg_data=dlg_data at entry=0x0) at dlg_cb.c:253
>> 253			if ( (cb->types)&type ) {
>> (gdb) bt
>> #0  run_dlg_callbacks (type=type at entry=64, dlg=dlg at entry=0x7fceb400e2f0,
>> req=req at entry=0x7fced4f093c8, rpl=rpl at entry=0x0, dir=<optimized out>,
>> dlg_data=dlg_data at entry=0x0) at dlg_cb.c:253
>> #1  0x00007fcead3648f9 in dlg_terminated (dir=<optimized out>,
>> dlg=0x7fceb400e2f0, req=0x7fced4f093c8) at dlg_handlers.c:368
>> #2  dlg_onroute (req=0x7fced4f093c8, route_params=<optimized out>,
>> param=<optimized out>) at dlg_handlers.c:1354
>> #3  0x00007fceb10ab0e2 in run_rr_callbacks
>> (req=req at entry=0x7fced4f093c8, rr_param=rr_param at entry=0x7fceb12b77a0)
>> at rr_cb.c:96
>> #4  0x00007fceb10952c5 in after_loose (_m=0x7fced4f093c8, preloaded=0)
>> at loose.c:919
>> #5  0x000000000042b618 in do_action (h=h at entry=0x7ffeb0b3ed80,
>> a=a at entry=0x7fced4cb4338, msg=msg at entry=0x7fced4f093c8) at action.c:1060
>> #6  0x000000000042a10a in run_actions (h=h at entry=0x7ffeb0b3ed80,
>> a=0x7fced4cb4338, msg=0x7fced4f093c8) at action.c:1549
>> #7  0x0000000000437544 in run_actions_safe (h=h at entry=0x7ffeb0b402b0,
>> a=<optimized out>, msg=<optimized out>) at action.c:1614
>> #8  0x000000000053b2e8 in rval_get_int (h=0x7ffeb0b402b0, msg=<optimized
>> out>, i=0x7ffeb0b3f1e0, rv=rv at entry=0x7fced4cb4d58,
>> cache=cache at entry=0x0) at rvalue.c:912
>> #9  0x000000000054261c in rval_expr_eval_int (h=h at entry=0x7ffeb0b402b0,
>> msg=msg at entry=0x7fced4f093c8, res=res at entry=0x7ffeb0b3f1e0,
>> rve=rve at entry=0x7fced4cb4d50) at rvalue.c:1910
>> #10 0x000000000042bc91 in do_action (h=h at entry=0x7ffeb0b402b0,
>> a=a at entry=0x7fced4cb8f88, msg=msg at entry=0x7fced4f093c8) at action.c:1030
>> #11 0x000000000042a10a in run_actions (h=h at entry=0x7ffeb0b402b0,
>> a=0x7fced4cb8f88, msg=msg at entry=0x7fced4f093c8) at action.c:1549
>> #12 0x000000000042bcf2 in do_action (h=h at entry=0x7ffeb0b402b0,
>> a=a at entry=0x7fced4cb91e8, msg=msg at entry=0x7fced4f093c8) at action.c:1049
>> #13 0x000000000042a10a in run_actions (h=h at entry=0x7ffeb0b402b0,
>> a=0x7fced4cb3f48, msg=msg at entry=0x7fced4f093c8) at action.c:1549
>> #14 0x000000000042bde0 in do_action (h=h at entry=0x7ffeb0b402b0,
>> a=a at entry=0x7fced4ac3d70, msg=msg at entry=0x7fced4f093c8) at action.c:678
>> #15 0x000000000042a10a in run_actions (h=h at entry=0x7ffeb0b402b0,
>> a=a at entry=0x7fced4ac1698, msg=msg at entry=0x7fced4f093c8) at action.c:1549
>> #16 0x00000000004375d0 in run_top_route (a=0x7fced4ac1698,
>> msg=msg at entry=0x7fced4f093c8, c=c at entry=0x0) at action.c:1635
>> #17 0x0000000000504386 in receive_msg (buf=<optimized out>,
>> len=<optimized out>, rcv_info=<optimized out>) at receive.c:240
>> #18 0x00000000005f5bd4 in udp_rcv_loop () at udp_server.c:495
>> #19 0x00000000004b2625 in main_loop () at main.c:1600
>> #20 0x0000000000427e2b in main (argc=<optimized out>, argv=<optimized
>> out>) at main.c:2616
>>
>>
>> And from syslog the relevant messages before this dump:
>>
>> Jul 15 08:55:03 server /usr/sbin/kamailio[16470]: WARNING: dialog
>> [dlg_handlers.c:1219]: dlg_onroute(): unable to find dialog for BYE with
>> route param 'd4c.26d1' [3149:7522]
>> Jul 15 08:56:01 server /usr/sbin/kamailio[16481]: WARNING: dialog
>> [dlg_handlers.c:1219]: dlg_onroute(): unable to find dialog for BYE with
>> route param 'fc.99f1' [207:8089]
>> Jul 15 08:56:27 server /usr/sbin/kamailio[16470]: CRITICAL: dialog
>> [dlg_timer.c:200]: update_dlg_timer(): Trying to update a bogus dlg
>> tl=0x7fceb3f7d920 tl->next=(nil) tl->prev=(nil)
>> Jul 15 08:56:27 server /usr/sbin/kamailio[16470]: ERROR: dialog
>> [dlg_handlers.c:1377]: dlg_onroute(): failed to update dialog lifetime
>> Jul 15 08:57:01 server /usr/sbin/kamailio[16482]: ERROR: db_mysql
>> [km_dbase.c:128]: db_mysql_submit_query(): driver error on query:
>> Duplicate entry '9584-3854-435' for key 'hash_index' (1062)
>> Jul 15 08:57:01 server /usr/sbin/kamailio[16482]: ERROR: <core>
>> [db_query.c:181]: db_do_raw_query(): error while submitting query
>> Jul 15 08:57:01 server /usr/sbin/kamailio[16482]: ERROR: sqlops
>> [sql_api.c:265]: sql_do_query(): cannot do the query [INSERT INTO
>> `dialog_extra` (`h_i]
>> Jul 15 08:57:01 server /usr/sbin/kamailio[16482]: ERROR: auth
>> [api.c:119]: auth_check_hdr_md5(): auth:pre_auth: Credentials are not
>> filled properly
>> Jul 15 08:57:01 server /usr/sbin/kamailio[16483]: ERROR: auth
>> [api.c:119]: auth_check_hdr_md5(): auth:pre_auth: Credentials are not
>> filled properly
>> Jul 15 08:57:54 server /usr/sbin/kamailio[16506]: NOTICE: dialog
>> [dlg_hash.c:245]: dlg_clean_run(): dialog in delete state is too old
>> (0x7fceb3f64470 ref 4)
>> Jul 15 08:57:54 server /usr/sbin/kamailio[16473]: WARNING: dialog
>> [dlg_handlers.c:1348]: dlg_onroute(): inconsitent dlg timer data on dlg
>> 0x7fceb3f64470 [1182:5803] with clid
>> '09ad128753e2535d24bde58e3d7eda04 at 192.168.10.232:5060' and tags
>> 'as1b497b34' '5788890C-EC6F55F-3E86ED0C'
>> Jul 15 08:57:54 server /usr/sbin/kamailio[16469]: ERROR: dialog
>> [dlg_handlers.c:334]: dlg_terminated_confirmed(): failed to get dialog
>> from params!
>> Jul 15 08:58:49 server /usr/sbin/kamailio[16467]: WARNING: dialog
>> [dlg_handlers.c:1219]: dlg_onroute(): unable to find dialog for BYE with
>> route param '6d2.2581' [726:6226]
>> Jul 15 08:59:24 server /usr/sbin/kamailio[16506]: NOTICE: dialog
>> [dlg_hash.c:245]: dlg_clean_run(): dialog in delete state is too old
>> (0x7fceb400e2f0 ref 4)
>> Jul 15 08:59:25 server /usr/sbin/kamailio[16464]: WARNING: dialog
>> [dlg_handlers.c:1219]: dlg_onroute(): unable to find dialog for BYE with
>> route param '3e4.b5c1' [1251:7259]
>> Jul 15 08:59:25 server /usr/sbin/kamailio[16465]: WARNING: dialog
>> [dlg_handlers.c:1348]: dlg_onroute(): inconsitent dlg timer data on dlg
>> 0x7fceb400e2f0 [1251:7259] with clid '87791a#015#012Call-ID:
>> 25750e286a5654361ef9405d72edbc' and tags '' 'as148f41b1'
>> Jul 15 08:59:25 server kernel: [203670.830521] kamailio[16465] general
>> protection ip:7fcead34b3a5 sp:7ffeb0b3e220 error:0 in
>> dialog.so[7fcead330000+88000]
>> Jul 15 08:59:26 server /usr/sbin/kamailio[16511]: CRITICAL: <core>
>> [pass_fd.c:275]: receive_fd(): EOF on 33
>> Jul 15 08:59:26 server /usr/sbin/kamailio[16458]: ALERT: <core>
>> [main.c:739]: handle_sigs(): child process 16465 exited by a signal 11
>> Jul 15 08:59:26 server /usr/sbin/kamailio[16458]: ALERT: <core>
>> [main.c:742]: handle_sigs(): core was generated
>> Jul 15 08:59:26 server /usr/sbin/kamailio[16458]: INFO: <core>
>> [main.c:754]: handle_sigs(): terminating due to SIGCHLD
>>
>> Any insight would be appreciated!
>>
>> Cheers,
>> Dirk
>>
>> _______________________________________________
>> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
>> sr-users at lists.sip-router.org
>> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
>>
> _______________________________________________
> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
> sr-users at lists.sip-router.org
> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users

-- 
Daniel-Constantin Mierla
http://www.asipto.com - http://www.kamailio.org
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda