[SR-Users] Multiple crashes of Kamailio 4.2.1
Daniel-Constantin Mierla
miconda at gmail.com
Fri Sep 11 16:10:30 CEST 2015
Hello,
this one is also from the cleanup process at shut down -- if you look at
the backtrace, if frames are executing shutdown_children(), cleanup(),
destory_modules()... then it is the shutdown. There should be another
core that is the reason for the runtime problem -- if there was
something wrong with the memory structures at runtime, then shutdown can
end in it as well.
Do you have other core files from where to extract the bt full?
Just to rule it out, are you having any private modules or patches?
Cheers,
Daniel
On 11/09/15 16:03, Igor Potjevlesch wrote:
>
> Hello,
>
>
>
> I got two more crashes but they seem to be different.
>
>
>
> The logs mentioned:
>
> kernel: kamailio[27096] general protection ip:7f0b3e0bae55
> sp:7fffdbce5fd0 error:0 in dialog.so[7f0b3e077000+8f000]
>
> /usr/local/sbin/kamailio[27056]: ALERT: <core> [main.c:784]:
> handle_sigs(): child process 27096 exited by a signal 11
>
>
>
> Both gdb are related to dialog module:
>
>
>
> (gdb) bt full
>
> #0 0x00007fb6a8984c0e in remove_dialog_timer_unsafe
> (tl=0x7fb6978e9060) at dlg_timer.c:156
>
> No locals.
>
> #1 0x00007fb6a8985001 in remove_dialog_timer (tl=0x7fb6978e9060) at
> dlg_timer.c:182
>
> __FUNCTION__ = "remove_dialog_timer"
>
> #2 0x00007fb6a8966bb7 in destroy_dlg (dlg=0x7fb6978e9008) at
> dlg_hash.c:357
>
> ret = 0
>
> var = 0x7fb6976154b0
>
> __FUNCTION__ = "destroy_dlg"
>
> #3 0x00007fb6a8967b35 in destroy_dlg_table () at dlg_hash.c:438
>
> dlg = 0xb02030a01201001
>
> l_dlg = 0x7fb6978e9008
>
> i = 2087
>
> __FUNCTION__ = "destroy_dlg_table"
>
> #4 0x00007fb6a8933263 in mod_destroy () at dialog.c:783
>
> No locals.
>
> #5 0x0000000000590d79 in destroy_modules () at sr_module.c:811
>
> t = 0x7fb6af43d670
>
> foo = 0x7fb6af43d440
>
> __FUNCTION__ = "destroy_modules"
>
> #6 0x000000000049bb43 in cleanup (show_status=1) at main.c:569
>
> memlog = 0
>
> __FUNCTION__ = "cleanup"
>
> #7 0x000000000049d10b in shutdown_children (sig=15, show_status=1) at
> main.c:711
>
> __FUNCTION__ = "shutdown_children"
>
> #8 0x000000000049f6e1 in handle_sigs () at main.c:802
>
> chld = 0
>
> chld_status = 139
>
> memlog = -1755228944
>
> __FUNCTION__ = "handle_sigs"
>
> #9 0x00000000004a6fbf in main_loop () at main.c:1757
>
> i = 8
>
> pid = 4424
>
> si = 0x0
>
> si_desc = "udp receiver child=7
> sock=91.213.145.60:5060\000\000\000\000\016\b\000\000\377\177\000\000\260Ta\227\266\177\000\000\000\000\000\020\004\000\000\000\260Ta\227\266\177\000\000\060SA\000\000\000\000\000\240\177\207\b\001\000\000\000\060}\207\b\377\177\000\000\032dN\000\000\000\000\000h\261@\257z\000\000\000\276}p\000\000\000\000"
>
> nrprocs = 8
>
> __FUNCTION__ = "main_loop"
>
> #10 0x00000000004ab8bf in main (argc=7, argv=0x7fff08877fa8) at
> main.c:2578
>
> cfg_stream = 0x18b4010
>
> c = -1
>
> r = 0
>
> tmp = 0x7fff08879f70 ""
>
> tmp_len = 0
>
> port = 0
>
> proto = 32767
>
> options = 0x6fcc00
> ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:"
>
> ret = -1
>
> seed = 2249241156
>
> rfd = 4
>
> debug_save = 0
>
> debug_flag = 0
>
> dont_fork_cnt = 0
>
> n_lst = 0xc2
>
> p = 0x7fff08877e7e ""
>
> __FUNCTION__ = "main"
>
>
>
> Regards,
>
>
>
> Igor.
>
>
>
> *De :*Igor Potjevlesch [mailto:igor.potjevlesch at gmail.com]
> *Envoyé :* jeudi 10 septembre 2015 16:43
> *À :* miconda at gmail.com; 'Kamailio (SER) - Users Mailing List'
> <sr-users at lists.sip-router.org>
> *Objet :* RE: [SR-Users] Multiple crashes of Kamailio 4.2.1
>
>
>
> Hello Daniel,
>
>
>
> There is no particular reason. It's just that our others Kamailio are
> running 4.2.3. So, we try to have the same version on all nodes.
>
>
>
> Regarding the backtrace, I'm surprised, because there was no manual
> shutdown.
>
>
>
> Unfortunately, for this occurrence of the crash, it's the only
> coredump available. There was another crash later this day with two
> coredumps and the following backtrace:
>
>
>
> […]
>
>
>
> Regards,
>
>
>
> Igor.
>
>
>
> *De :*sr-users [mailto:sr-users-bounces at lists.sip-router.org] *De la
> part de* Daniel-Constantin Mierla
> *Envoyé :* jeudi 10 septembre 2015 12:57
> *À :* Kamailio (SER) - Users Mailing List
> <sr-users at lists.sip-router.org <mailto:sr-users at lists.sip-router.org>>
> *Objet :* Re: [SR-Users] Multiple crashes of Kamailio 4.2.1
>
>
>
> Hello,
>
> there are newer releases in branch 4.2, was there any reason to stop
> at 4.2.3?
>
> Back to the backtrace, this one is from shutdown, happening during the
> cleanup, not showing the reason of the crash at runtime, unless there
> was a manual shut down triggered at that moment. Do you have another
> core file?
>
> Cheers,
> Daniel
>
> On 10/09/15 12:36, Igor Potjevlesch wrote:
>
> Hello,
>
>
>
> We got some others crashes even after updated to 4.2.3.
>
>
>
> From the last one, I got the following into "bt full":
>
> Core was generated by `/usr/local/sbin/kamailio -P
> /var/run/kamailio.pid -m 256 -M 64'.
>
> Program terminated with signal 11, Segmentation fault.
>
> #0 0x0000000000617612 in fm_insert_free (qm=0x7f7d4206d000,
> frag=0x7f7d422ea368) at mem/f_malloc.c:245
>
> 245 if (*f) (*f)->prv_free = &(frag->u.nxt_free);
>
> Missing separate debuginfos, use: debuginfo-install
> bzip2-libs-1.0.5-7.el6_0.x86_64 db4-4.7.25-18.el6_4.x86_64
> elfutils-libelf-0.152-1.el6.x86_64 glibc-2.12-1.132.el6.x86_64
> keyutils-libs-1.4-4.el6.x86_64 krb5-libs-1.10.3-10.el6_4.6.x86_64
> libacl-2.2.49-6.el6.x86_64 libattr-2.4.44-7.el6.x86_64
> libcap-2.16-5.5.el6.x86_64 libcom_err-1.41.12-18.el6.x86_64
> libgcc-4.4.7-11.el6.x86_64 libselinux-2.0.94-5.3.el6_4.1.x86_64
> lm_sensors-libs-3.1.1-17.el6.x86_64 lua-5.1.4-4.1.el6.x86_64
> mysql-libs-5.1.73-3.el6_5.x86_64
> net-snmp-libs-5.5-50.el6_6.1.x86_64 nspr-4.10.0-1.el6.x86_64
> nss-3.15.1-15.el6.x86_64 nss-softokn-freebl-3.14.3-9.el6.x86_64
> nss-util-3.15.1-3.el6.x86_64 openssl-1.0.1e-30.el6_6.4.x86_64
> pcre-7.8-6.el6.x86_64 perl-libs-5.10.1-136.el6.x86_64
> popt-1.13-7.el6.x86_64 rpm-libs-4.8.0-37.el6.x86_64
> tcp_wrappers-libs-7.6-57.el6.x86_64
> xz-libs-4.999.9-0.3.beta.20091007git.el6.x86_64
> zlib-1.2.3-29.el6.x86_64
>
>
>
>
>
> (gdb) bt full
>
> #0 0x0000000000617612 in fm_insert_free (qm=0x7f7d4206d000,
> frag=0x7f7d422ea368) at mem/f_malloc.c:245
>
> f = 0x7f7d4206d188
>
> hash = 4
>
> #1 0x000000000061a2fe in fm_free (qm=0x7f7d4206d000,
> p=0x7f7d422ea3a0, file=0x7f7d53408ff0 "dialog: dlg_timer.c",
> func=0x7f7d53409d00 "destroy_dlg_timer", line=95)
>
> at mem/f_malloc.c:614
>
> f = 0x7f7d422ea368
>
> __FUNCTION__ = "fm_free"
>
> #2 0x00007f7d533e441f in destroy_dlg_timer () at dlg_timer.c:95
>
> __FUNCTION__ = "destroy_dlg_timer"
>
> #3 0x00007f7d53393268 in mod_destroy () at dialog.c:784
>
> No locals.
>
> #4 0x0000000000590d79 in destroy_modules () at sr_module.c:811
>
> t = 0x7f7d59e9d670
>
> foo = 0x7f7d59e9d440
>
> __FUNCTION__ = "destroy_modules"
>
> #5 0x000000000049bb43 in cleanup (show_status=1) at main.c:569
>
> memlog = 0
>
> __FUNCTION__ = "cleanup"
>
> #6 0x000000000049d10b in shutdown_children (sig=15,
> show_status=1) at main.c:711
>
> __FUNCTION__ = "shutdown_children"
>
> #7 0x000000000049f6e1 in handle_sigs () at main.c:802
>
> chld = 0
>
> chld_status = 139
>
> memlog = 1107776752
>
> __FUNCTION__ = "handle_sigs"
>
> #8 0x00000000004a6fbf in main_loop () at main.c:1757
>
> i = 8
>
> pid = 4147
>
> si = 0x0
>
> si_desc = "udp receiver child=7
> sock=91.213.145.60:5060\000\000\000\000\016\b\000\000\377\177\000\000\260T\aB}\177\000\000\000\000\000\020\004\000\000\000\260T\aB}\177\000\000\060SA\000\000\000\000\000\260:\240\343\001\000\000\000 at 8\240\343\377\177\000\000\032dN\000\000\000\000\000h\261\346Yz\000\000\000\276}p\000\000\000\000"
>
> nrprocs = 8
>
> __FUNCTION__ = "main_loop"
>
> #9 0x00000000004ab8bf in main (argc=7, argv=0x7fffe3a03ab8) at
> main.c:2578
>
> cfg_stream = 0x1a04010
>
> c = -1
>
> r = 0
>
> tmp = 0x7fffe3a04f70 ""
>
> tmp_len = 0
>
> port = 0
>
> proto = 32767
>
> options = 0x6fcc00
> ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:"
>
> ret = -1
>
> seed = 3320376299
>
> rfd = 4
>
> debug_save = 0
>
> debug_flag = 0
>
> dont_fork_cnt = 0
>
> n_lst = 0xc2
>
> p = 0x7fffe3a0398e ""
>
> __FUNCTION__ = "main"
>
>
>
> Regards,
>
>
>
> Igor.
>
>
>
> *De :*Igor Potjevlesch [mailto:igor.potjevlesch at gmail.com]
> *Envoyé :* lundi 20 juillet 2015 16:22
> *À :* sr-users at lists.sip-router.org
> <mailto:sr-users at lists.sip-router.org>
> *Objet :* Multiple crashes of Kamailio 4.2.1
>
>
>
> Hello,
>
>
>
> 3 crashes occurred today (we never seen crashes on this server
> before) during a period of 20 minutes.
>
>
>
> I had a look to the coredumps. Here is a quick overview of the
> first entries of "bt full":
>
>
>
> core.27671 : #0 0x00007f5577048d65 in
> run_trans_callbacks_internal (cb_lst=0x7f5560abbd50, type=2,
> trans=0x7f5560abbce0, params=0x7fff3712acb0) at t_hooks.c:286
>
> core.27656 : #0 0x00007f557700a048 in free_cell
> (dead_cell=0x7f5560abbce0) at h_table.c:175
>
> core.27685 : #0 0x00007f5577048d65 in
> run_trans_callbacks_internal (cb_lst=0x7f5560abbd50, type=2,
> trans=0x7f5560abbce0, params=0x7fff3712acb0) at t_hooks.c:286
>
> core.27689 : #0 0x00007f5577048d65 in
> run_trans_callbacks_internal (cb_lst=0x7f5560abbd50, type=2,
> trans=0x7f5560abbce0, params=0x7fff3712acb0) at t_hooks.c:286
>
> core.27691 : #0 0x00007f5577048d65 in
> run_trans_callbacks_internal (cb_lst=0x7f5560abbd50, type=2,
> trans=0x7f5560abbce0, params=0x7fff3712acb0) at t_hooks.c:286
>
> core.31977 : #0 0x00007f038e66fdee in free_cell
> (dead_cell=0x7f037811ebe0) at h_table.c:157
>
> core.31994 : #0 0x00007f038e6ab65a in t_forward_nonack
> (t=0x7f037811ebe0, p_msg=0x7f038fdca438, proxy=0x0, proto=0) at
> t_fwd.c:1762
>
> core.3700 : #0 0x000000000061697a in fm_insert_free
> (qm=0x7f19861fa000, frag=0x7f19864d53e8) at mem/f_malloc.c:245
>
> core.3727 : #0 0x00000000006167a4 in fm_extract_free
> (qm=0x7f19861fa000, frag=0x7f19864f4cc8) at mem/f_malloc.c:208
>
>
>
> I don't know how to proceed now. Can you help me to highlight the
> issue here? Thank you.
>
> Regards,
>
>
>
> Igor.
>
>
>
> _______________________________________________
>
> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
>
> sr-users at lists.sip-router.org <mailto:sr-users at lists.sip-router.org>
>
> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
>
>
>
> --
> Daniel-Constantin Mierla
> http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> - http://www.linkedin.com/in/miconda
> Book: SIP Routing With Kamailio - http://www.asipto.com
> Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Book: SIP Routing With Kamailio - http://www.asipto.com
Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.sip-router.org/pipermail/sr-users/attachments/20150911/29128e9e/attachment.html>
More information about the sr-users
mailing list