[SR-Users] Kamailio 4.2.6 crash
Daniel-Constantin Mierla
miconda at gmail.com
Mon Jan 18 08:56:24 CET 2016
Hello,
are you using async transactions (e.g., t_suspend() or tsilo module)?
Cheers,
Daniel
On 15/01/16 10:51, Igor Potjevlesch wrote:
>
> I also seen this in /var/log/messages during the crash:
>
>
>
> Jan 15 10:37:41 tanus /usr/local/sbin/kamailio[24021]: : <core>
> [mem/q_malloc.c:149]: qm_debug_frag(): BUG: qm_*: fragm.
> 0x7f9abd767300 (address 0x7f9abd767330) end
> overwritten(d33762d69737465, 746e65746e6f430a)!
>
> Jan 15 10:37:41 tanus kernel: kamailio[24021] general protection
> ip:62245e sp:7fff39bc7900 error:0 in kamailio[400000+3c8000]
>
> Jan 15 10:37:42 tanus abrtd: Directory
> 'ccpp-2016-01-15-10:37:41-24021' creation detected
>
> Jan 15 10:37:42 tanus abrt[23992]: Saved core dump of pid 24021
> (/usr/local/sbin/kamailio) to
> /var/spool/abrt/ccpp-2016-01-15-10:37:41-24021 (339316736 bytes)
>
> Jan 15 10:37:42 tanus /usr/local/sbin/kamailio[23990]: ALERT: <core>
> [main.c:784]: handle_sigs(): child process 24021 exited by a signal 11
>
> Jan 15 10:37:42 tanus /usr/local/sbin/kamailio[23990]: ALERT: <core>
> [main.c:787]: handle_sigs(): core was generated
>
> Jan 15 10:37:42 tanus /usr/local/sbin/kamailio[23990]: INFO: <core>
> [main.c:799]: handle_sigs(): terminating due to SIGCHLD
>
> Jan 15 10:37:42 tanus /usr/local/sbin/kamailio[24019]: INFO: <core>
> [main.c:850]: sig_usr(): signal 15 received
>
> Jan 15 10:37:42 tanus /usr/local/sbin/kamailio[24013]: INFO: <core>
> [main.c:850]: sig_usr(): signal 15 received
>
> Jan 15 10:37:42 tanus /usr/local/sbin/kamailio[24017]: INFO: <core>
> [main.c:850]: sig_usr(): signal 15 received
>
> Jan 15 10:37:42 tanus /usr/local/sbin/kamailio[24026]: INFO: <core>
> [main.c:850]: sig_usr(): signal 15 received
>
> Jan 15 10:37:42 tanus /usr/local/sbin/kamailio[24023]: INFO: <core>
> [main.c:850]: sig_usr(): signal 15 received
>
> Jan 15 10:37:42 tanus /usr/local/sbin/kamailio[24030]: INFO: <core>
> [main.c:850]: sig_usr(): signal 15 received
>
> Jan 15 10:37:42 tanus /usr/local/sbin/kamailio[24005]: INFO: <core>
> [main.c:850]: sig_usr(): signal 15 received
>
> Jan 15 10:37:42 tanus /usr/local/sbin/kamailio[24009]: INFO: <core>
> [main.c:850]: sig_usr(): signal 15 received
>
> Jan 15 10:37:42 tanus /usr/local/sbin/kamailio[24011]: INFO: <core>
> [main.c:850]: sig_usr(): signal 15 received
>
> Jan 15 10:37:42 tanus /usr/local/sbin/kamailio[24007]: INFO: <core>
> [main.c:850]: sig_usr(): signal 15 received
>
> Jan 15 10:37:42 tanus /usr/local/sbin/kamailio[24015]: INFO: <core>
> [main.c:850]: sig_usr(): signal 15 received
>
> Jan 15 10:37:42 tanus /usr/local/sbin/kamailio[24003]: INFO: <core>
> [main.c:850]: sig_usr(): signal 15 received
>
> Jan 15 10:37:42 tanus /usr/local/sbin/kamailio[23990]: : <core>
> [mem/q_malloc.c:159]: qm_debug_frag(): BUG: qm_*: prev. fragm. tail
> overwritten(732d6369676f6c61,
> a0d2d2d312d77)[0x7f9abd767970:0x7f9abd7679a0]!
>
> Jan 15 10:37:42 tanus kernel: kamailio[23990] general protection
> ip:62245e sp:7fff39bc7690 error:0 in kamailio[400000+3c8000]
>
> Jan 15 10:37:42 tanus abrt[23994]: Not saving repeating crash in
> '/usr/local/sbin/kamailio'
>
> Jan 15 10:37:42 tanus abrtd: Executable '/usr/local/sbin/kamailio'
> doesn't belong to any package and ProcessUnpackaged is set to 'no'
>
> Jan 15 10:37:42 tanus abrtd: 'post-create' on
> '/var/spool/abrt/ccpp-2016-01-15-10:37:41-24021' exited with 1
>
> Jan 15 10:37:42 tanus abrtd: Deleting problem directory
> '/var/spool/abrt/ccpp-2016-01-15-10:37:41-24021'
>
> Jan 15 10:37:43 tanus abrt[23994]: Saved core dump of pid 23990 to
> core.23990 (339316736 bytes)
>
> Jan 15 10:37:47 tanus kamailio: INFO: <core> [sctp_core.c:70]:
> sctp_core_check_support(): SCTP API not enabled - if you want to use
> it, load sctp module
>
> Jan 15 10:37:47 tanus kamailio: WARNING: <core> [daemonize.c:360]:
> daemonize(): pid file contains old pid, replacing pid
>
> Jan 15 10:37:47 tanus /usr/local/sbin/kamailio[24042]: INFO: rr
> [../outbound/api.h:54]: ob_load_api(): unable to import bind_ob -
> maybe module is not loaded
>
> Jan 15 10:37:47 tanus /usr/local/sbin/kamailio[24042]: INFO: rr
> [rr_mod.c:160]: mod_init(): outbound module not available
>
> Jan 15 10:37:47 tanus /usr/local/sbin/kamailio[24042]: INFO: usrloc
> [hslot.c:53]: ul_init_locks(): locks array size 1024
>
>
>
> Regards,
>
>
>
> Igor.
>
>
>
> *De :*Igor Potjevlesch [mailto:igor.potjevlesch at gmail.com]
> *Envoyé :* vendredi 15 janvier 2016 10:47
> *À :* miconda at gmail.com; 'Kamailio (SER) - Users Mailing List'
> <sr-users at lists.sip-router.org>
> *Objet :* RE: [SR-Users] Kamailio 4.2.6 crash
>
>
>
> Hello Daniel,
>
>
>
> I move to 4.2.7. This morning a new crash occurred. I got two coredump:
>
>
>
> Core was generated by `/usr/local/sbin/kamailio -P
> /var/run/kamailio.pid -m 256 -M 64'.
>
> Program terminated with signal 11, Segmentation fault.
>
> #0 0x000000000062245e in qm_status (qm=0x7f9abd447000) at
> mem/q_malloc.c:788
>
> 788 f!=&(qm->free_hash[h].head);
> f=f->u.nxt_free, i++, j++){
>
>
>
> (gdb) bt full
>
> #0 0x000000000062245e in qm_status (qm=0x7f9abd447000) at
> mem/q_malloc.c:788
>
> f = 0x30a012010010a0d
>
> i = 57
>
> j = 4
>
> h = 4
>
> unused = 0
>
> memlog = 5
>
> mem_summary = 3
>
> __FUNCTION__ = "qm_status"
>
> #1 0x000000000061a795 in qm_debug_frag (qm=0x7f9abd447000,
> f=0x7f9abd767970) at mem/q_malloc.c:160
>
> __FUNCTION__ = "qm_debug_frag"
>
> #2 0x000000000061ca58 in qm_free (qm=0x7f9abd447000,
> p=0x7f9abd7679a0, file=0x7f9ace7e2662 "dialog: dlg_hash.c",
> func=0x7f9ace7e50e2 "destroy_dlg", line=380) at mem/q_malloc.c:468
>
> f = 0x7f9abd767970
>
> size = 176
>
> next = 0x400
>
> prev = 0x7fff39bc7910
>
> __FUNCTION__ = "qm_free"
>
> #3 0x00007f9ace7a64ef in destroy_dlg (dlg=0x7f9abd7660b8) at
> dlg_hash.c:380
>
> ret = 0
>
> var = 0x7f9ad54374e8
>
> __FUNCTION__ = "destroy_dlg"
>
> #4 0x00007f9ace7a67df in destroy_dlg_table () at dlg_hash.c:419
>
> dlg = 0x0
>
> l_dlg = 0x7f9abd7660b8
>
> i = 665
>
> __FUNCTION__ = "destroy_dlg_table"
>
> #5 0x00007f9ace771263 in mod_destroy () at dialog.c:783
>
> No locals.
>
> #6 0x00000000005929ee in destroy_modules () at sr_module.c:811
>
> t = 0x7f9ad52b8d00
>
> foo = 0x7f9ad52b8a30
>
> __FUNCTION__ = "destroy_modules"
>
> #7 0x000000000049c917 in cleanup (show_status=1) at main.c:569
>
> memlog = 0
>
> __FUNCTION__ = "cleanup"
>
> #8 0x000000000049dee4 in shutdown_children (sig=15, show_status=1) at
> main.c:711
>
> __FUNCTION__ = "shutdown_children"
>
> #9 0x00000000004a04ba in handle_sigs () at main.c:802
>
> chld = 0
>
> chld_status = 139
>
> memlog = -1119369840
>
> __FUNCTION__ = "handle_sigs"
>
> #10 0x00000000004a82eb in main_loop () at main.c:1757
>
> i = 8
>
> pid = 24021
>
> si = 0x0
>
> si_desc = "udp receiver child=7
> sock=91.213.145.60:5060\000\177\000\000`~\274\071\377\177\000\000\033{N\000\000\000\000\000\260~\274\071\377\177\000\000\004\000\000\000\000\000\000\000`TA\000\000\000\000\000(\305G\275\232\177",
> '\000' <repeats 14 times>,
> "\001\000\000\000\260~\274\071\377\177\000\000\276{N\000\000\000\000"
>
> nrprocs = 8
>
> __FUNCTION__ = "main_loop"
>
> #11 0x00000000004acfab in main (argc=7, argv=0x7fff39bc8128) at
> main.c:2581
>
> cfg_stream = 0x2392010
>
> c = -1
>
> r = 0
>
> tmp = 0x7fff39bc8f70 ""
>
> tmp_len = 32767
>
> port = 968654846
>
> proto = 0
>
> options = 0x7033b8
> ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:"
>
> ret = -1
>
> seed = 1451157380
>
> rfd = 4
>
> debug_save = 0
>
> debug_flag = 0
>
> dont_fork_cnt = 0
>
> n_lst = 0x40d134
>
> p = 0xc2 <Address 0xc2 out of bounds>
>
> __FUNCTION__ = "main"
>
>
>
> Second one:
>
>
>
> Core was generated by `/usr/local/sbin/kamailio -P
> /var/run/kamailio.pid -m 256 -M 64'.
>
> Program terminated with signal 11, Segmentation fault.
>
> #0 0x000000000062245e in qm_status (qm=0x7f9abd447000) at
> mem/q_malloc.c:788
>
> 788 f!=&(qm->free_hash[h].head);
> f=f->u.nxt_free, i++, j++){
>
>
>
> (gdb) bt full
>
> #0 0x000000000062245e in qm_status (qm=0x7f9abd447000) at
> mem/q_malloc.c:788
>
> f = 0x30a012010010a0d
>
> i = 16
>
> j = 4
>
> h = 4
>
> unused = 0
>
> memlog = 5
>
> mem_summary = 3
>
> __FUNCTION__ = "qm_status"
>
> #1 0x000000000061a420 in qm_debug_frag (qm=0x7f9abd447000,
> f=0x7f9abd767300) at mem/q_malloc.c:150
>
> __FUNCTION__ = "qm_debug_frag"
>
> #2 0x000000000061ca58 in qm_free (qm=0x7f9abd447000,
> p=0x7f9abd767330, file=0x7f9ad3d2f34d "tm: h_table.c",
> func=0x7f9ad3d2f628 "free_cell", line=186) at mem/q_malloc.c:468
>
> f = 0x7f9abd767300
>
> size = 40
>
> next = 0x400
>
> prev = 0x7fff39bc7b80
>
> __FUNCTION__ = "qm_free"
>
> #3 0x00007f9ad3c70c9d in free_cell (dead_cell=0x7f9abd79b5c0) at
> h_table.c:186
>
> b = 0x7f9abd767330 "INVITE sip:00447798156873 at goren
> SIP/2.0\r\nRecord-Route: <sip:A.B.C.D;lr;did=4b7.60a>\r\nCSeq: 1
> INVITE\r\nCall-ID: 729d-7e9-015201693735-DSQ-1-A.B.C_leg2\r\nFrom:
> <sip:0123456789 at D.C.B.A>;"...
>
> i = 0
>
> rpl = 0x0
>
> tt = 0x7f9abd5d8bf8
>
> foo = 0x7fff39bc7c50
>
> cbs = 0x0
>
> cbs_tmp = 0x7f9abd7600a0
>
> __FUNCTION__ = "free_cell"
>
> #4 0x00007f9ad3cb5a1c in wait_handler (ti=1300688687,
> wait_tl=0x7f9abd79b640, data=0x7f9abd79b5c0) at timer.c:675
>
> p_cell = 0x7f9abd79b5c0
>
> ret = 1
>
> #5 0x00000000005fd30f in timer_list_expire (t=1300688687,
> h=0x7f9abd4c0908, slow_l=0x7f9abd4c36d8, slow_mark=17084) at timer.c:888
>
> tl = 0x7f9abd79b640
>
> ret = 1300688687
>
> #6 0x00000000005fd757 in timer_handler () at timer.c:953
>
> saved_ticks = 1300688687
>
> run_slow_timer = 0
>
> i = 700
>
> __FUNCTION__ = "timer_handler"
>
> #7 0x00000000005fdbc5 in timer_main () at timer.c:992
>
> No locals.
>
> #8 0x00000000004a77e6 in main_loop () at main.c:1700
>
> i = 8
>
> pid = 0
>
> si = 0x0
>
> si_desc = "udp receiver child=7
> sock=91.213.145.60:5060\000\177\000\000`~\274\071\377\177\000\000\033{N\000\000\000\000\000\260~\274\071\377\177\000\000\004\000\000\000\000\000\000\000`TA\000\000\000\000\000(\305G\275\232\177",
> '\000' <repeats 14 times>,
> "\001\000\000\000\260~\274\071\377\177\000\000\276{N\000\000\000\000"
>
> nrprocs = 8
>
> __FUNCTION__ = "main_loop"
>
> #9 0x00000000004acfab in main (argc=7, argv=0x7fff39bc8128) at
> main.c:2581
>
> cfg_stream = 0x2392010
>
> c = -1
>
> r = 0
>
> tmp = 0x7fff39bc8f70 ""
>
> tmp_len = 32767
>
> port = 968654846
>
> proto = 0
>
> options = 0x7033b8
> ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:"
>
> ret = -1
>
> seed = 1451157380
>
> rfd = 4
>
> debug_save = 0
>
> debug_flag = 0
>
> dont_fork_cnt = 0
>
> n_lst = 0x40d134
>
> p = 0xc2 <Address 0xc2 out of bounds>
>
> __FUNCTION__ = "main"
>
>
>
> Regards,
>
>
>
> Igor.
>
>
>
> *De :*Igor Potjevlesch [mailto:igor.potjevlesch at gmail.com]
> *Envoyé :* lundi 4 janvier 2016 10:55
> *À :* miconda at gmail.com <mailto:miconda at gmail.com>; 'Kamailio (SER) -
> Users Mailing List' <sr-users at lists.sip-router.org
> <mailto:sr-users at lists.sip-router.org>>
> *Objet :* RE: [SR-Users] Kamailio 4.2.6 crash
>
>
>
> Hi Daniel,
>
>
>
> I wish you an happy new year! I will schedule this update in the next
> few days.
>
>
>
> Thank you for your support.
>
>
>
> Regards,
>
>
>
> Igor.
>
>
>
> *De :*Daniel-Constantin Mierla [mailto:miconda at gmail.com]
> *Envoyé :* vendredi 18 décembre 2015 11:28
> *À :* Igor Potjevlesch <igor.potjevlesch at gmail.com
> <mailto:igor.potjevlesch at gmail.com>>; 'Kamailio (SER) - Users Mailing
> List' <sr-users at lists.sip-router.org
> <mailto:sr-users at lists.sip-router.org>>
> *Objet :* Re: [SR-Users] Kamailio 4.2.6 crash
>
>
>
> Hello,
>
> can you upgrade to 4.2.7, it has some fixes that may be related to
> this issue -- there are no changes to config/database that need to be
> done.
>
> Cheers,
> Daniel
>
> On 16/12/15 15:06, Igor Potjevlesch wrote:
>
> Hello,
>
>
>
> Here is the output:
>
>
>
> kamailio -v
>
> version: kamailio 4.2.6 (x86_64/linux) db77ac
>
> flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS,
> USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM,
> SHM_MMAP, PKG_MALLOC, F_MALLOC, DBG_F_MALLOC, USE_FUTEX,
> FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER,
> USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES
>
> ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN
> 16, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 64MB
>
> poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
>
> id: db77ac
>
> compiled on 12:19:48 Oct 2 2015 with gcc 4.4.7
>
>
>
> Regards,
>
>
>
> Igor.
>
>
>
> *De :*Daniel-Constantin Mierla [mailto:miconda at gmail.com]
> *Envoyé :* lundi 14 décembre 2015 16:16
> *À :* Igor Potjevlesch <igor.potjevlesch at gmail.com>
> <mailto:igor.potjevlesch at gmail.com>; 'Kamailio (SER) - Users
> Mailing List' <sr-users at lists.sip-router.org>
> <mailto:sr-users at lists.sip-router.org>
> *Objet :* Re: [SR-Users] Kamailio 4.2.6 crash
>
>
>
> Hello,
>
> yes, it is the output I wanted.
>
> What is the exact version are you running? It is the output of
> 'kamailio -v'.
>
> Cheers,
> Daniel
>
> On 14/12/15 15:53, Igor Potjevlesch wrote:
>
> Hello Daniel,
>
>
>
> Is this the expected output?:
>
>
>
> (gdb) frame 0
>
> #0 0x0000000000619694 in fm_extract_free (qm=0x7f0114c07000,
> frag=0x7f0114f0e4b0) at mem/f_malloc.c:206
>
> 206 *pf=frag->u.nxt_free;
>
> (gdb) list
>
> 201 int hash;
>
> 202
>
> 203 pf = frag->prv_free;
>
> 204 hash = GET_HASH(frag->size);
>
> 205
>
> 206 *pf=frag->u.nxt_free;
>
> 207
>
> 208 if(frag->u.nxt_free)
> frag->u.nxt_free->prv_free = pf;
>
> 209
>
> 210 qm->ffrags--;
>
> (gdb)
>
>
>
> Regards,
>
>
>
> Igor.
>
>
>
> *De :*sr-users [mailto:sr-users-bounces at lists.sip-router.org]
> *De la part de* Daniel-Constantin Mierla
> *Envoyé :* lundi 14 décembre 2015 13:15
> *À :* Kamailio (SER) - Users Mailing List
> <sr-users at lists.sip-router.org>
> <mailto:sr-users at lists.sip-router.org>
> *Objet :* Re: [SR-Users] Kamailio 4.2.6 crash
>
>
>
> Hello,
>
> can you provide the list output in gdb for frame 0:
>
> frame 0
> list
>
> Cheers,
> Daniel
>
>
>
> --
>
> Daniel-Constantin Mierla
>
> http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> - http://www.linkedin.com/in/miconda
>
> Book: SIP Routing With Kamailio - http://www.asipto.com
>
> http://miconda.eu
>
>
>
> --
> Daniel-Constantin Mierla
> http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> - http://www.linkedin.com/in/miconda
> Book: SIP Routing With Kamailio - http://www.asipto.com
> http://miconda.eu
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Book: SIP Routing With Kamailio - http://www.asipto.com
http://miconda.eu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.sip-router.org/pipermail/sr-users/attachments/20160118/27b1b8fc/attachment.html>
More information about the sr-users
mailing list