[SR-Users] Kamailio 4.2.6 crash

Igor Potjevlesch igor.potjevlesch at gmail.com
Fri Jan 15 10:51:44 CET 2016


I also seen this in /var/log/messages during the crash:

 

Jan 15 10:37:41 tanus /usr/local/sbin/kamailio[24021]: : <core>
[mem/q_malloc.c:149]: qm_debug_frag(): BUG: qm_*: fragm. 0x7f9abd767300
(address 0x7f9abd767330) end overwritten(d33762d69737465, 746e65746e6f430a)!

Jan 15 10:37:41 tanus kernel: kamailio[24021] general protection ip:62245e
sp:7fff39bc7900 error:0 in kamailio[400000+3c8000]

Jan 15 10:37:42 tanus abrtd: Directory 'ccpp-2016-01-15-10:37:41-24021'
creation detected

Jan 15 10:37:42 tanus abrt[23992]: Saved core dump of pid 24021
(/usr/local/sbin/kamailio) to /var/spool/abrt/ccpp-2016-01-15-10:37:41-24021
(339316736 bytes)

Jan 15 10:37:42 tanus /usr/local/sbin/kamailio[23990]: ALERT: <core>
[main.c:784]: handle_sigs(): child process 24021 exited by a signal 11

Jan 15 10:37:42 tanus /usr/local/sbin/kamailio[23990]: ALERT: <core>
[main.c:787]: handle_sigs(): core was generated

Jan 15 10:37:42 tanus /usr/local/sbin/kamailio[23990]: INFO: <core>
[main.c:799]: handle_sigs(): terminating due to SIGCHLD

Jan 15 10:37:42 tanus /usr/local/sbin/kamailio[24019]: INFO: <core>
[main.c:850]: sig_usr(): signal 15 received

Jan 15 10:37:42 tanus /usr/local/sbin/kamailio[24013]: INFO: <core>
[main.c:850]: sig_usr(): signal 15 received

Jan 15 10:37:42 tanus /usr/local/sbin/kamailio[24017]: INFO: <core>
[main.c:850]: sig_usr(): signal 15 received

Jan 15 10:37:42 tanus /usr/local/sbin/kamailio[24026]: INFO: <core>
[main.c:850]: sig_usr(): signal 15 received

Jan 15 10:37:42 tanus /usr/local/sbin/kamailio[24023]: INFO: <core>
[main.c:850]: sig_usr(): signal 15 received

Jan 15 10:37:42 tanus /usr/local/sbin/kamailio[24030]: INFO: <core>
[main.c:850]: sig_usr(): signal 15 received

Jan 15 10:37:42 tanus /usr/local/sbin/kamailio[24005]: INFO: <core>
[main.c:850]: sig_usr(): signal 15 received

Jan 15 10:37:42 tanus /usr/local/sbin/kamailio[24009]: INFO: <core>
[main.c:850]: sig_usr(): signal 15 received

Jan 15 10:37:42 tanus /usr/local/sbin/kamailio[24011]: INFO: <core>
[main.c:850]: sig_usr(): signal 15 received

Jan 15 10:37:42 tanus /usr/local/sbin/kamailio[24007]: INFO: <core>
[main.c:850]: sig_usr(): signal 15 received

Jan 15 10:37:42 tanus /usr/local/sbin/kamailio[24015]: INFO: <core>
[main.c:850]: sig_usr(): signal 15 received

Jan 15 10:37:42 tanus /usr/local/sbin/kamailio[24003]: INFO: <core>
[main.c:850]: sig_usr(): signal 15 received

Jan 15 10:37:42 tanus /usr/local/sbin/kamailio[23990]: : <core>
[mem/q_malloc.c:159]: qm_debug_frag(): BUG: qm_*: prev. fragm. tail
overwritten(732d6369676f6c61, a0d2d2d312d77)[0x7f9abd767970:0x7f9abd7679a0]!

Jan 15 10:37:42 tanus kernel: kamailio[23990] general protection ip:62245e
sp:7fff39bc7690 error:0 in kamailio[400000+3c8000]

Jan 15 10:37:42 tanus abrt[23994]: Not saving repeating crash in
'/usr/local/sbin/kamailio'

Jan 15 10:37:42 tanus abrtd: Executable '/usr/local/sbin/kamailio' doesn't
belong to any package and ProcessUnpackaged is set to 'no'

Jan 15 10:37:42 tanus abrtd: 'post-create' on
'/var/spool/abrt/ccpp-2016-01-15-10:37:41-24021' exited with 1

Jan 15 10:37:42 tanus abrtd: Deleting problem directory
'/var/spool/abrt/ccpp-2016-01-15-10:37:41-24021'

Jan 15 10:37:43 tanus abrt[23994]: Saved core dump of pid 23990 to
core.23990 (339316736 bytes)

Jan 15 10:37:47 tanus kamailio: INFO: <core> [sctp_core.c:70]:
sctp_core_check_support(): SCTP API not enabled - if you want to use it,
load sctp module

Jan 15 10:37:47 tanus kamailio: WARNING: <core> [daemonize.c:360]:
daemonize(): pid file contains old pid, replacing pid

Jan 15 10:37:47 tanus /usr/local/sbin/kamailio[24042]: INFO: rr
[../outbound/api.h:54]: ob_load_api(): unable to import bind_ob - maybe
module is not loaded

Jan 15 10:37:47 tanus /usr/local/sbin/kamailio[24042]: INFO: rr
[rr_mod.c:160]: mod_init(): outbound module not available

Jan 15 10:37:47 tanus /usr/local/sbin/kamailio[24042]: INFO: usrloc
[hslot.c:53]: ul_init_locks(): locks array size 1024

 

Regards,

 

Igor.

 

De : Igor Potjevlesch [mailto:igor.potjevlesch at gmail.com] 
Envoyé : vendredi 15 janvier 2016 10:47
À : miconda at gmail.com; 'Kamailio (SER) - Users Mailing List'
<sr-users at lists.sip-router.org>
Objet : RE: [SR-Users] Kamailio 4.2.6 crash

 

Hello Daniel,

 

I move to 4.2.7. This morning a new crash occurred. I got two coredump:

 

Core was generated by `/usr/local/sbin/kamailio -P /var/run/kamailio.pid -m
256 -M 64'.

Program terminated with signal 11, Segmentation fault.

#0  0x000000000062245e in qm_status (qm=0x7f9abd447000) at
mem/q_malloc.c:788

788                                     f!=&(qm->free_hash[h].head);
f=f->u.nxt_free, i++, j++){

 

(gdb) bt full

#0  0x000000000062245e in qm_status (qm=0x7f9abd447000) at
mem/q_malloc.c:788

        f = 0x30a012010010a0d

        i = 57

        j = 4

        h = 4

        unused = 0

        memlog = 5

        mem_summary = 3

        __FUNCTION__ = "qm_status"

#1  0x000000000061a795 in qm_debug_frag (qm=0x7f9abd447000,
f=0x7f9abd767970) at mem/q_malloc.c:160

        __FUNCTION__ = "qm_debug_frag"

#2  0x000000000061ca58 in qm_free (qm=0x7f9abd447000, p=0x7f9abd7679a0,
file=0x7f9ace7e2662 "dialog: dlg_hash.c", func=0x7f9ace7e50e2 "destroy_dlg",
line=380) at mem/q_malloc.c:468

        f = 0x7f9abd767970

        size = 176

        next = 0x400

        prev = 0x7fff39bc7910

        __FUNCTION__ = "qm_free"

#3  0x00007f9ace7a64ef in destroy_dlg (dlg=0x7f9abd7660b8) at dlg_hash.c:380

        ret = 0

        var = 0x7f9ad54374e8

        __FUNCTION__ = "destroy_dlg"

#4  0x00007f9ace7a67df in destroy_dlg_table () at dlg_hash.c:419

        dlg = 0x0

        l_dlg = 0x7f9abd7660b8

        i = 665

        __FUNCTION__ = "destroy_dlg_table"

#5  0x00007f9ace771263 in mod_destroy () at dialog.c:783

No locals.

#6  0x00000000005929ee in destroy_modules () at sr_module.c:811

        t = 0x7f9ad52b8d00

        foo = 0x7f9ad52b8a30

        __FUNCTION__ = "destroy_modules"

#7  0x000000000049c917 in cleanup (show_status=1) at main.c:569

        memlog = 0

        __FUNCTION__ = "cleanup"

#8  0x000000000049dee4 in shutdown_children (sig=15, show_status=1) at
main.c:711

        __FUNCTION__ = "shutdown_children"

#9  0x00000000004a04ba in handle_sigs () at main.c:802

        chld = 0

        chld_status = 139

        memlog = -1119369840

        __FUNCTION__ = "handle_sigs"

#10 0x00000000004a82eb in main_loop () at main.c:1757

        i = 8

        pid = 24021

        si = 0x0

        si_desc = "udp receiver child=7
sock=91.213.145.60:5060\000\177\000\000`~\274\071\377\177\000\000\033{N\000\
000\000\000\000\260~\274\071\377\177\000\000\004\000\000\000\000\000\000\000
`TA\000\000\000\000\000(\305G\275\232\177", '\000' <repeats 14 times>,
"\001\000\000\000\260~\274\071\377\177\000\000\276{N\000\000\000\000"

        nrprocs = 8

        __FUNCTION__ = "main_loop"

#11 0x00000000004acfab in main (argc=7, argv=0x7fff39bc8128) at main.c:2581

        cfg_stream = 0x2392010

        c = -1

        r = 0

        tmp = 0x7fff39bc8f70 ""

        tmp_len = 32767

        port = 968654846

        proto = 0

        options = 0x7033b8
":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:"

        ret = -1

        seed = 1451157380

        rfd = 4

        debug_save = 0

        debug_flag = 0

        dont_fork_cnt = 0

        n_lst = 0x40d134

        p = 0xc2 <Address 0xc2 out of bounds>

        __FUNCTION__ = "main"

 

Second one:

 

Core was generated by `/usr/local/sbin/kamailio -P /var/run/kamailio.pid -m
256 -M 64'.

Program terminated with signal 11, Segmentation fault.

#0  0x000000000062245e in qm_status (qm=0x7f9abd447000) at
mem/q_malloc.c:788

788                                     f!=&(qm->free_hash[h].head);
f=f->u.nxt_free, i++, j++){

 

(gdb) bt full

#0  0x000000000062245e in qm_status (qm=0x7f9abd447000) at
mem/q_malloc.c:788

        f = 0x30a012010010a0d

        i = 16

        j = 4

        h = 4

        unused = 0

        memlog = 5

        mem_summary = 3

        __FUNCTION__ = "qm_status"

#1  0x000000000061a420 in qm_debug_frag (qm=0x7f9abd447000,
f=0x7f9abd767300) at mem/q_malloc.c:150

        __FUNCTION__ = "qm_debug_frag"

#2  0x000000000061ca58 in qm_free (qm=0x7f9abd447000, p=0x7f9abd767330,
file=0x7f9ad3d2f34d "tm: h_table.c", func=0x7f9ad3d2f628 "free_cell",
line=186) at mem/q_malloc.c:468

        f = 0x7f9abd767300

        size = 40

        next = 0x400

        prev = 0x7fff39bc7b80

        __FUNCTION__ = "qm_free"

#3  0x00007f9ad3c70c9d in free_cell (dead_cell=0x7f9abd79b5c0) at
h_table.c:186

        b = 0x7f9abd767330 "INVITE sip:00447798156873 at goren
SIP/2.0\r\nRecord-Route: <sip:A.B.C.D;lr;did=4b7.60a>\r\nCSeq: 1
INVITE\r\nCall-ID: 729d-7e9-015201693735-DSQ-1-A.B.C_leg2\r\nFrom:
<sip:0123456789 at D.C.B.A>;"...

        i = 0

        rpl = 0x0

        tt = 0x7f9abd5d8bf8

        foo = 0x7fff39bc7c50

        cbs = 0x0

        cbs_tmp = 0x7f9abd7600a0

        __FUNCTION__ = "free_cell"

#4  0x00007f9ad3cb5a1c in wait_handler (ti=1300688687,
wait_tl=0x7f9abd79b640, data=0x7f9abd79b5c0) at timer.c:675

        p_cell = 0x7f9abd79b5c0

        ret = 1

#5  0x00000000005fd30f in timer_list_expire (t=1300688687, h=0x7f9abd4c0908,
slow_l=0x7f9abd4c36d8, slow_mark=17084) at timer.c:888

        tl = 0x7f9abd79b640

        ret = 1300688687

#6  0x00000000005fd757 in timer_handler () at timer.c:953

        saved_ticks = 1300688687

        run_slow_timer = 0

        i = 700

        __FUNCTION__ = "timer_handler"

#7  0x00000000005fdbc5 in timer_main () at timer.c:992

No locals.

#8  0x00000000004a77e6 in main_loop () at main.c:1700

        i = 8

        pid = 0

        si = 0x0

        si_desc = "udp receiver child=7
sock=91.213.145.60:5060\000\177\000\000`~\274\071\377\177\000\000\033{N\000\
000\000\000\000\260~\274\071\377\177\000\000\004\000\000\000\000\000\000\000
`TA\000\000\000\000\000(\305G\275\232\177", '\000' <repeats 14 times>,
"\001\000\000\000\260~\274\071\377\177\000\000\276{N\000\000\000\000"

        nrprocs = 8

        __FUNCTION__ = "main_loop"

#9  0x00000000004acfab in main (argc=7, argv=0x7fff39bc8128) at main.c:2581

        cfg_stream = 0x2392010

        c = -1

        r = 0

        tmp = 0x7fff39bc8f70 ""

        tmp_len = 32767

        port = 968654846

        proto = 0

        options = 0x7033b8
":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:"

        ret = -1

        seed = 1451157380

        rfd = 4

        debug_save = 0

        debug_flag = 0

        dont_fork_cnt = 0

        n_lst = 0x40d134

        p = 0xc2 <Address 0xc2 out of bounds>

        __FUNCTION__ = "main"

 

Regards,

 

Igor.

 

De : Igor Potjevlesch [mailto:igor.potjevlesch at gmail.com] 
Envoyé : lundi 4 janvier 2016 10:55
À : miconda at gmail.com <mailto:miconda at gmail.com> ; 'Kamailio (SER) - Users
Mailing List' <sr-users at lists.sip-router.org
<mailto:sr-users at lists.sip-router.org> >
Objet : RE: [SR-Users] Kamailio 4.2.6 crash

 

Hi Daniel,

 

I wish you an happy new year! I will schedule this update in the next few
days.

 

Thank you for your support.

 

Regards,

 

Igor.

 

De : Daniel-Constantin Mierla [mailto:miconda at gmail.com] 
Envoyé : vendredi 18 décembre 2015 11:28
À : Igor Potjevlesch <igor.potjevlesch at gmail.com
<mailto:igor.potjevlesch at gmail.com> >; 'Kamailio (SER) - Users Mailing List'
<sr-users at lists.sip-router.org <mailto:sr-users at lists.sip-router.org> >
Objet : Re: [SR-Users] Kamailio 4.2.6 crash

 

Hello,

can you upgrade to 4.2.7, it has some fixes that may be related to this
issue -- there are no changes to config/database that need to be done.

Cheers,
Daniel

On 16/12/15 15:06, Igor Potjevlesch wrote:

Hello,

 

Here is the output:

 

kamailio -v

version: kamailio 4.2.6 (x86_64/linux) db77ac

flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS,
DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC,
F_MALLOC, DBG_F_MALLOC, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE,
USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES

ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16,
MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 64MB

poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.

id: db77ac 

compiled on 12:19:48 Oct  2 2015 with gcc 4.4.7

 

Regards,

 

Igor.

 

De : Daniel-Constantin Mierla [mailto:miconda at gmail.com] 
Envoyé : lundi 14 décembre 2015 16:16
À : Igor Potjevlesch  <mailto:igor.potjevlesch at gmail.com>
<igor.potjevlesch at gmail.com>; 'Kamailio (SER) - Users Mailing List'
<mailto:sr-users at lists.sip-router.org> <sr-users at lists.sip-router.org>
Objet : Re: [SR-Users] Kamailio 4.2.6 crash

 

Hello,

yes, it is the output I wanted.

What is the exact version are you running? It is the output of 'kamailio
-v'.

Cheers,
Daniel

On 14/12/15 15:53, Igor Potjevlesch wrote:

Hello Daniel,

 

Is this the expected output?:

 

(gdb) frame 0

#0  0x0000000000619694 in fm_extract_free (qm=0x7f0114c07000,
frag=0x7f0114f0e4b0) at mem/f_malloc.c:206

206             *pf=frag->u.nxt_free;

(gdb) list

201             int hash;

202

203             pf = frag->prv_free;

204             hash = GET_HASH(frag->size);

205

206             *pf=frag->u.nxt_free;

207

208             if(frag->u.nxt_free) frag->u.nxt_free->prv_free = pf;

209

210             qm->ffrags--;

(gdb)

 

Regards,

 

Igor.

 

De : sr-users [mailto:sr-users-bounces at lists.sip-router.org] De la part de
Daniel-Constantin Mierla
Envoyé : lundi 14 décembre 2015 13:15
À : Kamailio (SER) - Users Mailing List
<mailto:sr-users at lists.sip-router.org> <sr-users at lists.sip-router.org>
Objet : Re: [SR-Users] Kamailio 4.2.6 crash

 

Hello,

can you provide the list output in gdb for frame 0:

frame 0
list

Cheers,
Daniel

 

-- 
Daniel-Constantin Mierla
http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda>  -
http://www.linkedin.com/in/miconda
Book: SIP Routing With Kamailio - http://www.asipto.com
http://miconda.eu

 

-- 
Daniel-Constantin Mierla
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Book: SIP Routing With Kamailio - http://www.asipto.com
http://miconda.eu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.sip-router.org/pipermail/sr-users/attachments/20160115/5042eefc/attachment.html>


More information about the sr-users mailing list