14 Dec
2015
14 Dec
'15
8:45 a.m.
I have 3 kamailio 434 registrars replicating between each other using dmq usrloc module
I have recently noticed that all of the registrars crash at random intervals over the
course of the day, they all dont crash at the same time, but randomly, independantly of
each other
I cannot see any unusual traffic arriving at the systems that may be a cause for the
crash
I'd appreciate assistance trying to get to the bottom of the issue
Kamailio Version:
```
# /usr/sbin/kamailio -V
version: kamailio 434 (x86_64/linux) 0ec860
flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE,
USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, DBG_QM_MALLOC, USE_FUTEX,
FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST,
HAVE_RESOLV_RES
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024,
BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select
id: 0ec860
compiled on 17:42:12 Nov 25 2015 with gcc 447
```
Message in the log file when the kamailio application crashes:
```
Dec 14 01:46:54 /usr/sbin/kamailio[16205]: ALERT: <core> [mainc:728]: handle_sigs():
child process 16234 exited by a signal 11
Dec 14 01:46:54 /usr/sbin/kamailio[16205]: ALERT: <core> [mainc:731]: handle_sigs():
core was generated
Dec 14 01:46:54 /usr/sbin/kamailio[16205]: INFO: <core> [mainc:743]: handle_sigs():
terminating due to SIGCHLD
Dec 14 01:46:54 /usr/sbin/kamailio[16246]: INFO: <core> [mainc:798]: sig_usr():
signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16244]: INFO: <core> [mainc:798]: sig_usr():
signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16232]: INFO: <core> [mainc:798]: sig_usr():
signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16243]: INFO: <core> [mainc:798]: sig_usr():
signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16242]: INFO: <core> [mainc:798]: sig_usr():
signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16228]: INFO: <core> [mainc:798]: sig_usr():
signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16245]: INFO: <core> [mainc:798]: sig_usr():
signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16241]: INFO: <core> [mainc:798]: sig_usr():
signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16235]: INFO: <core> [mainc:798]: sig_usr():
signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16240]: INFO: <core> [mainc:798]: sig_usr():
signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16225]: INFO: <core> [mainc:798]: sig_usr():
signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16236]: INFO: <core> [mainc:798]: sig_usr():
signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16233]: INFO: <core> [mainc:798]: sig_usr():
signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16220]: INFO: <core> [mainc:798]: sig_usr():
signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16221]: INFO: <core> [mainc:798]: sig_usr():
signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16230]: INFO: <core> [mainc:798]: sig_usr():
signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16237]: INFO: <core> [mainc:798]: sig_usr():
signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16231]: INFO: <core> [mainc:798]: sig_usr():
signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16217]: INFO: <core> [mainc:798]: sig_usr():
signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16229]: INFO: <core> [mainc:798]: sig_usr():
signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16218]: INFO: <core> [mainc:798]: sig_usr():
signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16219]: INFO: <core> [mainc:798]: sig_usr():
signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16226]: INFO: <core> [mainc:798]: sig_usr():
signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16215]: INFO: <core> [mainc:798]: sig_usr():
signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16216]: INFO: <core> [mainc:798]: sig_usr():
signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16239]: INFO: <core> [mainc:798]: sig_usr():
signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16212]: INFO: <core> [mainc:798]: sig_usr():
signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16223]: INFO: <core> [mainc:798]: sig_usr():
signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16214]: INFO: <core> [mainc:798]: sig_usr():
signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16213]: INFO: <core> [mainc:798]: sig_usr():
signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16208]: INFO: <core> [mainc:798]: sig_usr():
signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16207]: INFO: <core> [mainc:798]: sig_usr():
signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16211]: INFO: <core> [mainc:798]: sig_usr():
signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16238]: INFO: <core> [mainc:798]: sig_usr():
signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16222]: INFO: <core> [mainc:798]: sig_usr():
signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16210]: INFO: <core> [mainc:798]: sig_usr():
signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16227]: INFO: <core> [mainc:798]: sig_usr():
signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16224]: INFO: <core> [mainc:798]: sig_usr():
signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16206]: INFO: <core> [mainc:798]: sig_usr():
signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16209]: INFO: <core> [mainc:798]: sig_usr():
signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16205]: WARNING: tm [h_tablec:130]: free_cell_helper():
removed cell 0x7f5a40a084f8 is still linked in hash table (h_tablec:440)
Dec 14 01:46:54 /usr/sbin/kamailio[16205]: WARNING: tm [h_tablec:130]: free_cell_helper():
removed cell 0x7f5a40a12390 is still linked in hash table (h_tablec:440)
```
Core file synopsis is:
```
Core was generated by `/usr/sbin/kamailio -P /var/run/kamailiopid -m 1024 -M 64 -u root -g
root'
Program terminated with signal 11, Segmentation fault
#0 0x00007f5a82f68985 in mem_insert_ucontact (_r=0x7f5a409ec948, _c=0x7ffcc26f3140,
_ci=0x7f5a81418500) at urecordc:152
152 if_update_stat( _r->slot, _r->slot->d->contacts, 1);
```
I still have the core file, however, it contains some sensitive info, so I can send
backtraces directly to you if required
Please let me know if you require any additional info
Thanks
---
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/440
14 Dec
14 Dec
3:36 p.m.
Can you give the output of:
```
p *_r
p *_r->slot
p *_r->slot->d
p *_r->slot->d->contacts
```
---
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/440#issuecomment-164469254
3:42 p.m.
```
(gdb) p *_r
$1 = {domain = 0x746d616e3a706973, aor = {s = 0x3931403130747365 <Address
0x3931403130747365 out of bounds>, len = 909192754}, aorhash = 893007672, contacts =
0x39383d7361696c61, slot = 0x3133312e3130312e, prev = 0x3336327e3032312e, next =
0x317e38}
(gdb) p *_r->slot
Cannot access memory at address 0x3133312e3130312e
(gdb) p *_r->slot->d
Cannot access memory at address 0x3133312e31303146
(gdb) p *_r->slot->d->contacts
Cannot access memory at address 0x3133312e31303146
```
---
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/440#issuecomment-164470877
16 Dec
16 Dec
7:18 a.m.
It looks like the structure has invalid content. Do you see any error messages in syslog
before the crash happened?
---
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/440#issuecomment-165020997
7:53 a.m.
all I see in the kamailio log is what is posted in the 1st post, I re-post it below again,
the previous log above these is for the 13 Dec at 20:57, with a warning, the next log
after these below is the first line of Kamailio starting up.
```
Dec 14 01:46:54 /usr/sbin/kamailio[16205]: ALERT: <core> [main.c:728]:
handle_sigs(): child process 16234 exited by a signal 11
Dec 14 01:46:54 /usr/sbin/kamailio[16205]: ALERT: <core> [main.c:731]:
handle_sigs(): core was generated
Dec 14 01:46:54 /usr/sbin/kamailio[16205]: INFO: <core> [main.c:743]: handle_sigs():
terminating due to SIGCHLD
Dec 14 01:46:54 /usr/sbin/kamailio[16246]: INFO: <core> [main.c:798]: sig_usr():
signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16244]: INFO: <core> [main.c:798]: sig_usr():
signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16232]: INFO: <core> [main.c:798]: sig_usr():
signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16243]: INFO: <core> [main.c:798]: sig_usr():
signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16242]: INFO: <core> [main.c:798]: sig_usr():
signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16228]: INFO: <core> [main.c:798]: sig_usr():
signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16245]: INFO: <core> [main.c:798]: sig_usr():
signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16241]: INFO: <core> [main.c:798]: sig_usr():
signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16235]: INFO: <core> [main.c:798]: sig_usr():
signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16240]: INFO: <core> [main.c:798]: sig_usr():
signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16225]: INFO: <core> [main.c:798]: sig_usr():
signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16236]: INFO: <core> [main.c:798]: sig_usr():
signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16233]: INFO: <core> [main.c:798]: sig_usr():
signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16220]: INFO: <core> [main.c:798]: sig_usr():
signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16221]: INFO: <core> [main.c:798]: sig_usr():
signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16230]: INFO: <core> [main.c:798]: sig_usr():
signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16237]: INFO: <core> [main.c:798]: sig_usr():
signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16231]: INFO: <core> [main.c:798]: sig_usr():
signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16217]: INFO: <core> [main.c:798]: sig_usr():
signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16229]: INFO: <core> [main.c:798]: sig_usr():
signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16218]: INFO: <core> [main.c:798]: sig_usr():
signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16219]: INFO: <core> [main.c:798]: sig_usr():
signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16226]: INFO: <core> [main.c:798]: sig_usr():
signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16215]: INFO: <core> [main.c:798]: sig_usr():
signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16216]: INFO: <core> [main.c:798]: sig_usr():
signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16239]: INFO: <core> [main.c:798]: sig_usr():
signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16212]: INFO: <core> [main.c:798]: sig_usr():
signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16223]: INFO: <core> [main.c:798]: sig_usr():
signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16214]: INFO: <core> [main.c:798]: sig_usr():
signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16213]: INFO: <core> [main.c:798]: sig_usr():
signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16208]: INFO: <core> [main.c:798]: sig_usr():
signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16207]: INFO: <core> [main.c:798]: sig_usr():
signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16211]: INFO: <core> [main.c:798]: sig_usr():
signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16238]: INFO: <core> [main.c:798]: sig_usr():
signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16222]: INFO: <core> [main.c:798]: sig_usr():
signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16210]: INFO: <core> [main.c:798]: sig_usr():
signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16227]: INFO: <core> [main.c:798]: sig_usr():
signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16224]: INFO: <core> [main.c:798]: sig_usr():
signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16206]: INFO: <core> [main.c:798]: sig_usr():
signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16209]: INFO: <core> [main.c:798]: sig_usr():
signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16205]: WARNING: tm [h_table.c:130]:
free_cell_helper(): removed cell 0x7f5a40a084f8 is still linked in hash table
(h_table.c:440)
Dec 14 01:46:54 /usr/sbin/kamailio[16205]: WARNING: tm [h_table.c:130]:
free_cell_helper(): removed cell 0x7f5a40a12390 is still linked in hash table
(h_table.c:440)
```
I have multiple core files, I could try another one if we cannot get any info out of this
one.
---
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/440#issuecomment-165025696
8:13 a.m.
To be more specific now: any log messages before the crash from the process with pid 16234
?
---
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/440#issuecomment-165028337
8:18 a.m.
no, not around that time, but an entire log search for that pid results in the following:
the log file date starts from 30th June, so I presume it would have caught anything if it
was logged.
```
# fgrep '\[16234\]' kamailio.log
# fgrep '16234' kamailio.log
Dec 13 15:23:08 /usr/sbin/kamailio[16234]: ERROR: dmq_usrloc [usrloc_sync.c:67]:
add_contact(): Invalid cseq
Dec 13 15:23:08 /usr/sbin/kamailio[16234]: ERROR: dmq_usrloc [usrloc_sync.c:67]:
add_contact(): Invalid cseq
Dec 13 15:23:08 /usr/sbin/kamailio[16234]: ERROR: dmq_usrloc [usrloc_sync.c:67]:
add_contact(): Invalid cseq
Dec 13 15:23:08 /usr/sbin/kamailio[16234]: ERROR: dmq_usrloc [usrloc_sync.c:67]:
add_contact(): Invalid cseq
Dec 14 01:46:54 /usr/sbin/kamailio[16205]: ALERT: <core> [main.c:728]:
handle_sigs(): child process 16234 exited by a signal 11
```
---
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/440#issuecomment-165029061
8:23 a.m.
actually looking at the log for the mesages around 15:23 on Dec 13, it looks like it
crashed and restarted at that time too, those dmq messages appear to be just after the
restart, when, I presume, dmq is replacting the contacts upon the restart, full log at
that time is as follows:
```
Dec 13 15:22:57 /usr/sbin/kamailio[15553]: ALERT: <core> [main.c:728]:
handle_sigs(): child process 15582 exited by a signal 11
Dec 13 15:22:57 /usr/sbin/kamailio[15553]: ALERT: <core> [main.c:731]:
handle_sigs(): core was generated
Dec 13 15:22:57 /usr/sbin/kamailio[15553]: INFO: <core> [main.c:743]:
handle_sigs(): terminating due to SIGCHLD
Dec 13 15:22:57 /usr/sbin/kamailio[15590]: INFO: <core> [main.c:798]: sig_usr():
signal 15 received
Dec 13 15:22:57 /usr/sbin/kamailio[15592]: INFO: <core> [main.c:798]: sig_usr():
signal 15 received
Dec 13 15:22:57 /usr/sbin/kamailio[15585]: INFO: <core> [main.c:798]: sig_usr():
signal 15 received
Dec 13 15:22:57 /usr/sbin/kamailio[15594]: INFO: <core> [main.c:798]: sig_usr():
signal 15 received
Dec 13 15:22:57 /usr/sbin/kamailio[15584]: INFO: <core> [main.c:798]: sig_usr():
signal 15 received
Dec 13 15:22:57 /usr/sbin/kamailio[15589]: INFO: <core> [main.c:798]: sig_usr():
signal 15 received
Dec 13 15:22:57 /usr/sbin/kamailio[15581]: INFO: <core> [main.c:798]: sig_usr():
signal 15 received
Dec 13 15:22:57 /usr/sbin/kamailio[15579]: INFO: <core> [main.c:798]: sig_usr():
signal 15 received
Dec 13 15:22:57 /usr/sbin/kamailio[15583]: INFO: <core> [main.c:798]: sig_usr():
signal 15 received
Dec 13 15:22:57 /usr/sbin/kamailio[15593]: INFO: <core> [main.c:798]: sig_usr():
signal 15 received
Dec 13 15:22:57 /usr/sbin/kamailio[15580]: INFO: <core> [main.c:798]: sig_usr():
signal 15 received
Dec 13 15:22:57 /usr/sbin/kamailio[15571]: INFO: <core> [main.c:798]: sig_usr():
signal 15 received
Dec 13 15:22:57 /usr/sbin/kamailio[15575]: INFO: <core> [main.c:798]: sig_usr():
signal 15 received
Dec 13 15:22:57 /usr/sbin/kamailio[15578]: INFO: <core> [main.c:798]: sig_usr():
signal 15 received
Dec 13 15:22:57 /usr/sbin/kamailio[15554]: INFO: <core> [main.c:798]: sig_usr():
signal 15 received
Dec 13 15:22:57 /usr/sbin/kamailio[15556]: INFO: <core> [main.c:798]: sig_usr():
signal 15 received
Dec 13 15:22:57 /usr/sbin/kamailio[15567]: INFO: <core> [main.c:798]: sig_usr():
signal 15 received
Dec 13 15:22:57 /usr/sbin/kamailio[15569]: INFO: <core> [main.c:798]: sig_usr():
signal 15 received
Dec 13 15:22:57 /usr/sbin/kamailio[15566]: INFO: <core> [main.c:798]: sig_usr():
signal 15 received
Dec 13 15:22:57 /usr/sbin/kamailio[15561]: INFO: <core> [main.c:798]: sig_usr():
signal 15 received
Dec 13 15:22:57 /usr/sbin/kamailio[15568]: INFO: <core> [main.c:798]: sig_usr():
signal 15 received
Dec 13 15:22:57 /usr/sbin/kamailio[15565]: INFO: <core> [main.c:798]: sig_usr():
signal 15 received
Dec 13 15:22:57 /usr/sbin/kamailio[15588]: INFO: <core> [main.c:798]: sig_usr():
signal 15 received
Dec 13 15:22:57 /usr/sbin/kamailio[15570]: INFO: <core> [main.c:798]: sig_usr():
signal 15 received
Dec 13 15:22:57 /usr/sbin/kamailio[15591]: INFO: <core> [main.c:798]: sig_usr():
signal 15 received
Dec 13 15:22:57 /usr/sbin/kamailio[15559]: INFO: <core> [main.c:798]: sig_usr():
signal 15 received
Dec 13 15:22:57 /usr/sbin/kamailio[15563]: INFO: <core> [main.c:798]: sig_usr():
signal 15 received
Dec 13 15:22:57 /usr/sbin/kamailio[15560]: INFO: <core> [main.c:798]: sig_usr():
signal 15 received
Dec 13 15:22:57 /usr/sbin/kamailio[15562]: INFO: <core> [main.c:798]: sig_usr():
signal 15 received
Dec 13 15:22:57 /usr/sbin/kamailio[15557]: INFO: <core> [main.c:798]: sig_usr():
signal 15 received
Dec 13 15:22:57 /usr/sbin/kamailio[15564]: INFO: <core> [main.c:798]: sig_usr():
signal 15 received
Dec 13 15:22:57 /usr/sbin/kamailio[15555]: INFO: <core> [main.c:798]: sig_usr():
signal 15 received
Dec 13 15:22:57 /usr/sbin/kamailio[15572]: INFO: <core> [main.c:798]: sig_usr():
signal 15 received
Dec 13 15:22:57 /usr/sbin/kamailio[15587]: INFO: <core> [main.c:798]: sig_usr():
signal 15 received
Dec 13 15:22:57 /usr/sbin/kamailio[15558]: INFO: <core> [main.c:798]: sig_usr():
signal 15 received
Dec 13 15:22:57 /usr/sbin/kamailio[15576]: INFO: <core> [main.c:798]: sig_usr():
signal 15 received
Dec 13 15:22:57 /usr/sbin/kamailio[15586]: INFO: <core> [main.c:798]: sig_usr():
signal 15 received
Dec 13 15:22:57 /usr/sbin/kamailio[15574]: INFO: <core> [main.c:798]: sig_usr():
signal 15 received
Dec 13 15:22:57 /usr/sbin/kamailio[15573]: INFO: <core> [main.c:798]: sig_usr():
signal 15 received
Dec 13 15:22:57 /usr/sbin/kamailio[15577]: INFO: <core> [main.c:798]: sig_usr():
signal 15 received
Dec 13 15:22:57 /usr/sbin/kamailio[15553]: WARNING: tm [h_table.c:130]:
free_cell_helper(): removed cell 0x7f380d9f91c8 is still linked in hash table
(h_table.c:440)
Dec 13 15:22:57 /usr/sbin/kamailio[15553]: WARNING: tm [h_table.c:130]:
free_cell_helper(): removed cell 0x7f380d9fdac0 is still linked in hash table
(h_table.c:440)
Dec 13 15:22:57 /usr/sbin/kamailio[15553]: WARNING: tm [h_table.c:130]:
free_cell_helper(): removed cell 0x7f380da1a8d0 is still linked in hash table
(h_table.c:440)
Dec 13 15:22:57 /usr/sbin/kamailio[15553]: WARNING: tm [h_table.c:130]:
free_cell_helper(): removed cell 0x7f380da19228 is still linked in hash table
(h_table.c:440)
Dec 13 15:22:57 /usr/sbin/kamailio[15553]: INFO: <core> [sctp_core.c:53]:
sctp_core_destroy(): SCTP API not initialized
Dec 13 15:23:08 kamailio: INFO: <core> [sctp_core.c:75]: sctp_core_check_support():
SCTP API not enabled - if you want to use it, load sctp module
Dec 13 15:23:08 kamailio: INFO: <core> [tcp_main.c:4657]: init_tcp(): using
epoll_lt as the io watch method (auto detected)
Dec 13 15:23:08 /usr/sbin/kamailio[16205]: INFO: rr [../outbound/api.h:54]:
ob_load_api(): Failed to import bind_ob
Dec 13 15:23:08 /usr/sbin/kamailio[16205]: INFO: rr [rr_mod.c:174]: mod_init(): outbound
module not available
Dec 13 15:23:08 /usr/sbin/kamailio[16205]: INFO: usrloc [hslot.c:51]: ul_init_locks():
locks array size 1024
Dec 13 15:23:08 /usr/sbin/kamailio[16205]: INFO: path [../outbound/api.h:54]:
ob_load_api(): Failed to import bind_ob
Dec 13 15:23:08 /usr/sbin/kamailio[16205]: INFO: path [path_mod.c:147]: mod_init():
outbound module not available
Dec 13 15:23:08 /usr/sbin/kamailio[16205]: INFO: dmq_usrloc [dmq_usrloc.c:67]:
mod_init(): dmq usrloc replication mode = 1
Dec 13 15:23:08 /usr/sbin/kamailio[16205]: INFO: permissions [permissions.c:608]:
mod_init(): default allow file (//etc/kamailio/permissions.allow) not found => empty
rule set
Dec 13 15:23:08 /usr/sbin/kamailio[16205]: INFO: permissions [permissions.c:617]:
mod_init(): default deny file (//etc/kamailio/permissions.deny) not found => empty rule
set
Dec 13 15:23:08 /usr/sbin/kamailio[16205]: INFO: db_text [dbt_base.c:92]: dbt_init():
using database at: /etc/kamailio/db_text
Dec 13 15:23:08 /usr/sbin/kamailio[16205]: INFO: db_text [dbt_base.c:92]: dbt_init():
using database at: /etc/kamailio/db_text
Dec 13 15:23:08 /usr/sbin/kamailio[16205]: INFO: <core> [udp_server.c:150]:
probe_max_receive_buffer(): SO_RCVBUF is initially 124928
Dec 13 15:23:08 /usr/sbin/kamailio[16205]: INFO: <core> [udp_server.c:200]:
probe_max_receive_buffer(): SO_RCVBUF is finally 249856
Dec 13 15:23:08 /usr/sbin/kamailio[16205]: INFO: <core> [udp_server.c:150]:
probe_max_receive_buffer(): SO_RCVBUF is initially 124928
Dec 13 15:23:08 /usr/sbin/kamailio[16205]: INFO: <core> [udp_server.c:200]:
probe_max_receive_buffer(): SO_RCVBUF is finally 249856
Dec 13 15:23:08 /usr/sbin/kamailio[16224]: INFO: db_text [dbt_base.c:92]: dbt_init():
using database at: /etc/kamailio/db_text
Dec 13 15:23:08 /usr/sbin/kamailio[16224]: INFO: db_text [dbt_base.c:92]: dbt_init():
using database at: /etc/kamailio/db_text
Dec 13 15:23:08 /usr/sbin/kamailio[16233]: INFO: ctl [io_listener.c:214]:
io_listen_loop(): io_listen_loop: using epoll_lt io watch method (config)
Dec 13 15:23:08 /usr/sbin/kamailio[16236]: ERROR: dmq_usrloc [usrloc_sync.c:67]:
add_contact(): Invalid cseq
Dec 13 15:23:08 /usr/sbin/kamailio[16234]: ERROR: dmq_usrloc [usrloc_sync.c:67]:
add_contact(): Invalid cseq
Dec 13 15:23:08 /usr/sbin/kamailio[16234]: ERROR: dmq_usrloc [usrloc_sync.c:67]:
add_contact(): Invalid cseq
Dec 13 15:23:08 /usr/sbin/kamailio[16235]: ERROR: dmq_usrloc [usrloc_sync.c:67]:
add_contact(): Invalid cseq
Dec 13 15:23:08 /usr/sbin/kamailio[16234]: ERROR: dmq_usrloc [usrloc_sync.c:67]:
add_contact(): Invalid cseq
Dec 13 15:23:08 /usr/sbin/kamailio[16234]: ERROR: dmq_usrloc [usrloc_sync.c:67]:
add_contact(): Invalid cseq
```
---
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/440#issuecomment-165029746
9:37 p.m.
Can you attach the backtrace (just 'bt', not 'bt full' for the moment)
from gdb here? Replace any IP/sensitive info there, but I would like to see the backtrace
here to correlate easier with the logs above.
---
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/440#issuecomment-165247807
9:44 p.m.
sure, here it is:
```
# gdb /usr/sbin/kamailio /core.16234
<snip loading symbols output>
Core was generated by `/usr/sbin/kamailio -P /var/run/kamailio.pid -m 1024 -M 64 -u root
-g root'.
Program terminated with signal 11, Segmentation fault.
#0 0x00007f5a82f68985 in mem_insert_ucontact (_r=0x7f5a409ec948, _c=0x7ffcc26f3140,
_ci=0x7f5a81418500) at urecord.c:152
152 if_update_stat( _r->slot, _r->slot->d->contacts, 1);
Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.166.el6_7.3.x86_64
libxml2-2.7.6-20.el6_7.1.x86_64 zlib-1.2.3-29.el6.x86_64
(gdb) bt
#0 0x00007f5a82f68985 in mem_insert_ucontact (_r=0x7f5a409ec948, _c=0x7ffcc26f3140,
_ci=0x7f5a81418500) at urecord.c:152
#1 0x00007f5a82f6bb9d in insert_ucontact (_r=0x7f5a409ec948, _contact=0x7ffcc26f3140,
_ci=0x7f5a81418500, _c=0x7ffcc26f3150) at urecord.c:568
#2 0x00007f5a8120b613 in add_contact (aor=..., ci=0x7f5a81418500) at usrloc_sync.c:73
#3 0x00007f5a8121178b in usrloc_dmq_handle_msg (msg=0x7f5a409f4000, resp=0x7ffcc26f33b0,
node=0x7f5a409e88a8) at usrloc_sync.c:390
#4 0x00007f5a8143d22b in worker_loop (id=0) at worker.c:105
#5 0x00007f5a8142128c in child_init (rank=0) at dmq.c:293
#6 0x0000000000595308 in init_mod_child (m=0x7f5a84f32088, rank=0) at sr_module.c:915
#7 0x0000000000595043 in init_mod_child (m=0x7f5a84f32570, rank=0) at sr_module.c:912
#8 0x0000000000595043 in init_mod_child (m=0x7f5a84f32838, rank=0) at sr_module.c:912
#9 0x0000000000595043 in init_mod_child (m=0x7f5a84f33440, rank=0) at sr_module.c:912
#10 0x0000000000595043 in init_mod_child (m=0x7f5a84f33928, rank=0) at sr_module.c:912
#11 0x0000000000595043 in init_mod_child (m=0x7f5a84f33d68, rank=0) at sr_module.c:912
#12 0x0000000000595043 in init_mod_child (m=0x7f5a84f34640, rank=0) at sr_module.c:912
#13 0x0000000000595612 in init_child (rank=0) at sr_module.c:941
#14 0x00000000004a8c7d in main_loop () at main.c:1654
#15 0x00000000004aecd2 in main (argc=11, argv=0x7ffcc26f3c88) at main.c:2553
```
---
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/440#issuecomment-165250172
21 Dec
21 Dec
1:52 p.m.
I pushed a patch for it - commit 5937b04 - as I could spot, there was no locking done
before the insert, but only unlocking. Would be good if you can test and report the
results.
---
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/440#issuecomment-166310109
5:13 p.m.
Thanks, I am away this week, I will test it out early next week and update you here.
Thanks for looking into it.
---
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/440#issuecomment-166362706
28 Dec
28 Dec
2:40 p.m.
I have updated the registars with v4.3.4 including the patch above. I will need to wait
for a couple days to see if we can get the system to generate any core dumps with the new
patch applied.
I'll update the issue if/when there is a dump generated.
Thanks for looking into this.
---
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/440#issuecomment-167581996
7 Jan
7 Jan
10:12 a.m.
So far no core files have been generated on the registrar's since the patch. I think
we can safely close this issue now.
Thanks for taking the time to look into it, much appreciated.
---
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/440#issuecomment-169617890
10:12 a.m.
Closed #440.
---
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/440#event-508074090
3050
days inactive
3074
days old
14 comments
2 participants
participants (2)
-
00Asgaroth00 -
Daniel-Constantin Mierla