[sr-dev] [kamailio] dmq_usrloc/usrloc core dumping (#440)

[00Asgaroth00]

I have 3 kamailio 434 registrars replicating between each other using dmq usrloc module

I have recently noticed that all of the registrars crash at random intervals over the course of the day, they all dont crash at the same time, but randomly, independantly of each other

I cannot see any unusual traffic arriving at the systems that may be a cause for the crash

I'd appreciate assistance trying to get to the bottom of the issue

Kamailio Version:
```
# /usr/sbin/kamailio -V
version: kamailio 434 (x86_64/linux) 0ec860
flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, DBG_QM_MALLOC, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select
id: 0ec860 
compiled on 17:42:12 Nov 25 2015 with gcc 447
```

Message in the log file when the kamailio application crashes:
```
Dec 14 01:46:54 /usr/sbin/kamailio[16205]: ALERT: <core> [mainc:728]: handle_sigs(): child process 16234 exited by a signal 11
Dec 14 01:46:54 /usr/sbin/kamailio[16205]: ALERT: <core> [mainc:731]: handle_sigs(): core was generated
Dec 14 01:46:54 /usr/sbin/kamailio[16205]: INFO: <core> [mainc:743]: handle_sigs(): terminating due to SIGCHLD
Dec 14 01:46:54 /usr/sbin/kamailio[16246]: INFO: <core> [mainc:798]: sig_usr(): signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16244]: INFO: <core> [mainc:798]: sig_usr(): signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16232]: INFO: <core> [mainc:798]: sig_usr(): signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16243]: INFO: <core> [mainc:798]: sig_usr(): signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16242]: INFO: <core> [mainc:798]: sig_usr(): signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16228]: INFO: <core> [mainc:798]: sig_usr(): signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16245]: INFO: <core> [mainc:798]: sig_usr(): signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16241]: INFO: <core> [mainc:798]: sig_usr(): signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16235]: INFO: <core> [mainc:798]: sig_usr(): signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16240]: INFO: <core> [mainc:798]: sig_usr(): signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16225]: INFO: <core> [mainc:798]: sig_usr(): signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16236]: INFO: <core> [mainc:798]: sig_usr(): signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16233]: INFO: <core> [mainc:798]: sig_usr(): signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16220]: INFO: <core> [mainc:798]: sig_usr(): signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16221]: INFO: <core> [mainc:798]: sig_usr(): signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16230]: INFO: <core> [mainc:798]: sig_usr(): signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16237]: INFO: <core> [mainc:798]: sig_usr(): signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16231]: INFO: <core> [mainc:798]: sig_usr(): signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16217]: INFO: <core> [mainc:798]: sig_usr(): signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16229]: INFO: <core> [mainc:798]: sig_usr(): signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16218]: INFO: <core> [mainc:798]: sig_usr(): signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16219]: INFO: <core> [mainc:798]: sig_usr(): signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16226]: INFO: <core> [mainc:798]: sig_usr(): signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16215]: INFO: <core> [mainc:798]: sig_usr(): signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16216]: INFO: <core> [mainc:798]: sig_usr(): signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16239]: INFO: <core> [mainc:798]: sig_usr(): signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16212]: INFO: <core> [mainc:798]: sig_usr(): signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16223]: INFO: <core> [mainc:798]: sig_usr(): signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16214]: INFO: <core> [mainc:798]: sig_usr(): signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16213]: INFO: <core> [mainc:798]: sig_usr(): signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16208]: INFO: <core> [mainc:798]: sig_usr(): signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16207]: INFO: <core> [mainc:798]: sig_usr(): signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16211]: INFO: <core> [mainc:798]: sig_usr(): signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16238]: INFO: <core> [mainc:798]: sig_usr(): signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16222]: INFO: <core> [mainc:798]: sig_usr(): signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16210]: INFO: <core> [mainc:798]: sig_usr(): signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16227]: INFO: <core> [mainc:798]: sig_usr(): signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16224]: INFO: <core> [mainc:798]: sig_usr(): signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16206]: INFO: <core> [mainc:798]: sig_usr(): signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16209]: INFO: <core> [mainc:798]: sig_usr(): signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16205]: WARNING: tm [h_tablec:130]: free_cell_helper(): removed cell 0x7f5a40a084f8 is still linked in hash table (h_tablec:440)
Dec 14 01:46:54 /usr/sbin/kamailio[16205]: WARNING: tm [h_tablec:130]: free_cell_helper(): removed cell 0x7f5a40a12390 is still linked in hash table (h_tablec:440)
```

Core file synopsis is:
```
Core was generated by `/usr/sbin/kamailio -P /var/run/kamailiopid -m 1024 -M 64 -u root -g root'
Program terminated with signal 11, Segmentation fault
#0  0x00007f5a82f68985 in mem_insert_ucontact (_r=0x7f5a409ec948, _c=0x7ffcc26f3140, _ci=0x7f5a81418500) at urecordc:152
152             if_update_stat( _r->slot, _r->slot->d->contacts, 1);
```

I still have the core file, however, it contains some sensitive info, so I can send backtraces directly to you if required

Please let me know if you require any additional info

Thanks


---
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/440
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.sip-router.org/pipermail/sr-dev/attachments/20151214/66f05869/attachment.html>