### Description
Not sure, found 2 cores generated today, only suggesting it's DMQ related by what I see in the backtrace. They could have been generated during a restart of kamailio, not 100% sure though.
### Troubleshooting
TBD
#### Reproduction
TBD
#### Debugging Data
Attaching *bt full*, *info locals* and *list* from:
``` -rw------- 1 root kamailio 549019648 Jul 13 16:29 core.kamailio.22907.sbc01.1531517373 -rw------- 1 root kamailio 549019648 Jul 13 17:13 core.kamailio.777.sbc01.1531520031 ```
[core1__bt_full.txt](https://github.com/kamailio/kamailio/files/2194484/core1__bt_full.txt) [core1__info_locals.txt](https://github.com/kamailio/kamailio/files/2194485/core1__info_locals.txt) [core1__list.txt](https://github.com/kamailio/kamailio/files/2194486/core1__list.txt) [core2__bt_full.txt](https://github.com/kamailio/kamailio/files/2194487/core2__bt_full.txt) [core2__info_locals.txt](https://github.com/kamailio/kamailio/files/2194488/core2__info_locals.txt) [core2__list.txt](https://github.com/kamailio/kamailio/files/2194489/core2__list.txt)
### Additional Information
* **Kamailio Version** - output of `kamailio -v`
``` root@sbc01:~# kamailio -V version: kamailio 5.1.4 (x86_64/linux) flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: unknown compiled with gcc 6.3.0 root@sbc01:~# ```
* **Operating System**:
``` root@sbc01:~# lsb_release -a No LSB modules are available. Distributor ID: Debian Description: Debian GNU/Linux 9.4 (stretch) Release: 9.4 Codename: stretch root@sbc01:~#
root@sbc01:~# uname -a Linux sbc01 4.9.0-6-amd64 #1 SMP Debian 4.9.88-1+deb9u1 (2018-05-07) x86_64 GNU/Linux root@sbc01:~# ```
Maybe @charlesrchance can have a look as well.
@joelsdc - if you still have the core file used to get the content for core2_* files, can you grab the output for next gdb command in frame 0:
``` p *callid p *ftag p *ttag p *dir p d_table p *d_table ```
@miconda:
``` (gdb) frame 0 #0 0x00007f4e12ce6c68 in get_dlg (callid=0x7ffdc5993850, ftag=0x7ffdc5993860, ttag=0x7ffdc5993870, dir=0x7ffdc5993840) at dlg_hash.c:797 797 in dlg_hash.c (gdb) p *callid $1 = {s = 0x7f4dec572b2e "72a1671957944647-777@138.99.136.21\r\nContent-Length: 70\r\nMax-Forwards: 1\r\nContent-Type: text/plain\r\n\r\nsip:10.2.1.24:5080;status=active\r\nsip:10.2.1.23:5080;status=disabled\r\n", len = 34} (gdb) p *ftag $2 = { s = 0x7f4dec572aef "053226fbdabb3180cd6e80991bee5f34-940d\r\nCSeq: 10 KDMQ\r\nCall-ID: 72a1671957944647-777@138.99.136.21\r\nContent-Length: 70\r\nMax-Forwards: 1\r\nContent-Type: text/plain\r\n\r\nsip:10.2.1.24:5080;status=active\r\nsi"..., len = 37} (gdb) p *ttag $3 = {s = 0x0, len = 0} (gdb) p *dir $4 = 0 (gdb) p d_table $5 = (dlg_table_t *) 0x0 (gdb) p *d_table Cannot access memory at address 0x0 (gdb) ```
Hmm, dialog hash table not initialized or not available, get also:
``` p process_no p pt[process_no] ```
Hi Daniel:
``` (gdb) p process_no $1 = 0 (gdb) p pt[process_no] $2 = {pid = 777, unix_sock = -1, idx = -1, desc = "main process - attendant", '\000' <repeats 103 times>} (gdb) ```
Did you start kamailio with `fork=yes` in kamailio.cfg or have you provide any `-D` parameters to command line?
@miconda - apologies, somehow missed this - doesn't appear to be directly related to dmq though from what I can tell?
@charlesrchance - the core2 is generated when sending a KDMQ request, the core1 is generated during cleanup at shut down.
The first thing to sort out with core2 is why sending KDMQ it is done in main process -- either it is a kamailio with fork=no or some procedure done at startup. Sending happens when dialog hash table is still null.
I pushed a safety patch, but overall, dealing with sip traffic should be avoided in main process -- not sure if it is related to dmq in general or the dmq extension for dialog.
@miconda - from core2 it looks to me like kamailio was in the process of shutting down and the KDMQ message was to notify others of its new 'disabled' status (dmq.c:329).
This is clearly done in main process - is there an alternative? Does module load order have any impact on the order in which they're destroyed?
@charlesrchance - right, I checked the first core report and it was for shutdown, then for second I assumed is from another kamailio process of the same instance, but it looks like being from another instance
@joelsdc - given the above, did you get more core files from same instance, apparently the report for core1 and core2 are not from same instance, but from different once. If you got only one, maybe you have to enable once core file per pid, otherwise if there are more core files to be generated, they get overwritten and only one is at the end, but most relevant are those from non-main process.
@miconda: they are both (core1 and core2) from the same server, just later in time. So core1 was generated at say for example 1pm, then kamailio is restarted by watchdog, and at 3pm core2 is generated.
Regarding core-per-pid:
I have the following in `/etc/kamailio/kamailio.cfg`:
``` fork=yes ```
And in `/etc/sysctl.d/kamailio.conf`:
``` # Kamailio debugging kernel.core_pattern = /var/tmp/core.%e.%p.%h.%t kernel.core_uses_pid = 1 fs.suid_dumpable = 2 ```
Just to verify:
``` root@sbc01:~# cat /proc/sys/kernel/core_uses_pid 1 root@sbc01:~# ```
Am I missing anything to get the core-per-pid ??
If no other core files, then those were generated on stop/restart and should be fixed by commit 4f13c9a.
Hi Daniel, just checked and so far no more core files. Closing this, if I find new ones I'll open a new ticket. Thanks for the help!
Closed #1594.
-
Charles Chance -
Daniel-Constantin Mierla
-
Joel Serrano