13 Jul
2018
13 Jul
'18
8:09 p.m.
### Description
Not sure, found 2 cores generated today, only suggesting it's DMQ related by what I
see in the backtrace. They could have been generated during a restart of kamailio, not
100% sure though.
### Troubleshooting
TBD
#### Reproduction
TBD
#### Debugging Data
Attaching *bt full*, *info locals* and *list* from:
```
-rw------- 1 root kamailio 549019648 Jul 13 16:29 core.kamailio.22907.sbc01.1531517373
-rw------- 1 root kamailio 549019648 Jul 13 17:13 core.kamailio.777.sbc01.1531520031
```
[core1__bt_full.txt](https://github.com/kamailio/kamailio/files/2194484/core…
[core1__info_locals.txt](https://github.com/kamailio/kamailio/files/2194485/…
[core1__list.txt](https://github.com/kamailio/kamailio/files/2194486/core1__…
[core2__bt_full.txt](https://github.com/kamailio/kamailio/files/2194487/core…
[core2__info_locals.txt](https://github.com/kamailio/kamailio/files/2194488/…
[core2__list.txt](https://github.com/kamailio/kamailio/files/2194489/core2__…
### Additional Information
* **Kamailio Version** - output of `kamailio -v`
```
root@sbc01:~# kamailio -V
version: kamailio 5.1.4 (x86_64/linux)
flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, DISABLE_NAGLE, USE_MCAST,
DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC,
DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER,
USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024,
BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
id: unknown
compiled with gcc 6.3.0
root@sbc01:~#
```
* **Operating System**:
```
root@sbc01:~# lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description: Debian GNU/Linux 9.4 (stretch)
Release: 9.4
Codename: stretch
root@sbc01:~#
root@sbc01:~# uname -a
Linux sbc01 4.9.0-6-amd64 #1 SMP Debian 4.9.88-1+deb9u1 (2018-05-07) x86_64 GNU/Linux
root@sbc01:~#
```
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1594
16 Jul
16 Jul
2:29 p.m.
Maybe @charlesrchance can have a look as well.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1594#issuecomment-405332918
18 Jul
18 Jul
11:21 a.m.
@joelsdc - if you still have the core file used to get the content for core2_* files, can
you grab the output for next gdb command in frame 0:
```
p *callid
p *ftag
p *ttag
p *dir
p d_table
p *d_table
```
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1594#issuecomment-405970034
12:42 p.m.
@miconda:
```
(gdb) frame 0
#0 0x00007f4e12ce6c68 in get_dlg (callid=0x7ffdc5993850, ftag=0x7ffdc5993860,
ttag=0x7ffdc5993870, dir=0x7ffdc5993840) at dlg_hash.c:797
797 in dlg_hash.c
(gdb) p *callid
$1 = {s = 0x7f4dec572b2e "72a1671957944647-777(a)138.99.136.21\r\nContent-Lengthength:
70\r\nMax-Forwards: 1\r\nContent-Type:
text/plain\r\n\r\nsip:10.2.1.24:5080;status=active\r\nsip:10.2.1.23:5080;status=disabled\r\n",
len = 34}
(gdb) p *ftag
$2 = {
s = 0x7f4dec572aef "053226fbdabb3180cd6e80991bee5f34-940d\r\nCSeq: 10
KDMQ\r\nCall-ID: 72a1671957944647-777(a)138.99.136.21\r\nContent-Length: 70\r\nMax-Forwards:
1\r\nContent-Type: text/plain\r\n\r\nsip:10.2.1.24:5080;status=active\r\nsi"..., len
= 37}
(gdb) p *ttag
$3 = {s = 0x0, len = 0}
(gdb) p *dir
$4 = 0
(gdb) p d_table
$5 = (dlg_table_t *) 0x0
(gdb) p *d_table
Cannot access memory at address 0x0
(gdb)
```
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1594#issuecomment-405997714
19 Jul
19 Jul
4:23 a.m.
Hmm, dialog hash table not initialized or not available, get also:
```
p process_no
p pt[process_no]
```
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1594#issuecomment-406196110
5:09 a.m.
Hi Daniel:
```
(gdb) p process_no
$1 = 0
(gdb) p pt[process_no]
$2 = {pid = 777, unix_sock = -1, idx = -1, desc = "main process - attendant",
'\000' <repeats 103 times>}
(gdb)
```
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1594#issuecomment-406209235
6:09 a.m.
Did you start kamailio with `fork=yes` in kamailio.cfg or have you provide any `-D`
parameters to command line?
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1594#issuecomment-406226025
7 a.m.
@miconda - apologies, somehow missed this - doesn't appear to be directly related to
dmq though from what I can tell?
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1594#issuecomment-406238320
7:50 a.m.
@charlesrchance - the core2 is generated when sending a KDMQ request, the core1 is
generated during cleanup at shut down.
The first thing to sort out with core2 is why sending KDMQ it is done in main process --
either it is a kamailio with fork=no or some procedure done at startup. Sending happens
when dialog hash table is still null.
I pushed a safety patch, but overall, dealing with sip traffic should be avoided in main
process -- not sure if it is related to dmq in general or the dmq extension for dialog.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1594#issuecomment-406249268
8:22 a.m.
@miconda - from core2 it looks to me like kamailio was in the process of shutting down and
the KDMQ message was to notify others of its new 'disabled' status (dmq.c:329).
This is clearly done in main process - is there an alternative? Does module load order
have any impact on the order in which they're destroyed?
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1594#issuecomment-406257601
10:22 a.m.
@charlesrchance - right, I checked the first core report and it was for shutdown, then for
second I assumed is from another kamailio process of the same instance, but it looks like
being from another instance
@joelsdc - given the above, did you get more core files from same instance, apparently the
report for core1 and core2 are not from same instance, but from different once. If you got
only one, maybe you have to enable once core file per pid, otherwise if there are more
core files to be generated, they get overwritten and only one is at the end, but most
relevant are those from non-main process.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1594#issuecomment-406293587
8:05 p.m.
@miconda: they are both (core1 and core2) from the same server, just later in time. So
core1 was generated at say for example 1pm, then kamailio is restarted by watchdog, and at
3pm core2 is generated.
Regarding core-per-pid:
I have the following in `/etc/kamailio/kamailio.cfg`:
```
fork=yes
```
And in `/etc/sysctl.d/kamailio.conf`:
```
# Kamailio debugging
kernel.core_pattern = /var/tmp/core.%e.%p.%h.%t
kernel.core_uses_pid = 1
fs.suid_dumpable = 2
```
Just to verify:
```
root@sbc01:~# cat /proc/sys/kernel/core_uses_pid
1
root@sbc01:~#
```
Am I missing anything to get the core-per-pid ??
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1594#issuecomment-406449916
24 Jul
24 Jul
5:55 p.m.
If no other core files, then those were generated on stop/restart and should be fixed by
commit 4f13c9a.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1594#issuecomment-407564828
25 Jul
25 Jul
10:26 a.m.
Hi Daniel, just checked and so far no more core files. Closing this, if I find new ones
I'll open a new ticket. Thanks for the help!
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1594#issuecomment-407773395
10:26 a.m.
Closed #1594.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1594#event-1752743784
2139
days inactive
2150
days old
14 comments
3 participants
participants (3)
-
Charles Chance -
Daniel-Constantin Mierla
-
Joel Serrano