[SR-Users] Memory issue?

Thu Jan 21 11:37:26 CET 2016

Hello,

I got several errors this morning on a Kamailio instance.

The first occurred without a complete crash of Kamailio but 4 coredumps have
been created:

BFD: Warning: /core.3818 is truncated: expected core file size >= 342618112,
found: 61579264.

[New Thread 3818]

Cannot access memory at address 0x7f66e0557658

BFD: Warning: /core.3821 is truncated: expected core file size >= 343044096,
found: 21229568.

[New Thread 3821]

Cannot access memory at address 0x7f66e0557658

BFD: Warning: /core.3822 is truncated: expected core file size >= 342675456,
found: 120066048.

[New Thread 3822]

Cannot access memory at address 0x7f66e0557658

BFD: Warning: /core.3839 is truncated: expected core file size >= 342401024,
found: 2691072.

[New Thread 3839]

Cannot access memory at address 0x3d6f021168

Cannot access memory at address 0x3d6f021168

Cannot access memory at address 0x3d6f021168

It seems that Kamailio auto recovered without fully crashed during this
issue. But the Kamailio FIFO was deleted and kamctl didn't worked anymore. I
had to properly restart.

I have seen in the logs:

Jan 21 10:21:11 /usr/local/sbin/kamailio[3843]: ERROR: <core> [dtrie.c:153]:
dtrie_insert(): could not allocate shared memory from available pool

Jan 21 10:21:11 /usr/local/sbin/kamailio[3839]: ERROR: <core> [dtrie.c:144]:
dtrie_insert(): could not allocate shared memory from available pool

Jan 21 10:21:11 /usr/local/sbin/kamailio[3839]: ERROR: userblacklist
[db.c:91]: db_build_userbl_tree(): could not insert values into trie.

Jan 21 10:21:11 /usr/local/sbin/kamailio[3843]: ERROR: userblacklist
[db.c:91]: db_build_userbl_tree(): could not insert values into trie.

Jan 21 10:21:11 /usr/local/sbin/kamailio[3831]: ERROR: <core> [dtrie.c:144]:
dtrie_insert(): could not allocate shared memory from available pool

Jan 21 10:21:11 /usr/local/sbin/kamailio[3831]: ERROR: userblacklist
[db.c:91]: db_build_userbl_tree(): could not insert values into trie.

Jan 21 10:21:11 /usr/local/sbin/kamailio[3841]: ERROR: <core> [dtrie.c:153]:
dtrie_insert(): could not allocate shared memory from available pool

Jan 21 10:21:12 /usr/local/sbin/kamailio[3821]: ERROR: <core> [dtrie.c:153]:
dtrie_insert(): could not allocate shared memory from available pool

Jan 21 10:21:12 /usr/local/sbin/kamailio[3821]: ERROR: userblacklist
[db.c:91]: db_build_userbl_tree(): could not insert values into trie.

Jan 21 10:21:12 /usr/local/sbin/kamailio[3828]: ERROR: <core> [dtrie.c:153]:
dtrie_insert(): could not allocate shared memory from available pool

Jan 21 10:21:12 /usr/local/sbin/kamailio[3843]: ERROR: <core> [dtrie.c:153]:
dtrie_insert(): could not allocate shared memory from available pool

Jan 21 10:21:12 /usr/local/sbin/kamailio[3818]: ERROR: <core> [dtrie.c:144]:
dtrie_insert(): could not allocate shared memory from available pool

Jan 21 10:21:12 /usr/local/sbin/kamailio[3818]: ERROR: userblacklist
[db.c:91]: db_build_userbl_tree(): could not insert values into trie.

[.]

Jan 21 10:21:13 kernel: kamailio[3818]: segfault at 40 ip 00007f66d8812af2
sp 00007fff09c83380 error 4 in libtrie.so.1.0[7f66d8810000+5000]

Jan 21 10:21:13 kernel: kamailio[3822]: segfault at 20 ip 00007f66d8812af2
sp 00007fff09c83380 error 4 in libtrie.so.1.0[7f66d8810000+5000]

Jan 21 10:21:14 kernel: kamailio[3821]: segfault at 0 ip 00007f66d8812af2 sp
00007fff09c83380 error 4 in libtrie.so.1.0[7f66d8810000+5000]

[.]

Jan 21 10:21:14 /usr/local/sbin/kamailio[3807]: ALERT: <core> [main.c:784]:
handle_sigs(): child process 3828 exited by a signal 11

Jan 21 10:21:14 /usr/local/sbin/kamailio[3807]: ALERT: <core> [main.c:787]:
handle_sigs(): core was not generated

Jan 21 10:21:14 /usr/local/sbin/kamailio[3807]: ALERT: <core> [main.c:784]:
handle_sigs(): child process 3825 exited by a signal 11

Jan 21 10:21:14 /usr/local/sbin/kamailio[3807]: ALERT: <core> [main.c:787]:
handle_sigs(): core was not generated

Jan 21 10:21:14 /usr/local/sbin/kamailio[3807]: INFO: <core> [main.c:799]:
handle_sigs(): terminating due to SIGCHLD

Jan 21 10:21:14 /usr/local/sbin/kamailio[3832]: INFO: <core> [main.c:850]:
sig_usr(): signal 15 received

Jan 21 10:21:14 /usr/local/sbin/kamailio[3841]: INFO: <core> [main.c:850]:
sig_usr(): signal 15 received

Jan 21 10:21:14 /usr/local/sbin/kamailio[3876]: INFO: <core> [main.c:850]:
sig_usr(): signal 15 received

1 hour later a new crash but here Kamailio has completely restarted with two
coredump showing the same error than previous one.

And 2 coredumps readable:

Core was generated by `/usr/local/sbin/kamailio -P /var/run/kamailio.pid -m
256 -M 64'.

Program terminated with signal 11, Segmentation fault.

#0  0x00007f13a1550af2 in dtrie_insert (root=0x7f138d7ffb38,
number=0x22ba6c0 "003716716", numberlen=9, data=0x2, branches=10) at
dtrie.c:141

141                     if (node->child[digit] == NULL) {

Core was generated by `/usr/local/sbin/kamailio -P /var/run/kamailio.pid -m
256 -M 64'.

Program terminated with signal 11, Segmentation fault.

#0  0x00007f13a1550af2 in dtrie_insert (root=0x7f138d7ff998,
number=0x22b57a8 "0035587482", numberlen=10, data=0x2, branches=10) at
dtrie.c:141

141                     if (node->child[digit] == NULL) {

Kamailio is 4.2.6 on this instance. 

Regards,

Igor.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.sip-router.org/pipermail/sr-users/attachments/20160121/cfe35e56/attachment.html>