Hello,
I got several errors this morning on a Kamailio instance.
The first occurred without a complete crash of Kamailio but 4 coredumps have been created:
BFD: Warning: /core.3818 is truncated: expected core file size >= 342618112, found: 61579264.
[New Thread 3818]
Cannot access memory at address 0x7f66e0557658
BFD: Warning: /core.3821 is truncated: expected core file size >= 343044096, found: 21229568.
[New Thread 3821]
Cannot access memory at address 0x7f66e0557658
BFD: Warning: /core.3822 is truncated: expected core file size >= 342675456, found: 120066048.
[New Thread 3822]
Cannot access memory at address 0x7f66e0557658
BFD: Warning: /core.3839 is truncated: expected core file size >= 342401024, found: 2691072.
[New Thread 3839]
Cannot access memory at address 0x3d6f021168
Cannot access memory at address 0x3d6f021168
Cannot access memory at address 0x3d6f021168
It seems that Kamailio auto recovered without fully crashed during this issue. But the Kamailio FIFO was deleted and kamctl didn't worked anymore. I had to properly restart.
I have seen in the logs:
Jan 21 10:21:11 /usr/local/sbin/kamailio[3843]: ERROR: <core> [dtrie.c:153]: dtrie_insert(): could not allocate shared memory from available pool
Jan 21 10:21:11 /usr/local/sbin/kamailio[3839]: ERROR: <core> [dtrie.c:144]: dtrie_insert(): could not allocate shared memory from available pool
Jan 21 10:21:11 /usr/local/sbin/kamailio[3839]: ERROR: userblacklist [db.c:91]: db_build_userbl_tree(): could not insert values into trie.
Jan 21 10:21:11 /usr/local/sbin/kamailio[3843]: ERROR: userblacklist [db.c:91]: db_build_userbl_tree(): could not insert values into trie.
Jan 21 10:21:11 /usr/local/sbin/kamailio[3831]: ERROR: <core> [dtrie.c:144]: dtrie_insert(): could not allocate shared memory from available pool
Jan 21 10:21:11 /usr/local/sbin/kamailio[3831]: ERROR: userblacklist [db.c:91]: db_build_userbl_tree(): could not insert values into trie.
Jan 21 10:21:11 /usr/local/sbin/kamailio[3841]: ERROR: <core> [dtrie.c:153]: dtrie_insert(): could not allocate shared memory from available pool
Jan 21 10:21:12 /usr/local/sbin/kamailio[3821]: ERROR: <core> [dtrie.c:153]: dtrie_insert(): could not allocate shared memory from available pool
Jan 21 10:21:12 /usr/local/sbin/kamailio[3821]: ERROR: userblacklist [db.c:91]: db_build_userbl_tree(): could not insert values into trie.
Jan 21 10:21:12 /usr/local/sbin/kamailio[3828]: ERROR: <core> [dtrie.c:153]: dtrie_insert(): could not allocate shared memory from available pool
Jan 21 10:21:12 /usr/local/sbin/kamailio[3843]: ERROR: <core> [dtrie.c:153]: dtrie_insert(): could not allocate shared memory from available pool
Jan 21 10:21:12 /usr/local/sbin/kamailio[3818]: ERROR: <core> [dtrie.c:144]: dtrie_insert(): could not allocate shared memory from available pool
Jan 21 10:21:12 /usr/local/sbin/kamailio[3818]: ERROR: userblacklist [db.c:91]: db_build_userbl_tree(): could not insert values into trie.
[.]
Jan 21 10:21:13 kernel: kamailio[3818]: segfault at 40 ip 00007f66d8812af2 sp 00007fff09c83380 error 4 in libtrie.so.1.0[7f66d8810000+5000]
Jan 21 10:21:13 kernel: kamailio[3822]: segfault at 20 ip 00007f66d8812af2 sp 00007fff09c83380 error 4 in libtrie.so.1.0[7f66d8810000+5000]
Jan 21 10:21:14 kernel: kamailio[3821]: segfault at 0 ip 00007f66d8812af2 sp 00007fff09c83380 error 4 in libtrie.so.1.0[7f66d8810000+5000]
[.]
Jan 21 10:21:14 /usr/local/sbin/kamailio[3807]: ALERT: <core> [main.c:784]: handle_sigs(): child process 3828 exited by a signal 11
Jan 21 10:21:14 /usr/local/sbin/kamailio[3807]: ALERT: <core> [main.c:787]: handle_sigs(): core was not generated
Jan 21 10:21:14 /usr/local/sbin/kamailio[3807]: ALERT: <core> [main.c:784]: handle_sigs(): child process 3825 exited by a signal 11
Jan 21 10:21:14 /usr/local/sbin/kamailio[3807]: ALERT: <core> [main.c:787]: handle_sigs(): core was not generated
Jan 21 10:21:14 /usr/local/sbin/kamailio[3807]: INFO: <core> [main.c:799]: handle_sigs(): terminating due to SIGCHLD
Jan 21 10:21:14 /usr/local/sbin/kamailio[3832]: INFO: <core> [main.c:850]: sig_usr(): signal 15 received
Jan 21 10:21:14 /usr/local/sbin/kamailio[3841]: INFO: <core> [main.c:850]: sig_usr(): signal 15 received
Jan 21 10:21:14 /usr/local/sbin/kamailio[3876]: INFO: <core> [main.c:850]: sig_usr(): signal 15 received
1 hour later a new crash but here Kamailio has completely restarted with two coredump showing the same error than previous one.
And 2 coredumps readable:
Core was generated by `/usr/local/sbin/kamailio -P /var/run/kamailio.pid -m 256 -M 64'.
Program terminated with signal 11, Segmentation fault.
#0 0x00007f13a1550af2 in dtrie_insert (root=0x7f138d7ffb38, number=0x22ba6c0 "003716716", numberlen=9, data=0x2, branches=10) at dtrie.c:141
141 if (node->child[digit] == NULL) {
Core was generated by `/usr/local/sbin/kamailio -P /var/run/kamailio.pid -m 256 -M 64'.
Program terminated with signal 11, Segmentation fault.
#0 0x00007f13a1550af2 in dtrie_insert (root=0x7f138d7ff998, number=0x22b57a8 "0035587482", numberlen=10, data=0x2, branches=10) at dtrie.c:141
141 if (node->child[digit] == NULL) {
Kamailio is 4.2.6 on this instance.
Regards,
Igor.
On 21.01.2016 12:37, Igor Potjevlesch wrote:
I remember something related, solved and backported in commit [1].
Now that I'm looking at the segfault line I think node should be NULL checked also after ```struct dtrie_node_t *node = root;```
Stefan
[1] https://github.com/kamailio/kamailio/commit/b399bb34cebdb1002d27bb0cf554f2d2...
Hi Stefan,
Indeed, It looks to be the issue.
But, the root cause is the lack of SHM memory, isn't it?
Actually, Kamailio is running with 256M of SHM (-m 256). What would be the good new value?
Regards,
Igor.
De : sr-users [mailto:sr-users-bounces@lists.sip-router.org] De la part de smititelu Envoyé : jeudi 21 janvier 2016 12:22 À : Kamailio (SER) - Users Mailing List sr-users@lists.sip-router.org Objet : Re: [SR-Users] Memory issue?
On 21.01.2016 12:37, Igor Potjevlesch wrote:
And 2 coredumps readable:
Core was generated by `/usr/local/sbin/kamailio -P /var/run/kamailio.pid -m 256 -M 64'.
Program terminated with signal 11, Segmentation fault.
#0 0x00007f13a1550af2 in dtrie_insert (root=0x7f138d7ffb38, number=0x22ba6c0 "003716716", numberlen=9, data=0x2, branches=10) at dtrie.c:141
141 if (node->child[digit] == NULL) {
Core was generated by `/usr/local/sbin/kamailio -P /var/run/kamailio.pid -m 256 -M 64'.
Program terminated with signal 11, Segmentation fault.
#0 0x00007f13a1550af2 in dtrie_insert (root=0x7f138d7ff998, number=0x22b57a8 "0035587482", numberlen=10, data=0x2, branches=10) at dtrie.c:141
141 if (node->child[digit] == NULL) {
I remember something related, solved and backported in commit [1].
Now that I'm looking at the segfault line I think node should be NULL checked also after ```struct dtrie_node_t *node = root;```
Stefan
[1] https://github.com/kamailio/kamailio/commit/b399bb34cebdb1002d27bb0cf554f2d2 e683c40d
Hi Igor,
Yes, it's caused by lack of shm.
I don't know if there is a proper shm value. Depending on how many modules are loaded on kamailio and how much shm a module might use over time, more or less shm should be used. In your particular case, depending on how much userblacklist table should grow, you should adjust the startup shm value.
Thank you for your detailed feedback on the segfault; the fix it's on its way and will be backported.
Stefan
On Fri, Jan 22, 2016 at 1:12 PM, Igor Potjevlesch < igor.potjevlesch@gmail.com> wrote:
Hi Stefan,
The fix must be applied in addition of the growth of SHM?
What exactly the fix will avoid?
Regards,
Igor.
De : sr-users [mailto:sr-users-bounces@lists.sip-router.org] De la part de Mititelu Stefan Envoyé : vendredi 22 janvier 2016 15:26 À : Kamailio (SER) - Users Mailing List sr-users@lists.sip-router.org Objet : Re: [SR-Users] Memory issue?
Hi Igor,
Yes, it's caused by lack of shm.
I don't know if there is a proper shm value. Depending on how many modules are loaded on kamailio and how much shm a module might use over time, more or less shm should be used. In your particular case, depending on how much userblacklist table should grow, you should adjust the startup shm value.
Thank you for your detailed feedback on the segfault; the fix it's on its way and will be backported.
Stefan
On Fri, Jan 22, 2016 at 1:12 PM, Igor Potjevlesch <igor.potjevlesch@gmail.com mailto:igor.potjevlesch@gmail.com > wrote:
Hi Stefan,
Indeed, It looks to be the issue.
But, the root cause is the lack of SHM memory, isn't it?
Actually, Kamailio is running with 256M of SHM (-m 256). What would be the good new value?
Regards,
Igor.
De : sr-users [mailto:sr-users-bounces@lists.sip-router.org mailto:sr-users-bounces@lists.sip-router.org ] De la part de smititelu Envoyé : jeudi 21 janvier 2016 12:22 À : Kamailio (SER) - Users Mailing List <sr-users@lists.sip-router.org mailto:sr-users@lists.sip-router.org > Objet : Re: [SR-Users] Memory issue?
On 21.01.2016 12:37, Igor Potjevlesch wrote:
And 2 coredumps readable:
Core was generated by `/usr/local/sbin/kamailio -P /var/run/kamailio.pid -m 256 -M 64'.
Program terminated with signal 11, Segmentation fault.
#0 0x00007f13a1550af2 in dtrie_insert (root=0x7f138d7ffb38, number=0x22ba6c0 "003716716", numberlen=9, data=0x2, branches=10) at dtrie.c:141
141 if (node->child[digit] == NULL) {
Core was generated by `/usr/local/sbin/kamailio -P /var/run/kamailio.pid -m 256 -M 64'.
Program terminated with signal 11, Segmentation fault.
#0 0x00007f13a1550af2 in dtrie_insert (root=0x7f138d7ff998, number=0x22b57a8 "0035587482", numberlen=10, data=0x2, branches=10) at dtrie.c:141
141 if (node->child[digit] == NULL) {
I remember something related, solved and backported in commit [1].
Now that I'm looking at the segfault line I think node should be NULL checked also after ```struct dtrie_node_t *node = root;```
Stefan
[1] https://github.com/kamailio/kamailio/commit/b399bb34cebdb1002d27bb0cf554f2d2...
_______________________________________________ SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org mailto:sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
-
Igor Potjevlesch -
Mititelu Stefan -
smititelu