21 Jan
2016
21 Jan
'16
11:37 a.m.
Hello,
I got several errors this morning on a Kamailio instance.
The first occurred without a complete crash of Kamailio but 4 coredumps have
been created:
BFD: Warning: /core.3818 is truncated: expected core file size >= 342618112,
found: 61579264.
[New Thread 3818]
Cannot access memory at address 0x7f66e0557658
BFD: Warning: /core.3821 is truncated: expected core file size >= 343044096,
found: 21229568.
[New Thread 3821]
Cannot access memory at address 0x7f66e0557658
BFD: Warning: /core.3822 is truncated: expected core file size >= 342675456,
found: 120066048.
[New Thread 3822]
Cannot access memory at address 0x7f66e0557658
BFD: Warning: /core.3839 is truncated: expected core file size >= 342401024,
found: 2691072.
[New Thread 3839]
Cannot access memory at address 0x3d6f021168
Cannot access memory at address 0x3d6f021168
Cannot access memory at address 0x3d6f021168
It seems that Kamailio auto recovered without fully crashed during this
issue. But the Kamailio FIFO was deleted and kamctl didn't worked anymore. I
had to properly restart.
I have seen in the logs:
Jan 21 10:21:11 /usr/local/sbin/kamailio[3843]: ERROR: <core> [dtrie.c:153]:
dtrie_insert(): could not allocate shared memory from available pool
Jan 21 10:21:11 /usr/local/sbin/kamailio[3839]: ERROR: <core> [dtrie.c:144]:
dtrie_insert(): could not allocate shared memory from available pool
Jan 21 10:21:11 /usr/local/sbin/kamailio[3839]: ERROR: userblacklist
[db.c:91]: db_build_userbl_tree(): could not insert values into trie.
Jan 21 10:21:11 /usr/local/sbin/kamailio[3843]: ERROR: userblacklist
[db.c:91]: db_build_userbl_tree(): could not insert values into trie.
Jan 21 10:21:11 /usr/local/sbin/kamailio[3831]: ERROR: <core> [dtrie.c:144]:
dtrie_insert(): could not allocate shared memory from available pool
Jan 21 10:21:11 /usr/local/sbin/kamailio[3831]: ERROR: userblacklist
[db.c:91]: db_build_userbl_tree(): could not insert values into trie.
Jan 21 10:21:11 /usr/local/sbin/kamailio[3841]: ERROR: <core> [dtrie.c:153]:
dtrie_insert(): could not allocate shared memory from available pool
Jan 21 10:21:12 /usr/local/sbin/kamailio[3821]: ERROR: <core> [dtrie.c:153]:
dtrie_insert(): could not allocate shared memory from available pool
Jan 21 10:21:12 /usr/local/sbin/kamailio[3821]: ERROR: userblacklist
[db.c:91]: db_build_userbl_tree(): could not insert values into trie.
Jan 21 10:21:12 /usr/local/sbin/kamailio[3828]: ERROR: <core> [dtrie.c:153]:
dtrie_insert(): could not allocate shared memory from available pool
Jan 21 10:21:12 /usr/local/sbin/kamailio[3843]: ERROR: <core> [dtrie.c:153]:
dtrie_insert(): could not allocate shared memory from available pool
Jan 21 10:21:12 /usr/local/sbin/kamailio[3818]: ERROR: <core> [dtrie.c:144]:
dtrie_insert(): could not allocate shared memory from available pool
Jan 21 10:21:12 /usr/local/sbin/kamailio[3818]: ERROR: userblacklist
[db.c:91]: db_build_userbl_tree(): could not insert values into trie.
[.]
Jan 21 10:21:13 kernel: kamailio[3818]: segfault at 40 ip 00007f66d8812af2
sp 00007fff09c83380 error 4 in libtrie.so.1.0[7f66d8810000+5000]
Jan 21 10:21:13 kernel: kamailio[3822]: segfault at 20 ip 00007f66d8812af2
sp 00007fff09c83380 error 4 in libtrie.so.1.0[7f66d8810000+5000]
Jan 21 10:21:14 kernel: kamailio[3821]: segfault at 0 ip 00007f66d8812af2 sp
00007fff09c83380 error 4 in libtrie.so.1.0[7f66d8810000+5000]
[.]
Jan 21 10:21:14 /usr/local/sbin/kamailio[3807]: ALERT: <core> [main.c:784]:
handle_sigs(): child process 3828 exited by a signal 11
Jan 21 10:21:14 /usr/local/sbin/kamailio[3807]: ALERT: <core> [main.c:787]:
handle_sigs(): core was not generated
Jan 21 10:21:14 /usr/local/sbin/kamailio[3807]: ALERT: <core> [main.c:784]:
handle_sigs(): child process 3825 exited by a signal 11
Jan 21 10:21:14 /usr/local/sbin/kamailio[3807]: ALERT: <core> [main.c:787]:
handle_sigs(): core was not generated
Jan 21 10:21:14 /usr/local/sbin/kamailio[3807]: INFO: <core> [main.c:799]:
handle_sigs(): terminating due to SIGCHLD
Jan 21 10:21:14 /usr/local/sbin/kamailio[3832]: INFO: <core> [main.c:850]:
sig_usr(): signal 15 received
Jan 21 10:21:14 /usr/local/sbin/kamailio[3841]: INFO: <core> [main.c:850]:
sig_usr(): signal 15 received
Jan 21 10:21:14 /usr/local/sbin/kamailio[3876]: INFO: <core> [main.c:850]:
sig_usr(): signal 15 received
1 hour later a new crash but here Kamailio has completely restarted with two
coredump showing the same error than previous one.
And 2 coredumps readable:
Core was generated by `/usr/local/sbin/kamailio -P /var/run/kamailio.pid -m
256 -M 64'.
Program terminated with signal 11, Segmentation fault.
#0 0x00007f13a1550af2 in dtrie_insert (root=0x7f138d7ffb38,
number=0x22ba6c0 "003716716", numberlen=9, data=0x2, branches=10) at
dtrie.c:141
141 if (node->child[digit] == NULL) {
Core was generated by `/usr/local/sbin/kamailio -P /var/run/kamailio.pid -m
256 -M 64'.
Program terminated with signal 11, Segmentation fault.
#0 0x00007f13a1550af2 in dtrie_insert (root=0x7f138d7ff998,
number=0x22b57a8 "0035587482", numberlen=10, data=0x2, branches=10) at
dtrie.c:141
141 if (node->child[digit] == NULL) {
Kamailio is 4.2.6 on this instance.
Regards,
Igor.
21 Jan
21 Jan
12:22 p.m.
On 21.01.2016 12:37, Igor Potjevlesch wrote:
And 2 coredumps readable:
Core was generated by `/usr/local/sbin/kamailio -P
/var/run/kamailio.pid -m 256 -M 64'.
Program terminated with signal 11, Segmentation fault.
#0 0x00007f13a1550af2 in dtrie_insert (root=0x7f138d7ffb38,
number=0x22ba6c0 "003716716", numberlen=9, data=0x2, branches=10) at
dtrie.c:141
141 if (node->child[digit] == NULL) {
Core was generated by `/usr/local/sbin/kamailio -P
/var/run/kamailio.pid -m 256 -M 64'.
Program terminated with signal 11, Segmentation fault.
#0 0x00007f13a1550af2 in dtrie_insert (root=0x7f138d7ff998,
number=0x22b57a8 "0035587482", numberlen=10, data=0x2, branches=10) at
dtrie.c:141
141 if (node->child[digit] == NULL) {
I remember something related, solved and backported in commit [1].
Now that I'm looking at the segfault line I think node should be NULL
checked also after ```struct dtrie_node_t *node = root;```
Stefan
[1]
https://github.com/kamailio/kamailio/commit/b399bb34cebdb1002d27bb0cf554f2d…
22 Jan
22 Jan
12:12 p.m.
Hi Stefan,
Indeed, It looks to be the issue.
But, the root cause is the lack of SHM memory, isn't it?
Actually, Kamailio is running with 256M of SHM (-m 256). What would be the
good new value?
Regards,
Igor.
De : sr-users [mailto:sr-users-bounces@lists.sip-router.org] De la part de
smititelu
Envoyé : jeudi 21 janvier 2016 12:22
À : Kamailio (SER) - Users Mailing List <sr-users(a)lists.sip-router.org>
Objet : Re: [SR-Users] Memory issue?
On 21.01.2016 12:37, Igor Potjevlesch wrote:
And 2 coredumps readable:
Core was generated by `/usr/local/sbin/kamailio -P /var/run/kamailio.pid -m
256 -M 64'.
Program terminated with signal 11, Segmentation fault.
#0 0x00007f13a1550af2 in dtrie_insert (root=0x7f138d7ffb38,
number=0x22ba6c0 "003716716", numberlen=9, data=0x2, branches=10) at
dtrie.c:141
141 if (node->child[digit] == NULL) {
Core was generated by `/usr/local/sbin/kamailio -P /var/run/kamailio.pid -m
256 -M 64'.
Program terminated with signal 11, Segmentation fault.
#0 0x00007f13a1550af2 in dtrie_insert (root=0x7f138d7ff998,
number=0x22b57a8 "0035587482", numberlen=10, data=0x2, branches=10) at
dtrie.c:141
141 if (node->child[digit] == NULL) {
I remember something related, solved and backported in commit [1].
Now that I'm looking at the segfault line I think node should be NULL
checked also after ```struct dtrie_node_t *node = root;```
Stefan
[1]
https://github.com/kamailio/kamailio/commit/b399bb34cebdb1002d27bb0cf554f2d2
e683c40d
3:25 p.m.
Hi Igor,
Yes, it's caused by lack of shm.
I don't know if there is a proper shm value. Depending on how many modules
are loaded on kamailio and how much shm a module might use over time, more
or less shm should be used. In your particular case, depending on how much
userblacklist table should grow, you should adjust the startup shm value.
Thank you for your detailed feedback on the segfault; the fix it's on its
way and will be backported.
Stefan
On Fri, Jan 22, 2016 at 1:12 PM, Igor Potjevlesch <
igor.potjevlesch(a)gmail.com> wrote:
Hi Stefan,
Indeed, It looks to be the issue.
But, the root cause is the lack of SHM memory, isn't it?
Actually, Kamailio is running with 256M of SHM (-m 256). What would be the
good new value?
Regards,
Igor.
*De :* sr-users [mailto:sr-users-bounces@lists.sip-router.org] *De la
part de* smititelu
*Envoyé :* jeudi 21 janvier 2016 12:22
*À :* Kamailio (SER) - Users Mailing List <sr-users(a)lists.sip-router.org>
*Objet :* Re: [SR-Users] Memory issue?
On 21.01.2016 12:37, Igor Potjevlesch wrote:
And 2 coredumps readable:
Core was generated by `/usr/local/sbin/kamailio -P /var/run/kamailio.pid
-m 256 -M 64'.
Program terminated with signal 11, Segmentation fault.
#0 0x00007f13a1550af2 in dtrie_insert (root=0x7f138d7ffb38,
number=0x22ba6c0 "003716716", numberlen=9, data=0x2, branches=10) at
dtrie.c:141
141 if (node->child[digit] == NULL) {
Core was generated by `/usr/local/sbin/kamailio -P /var/run/kamailio.pid
-m 256 -M 64'.
Program terminated with signal 11, Segmentation fault.
#0 0x00007f13a1550af2 in dtrie_insert (root=0x7f138d7ff998,
number=0x22b57a8 "0035587482", numberlen=10, data=0x2, branches=10) at
dtrie.c:141
141 if (node->child[digit] == NULL) {
I remember something related, solved and backported in commit [1].
Now that I'm looking at the segfault line I think node should be NULL
checked also after ```struct dtrie_node_t *node = root;```
Stefan
[1]
https://github.com/kamailio/kamailio/commit/b399bb34cebdb1002d27bb0cf554f2d…
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users(a)lists.sip-router.org
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
29 Jan
29 Jan
5:31 p.m.
Hi Stefan,
The fix must be applied in addition of the growth of SHM?
What exactly the fix will avoid?
Regards,
Igor.
De : sr-users [mailto:sr-users-bounces@lists.sip-router.org] De la part de Mititelu
Stefan
Envoyé : vendredi 22 janvier 2016 15:26
À : Kamailio (SER) - Users Mailing List <sr-users(a)lists.sip-router.org>
Objet : Re: [SR-Users] Memory issue?
Hi Igor,
Yes, it's caused by lack of shm.
I don't know if there is a proper shm value. Depending on how many modules are loaded
on kamailio and how much shm a module might use over time, more or less shm should be
used. In your particular case, depending on how much userblacklist table should grow, you
should adjust the startup shm value.
Thank you for your detailed feedback on the segfault; the fix it's on its way and will
be backported.
Stefan
On Fri, Jan 22, 2016 at 1:12 PM, Igor Potjevlesch <igor.potjevlesch(a)gmail.com
<mailto:igor.potjevlesch@gmail.com> > wrote:
Hi Stefan,
Indeed, It looks to be the issue.
But, the root cause is the lack of SHM memory, isn't it?
Actually, Kamailio is running with 256M of SHM (-m 256). What would be the good new
value?
Regards,
Igor.
De : sr-users [mailto:sr-users-bounces@lists.sip-router.org
<mailto:sr-users-bounces@lists.sip-router.org> ] De la part de smititelu
Envoyé : jeudi 21 janvier 2016 12:22
À : Kamailio (SER) - Users Mailing List <sr-users(a)lists.sip-router.org
<mailto:sr-users@lists.sip-router.org> >
Objet : Re: [SR-Users] Memory issue?
On 21.01.2016 12:37, Igor Potjevlesch wrote:
And 2 coredumps readable:
Core was generated by `/usr/local/sbin/kamailio -P /var/run/kamailio.pid -m 256 -M
64'.
Program terminated with signal 11, Segmentation fault.
#0 0x00007f13a1550af2 in dtrie_insert (root=0x7f138d7ffb38, number=0x22ba6c0
"003716716", numberlen=9, data=0x2, branches=10) at dtrie.c:141
141 if (node->child[digit] == NULL) {
Core was generated by `/usr/local/sbin/kamailio -P /var/run/kamailio.pid -m 256 -M
64'.
Program terminated with signal 11, Segmentation fault.
#0 0x00007f13a1550af2 in dtrie_insert (root=0x7f138d7ff998, number=0x22b57a8
"0035587482", numberlen=10, data=0x2, branches=10) at dtrie.c:141
141 if (node->child[digit] == NULL) {
I remember something related, solved and backported in commit [1].
Now that I'm looking at the segfault line I think node should be NULL checked also
after ```struct dtrie_node_t *node = root;```
Stefan
[1] https://github.com/kamailio/kamailio/commit/b399bb34cebdb1002d27bb0cf554f2d…
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users(a)lists.sip-router.org <mailto:sr-users@lists.sip-router.org>
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
3233
days inactive
3241
days old
4 comments
3 participants
participants (3)
-
Igor Potjevlesch -
Mititelu Stefan -
smititelu