17 May
2024
17 May
'24
7:37 p.m.
Hi there,
We are encountering consistent segfaults after rebooting our Kamailio instance with
incoming traffic, specifically when using Kamailio 5.7.4. We think this issue did not
occur with version 5.7.2, so it seems to have been introduced in either 5.7.3 or 5.7.4.
Due to team bandwidth constraints and the potential impact on production traffic, we
don't want to spend time on trying to reproduce the issue. So we have decided to
downgrade to 5.6.4, which we confirmed to be stable. (Probably 5.7.2 would be too - but we
didn't try).
Unfortunately, our logging was only set to WARNING level, and we did not capture a core
dump, so we cannot provide additional details beyond the following logs:
This was with tcp_reuse_ports=yes:
2024-05-17T15:42:55.582475541Z Listening on
2024-05-17T15:42:55.582512370Z [redacted]
2024-05-17T15:42:55.582538161Z tls: 10.X.X.X:5061 advertise Y.Y.Y:5061
2024-05-17T15:42:55.582543750Z Aliases:
2024-05-17T15:42:55.582549081Z tls: [redacted]:5061
2024-05-17T15:42:55.582574890Z
2024-05-17T15:42:55.587876630Z 0(1) WARNING: tls [tls_init.c:978]: tls_h_mod_init_f():
openssl bug #1491 (crash/mem leaks on low memory) workaround enabled (on low memory tls
operations will fail preemptively) with free memory thresholds 18874368 and 9437184 bytes
2024-05-17T15:42:55.703927049Z 35(41) CRITICAL: <core> [core/pass_fd.c:281]:
receive_fd(): EOF on 23
2024-05-17T15:42:55.703972029Z 0(1) ALERT: <core> [main.c:791]: handle_sigs():
child process 15 exited by a signal 11
2024-05-17T15:42:55.703978409Z 0(1) ALERT: <core> [main.c:795]: handle_sigs(): core
was generated
2024-05-17T15:42:55.705049839Z 35(41) CRITICAL: <core> [core/pass_fd.c:281]:
receive_fd(): EOF on 17
2024-05-17T15:42:55.705074209Z 35(41) CRITICAL: <core> [core/pass_fd.c:281]:
receive_fd(): EOF on 21
2024-05-17T15:42:55.705081209Z 35(41) CRITICAL: <core> [core/pass_fd.c:281]:
receive_fd(): EOF on 22
2024-05-17T15:42:55.705085879Z 35(41) CRITICAL: <core> [core/pass_fd.c:281]:
receive_fd(): EOF on 20
2024-05-17T15:42:55.705090319Z 35(41) CRITICAL: <core> [core/pass_fd.c:281]:
receive_fd(): EOF on 18
2024-05-17T15:42:55.705094649Z 35(41) CRITICAL: <core> [core/pass_fd.c:281]:
receive_fd(): EOF on 19
2024-05-17T15:42:55.705098879Z 35(41) CRITICAL: <core> [core/pass_fd.c:281]:
receive_fd(): EOF on 16
2024-05-17T15:42:55.705207399Z 35(41) CRITICAL: <core> [core/pass_fd.c:281]:
receive_fd(): EOF on 15
2024-05-17T15:42:55.705459439Z 35(41) CRITICAL: <core> [core/pass_fd.c:281]:
receive_fd(): EOF on 27
Without tcp_reuse_ports=yes, the segfault was always preceded by the following line if any
existing TLS connections were stuck in TIME_WAIT:
2024-05-16T19:18:51.654447639Z 9(14) WARNING: {1 1 INVITE XXX(a)0.0.0.0} <core>
[core/tcp_main.c:1301]: find_listening_sock_info(): binding to source address
10.X.X>X:5061 failed: Address already in use [98]
2024-05-16T19:18:51.746994728Z 0(1) ALERT: <core> [main.c:791]: handle_sigs():
child process 14 exited by a signal 11
When the server wasn't handling any traffic, the issue didn't occur even in
5.7.4.
Does anyone have any insights or suggestions on how to address this issue?
Kind regards
Stefan
17 May
17 May
7:59 p.m.
For additional context:
- Our Kamailio setup receives SIP messages from one endpoint over UDP and forwards them to
another endpoint over TLS, with rtpengine for RTP proxying.
- The issue only occurs on startup. E.g., after a VM reboot hosting our Kamailio
container, after deploying an updated Docker container onto the VM, or just after
restarting the docker container.
- Out of business hours, when the instance doesn't handle any traffic, we can't
reproduce the issue.
- We've been running 5.7.4 for about two months. We did the upgrade out of hours so
the initial upgrade didn't trigger the issue. The issue occurred during redeployments
yesterday and today, while the instance was handling traffic. We saw a few dozen segfaults
as we troubleshooted the issue during the incidents yesterday and today. After business
hours, we couldn't reproduce the issue.
- We've been doing regular kamailio upgrades to the latest 5.4.x-5.7.x versions and
this instance has been around for years without any similar issues or significant
configuration changes.
8:52 p.m.
Hello,
In the releases after 5.7.2 there have been a lot of TLS related changes. There were
necessary due to several critical memory corruption bugs due to implementation decisions
from the OpenSSL team in version 3.x. These changes were somewhat larger as usually
expected in minor releases, but ultimately necessary due to the mentioned problems. Due to
the complexity of the problems, several iterations were necessary to solve it completely.
So, without looking too much into the details of your issue - I suspect that the problems
you are observing might be caused from these changes. It might that you did a security
package update that changes some memory layout, for example, that triggered it.
I think that these TLS changes are now done in the releases 5.7.5 and 5.8.1, and these
releases should be stable again. This has been confirmed on multiples reports on our issue
tracker and also in some of our customer environments.
So, I would suggest you give the 5.7.5 a try. If there are still crashes on startup,
please let provide an update on the list or on the issue tracker.
Cheers,
Henning
--
Henning Westerholt - https://skalatan.de/blog/
Kamailio services - https://gilawa.com
-----Original Message-----
From: stefanr--- via sr-users <sr-users(a)lists.kamailio.org>
Sent: Freitag, 17. Mai 2024 19:00
To: sr-users(a)lists.kamailio.org
Cc: stefanr(a)wave.com
Subject: [SR-Users] Re: Segfault on startup when using TLS in kamailio 5.7.4
For additional context:
- Our Kamailio setup receives SIP messages from one endpoint over UDP and
forwards them to another endpoint over TLS, with rtpengine for RTP proxying.
- The issue only occurs on startup. E.g., after a VM reboot hosting our Kamailio
container, after deploying an updated Docker container onto the VM, or just after
restarting the docker container.
- Out of business hours, when the instance doesn't handle any traffic, we can't
reproduce the issue.
- We've been running 5.7.4 for about two months. We did the upgrade out of
hours so the initial upgrade didn't trigger the issue. The issue occurred during
redeployments yesterday and today, while the instance was handling traffic. We
saw a few dozen segfaults as we troubleshooted the issue during the incidents
yesterday and today. After business hours, we couldn't reproduce the issue.
- We've been doing regular kamailio upgrades to the latest 5.4.x-5.7.x versions
and this instance has been around for years without any similar issues or
significant configuration changes.
__________________________________________________________
Kamailio - Users Mailing List - Non Commercial Discussions To unsubscribe send
an email to sr-users-leave(a)lists.kamailio.org
Important: keep the mailing list in the recipients, do not reply only to the sender!
Edit mailing list options or unsubscribe:
195
days inactive
195
days old
2 comments
2 participants
participants (2)
-
Henning Westerholt -
stefanr＠wave.com