[SR-Users] Kamailio 5.4.3 Troubleshooting Traffic Blocking Issue After Docker and OpenSSL Update

27 Oct 2023

Hi All,

We have recently encountered a challenging issue in our environment that I
wanted to bring to your attention. A few months ago, we performed updates
to our Docker and OpenSSL installations, upgrading OpenSSL to version
1.1.1u. Since then, we have been experiencing intermittent traffic blocking
(TLS and UDP both for around 10 minutes and getting recovered) on some of
our instances.

Prior to the OpenSSL upgrade, we had not observed any such issues. However,
the situation has become more complex due to a substantial increase in
traffic, with each instance now handling 4K-5K concurrent calls and a daily
total of approximately 200,000 calls per instance.

Our initial suspicion was related to a Docker iptable issue. To investigate
further, we conducted a GDB trace using "kamctl pstrap" and discovered
specific log entries indicating potential issues within the
"pthread_rwlock_wrlock" function. This has prompted us to explore whether
the problems are linked to the OpenSSL upgrade.

We are considering upgrading Kamailio to the latest version, although this
process typically requires a significant amount of time for proper planning
and testing. Given the high volume of calls we are currently handling, we
are also exploring the possibility of a temporary solution involving a
rollback of the OpenSSL version to see if it resolves the issue.

Curiously, during a recent load test using SIPP, which simulated 10,000
concurrent calls with a rate of 100 calls per second, we did not encounter
the same problem.

any hints or insights you may have that could help us pinpoint the root
cause of the problem .

We do have a complete GDB trace available, but its size necessitates
sharing a truncated version for now.

version: kamailio 5.4.3 (x86_64/linux) 0178a5

flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE,
USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC,
TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT,
USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST,
HAVE_RESOLV_RES

ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144, MAX_URI_SIZE 1024,
BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB

poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.

id: 0178a5

compiled on 12:36:56 Oct 22 2023 with gcc 4.8.5

Operating System Host :
LSB Version: :core-4.1-amd64:core-4.1-noarch

Distributor ID: Rocky

Description: Rocky Linux release 8.8 (Green Obsidian)

Release: 8.8

Codename: GreenObsidian

container os (Centos7)

Kamctl pstraip output (truncated to last few lines)

---start 154 -----------------------------------------------------

kamailio: No such file or directory.

[Thread debugging using libthread_db enabled]

Using host libthread_db library "/lib64/libthread_db.so.1".

0x00007fbf6f38a0c3 in __epoll_wait_nocancel () from /lib64/libc.so.6

$1 = 82

$2 = {pid = 154, unix_sock = 182, idx = 27, status = 1, rank = 48, desc =
"tcp receiver (tcp:xxx.xxx.xxx.xx:5060 <http://223.62.234.57:5060/>)
child=27", '\000' <repeats 81 times>}

#0  0x00007fbf6f38a0c3 in __epoll_wait_nocancel () from /lib64/libc.so.6