[SR-Users] Kamailio not responding after stress test

[Ihor Olkhovskyi]

Daniel,

Just openssl compiled with ./config -d option and statically linked to 
tlsa module. This module is not installed system-wide, only used for 
Kamailio.

libpthread, I believe, is taken from the system.

Le 02/12/2022 à 11:15, Daniel-Constantin Mierla a écrit :
>
> Still now showing the debug symbols for libssl/libcrypto:
>
> 1.
>     0x00007fdc7d07754d in __lll_lock_wait () from /lib64/libpthread.so.0
> 2.
>     #0 0x00007fdc7d07754d in __lll_lock_wait () from
>     /lib64/libpthread.so.0
> 3.
>     No symbol table info available.
> 4.
>     #1 0x00007fdc7d0743d2 in pthread_rwlock_wrlock () from
>     /lib64/libpthread.so.0
> 5.
>     No symbol table info available.
> 6.
>     #2 0x00007fdc737f2c59 in CRYPTO_THREAD_write_lock () from
>     /usr/local/lib64/kamailio/modules/tlsa.so
> 7.
>     No symbol table info available.
> 8.
>     #3 0x00007fdc7379e32e in CRYPTO_free_ex_data () from
>     /usr/local/lib64/kamailio/modules/tlsa.so
> 9.
>     No symbol table info available.
>10.
>     #4 0x00007fdc73708ae2 in SSL_free () from
>     /usr/local/lib64/kamailio/modules/tlsa.so
>11.
>     No symbol table info available.
>12.
>
> How are these libs installed?
>
> Cheers,
> Daniel
>
> On 29.11.22 10:38, Ihor Olkhovskyi wrote:
>>
>> Daniel,
>>
>> Thanks for the tip, recompiled and got a new trace
>>
>> https://pastebin.com/NxCXA1gT
>>
>> Not sure what happened, but sometimes running kamctl trap restores 
>> Kamailio functionality to accept connections.
>>
>> Thanks in advance!
>>
>> Le 29/11/2022 à 08:07, Daniel-Constantin Mierla a écrit :
>>>
>>> Hello,
>>>
>>> is libssl/libcrypto compiled/installed with debugging symbols? They 
>>> are not shown in the backtrace.
>>>
>>> Cheers,
>>> Daniel
>>>
>>> On 28.11.22 17:38, Igor Olhovskiy wrote:
>>>> Hello,
>>>>
>>>> I'm continuing investigations on Kamailio and stress test. Got it 
>>>> again in the state where it's not accepting any new TCP/TLS 
>>>> connections (UDP still works though), but all looks good from 
>>>> lsof/netnstat part, like system is not reporting any zombie 
>>>> connections. Restart of Kamailio process helps
>>>>
>>>> This time I got output of kamctl trap
>>>>
>>>> Put it here: https://pastebin.com/iYrNZ8U9
>>>>
>>>> kamailio --version
>>>> version: kamailio 5.6.2 (x86_64/linux) 54a9c1
>>>> flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, 
>>>> DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, 
>>>> Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, 
>>>> FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, 
>>>> USE_NAPTR, USE_DST_BLOCKLIST, HAVE_RESOLV_RES
>>>> ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144, MAX_URI_SIZE 
>>>> 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB
>>>> poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
>>>> id: 54a9c1
>>>> compiled on 14:01:01 Oct 18 2022 with gcc 4.8.5
>>>>
>>>> It's statically linked with tlsa pointing on openssl-1.1.1q
>>>>
>>>> Settings related to TLS are:
>>>>
>>>> fork=yes
>>>> children=4
>>>> tcp_children=12
>>>> enable_tls=yes
>>>>
>>>> tcp_accept_no_cl=yes
>>>> tcp_max_connections=63536
>>>> tls_max_connections=63536
>>>> tcp_accept_aliases=no
>>>> tcp_async=yes
>>>> tcp_connect_timeout=10
>>>> tcp_conn_wq_max=63536
>>>> tcp_crlf_ping=yes
>>>> tcp_delayed_ack=yes
>>>> tcp_fd_cache=yes
>>>> tcp_keepalive=yes
>>>> tcp_keepcnt=3
>>>> tcp_keepidle=30
>>>> tcp_keepintvl=10
>>>> tcp_linger2=30
>>>> tcp_rd_buf_size=80000
>>>> tcp_send_timeout=10
>>>> tcp_wq_blk_size=2100
>>>> tcp_wq_max=10485760
>>>> open_files_limit=63536
>>>>
>>>>
>>>> Can you please help to read gdb output and understand where I 
>>>> missed in config?
>>>>
>>>> Thanks in advance!
>>>>
>>>>
>>>> -- 
>>>> Best regards,
>>>> Ihor (Igor)
>>>>
>>>> __________________________________________________________
>>>> Kamailio - Users Mailing List - Non Commercial Discussions
>>>> sr-users at lists.kamailio.org
>>>> Important: keep the mailing list in the recipients, do not reply only to the sender!
>>>> Edit mailing list options or unsubscribe:
>>>> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
>>> -- 
>>> Daniel-Constantin Mierla --www.asipto.com
>>> www.twitter.com/miconda  --www.linkedin.com/in/miconda
> -- 
> Daniel-Constantin Mierla --www.asipto.com
> www.twitter.com/miconda  --www.linkedin.com/in/miconda
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.kamailio.org/pipermail/sr-users/attachments/20221208/5032055f/attachment.htm>