Hello,
I'm continuing investigations on Kamailio and stress test. Got it again in the state where it's not accepting any new TCP/TLS connections (UDP still works though), but all looks good from lsof/netnstat part, like system is not reporting any zombie connections. Restart of Kamailio process helps
This time I got output of kamctl trap
Put it here: https://pastebin.com/iYrNZ8U9
kamailio --version version: kamailio 5.6.2 (x86_64/linux) 54a9c1 flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLOCKLIST, HAVE_RESOLV_RES ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: 54a9c1 compiled on 14:01:01 Oct 18 2022 with gcc 4.8.5
It's statically linked with tlsa pointing on openssl-1.1.1q
Settings related to TLS are:
fork=yes children=4 tcp_children=12 enable_tls=yes
tcp_accept_no_cl=yes tcp_max_connections=63536 tls_max_connections=63536 tcp_accept_aliases=no tcp_async=yes tcp_connect_timeout=10 tcp_conn_wq_max=63536 tcp_crlf_ping=yes tcp_delayed_ack=yes tcp_fd_cache=yes tcp_keepalive=yes tcp_keepcnt=3 tcp_keepidle=30 tcp_keepintvl=10 tcp_linger2=30 tcp_rd_buf_size=80000 tcp_send_timeout=10 tcp_wq_blk_size=2100 tcp_wq_max=10485760 open_files_limit=63536
Can you please help to read gdb output and understand where I missed in config?
Thanks in advance!
Hello,
is libssl/libcrypto compiled/installed with debugging symbols? They are not shown in the backtrace.
Cheers, Daniel
On 28.11.22 17:38, Igor Olhovskiy wrote:
Hello,
I'm continuing investigations on Kamailio and stress test. Got it again in the state where it's not accepting any new TCP/TLS connections (UDP still works though), but all looks good from lsof/netnstat part, like system is not reporting any zombie connections. Restart of Kamailio process helps
This time I got output of kamctl trap
Put it here: https://pastebin.com/iYrNZ8U9
kamailio --version version: kamailio 5.6.2 (x86_64/linux) 54a9c1 flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLOCKLIST, HAVE_RESOLV_RES ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: 54a9c1 compiled on 14:01:01 Oct 18 2022 with gcc 4.8.5
It's statically linked with tlsa pointing on openssl-1.1.1q
Settings related to TLS are:
fork=yes children=4 tcp_children=12 enable_tls=yes
tcp_accept_no_cl=yes tcp_max_connections=63536 tls_max_connections=63536 tcp_accept_aliases=no tcp_async=yes tcp_connect_timeout=10 tcp_conn_wq_max=63536 tcp_crlf_ping=yes tcp_delayed_ack=yes tcp_fd_cache=yes tcp_keepalive=yes tcp_keepcnt=3 tcp_keepidle=30 tcp_keepintvl=10 tcp_linger2=30 tcp_rd_buf_size=80000 tcp_send_timeout=10 tcp_wq_blk_size=2100 tcp_wq_max=10485760 open_files_limit=63536
Can you please help to read gdb output and understand where I missed in config?
Thanks in advance!
-- Best regards, Ihor (Igor)
Kamailio - Users Mailing List - Non Commercial Discussions sr-users@lists.kamailio.org Important: keep the mailing list in the recipients, do not reply only to the sender! Edit mailing list options or unsubscribe: https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Daniel,
Thanks for the tip, recompiled and got a new trace
Not sure what happened, but sometimes running kamctl trap restores Kamailio functionality to accept connections.
Thanks in advance!
Le 29/11/2022 à 08:07, Daniel-Constantin Mierla a écrit :
Hello,
is libssl/libcrypto compiled/installed with debugging symbols? They are not shown in the backtrace.
Cheers, Daniel
On 28.11.22 17:38, Igor Olhovskiy wrote:
Hello,
I'm continuing investigations on Kamailio and stress test. Got it again in the state where it's not accepting any new TCP/TLS connections (UDP still works though), but all looks good from lsof/netnstat part, like system is not reporting any zombie connections. Restart of Kamailio process helps
This time I got output of kamctl trap
Put it here: https://pastebin.com/iYrNZ8U9
kamailio --version version: kamailio 5.6.2 (x86_64/linux) 54a9c1 flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLOCKLIST, HAVE_RESOLV_RES ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: 54a9c1 compiled on 14:01:01 Oct 18 2022 with gcc 4.8.5
It's statically linked with tlsa pointing on openssl-1.1.1q
Settings related to TLS are:
fork=yes children=4 tcp_children=12 enable_tls=yes
tcp_accept_no_cl=yes tcp_max_connections=63536 tls_max_connections=63536 tcp_accept_aliases=no tcp_async=yes tcp_connect_timeout=10 tcp_conn_wq_max=63536 tcp_crlf_ping=yes tcp_delayed_ack=yes tcp_fd_cache=yes tcp_keepalive=yes tcp_keepcnt=3 tcp_keepidle=30 tcp_keepintvl=10 tcp_linger2=30 tcp_rd_buf_size=80000 tcp_send_timeout=10 tcp_wq_blk_size=2100 tcp_wq_max=10485760 open_files_limit=63536
Can you please help to read gdb output and understand where I missed in config?
Thanks in advance!
-- Best regards, Ihor (Igor)
Kamailio - Users Mailing List - Non Commercial Discussions sr-users@lists.kamailio.org Important: keep the mailing list in the recipients, do not reply only to the sender! Edit mailing list options or unsubscribe: https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
-- Daniel-Constantin Mierla --www.asipto.com www.twitter.com/miconda --www.linkedin.com/in/miconda
Still now showing the debug symbols for libssl/libcrypto:
1. 0x00007fdc7d07754d in __lll_lock_wait () from /lib64/libpthread.so.0 2. #0 0x00007fdc7d07754d in __lll_lock_wait () from /lib64/libpthread.so.0 3. No symbol table info available. 4. #1 0x00007fdc7d0743d2 in pthread_rwlock_wrlock () from /lib64/libpthread.so.0 5. No symbol table info available. 6. #2 0x00007fdc737f2c59 in CRYPTO_THREAD_write_lock () from /usr/local/lib64/kamailio/modules/tlsa.so 7. No symbol table info available. 8. #3 0x00007fdc7379e32e in CRYPTO_free_ex_data () from /usr/local/lib64/kamailio/modules/tlsa.so 9. No symbol table info available. 10. #4 0x00007fdc73708ae2 in SSL_free () from /usr/local/lib64/kamailio/modules/tlsa.so 11. No symbol table info available. 12.
How are these libs installed?
Cheers, Daniel
On 29.11.22 10:38, Ihor Olkhovskyi wrote:
Daniel,
Thanks for the tip, recompiled and got a new trace
Not sure what happened, but sometimes running kamctl trap restores Kamailio functionality to accept connections.
Thanks in advance!
Le 29/11/2022 à 08:07, Daniel-Constantin Mierla a écrit :
Hello,
is libssl/libcrypto compiled/installed with debugging symbols? They are not shown in the backtrace.
Cheers, Daniel
On 28.11.22 17:38, Igor Olhovskiy wrote:
Hello,
I'm continuing investigations on Kamailio and stress test. Got it again in the state where it's not accepting any new TCP/TLS connections (UDP still works though), but all looks good from lsof/netnstat part, like system is not reporting any zombie connections. Restart of Kamailio process helps
This time I got output of kamctl trap
Put it here: https://pastebin.com/iYrNZ8U9
kamailio --version version: kamailio 5.6.2 (x86_64/linux) 54a9c1 flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLOCKLIST, HAVE_RESOLV_RES ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: 54a9c1 compiled on 14:01:01 Oct 18 2022 with gcc 4.8.5
It's statically linked with tlsa pointing on openssl-1.1.1q
Settings related to TLS are:
fork=yes children=4 tcp_children=12 enable_tls=yes
tcp_accept_no_cl=yes tcp_max_connections=63536 tls_max_connections=63536 tcp_accept_aliases=no tcp_async=yes tcp_connect_timeout=10 tcp_conn_wq_max=63536 tcp_crlf_ping=yes tcp_delayed_ack=yes tcp_fd_cache=yes tcp_keepalive=yes tcp_keepcnt=3 tcp_keepidle=30 tcp_keepintvl=10 tcp_linger2=30 tcp_rd_buf_size=80000 tcp_send_timeout=10 tcp_wq_blk_size=2100 tcp_wq_max=10485760 open_files_limit=63536
Can you please help to read gdb output and understand where I missed in config?
Thanks in advance!
-- Best regards, Ihor (Igor)
Kamailio - Users Mailing List - Non Commercial Discussions sr-users@lists.kamailio.org Important: keep the mailing list in the recipients, do not reply only to the sender! Edit mailing list options or unsubscribe: https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
-- Daniel-Constantin Mierla -- www.asipto.com www.twitter.com/miconda -- www.linkedin.com/in/miconda
Daniel,
Just openssl compiled with ./config -d option and statically linked to tlsa module. This module is not installed system-wide, only used for Kamailio.
libpthread, I believe, is taken from the system.
Le 02/12/2022 à 11:15, Daniel-Constantin Mierla a écrit :
Still now showing the debug symbols for libssl/libcrypto:
- 0x00007fdc7d07754d in __lll_lock_wait () from /lib64/libpthread.so.0
- #0 0x00007fdc7d07754d in __lll_lock_wait () from /lib64/libpthread.so.0
- No symbol table info available.
- #1 0x00007fdc7d0743d2 in pthread_rwlock_wrlock () from /lib64/libpthread.so.0
- No symbol table info available.
- #2 0x00007fdc737f2c59 in CRYPTO_THREAD_write_lock () from /usr/local/lib64/kamailio/modules/tlsa.so
- No symbol table info available.
- #3 0x00007fdc7379e32e in CRYPTO_free_ex_data () from /usr/local/lib64/kamailio/modules/tlsa.so
- No symbol table info available.
- #4 0x00007fdc73708ae2 in SSL_free () from /usr/local/lib64/kamailio/modules/tlsa.so
- No symbol table info available.
How are these libs installed?
Cheers, Daniel
On 29.11.22 10:38, Ihor Olkhovskyi wrote:
Daniel,
Thanks for the tip, recompiled and got a new trace
Not sure what happened, but sometimes running kamctl trap restores Kamailio functionality to accept connections.
Thanks in advance!
Le 29/11/2022 à 08:07, Daniel-Constantin Mierla a écrit :
Hello,
is libssl/libcrypto compiled/installed with debugging symbols? They are not shown in the backtrace.
Cheers, Daniel
On 28.11.22 17:38, Igor Olhovskiy wrote:
Hello,
I'm continuing investigations on Kamailio and stress test. Got it again in the state where it's not accepting any new TCP/TLS connections (UDP still works though), but all looks good from lsof/netnstat part, like system is not reporting any zombie connections. Restart of Kamailio process helps
This time I got output of kamctl trap
Put it here: https://pastebin.com/iYrNZ8U9
kamailio --version version: kamailio 5.6.2 (x86_64/linux) 54a9c1 flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLOCKLIST, HAVE_RESOLV_RES ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: 54a9c1 compiled on 14:01:01 Oct 18 2022 with gcc 4.8.5
It's statically linked with tlsa pointing on openssl-1.1.1q
Settings related to TLS are:
fork=yes children=4 tcp_children=12 enable_tls=yes
tcp_accept_no_cl=yes tcp_max_connections=63536 tls_max_connections=63536 tcp_accept_aliases=no tcp_async=yes tcp_connect_timeout=10 tcp_conn_wq_max=63536 tcp_crlf_ping=yes tcp_delayed_ack=yes tcp_fd_cache=yes tcp_keepalive=yes tcp_keepcnt=3 tcp_keepidle=30 tcp_keepintvl=10 tcp_linger2=30 tcp_rd_buf_size=80000 tcp_send_timeout=10 tcp_wq_blk_size=2100 tcp_wq_max=10485760 open_files_limit=63536
Can you please help to read gdb output and understand where I missed in config?
Thanks in advance!
-- Best regards, Ihor (Igor)
Kamailio - Users Mailing List - Non Commercial Discussions sr-users@lists.kamailio.org Important: keep the mailing list in the recipients, do not reply only to the sender! Edit mailing list options or unsubscribe: https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
-- Daniel-Constantin Mierla --www.asipto.com www.twitter.com/miconda --www.linkedin.com/in/miconda
-- Daniel-Constantin Mierla --www.asipto.com www.twitter.com/miconda --www.linkedin.com/in/miconda
Maybe you can try with git master branch and setting tls moduparam lock_mode to 1 -- let's see if there is any change in behaviour.
Cheers, Daniel
On 08.12.22 10:46, Ihor Olkhovskyi wrote:
Daniel,
Just openssl compiled with ./config -d option and statically linked to tlsa module. This module is not installed system-wide, only used for Kamailio.
libpthread, I believe, is taken from the system.
Le 02/12/2022 à 11:15, Daniel-Constantin Mierla a écrit :
Still now showing the debug symbols for libssl/libcrypto:
- 0x00007fdc7d07754d in __lll_lock_wait () from /lib64/libpthread.so.0
- #0 0x00007fdc7d07754d in __lll_lock_wait () from /lib64/libpthread.so.0
- No symbol table info available.
- #1 0x00007fdc7d0743d2 in pthread_rwlock_wrlock () from /lib64/libpthread.so.0
- No symbol table info available.
- #2 0x00007fdc737f2c59 in CRYPTO_THREAD_write_lock () from /usr/local/lib64/kamailio/modules/tlsa.so
- No symbol table info available.
- #3 0x00007fdc7379e32e in CRYPTO_free_ex_data () from /usr/local/lib64/kamailio/modules/tlsa.so
- No symbol table info available.
- #4 0x00007fdc73708ae2 in SSL_free () from /usr/local/lib64/kamailio/modules/tlsa.so
- No symbol table info available.
How are these libs installed?
Cheers, Daniel
On 29.11.22 10:38, Ihor Olkhovskyi wrote:
Daniel,
Thanks for the tip, recompiled and got a new trace
Not sure what happened, but sometimes running kamctl trap restores Kamailio functionality to accept connections.
Thanks in advance!
Le 29/11/2022 à 08:07, Daniel-Constantin Mierla a écrit :
Hello,
is libssl/libcrypto compiled/installed with debugging symbols? They are not shown in the backtrace.
Cheers, Daniel
On 28.11.22 17:38, Igor Olhovskiy wrote:
Hello,
I'm continuing investigations on Kamailio and stress test. Got it again in the state where it's not accepting any new TCP/TLS connections (UDP still works though), but all looks good from lsof/netnstat part, like system is not reporting any zombie connections. Restart of Kamailio process helps
This time I got output of kamctl trap
Put it here: https://pastebin.com/iYrNZ8U9
kamailio --version version: kamailio 5.6.2 (x86_64/linux) 54a9c1 flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLOCKLIST, HAVE_RESOLV_RES ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: 54a9c1 compiled on 14:01:01 Oct 18 2022 with gcc 4.8.5
It's statically linked with tlsa pointing on openssl-1.1.1q
Settings related to TLS are:
fork=yes children=4 tcp_children=12 enable_tls=yes
tcp_accept_no_cl=yes tcp_max_connections=63536 tls_max_connections=63536 tcp_accept_aliases=no tcp_async=yes tcp_connect_timeout=10 tcp_conn_wq_max=63536 tcp_crlf_ping=yes tcp_delayed_ack=yes tcp_fd_cache=yes tcp_keepalive=yes tcp_keepcnt=3 tcp_keepidle=30 tcp_keepintvl=10 tcp_linger2=30 tcp_rd_buf_size=80000 tcp_send_timeout=10 tcp_wq_blk_size=2100 tcp_wq_max=10485760 open_files_limit=63536
Can you please help to read gdb output and understand where I missed in config?
Thanks in advance!
-- Best regards, Ihor (Igor)
Kamailio - Users Mailing List - Non Commercial Discussions sr-users@lists.kamailio.org Important: keep the mailing list in the recipients, do not reply only to the sender! Edit mailing list options or unsubscribe: https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
-- Daniel-Constantin Mierla -- www.asipto.com www.twitter.com/miconda -- www.linkedin.com/in/miconda
-- Daniel-Constantin Mierla -- www.asipto.com www.twitter.com/miconda -- www.linkedin.com/in/miconda
Kamailio - Users Mailing List - Non Commercial Discussions sr-users@lists.kamailio.org Important: keep the mailing list in the recipients, do not reply only to the sender! Edit mailing list options or unsubscribe: https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Adding a note that the `lock_mode` parameter was renamed to `init_mode`, as I plan to use it for other purposes.
Cheers, Daniel
On 23.01.23 15:23, Daniel-Constantin Mierla wrote:
Maybe you can try with git master branch and setting tls moduparam lock_mode to 1 -- let's see if there is any change in behaviour.
Cheers, Daniel
On 08.12.22 10:46, Ihor Olkhovskyi wrote:
Daniel,
Just openssl compiled with ./config -d option and statically linked to tlsa module. This module is not installed system-wide, only used for Kamailio.
libpthread, I believe, is taken from the system.
Le 02/12/2022 à 11:15, Daniel-Constantin Mierla a écrit :
Still now showing the debug symbols for libssl/libcrypto:
- 0x00007fdc7d07754d in __lll_lock_wait () from /lib64/libpthread.so.0
- #0 0x00007fdc7d07754d in __lll_lock_wait () from /lib64/libpthread.so.0
- No symbol table info available.
- #1 0x00007fdc7d0743d2 in pthread_rwlock_wrlock () from /lib64/libpthread.so.0
- No symbol table info available.
- #2 0x00007fdc737f2c59 in CRYPTO_THREAD_write_lock () from /usr/local/lib64/kamailio/modules/tlsa.so
- No symbol table info available.
- #3 0x00007fdc7379e32e in CRYPTO_free_ex_data () from /usr/local/lib64/kamailio/modules/tlsa.so
- No symbol table info available.
- #4 0x00007fdc73708ae2 in SSL_free () from /usr/local/lib64/kamailio/modules/tlsa.so
- No symbol table info available.
How are these libs installed?
Cheers, Daniel
On 29.11.22 10:38, Ihor Olkhovskyi wrote:
Daniel,
Thanks for the tip, recompiled and got a new trace
Not sure what happened, but sometimes running kamctl trap restores Kamailio functionality to accept connections.
Thanks in advance!
Le 29/11/2022 à 08:07, Daniel-Constantin Mierla a écrit :
Hello,
is libssl/libcrypto compiled/installed with debugging symbols? They are not shown in the backtrace.
Cheers, Daniel
On 28.11.22 17:38, Igor Olhovskiy wrote:
Hello,
I'm continuing investigations on Kamailio and stress test. Got it again in the state where it's not accepting any new TCP/TLS connections (UDP still works though), but all looks good from lsof/netnstat part, like system is not reporting any zombie connections. Restart of Kamailio process helps
This time I got output of kamctl trap
Put it here: https://pastebin.com/iYrNZ8U9
kamailio --version version: kamailio 5.6.2 (x86_64/linux) 54a9c1 flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLOCKLIST, HAVE_RESOLV_RES ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: 54a9c1 compiled on 14:01:01 Oct 18 2022 with gcc 4.8.5
It's statically linked with tlsa pointing on openssl-1.1.1q
Settings related to TLS are:
fork=yes children=4 tcp_children=12 enable_tls=yes
tcp_accept_no_cl=yes tcp_max_connections=63536 tls_max_connections=63536 tcp_accept_aliases=no tcp_async=yes tcp_connect_timeout=10 tcp_conn_wq_max=63536 tcp_crlf_ping=yes tcp_delayed_ack=yes tcp_fd_cache=yes tcp_keepalive=yes tcp_keepcnt=3 tcp_keepidle=30 tcp_keepintvl=10 tcp_linger2=30 tcp_rd_buf_size=80000 tcp_send_timeout=10 tcp_wq_blk_size=2100 tcp_wq_max=10485760 open_files_limit=63536
Can you please help to read gdb output and understand where I missed in config?
Thanks in advance!
-- Best regards, Ihor (Igor)
Kamailio - Users Mailing List - Non Commercial Discussions sr-users@lists.kamailio.org Important: keep the mailing list in the recipients, do not reply only to the sender! Edit mailing list options or unsubscribe: https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
-- Daniel-Constantin Mierla -- www.asipto.com www.twitter.com/miconda -- www.linkedin.com/in/miconda
-- Daniel-Constantin Mierla -- www.asipto.com www.twitter.com/miconda -- www.linkedin.com/in/miconda
Kamailio - Users Mailing List - Non Commercial Discussions sr-users@lists.kamailio.org Important: keep the mailing list in the recipients, do not reply only to the sender! Edit mailing list options or unsubscribe: https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
-- Daniel-Constantin Mierla -- www.asipto.com www.twitter.com/miconda -- www.linkedin.com/in/miconda Kamailio World Conference - June 5-7, 2023 - www.kamailioworld.com
Daniel,
It does not change the situation, sorry (
Only thing I can confirm, that it's not listener-specific. Means if Kamailio stops answer on TLS on one interface, other interfaces over TLS are also affected (I have explicit listen=)
Le 24/01/2023 à 09:39, Daniel-Constantin Mierla a écrit :
Adding a note that the `lock_mode` parameter was renamed to `init_mode`, as I plan to use it for other purposes.
Cheers, Daniel
On 23.01.23 15:23, Daniel-Constantin Mierla wrote:
Maybe you can try with git master branch and setting tls moduparam lock_mode to 1 -- let's see if there is any change in behaviour.
Cheers, Daniel
On 08.12.22 10:46, Ihor Olkhovskyi wrote:
Daniel,
Just openssl compiled with ./config -d option and statically linked to tlsa module. This module is not installed system-wide, only used for Kamailio.
libpthread, I believe, is taken from the system.
Le 02/12/2022 à 11:15, Daniel-Constantin Mierla a écrit :
Still now showing the debug symbols for libssl/libcrypto:
- 0x00007fdc7d07754d in __lll_lock_wait () from /lib64/libpthread.so.0
- #0 0x00007fdc7d07754d in __lll_lock_wait () from /lib64/libpthread.so.0
- No symbol table info available.
- #1 0x00007fdc7d0743d2 in pthread_rwlock_wrlock () from /lib64/libpthread.so.0
- No symbol table info available.
- #2 0x00007fdc737f2c59 in CRYPTO_THREAD_write_lock () from /usr/local/lib64/kamailio/modules/tlsa.so
- No symbol table info available.
- #3 0x00007fdc7379e32e in CRYPTO_free_ex_data () from /usr/local/lib64/kamailio/modules/tlsa.so
- No symbol table info available.
- #4 0x00007fdc73708ae2 in SSL_free () from /usr/local/lib64/kamailio/modules/tlsa.so
- No symbol table info available.
How are these libs installed?
Cheers, Daniel
On 29.11.22 10:38, Ihor Olkhovskyi wrote:
Daniel,
Thanks for the tip, recompiled and got a new trace
Not sure what happened, but sometimes running kamctl trap restores Kamailio functionality to accept connections.
Thanks in advance!
Le 29/11/2022 à 08:07, Daniel-Constantin Mierla a écrit :
Hello,
is libssl/libcrypto compiled/installed with debugging symbols? They are not shown in the backtrace.
Cheers, Daniel
On 28.11.22 17:38, Igor Olhovskiy wrote: > Hello, > > I'm continuing investigations on Kamailio and stress test. Got > it again in the state where it's not accepting any new TCP/TLS > connections (UDP still works though), but all looks good from > lsof/netnstat part, like system is not reporting any zombie > connections. Restart of Kamailio process helps > > This time I got output of kamctl trap > > Put it here: https://pastebin.com/iYrNZ8U9 > > kamailio --version > version: kamailio 5.6.2 (x86_64/linux) 54a9c1 > flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, > DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, > Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, > FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, > USE_NAPTR, USE_DST_BLOCKLIST, HAVE_RESOLV_RES > ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144, > MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB > poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. > id: 54a9c1 > compiled on 14:01:01 Oct 18 2022 with gcc 4.8.5 > > It's statically linked with tlsa pointing on openssl-1.1.1q > > Settings related to TLS are: > > fork=yes > children=4 > tcp_children=12 > enable_tls=yes > > tcp_accept_no_cl=yes > tcp_max_connections=63536 > tls_max_connections=63536 > tcp_accept_aliases=no > tcp_async=yes > tcp_connect_timeout=10 > tcp_conn_wq_max=63536 > tcp_crlf_ping=yes > tcp_delayed_ack=yes > tcp_fd_cache=yes > tcp_keepalive=yes > tcp_keepcnt=3 > tcp_keepidle=30 > tcp_keepintvl=10 > tcp_linger2=30 > tcp_rd_buf_size=80000 > tcp_send_timeout=10 > tcp_wq_blk_size=2100 > tcp_wq_max=10485760 > open_files_limit=63536 > > > Can you please help to read gdb output and understand where I > missed in config? > > Thanks in advance! > > > -- > Best regards, > Ihor (Igor) > > __________________________________________________________ > Kamailio - Users Mailing List - Non Commercial Discussions > sr-users@lists.kamailio.org > Important: keep the mailing list in the recipients, do not reply only to the sender! > Edit mailing list options or unsubscribe:
> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Daniel-Constantin Mierla --www.asipto.com www.twitter.com/miconda --www.linkedin.com/in/miconda
-- Daniel-Constantin Mierla --www.asipto.com www.twitter.com/miconda --www.linkedin.com/in/miconda
Kamailio - Users Mailing List - Non Commercial Discussions sr-users@lists.kamailio.org Important: keep the mailing list in the recipients, do not reply only to the sender! Edit mailing list options or unsubscribe: https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
-- Daniel-Constantin Mierla --www.asipto.com www.twitter.com/miconda --www.linkedin.com/in/miconda Kamailio World Conference - June 5-7, 2023 -www.kamailioworld.com
-- Daniel-Constantin Mierla --www.asipto.com www.twitter.com/miconda --www.linkedin.com/in/miconda Kamailio World Conference - June 5-7, 2023 -www.kamailioworld.com
-
Daniel-Constantin Mierla -
Igor Olhovskiy -
Ihor Olkhovskyi