Hi all, I’ve been using Kamailio 4.4.4 for 8 months without any problems but in recent days, I’m having some issues with our server and, when it happens, it cannot get new registrations. Using top I saw a kamailio process is eating all CPU.
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 1556 kamailio 20 0 1010884 27624 24520 R 98.5 1.4 150:07.59 kamailio
I got backtrace attaching to Kamailio with gdb. Here is the backtrace:
#0 0x00007fb996992d6d in accept () at ../sysdeps/unix/syscall-template.S:81 #1 0x00000000005f3615 in handle_new_connect (si=0x7fb9960b4fa8) at tcp_main.c:3894 #2 0x00000000005f7029 in handle_io (fm=0x7fb9961da520, ev=1, idx=-1) at tcp_main.c:4199 #3 0x00000000005c68b2 in io_wait_loop_epoll (h=0xa1fdc0 <io_h>, t=5, repeat=0) at io_wait.h:1065 #4 0x00000000005fa04a in tcp_main_loop () at tcp_main.c:4490 #5 0x00000000004b48ba in main_loop () at main.c:1712 #6 0x00000000004bad12 in main (argc=13, argv=0x7ffd14247c18) at main.c:2631 (gdb) bt
At the moment I found the issue, there were 862 sip client registered correctly. Our server is configured to use only TLS and mysql as backend.
/usr/local/sbin/kamailio -v version: kamailio 4.4.4 (x86_64/linux) 852c47 flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: 852c47 compiled on 15:56:18 Dec 15 2016 with gcc 4.8.4
Can anyone address me to find out why this happens?
Many thanks, T.
Tomas Zanet R&D Software designer tzanet@came.com CAME S.p.A.
Hello,
can you monitor the traffic and see if you got exposed to someone stressing your server with opening new connections?
The accept() function is from libc, not something implemented in kamailio. So if the process is using a lot of CPU and it is stuck in accept(), then it is something related to the connection layer ...
Can you share the params for tls module? Another case that can be exploited in tls related attacks is cypher renegotiation, but that should be disabled by default in kamailio tls module.
Cheers, Daniel
On 15.12.17 15:11, Tomas Zanet wrote:
Hi all, I’ve been using Kamailio 4.4.4 for 8 months without any problems but in recent days, I’m having some issues with our server and, when it happens, it cannot get new registrations. Using top I saw a kamailio process is eating all CPU.
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 1556 kamailio 20 0 1010884 27624 24520 R 98.5 1.4 150:07.59 kamailio
I got backtrace attaching to Kamailio with gdb. Here is the backtrace:
#0 0x00007fb996992d6d in accept () at ../sysdeps/unix/syscall-template.S:81 #1 0x00000000005f3615 in handle_new_connect (si=0x7fb9960b4fa8) at tcp_main.c:3894 #2 0x00000000005f7029 in handle_io (fm=0x7fb9961da520, ev=1, idx=-1) at tcp_main.c:4199 #3 0x00000000005c68b2 in io_wait_loop_epoll (h=0xa1fdc0 <io_h>, t=5, repeat=0) at io_wait.h:1065 #4 0x00000000005fa04a in tcp_main_loop () at tcp_main.c:4490 #5 0x00000000004b48ba in main_loop () at main.c:1712 #6 0x00000000004bad12 in main (argc=13, argv=0x7ffd14247c18) at main.c:2631 (gdb) bt
At the moment I found the issue, there were 862 sip client registered correctly. Our server is configured to use only TLS and mysql as backend.
/usr/local/sbin/kamailio -v version: kamailio 4.4.4 (x86_64/linux) 852c47 flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: 852c47 compiled on 15:56:18 Dec 15 2016 with gcc 4.8.4
Can anyone address me to find out why this happens?
Many thanks, T.
Tomas Zanet R&D Software designer tzanet@came.com CAME S.p.A.
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Hello, I found the issue which was due to a misconfiguration of kamalio. I forgot to set open_files_limit value which was set to default values 1024, not enough for our installation. Increasing it, now everything works fine.
Maybe this could help anyone who use TLS connections.
BR.
T.
Tomas Zanet R&D Software designer tzanet@came.com CAME S.p.A.
-----Messaggio originale----- Da: sr-users [mailto:sr-users-bounces@lists.kamailio.org] Per conto di Tomas Zanet Inviato: venerdì 15 dicembre 2017 15:11 A: Kamailio (SER) - Users Mailing List sr-users@lists.sip-router.org Oggetto: [SR-Users] 4.4.4 CPU 100%
Hi all, I’ve been using Kamailio 4.4.4 for 8 months without any problems but in recent days, I’m having some issues with our server and, when it happens, it cannot get new registrations. Using top I saw a kamailio process is eating all CPU.
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 1556 kamailio 20 0 1010884 27624 24520 R 98.5 1.4 150:07.59 kamailio
I got backtrace attaching to Kamailio with gdb. Here is the backtrace:
#0 0x00007fb996992d6d in accept () at ../sysdeps/unix/syscall-template.S:81 #1 0x00000000005f3615 in handle_new_connect (si=0x7fb9960b4fa8) at tcp_main.c:3894 #2 0x00000000005f7029 in handle_io (fm=0x7fb9961da520, ev=1, idx=-1) at tcp_main.c:4199 #3 0x00000000005c68b2 in io_wait_loop_epoll (h=0xa1fdc0 <io_h>, t=5, repeat=0) at io_wait.h:1065 #4 0x00000000005fa04a in tcp_main_loop () at tcp_main.c:4490 #5 0x00000000004b48ba in main_loop () at main.c:1712 #6 0x00000000004bad12 in main (argc=13, argv=0x7ffd14247c18) at main.c:2631 (gdb) bt
At the moment I found the issue, there were 862 sip client registered correctly. Our server is configured to use only TLS and mysql as backend.
/usr/local/sbin/kamailio -v version: kamailio 4.4.4 (x86_64/linux) 852c47 flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: 852c47 compiled on 15:56:18 Dec 15 2016 with gcc 4.8.4
Can anyone address me to find out why this happens?
Many thanks, T.
Tomas Zanet R&D Software designer tzanet@came.com CAME S.p.A.
_______________________________________________ Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
-
Daniel-Constantin Mierla -
Tomas Zanet