You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.
Kamailio pipelimit module causes high CPU load (20 - 30 %) on servers with > 48 GB RAM even if Kamailio is idle (no active calls).
strace shows that every second open /pro/stat is called even if timer_interval is set to 10 seconds
modparam("pipelimit", "timer_interval", 10)
1539343630.677932 open("/proc/stat", O_RDONLY) = 6
1539343630.678278 open("/sys/devices/system/cpu/online", O_RDONLY|O_CLOEXEC) = 6
1539343630.678418 open("/proc/net/udp", O_RDONLY) = 6
1539343630.679448 open("/proc/net/tcp", O_RDONLY) = 6
1539343630.722522 open("/proc/net/tcp", O_RDONLY) = 6
1539343630.753095 open("/proc/net/udp", O_RDONLY) = 6
1539343630.753838 open("/proc/net/tcp", O_RDONLY) = 6
1539343630.783710 open("/proc/net/tcp", O_RDONLY) = 6
1539343631.677966 open("/proc/stat", O_RDONLY) = 6
1539343631.678378 open("/sys/devices/system/cpu/online", O_RDONLY|O_CLOEXEC) = 6
1539343631.678534 open("/proc/net/udp", O_RDONLY) = 6
1539343631.679784 open("/proc/net/tcp", O_RDONLY) = 6
1539343631.717329 open("/proc/net/tcp", O_RDONLY) = 6
1539343631.760763 open("/proc/net/udp", O_RDONLY) = 6
1539343631.761786 open("/proc/net/tcp", O_RDONLY) = 6
1539343631.805426 open("/proc/net/tcp", O_RDONLY) = 6
strace shows that with 8 GB RAM the read syscall takes 100us (average) - with 48GB it takes 586us (average)!
see 5 seconds interval strace for 48GB:
2018-09-21 07:55:38,201
strace: Process 6907 attached
strace: Process 6907 detached
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
99.79 1.259694 586 2148 read
0.06 0.000761 19 41 open
0.04 0.000469 8 61 61 pause
0.04 0.000462 13 36 munmap
0.03 0.000336 8 41 close
0.02 0.000257 7 36 mmap
0.02 0.000234 3 80 61 rt_sigreturn
0.01 0.000113 3 36 fstat
0.00 0.000017 3 5 kill
------ ----------- ----------- --------- --------- ----------------
100.00 1.262343 2484 122 total
2018-09-21 07:55:43,225
In the documentation (https://kamailio.org/docs/modules/5.1.x/modules/pipelimit.html) I found the following:
As reading the CPU load average is relatively expensive (opening /proc/stat, parsing it, etc), this only happens once every timer_interval seconds and consequently the FEEDBACK value is only at these intervals recomputed. This in turn makes it difficult for the drop rate to adjust quickly. Worst case scenarios are request rates going up/down instantly by thousands - it takes up to 20 seconds for the controller to adapt to the new request rate.
IMHO the doc is misleading because the periodic timer used for this is hardcoded with 1000ms.
see
( 349 pipelimit.c ) timer_add(pl_timer, MS_TO_TICKS(1000)); /* Start it after 1000ms */
Conclusion:
Pipelimit polls periodically every second /proc/stats and /proc/net/udp|tcp. The modparam timer_interval does not affect the timer period.
On servers with >48 GB this causes high CPU load.
always reproducible.
Use modparam timer_interval to initialize the pl_timer interval
version: kamailio 5.1.2 (x86_64/linux)
flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
id: unknown
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.