New subject: [kamailio/kamailio] pipelimit: module causes high CPU load due to periodic 1s /proc/net/upd|tcp and /proc/stat system call if server RAM >= 48GB (#1675)

12 Oct 2018

### Description

Kamailio pipelimit module causes high CPU load (20 - 30 %) on servers with > 48 GB RAM
even if Kamailio is idle (no active calls).

strace shows that every second `open /pro/stat` is called even if timer_interval is set to
10 seconds
`modparam("pipelimit", "timer_interval", 10)`

`1539343630.677932 open("/proc/stat", O_RDONLY) = 6`
`1539343630.678278 open("/sys/devices/system/cpu/online", O_RDONLY|O_CLOEXEC) =
6`
`1539343630.678418 open("/proc/net/udp", O_RDONLY) = 6`
`1539343630.679448 open("/proc/net/tcp", O_RDONLY) = 6`
`1539343630.722522 open("/proc/net/tcp", O_RDONLY) = 6`
`1539343630.753095 open("/proc/net/udp", O_RDONLY) = 6`
`1539343630.753838 open("/proc/net/tcp", O_RDONLY) = 6`
`1539343630.783710 open("/proc/net/tcp", O_RDONLY) = 6`
`1539343631.677966 open("/proc/stat", O_RDONLY) = 6`
`1539343631.678378 open("/sys/devices/system/cpu/online", O_RDONLY|O_CLOEXEC) =
6`
`1539343631.678534 open("/proc/net/udp", O_RDONLY) = 6`
`1539343631.679784 open("/proc/net/tcp", O_RDONLY) = 6`
`1539343631.717329 open("/proc/net/tcp", O_RDONLY) = 6`
`1539343631.760763 open("/proc/net/udp", O_RDONLY) = 6`
`1539343631.761786 open("/proc/net/tcp", O_RDONLY) = 6`
`1539343631.805426 open("/proc/net/tcp", O_RDONLY) = 6`

strace shows that with 8 GB  RAM the read syscall takes 100us (average)  - with 48GB it
takes 586us (average)! 

see 5 seconds interval strace for 48GB:

`2018-09-21 07:55:38,201`
`strace: Process 6907 attached`
`strace: Process 6907 detached`
`% time     seconds  usecs/call     calls    errors syscall`
`------ ----------- ----------- --------- --------- ----------------`
`99.79    1.259694         586      2148           read`
 ` 0.06    0.000761          19        41           open`
 ` 0.04    0.000469           8        61        61 pause`
 ` 0.04    0.000462          13        36           munmap`
 ` 0.03    0.000336           8        41           close`
 ` 0.02    0.000257           7        36           mmap`
 ` 0.02    0.000234           3        80        61 rt_sigreturn`
 ` 0.01    0.000113           3        36           fstat`
 ` 0.00    0.000017           3         5           kill`
`------ ----------- ----------- --------- --------- ----------------`
`100.00    1.262343                  2484       122 total`
`2018-09-21 07:55:43,225`

In the documentation (https://kamailio.org/docs/modules/5.1.x/modules/pipelimit.html) I
found the following:

...
  As reading the CPU load average is relatively
expensive (opening /proc/stat, parsing it, etc), this only happens once every
timer_interval seconds and consequently the FEEDBACK value is only at these intervals
recomputed. This in turn makes it difficult for the drop rate to adjust quickly. Worst
case scenarios are request rates going up/down instantly by thousands - it takes up to 20
seconds for the controller to adapt to the new request rate.  
IMHO the doc is misleading because the periodic timer used for this is hardcoded with
1000ms.
see
`( 349 pipelimit.c ) timer_add(pl_timer, MS_TO_TICKS(1000)); /* Start it after 1000ms */`

Conclusion:
Pipelimit polls periodically every second /proc/stats and /proc/net/udp|tcp. The modparam
timer_interval does not affect the timer period.
On servers with >48 GB this causes high CPU load.

#### Reproduction

always reproducible.

### Possible Solutions

Use modparam timer_interval to initialize the `pl_timer` interval

### Additional Information

 version: kamailio 5.1.2 (x86_64/linux)
flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, DISABLE_NAGLE, USE_MCAST,
DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC,
DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER,
USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024,
BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
id: unknown

* **Operating System**:  Centos 7.4

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1675

[kamailio/kamailio] pipelimit: module causes high CPU load due to periodic 1s /pro/net/upd|tcp and /pro/stat system call if server RAM >= 48GB (#1675)