[kamailio/kamailio] Problem with process limit of count wtimer secondary process (#1237)
### Description Kamailio won't start after pulling from master
Commit triggering the problem ``` git show ec15b23f25ba2502a661157ae69b4dc1db923f66 Author: Daniel-Constantin Mierla miconda@gmail.com Date: Fri Sep 1 12:00:06 2017 +0200
core: main - don't count wtimer secondary process separately
- it is registered via sr_wtimer_init() - fixes the extra empty entry in process table reported by Juha Heinanen ```
#### Reproduction
Always with specific configuration (I can diagnose further is needed)
#### Log Messages When trying to start Kamailio ``` 0(26583) CRITICAL: <core> [core/pt.c:285]: fork_process(): Process limit of 13 exceeded. Will simulate fork fail. 0(26583) CRITICAL: <core> [main.c:1720]: main_loop(): cannot fork tcp main process: Invalid argument ```
### Possible Solutions Not sure is the config I am using is not supported or if Kamailio code should be fixed. rolling back the commit is fixing the issue `git revert ec15b23f25ba2502a661157ae69b4dc1db923f66`
Can you list here the names of the modules you loaded in your config? Likely there is one that creates/forks an extra process, but it doesn't declare it.
@jchavanton -- were you able to look more into this one as per my previous comment?
sorry for the delay, I was thinking I would receive an email notification
14 processes ``` / # ps | grep kamailio 7 root 0:03 /usr/sbin/kamailio 24 root 0:00 /usr/sbin/kamailio 25 root 0:00 /usr/sbin/kamailio 26 root 0:00 /usr/sbin/kamailio 27 root 0:00 /usr/sbin/kamailio 28 root 0:00 /usr/sbin/kamailio 29 root 0:00 /usr/sbin/kamailio 30 root 0:00 /usr/sbin/kamailio 31 root 0:00 /usr/sbin/kamailio 32 root 0:00 /usr/sbin/kamailio 33 root 0:00 /usr/sbin/kamailio 34 root 0:00 /usr/sbin/kamailio 35 root 0:00 /usr/sbin/kamailio 36 root 0:00 /usr/sbin/kamailio ```
``` / # kamcmd ps 7 main process - attendant 24 udp receiver child=0 sock=117.75.69.33:5060 25 udp receiver child=0 sock=117.75.69.33:5160 26 slow timer 27 timer 28 secondary timer 29 JSONRPC-S FIFO 30 JSONRPC-S DATAGRAM 31 ctl handler 32 RTIMER EXEC child=0 timer=tmetrics 33 HTable Timer 34 Dialog Clean Timer 35 tcp receiver (generic) child=0 36 tcp main process ``` If you do not find the answer spontaneously, I will troubleshoot it further
This is on Linux Alpine in docker
I asked for the list of kamailio modules you load in config.
``` loadmodule "db_flatstore" loadmodule "acc" loadmodule "kex" loadmodule "jsonrpcs" loadmodule "mi_fifo" loadmodule "mi_rpc" loadmodule "ctl" loadmodule "tm" loadmodule "tmx" loadmodule "sl" loadmodule "diversion" loadmodule "rr" loadmodule "db_postgres" loadmodule "db_sqlite" loadmodule "pv" loadmodule "avpops" loadmodule "maxfwd" loadmodule "textops" loadmodule "siputils" loadmodule "xlog" loadmodule "permissions" loadmodule "dispatcher" loadmodule "sanity" loadmodule "auth" loadmodule "statistics" loadmodule "cfgutils" loadmodule "uac_redirect" loadmodule "statsd" loadmodule "sqlops" loadmodule "rtimer" loadmodule "htable" loadmodule "mqueue" loadmodule "http_client" loadmodule "app_lua" loadmodule "xhttp" loadmodule "jsonrpc-s" loadmodule "ipops" loadmodule "sqlops" loadmodule "cfgutils" loadmodule "statsd" loadmodule "path" loadmodule "textopsx" loadmodule "nathelper" loadmodule "uac" loadmodule "dialog" loadmodule "pike" loadmodule "ratelimit" loadmodule "xmlrpc" loadmodule "ndb_redis" loadmodule "topos" loadmodule "topos_redis" loadmodule "tm" loadmodule "statsd" ```
I thought that since `kamcmd ps` was matching the amount of Kamailio running processes we could be able to find why the count was not matching, in this
For the last test I lowered the amount of childs
``` 0(21919) CRITICAL: <core> [core/pt.c:285]: fork_process(): Process limit of 13 exceeded. Will simulate fork fail. ``` As seen above there is 14 processes when the server is starting properly, I concluded that is must be one of the following :
``` / # kamcmd ps 7 main process - attendant 24 udp receiver child=0 sock=117.75.69.33:5060 25 udp receiver child=0 sock=117.75.69.33:5160 26 slow timer 27 timer 28 secondary timer 29 JSONRPC-S FIFO 30 JSONRPC-S DATAGRAM 31 ctl handler 32 RTIMER EXEC child=0 timer=tmetrics 33 HTable Timer 34 Dialog Clean Timer 35 tcp receiver (generic) child=0 36 tcp main process ```
found it ! `modparam("htable", "timer_procs", 0)` # problem when set to 1
``` if(rank==PROC_MAIN) { if(ht_timer_procs>0) { for(i=0; i<ht_timer_procs; i++) { if(fork_sync_timer(PROC_TIMER, "HTable Timer", 1 /*socks flag*/, ht_timer, (void*)(long)i, ht_timer_interval)<0) { LM_ERR("failed to start timer routine as process\n"); return -1; /* error */ } } } } ```
Good catch, thanks for troubleshooting. The issue happened when asking for an extra clean up timer for hash tables but no hash table was defined with auto-expire (so the extra timer was not needed). Hopefully it is fixed with the patch referenced above.
I see, kind of a miss config in the end, thanks for the fix
Closed #1237.
-
Daniel-Constantin Mierla -
Julien Chavanton