27 May
2009
27 May
'09
6:30 p.m.
Hello,
we have kamailio 1.4 (r. 5728) running in production. We compiled
kamailio with 4 times the default pkg memory pool size (#define
PKG_MEM_POOL_SIZE 4*1024*1024 ).
We use snmpstats to monitor registration and number of calls using
Nagios and PRTG.
Some days ago, after almost 30 days of continuous operation without
problems, in the shell we executed "snmpwalk -v2c -c public
192.168.88.22 .1.3.6.1.4.1.27483" as we routinely do for a quick check
of SNMP in the command line and it failed. Then we checked kamailio
logs and we found this:
May 21 16:25:35 ipx022 /usr/local/sbin/kamailio[8781]:
ERROR:snmpstats:handle_openserSIPServiceStartTime: failed to read
sysUpTime file at /tmp/openSER_SNMPAgent.txt
I don't know what caused the above (maybe the file was mistakenly rm'd
by someone. I failed to check it at that moment) but it was followed
with this:
May 21 16:25:35 ipx022 /usr/local/sbin/kamailio[8781]:
ERROR:snmpstats:executeInterprocessBufferCmd: Received a request for
contact: [protected]@[protected] for user:
sip:[protected]@[protected] who doesn't exists
May 21 16:25:35 ipx022 /usr/local/sbin/kamailio[8781]:
ERROR:snmpstats:executeInterprocessBufferCmd: Received a request to
delete contact: [protected]@[protected] for user:
sip:[protected]@[protected] who doesn't exist
... then the same log "contact XXX who doesn't exist" happened several
times
per second till:
May 21 16:36:30 ipx022 /usr/local/sbin/kamailio[8781]:
ERROR:snmpstats:createRegUserRow: failed to create a row for
openserSIPRegUserTable
May 21 16:36:30 ipx022 /usr/local/sbin/kamailio[8781]:
ERROR:snmpstats:updateUser: openserSIPRegUserTable ran out of memory.
Not able to add user: [protected]@[protected]
May 21 16:36:30 ipx022 /usr/local/sbin/kamailio[8781]:
ERROR:snmpstats:executeInterprocessBufferCmd: Received a request for
contact: [protected]@[protected] for user:
sip:[protected]@[protected] who doesn't exists
May 21 16:36:30 ipx022 /usr/local/sbin/kamailio[8781]:
ERROR:snmpstats:insertContactRecord: no more pkg memory
May 21 16:36:30 ipx022 /usr/local/sbin/kamailio[8781]:
ERROR:snmpstats:executeInterprocessBufferCmd: openserSIPRegUserTable
was unable to allocate memory for adding contact:
[protected]@[protected] to user
sip:[protected]@[protected].
During this time, snmpget queries issued by Nagios and PRTG were
failing and the process spawned by module snmpstats was making heavy
use of CPU.
No other parts of kamailio were affected (calls and registration were
OK), but snmpstats didn't normalize by itself so we decided to restart
kamailio.
I'm trying to recreate this in a lab machine but no luck so far.
I am keeping the lab server busy with registration and calls while
running snmpwalk in a loop. I've also ran "rm
/tmp/openSER_SNMPAgent.txt" to check if could trigger the problem but
the problem could not be recreated.
If I push the lab server enough, I can see similar logs like this:
May 27 20:16:54 ipx029 /usr/local/sbin/kamailio[14487]:
ERROR:snmpstats:updateUser: openserSIPRegUserTable ran out of memory.
Not able to add user: [protected]@[protected]
May 27 20:16:54 ipx029 /usr/local/sbin/kamailio[14487]:
ERROR:snmpstats:executeInterprocessBufferCmd: Received a request for
contact: [protected]@[protected] for user: sip:[protected]@[protected]
who doesn't exists
May 27 20:17:05 ipx029 /usr/local/sbin/kamailio[14487]:
ERROR:snmpstats:get_socket_list_from_proto: no more pkg memory
May 27 20:17:05 ipx029 last message repeated 9 times
May 27 20:17:59 ipx029 /usr/local/sbin/kamailio[14487]:
ERROR:snmpstats:handle_openserSIPServiceStartTime: failed to read
sysUpTime file at /tmp/openSER_SNMPAgent.txt
May 27 20:40:25 ipx029 /usr/local/sbin/kamailio[14487]:
ERROR:snmpstats:handle_openserSIPServiceStartTime: failed to read
sysUpTime file at /tmp/openSER_SNMPAgent.txt
But even with those messages, snmpget/snmpwalk doesn't stop to return responses.
I was hoping someone could take a look at snmpstats code; the logs for
snmpstats:executeInterprocessBufferCmd are marked with comments like :
/* This should never happen. This is more of a sanity check. */
so probably my server hit some bug.
regards,
takeshi