Hey all,
looking for a little help.
I've been tracking an issue, where a number of SIP Messages ( typically INVITE ) are sent, but Kamailio only see one of them ( the last one). I have verified that the messages are definitely received on the host, as verified with TCPDUMP, but the INVITES never hit the request route block. ( the very first thing is to log the message )
What i've managed to find is, that if I increase worker children, the problem goes away.
So I've got a problem, I think I've found the solution, but what I'm struggling with is how to monitor better, so I can be alerted to this in the future.
I started by looking at "kamctl stats | grep drop"
"core:drop_replies = 14",
"core:drop_requests = 10577"
however drop_requests seems to include explicit drops in my config.
which I do for many reasons, but mainly bad UA or blocked IP's, so this dosnt seem to be what im after.
I then went digging in the code ( often the best way to find things ).
in receive.c receive_msg() we find some conditions, where we drop the packet ( before calling request route ).
right now im looking around line 251 at
LM_DBG("dropping the received message\n");
goto error00;
Following error00 I see it calls STATS_RX_DROPS which increments stats->received_drops.
This is where it gets fun, I cant find where received_drops is ever returned in the stats output.
I can see where -SIGUSR1 will trigger this to be dumped, if Kamailio is compiled with that option ( dosnt seem the debian packages are ). however, what im seeking clarification on is, does this get reported in "kamctl stats"
Am I just going crazy or is something not quite right here?
TLDNR; how do I monitor for Kamailio not having enough worker threads to process incoming messages.
FYI I'm seeing about 50Mbps of incoming SIP on this box, and I do perform a decent amount of work on some message types.