[SR-Users] lost sip messages

[Henning Westerholt]

Hello Jay,

maybe this is interesting for you:

https://www.kamailio.org/wiki/cookbooks/5.2.x/core#latency_limit_db

You can log an error message if a DB operation from Kamailio takes a long time.

Cheers,

Henning

Am 02.08.19 um 01:52 schrieb jay binks:
Firstly, thankyou for fixing my missing subject... oops :(


is the traffic sent over UDP?

Yes the traffic is UDP, but im 100% sure the traffic arrived at the server ( verified by TCPDump on the server ).


Can you look with netstat and see if the receiving queue for the socket of kamailio is buffering data?

 Sure, to avoid confusion, can you point me in the right direction here, as im not familiar with this metric.

What is the operating system? Do you have contrack (kernel module) or selinux enabled in the system?

Debian 9, on a dual Xeon(R) Silver 4114 CPU @ 2.20GHz ( 40 cores ).
32 gig of ram,  GigE NIC.
No Selinux
following contract modules are loaded :
nf_conntrack_ipv6      20480  1
nf_defrag_ipv6         20480  1 nf_conntrack_ipv6
nf_conntrack_ipv4      16384  1
nf_defrag_ipv4         16384  1 nf_conntrack_ipv4
xt_conntrack           16384  2
nf_conntrack          114688  3 nf_conntrack_ipv6,nf_conntrack_ipv4,xt_conntrack
x_tables               36864  11 xt_comment,xt_multiport,ipt_REJECT,iptable_mangle,ip_tables,iptable_filter,ip6t_REJECT,ip6table_filter,xt_DSCP,xt_conntrack,ip6_tables


Verify also that you don't have pike module enabled with a low rate limit.

I have removed pike, and replicated without pike loaded.

Just for clarity sake, adding more workers 100% fixed the issue.
What im really seeking is a way to monitor for this scenario so I can graph it and not get caught out by surprise.

In the last day or 2, ive found that some of the DB Queries in my kam config were taking longer than they should ( for a few reasons ), so this will be a largly contributing factor, however ... it still comes back to, how do I monitor for packets being dropped because there was no worker.

Thanks all

Jay


Cheers,
Daniel

On 31.07.19 04:57, jay binks wrote:
Hey all,
looking for a little help.

I've been tracking an issue, where a number of SIP Messages ( typically INVITE ) are sent, but Kamailio only see one of them ( the last one).  I have verified that the messages are definitely received on the host, as verified with TCPDUMP, but the INVITES never hit the request route block. ( the very first thing is to log the message )

What i've managed to find is, that if I increase worker children, the problem goes away.

So I've got a problem, I think I've found the solution, but what I'm struggling with is how to monitor better, so I can be alerted to this in the future.

I started by looking at "kamctl stats | grep drop"
    "core:drop_replies = 14",
    "core:drop_requests = 10577"

however drop_requests seems to include explicit drops in my config.
which I do for many reasons, but mainly bad UA or blocked IP's, so this dosnt seem to be what im after.

I then went digging in the code ( often the best way to find things ).
in receive.c receive_msg() we find some conditions, where we drop the packet ( before calling request route ).

right now im looking around line 251 at
LM_DBG("dropping the received message\n");
goto error00;

Following error00 I see it calls STATS_RX_DROPS which increments stats->received_drops.
This is where it gets fun, I cant find where received_drops is ever returned in the stats output.

I can see where -SIGUSR1 will trigger this to be dumped, if Kamailio is compiled with that option ( dosnt seem the debian packages are ).  however, what im seeking clarification on is, does this get reported in "kamctl stats"

Am I just going crazy or is something not quite right here?

TLDNR;  how do I monitor for Kamailio not having enough worker threads to process incoming messages.

FYI I'm seeing about 50Mbps of incoming SIP on this box, and I do perform a decent amount of work on some message types.

--
Sincerely

Jay



_______________________________________________
Kamailio (SER) - Users Mailing List
sr-users at lists.kamailio.org<mailto:sr-users at lists.kamailio.org>
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users


--
Daniel-Constantin Mierla -- www.asipto.com<http://www.asipto.com>
www.twitter.com/miconda<http://www.twitter.com/miconda> -- www.linkedin.com/in/miconda<http://www.linkedin.com/in/miconda>


--
Sincerely

Jay



_______________________________________________
Kamailio (SER) - Users Mailing List
sr-users at lists.kamailio.org<mailto:sr-users at lists.kamailio.org>
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users


--
Henning Westerholt - https://skalatan.de/blog/
Kamailio services - https://skalatan.de/services
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.kamailio.org/pipermail/sr-users/attachments/20190802/88fc3b74/attachment-0001.html>