[SR-Users] lost sip messages

Fri Aug 2 10:22:01 CEST 2019

On 02.08.19 01:52, jay binks wrote:
> Firstly, thankyou for fixing my missing subject... oops :(
>
>     is the traffic sent over UDP?
>
> Yes the traffic is UDP, but im 100% sure the traffic arrived at the
> server ( verified by TCPDump on the server ).

If you see the traffic with ngrep. tcpdump, etc ... it still does not
mean it reaches the application layer, these tools hook into network
traffic callbacks and from there the kernel or firewall can still drop
the traffic and the application (kamailio) won't get it. Might not be
the case here, but that can happen.

>  
>
>     Can you look with netstat and see if the receiving queue for the
>     socket of kamailio is buffering data?
>
>  Sure, to avoid confusion, can you point me in the right direction
> here, as im not familiar with this metric.

If it is udp traffic, do:

netstat -alupn

Look for the lines of kamailio and the sip port (5060), something like:

Proto Recv-Q Send-Q Local Address           Foreign Address        
State       PID/Program name

udp        0      0 127.0.0.1:5060         
0.0.0.0:*                           4960/kamailio

If the Recv-Q value is high, then kamailio cannot handle the traffic
there in a timely manner, so some of them can get dropped at some point,
when the socket buffer is full.

Cheers,
Daniel

>     What is the operating system? Do you have contrack (kernel module)
>     or selinux enabled in the system?
>
> Debian 9, on a dual Xeon(R) Silver 4114 CPU @ 2.20GHz ( 40 cores ).
> 32 gig of ram,  GigE NIC.
> No Selinux 
> following contract modules are loaded : 
>
>     nf_conntrack_ipv6      20480  1
>     nf_defrag_ipv6         20480  1 nf_conntrack_ipv6
>     nf_conntrack_ipv4      16384  1
>     nf_defrag_ipv4         16384  1 nf_conntrack_ipv4
>     xt_conntrack           16384  2
>     nf_conntrack          114688  3
>     nf_conntrack_ipv6,nf_conntrack_ipv4,xt_conntrack
>     x_tables               36864  11
>     xt_comment,xt_multiport,ipt_REJECT,iptable_mangle,ip_tables,iptable_filter,ip6t_REJECT,ip6table_filter,xt_DSCP,xt_conntrack,ip6_tables
>
>
>     Verify also that you don't have pike module enabled with a low
>     rate limit.
>
> I have removed pike, and replicated without pike loaded.
>
> Just for clarity sake, adding more workers 100% fixed the issue.
> What im really seeking is a way to monitor for this scenario so I can
> graph it and not get caught out by surprise.
>
> In the last day or 2, ive found that some of the DB Queries in my kam
> config were taking longer than they should ( for a few reasons ), so
> this will be a largly contributing factor, however ... it still comes
> back to, how do I monitor for packets being dropped because there was
> no worker.
>
> Thanks all
>
> Jay
>  
>
>     Cheers,
>     Daniel
>
>     On 31.07.19 04:57, jay binks wrote:
>>     Hey all,
>>     looking for a little help.
>>
>>     I've been tracking an issue, where a number of SIP Messages (
>>     typically INVITE ) are sent, but Kamailio only see one of them (
>>     the last one).  I have verified that the messages are definitely
>>     received on the host, as verified with TCPDUMP, but the INVITES
>>     never hit the request route block. ( the very first thing is to
>>     log the message )
>>
>>     What i've managed to find is, that if I increase worker children,
>>     the problem goes away.
>>
>>     So I've got a problem, I think I've found the solution, but what
>>     I'm struggling with is how to monitor better, so I can be alerted
>>     to this in the future.
>>
>>     I started by looking at "kamctl stats | grep drop"
>>         "core:drop_replies = 14",
>>         "core:drop_requests = 10577"
>>
>>     however drop_requests seems to include explicit drops in my config.
>>     which I do for many reasons, but mainly bad UA or blocked IP's,
>>     so this dosnt seem to be what im after.
>>
>>     I then went digging in the code ( often the best way to find
>>     things ).
>>     in receive.c receive_msg() we find some conditions, where we drop
>>     the packet ( before calling request route ).
>>
>>     right now im looking around line 251 at 
>>     LM_DBG("dropping the received message\n"); 
>>     goto error00;
>>
>>     Following error00 I see it calls STATS_RX_DROPS which increments
>>     stats->received_drops.
>>     This is where it gets fun, I cant find where received_drops is
>>     ever returned in the stats output.
>>
>>     I can see where -SIGUSR1 will trigger this to be dumped, if
>>     Kamailio is compiled with that option ( dosnt seem the debian
>>     packages are ).  however, what im seeking clarification on is,
>>     does this get reported in "kamctl stats"
>>
>>     Am I just going crazy or is something not quite right here?
>>
>>     TLDNR;  how do I monitor for Kamailio not having enough worker
>>     threads to process incoming messages.
>>
>>     FYI I'm seeing about 50Mbps of incoming SIP on this box, and I do
>>     perform a decent amount of work on some message types.
>>
>>     -- 
>>     Sincerely
>>
>>     Jay
>>
>>     _______________________________________________
>>     Kamailio (SER) - Users Mailing List
>>     sr-users at lists.kamailio.org <mailto:sr-users at lists.kamailio.org>
>>     https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
>
>     -- 
>     Daniel-Constantin Mierla -- www.asipto.com <http://www.asipto.com>
>     www.twitter.com/miconda <http://www.twitter.com/miconda> -- www.linkedin.com/in/miconda <http://www.linkedin.com/in/miconda>
>
>
>
> -- 
> Sincerely
>
> Jay

-- 
Daniel-Constantin Mierla -- www.asipto.com
www.twitter.com/miconda -- www.linkedin.com/in/miconda

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.kamailio.org/pipermail/sr-users/attachments/20190802/63867050/attachment.html>