2 Aug
2019
2 Aug
'19
1:52 a.m.
Firstly, thankyou for fixing my missing subject... oops :(
is the traffic sent over UDP?
Yes the traffic is UDP, but im 100% sure the
traffic arrived at the server
( verified by TCPDump on the server ).
> Can you look with netstat and see if the receiving queue for the socket of
> kamailio is buffering data?
Sure, to avoid confusion, can you point me in
the right direction here, as
im not familiar with this metric.
> What is the operating system? Do you have contrack (kernel module) or
> selinux enabled in the system?
Debian 9, on a dual Xeon(R) Silver 4114 CPU @
2.20GHz ( 40 cores ).
32 gig of ram, GigE NIC.
No Selinux
following contract modules are loaded :
nf_conntrack_ipv6 20480 1
nf_defrag_ipv6 20480 1 nf_conntrack_ipv6
nf_conntrack_ipv4 16384 1
nf_defrag_ipv4 16384 1 nf_conntrack_ipv4
xt_conntrack 16384 2
nf_conntrack 114688 3
nf_conntrack_ipv6,nf_conntrack_ipv4,xt_conntrack
x_tables 36864 11
xt_comment,xt_multiport,ipt_REJECT,iptable_mangle,ip_tables,iptable_filter,ip6t_REJECT,ip6table_filter,xt_DSCP,xt_conntrack,ip6_tables
Verify also that you don't have pike module enabled with a low rate limit.
I have removed pike, and replicated without pike
loaded.
Just for clarity sake, adding more workers 100% fixed the issue.
What im really seeking is a way to monitor for this scenario so I can graph
it and not get caught out by surprise.
In the last day or 2, ive found that some of the DB Queries in my kam
config were taking longer than they should ( for a few reasons ), so this
will be a largly contributing factor, however ... it still comes back to,
how do I monitor for packets being dropped because there was no worker.
Thanks all
Jay
> Cheers,
> Daniel
> On 31.07.19 04:57, jay binks wrote:
> Hey all,
> looking for a little help.
> I've been tracking an issue, where a
number of SIP Messages ( typically
> INVITE ) are sent, but Kamailio only see one of them ( the last one). I
> have verified that the messages are definitely received on the host, as
> verified with TCPDUMP, but the INVITES never hit the request route block. (
> the very first thing is to log the message )
> What i've managed to find is, that if I
increase worker children, the
> problem goes away.
> So I've got a problem, I think I've
found the solution, but what I'm
> struggling with is how to monitor better, so I can be alerted to this in
> the future.
> I started by looking at "kamctl stats
| grep drop"
> "core:drop_replies = 14",
> "core:drop_requests = 10577"
> however drop_requests seems to include
explicit drops in my config.
> which I do for many reasons, but mainly bad UA or blocked IP's, so this
> dosnt seem to be what im after.
> I then went digging in the code ( often the
best way to find things ).
> in receive.c receive_msg() we find some conditions, where we drop the
> packet ( before calling request route ).
> right now im looking around line 251 at
> LM_DBG("dropping the received message\n");
> goto error00;
> Following error00 I see it calls
STATS_RX_DROPS which increments
> stats->received_drops.
> This is where it gets fun, I cant find where received_drops is ever
> returned in the stats output.
> I can see where -SIGUSR1 will trigger this
to be dumped, if Kamailio is
> compiled with that option ( dosnt seem the debian packages are ). however,
> what im seeking clarification on is, does this get reported in "kamctl
> stats"
> Am I just going crazy or is something not
quite right here?
> TLDNR; how do I monitor for Kamailio not
having enough worker threads to
> process incoming messages.
> FYI I'm seeing about 50Mbps of incoming
SIP on this box, and I do perform
> a decent amount of work on some message types.
> --
> Sincerely
> Jay
>
_______________________________________________
> Kamailio (SER) - Users Mailing
Listsr-users@lists.kamailio.orghttps://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
> --
> Daniel-Constantin Mierla -- www.asipto.comwww.twitter.com/miconda --
www.linkedin.com/in/miconda
--
Sincerely
Jay