28 Aug
2018
28 Aug
'18
10:42 a.m.
Hello,
I am using Kamailio as a SIP proxy.So it receives SIP packet from internet
and forwards it to FS servers in local network.When i execute "ss" command
i see very high value in RECV-Q column.I THINK IT IS NOT NORMAL.PLEASE
CORRECT ME IF I AM WRONG.
====================================================
[root@fep-1 proc]# ss -u -a -n -e | grep 5060
UNCONN 0 0 10.50.8.1:5060 *:*
ino:6831630 sk:fd <->
UNCONN 0 0 10.50.7.18:5060 *:*
ino:6831629 sk:fe <->
UNCONN *1183104* 0 10.50.7.254:5060 *:*
ino:6831627 sk:ff <->
UNCONN *84864* 0 2607:f900:1:3::254:5060
:::* ino:6831628 sk:100 v6only:1 <->
======================================================
Initially i thought that there is something in script which must be causing
kamailio to process UDP request slower but i faced same issue with a very
simple script where i simply reply with stateless 200 OK for each sip
request :
request_route {
sl_send_reply("200","OK");exit;
$avp(uuid) = $rm + "-" + $ci;
........
.........
.........
.......
Server configuration :
OS : CENTOS 7
Kernel : 4.16
CPU : 5 X Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz
RAM : 32 GB
Please help me debugging this issue.Thanks in advance
28 Aug
28 Aug
10:50 a.m.
New subject: Kamailio consuming UDP packets very slowly leading to high RECV - Q
No, this is not normal. But you almost certainly have some external I/O operation which is
blocking the worker thread. There isn't much explanation otherwise.
This article (disclaimer: I wrote it) may help:
http://www.evaristesys.com/blog/tuning-kamailio-for-high-throughput-and-per…
--
Sent from mobile. Kindly excuse brevity and errors.
11:31 a.m.
New subject: Kamailio consuming UDP packets very slowly leading to high RECV - Q
Hello,
On 28.08.18 12:42, sagar malam wrote:
Hello,
I am using Kamailio as a SIP proxy.So it receives SIP packet from
internet and forwards it to FS servers in local network.When i execute
"ss" command i see very high value in RECV-Q column.I THINK IT IS NOT
NORMAL.PLEASE CORRECT ME IF I AM WRONG.
====================================================
[root@fep-1 proc]# ss -u -a -n -e | grep 5060
UNCONN 0 0 10.50.8.1:5060 <http://10.50.8.1:5060>
*:* ino:6831630 sk:fd <->
UNCONN 0 0 10.50.7.18:5060 <http://10.50.7.18:5060>
*:* ino:6831629 sk:fe <->
UNCONN *1183104* 0 10.50.7.254:5060
<http://10.50.7.254:5060> *:*
ino:6831627 sk:ff <->
UNCONN *84864* 0 2607:f900:1:3::254:5060
:::* ino:6831628 sk:100 v6only:1 <->
======================================================
Initially i thought that there is something in script which must be
causing kamailio to process UDP request slower but i faced same issue
with a very simple script where i simply reply with stateless 200 OK
for each sip request :
request_route {
sl_send_reply("200","OK");exit;
$avp(uuid) = $rm + "-" + $ci;
........
.........
.........
.......
Server configuration :
OS : CENTOS 7
Kernel : 4.16
CPU : 5 X Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz
RAM : 32 GB
Please help me debugging this issue.Thanks in advance
if you have performance issues just with a very simple config sending a
stateless sip reply, then check your system/firewall
configuration/limits. Specially on centos, I have seen a lot of
restrictive traffic rates limits set by selinux. Also, if you run in a
virtual machine, there can be limits enforced by the vm platform.
If you still cannot sort out, I would just run similar tests on a
vanilla debian.
Cheers,
Daniel
--
Daniel-Constantin Mierla -- www.asipto.com
www.twitter.com/miconda -- www.linkedin.com/in/miconda
Kamailio World Conference -- www.kamailioworld.com
11:42 a.m.
New subject: Kamailio consuming UDP packets very slowly leading to high RECV - Q
Thanks for prompt response.
Some more information :
We are running it on virtual machine(Xenserver).
Selinux and IPtables are disabled.I will check if VM platform is putting
some limits or not.
Further i have same Kamailio script in another server and i am not facing
this issue there.
OS max UDP buffer limit parameters is configured as below :
net.core.rmem_max=26214400
net.core.rmem_default=26214400
Any other clue is most welcome.
On Tue, Aug 28, 2018 at 5:01 PM Daniel-Constantin Mierla <miconda(a)gmail.com>
wrote:
Hello,
On 28.08.18 12:42, sagar malam wrote:
Hello,
I am using Kamailio as a SIP proxy.So it receives SIP packet from internet
and forwards it to FS servers in local network.When i execute "ss" command
i see very high value in RECV-Q column.I THINK IT IS NOT NORMAL.PLEASE
CORRECT ME IF I AM WRONG.
====================================================
[root@fep-1 proc]# ss -u -a -n -e | grep 5060
UNCONN 0 0 10.50.8.1:5060 *:*
ino:6831630 sk:fd <->
UNCONN 0 0 10.50.7.18:5060 *:*
ino:6831629 sk:fe <->
UNCONN *1183104* 0 10.50.7.254:5060 *:*
ino:6831627 sk:ff <->
UNCONN *84864* 0 2607:f900:1:3::254:5060
:::* ino:6831628 sk:100 v6only:1 <->
======================================================
Initially i thought that there is something in script which must be
causing kamailio to process UDP request slower but i faced same issue with
a very simple script where i simply reply with stateless 200 OK for each
sip request :
request_route {
sl_send_reply("200","OK");exit;
$avp(uuid) = $rm + "-" + $ci;
........
.........
.........
.......
Server configuration :
OS : CENTOS 7
Kernel : 4.16
CPU : 5 X Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz
RAM : 32 GB
Please help me debugging this issue.Thanks in advance
if you have performance issues just with a very simple config sending a
stateless sip reply, then check your system/firewall configuration/limits.
Specially on centos, I have seen a lot of restrictive traffic rates limits
set by selinux. Also, if you run in a virtual machine, there can be limits
enforced by the vm platform.
If you still cannot sort out, I would just run similar tests on a vanilla
debian.
Cheers,
Daniel
--
Daniel-Constantin Mierla -- www.asipto.comwww.twitter.com/miconda --
www.linkedin.com/in/miconda
Kamailio World Conference -- www.kamailioworld.com
11:44 a.m.
New subject: Kamailio consuming UDP packets very slowly leading to high RECV - Q
On Tue, Aug 28, 2018 at 01:31:04PM +0200, Daniel-Constantin Mierla wrote:
if you have performance issues just with a very simple
config sending
a stateless sip reply, then check your system/firewall
configuration/limits. Specially on centos, I have seen a lot of
restrictive traffic rates limits set by selinux. Also, if you run in a
virtual machine, there can be limits enforced by the vm platform.
Really? When did SELinux get into the PPS enforcement business?
--
Alex Balashov | Principal | Evariste Systems LLC
Tel: +1-706-510-6800 / +1-800-250-5920 (toll-free)
Web: http://www.evaristesys.com/, http://www.csrpswitch.com/
11:45 a.m.
New subject: Kamailio consuming UDP packets very slowly leading to high RECV - Q
On Tue, Aug 28, 2018 at 05:12:01PM +0530, sagar malam wrote:
Further i have same Kamailio script in another server
and i am not
facing this issue there.
That pretty well points to the problem being caused by the runtime
environment rather than the Kamailio config or anything it does.
--
Alex Balashov | Principal | Evariste Systems LLC
Tel: +1-706-510-6800 / +1-800-250-5920 (toll-free)
Web: http://www.evaristesys.com/, http://www.csrpswitch.com/
11:52 a.m.
New subject: Kamailio consuming UDP packets very slowly leading to high RECV - Q
Alex,
I agree but i want to find root cause of this problem.Can you suggest me
what next i need to look at OS level ?
Thanks
On Tue, Aug 28, 2018 at 5:15 PM Alex Balashov <abalashov(a)evaristesys.com>
wrote:
On Tue, Aug 28, 2018 at 05:12:01PM +0530, sagar malam
wrote:
Further i have same Kamailio script in another
server and i am not
facing this issue there.
That pretty well points to the problem being caused by the runtime
environment rather than the Kamailio config or anything it does.
--
Alex Balashov | Principal | Evariste Systems LLC
Tel: +1-706-510-6800 <(706)%20510-6800> / +1-800-250-5920
<(800)%20250-5920> (toll-free)
Web: http://www.evaristesys.com/, http://www.csrpswitch.com/
_______________________________________________
Kamailio (SER) - Users Mailing List
sr-users(a)lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
12:30 p.m.
New subject: Kamailio consuming UDP packets very slowly leading to high RECV - Q
Well, I can't really say what limits selinux puts in place or other CentOS
specific apps, but I encountered various strange situations where no centos
sysadmin I interacted with could give a solution, even with selinux
disabled, so could be something else...
Coming in my mind: limit on number of new TCP connections per second or
number of forked processes per second - they are the reason for these
"delay" core parameters for mod/child init.
Using Debian stable was always smooth an without such strange limits...
Cheers,
Daniel
On Tue, 28 Aug 2018, 13:45 Alex Balashov, <abalashov(a)evaristesys.com> wrote:
On Tue, Aug 28, 2018 at 01:31:04PM +0200,
Daniel-Constantin Mierla wrote:
if you have performance issues just with a very
simple config sending
a stateless sip reply, then check your system/firewall
configuration/limits. Specially on centos, I have seen a lot of
restrictive traffic rates limits set by selinux. Also, if you run in a
virtual machine, there can be limits enforced by the vm platform.
Really? When did SELinux get into the PPS enforcement business?
--
Alex Balashov | Principal | Evariste Systems LLC
Tel: +1-706-510-6800 / +1-800-250-5920 (toll-free)
Web: http://www.evaristesys.com/, http://www.csrpswitch.com/
_______________________________________________
Kamailio (SER) - Users Mailing List
sr-users(a)lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
7:39 p.m.
New subject: Kamailio consuming UDP packets very slowly leading to high RECV - Q
Am Dienstag, 28. August 2018, 13:52:09 CEST schrieb sagar malam:
I agree but i want to find root cause of this
problem.Can you suggest me
what next i need to look at OS level ?
Hello Sagar,
as already suggested - I would try to sort out the other variables in this
setup:
- compare the server that you have that is working correctly to the one that
is not working correctly
- try with this particular setup on a "bare metal" instance or move for test
to a VM host without any other I/O heavy virtual instances
- try with another Linux (Debian)
- try to start Kamailio with debug logging enabled and have a closer look to
the logging timestamps, if there is maybe a suspicious delay during some
operations
Best regards,
Henning
--
Henning Westerholt
https://skalatan.de/blog/
29 Aug
29 Aug
9:59 a.m.
New subject: Kamailio consuming UDP packets very slowly leading to high RECV - Q
Hello Henning, Thanks for your inputs.
Issue got fixed after upgrading kernel version to 4.18.5( latest one). So
there was something wrong in Kernel 4.16.7.
On Wed, Aug 29, 2018 at 1:09 AM Henning Westerholt <hw(a)kamailio.org> wrote:
Am Dienstag, 28. August 2018, 13:52:09 CEST schrieb
sagar malam:
I agree but i want to find root cause of this
problem.Can you suggest me
what next i need to look at OS level ?
Hello Sagar,
as already suggested - I would try to sort out the other variables in this
setup:
- compare the server that you have that is working correctly to the one
that
is not working correctly
- try with this particular setup on a "bare metal" instance or move for
test
to a VM host without any other I/O heavy virtual instances
- try with another Linux (Debian)
- try to start Kamailio with debug logging enabled and have a closer look
to
the logging timestamps, if there is maybe a suspicious delay during some
operations
Best regards,
Henning
--
Henning Westerholt
https://skalatan.de/blog/
2084
days inactive
2085
days old
9 comments
4 participants
participants (4)
-
Alex Balashov -
Daniel-Constantin Mierla -
Henning Westerholt -
sagar malam