20 Dec
2021
20 Dec
'21
5:05 p.m.
Hi,
I have been doing some load testing with Kamailio and having some issues.
I have been trying with 100 CPS and at the beginning, everything is working
well but after some time (it can be 10 minutes or for example 40 minutes,
is just random but normally happens more in the first 10 minutes) Kamailio
stop replying to all SIP messages but still processing HTTP requests, some
commands, etc.
If I use "kamcmd dlg.list" no output happens and after that, all kamcmd
commands just keep loading with no output.
Kamailio is sharing Dialogs and htables using DMQ with other Kamailio that
is the failover, if I shut down the failover one (so no DMQ replication),
the test works well with the 100 CPS.
Also tried with modparam("dmq", "worker_usleep", 1000) but the
behavior is
the same, it will stop processing traffic.
This is some of configuration used:
# ----------------- setting module-specific parameters ---------------
modparam("dmq", "server_address",
"sip:INTERNAL_INSTANCE_IP:5060")
modparam("dmq", "notification_address",
"DMQ_NOTIFICATION_ADDRESS")
modparam("dmq", "multi_notify", 1)
modparam("dmq", "ping_interval", 10)
modparam("dmq", "num_workers",4)
modparam("htable", "enable_dmq", 1)
modparam("htable", "dmq_init_sync", 1)
# ----- dialog params -----
modparam("dialog", "dlg_flag", FLD_DLG)
modparam("dialog", "dlg_match_mode", 1)
modparam("dialog", "db_url", DBURL_RW)
modparam("dialog", "db_mode", 0)
modparam("dialog", "enable_dmq", 1)
modparam("dialog", "db_update_period", 10)
modparam("dialog", "h_id_start", H_ID_START)
modparam("dialog", "h_id_step", H_ID_STEP)
Memory:
Shared: 4096
Private: 512
kamailio -v
version: kamailio 5.5.3 (x86_64/linux) 473cef
flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE,
USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC,
TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT,
USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLOCKLIST,
HAVE_RESOLV_RES, TLS_PTHREAD_MUTEX_SHARED
ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144, MAX_URI_SIZE 1024,
BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
id: 473cef
compiled on 09:34:57 Dec 20 2021 with gcc 10.2.1
In attach is the trap collected after the issue happens.
Any more logs or configurations that can help identify or solve the issue?
Thanks for the help,
Regards,
Tiago
21 Dec
21 Dec
3:28 p.m.
What is the call length of each call? How many NICs you have on that
server? what networking switches are you using? all these variables have
huge impacts on the call flow!
Let's say you make 100CPS and the call length is 10 minutes.
At 10 seconds you will have 1.000 ongoing calls using g711 (64kbps). You
will have around 64Mbps bandwidth being used.
At 1 minute you get 6.000 ongoing calls and will use 390Mbps bandwidth!
Near the 10th minute you will be using almost 4Gbps bandwidth and 60.000
ongoing calls!!
Maybe the issue is not within Kamailio and/or the media server, but within
your network capacity ...
On my side, on a 4 vCPU and 4GB Ram, Kamailio 5.2 gets me around 400cps
with calls up to 30seconds length, and RTPEngine goes OK too. I am sure I
could do more, but I don't bother much, this is more than enough!
But I have an enormous amount of bandwidth and 3 NICs on the server. NICs
are the key . Remember that Kamailio only processes SIP messages.
At 5 minutes, using the above call details, you've already sent 30.000 SIP
INVITES minimum and many more SIP messages during that time. You need a
very good network interface to do that kind of job!
Add 30.000 channels / RTP ports to your media server and you get a whole
lot of data going on in your network. I bet even the UTP5 cables won't
handle the heat =:o)
Also, check the RTP port range your media server was configured with.
On Asterisk, usually ports 10.000 to 20.000 are set, which is about 10.000
calls. In the above example, you would stop receiving calls at 2minutes
because Asterisk has all RTP ports busy/in use!
I don't know about other media servers, but I would bet this is pretty much
the same...
Hope this helps!
*Sérgio Charrua*
*www.voip.pt <http://www.voip.pt/>*
Tel.: +351 <callto:+351+91+104+12+66>21 130 71 77
Email : *sergio.charrua(a)voip.pt <sergio.charrua(a)voip.pt>*
This message and any files or documents attached are strictly confidential
or otherwise legally protected.
It is intended only for the individual or entity named. If you are not the
named addressee or have received this email in error, please inform the
sender immediately, delete it from your system and do not copy or disclose
it or its contents or use it for any purpose. Please also note that
transmission cannot be guaranteed to be secure or error-free.
On Tue, Dec 21, 2021 at 11:54 AM pwerspire <pwerspire(a)gmail.com> wrote:
Hi,
I have been doing some load testing with Kamailio and having some issues.
I have been trying with 100 CPS and at the beginning, everything is
working well but after some time (it can be 10 minutes or for example 40
minutes, is just random but normally happens more in the first 10 minutes)
Kamailio stop replying to all SIP messages but still processing HTTP
requests, some commands, etc.
If I use "kamcmd dlg.list" no output happens and after that, all kamcmd
commands just keep loading with no output.
Kamailio is sharing Dialogs and htables using DMQ with other Kamailio that
is the failover, if I shut down the failover one (so no DMQ replication),
the test works well with the 100 CPS.
Also tried with modparam("dmq", "worker_usleep", 1000) but the
behavior is
the same, it will stop processing traffic.
This is some of configuration used:
# ----------------- setting module-specific parameters ---------------
modparam("dmq", "server_address",
"sip:INTERNAL_INSTANCE_IP:5060")
modparam("dmq", "notification_address",
"DMQ_NOTIFICATION_ADDRESS")
modparam("dmq", "multi_notify", 1)
modparam("dmq", "ping_interval", 10)
modparam("dmq", "num_workers",4)
modparam("htable", "enable_dmq", 1)
modparam("htable", "dmq_init_sync", 1)
# ----- dialog params -----
modparam("dialog", "dlg_flag", FLD_DLG)
modparam("dialog", "dlg_match_mode", 1)
modparam("dialog", "db_url", DBURL_RW)
modparam("dialog", "db_mode", 0)
modparam("dialog", "enable_dmq", 1)
modparam("dialog", "db_update_period", 10)
modparam("dialog", "h_id_start", H_ID_START)
modparam("dialog", "h_id_step", H_ID_STEP)
Memory:
Shared: 4096
Private: 512
kamailio -v
version: kamailio 5.5.3 (x86_64/linux) 473cef
flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS,
DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, Q_MALLOC,
F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT,
USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLOCKLIST,
HAVE_RESOLV_RES, TLS_PTHREAD_MUTEX_SHARED
ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144, MAX_URI_SIZE 1024,
BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
id: 473cef
compiled on 09:34:57 Dec 20 2021 with gcc 10.2.1
In attach is the trap collected after the issue happens.
Any more logs or configurations that can help identify or solve the issue?
Thanks for the help,
Regards,
Tiago
__________________________________________________________
Kamailio - Users Mailing List - Non Commercial Discussions
* sr-users(a)lists.kamailio.org
Important: keep the mailing list in the recipients, do not reply only to
the sender!
Edit mailing list options or unsubscribe:
* https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
22 Dec
22 Dec
6:50 p.m.
Hi Sergio,
Thanks for your reply.
This is a google cloud VM with 8 vCPU, I'm using 100 CPS but the number of
simultaneous calls is stable at around 1071 at any given time.
Also, I can always see INVITES going inside the server but Kamailio
stops answering them, even if you stop sending calls for a long time, and
then send a single call, Kamailio will not reply anymore to the Invites.
When Kamailio stops replying to INVITES, If I do for example a "kamcmd
dispatcher.list" I have the correct output, but if I do a "kamcmd
dlg.list"
it will not provide any output and not give back the prompt( before it
would just answer that the reply was too big).
I will need to use "control + c" to get the prompt of the shell back and
after that, all kamcmd commands stop returning any output or giving back
the prompt.
If the Kamailio is then restarted, everything starts working correctly
again. Also, another note is that this seems only to happen when DMQ is in
use and replicating to the fallback Kamailio.
Regards,
Tiago
Sergio Charrua <sergio.charrua(a)voip.pt> escreveu no dia terça, 21/12/2021
à(s) 13:38:
What is the call length of each call? How many NICs
you have on that
server? what networking switches are you using? all these variables have
huge impacts on the call flow!
Let's say you make 100CPS and the call length is 10 minutes.
At 10 seconds you will have 1.000 ongoing calls using g711 (64kbps). You
will have around 64Mbps bandwidth being used.
At 1 minute you get 6.000 ongoing calls and will use 390Mbps bandwidth!
Near the 10th minute you will be using almost 4Gbps bandwidth and 60.000
ongoing calls!!
Maybe the issue is not within Kamailio and/or the media server, but within
your network capacity ...
On my side, on a 4 vCPU and 4GB Ram, Kamailio 5.2 gets me around 400cps
with calls up to 30seconds length, and RTPEngine goes OK too. I am sure I
could do more, but I don't bother much, this is more than enough!
But I have an enormous amount of bandwidth and 3 NICs on the server. NICs
are the key . Remember that Kamailio only processes SIP messages.
At 5 minutes, using the above call details, you've already sent 30.000 SIP
INVITES minimum and many more SIP messages during that time. You need a
very good network interface to do that kind of job!
Add 30.000 channels / RTP ports to your media server and you get a whole
lot of data going on in your network. I bet even the UTP5 cables won't
handle the heat =:o)
Also, check the RTP port range your media server was configured with.
On Asterisk, usually ports 10.000 to 20.000 are set, which is about 10.000
calls. In the above example, you would stop receiving calls at 2minutes
because Asterisk has all RTP ports busy/in use!
I don't know about other media servers, but I would bet this is pretty
much the same...
Hope this helps!
*Sérgio Charrua*
*www.voip.pt <http://www.voip.pt/>*
Tel.: +351 <callto:+351+91+104+12+66>21 130 71 77
Email : *sergio.charrua(a)voip.pt <sergio.charrua(a)voip.pt>*
This message and any files or documents attached are strictly confidential
or otherwise legally protected.
It is intended only for the individual or entity named. If you are not the
named addressee or have received this email in error, please inform the
sender immediately, delete it from your system and do not copy or disclose
it or its contents or use it for any purpose. Please also note that
transmission cannot be guaranteed to be secure or error-free.
On Tue, Dec 21, 2021 at 11:54 AM pwerspire <pwerspire(a)gmail.com> wrote:
Hi,
I have been doing some load testing with Kamailio and having some issues.
I have been trying with 100 CPS and at the beginning, everything is
working well but after some time (it can be 10 minutes or for example 40
minutes, is just random but normally happens more in the first 10 minutes)
Kamailio stop replying to all SIP messages but still processing HTTP
requests, some commands, etc.
If I use "kamcmd dlg.list" no output happens and after that, all kamcmd
commands just keep loading with no output.
Kamailio is sharing Dialogs and htables using DMQ with other Kamailio
that is the failover, if I shut down the failover one (so no DMQ
replication), the test works well with the 100 CPS.
Also tried with modparam("dmq", "worker_usleep", 1000) but the
behavior
is the same, it will stop processing traffic.
This is some of configuration used:
# ----------------- setting module-specific parameters ---------------
modparam("dmq", "server_address",
"sip:INTERNAL_INSTANCE_IP:5060")
modparam("dmq", "notification_address",
"DMQ_NOTIFICATION_ADDRESS")
modparam("dmq", "multi_notify", 1)
modparam("dmq", "ping_interval", 10)
modparam("dmq", "num_workers",4)
modparam("htable", "enable_dmq", 1)
modparam("htable", "dmq_init_sync", 1)
# ----- dialog params -----
modparam("dialog", "dlg_flag", FLD_DLG)
modparam("dialog", "dlg_match_mode", 1)
modparam("dialog", "db_url", DBURL_RW)
modparam("dialog", "db_mode", 0)
modparam("dialog", "enable_dmq", 1)
modparam("dialog", "db_update_period", 10)
modparam("dialog", "h_id_start", H_ID_START)
modparam("dialog", "h_id_step", H_ID_STEP)
Memory:
Shared: 4096
Private: 512
kamailio -v
version: kamailio 5.5.3 (x86_64/linux) 473cef
flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS,
DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, Q_MALLOC,
F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT,
USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLOCKLIST,
HAVE_RESOLV_RES, TLS_PTHREAD_MUTEX_SHARED
ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144, MAX_URI_SIZE 1024,
BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
id: 473cef
compiled on 09:34:57 Dec 20 2021 with gcc 10.2.1
In attach is the trap collected after the issue happens.
Any more logs or configurations that can help identify or solve the issue?
Thanks for the help,
Regards,
Tiago
__________________________________________________________
Kamailio - Users Mailing List - Non Commercial Discussions
* sr-users(a)lists.kamailio.org
Important: keep the mailing list in the recipients, do not reply only to
the sender!
Edit mailing list options or unsubscribe:
* https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
__________________________________________________________
Kamailio - Users Mailing List - Non Commercial Discussions
* sr-users(a)lists.kamailio.org
Important: keep the mailing list in the recipients, do not reply only to
the sender!
Edit mailing list options or unsubscribe:
* https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
7:23 p.m.
any statistics regarding CPU use? how many threads?
*Sérgio Charrua*
*www.voip.pt <http://www.voip.pt/>*
Tel.: +351 <callto:+351+91+104+12+66>21 130 71 77
Email : *sergio.charrua(a)voip.pt <sergio.charrua(a)voip.pt>*
This message and any files or documents attached are strictly confidential
or otherwise legally protected.
It is intended only for the individual or entity named. If you are not the
named addressee or have received this email in error, please inform the
sender immediately, delete it from your system and do not copy or disclose
it or its contents or use it for any purpose. Please also note that
transmission cannot be guaranteed to be secure or error-free.
On Wed, Dec 22, 2021 at 3:57 PM pwerspire <pwerspire(a)gmail.com> wrote:
Hi Sergio,
Thanks for your reply.
This is a google cloud VM with 8 vCPU, I'm using 100 CPS but the number
of simultaneous calls is stable at around 1071 at any given time.
Also, I can always see INVITES going inside the server but Kamailio
stops answering them, even if you stop sending calls for a long time, and
then send a single call, Kamailio will not reply anymore to the Invites.
When Kamailio stops replying to INVITES, If I do for example a "kamcmd
dispatcher.list" I have the correct output, but if I do a "kamcmd
dlg.list"
it will not provide any output and not give back the prompt( before it
would just answer that the reply was too big).
I will need to use "control + c" to get the prompt of the shell back and
after that, all kamcmd commands stop returning any output or giving back
the prompt.
If the Kamailio is then restarted, everything starts working correctly
again. Also, another note is that this seems only to happen when DMQ is in
use and replicating to the fallback Kamailio.
Regards,
Tiago
Sergio Charrua <sergio.charrua(a)voip.pt> escreveu no dia terça, 21/12/2021
à(s) 13:38:
What is the call length of each call? How many
NICs you have on that
server? what networking switches are you using? all these variables have
huge impacts on the call flow!
Let's say you make 100CPS and the call length is 10 minutes.
At 10 seconds you will have 1.000 ongoing calls using g711 (64kbps). You
will have around 64Mbps bandwidth being used.
At 1 minute you get 6.000 ongoing calls and will use 390Mbps bandwidth!
Near the 10th minute you will be using almost 4Gbps bandwidth and 60.000
ongoing calls!!
Maybe the issue is not within Kamailio and/or the media server, but
within your network capacity ...
On my side, on a 4 vCPU and 4GB Ram, Kamailio 5.2 gets me around 400cps
with calls up to 30seconds length, and RTPEngine goes OK too. I am sure I
could do more, but I don't bother much, this is more than enough!
But I have an enormous amount of bandwidth and 3 NICs on the server. NICs
are the key . Remember that Kamailio only processes SIP messages.
At 5 minutes, using the above call details, you've already sent 30.000
SIP INVITES minimum and many more SIP messages during that time. You need a
very good network interface to do that kind of job!
Add 30.000 channels / RTP ports to your media server and you get a whole
lot of data going on in your network. I bet even the UTP5 cables won't
handle the heat =:o)
Also, check the RTP port range your media server was configured with.
On Asterisk, usually ports 10.000 to 20.000 are set, which is about
10.000 calls. In the above example, you would stop receiving calls at
2minutes because Asterisk has all RTP ports busy/in use!
I don't know about other media servers, but I would bet this is pretty
much the same...
Hope this helps!
*Sérgio Charrua*
*www.voip.pt <http://www.voip.pt/>*
Tel.: +351 <callto:+351+91+104+12+66>21 130 71 77
Email : *sergio.charrua(a)voip.pt <sergio.charrua(a)voip.pt>*
This message and any files or documents attached are strictly
confidential or otherwise legally protected.
It is intended only for the individual or entity named. If you are not
the named addressee or have received this email in error, please inform the
sender immediately, delete it from your system and do not copy or disclose
it or its contents or use it for any purpose. Please also note that
transmission cannot be guaranteed to be secure or error-free.
On Tue, Dec 21, 2021 at 11:54 AM pwerspire <pwerspire(a)gmail.com> wrote:
__________________________________________________________
Kamailio - Users Mailing List - Non Commercial Discussions
* sr-users(a)lists.kamailio.org
Important: keep the mailing list in the recipients, do not reply only to
the sender!
Edit mailing list options or unsubscribe:
* https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Hi,
I have been doing some load testing with Kamailio and having some issues.
I have been trying with 100 CPS and at the beginning, everything is
working well but after some time (it can be 10 minutes or for example 40
minutes, is just random but normally happens more in the first 10 minutes)
Kamailio stop replying to all SIP messages but still processing HTTP
requests, some commands, etc.
If I use "kamcmd dlg.list" no output happens and after that, all kamcmd
commands just keep loading with no output.
Kamailio is sharing Dialogs and htables using DMQ with other Kamailio
that is the failover, if I shut down the failover one (so no DMQ
replication), the test works well with the 100 CPS.
Also tried with modparam("dmq", "worker_usleep", 1000) but the
behavior
is the same, it will stop processing traffic.
This is some of configuration used:
# ----------------- setting module-specific parameters ---------------
modparam("dmq", "server_address",
"sip:INTERNAL_INSTANCE_IP:5060")
modparam("dmq", "notification_address",
"DMQ_NOTIFICATION_ADDRESS")
modparam("dmq", "multi_notify", 1)
modparam("dmq", "ping_interval", 10)
modparam("dmq", "num_workers",4)
modparam("htable", "enable_dmq", 1)
modparam("htable", "dmq_init_sync", 1)
# ----- dialog params -----
modparam("dialog", "dlg_flag", FLD_DLG)
modparam("dialog", "dlg_match_mode", 1)
modparam("dialog", "db_url", DBURL_RW)
modparam("dialog", "db_mode", 0)
modparam("dialog", "enable_dmq", 1)
modparam("dialog", "db_update_period", 10)
modparam("dialog", "h_id_start", H_ID_START)
modparam("dialog", "h_id_step", H_ID_STEP)
Memory:
Shared: 4096
Private: 512
kamailio -v
version: kamailio 5.5.3 (x86_64/linux) 473cef
flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS,
DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, Q_MALLOC,
F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT,
USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLOCKLIST,
HAVE_RESOLV_RES, TLS_PTHREAD_MUTEX_SHARED
ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144, MAX_URI_SIZE
1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
id: 473cef
compiled on 09:34:57 Dec 20 2021 with gcc 10.2.1
In attach is the trap collected after the issue happens.
Any more logs or configurations that can help identify or solve the
issue?
Thanks for the help,
Regards,
Tiago
__________________________________________________________
Kamailio - Users Mailing List - Non Commercial Discussions
* sr-users(a)lists.kamailio.org
Important: keep the mailing list in the recipients, do not reply only to
the sender!
Edit mailing list options or unsubscribe:
* https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
__________________________________________________________
Kamailio - Users Mailing List - Non Commercial Discussions
* sr-users(a)lists.kamailio.org
Important: keep the mailing list in the recipients, do not reply only to
the sender!
Edit mailing list options or unsubscribe:
* https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
7:41 p.m.
The CPU is around 30% during the test, is always really stable and memory
doesn't increase much.
I'm using children=8.
Sergio Charrua <sergio.charrua(a)voip.pt> escreveu no dia quarta, 22/12/2021
à(s) 16:27:
any statistics regarding CPU use? how many threads?
*Sérgio Charrua*
*www.voip.pt <http://www.voip.pt/>*
Tel.: +351 <callto:+351+91+104+12+66>21 130 71 77
Email : *sergio.charrua(a)voip.pt <sergio.charrua(a)voip.pt>*
This message and any files or documents attached are strictly confidential
or otherwise legally protected.
It is intended only for the individual or entity named. If you are not the
named addressee or have received this email in error, please inform the
sender immediately, delete it from your system and do not copy or disclose
it or its contents or use it for any purpose. Please also note that
transmission cannot be guaranteed to be secure or error-free.
On Wed, Dec 22, 2021 at 3:57 PM pwerspire <pwerspire(a)gmail.com> wrote:
Hi Sergio,
Thanks for your reply.
This is a google cloud VM with 8 vCPU, I'm using 100 CPS but the number
of simultaneous calls is stable at around 1071 at any given time.
Also, I can always see INVITES going inside the server but Kamailio
stops answering them, even if you stop sending calls for a long time, and
then send a single call, Kamailio will not reply anymore to the Invites.
When Kamailio stops replying to INVITES, If I do for example a "kamcmd
dispatcher.list" I have the correct output, but if I do a "kamcmd
dlg.list"
it will not provide any output and not give back the prompt( before it
would just answer that the reply was too big).
I will need to use "control + c" to get the prompt of the shell back and
after that, all kamcmd commands stop returning any output or giving back
the prompt.
If the Kamailio is then restarted, everything starts working correctly
again. Also, another note is that this seems only to happen when DMQ is in
use and replicating to the fallback Kamailio.
Regards,
Tiago
Sergio Charrua <sergio.charrua(a)voip.pt> escreveu no dia terça,
21/12/2021 à(s) 13:38:
__________________________________________________________
Kamailio - Users Mailing List - Non Commercial Discussions
* sr-users(a)lists.kamailio.org
Important: keep the mailing list in the recipients, do not reply only to
the sender!
Edit mailing list options or unsubscribe:
* https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
What is the call length of each call? How many
NICs you have on that
server? what networking switches are you using? all these variables have
huge impacts on the call flow!
Let's say you make 100CPS and the call length is 10 minutes.
At 10 seconds you will have 1.000 ongoing calls using g711 (64kbps). You
will have around 64Mbps bandwidth being used.
At 1 minute you get 6.000 ongoing calls and will use 390Mbps bandwidth!
Near the 10th minute you will be using almost 4Gbps bandwidth and 60.000
ongoing calls!!
Maybe the issue is not within Kamailio and/or the media server, but
within your network capacity ...
On my side, on a 4 vCPU and 4GB Ram, Kamailio 5.2 gets me around 400cps
with calls up to 30seconds length, and RTPEngine goes OK too. I am sure I
could do more, but I don't bother much, this is more than enough!
But I have an enormous amount of bandwidth and 3 NICs on the server.
NICs are the key . Remember that Kamailio only processes SIP messages.
At 5 minutes, using the above call details, you've already sent 30.000
SIP INVITES minimum and many more SIP messages during that time. You need a
very good network interface to do that kind of job!
Add 30.000 channels / RTP ports to your media server and you get a whole
lot of data going on in your network. I bet even the UTP5 cables won't
handle the heat =:o)
Also, check the RTP port range your media server was configured with.
On Asterisk, usually ports 10.000 to 20.000 are set, which is about
10.000 calls. In the above example, you would stop receiving calls at
2minutes because Asterisk has all RTP ports busy/in use!
I don't know about other media servers, but I would bet this is pretty
much the same...
Hope this helps!
*Sérgio Charrua*
*www.voip.pt <http://www.voip.pt/>*
Tel.: +351 <callto:+351+91+104+12+66>21 130 71 77
Email : *sergio.charrua(a)voip.pt <sergio.charrua(a)voip.pt>*
This message and any files or documents attached are strictly
confidential or otherwise legally protected.
It is intended only for the individual or entity named. If you are not
the named addressee or have received this email in error, please inform the
sender immediately, delete it from your system and do not copy or disclose
it or its contents or use it for any purpose. Please also note that
transmission cannot be guaranteed to be secure or error-free.
On Tue, Dec 21, 2021 at 11:54 AM pwerspire <pwerspire(a)gmail.com> wrote:
__________________________________________________________
Kamailio - Users Mailing List - Non Commercial Discussions
* sr-users(a)lists.kamailio.org
Important: keep the mailing list in the recipients, do not reply only to
the sender!
Edit mailing list options or unsubscribe:
* https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Hi,
I have been doing some load testing with Kamailio and having some
issues.
I have been trying with 100 CPS and at the beginning, everything is
working well but after some time (it can be 10 minutes or for example 40
minutes, is just random but normally happens more in the first 10 minutes)
Kamailio stop replying to all SIP messages but still processing HTTP
requests, some commands, etc.
If I use "kamcmd dlg.list" no output happens and after that, all
kamcmd commands just keep loading with no output.
Kamailio is sharing Dialogs and htables using DMQ with other Kamailio
that is the failover, if I shut down the failover one (so no DMQ
replication), the test works well with the 100 CPS.
Also tried with modparam("dmq", "worker_usleep", 1000) but the
behavior
is the same, it will stop processing traffic.
This is some of configuration used:
# ----------------- setting module-specific parameters ---------------
modparam("dmq", "server_address",
"sip:INTERNAL_INSTANCE_IP:5060")
modparam("dmq", "notification_address",
"DMQ_NOTIFICATION_ADDRESS")
modparam("dmq", "multi_notify", 1)
modparam("dmq", "ping_interval", 10)
modparam("dmq", "num_workers",4)
modparam("htable", "enable_dmq", 1)
modparam("htable", "dmq_init_sync", 1)
# ----- dialog params -----
modparam("dialog", "dlg_flag", FLD_DLG)
modparam("dialog", "dlg_match_mode", 1)
modparam("dialog", "db_url", DBURL_RW)
modparam("dialog", "db_mode", 0)
modparam("dialog", "enable_dmq", 1)
modparam("dialog", "db_update_period", 10)
modparam("dialog", "h_id_start", H_ID_START)
modparam("dialog", "h_id_step", H_ID_STEP)
Memory:
Shared: 4096
Private: 512
kamailio -v
version: kamailio 5.5.3 (x86_64/linux) 473cef
flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS,
DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, Q_MALLOC,
F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT,
USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLOCKLIST,
HAVE_RESOLV_RES, TLS_PTHREAD_MUTEX_SHARED
ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144, MAX_URI_SIZE
1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
id: 473cef
compiled on 09:34:57 Dec 20 2021 with gcc 10.2.1
In attach is the trap collected after the issue happens.
Any more logs or configurations that can help identify or solve the
issue?
Thanks for the help,
Regards,
Tiago
__________________________________________________________
Kamailio - Users Mailing List - Non Commercial Discussions
* sr-users(a)lists.kamailio.org
Important: keep the mailing list in the recipients, do not reply only
to the sender!
Edit mailing list options or unsubscribe:
* https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
__________________________________________________________
Kamailio - Users Mailing List - Non Commercial Discussions
* sr-users(a)lists.kamailio.org
Important: keep the mailing list in the recipients, do not reply only to
the sender!
Edit mailing list options or unsubscribe:
* https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
23 Dec
23 Dec
6:14 a.m.
Check with htop if you don't have any kamailio procs at 100% CPU (due
to a deadlock).
Do you see any errors in the logs?
-ovidiu
On Wed, Dec 22, 2021 at 11:44 AM pwerspire <pwerspire(a)gmail.com> wrote:
The CPU is around 30% during the test, is always really stable and memory doesn't
increase much.
I'm using children=8.
Sergio Charrua <sergio.charrua(a)voip.pt> escreveu no dia quarta, 22/12/2021 à(s)
16:27:
--
VoIP Embedded, Inc.
http://www.voipembedded.com
any statistics regarding CPU use? how many threads?
Sérgio Charrua
www.voip.pt
Tel.: +351 21 130 71 77
Email : sergio.charrua(a)voip.pt
This message and any files or documents attached are strictly confidential or otherwise
legally protected.
It is intended only for the individual or entity named. If you are not the named
addressee or have received this email in error, please inform the sender immediately,
delete it from your system and do not copy or disclose it or its contents or use it for
any purpose. Please also note that transmission cannot be guaranteed to be secure or
error-free.
On Wed, Dec 22, 2021 at 3:57 PM pwerspire <pwerspire(a)gmail.com> wrote:
__________________________________________________________
Kamailio - Users Mailing List - Non Commercial Discussions
* sr-users(a)lists.kamailio.org
Important: keep the mailing list in the recipients, do not reply only to the sender!
Edit mailing list options or unsubscribe:
* https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Hi Sergio,
Thanks for your reply.
This is a google cloud VM with 8 vCPU, I'm using 100 CPS but the number of
simultaneous calls is stable at around 1071 at any given time.
Also, I can always see INVITES going inside the server but Kamailio stops answering them,
even if you stop sending calls for a long time, and then send a single call, Kamailio will
not reply anymore to the Invites.
When Kamailio stops replying to INVITES, If I do for example a "kamcmd
dispatcher.list" I have the correct output, but if I do a "kamcmd dlg.list"
it will not provide any output and not give back the prompt( before it would just answer
that the reply was too big).
I will need to use "control + c" to get the prompt of the shell back and after
that, all kamcmd commands stop returning any output or giving back the prompt.
If the Kamailio is then restarted, everything starts working correctly again. Also,
another note is that this seems only to happen when DMQ is in use and replicating to the
fallback Kamailio.
Regards,
Tiago
Sergio Charrua <sergio.charrua(a)voip.pt> escreveu no dia terça, 21/12/2021 à(s)
13:38:
__________________________________________________________
Kamailio - Users Mailing List - Non Commercial Discussions
* sr-users(a)lists.kamailio.org
Important: keep the mailing list in the recipients, do not reply only to the sender!
Edit mailing list options or unsubscribe:
* https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
What is the call length of each call? How many NICs you have on that server? what
networking switches are you using? all these variables have huge impacts on the call
flow!
Let's say you make 100CPS and the call length is 10 minutes.
At 10 seconds you will have 1.000 ongoing calls using g711 (64kbps). You will have around
64Mbps bandwidth being used.
At 1 minute you get 6.000 ongoing calls and will use 390Mbps bandwidth!
Near the 10th minute you will be using almost 4Gbps bandwidth and 60.000 ongoing calls!!
Maybe the issue is not within Kamailio and/or the media server, but within your network
capacity ...
On my side, on a 4 vCPU and 4GB Ram, Kamailio 5.2 gets me around 400cps with calls up to
30seconds length, and RTPEngine goes OK too. I am sure I could do more, but I don't
bother much, this is more than enough!
But I have an enormous amount of bandwidth and 3 NICs on the server. NICs are the key .
Remember that Kamailio only processes SIP messages.
At 5 minutes, using the above call details, you've already sent 30.000 SIP INVITES
minimum and many more SIP messages during that time. You need a very good network
interface to do that kind of job!
Add 30.000 channels / RTP ports to your media server and you get a whole lot of data
going on in your network. I bet even the UTP5 cables won't handle the heat =:o)
Also, check the RTP port range your media server was configured with.
On Asterisk, usually ports 10.000 to 20.000 are set, which is about 10.000 calls. In the
above example, you would stop receiving calls at 2minutes because Asterisk has all RTP
ports busy/in use!
I don't know about other media servers, but I would bet this is pretty much the
same...
Hope this helps!
Sérgio Charrua
www.voip.pt
Tel.: +351 21 130 71 77
Email : sergio.charrua(a)voip.pt
This message and any files or documents attached are strictly confidential or otherwise
legally protected.
It is intended only for the individual or entity named. If you are not the named
addressee or have received this email in error, please inform the sender immediately,
delete it from your system and do not copy or disclose it or its contents or use it for
any purpose. Please also note that transmission cannot be guaranteed to be secure or
error-free.
On Tue, Dec 21, 2021 at 11:54 AM pwerspire <pwerspire(a)gmail.com> wrote:
>
> Hi,
>
> I have been doing some load testing with Kamailio and having some issues.
>
> I have been trying with 100 CPS and at the beginning, everything is working well but
after some time (it can be 10 minutes or for example 40 minutes, is just random but
normally happens more in the first 10 minutes) Kamailio stop replying to all SIP messages
but still processing HTTP requests, some commands, etc.
>
> If I use "kamcmd dlg.list" no output happens and after that, all kamcmd
commands just keep loading with no output.
>
> Kamailio is sharing Dialogs and htables using DMQ with other Kamailio that is the
failover, if I shut down the failover one (so no DMQ replication), the test works well
with the 100 CPS.
>
> Also tried with modparam("dmq", "worker_usleep", 1000) but the
behavior is the same, it will stop processing traffic.
>
> This is some of configuration used:
>
> # ----------------- setting module-specific parameters ---------------
> modparam("dmq", "server_address",
"sip:INTERNAL_INSTANCE_IP:5060")
> modparam("dmq", "notification_address",
"DMQ_NOTIFICATION_ADDRESS")
>
> modparam("dmq", "multi_notify", 1)
> modparam("dmq", "ping_interval", 10)
> modparam("dmq", "num_workers",4)
>
>
> modparam("htable", "enable_dmq", 1)
> modparam("htable", "dmq_init_sync", 1)
>
>
> # ----- dialog params -----
> modparam("dialog", "dlg_flag", FLD_DLG)
> modparam("dialog", "dlg_match_mode", 1)
> modparam("dialog", "db_url", DBURL_RW)
> modparam("dialog", "db_mode", 0)
> modparam("dialog", "enable_dmq", 1)
> modparam("dialog", "db_update_period", 10)
> modparam("dialog", "h_id_start", H_ID_START)
> modparam("dialog", "h_id_step", H_ID_STEP)
>
> Memory:
>
> Shared: 4096
> Private: 512
>
> kamailio -v
> version: kamailio 5.5.3 (x86_64/linux) 473cef
> flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE,
USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC,
DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER,
USE_NAPTR, USE_DST_BLOCKLIST, HAVE_RESOLV_RES, TLS_PTHREAD_MUTEX_SHARED
> ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144, MAX_URI_SIZE 1024, BUF_SIZE
65535, DEFAULT PKG_SIZE 8MB
> poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
> id: 473cef
> compiled on 09:34:57 Dec 20 2021 with gcc 10.2.1
>
>
> In attach is the trap collected after the issue happens.
> Any more logs or configurations that can help identify or solve the issue?
>
> Thanks for the help,
> Regards,
> Tiago
> __________________________________________________________
> Kamailio - Users Mailing List - Non Commercial Discussions
> * sr-users(a)lists.kamailio.org
> Important: keep the mailing list in the recipients, do not reply only to the sender!
> Edit mailing list options or unsubscribe:
> * https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
__________________________________________________________
Kamailio - Users Mailing List - Non Commercial Discussions
* sr-users(a)lists.kamailio.org
Important: keep the mailing list in the recipients, do not reply only to the sender!
Edit mailing list options or unsubscribe:
* https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
__________________________________________________________
Kamailio - Users Mailing List - Non Commercial Discussions
* sr-users(a)lists.kamailio.org
Important: keep the mailing list in the recipients, do not reply only to the sender!
Edit mailing list options or unsubscribe:
* https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
27 Dec
27 Dec
3:12 p.m.
Hi,
I just repeated the test and there is no Kamailio process at 100%, they are
low all the time even when the issue starts to happen.
I went through the logs and couldn't find any error, unfortunately, with
100 CPS I can't put it on debug mode but only in info.
Regards,
Tiago
Ovidiu Sas <osas(a)voipembedded.com> escreveu no dia quinta, 23/12/2021 à(s)
03:15:
Check with htop if you don't have any kamailio
procs at 100% CPU (due
to a deadlock).
Do you see any errors in the logs?
-ovidiu
On Wed, Dec 22, 2021 at 11:44 AM pwerspire <pwerspire(a)gmail.com> wrote:
the
sender!
The CPU is around 30% during the test, is always really stable and
memory
doesn't increase much.
I'm using children=8.
Sergio Charrua <sergio.charrua(a)voip.pt> escreveu no dia quarta,
22/12/2021
à(s) 16:27:
>
> any statistics regarding CPU use? how many threads?
>
> Sérgio Charrua
>
>
> www.voip.pt
> Tel.: +351 21 130 71 77
>
> Email : sergio.charrua(a)voip.pt
>
> This message and any files or documents attached are strictly
confidential or
otherwise legally protected.
>
> It is intended only for the individual or entity named. If you are not
the
named addressee or have received this email in error, please inform the
sender immediately, delete it from your system and do not copy or disclose
it or its contents or use it for any purpose. Please also note that
transmission cannot be guaranteed to be secure or error-free.
>
>
>
>
>
>
>
>
>
> On Wed, Dec 22, 2021 at 3:57 PM pwerspire <pwerspire(a)gmail.com> wrote:
>>
>> Hi Sergio,
>>
>> Thanks for your reply.
>>
>> This is a google cloud VM with 8 vCPU, I'm using 100 CPS but the
number of simultaneous calls is stable at around 1071 at any given time.
>>
>> Also, I can always see INVITES going inside the server but Kamailio
stops
answering them, even if you stop sending calls for a long time, and
then send a single call, Kamailio will not reply anymore to the Invites.
>>
>> When Kamailio stops replying to INVITES, If I do for example a "kamcmd
dispatcher.list" I have the correct output, but if I do a "kamcmd
dlg.list"
it will not provide any output and not give back the prompt( before it
would just answer that the reply was too big).
>> I will need to use "control +
c" to get the prompt of the shell back
and after that, all kamcmd commands
stop returning any output or giving
back the prompt.
>>
>> If the Kamailio is then restarted, everything starts working correctly
again. Also, another note is that this seems only to happen when DMQ is in
use and replicating to the fallback Kamailio.
>>
>> Regards,
>> Tiago
>>
>>
>> Sergio Charrua <sergio.charrua(a)voip.pt> escreveu no dia terça,
21/12/2021 à(s) 13:38:
>>>
>>> What is the call length of each call? How many NICs you have on that
server? what networking switches are you using? all these variables have
huge impacts on the call flow!
>>>
>>> Let's say you make 100CPS and the call length is 10 minutes.
>>> At 10 seconds you will have 1.000 ongoing calls using g711 (64kbps).
You will have around 64Mbps bandwidth being used.
>>> At 1 minute you get 6.000 ongoing
calls and will use 390Mbps
bandwidth!
>>> Near the 10th minute you will be
using almost 4Gbps bandwidth and
60.000 ongoing calls!!
>>>
>>> Maybe the issue is not within Kamailio and/or the media server, but
within your network capacity ...
>>>
>>> On my side, on a 4 vCPU and 4GB Ram, Kamailio 5.2 gets me around
400cps with calls up to 30seconds length, and RTPEngine goes OK too. I am
sure I could do more, but I don't bother much, this is more than enough!
>>> But I have an enormous amount of
bandwidth and 3 NICs on the server.
NICs are the key . Remember that Kamailio only
processes SIP messages.
>>> At 5 minutes, using the above call
details, you've already sent
30.000 SIP INVITES minimum and many more SIP
messages during that time. You
need a very good network interface to do that kind of job!
>>> Add 30.000 channels / RTP ports to
your media server and you get a
whole lot of data going on in your network. I bet
even the UTP5 cables
won't handle the heat =:o)
>>>
>>> Also, check the RTP port range your media server was configured with.
>>> On Asterisk, usually ports 10.000 to 20.000 are set, which is about
10.000 calls. In the above example, you would stop receiving calls at
2minutes because Asterisk has all RTP ports busy/in use!
>>> I don't know about other media
servers, but I would bet this is
pretty much the same...
>>>
>>> Hope this helps!
>>>
>>>
>>> Sérgio Charrua
>>>
>>>
>>> www.voip.pt
>>> Tel.: +351 21 130 71 77
>>>
>>> Email : sergio.charrua(a)voip.pt
>>>
>>> This message and any files or documents attached are strictly
confidential or otherwise legally protected.
>>>
>>> It is intended only for the individual or entity named. If you are
not
the named addressee or have received this email in error, please inform
the sender immediately, delete it from your system and do not copy or
disclose it or its contents or use it for any purpose. Please also note
that transmission cannot be guaranteed to be secure or error-free.
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Tue, Dec 21, 2021 at 11:54 AM pwerspire <pwerspire(a)gmail.com>
wrote:
>>>>
>>>> Hi,
>>>>
>>>> I have been doing some load testing with Kamailio and having some
issues.
>>>>
>>>> I have been trying with 100 CPS and at the beginning, everything is
working well but after some time (it can be 10 minutes or for example 40
minutes, is just random but normally happens more in the first 10 minutes)
Kamailio stop replying to all SIP messages but still processing HTTP
requests, some commands, etc.
>>>>
>>>> If I use "kamcmd dlg.list" no output happens and after that,
all
kamcmd commands just keep loading with no output.
>>>>
>>>> Kamailio is sharing Dialogs and htables using DMQ with other
Kamailio that is the failover, if I shut down the failover one (so no DMQ
replication), the test works well with the 100 CPS.
>>>>
>>>> Also tried with modparam("dmq", "worker_usleep",
1000) but the
behavior is the same, it will stop processing traffic.
>>>>
>>>> This is some of configuration used:
>>>>
>>>> # ----------------- setting module-specific parameters
---------------
>>>> modparam("dmq",
"server_address", "sip:INTERNAL_INSTANCE_IP:5060")
>>>> modparam("dmq", "notification_address",
"DMQ_NOTIFICATION_ADDRESS")
>>>>
>>>> modparam("dmq", "multi_notify", 1)
>>>> modparam("dmq", "ping_interval", 10)
>>>> modparam("dmq", "num_workers",4)
>>>>
>>>>
>>>> modparam("htable", "enable_dmq", 1)
>>>> modparam("htable", "dmq_init_sync", 1)
>>>>
>>>>
>>>> # ----- dialog params -----
>>>> modparam("dialog", "dlg_flag", FLD_DLG)
>>>> modparam("dialog", "dlg_match_mode", 1)
>>>> modparam("dialog", "db_url", DBURL_RW)
>>>> modparam("dialog", "db_mode", 0)
>>>> modparam("dialog", "enable_dmq", 1)
>>>> modparam("dialog", "db_update_period", 10)
>>>> modparam("dialog", "h_id_start", H_ID_START)
>>>> modparam("dialog", "h_id_step", H_ID_STEP)
>>>>
>>>> Memory:
>>>>
>>>> Shared: 4096
>>>> Private: 512
>>>>
>>>> kamailio -v
>>>> version: kamailio 5.5.3 (x86_64/linux) 473cef
>>>> flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS,
DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, Q_MALLOC,
F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT,
USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLOCKLIST,
HAVE_RESOLV_RES, TLS_PTHREAD_MUTEX_SHARED
>>>> ADAPTIVE_WAIT_LOOPS 1024,
MAX_RECV_BUFFER_SIZE 262144, MAX_URI_SIZE
1024, BUF_SIZE 65535, DEFAULT PKG_SIZE
8MB
>>>> poll method support: poll,
epoll_lt, epoll_et, sigio_rt, select.
>>>> id: 473cef
>>>> compiled on 09:34:57 Dec 20 2021 with gcc 10.2.1
>>>>
>>>>
>>>> In attach is the trap collected after the issue happens.
>>>> Any more logs or configurations that can help identify or solve the
issue?
>>>>
>>>> Thanks for the help,
>>>> Regards,
>>>> Tiago
>>>> __________________________________________________________
>>>> Kamailio - Users Mailing List - Non Commercial Discussions
>>>> * sr-users(a)lists.kamailio.org
>>>> Important: keep the mailing list in the recipients, do not reply
only to the sender!
>>>> Edit mailing list options or
unsubscribe:
>>>> * https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
>>>
>>> __________________________________________________________
>>> Kamailio - Users Mailing List - Non Commercial Discussions
>>> * sr-users(a)lists.kamailio.org
>>> Important: keep the mailing list in the recipients, do not reply only
to the sender!
>>> Edit mailing list options or
unsubscribe:
>>> * https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
>>
>> __________________________________________________________
>> Kamailio - Users Mailing List - Non Commercial Discussions
>> * sr-users(a)lists.kamailio.org
>> Important: keep the mailing list in the recipients, do not reply only
to
the sender!
>> Edit mailing list options or
unsubscribe:
>> * https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
>
> __________________________________________________________
> Kamailio - Users Mailing List - Non Commercial Discussions
> * sr-users(a)lists.kamailio.org
> Important: keep the mailing list in the recipients, do not reply only
to the
sender!
Edit
mailing list options or unsubscribe:
* https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
__________________________________________________________
Kamailio - Users Mailing List - Non Commercial Discussions
* sr-users(a)lists.kamailio.org
Important: keep the mailing list in the recipients, do not reply only to Edit mailing list options or unsubscribe:
* https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
--
VoIP Embedded, Inc.
http://www.voipembedded.com
__________________________________________________________
Kamailio - Users Mailing List - Non Commercial Discussions
* sr-users(a)lists.kamailio.org
Important: keep the mailing list in the recipients, do not reply only to
the sender!
Edit mailing list options or unsubscribe:
* https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
5:09 p.m.
ok, let's go 1 step at a time.
stop all firewalls, if any.
install sngrep SIP monitoring tool on your server
start sngrep and then (re)start your test script until you see Kamailio
stoping accepting calls
check if sngrep receives SIP Invites
Also, please share your kamailio.cfg . It might help too.
obrigado,
*Sérgio Charrua*
*www.voip.pt <http://www.voip.pt/>*
Tel.: +351 <callto:+351+91+104+12+66>21 130 71 77
Email : *sergio.charrua(a)voip.pt <sergio.charrua(a)voip.pt>*
This message and any files or documents attached are strictly confidential
or otherwise legally protected.
It is intended only for the individual or entity named. If you are not the
named addressee or have received this email in error, please inform the
sender immediately, delete it from your system and do not copy or disclose
it or its contents or use it for any purpose. Please also note that
transmission cannot be guaranteed to be secure or error-free.
On Mon, Dec 27, 2021 at 12:23 PM pwerspire <pwerspire(a)gmail.com> wrote:
Hi,
I just repeated the test and there is no Kamailio process at 100%, they
are low all the time even when the issue starts to happen.
I went through the logs and couldn't find any error, unfortunately, with
100 CPS I can't put it on debug mode but only in info.
Regards,
Tiago
Ovidiu Sas <osas(a)voipembedded.com> escreveu no dia quinta, 23/12/2021
à(s) 03:15:
Check with htop if you don't have any
kamailio procs at 100% CPU (due
to a deadlock).
Do you see any errors in the logs?
-ovidiu
On Wed, Dec 22, 2021 at 11:44 AM pwerspire <pwerspire(a)gmail.com> wrote:
to the
sender!
__________________________________________________________
Kamailio - Users Mailing List - Non Commercial Discussions
* sr-users(a)lists.kamailio.org
Important: keep the mailing list in the recipients, do not reply only to
the sender!
Edit mailing list options or unsubscribe:
* https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
The CPU is around 30% during the test, is always really stable and
memory
doesn't increase much.
I'm using children=8.
Sergio Charrua <sergio.charrua(a)voip.pt> escreveu no dia quarta,
22/12/2021
à(s) 16:27:
>
> any statistics regarding CPU use? how many threads?
>
> Sérgio Charrua
>
>
> www.voip.pt
> Tel.: +351 21 130 71 77
>
> Email : sergio.charrua(a)voip.pt
>
> This message and any files or documents attached are strictly
confidential or
otherwise legally protected.
>
> It is intended only for the individual or entity named. If you are not
the
named addressee or have received this email in error, please inform the
sender immediately, delete it from your system and do not copy or disclose
it or its contents or use it for any purpose. Please also note that
transmission cannot be guaranteed to be secure or error-free.
>
>
>
>
>
>
>
>
>
> On Wed, Dec 22, 2021 at 3:57 PM pwerspire <pwerspire(a)gmail.com> wrote:
>>
>> Hi Sergio,
>>
>> Thanks for your reply.
>>
>> This is a google cloud VM with 8 vCPU, I'm using 100 CPS but the
number of simultaneous calls is stable at around 1071 at any given time.
>>
>> Also, I can always see INVITES going inside the server but Kamailio
stops
answering them, even if you stop sending calls for a long time, and
then send a single call, Kamailio will not reply anymore to the Invites.
>>
>> When Kamailio stops replying to INVITES, If I do for example a
"kamcmd dispatcher.list" I have the correct output, but if I do a "kamcmd
dlg.list" it will not provide any output and not give back the prompt(
before it would just answer that the reply was too big).
>> I will need to use "control +
c" to get the prompt of the shell back
and after that, all kamcmd commands
stop returning any output or giving
back the prompt.
>>
>> If the Kamailio is then restarted, everything starts working
correctly
again. Also, another note is that this seems only to happen when
DMQ is in use and replicating to the fallback Kamailio.
>>
>> Regards,
>> Tiago
>>
>>
>> Sergio Charrua <sergio.charrua(a)voip.pt> escreveu no dia terça,
21/12/2021 à(s) 13:38:
>>>
>>> What is the call length of each call? How many NICs you have on that
server? what networking switches are you using? all these variables have
huge impacts on the call flow!
>>>
>>> Let's say you make 100CPS and the call length is 10 minutes.
>>> At 10 seconds you will have 1.000 ongoing calls using g711 (64kbps).
You will have around 64Mbps bandwidth being used.
>>> At 1 minute you get 6.000 ongoing
calls and will use 390Mbps
bandwidth!
>>> Near the 10th minute you will be
using almost 4Gbps bandwidth and
60.000 ongoing calls!!
>>>
>>> Maybe the issue is not within Kamailio and/or the media server, but
within your network capacity ...
>>>
>>> On my side, on a 4 vCPU and 4GB Ram, Kamailio 5.2 gets me around
400cps with calls up to 30seconds length, and RTPEngine goes OK too. I am
sure I could do more, but I don't bother much, this is more than enough!
>>> But I have an enormous amount of
bandwidth and 3 NICs on the server.
NICs are the key . Remember that Kamailio only
processes SIP messages.
>>> At 5 minutes, using the above call
details, you've already sent
30.000 SIP INVITES minimum and many more SIP
messages during that time. You
need a very good network interface to do that kind of job!
>>> Add 30.000 channels / RTP ports to
your media server and you get a
whole lot of data going on in your network. I bet
even the UTP5 cables
won't handle the heat =:o)
>>>
>>> Also, check the RTP port range your media server was configured with.
>>> On Asterisk, usually ports 10.000 to 20.000 are set, which is about
10.000 calls. In the above example, you would stop receiving calls at
2minutes because Asterisk has all RTP ports busy/in use!
>>> I don't know about other media
servers, but I would bet this is
pretty much the same...
>>>
>>> Hope this helps!
>>>
>>>
>>> Sérgio Charrua
>>>
>>>
>>> www.voip.pt
>>> Tel.: +351 21 130 71 77
>>>
>>> Email : sergio.charrua(a)voip.pt
>>>
>>> This message and any files or documents attached are strictly
confidential or otherwise legally protected.
>>>
>>> It is intended only for the individual or entity named. If you are
not
the named addressee or have received this email in error, please inform
the sender immediately, delete it from your system and do not copy or
disclose it or its contents or use it for any purpose. Please also note
that transmission cannot be guaranteed to be secure or error-free.
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Tue, Dec 21, 2021 at 11:54 AM pwerspire <pwerspire(a)gmail.com>
wrote:
>>>>
>>>> Hi,
>>>>
>>>> I have been doing some load testing with Kamailio and having some
issues.
>>>>
>>>> I have been trying with 100 CPS and at the beginning, everything is
working well but after some time (it can be 10 minutes or for example 40
minutes, is just random but normally happens more in the first 10 minutes)
Kamailio stop replying to all SIP messages but still processing HTTP
requests, some commands, etc.
>>>>
>>>> If I use "kamcmd dlg.list" no output happens and after that,
all
kamcmd commands just keep loading with no output.
>>>>
>>>> Kamailio is sharing Dialogs and htables using DMQ with other
Kamailio that is the failover, if I shut down the failover one (so no DMQ
replication), the test works well with the 100 CPS.
>>>>
>>>> Also tried with modparam("dmq", "worker_usleep",
1000) but the
behavior is the same, it will stop processing traffic.
>>>>
>>>> This is some of configuration used:
>>>>
>>>> # ----------------- setting module-specific parameters
---------------
>>>> modparam("dmq",
"server_address", "sip:INTERNAL_INSTANCE_IP:5060")
>>>> modparam("dmq", "notification_address",
"DMQ_NOTIFICATION_ADDRESS")
>>>>
>>>> modparam("dmq", "multi_notify", 1)
>>>> modparam("dmq", "ping_interval", 10)
>>>> modparam("dmq", "num_workers",4)
>>>>
>>>>
>>>> modparam("htable", "enable_dmq", 1)
>>>> modparam("htable", "dmq_init_sync", 1)
>>>>
>>>>
>>>> # ----- dialog params -----
>>>> modparam("dialog", "dlg_flag", FLD_DLG)
>>>> modparam("dialog", "dlg_match_mode", 1)
>>>> modparam("dialog", "db_url", DBURL_RW)
>>>> modparam("dialog", "db_mode", 0)
>>>> modparam("dialog", "enable_dmq", 1)
>>>> modparam("dialog", "db_update_period", 10)
>>>> modparam("dialog", "h_id_start", H_ID_START)
>>>> modparam("dialog", "h_id_step", H_ID_STEP)
>>>>
>>>> Memory:
>>>>
>>>> Shared: 4096
>>>> Private: 512
>>>>
>>>> kamailio -v
>>>> version: kamailio 5.5.3 (x86_64/linux) 473cef
>>>> flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS,
DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, Q_MALLOC,
F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT,
USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLOCKLIST,
HAVE_RESOLV_RES, TLS_PTHREAD_MUTEX_SHARED
>>>> ADAPTIVE_WAIT_LOOPS 1024,
MAX_RECV_BUFFER_SIZE 262144, MAX_URI_SIZE
1024, BUF_SIZE 65535, DEFAULT PKG_SIZE
8MB
>>>> poll method support: poll,
epoll_lt, epoll_et, sigio_rt, select.
>>>> id: 473cef
>>>> compiled on 09:34:57 Dec 20 2021 with gcc 10.2.1
>>>>
>>>>
>>>> In attach is the trap collected after the issue happens.
>>>> Any more logs or configurations that can help identify or solve the
issue?
>>>>
>>>> Thanks for the help,
>>>> Regards,
>>>> Tiago
>>>> __________________________________________________________
>>>> Kamailio - Users Mailing List - Non Commercial Discussions
>>>> * sr-users(a)lists.kamailio.org
>>>> Important: keep the mailing list in the recipients, do not reply
only to the sender!
>>>> Edit mailing list options or
unsubscribe:
>>>> * https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
>>>
>>> __________________________________________________________
>>> Kamailio - Users Mailing List - Non Commercial Discussions
>>> * sr-users(a)lists.kamailio.org
>>> Important: keep the mailing list in the recipients, do not reply
only
to the sender!
>>> Edit mailing list options or
unsubscribe:
>>> * https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
>>
>> __________________________________________________________
>> Kamailio - Users Mailing List - Non Commercial Discussions
>> * sr-users(a)lists.kamailio.org
>> Important: keep the mailing list in the recipients, do not reply only
to
the sender!
>> Edit mailing list options or
unsubscribe:
>> * https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
>
> __________________________________________________________
> Kamailio - Users Mailing List - Non Commercial Discussions
> * sr-users(a)lists.kamailio.org
> Important: keep the mailing list in the recipients, do not reply only
to the
sender!
Edit
mailing list options or unsubscribe:
* https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
__________________________________________________________
Kamailio - Users Mailing List - Non Commercial Discussions
* sr-users(a)lists.kamailio.org
Important: keep the mailing list in the recipients, do not reply only Edit mailing list options or unsubscribe:
* https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
--
VoIP Embedded, Inc.
http://www.voipembedded.com
__________________________________________________________
Kamailio - Users Mailing List - Non Commercial Discussions
* sr-users(a)lists.kamailio.org
Important: keep the mailing list in the recipients, do not reply only to
the sender!
Edit mailing list options or unsubscribe:
* https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
7:44 p.m.
Already checked with sngrep and the INVITES are arriving.
I don't believe that is something related to the server because is not
normal that when the issue happens, dialog commands like "kamcmd dlg.list"
stop returning any prompt. During the time that the issue is not happening
yet, the command just returns that is too big, then when Kamailio stops
processing the Sip messages, the command will not work anymore.
Unfortunately, I cannot share the kamailio.cfg file.
Thanks,
Tiago
Sergio Charrua <sergio.charrua(a)voip.pt> escreveu no dia segunda, 27/12/2021
à(s) 14:12:
ok, let's go 1 step at a time.
stop all firewalls, if any.
install sngrep SIP monitoring tool on your server
start sngrep and then (re)start your test script until you see Kamailio
stoping accepting calls
check if sngrep receives SIP Invites
Also, please share your kamailio.cfg . It might help too.
obrigado,
*Sérgio Charrua*
*www.voip.pt <http://www.voip.pt/>*
Tel.: +351 <callto:+351+91+104+12+66>21 130 71 77
Email : *sergio.charrua(a)voip.pt <sergio.charrua(a)voip.pt>*
This message and any files or documents attached are strictly confidential
or otherwise legally protected.
It is intended only for the individual or entity named. If you are not the
named addressee or have received this email in error, please inform the
sender immediately, delete it from your system and do not copy or disclose
it or its contents or use it for any purpose. Please also note that
transmission cannot be guaranteed to be secure or error-free.
On Mon, Dec 27, 2021 at 12:23 PM pwerspire <pwerspire(a)gmail.com> wrote:
Hi,
I just repeated the test and there is no Kamailio process at 100%, they
are low all the time even when the issue starts to happen.
I went through the logs and couldn't find any error, unfortunately, with
100 CPS I can't put it on debug mode but only in info.
Regards,
Tiago
Ovidiu Sas <osas(a)voipembedded.com> escreveu no dia quinta, 23/12/2021
à(s) 03:15:
__________________________________________________________
Kamailio - Users Mailing List - Non Commercial Discussions
* sr-users(a)lists.kamailio.org
Important: keep the mailing list in the recipients, do not reply only to
the sender!
Edit mailing list options or unsubscribe:
* https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Check with htop if you don't have any
kamailio procs at 100% CPU (due
to a deadlock).
Do you see any errors in the logs?
-ovidiu
On Wed, Dec 22, 2021 at 11:44 AM pwerspire <pwerspire(a)gmail.com> wrote:
__________________________________________________________
Kamailio - Users Mailing List - Non Commercial Discussions
* sr-users(a)lists.kamailio.org
Important: keep the mailing list in the recipients, do not reply only to
the sender!
Edit mailing list options or unsubscribe:
* https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
The CPU is around 30% during the test, is always really stable and
memory
doesn't increase much.
I'm using children=8.
Sergio Charrua <sergio.charrua(a)voip.pt> escreveu no dia quarta,
22/12/2021
à(s) 16:27:
>
> any statistics regarding CPU use? how many threads?
>
> Sérgio Charrua
>
>
> www.voip.pt
> Tel.: +351 21 130 71 77
>
> Email : sergio.charrua(a)voip.pt
>
> This message and any files or documents attached are strictly
confidential or
otherwise legally protected.
>
> It is intended only for the individual or entity named. If you are
not the
named addressee or have received this email in error, please inform
the sender immediately, delete it from your system and do not copy or
disclose it or its contents or use it for any purpose. Please also note
that transmission cannot be guaranteed to be secure or error-free.
>
>
>
>
>
>
>
>
>
> On Wed, Dec 22, 2021 at 3:57 PM pwerspire <pwerspire(a)gmail.com>
wrote:
>>
>> Hi Sergio,
>>
>> Thanks for your reply.
>>
>> This is a google cloud VM with 8 vCPU, I'm using 100 CPS but the
number of simultaneous calls is stable at around 1071 at any given time.
>>
>> Also, I can always see INVITES going inside the server but Kamailio
stops
answering them, even if you stop sending calls for a long time, and
then send a single call, Kamailio will not reply anymore to the Invites.
>>
>> When Kamailio stops replying to INVITES, If I do for example a
"kamcmd dispatcher.list" I have the correct output, but if I do a "kamcmd
dlg.list" it will not provide any output and not give back the prompt(
before it would just answer that the reply was too big).
>> I will need to use "control +
c" to get the prompt of the shell back
and after that, all kamcmd commands
stop returning any output or giving
back the prompt.
>>
>> If the Kamailio is then restarted, everything starts working
correctly
again. Also, another note is that this seems only to happen when
DMQ is in use and replicating to the fallback Kamailio.
>>
>> Regards,
>> Tiago
>>
>>
>> Sergio Charrua <sergio.charrua(a)voip.pt> escreveu no dia terça,
21/12/2021 à(s) 13:38:
>>>
>>> What is the call length of each call? How many NICs you have on
that
server? what networking switches are you using? all these variables
have huge impacts on the call flow!
>>>
>>> Let's say you make 100CPS and the call length is 10 minutes.
>>> At 10 seconds you will have 1.000 ongoing calls using g711
(64kbps).
You will have around 64Mbps bandwidth being used.
>>> At 1 minute you get 6.000 ongoing
calls and will use 390Mbps
bandwidth!
>>> Near the 10th minute you will be
using almost 4Gbps bandwidth and
60.000 ongoing calls!!
>>>
>>> Maybe the issue is not within Kamailio and/or the media server, but
within your network capacity ...
>>>
>>> On my side, on a 4 vCPU and 4GB Ram, Kamailio 5.2 gets me around
400cps with calls up to 30seconds length, and RTPEngine goes OK too. I am
sure I could do more, but I don't bother much, this is more than enough!
>>> But I have an enormous amount of
bandwidth and 3 NICs on the
server. NICs are the key . Remember that Kamailio only
processes SIP
messages.
>>> At 5 minutes, using the above call
details, you've already sent
30.000 SIP INVITES minimum and many more SIP
messages during that time. You
need a very good network interface to do that kind of job!
>>> Add 30.000 channels / RTP ports to
your media server and you get a
whole lot of data going on in your network. I bet
even the UTP5 cables
won't handle the heat =:o)
>>>
>>> Also, check the RTP port range your media server was configured
with.
>>> On Asterisk, usually ports 10.000 to
20.000 are set, which is about
10.000 calls. In the above example, you would stop
receiving calls at
2minutes because Asterisk has all RTP ports busy/in use!
>>> I don't know about other media
servers, but I would bet this is
pretty much the same...
>>>
>>> Hope this helps!
>>>
>>>
>>> Sérgio Charrua
>>>
>>>
>>> www.voip.pt
>>> Tel.: +351 21 130 71 77
>>>
>>> Email : sergio.charrua(a)voip.pt
>>>
>>> This message and any files or documents attached are strictly
confidential or otherwise legally protected.
>>>
>>> It is intended only for the individual or entity named. If you are
not
the named addressee or have received this email in error, please inform
the sender immediately, delete it from your system and do not copy or
disclose it or its contents or use it for any purpose. Please also note
that transmission cannot be guaranteed to be secure or error-free.
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Tue, Dec 21, 2021 at 11:54 AM pwerspire <pwerspire(a)gmail.com>
wrote:
>>>>
>>>> Hi,
>>>>
>>>> I have been doing some load testing with Kamailio and having some
issues.
>>>>
>>>> I have been trying with 100 CPS and at the beginning, everything
is working well but after some time (it can be 10 minutes or for example 40
minutes, is just random but normally happens more in the first 10 minutes)
Kamailio stop replying to all SIP messages but still processing HTTP
requests, some commands, etc.
>>>>
>>>> If I use "kamcmd dlg.list" no output happens and after that,
all
kamcmd commands just keep loading with no output.
>>>>
>>>> Kamailio is sharing Dialogs and htables using DMQ with other
Kamailio that is the failover, if I shut down the failover one (so no DMQ
replication), the test works well with the 100 CPS.
>>>>
>>>> Also tried with modparam("dmq", "worker_usleep",
1000) but the
behavior is the same, it will stop processing traffic.
>>>>
>>>> This is some of configuration used:
>>>>
>>>> # ----------------- setting module-specific parameters
---------------
>>>> modparam("dmq",
"server_address", "sip:INTERNAL_INSTANCE_IP:5060")
>>>> modparam("dmq", "notification_address",
"DMQ_NOTIFICATION_ADDRESS")
>>>>
>>>> modparam("dmq", "multi_notify", 1)
>>>> modparam("dmq", "ping_interval", 10)
>>>> modparam("dmq", "num_workers",4)
>>>>
>>>>
>>>> modparam("htable", "enable_dmq", 1)
>>>> modparam("htable", "dmq_init_sync", 1)
>>>>
>>>>
>>>> # ----- dialog params -----
>>>> modparam("dialog", "dlg_flag", FLD_DLG)
>>>> modparam("dialog", "dlg_match_mode", 1)
>>>> modparam("dialog", "db_url", DBURL_RW)
>>>> modparam("dialog", "db_mode", 0)
>>>> modparam("dialog", "enable_dmq", 1)
>>>> modparam("dialog", "db_update_period", 10)
>>>> modparam("dialog", "h_id_start", H_ID_START)
>>>> modparam("dialog", "h_id_step", H_ID_STEP)
>>>>
>>>> Memory:
>>>>
>>>> Shared: 4096
>>>> Private: 512
>>>>
>>>> kamailio -v
>>>> version: kamailio 5.5.3 (x86_64/linux) 473cef
>>>> flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS,
DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, Q_MALLOC,
F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT,
USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLOCKLIST,
HAVE_RESOLV_RES, TLS_PTHREAD_MUTEX_SHARED
>>>> ADAPTIVE_WAIT_LOOPS 1024,
MAX_RECV_BUFFER_SIZE 262144,
MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE
8MB
>>>> poll method support: poll,
epoll_lt, epoll_et, sigio_rt, select.
>>>> id: 473cef
>>>> compiled on 09:34:57 Dec 20 2021 with gcc 10.2.1
>>>>
>>>>
>>>> In attach is the trap collected after the issue happens.
>>>> Any more logs or configurations that can help identify or solve
the issue?
>>>>
>>>> Thanks for the help,
>>>> Regards,
>>>> Tiago
>>>> __________________________________________________________
>>>> Kamailio - Users Mailing List - Non Commercial Discussions
>>>> * sr-users(a)lists.kamailio.org
>>>> Important: keep the mailing list in the recipients, do not reply
only to the sender!
>>>> Edit mailing list options or
unsubscribe:
>>>> * https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
>>>
>>> __________________________________________________________
>>> Kamailio - Users Mailing List - Non Commercial Discussions
>>> * sr-users(a)lists.kamailio.org
>>> Important: keep the mailing list in the recipients, do not reply
only
to the sender!
>>> Edit mailing list options or
unsubscribe:
>>> * https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
>>
>> __________________________________________________________
>> Kamailio - Users Mailing List - Non Commercial Discussions
>> * sr-users(a)lists.kamailio.org
>> Important: keep the mailing list in the recipients, do not reply
only to
the sender!
>> Edit mailing list options or
unsubscribe:
>> * https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
>
> __________________________________________________________
> Kamailio - Users Mailing List - Non Commercial Discussions
> * sr-users(a)lists.kamailio.org
> Important: keep the mailing list in the recipients, do not reply only
to the
sender!
> Edit mailing list options or unsubscribe:
> * https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
__________________________________________________________
Kamailio - Users Mailing List - Non Commercial Discussions
* sr-users(a)lists.kamailio.org
Important: keep the mailing list in the recipients, do not reply only
to the
sender!
Edit mailing list options or unsubscribe:
* https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
--
VoIP Embedded, Inc.
http://www.voipembedded.com
__________________________________________________________
Kamailio - Users Mailing List - Non Commercial Discussions
* sr-users(a)lists.kamailio.org
Important: keep the mailing list in the recipients, do not reply only to
the sender!
Edit mailing list options or unsubscribe:
* https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
7:54 p.m.
If you can't share your kamailio config for reasons of corporate policy, etc. that
makes sense. Perhaps you can share "the simplest configuration possible in which the
error occurs"? There shouldn't be any need within the troubleshooting to have
the "secret recipe" of your LCR selections, etc. present. Additionally,
reducing the problem to the simplest configuration in which it can occur is usually an
invaluable tool for problem isolation.
Regards,
Ben Kaufman
From: sr-users <sr-users-bounces(a)lists.kamailio.org> On Behalf Of pwerspire
Sent: Monday, December 27, 2021 10:45 AM
To: Kamailio (SER) - Users Mailing List <sr-users(a)lists.kamailio.org>
Subject: Re: [SR-Users] Kamailio Stop processing SIP traffic
Already checked with sngrep and the INVITES are arriving.
I don't believe that is something related to the server because is not normal that
when the issue happens, dialog commands like "kamcmd dlg.list" stop returning
any prompt. During the time that the issue is not happening yet, the command just returns
that is too big, then when Kamailio stops processing the Sip messages, the command will
not work anymore.
Unfortunately, I cannot share the kamailio.cfg file.
Thanks,
Tiago
Sergio Charrua <sergio.charrua@voip.pt<mailto:sergio.charrua@voip.pt>>
escreveu no dia segunda, 27/12/2021 à(s) 14:12:
ok, let's go 1 step at a time.
stop all firewalls, if any.
install sngrep SIP monitoring tool on your server
start sngrep and then (re)start your test script until you see Kamailio stoping accepting
calls
check if sngrep receives SIP Invites
Also, please share your kamailio.cfg . It might help too.
obrigado,
Sérgio Charrua
[http://cdn.shopify.com/s/files/1/0380/5305/t/11/assets/logo.png?11469850511…]
www.voip.pt<https://nam11.safelinks.protection.outlook.com/?url=http%3A%…
Tel.: +351 <callto:+351+91+104+12+66> 21 130 71 77
Email : sergio.charrua@voip.pt<mailto:sergio.charrua@voip.pt>
This message and any files or documents attached are strictly confidential or otherwise
legally protected.
It is intended only for the individual or entity named. If you are not the named addressee
or have received this email in error, please inform the sender immediately, delete it from
your system and do not copy or disclose it or its contents or use it for any purpose.
Please also note that transmission cannot be guaranteed to be secure or error-free.
On Mon, Dec 27, 2021 at 12:23 PM pwerspire
<pwerspire@gmail.com<mailto:pwerspire@gmail.com>> wrote:
Hi,
I just repeated the test and there is no Kamailio process at 100%, they are low all the
time even when the issue starts to happen.
I went through the logs and couldn't find any error, unfortunately, with 100 CPS I
can't put it on debug mode but only in info.
Regards,
Tiago
Ovidiu Sas <osas@voipembedded.com<mailto:osas@voipembedded.com>> escreveu no
dia quinta, 23/12/2021 à(s) 03:15:
Check with htop if you don't have any kamailio procs at 100% CPU (due
to a deadlock).
Do you see any errors in the logs?
-ovidiu
On Wed, Dec 22, 2021 at 11:44 AM pwerspire
<pwerspire@gmail.com<mailto:pwerspire@gmail.com>> wrote:
The CPU is around 30% during the test, is always really stable and memory doesn't
increase much.
I'm using children=8.
Sergio Charrua <sergio.charrua@voip.pt<mailto:sergio.charrua@voip.pt>>
escreveu no dia quarta, 22/12/2021 à(s) 16:27:
--
VoIP Embedded, Inc.
http://www.voipembedded.com<https://nam11.safelinks.protection.outlook.c…
__________________________________________________________
Kamailio - Users Mailing List - Non Commercial Discussions
* sr-users@lists.kamailio.org<mailto:sr-users@lists.kamailio.org>
Important: keep the mailing list in the recipients, do not reply only to the sender!
Edit mailing list options or unsubscribe:
*
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users<https://nam…
__________________________________________________________
Kamailio - Users Mailing List - Non Commercial Discussions
* sr-users@lists.kamailio.org<mailto:sr-users@lists.kamailio.org>
Important: keep the mailing list in the recipients, do not reply only to the sender!
Edit mailing list options or unsubscribe:
*
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users<https://nam…
__________________________________________________________
Kamailio - Users Mailing List - Non Commercial Discussions
* sr-users@lists.kamailio.org<mailto:sr-users@lists.kamailio.org>
Important: keep the mailing list in the recipients, do not reply only to the sender!
Edit mailing list options or unsubscribe:
*
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users<https://nam…
any statistics regarding CPU use? how many threads?
Sérgio Charrua
www.voip.pt<https://nam11.safelinks.protection.outlook.com/?url=http%3A%…
Tel.: +351 21 130 71 77
Email : sergio.charrua@voip.pt<mailto:sergio.charrua@voip.pt>
This message and any files or documents attached are strictly confidential or otherwise
legally protected.
It is intended only for the individual or entity named. If you are not the named
addressee or have received this email in error, please inform the sender immediately,
delete it from your system and do not copy or disclose it or its contents or use it for
any purpose. Please also note that transmission cannot be guaranteed to be secure or
error-free.
On Wed, Dec 22, 2021 at 3:57 PM pwerspire
<pwerspire@gmail.com<mailto:pwerspire@gmail.com>> wrote:
__________________________________________________________
Kamailio - Users Mailing List - Non Commercial Discussions
* sr-users@lists.kamailio.org<mailto:sr-users@lists.kamailio.org>
Important: keep the mailing list in the recipients, do not reply only to the sender!
Edit mailing list options or unsubscribe:
*
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users<https://nam…
Hi Sergio,
Thanks for your reply.
This is a google cloud VM with 8 vCPU, I'm using 100 CPS but the number of
simultaneous calls is stable at around 1071 at any given time.
Also, I can always see INVITES going inside the server but Kamailio stops answering them,
even if you stop sending calls for a long time, and then send a single call, Kamailio will
not reply anymore to the Invites.
When Kamailio stops replying to INVITES, If I do for example a "kamcmd
dispatcher.list" I have the correct output, but if I do a "kamcmd dlg.list"
it will not provide any output and not give back the prompt( before it would just answer
that the reply was too big).
I will need to use "control + c" to get the prompt of the shell back and after
that, all kamcmd commands stop returning any output or giving back the prompt.
If the Kamailio is then restarted, everything starts working correctly again. Also,
another note is that this seems only to happen when DMQ is in use and replicating to the
fallback Kamailio.
Regards,
Tiago
Sergio Charrua <sergio.charrua@voip.pt<mailto:sergio.charrua@voip.pt>>
escreveu no dia terça, 21/12/2021 à(s) 13:38:
__________________________________________________________
Kamailio - Users Mailing List - Non Commercial Discussions
* sr-users@lists.kamailio.org<mailto:sr-users@lists.kamailio.org>
Important: keep the mailing list in the recipients, do not reply only to the sender!
Edit mailing list options or unsubscribe:
*
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users<https://nam…
What is the call length of each call? How many NICs you have on that server? what
networking switches are you using? all these variables have huge impacts on the call
flow!
Let's say you make 100CPS and the call length is 10 minutes.
At 10 seconds you will have 1.000 ongoing calls using g711 (64kbps). You will have around
64Mbps bandwidth being used.
At 1 minute you get 6.000 ongoing calls and will use 390Mbps bandwidth!
Near the 10th minute you will be using almost 4Gbps bandwidth and 60.000 ongoing calls!!
Maybe the issue is not within Kamailio and/or the media server, but within your network
capacity ...
On my side, on a 4 vCPU and 4GB Ram, Kamailio 5.2 gets me around 400cps with calls up to
30seconds length, and RTPEngine goes OK too. I am sure I could do more, but I don't
bother much, this is more than enough!
But I have an enormous amount of bandwidth and 3 NICs on the server. NICs are the key .
Remember that Kamailio only processes SIP messages.
At 5 minutes, using the above call details, you've already sent 30.000 SIP INVITES
minimum and many more SIP messages during that time. You need a very good network
interface to do that kind of job!
Add 30.000 channels / RTP ports to your media server and you get a whole lot of data
going on in your network. I bet even the UTP5 cables won't handle the heat =:o)
Also, check the RTP port range your media server was configured with.
On Asterisk, usually ports 10.000 to 20.000 are set, which is about 10.000 calls. In the
above example, you would stop receiving calls at 2minutes because Asterisk has all RTP
ports busy/in use!
I don't know about other media servers, but I would bet this is pretty much the
same...
Hope this helps!
Sérgio Charrua
www.voip.pt<https://nam11.safelinks.protection.outlook.com/?url=http%3A%…
Tel.: +351 21 130 71 77
Email : sergio.charrua@voip.pt<mailto:sergio.charrua@voip.pt>
This message and any files or documents attached are strictly confidential or otherwise
legally protected.
It is intended only for the individual or entity named. If you are not the named
addressee or have received this email in error, please inform the sender immediately,
delete it from your system and do not copy or disclose it or its contents or use it for
any purpose. Please also note that transmission cannot be guaranteed to be secure or
error-free.
On Tue, Dec 21, 2021 at 11:54 AM pwerspire
<pwerspire@gmail.com<mailto:pwerspire@gmail.com>> wrote:
>
> Hi,
>
> I have been doing some load testing with Kamailio and having some issues.
>
> I have been trying with 100 CPS and at the beginning, everything is working well but
after some time (it can be 10 minutes or for example 40 minutes, is just random but
normally happens more in the first 10 minutes) Kamailio stop replying to all SIP messages
but still processing HTTP requests, some commands, etc.
>
> If I use "kamcmd dlg.list" no output happens and after that, all kamcmd
commands just keep loading with no output.
>
> Kamailio is sharing Dialogs and htables using DMQ with other Kamailio that is the
failover, if I shut down the failover one (so no DMQ replication), the test works well
with the 100 CPS.
>
> Also tried with modparam("dmq", "worker_usleep", 1000) but the
behavior is the same, it will stop processing traffic.
>
> This is some of configuration used:
>
> # ----------------- setting module-specific parameters ---------------
> modparam("dmq", "server_address",
"sip:INTERNAL_INSTANCE_IP:5060")
> modparam("dmq", "notification_address",
"DMQ_NOTIFICATION_ADDRESS")
>
> modparam("dmq", "multi_notify", 1)
> modparam("dmq", "ping_interval", 10)
> modparam("dmq", "num_workers",4)
>
>
> modparam("htable", "enable_dmq", 1)
> modparam("htable", "dmq_init_sync", 1)
>
>
> # ----- dialog params -----
> modparam("dialog", "dlg_flag", FLD_DLG)
> modparam("dialog", "dlg_match_mode", 1)
> modparam("dialog", "db_url", DBURL_RW)
> modparam("dialog", "db_mode", 0)
> modparam("dialog", "enable_dmq", 1)
> modparam("dialog", "db_update_period", 10)
> modparam("dialog", "h_id_start", H_ID_START)
> modparam("dialog", "h_id_step", H_ID_STEP)
>
> Memory:
>
> Shared: 4096
> Private: 512
>
> kamailio -v
> version: kamailio 5.5.3 (x86_64/linux) 473cef
> flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE,
USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC,
DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER,
USE_NAPTR, USE_DST_BLOCKLIST, HAVE_RESOLV_RES, TLS_PTHREAD_MUTEX_SHARED
> ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144, MAX_URI_SIZE 1024, BUF_SIZE
65535, DEFAULT PKG_SIZE 8MB
> poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
> id: 473cef
> compiled on 09:34:57 Dec 20 2021 with gcc 10.2.1
>
>
> In attach is the trap collected after the issue happens.
> Any more logs or configurations that can help identify or solve the issue?
>
> Thanks for the help,
> Regards,
> Tiago
> __________________________________________________________
> Kamailio - Users Mailing List - Non Commercial Discussions
> * sr-users@lists.kamailio.org<mailto:sr-users@lists.kamailio.org>
> Important: keep the mailing list in the recipients, do not reply only to the sender!
> Edit mailing list options or unsubscribe:
> *
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users<https://nam…
__________________________________________________________
Kamailio - Users Mailing List - Non Commercial Discussions
* sr-users@lists.kamailio.org<mailto:sr-users@lists.kamailio.org>
Important: keep the mailing list in the recipients, do not reply only to the sender!
Edit mailing list options or unsubscribe:
*
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users<https://nam…
__________________________________________________________
Kamailio - Users Mailing List - Non Commercial Discussions
* sr-users@lists.kamailio.org<mailto:sr-users@lists.kamailio.org>
Important: keep the mailing list in the recipients, do not reply only to the sender!
Edit mailing list options or unsubscribe:
*
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users<https://nam…
8:03 p.m.
On Dec 27, 2021, at 11:54 AM, Ben Kaufman
<bkaufman(a)bcmone.com> wrote:
Additionally, reducing the problem to the simplest configuration in which it can occur is
usually an invaluable tool for problem isolation.
+100!
--
Alex Balashov | Principal | Evariste Systems LLC
Tel: +1-706-510-6800 / +1-800-250-5920 (toll-free)
Web: http://www.evaristesys.com/, http://www.csrpswitch.com/
8:39 p.m.
Hello,
are you using dialog profiles by any chances? I have encountered an issue here before:
https://github.com/kamailio/kamailio/issues/2923
Cheers,
Henning
--
Henning Westerholt – https://skalatan.de/blog/
Kamailio services – https://gilawa.com<https://gilawa.com/>
From: sr-users <sr-users-bounces(a)lists.kamailio.org> On Behalf Of pwerspire
Sent: Monday, December 27, 2021 5:45 PM
To: Kamailio (SER) - Users Mailing List <sr-users(a)lists.kamailio.org>
Subject: Re: [SR-Users] Kamailio Stop processing SIP traffic
Already checked with sngrep and the INVITES are arriving.
I don't believe that is something related to the server because is not normal that
when the issue happens, dialog commands like "kamcmd dlg.list" stop returning
any prompt. During the time that the issue is not happening yet, the command just returns
that is too big, then when Kamailio stops processing the Sip messages, the command will
not work anymore.
Unfortunately, I cannot share the kamailio.cfg file.
Thanks,
Tiago
Sergio Charrua <sergio.charrua@voip.pt<mailto:sergio.charrua@voip.pt>>
escreveu no dia segunda, 27/12/2021 à(s) 14:12:
ok, let's go 1 step at a time.
stop all firewalls, if any.
install sngrep SIP monitoring tool on your server
start sngrep and then (re)start your test script until you see Kamailio stoping accepting
calls
check if sngrep receives SIP Invites
Also, please share your kamailio.cfg . It might help too.
obrigado,
Sérgio Charrua
[http://cdn.shopify.com/s/files/1/0380/5305/t/11/assets/logo.png?11469850511…]
www.voip.pt<http://www.voip.pt/>
Tel.: +351 <callto:+351+91+104+12+66> 21 130 71 77
Email : sergio.charrua@voip.pt<mailto:sergio.charrua@voip.pt>
This message and any files or documents attached are strictly confidential or otherwise
legally protected.
It is intended only for the individual or entity named. If you are not the named addressee
or have received this email in error, please inform the sender immediately, delete it from
your system and do not copy or disclose it or its contents or use it for any purpose.
Please also note that transmission cannot be guaranteed to be secure or error-free.
On Mon, Dec 27, 2021 at 12:23 PM pwerspire
<pwerspire@gmail.com<mailto:pwerspire@gmail.com>> wrote:
Hi,
I just repeated the test and there is no Kamailio process at 100%, they are low all the
time even when the issue starts to happen.
I went through the logs and couldn't find any error, unfortunately, with 100 CPS I
can't put it on debug mode but only in info.
Regards,
Tiago
Ovidiu Sas <osas@voipembedded.com<mailto:osas@voipembedded.com>> escreveu no
dia quinta, 23/12/2021 à(s) 03:15:
Check with htop if you don't have any kamailio procs at 100% CPU (due
to a deadlock).
Do you see any errors in the logs?
-ovidiu
On Wed, Dec 22, 2021 at 11:44 AM pwerspire
<pwerspire@gmail.com<mailto:pwerspire@gmail.com>> wrote:
The CPU is around 30% during the test, is always really stable and memory doesn't
increase much.
I'm using children=8.
Sergio Charrua <sergio.charrua@voip.pt<mailto:sergio.charrua@voip.pt>>
escreveu no dia quarta, 22/12/2021 à(s) 16:27:
--
VoIP Embedded, Inc.
http://www.voipembedded.com
__________________________________________________________
Kamailio - Users Mailing List - Non Commercial Discussions
* sr-users@lists.kamailio.org<mailto:sr-users@lists.kamailio.org>
Important: keep the mailing list in the recipients, do not reply only to the sender!
Edit mailing list options or unsubscribe:
* https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
__________________________________________________________
Kamailio - Users Mailing List - Non Commercial Discussions
* sr-users@lists.kamailio.org<mailto:sr-users@lists.kamailio.org>
Important: keep the mailing list in the recipients, do not reply only to the sender!
Edit mailing list options or unsubscribe:
* https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
__________________________________________________________
Kamailio - Users Mailing List - Non Commercial Discussions
* sr-users@lists.kamailio.org<mailto:sr-users@lists.kamailio.org>
Important: keep the mailing list in the recipients, do not reply only to the sender!
Edit mailing list options or unsubscribe:
* https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
any statistics regarding CPU use? how many threads?
Sérgio Charrua
www.voip.pt<http://www.voip.pt>
Tel.: +351 21 130 71 77
Email : sergio.charrua@voip.pt<mailto:sergio.charrua@voip.pt>
This message and any files or documents attached are strictly confidential or otherwise
legally protected.
It is intended only for the individual or entity named. If you are not the named
addressee or have received this email in error, please inform the sender immediately,
delete it from your system and do not copy or disclose it or its contents or use it for
any purpose. Please also note that transmission cannot be guaranteed to be secure or
error-free.
On Wed, Dec 22, 2021 at 3:57 PM pwerspire
<pwerspire@gmail.com<mailto:pwerspire@gmail.com>> wrote:
__________________________________________________________
Kamailio - Users Mailing List - Non Commercial Discussions
* sr-users@lists.kamailio.org<mailto:sr-users@lists.kamailio.org>
Important: keep the mailing list in the recipients, do not reply only to the sender!
Edit mailing list options or unsubscribe:
* https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Hi Sergio,
Thanks for your reply.
This is a google cloud VM with 8 vCPU, I'm using 100 CPS but the number of
simultaneous calls is stable at around 1071 at any given time.
Also, I can always see INVITES going inside the server but Kamailio stops answering them,
even if you stop sending calls for a long time, and then send a single call, Kamailio will
not reply anymore to the Invites.
When Kamailio stops replying to INVITES, If I do for example a "kamcmd
dispatcher.list" I have the correct output, but if I do a "kamcmd dlg.list"
it will not provide any output and not give back the prompt( before it would just answer
that the reply was too big).
I will need to use "control + c" to get the prompt of the shell back and after
that, all kamcmd commands stop returning any output or giving back the prompt.
If the Kamailio is then restarted, everything starts working correctly again. Also,
another note is that this seems only to happen when DMQ is in use and replicating to the
fallback Kamailio.
Regards,
Tiago
Sergio Charrua <sergio.charrua@voip.pt<mailto:sergio.charrua@voip.pt>>
escreveu no dia terça, 21/12/2021 à(s) 13:38:
__________________________________________________________
Kamailio - Users Mailing List - Non Commercial Discussions
* sr-users@lists.kamailio.org<mailto:sr-users@lists.kamailio.org>
Important: keep the mailing list in the recipients, do not reply only to the sender!
Edit mailing list options or unsubscribe:
* https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
What is the call length of each call? How many NICs you have on that server? what
networking switches are you using? all these variables have huge impacts on the call
flow!
Let's say you make 100CPS and the call length is 10 minutes.
At 10 seconds you will have 1.000 ongoing calls using g711 (64kbps). You will have around
64Mbps bandwidth being used.
At 1 minute you get 6.000 ongoing calls and will use 390Mbps bandwidth!
Near the 10th minute you will be using almost 4Gbps bandwidth and 60.000 ongoing
calls!!
Maybe the issue is not within Kamailio and/or the media server, but within your network
capacity ...
On my side, on a 4 vCPU and 4GB Ram, Kamailio 5.2 gets me around 400cps with calls up to
30seconds length, and RTPEngine goes OK too. I am sure I could do more, but I don't
bother much, this is more than enough!
But I have an enormous amount of bandwidth and 3 NICs on the server. NICs are the key .
Remember that Kamailio only processes SIP messages.
At 5 minutes, using the above call details, you've already sent 30.000 SIP INVITES
minimum and many more SIP messages during that time. You need a very good network
interface to do that kind of job!
Add 30.000 channels / RTP ports to your media server and you get a whole lot of data
going on in your network. I bet even the UTP5 cables won't handle the heat =:o)
Also, check the RTP port range your media server was configured with.
On Asterisk, usually ports 10.000 to 20.000 are set, which is about 10.000 calls. In the
above example, you would stop receiving calls at 2minutes because Asterisk has all RTP
ports busy/in use!
I don't know about other media servers, but I would bet this is pretty much the
same...
Hope this helps!
Sérgio Charrua
www.voip.pt<http://www.voip.pt>
Tel.: +351 21 130 71 77
Email : sergio.charrua@voip.pt<mailto:sergio.charrua@voip.pt>
This message and any files or documents attached are strictly confidential or otherwise
legally protected.
It is intended only for the individual or entity named. If you are not the named
addressee or have received this email in error, please inform the sender immediately,
delete it from your system and do not copy or disclose it or its contents or use it for
any purpose. Please also note that transmission cannot be guaranteed to be secure or
error-free.
On Tue, Dec 21, 2021 at 11:54 AM pwerspire
<pwerspire@gmail.com<mailto:pwerspire@gmail.com>> wrote:
>
> Hi,
>
> I have been doing some load testing with Kamailio and having some issues.
>
> I have been trying with 100 CPS and at the beginning, everything is working well but
after some time (it can be 10 minutes or for example 40 minutes, is just random but
normally happens more in the first 10 minutes) Kamailio stop replying to all SIP messages
but still processing HTTP requests, some commands, etc.
>
> If I use "kamcmd dlg.list" no output happens and after that, all kamcmd
commands just keep loading with no output.
>
> Kamailio is sharing Dialogs and htables using DMQ with other Kamailio that is the
failover, if I shut down the failover one (so no DMQ replication), the test works well
with the 100 CPS.
>
> Also tried with modparam("dmq", "worker_usleep", 1000) but the
behavior is the same, it will stop processing traffic.
>
> This is some of configuration used:
>
> # ----------------- setting module-specific parameters ---------------
> modparam("dmq", "server_address",
"sip:INTERNAL_INSTANCE_IP:5060")
> modparam("dmq", "notification_address",
"DMQ_NOTIFICATION_ADDRESS")
>
> modparam("dmq", "multi_notify", 1)
> modparam("dmq", "ping_interval", 10)
> modparam("dmq", "num_workers",4)
>
>
> modparam("htable", "enable_dmq", 1)
> modparam("htable", "dmq_init_sync", 1)
>
>
> # ----- dialog params -----
> modparam("dialog", "dlg_flag", FLD_DLG)
> modparam("dialog", "dlg_match_mode", 1)
> modparam("dialog", "db_url", DBURL_RW)
> modparam("dialog", "db_mode", 0)
> modparam("dialog", "enable_dmq", 1)
> modparam("dialog", "db_update_period", 10)
> modparam("dialog", "h_id_start", H_ID_START)
> modparam("dialog", "h_id_step", H_ID_STEP)
>
> Memory:
>
> Shared: 4096
> Private: 512
>
> kamailio -v
> version: kamailio 5.5.3 (x86_64/linux) 473cef
> flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE,
USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC,
DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER,
USE_NAPTR, USE_DST_BLOCKLIST, HAVE_RESOLV_RES, TLS_PTHREAD_MUTEX_SHARED
> ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144, MAX_URI_SIZE 1024, BUF_SIZE
65535, DEFAULT PKG_SIZE 8MB
> poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
> id: 473cef
> compiled on 09:34:57 Dec 20 2021 with gcc 10.2.1
>
>
> In attach is the trap collected after the issue happens.
> Any more logs or configurations that can help identify or solve the issue?
>
> Thanks for the help,
> Regards,
> Tiago
> __________________________________________________________
> Kamailio - Users Mailing List - Non Commercial Discussions
> * sr-users@lists.kamailio.org<mailto:sr-users@lists.kamailio.org>
> Important: keep the mailing list in the recipients, do not reply only to the
sender!
> Edit mailing list options or unsubscribe:
> * https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
__________________________________________________________
Kamailio - Users Mailing List - Non Commercial Discussions
* sr-users@lists.kamailio.org<mailto:sr-users@lists.kamailio.org>
Important: keep the mailing list in the recipients, do not reply only to the sender!
Edit mailing list options or unsubscribe:
* https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
__________________________________________________________
Kamailio - Users Mailing List - Non Commercial Discussions
* sr-users@lists.kamailio.org<mailto:sr-users@lists.kamailio.org>
Important: keep the mailing list in the recipients, do not reply only to the sender!
Edit mailing list options or unsubscribe:
* https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
1067
days inactive
1074
days old
11 comments
6 participants
participants (6)
-
Alex Balashov -
Ben Kaufman -
Henning Westerholt -
Ovidiu Sas -
pwerspire -
Sergio Charrua