[SR-Users] Kamailio Stop processing SIP traffic
pwerspire
pwerspire at gmail.com
Mon Dec 27 13:12:14 CET 2021
Hi,
I just repeated the test and there is no Kamailio process at 100%, they are
low all the time even when the issue starts to happen.
I went through the logs and couldn't find any error, unfortunately, with
100 CPS I can't put it on debug mode but only in info.
Regards,
Tiago
Ovidiu Sas <osas at voipembedded.com> escreveu no dia quinta, 23/12/2021 à(s)
03:15:
> Check with htop if you don't have any kamailio procs at 100% CPU (due
> to a deadlock).
> Do you see any errors in the logs?
>
> -ovidiu
>
>
> On Wed, Dec 22, 2021 at 11:44 AM pwerspire <pwerspire at gmail.com> wrote:
> >
> > The CPU is around 30% during the test, is always really stable and
> memory doesn't increase much.
> >
> > I'm using children=8.
> >
> >
> >
> > Sergio Charrua <sergio.charrua at voip.pt> escreveu no dia quarta,
> 22/12/2021 à(s) 16:27:
> >>
> >> any statistics regarding CPU use? how many threads?
> >>
> >> Sérgio Charrua
> >>
> >>
> >> www.voip.pt
> >> Tel.: +351 21 130 71 77
> >>
> >> Email : sergio.charrua at voip.pt
> >>
> >> This message and any files or documents attached are strictly
> confidential or otherwise legally protected.
> >>
> >> It is intended only for the individual or entity named. If you are not
> the named addressee or have received this email in error, please inform the
> sender immediately, delete it from your system and do not copy or disclose
> it or its contents or use it for any purpose. Please also note that
> transmission cannot be guaranteed to be secure or error-free.
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >> On Wed, Dec 22, 2021 at 3:57 PM pwerspire <pwerspire at gmail.com> wrote:
> >>>
> >>> Hi Sergio,
> >>>
> >>> Thanks for your reply.
> >>>
> >>> This is a google cloud VM with 8 vCPU, I'm using 100 CPS but the
> number of simultaneous calls is stable at around 1071 at any given time.
> >>>
> >>> Also, I can always see INVITES going inside the server but Kamailio
> stops answering them, even if you stop sending calls for a long time, and
> then send a single call, Kamailio will not reply anymore to the Invites.
> >>>
> >>> When Kamailio stops replying to INVITES, If I do for example a "kamcmd
> dispatcher.list" I have the correct output, but if I do a "kamcmd dlg.list"
> it will not provide any output and not give back the prompt( before it
> would just answer that the reply was too big).
> >>> I will need to use "control + c" to get the prompt of the shell back
> and after that, all kamcmd commands stop returning any output or giving
> back the prompt.
> >>>
> >>> If the Kamailio is then restarted, everything starts working correctly
> again. Also, another note is that this seems only to happen when DMQ is in
> use and replicating to the fallback Kamailio.
> >>>
> >>> Regards,
> >>> Tiago
> >>>
> >>>
> >>> Sergio Charrua <sergio.charrua at voip.pt> escreveu no dia terça,
> 21/12/2021 à(s) 13:38:
> >>>>
> >>>> What is the call length of each call? How many NICs you have on that
> server? what networking switches are you using? all these variables have
> huge impacts on the call flow!
> >>>>
> >>>> Let's say you make 100CPS and the call length is 10 minutes.
> >>>> At 10 seconds you will have 1.000 ongoing calls using g711 (64kbps).
> You will have around 64Mbps bandwidth being used.
> >>>> At 1 minute you get 6.000 ongoing calls and will use 390Mbps
> bandwidth!
> >>>> Near the 10th minute you will be using almost 4Gbps bandwidth and
> 60.000 ongoing calls!!
> >>>>
> >>>> Maybe the issue is not within Kamailio and/or the media server, but
> within your network capacity ...
> >>>>
> >>>> On my side, on a 4 vCPU and 4GB Ram, Kamailio 5.2 gets me around
> 400cps with calls up to 30seconds length, and RTPEngine goes OK too. I am
> sure I could do more, but I don't bother much, this is more than enough!
> >>>> But I have an enormous amount of bandwidth and 3 NICs on the server.
> NICs are the key . Remember that Kamailio only processes SIP messages.
> >>>> At 5 minutes, using the above call details, you've already sent
> 30.000 SIP INVITES minimum and many more SIP messages during that time. You
> need a very good network interface to do that kind of job!
> >>>> Add 30.000 channels / RTP ports to your media server and you get a
> whole lot of data going on in your network. I bet even the UTP5 cables
> won't handle the heat =:o)
> >>>>
> >>>> Also, check the RTP port range your media server was configured with.
> >>>> On Asterisk, usually ports 10.000 to 20.000 are set, which is about
> 10.000 calls. In the above example, you would stop receiving calls at
> 2minutes because Asterisk has all RTP ports busy/in use!
> >>>> I don't know about other media servers, but I would bet this is
> pretty much the same...
> >>>>
> >>>> Hope this helps!
> >>>>
> >>>>
> >>>> Sérgio Charrua
> >>>>
> >>>>
> >>>> www.voip.pt
> >>>> Tel.: +351 21 130 71 77
> >>>>
> >>>> Email : sergio.charrua at voip.pt
> >>>>
> >>>> This message and any files or documents attached are strictly
> confidential or otherwise legally protected.
> >>>>
> >>>> It is intended only for the individual or entity named. If you are
> not the named addressee or have received this email in error, please inform
> the sender immediately, delete it from your system and do not copy or
> disclose it or its contents or use it for any purpose. Please also note
> that transmission cannot be guaranteed to be secure or error-free.
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> On Tue, Dec 21, 2021 at 11:54 AM pwerspire <pwerspire at gmail.com>
> wrote:
> >>>>>
> >>>>> Hi,
> >>>>>
> >>>>> I have been doing some load testing with Kamailio and having some
> issues.
> >>>>>
> >>>>> I have been trying with 100 CPS and at the beginning, everything is
> working well but after some time (it can be 10 minutes or for example 40
> minutes, is just random but normally happens more in the first 10 minutes)
> Kamailio stop replying to all SIP messages but still processing HTTP
> requests, some commands, etc.
> >>>>>
> >>>>> If I use "kamcmd dlg.list" no output happens and after that, all
> kamcmd commands just keep loading with no output.
> >>>>>
> >>>>> Kamailio is sharing Dialogs and htables using DMQ with other
> Kamailio that is the failover, if I shut down the failover one (so no DMQ
> replication), the test works well with the 100 CPS.
> >>>>>
> >>>>> Also tried with modparam("dmq", "worker_usleep", 1000) but the
> behavior is the same, it will stop processing traffic.
> >>>>>
> >>>>> This is some of configuration used:
> >>>>>
> >>>>> # ----------------- setting module-specific parameters
> ---------------
> >>>>> modparam("dmq", "server_address", "sip:INTERNAL_INSTANCE_IP:5060")
> >>>>> modparam("dmq", "notification_address", "DMQ_NOTIFICATION_ADDRESS")
> >>>>>
> >>>>> modparam("dmq", "multi_notify", 1)
> >>>>> modparam("dmq", "ping_interval", 10)
> >>>>> modparam("dmq", "num_workers",4)
> >>>>>
> >>>>>
> >>>>> modparam("htable", "enable_dmq", 1)
> >>>>> modparam("htable", "dmq_init_sync", 1)
> >>>>>
> >>>>>
> >>>>> # ----- dialog params -----
> >>>>> modparam("dialog", "dlg_flag", FLD_DLG)
> >>>>> modparam("dialog", "dlg_match_mode", 1)
> >>>>> modparam("dialog", "db_url", DBURL_RW)
> >>>>> modparam("dialog", "db_mode", 0)
> >>>>> modparam("dialog", "enable_dmq", 1)
> >>>>> modparam("dialog", "db_update_period", 10)
> >>>>> modparam("dialog", "h_id_start", H_ID_START)
> >>>>> modparam("dialog", "h_id_step", H_ID_STEP)
> >>>>>
> >>>>> Memory:
> >>>>>
> >>>>> Shared: 4096
> >>>>> Private: 512
> >>>>>
> >>>>> kamailio -v
> >>>>> version: kamailio 5.5.3 (x86_64/linux) 473cef
> >>>>> flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS,
> DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, Q_MALLOC,
> F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT,
> USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLOCKLIST,
> HAVE_RESOLV_RES, TLS_PTHREAD_MUTEX_SHARED
> >>>>> ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144, MAX_URI_SIZE
> 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB
> >>>>> poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
> >>>>> id: 473cef
> >>>>> compiled on 09:34:57 Dec 20 2021 with gcc 10.2.1
> >>>>>
> >>>>>
> >>>>> In attach is the trap collected after the issue happens.
> >>>>> Any more logs or configurations that can help identify or solve the
> issue?
> >>>>>
> >>>>> Thanks for the help,
> >>>>> Regards,
> >>>>> Tiago
> >>>>> __________________________________________________________
> >>>>> Kamailio - Users Mailing List - Non Commercial Discussions
> >>>>> * sr-users at lists.kamailio.org
> >>>>> Important: keep the mailing list in the recipients, do not reply
> only to the sender!
> >>>>> Edit mailing list options or unsubscribe:
> >>>>> * https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
> >>>>
> >>>> __________________________________________________________
> >>>> Kamailio - Users Mailing List - Non Commercial Discussions
> >>>> * sr-users at lists.kamailio.org
> >>>> Important: keep the mailing list in the recipients, do not reply only
> to the sender!
> >>>> Edit mailing list options or unsubscribe:
> >>>> * https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
> >>>
> >>> __________________________________________________________
> >>> Kamailio - Users Mailing List - Non Commercial Discussions
> >>> * sr-users at lists.kamailio.org
> >>> Important: keep the mailing list in the recipients, do not reply only
> to the sender!
> >>> Edit mailing list options or unsubscribe:
> >>> * https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
> >>
> >> __________________________________________________________
> >> Kamailio - Users Mailing List - Non Commercial Discussions
> >> * sr-users at lists.kamailio.org
> >> Important: keep the mailing list in the recipients, do not reply only
> to the sender!
> >> Edit mailing list options or unsubscribe:
> >> * https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
> >
> > __________________________________________________________
> > Kamailio - Users Mailing List - Non Commercial Discussions
> > * sr-users at lists.kamailio.org
> > Important: keep the mailing list in the recipients, do not reply only to
> the sender!
> > Edit mailing list options or unsubscribe:
> > * https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
>
>
>
> --
> VoIP Embedded, Inc.
> http://www.voipembedded.com
>
> __________________________________________________________
> Kamailio - Users Mailing List - Non Commercial Discussions
> * sr-users at lists.kamailio.org
> Important: keep the mailing list in the recipients, do not reply only to
> the sender!
> Edit mailing list options or unsubscribe:
> * https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.kamailio.org/pipermail/sr-users/attachments/20211227/56baa1bc/attachment.htm>
More information about the sr-users
mailing list