What do you have as a UAS? The UDP receive buffer increase that Ovidiu suggested would be necessary if you see traffic piling up/getting dropped, e.g. look at `netstat -s | grep "receive buffer errors"` or watch for 2nd column Recv-Q of netstat. Cheers, On Fri, Mar 22, 2024 at 1:31 PM Ovidiu Sas via sr-users < sr-users(a)lists.kamailio.org&gt; wrote: __________________________________________________________ Kamailio - Users Mailing List - Non Commercial Discussions To unsubscribe send an email to sr-users-leave(a)lists.kamailio.org Important: keep the mailing list in the recipients, do not reply only to the sender! Edit mailing list options or unsubscribe:

I don't know that I would do that. If your server isn't coping with the load, making the kernel receive queue deeper doesn't increase throughput, it just relocates the queueing -- or rather, adds another layer of queueing. Squeeze the balloon in one place, it inflates in another. -- Alex sr-users(a)lists.kamailio.org&gt; wrote: https://medium.com/@CameronSparr/increase-os-udp-buffers-to-improve-perform… sr-users(a)lists.kamailio.org&gt; wrote: 4216 CPU @ 2.10GHz and, 2 NICs and 4Gb RAM and MariaDB 10.6 as DB Backend., all running on a HP G380 host with a gazillion CPUs and a googol disk space! engine and accounting DB, Above that value, I get retransmissions everywhere. lot of retransmissions, while CPU/Core usage on Kamailio server stays below 10%. article found at templates available on the article, hoping for better results. far from the 7000CPS found in the article! article, which is pretty old already, were made on physical servers. Still, I would not expect 95% of lower performance! the sender! the sender! -- Alex Balashov Principal Consultant Evariste Systems LLC Web: https://evaristesys.com Tel: +1-706-510-6800 __________________________________________________________ Kamailio - Users Mailing List - Non Commercial Discussions To unsubscribe send an email to sr-users-leave(a)lists.kamailio.org Important: keep the mailing list in the recipients, do not reply only to the sender! Edit mailing list options or unsubscribe:

Sergio, You may consider giving this a read: https://blog.evaristesys.com/2016/02/15/tuning-kamailio-for-high-throughput… Child processes will make a big difference, especially in your scenario #1, where you're waiting on outside services. But it's possible you've just run into a fundamental throughput limit of your environment, and if you have, I would not take the advice to increase receive queue depth. -- Alex wrote: sr-users(a)lists.kamailio.org&gt; wrote: load, making the kernel receive queue deeper doesn't increase throughput, it just relocates the queueing -- or rather, adds another layer of queueing. sr-users(a)lists.kamailio.org&gt; wrote: https://medium.com/@CameronSparr/increase-os-udp-buffers-to-improve-perform… sr-users(a)lists.kamailio.org&gt; wrote: 4216 CPU @ 2.10GHz and, 2 NICs and 4Gb RAM and MariaDB 10.6 as DB Backend., all running on a HP G380 host with a gazillion CPUs and a googol disk space! engine and accounting no DB, Above that value, I get retransmissions everywhere. lot of retransmissions, while CPU/Core usage on Kamailio server stays below 10%. article found at templates available on the article, hoping for better results. far from the 7000CPS found in the article! article, which is pretty old already, were made on physical servers. Still, I would not expect 95% of lower performance! to the sender! to the sender! the sender! -- Alex Balashov Principal Consultant Evariste Systems LLC Web: https://evaristesys.com Tel: +1-706-510-6800 __________________________________________________________ Kamailio - Users Mailing List - Non Commercial Discussions To unsubscribe send an email to sr-users-leave(a)lists.kamailio.org Important: keep the mailing list in the recipients, do not reply only to the sender! Edit mailing list options or unsubscribe:

On Mar 22, 2024, at 10:21 PM, Ovidiu Sas &lt;osas(a)voipembedded.com&gt; wrote: The default udp queue length is not enough for high cps.

On Mar 23, 2024, at 1:54 PM, Fred Posner &lt;fred(a)pgpx.io&gt; wrote: The default queue I believe is about 2MB. Not very hard to exceed that queue while still having a good amount of CPU/processor available. -- Fred Posner Sent from mobile Phone: +1 (352) 664-3733 qxork.com > On Mar 23, 2024, at 11:47 AM, Alex Balashov via sr-users &lt;sr-users(a)lists.kamailio.org&gt; wrote: > >  >> On Mar 22, 2024, at 10:21 PM, Ovidiu Sas &lt;osas(a)voipembedded.com&gt; wrote: >> >> The default udp queue length is not enough for high cps. > > That's interesting. I'm not necessarily saying you're wrong, but I'd be curious to know more about what informs this theory. > > -- > Alex Balashov > Principal Consultant > Evariste Systems LLC > Web: https://evaristesys.com > Tel: +1-706-510-6800 > > __________________________________________________________ > Kamailio - Users Mailing List - Non Commercial Discussions > To unsubscribe send an email to sr-users-leave(a)lists.kamailio.org > Important: keep the mailing list in the recipients, do not reply only to the sender! > Edit mailing list options or unsubscribe:

On Mar 23, 2024, at 4:53 PM, Fred Posner &lt;fred(a)pgpx.io&gt; wrote: That could be one scenario. But when I’ve seen there being more than enough CPU it has been there’s a more for 2MB worth of traffic and the OS is making an issue that is easily solved by letting the system handle more traffic. It’s a bottleneck before kamailio. The analogy would be a funnel. At some point, when you’re pouring a lot of liquid, you need to use a wider funnel. Ain’t nothing you do after the funnel will matter. The funnel is the problem. -- Fred Posner Sent from mobile Phone: +1 (352) 664-3733 qxork.com > On Mar 23, 2024, at 4:13 PM, Alex Balashov via sr-users &lt;sr-users(a)lists.kamailio.org&gt; wrote: > > Sure, but if you're blowing through the default, that means you're not consistently coping with the load due to some other factors. Does it not follow that enlarging the queue will just give you more backlog without increasing throughput? > > -- Alex > >> On Mar 23, 2024, at 1:54 PM, Fred Posner &lt;fred(a)pgpx.io&gt; wrote: >> >> The default queue I believe is about 2MB. Not very hard to exceed that queue while still having a good amount of CPU/processor available. >> >> -- Fred Posner >> Sent from mobile >> Phone: +1 (352) 664-3733 >> qxork.com >> >> >> >>>> On Mar 23, 2024, at 11:47 AM, Alex Balashov via sr-users &lt;sr-users(a)lists.kamailio.org&gt; wrote: >>> >>>  >>>> On Mar 22, 2024, at 10:21 PM, Ovidiu Sas &lt;osas(a)voipembedded.com&gt; wrote: >>>> >>>> The default udp queue length is not enough for high cps. >>> >>> That's interesting. I'm not necessarily saying you're wrong, but I'd be curious to know more about what informs this theory. >>> >>> -- >>> Alex Balashov >>> Principal Consultant >>> Evariste Systems LLC >>> Web: https://evaristesys.com >>> Tel: +1-706-510-6800 >>> >>> __________________________________________________________ >>> Kamailio - Users Mailing List - Non Commercial Discussions >>> To unsubscribe send an email to sr-users-leave(a)lists.kamailio.org >>> Important: keep the mailing list in the recipients, do not reply only to the sender! >>> Edit mailing list options or unsubscribe: > > -- > Alex Balashov > Principal Consultant > Evariste Systems LLC > Web: https://evaristesys.com > Tel: +1-706-510-6800 > > __________________________________________________________ > Kamailio - Users Mailing List - Non Commercial Discussions > To unsubscribe send an email to sr-users-leave(a)lists.kamailio.org > Important: keep the mailing list in the recipients, do not reply only to the sender! > Edit mailing list options or unsubscribe:

On Mar 23, 2024, at 5:19 PM, Ovidiu Sas &lt;osas(a)voipembedded.com&gt; wrote: But the CPU is not the limiting factor.

Back to the funnel analogy: you have a big bottle and you are using a small funnel. That will not work well. If you use a bigger funnel with the same bottle (the bigger funnel must fit the bottle), than you can handle more liquid. If the funnel is too big for the bottle, then you are spilling out and this is the scenario that you are referring to.

On Mar 23, 2024, at 6:05 PM, Ovidiu Sas &lt;osas(a)voipembedded.com&gt; wrote: if most of the time the udp queue is empty and dropped packets are observed, then the size of the udp queue is too small.

observed, then the size of the udp queue is too small. Hmm.. why would you need to increase the queue size if nothing gets queued up? On Sat, Mar 23, 2024 at 6:38 PM Alex Balashov via sr-users < sr-users(a)lists.kamailio.org&gt; wrote: __________________________________________________________ Kamailio - Users Mailing List - Non Commercial Discussions To unsubscribe send an email to sr-users-leave(a)lists.kamailio.org Important: keep the mailing list in the recipients, do not reply only to the sender! Edit mailing list options or unsubscribe:

sr-users(a)lists.kamailio.org&gt; wrote: milliseconds and that’s the moment when you experience dropped packets. Totally fair. Just wondering if that's what we're discussing here. OP said he's running sipp, and sipp doesn't push out volatile workloads that are offset by periods of little to no activity to rapidly drain the queue. -- Alex -- Alex Balashov Principal Consultant Evariste Systems LLC Web: https://evaristesys.com Tel: +1-706-510-6800 __________________________________________________________ Kamailio - Users Mailing List - Non Commercial Discussions To unsubscribe send an email to sr-users-leave(a)lists.kamailio.org Important: keep the mailing list in the recipients, do not reply only to the sender! Edit mailing list options or unsubscribe:

repeating the same concept. Yeah, I think that's mostly right. Although I don't deal with very high-CPS deployments (500-1000 CPS) much these days, I used to, and my experiences there led me to the diametrically opposite conclusion: one should never increase the UDP queue size, and if you find yourself doing that, you're doing something wrong, _except_ in the occasional burst case we discussed. You can be absolutely sure that when I first encountered the problem, my first impulse was to increase the receive queue as high as it will go, then, gradually, to a lesser extent. I ultimately found that the proper amount by which to raise it is 0. ;) -- Alex -- Alex Balashov Principal Consultant Evariste Systems LLC Web: https://evaristesys.com Tel: +1-706-510-6800 __________________________________________________________ Kamailio - Users Mailing List - Non Commercial Discussions To unsubscribe send an email to sr-users-leave(a)lists.kamailio.org Important: keep the mailing list in the recipients, do not reply only to the sender! Edit mailing list options or unsubscribe:

Not really related to the ongoing discussion, but: Going to that kind of CPS might exceed the natural limits of all but the most exquisitely tuned execution environments. It probably wouldn't work at all on the average moderately oversubscribed public cloud VM, even a generously resourced one. Once you get to that point, you might be better off just scaling horizontally. -- Alex cps you run into this kind of issues. sr-users(a)lists.kamailio.org&gt; wrote: repeating the same concept. these days, I used to, and my experiences there led me to the diametrically opposite conclusion: one should never increase the UDP queue size, and if you find yourself doing that, you're doing something wrong, _except_ in the occasional burst case we discussed. first impulse was to increase the receive queue as high as it will go, then, gradually, to a lesser extent. I ultimately found that the proper amount by which to raise it is 0. ;) the sender! -- Alex Balashov Principal Consultant Evariste Systems LLC Web: https://evaristesys.com Tel: +1-706-510-6800 __________________________________________________________ Kamailio - Users Mailing List - Non Commercial Discussions To unsubscribe send an email to sr-users-leave(a)lists.kamailio.org Important: keep the mailing list in the recipients, do not reply only to the sender! Edit mailing list options or unsubscribe:

Hello, you should be able to decrease the private memory substantially, as this is per process. This much is never needed. On the other hand, you should probably increase the shared memory if you are having a lot of transactions going on, TLS etc.. This is per server, so you can configure more. Cheers, Henning *From:* Sergio Charrua via sr-users &lt;sr-users(a)lists.kamailio.org&gt; *Sent:* Freitag, 5. April 2024 11:45 *To:* Kamailio (SER) - Users Mailing List &lt;sr-users(a)lists.kamailio.org&gt; *Cc:* Sergio Charrua &lt;sergio.charrua(a)voip.pt&gt; *Subject:* [SR-Users] Re: low performance with no apparent reason Thank you all for helping! I wasn't expecting such a large number of replies! I ended up partially solving the issue with a different approach. Modifying the size of the UDP Buffer did not reveal any improvement. However, modifying the memory management did improve a lot: from 330 CPS to 1800 CPS in stateful mode. So, starting kamailio with the following command: kamailio -M 256 -m 128 -f <script.cfg> did the trick! And the VM is still running with 6 vCPU. Still very far from the test results described in https://www.kamailio.org/docs/openser-performance-tests/#tm-tests-c but a lot better and meets our requirements Thanks guys for your help! Greatly appreciated! *Sérgio Charrua* On Sun, Mar 24, 2024 at 4:28 PM Alex Balashov via sr-users < sr-users(a)lists.kamailio.org&gt; wrote: Not really related to the ongoing discussion, but: Going to that kind of CPS might exceed the natural limits of all but the most exquisitely tuned execution environments. It probably wouldn't work at all on the average moderately oversubscribed public cloud VM, even a generously resourced one. Once you get to that point, you might be better off just scaling horizontally. -- Alex cps you run into this kind of issues. sr-users(a)lists.kamailio.org&gt; wrote: repeating the same concept. these days, I used to, and my experiences there led me to the diametrically opposite conclusion: one should never increase the UDP queue size, and if you find yourself doing that, you're doing something wrong, _except_ in the occasional burst case we discussed. first impulse was to increase the receive queue as high as it will go, then, gradually, to a lesser extent. I ultimately found that the proper amount by which to raise it is 0. ;) the sender! -- Alex Balashov Principal Consultant Evariste Systems LLC Web: https://evaristesys.com Tel: +1-706-510-6800 __________________________________________________________ Kamailio - Users Mailing List - Non Commercial Discussions To unsubscribe send an email to sr-users-leave(a)lists.kamailio.org Important: keep the mailing list in the recipients, do not reply only to the sender! Edit mailing list options or unsubscribe:

On Mar 23, 2024, at 7:19 PM, Ovidiu Sas &lt;osas(a)voipembedded.com&gt; wrote: Things could cascade due to retransmission. It all depends on how much the system is loaded and the type of the traffic and message handling in kamailio. Sometimes the system recovers and works ok for a while until the next burst.

On Mar 23, 2024, at 5:14 PM, Ovidiu Sas &lt;osas(a)voipembedded.com&gt; wrote: But if you are dealing with network jitter and several clients pumping traffic at different rates, for a short period of time all the workers will be busy while new SIP messages will pile up and the default queue will not be able to hold all of them.

An increased UDP queue will hold all the SIP messages giving time the workers to consume them. The idea is that you have the UDP queue empty all most of the time, and only when there’s a short temporary traffic burst, it will come to the rescue.

If the UDP queue of full most of the time, then increasing it obviously won’t help. The increased UDP queue works only for coping with short traffic bursts.