Testing out KEMI functionality and running into performance issues. If I exceed 150 calls per second the network receive queue grows and Kamailio is unable to keep up with requests and they begin dropping.
KEMI script for testing is just doing a stateless reply to invites.
Using python3s module.
I've played with Kamailio child processes and memory allocations, but there is no impact. I've also attempted some buffer / memory tweaking at the OS level, again with no impact. Increasing CPU cores and even running the test on bare metal results in the same.
Example of receive queue at 150 calls per second - Netid State Recv-Q Send-Q Local Address:Port udp UNCONN 337280 0 x.x.x.x:5060
Just wondering if anyone has experienced similar issues or has an example of the performance they are seeing before I continue down this path.
Thanks,
- dan
Hello Daniel,
I am wondering if your issues are specific to KEMI, e.g. if you’ve also tried the same script logic with a native cfg and observing similar numbers? If it’s a simple script, you can maybe just repeat the same test. There were benchmarks done for KEMI some years ago which only showed a small performance difference.
Or do you have generic performance issues which you just happened to observe in your test with KEMI? In this case it would be more of a generic performance optimization topic.
Cheers,
Henning
-- Henning Westerholt – https://skalatan.de/blog/ Kamailio services – https://gilawa.comhttps://gilawa.com/
From: Daniel W. Graham via sr-users sr-users@lists.kamailio.org Sent: Sonntag, 12. Januar 2025 08:21 To: sr-users@lists.kamailio.org Cc: Daniel W. Graham dan@cmsinter.net Subject: [SR-Users] Performance issues with KEMI
Testing out KEMI functionality and running into performance issues. If I exceed 150 calls per second the network receive queue grows and Kamailio is unable to keep up with requests and they begin dropping.
KEMI script for testing is just doing a stateless reply to invites.
Using python3s module.
I've played with Kamailio child processes and memory allocations, but there is no impact. I've also attempted some buffer / memory tweaking at the OS level, again with no impact. Increasing CPU cores and even running the test on bare metal results in the same.
Example of receive queue at 150 calls per second - Netid State Recv-Q Send-Q Local Address:Port udp UNCONN 337280 0 x.x.x.x:5060
Just wondering if anyone has experienced similar issues or has an example of the performance they are seeing before I continue down this path.
Thanks,
- dan
I haven't converted this test to native configuration yet. That is my next step to rule out any general issues. I've never experienced any performance issues like this before and since this was my first test using KEMI I assumed it was related in some fashion. I've read through performance optimization related posts and documents that I've been able to find.
-dan
________________________________ From: Henning Westerholt hw@gilawa.com Sent: Sunday, January 12, 2025 03:52 To: Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org Cc: Daniel W. Graham dan@cmsinter.net Subject: RE: Performance issues with KEMI
Caution: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe
Hello Daniel,
I am wondering if your issues are specific to KEMI, e.g. if you’ve also tried the same script logic with a native cfg and observing similar numbers? If it’s a simple script, you can maybe just repeat the same test. There were benchmarks done for KEMI some years ago which only showed a small performance difference.
Or do you have generic performance issues which you just happened to observe in your test with KEMI? In this case it would be more of a generic performance optimization topic.
Cheers,
Henning
-- Henning Westerholt – https://skalatan.de/blog/ Kamailio services – https://gilawa.comhttps://gilawa.com/
From: Daniel W. Graham via sr-users sr-users@lists.kamailio.org Sent: Sonntag, 12. Januar 2025 08:21 To: sr-users@lists.kamailio.org Cc: Daniel W. Graham dan@cmsinter.net Subject: [SR-Users] Performance issues with KEMI
Testing out KEMI functionality and running into performance issues. If I exceed 150 calls per second the network receive queue grows and Kamailio is unable to keep up with requests and they begin dropping.
KEMI script for testing is just doing a stateless reply to invites.
Using python3s module.
I've played with Kamailio child processes and memory allocations, but there is no impact. I've also attempted some buffer / memory tweaking at the OS level, again with no impact. Increasing CPU cores and even running the test on bare metal results in the same.
Example of receive queue at 150 calls per second - Netid State Recv-Q Send-Q Local Address:Port udp UNCONN 337280 0 x.x.x.x:5060
Just wondering if anyone has experienced similar issues or has an example of the performance they are seeing before I continue down this path.
Thanks,
- dan
Native config resulted in over 10k calls per second without any drops.
I made some adjustments to the python config, looks like there was an issue with the way I had been using return. That is handling almost 10k calls now.
However, when adding in other logic such as database calls using python libraries, back to a few dozen calls before the queue grows and drops occur.
If performance is expected to be close to native, then I probably need to continue identifying configuration issues.
-dan
________________________________ From: Daniel W. Graham via sr-users sr-users@lists.kamailio.org Sent: Sunday, January 12, 2025 10:32 To: Henning Westerholt hw@gilawa.com; Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org Cc: Daniel W. Graham dan@cmsinter.net Subject: [SR-Users] Re: Performance issues with KEMI
Caution: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe
I haven't converted this test to native configuration yet. That is my next step to rule out any general issues. I've never experienced any performance issues like this before and since this was my first test using KEMI I assumed it was related in some fashion. I've read through performance optimization related posts and documents that I've been able to find.
-dan
________________________________ From: Henning Westerholt hw@gilawa.com Sent: Sunday, January 12, 2025 03:52 To: Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org Cc: Daniel W. Graham dan@cmsinter.net Subject: RE: Performance issues with KEMI
Caution: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe
Hello Daniel,
I am wondering if your issues are specific to KEMI, e.g. if you’ve also tried the same script logic with a native cfg and observing similar numbers? If it’s a simple script, you can maybe just repeat the same test. There were benchmarks done for KEMI some years ago which only showed a small performance difference.
Or do you have generic performance issues which you just happened to observe in your test with KEMI? In this case it would be more of a generic performance optimization topic.
Cheers,
Henning
--
Henning Westerholt – https://skalatan.de/blog/
Kamailio services – https://gilawa.comhttps://gilawa.com/
From: Daniel W. Graham via sr-users sr-users@lists.kamailio.org Sent: Sonntag, 12. Januar 2025 08:21 To: sr-users@lists.kamailio.org Cc: Daniel W. Graham dan@cmsinter.net Subject: [SR-Users] Performance issues with KEMI
Testing out KEMI functionality and running into performance issues. If I exceed 150 calls per second the network receive queue grows and Kamailio is unable to keep up with requests and they begin dropping.
KEMI script for testing is just doing a stateless reply to invites.
Using python3s module.
I've played with Kamailio child processes and memory allocations, but there is no impact. I've also attempted some buffer / memory tweaking at the OS level, again with no impact. Increasing CPU cores and even running the test on bare metal results in the same.
Example of receive queue at 150 calls per second -
Netid State Recv-Q Send-Q Local Address:Port
udp UNCONN 337280 0 x.x.x.x:5060
Just wondering if anyone has experienced similar issues or has an example of the performance they are seeing before I continue down this path.
Thanks,
- dan
Hello,
the performance of KEMI was compared when using Kamailio exported functions, in the way that a function implemented inside Kamailio has pretty much same performance when using it inside a kemi script or inside native config file. For example, if one uses KSR.auth.pv_auth_check(...) inside kemi script or pv_auth_check(...) inside kamailio.cfg, then the performance is very much the same.
Now, if you use a pure python/lua/... library inside your kemi script and it has poor performances, then of course kamailio is slowed down, it is nothing that can be done by kamailio code to improve that.
If you need to access database, sqlops functions are also exported to kemi.
To summarize, the goal was that the kemi engine does not add any (or at least not significant) performance to execute Kamailio-specific functions compared to execution by native config interpreter.
Cheers, Daniel
On 13.01.25 03:56, Daniel W. Graham via sr-users wrote:
Native config resulted in over 10k calls per second without any drops.
I made some adjustments to the python config, looks like there was an issue with the way I had been using return. That is handling almost 10k calls now.
However, when adding in other logic such as database calls using python libraries, back to a few dozen calls before the queue grows and drops occur.
If performance is expected to be close to native, then I probably need to continue identifying configuration issues.
-dan
*From:* Daniel W. Graham via sr-users sr-users@lists.kamailio.org *Sent:* Sunday, January 12, 2025 10:32 *To:* Henning Westerholt hw@gilawa.com; Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org *Cc:* Daniel W. Graham dan@cmsinter.net *Subject:* [SR-Users] Re: Performance issues with KEMI
*Caution:* This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe
I haven't converted this test to native configuration yet. That is my next step to rule out any general issues. I've never experienced any performance issues like this before and since this was my first test using KEMI I assumed it was related in some fashion. I've read through performance optimization related posts and documents that I've been able to find.
-dan
*From:* Henning Westerholt hw@gilawa.com *Sent:* Sunday, January 12, 2025 03:52 *To:* Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org *Cc:* Daniel W. Graham dan@cmsinter.net *Subject:* RE: Performance issues with KEMI
*Caution:* This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe
Hello Daniel,
I am wondering if your issues are specific to KEMI, e.g. if you’ve also tried the same script logic with a native cfg and observing similar numbers? If it’s a simple script, you can maybe just repeat the same test. There were benchmarks done for KEMI some years ago which only showed a small performance difference.
Or do you have generic performance issues which you just happened to observe in your test with KEMI? In this case it would be more of a generic performance optimization topic.
Cheers,
Henning
--
Henning Westerholt – https://skalatan.de/blog/ https://skalatan.de/blog/
Kamailio services – https://gilawa.com https://gilawa.com/
*From:* Daniel W. Graham via sr-users sr-users@lists.kamailio.org *Sent:* Sonntag, 12. Januar 2025 08:21 *To:* sr-users@lists.kamailio.org *Cc:* Daniel W. Graham dan@cmsinter.net *Subject:* [SR-Users] Performance issues with KEMI
Testing out KEMI functionality and running into performance issues. If I exceed 150 calls per second the network receive queue grows and Kamailio is unable to keep up with requests and they begin dropping.
KEMI script for testing is just doing a stateless reply to invites.
Using python3s module.
I've played with Kamailio child processes and memory allocations, but there is no impact. I've also attempted some buffer / memory tweaking at the OS level, again with no impact. Increasing CPU cores and even running the test on bare metal results in the same.
Example of receive queue at 150 calls per second -
Netid State Recv-Q Send-Q Local Address:Port
udp UNCONN 337280 0 x.x.x.x:5060
Just wondering if anyone has experienced similar issues or has an example of the performance they are seeing before I continue down this path.
Thanks,
- dan
Kamailio - Users Mailing List - Non Commercial Discussions -- sr-users@lists.kamailio.org To unsubscribe send an email to sr-users-leave@lists.kamailio.org Important: keep the mailing list in the recipients, do not reply only to the sender!
Yes, on the same page with all of this. Good to know performance is comparable and I can focus optimization efforts on non kemi functions.
In this case clickhouse access was needed, so that's where the impact is occurring now that kemi scripting issues are under control. Performance is continuing to improve.
Thanks Daniel
-dan
________________________________ From: Daniel-Constantin Mierla miconda@gmail.com Sent: Monday, January 13, 2025 02:31 To: Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org; Henning Westerholt hw@gilawa.com Cc: Daniel W. Graham dan@cmsinter.net Subject: Re: [SR-Users] Re: Performance issues with KEMI
Caution: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe
Hello,
the performance of KEMI was compared when using Kamailio exported functions, in the way that a function implemented inside Kamailio has pretty much same performance when using it inside a kemi script or inside native config file. For example, if one uses KSR.auth.pv_auth_check(...) inside kemi script or pv_auth_check(...) inside kamailio.cfg, then the performance is very much the same.
Now, if you use a pure python/lua/... library inside your kemi script and it has poor performances, then of course kamailio is slowed down, it is nothing that can be done by kamailio code to improve that.
If you need to access database, sqlops functions are also exported to kemi.
To summarize, the goal was that the kemi engine does not add any (or at least not significant) performance to execute Kamailio-specific functions compared to execution by native config interpreter.
Cheers, Daniel
On 13.01.25 03:56, Daniel W. Graham via sr-users wrote: Native config resulted in over 10k calls per second without any drops.
I made some adjustments to the python config, looks like there was an issue with the way I had been using return. That is handling almost 10k calls now.
However, when adding in other logic such as database calls using python libraries, back to a few dozen calls before the queue grows and drops occur.
If performance is expected to be close to native, then I probably need to continue identifying configuration issues.
-dan
________________________________ From: Daniel W. Graham via sr-users sr-users@lists.kamailio.orgmailto:sr-users@lists.kamailio.org Sent: Sunday, January 12, 2025 10:32 To: Henning Westerholt hw@gilawa.commailto:hw@gilawa.com; Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.orgmailto:sr-users@lists.kamailio.org Cc: Daniel W. Graham dan@cmsinter.netmailto:dan@cmsinter.net Subject: [SR-Users] Re: Performance issues with KEMI
Caution: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe
I haven't converted this test to native configuration yet. That is my next step to rule out any general issues. I've never experienced any performance issues like this before and since this was my first test using KEMI I assumed it was related in some fashion. I've read through performance optimization related posts and documents that I've been able to find.
-dan
________________________________ From: Henning Westerholt hw@gilawa.commailto:hw@gilawa.com Sent: Sunday, January 12, 2025 03:52 To: Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.orgmailto:sr-users@lists.kamailio.org Cc: Daniel W. Graham dan@cmsinter.netmailto:dan@cmsinter.net Subject: RE: Performance issues with KEMI
Caution: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe
Hello Daniel,
I am wondering if your issues are specific to KEMI, e.g. if you’ve also tried the same script logic with a native cfg and observing similar numbers? If it’s a simple script, you can maybe just repeat the same test. There were benchmarks done for KEMI some years ago which only showed a small performance difference.
Or do you have generic performance issues which you just happened to observe in your test with KEMI? In this case it would be more of a generic performance optimization topic.
Cheers,
Henning
--
Henning Westerholt – https://skalatan.de/blog/
Kamailio services – https://gilawa.comhttps://gilawa.com/
From: Daniel W. Graham via sr-users sr-users@lists.kamailio.orgmailto:sr-users@lists.kamailio.org Sent: Sonntag, 12. Januar 2025 08:21 To: sr-users@lists.kamailio.orgmailto:sr-users@lists.kamailio.org Cc: Daniel W. Graham dan@cmsinter.netmailto:dan@cmsinter.net Subject: [SR-Users] Performance issues with KEMI
Testing out KEMI functionality and running into performance issues. If I exceed 150 calls per second the network receive queue grows and Kamailio is unable to keep up with requests and they begin dropping.
KEMI script for testing is just doing a stateless reply to invites.
Using python3s module.
I've played with Kamailio child processes and memory allocations, but there is no impact. I've also attempted some buffer / memory tweaking at the OS level, again with no impact. Increasing CPU cores and even running the test on bare metal results in the same.
Example of receive queue at 150 calls per second -
Netid State Recv-Q Send-Q Local Address:Port
udp UNCONN 337280 0 x.x.x.x:5060
Just wondering if anyone has experienced similar issues or has an example of the performance they are seeing before I continue down this path.
Thanks,
- dan
__________________________________________________________ Kamailio - Users Mailing List - Non Commercial Discussions -- sr-users@lists.kamailio.orgmailto:sr-users@lists.kamailio.org To unsubscribe send an email to sr-users-leave@lists.kamailio.orgmailto:sr-users-leave@lists.kamailio.org Important: keep the mailing list in the recipients, do not reply only to the sender!
-- Daniel-Constantin Mierla (@ asipto.com) twitter.com/miconda -- linkedin.com/in/miconda Kamailio Consultancy, Training and Development Services -- asipto.com
You can do pull requests with what you think it worth adding to the kemi tutorial:
- https://github.com/kamailio/kamailio-docs/tree/master/kamailio-kemi-framewor...
Or the docs of the kemi modules.
Cheers, Daniel
On 14.01.25 13:49, Alex Balashov via sr-users wrote:
Daniel-Constantin’s point that performance claims and commitments can only be applied to native functions probably should be loudly underscored in the KEMI documentation, as well as in the documentation for the relevant config interpreter modules.
Many users will not understand Kamailio architecture well enough, and will assume that because KEMI unlocks the ability to use general language runtimes and third-party libraries (with unknown concurrency architecture and performance characteristics), e.g. in Python, it also allows one to benefit from Kamailio’s high performance with anything that one might embed in it, in any language.
— Alex
— Sent from my iPad Web: https://evaristesys.com
On 14 Jan 2025, at 05:12, Daniel W. Graham via sr-users sr-users@lists.kamailio.org wrote:
Yes, on the same page with all of this. Good to know performance is comparable and I can focus optimization efforts on non kemi functions.
In this case clickhouse access was needed, so that's where the impact is occurring now that kemi scripting issues are under control. Performance is continuing to improve.
Thanks Daniel
-dan
*From:* Daniel-Constantin Mierla miconda@gmail.com *Sent:* Monday, January 13, 2025 02:31 *To:* Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org; Henning Westerholt hw@gilawa.com *Cc:* Daniel W. Graham dan@cmsinter.net *Subject:* Re: [SR-Users] Re: Performance issues with KEMI
*Caution:* This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe
Hello,
the performance of KEMI was compared when using Kamailio exported functions, in the way that a function implemented inside Kamailio has pretty much same performance when using it inside a kemi script or inside native config file. For example, if one uses KSR.auth.pv_auth_check(...) inside kemi script or pv_auth_check(...) inside kamailio.cfg, then the performance is very much the same.
Now, if you use a pure python/lua/... library inside your kemi script and it has poor performances, then of course kamailio is slowed down, it is nothing that can be done by kamailio code to improve that.
If you need to access database, sqlops functions are also exported to kemi.
To summarize, the goal was that the kemi engine does not add any (or at least not significant) performance to execute Kamailio-specific functions compared to execution by native config interpreter.
Cheers, Daniel
On 13.01.25 03:56, Daniel W. Graham via sr-users wrote:
Native config resulted in over 10k calls per second without any drops. I made some adjustments to the python config, looks like there was an issue with the way I had been using return. That is handling almost 10k calls now. However, when adding in other logic such as database calls using python libraries, back to a few dozen calls before the queue grows and drops occur. If performance is expected to be close to native, then I probably need to continue identifying configuration issues. -dan ------------------------------------------------------------------------ *From:* Daniel W. Graham via sr-users <sr-users@lists.kamailio.org> <mailto:sr-users@lists.kamailio.org> *Sent:* Sunday, January 12, 2025 10:32 *To:* Henning Westerholt <hw@gilawa.com> <mailto:hw@gilawa.com>; Kamailio (SER) - Users Mailing List <sr-users@lists.kamailio.org> <mailto:sr-users@lists.kamailio.org> *Cc:* Daniel W. Graham <dan@cmsinter.net> <mailto:dan@cmsinter.net> *Subject:* [SR-Users] Re: Performance issues with KEMI *Caution:* This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe I haven't converted this test to native configuration yet. That is my next step to rule out any general issues. I've never experienced any performance issues like this before and since this was my first test using KEMI I assumed it was related in some fashion. I've read through performance optimization related posts and documents that I've been able to find. -dan ------------------------------------------------------------------------ *From:* Henning Westerholt <hw@gilawa.com> <mailto:hw@gilawa.com> *Sent:* Sunday, January 12, 2025 03:52 *To:* Kamailio (SER) - Users Mailing List <sr-users@lists.kamailio.org> <mailto:sr-users@lists.kamailio.org> *Cc:* Daniel W. Graham <dan@cmsinter.net> <mailto:dan@cmsinter.net> *Subject:* RE: Performance issues with KEMI *Caution:* This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe Hello Daniel, I am wondering if your issues are specific to KEMI, e.g. if you’ve also tried the same script logic with a native cfg and observing similar numbers? If it’s a simple script, you can maybe just repeat the same test. There were benchmarks done for KEMI some years ago which only showed a small performance difference. Or do you have generic performance issues which you just happened to observe in your test with KEMI? In this case it would be more of a generic performance optimization topic. Cheers, Henning -- Henning Westerholt – https://skalatan.de/blog/ <https://skalatan.de/blog/> Kamailio services – https://gilawa.com <https://gilawa.com/> *From:* Daniel W. Graham via sr-users <sr-users@lists.kamailio.org> <mailto:sr-users@lists.kamailio.org> *Sent:* Sonntag, 12. Januar 2025 08:21 *To:* sr-users@lists.kamailio.org <mailto:sr-users@lists.kamailio.org> *Cc:* Daniel W. Graham <dan@cmsinter.net> <mailto:dan@cmsinter.net> *Subject:* [SR-Users] Performance issues with KEMI Testing out KEMI functionality and running into performance issues. If I exceed 150 calls per second the network receive queue grows and Kamailio is unable to keep up with requests and they begin dropping. KEMI script for testing is just doing a stateless reply to invites. Using python3s module. I've played with Kamailio child processes and memory allocations, but there is no impact. I've also attempted some buffer / memory tweaking at the OS level, again with no impact. Increasing CPU cores and even running the test on bare metal results in the same. Example of receive queue at 150 calls per second - Netid State Recv-Q Send-Q Local Address:Port udp UNCONN 337280 0 x.x.x.x:5060 Just wondering if anyone has experienced similar issues or has an example of the performance they are seeing before I continue down this path. Thanks, - dan __________________________________________________________ Kamailio - Users Mailing List - Non Commercial Discussions -- sr-users@lists.kamailio.org <mailto:sr-users@lists.kamailio.org> To unsubscribe send an email to sr-users-leave@lists.kamailio.org <mailto:sr-users-leave@lists.kamailio.org> Important: keep the mailing list in the recipients, do not reply only to the sender!-- Daniel-Constantin Mierla (@ asipto.com) twitter.com/miconda -- linkedin.com/in/miconda Kamailio Consultancy, Training and Development Services -- asipto.com __________________________________________________________ Kamailio - Users Mailing List - Non Commercial Discussions -- sr-users@lists.kamailio.org To unsubscribe send an email to sr-users-leave@lists.kamailio.org Important: keep the mailing list in the recipients, do not reply only to the sender!
Kamailio - Users Mailing List - Non Commercial Discussions -- sr-users@lists.kamailio.org To unsubscribe send an email to sr-users-leave@lists.kamailio.org Important: keep the mailing list in the recipients, do not reply only to the sender!
Hello Daniel,
great that you solved it for the existing KEMI configuration with some adaptions.
Cheers,
Henning
From: Daniel W. Graham dan@cmsinter.net Sent: Montag, 13. Januar 2025 03:57 To: Henning Westerholt hw@gilawa.com; Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org Subject: Re: Performance issues with KEMI
Native config resulted in over 10k calls per second without any drops.
I made some adjustments to the python config, looks like there was an issue with the way I had been using return. That is handling almost 10k calls now.
However, when adding in other logic such as database calls using python libraries, back to a few dozen calls before the queue grows and drops occur.
If performance is expected to be close to native, then I probably need to continue identifying configuration issues.
-dan
________________________________ From: Daniel W. Graham via sr-users <sr-users@lists.kamailio.orgmailto:sr-users@lists.kamailio.org> Sent: Sunday, January 12, 2025 10:32 To: Henning Westerholt <hw@gilawa.commailto:hw@gilawa.com>; Kamailio (SER) - Users Mailing List <sr-users@lists.kamailio.orgmailto:sr-users@lists.kamailio.org> Cc: Daniel W. Graham <dan@cmsinter.netmailto:dan@cmsinter.net> Subject: [SR-Users] Re: Performance issues with KEMI
Caution: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe
I haven't converted this test to native configuration yet. That is my next step to rule out any general issues. I've never experienced any performance issues like this before and since this was my first test using KEMI I assumed it was related in some fashion. I've read through performance optimization related posts and documents that I've been able to find.
-dan
________________________________ From: Henning Westerholt <hw@gilawa.commailto:hw@gilawa.com> Sent: Sunday, January 12, 2025 03:52 To: Kamailio (SER) - Users Mailing List <sr-users@lists.kamailio.orgmailto:sr-users@lists.kamailio.org> Cc: Daniel W. Graham <dan@cmsinter.netmailto:dan@cmsinter.net> Subject: RE: Performance issues with KEMI
Caution: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe
Hello Daniel,
I am wondering if your issues are specific to KEMI, e.g. if you’ve also tried the same script logic with a native cfg and observing similar numbers? If it’s a simple script, you can maybe just repeat the same test. There were benchmarks done for KEMI some years ago which only showed a small performance difference.
Or do you have generic performance issues which you just happened to observe in your test with KEMI? In this case it would be more of a generic performance optimization topic.
Cheers,
Henning
--
Henning Westerholt – https://skalatan.de/blog/
Kamailio services – https://gilawa.comhttps://gilawa.com/
From: Daniel W. Graham via sr-users <sr-users@lists.kamailio.orgmailto:sr-users@lists.kamailio.org> Sent: Sonntag, 12. Januar 2025 08:21 To: sr-users@lists.kamailio.orgmailto:sr-users@lists.kamailio.org Cc: Daniel W. Graham <dan@cmsinter.netmailto:dan@cmsinter.net> Subject: [SR-Users] Performance issues with KEMI
Testing out KEMI functionality and running into performance issues. If I exceed 150 calls per second the network receive queue grows and Kamailio is unable to keep up with requests and they begin dropping.
KEMI script for testing is just doing a stateless reply to invites.
Using python3s module.
I've played with Kamailio child processes and memory allocations, but there is no impact. I've also attempted some buffer / memory tweaking at the OS level, again with no impact. Increasing CPU cores and even running the test on bare metal results in the same.
Example of receive queue at 150 calls per second -
Netid State Recv-Q Send-Q Local Address:Port
udp UNCONN 337280 0 x.x.x.x:5060
Just wondering if anyone has experienced similar issues or has an example of the performance they are seeing before I continue down this path.
Thanks,
- dan
I made some adjustments to the python config, looks like there was an issue with the way I had been using return. That is handling almost 10k calls now.
Can you share more details on this please? We're migrating to Python3s KEMI, just to make sure we don't run into some known issue. Thanks
пн, 13 янв. 2025 г. в 06:35, Daniel W. Graham via sr-users < sr-users@lists.kamailio.org>:
Native config resulted in over 10k calls per second without any drops.
I made some adjustments to the python config, looks like there was an issue with the way I had been using return. That is handling almost 10k calls now.
However, when adding in other logic such as database calls using python libraries, back to a few dozen calls before the queue grows and drops occur.
If performance is expected to be close to native, then I probably need to continue identifying configuration issues.
-dan
*From:* Daniel W. Graham via sr-users sr-users@lists.kamailio.org *Sent:* Sunday, January 12, 2025 10:32 *To:* Henning Westerholt hw@gilawa.com; Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org *Cc:* Daniel W. Graham dan@cmsinter.net *Subject:* [SR-Users] Re: Performance issues with KEMI
*Caution:* This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe
I haven't converted this test to native configuration yet. That is my next step to rule out any general issues. I've never experienced any performance issues like this before and since this was my first test using KEMI I assumed it was related in some fashion. I've read through performance optimization related posts and documents that I've been able to find.
-dan
*From:* Henning Westerholt hw@gilawa.com *Sent:* Sunday, January 12, 2025 03:52 *To:* Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org *Cc:* Daniel W. Graham dan@cmsinter.net *Subject:* RE: Performance issues with KEMI
*Caution:* This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe
Hello Daniel,
I am wondering if your issues are specific to KEMI, e.g. if you’ve also tried the same script logic with a native cfg and observing similar numbers? If it’s a simple script, you can maybe just repeat the same test. There were benchmarks done for KEMI some years ago which only showed a small performance difference.
Or do you have generic performance issues which you just happened to observe in your test with KEMI? In this case it would be more of a generic performance optimization topic.
Cheers,
Henning
--
Henning Westerholt – https://skalatan.de/blog/
Kamailio services – https://gilawa.com
*From:* Daniel W. Graham via sr-users sr-users@lists.kamailio.org *Sent:* Sonntag, 12. Januar 2025 08:21 *To:* sr-users@lists.kamailio.org *Cc:* Daniel W. Graham dan@cmsinter.net *Subject:* [SR-Users] Performance issues with KEMI
Testing out KEMI functionality and running into performance issues. If I exceed 150 calls per second the network receive queue grows and Kamailio is unable to keep up with requests and they begin dropping.
KEMI script for testing is just doing a stateless reply to invites.
Using python3s module.
I've played with Kamailio child processes and memory allocations, but there is no impact. I've also attempted some buffer / memory tweaking at the OS level, again with no impact. Increasing CPU cores and even running the test on bare metal results in the same.
Example of receive queue at 150 calls per second -
Netid State Recv-Q Send-Q Local Address:Port
udp UNCONN 337280 0 x.x.x.x:5060
Just wondering if anyone has experienced similar issues or has an example of the performance they are seeing before I continue down this path.
Thanks,
- dan
Kamailio - Users Mailing List - Non Commercial Discussions -- sr-users@lists.kamailio.org To unsubscribe send an email to sr-users-leave@lists.kamailio.org Important: keep the mailing list in the recipients, do not reply only to the sender!
-
Alex Balashov -
Daniel W. Graham -
Daniel-Constantin Mierla -
Dmitry Sytchev -
Henning Westerholt