Hello
We are currently using EVAPI to push messages into kamailio from a go app. For the most part ift works without problems, messages get delivered into the event route and we use it to update presence status.
Recently we started noticing delays on the evapi processing. After 500 requests x min the pid corresponding to the EVAPI dispatcher gets pegged at 100% CPU and processing of requests starts slowing down until in some cases the connection just drops and we have to reconnect. After that pid drops, the dispatcher goes back to working.
Unfortunately the error we get back is very generic and is not telling us much
logger.go:39: ERR Handler returned error (write tcp 127.0.0.1:48448->127.0.0.1:8228: write: broken pipe)
This only happens on prod servers and with high load which makes it hard to debug. There is no other slowness or increase of traffic or drop SIP traffic, only evapi dispatcher at 100% and the EVAPI workers on idle.
Curious if anyone has run into this issue. We've tried different versions (5.4 , 5.5 and 5.6) and it happens on all of them.
Any feedback appreciated ! Thanks !
Hi Sebastian,
This is very good information, thank you for reporting.
1) Do you think this phenomenon corresponds to pushing messages into Kamailio and actuating the event_route to process them? Or might it be triggered by activity in the other direction, e.g. sending EVAPI messages out of Kamailio?
2) Does the problem recede if the throughput drops below 500 RPM?
-- Alex
On Jan 30, 2023, at 5:40 PM, Sebastian Sastre sebastian.sastre@getweave.com wrote:
Hello
We are currently using EVAPI to push messages into kamailio from a go app. For the most part ift works without problems, messages get delivered into the event route and we use it to update presence status.
Recently we started noticing delays on the evapi processing. After 500 requests x min the pid corresponding to the EVAPI dispatcher gets pegged at 100% CPU and processing of requests starts slowing down until in some cases the connection just drops and we have to reconnect. After that pid drops, the dispatcher goes back to working.
Unfortunately the error we get back is very generic and is not telling us much logger.go:39: ERR Handler returned error (write tcp 127.0.0.1:48448->127.0.0.1:8228: write: broken pipe)
This only happens on prod servers and with high load which makes it hard to debug. There is no other slowness or increase of traffic or drop SIP traffic, only evapi dispatcher at 100% and the EVAPI workers on idle.
Curious if anyone has run into this issue. We've tried different versions (5.4 , 5.5 and 5.6) and it happens on all of them.
Any feedback appreciated ! Thanks !
Kamailio - Users Mailing List - Non Commercial Discussions To unsubscribe send an email to sr-users-leave@lists.kamailio.org Important: keep the mailing list in the recipients, do not reply only to the sender! Edit mailing list options or unsubscribe:
-
Alex Balashov -
Sebastian Sastre