Hello, I'm exploring the evapi module for my kamailio to interface with an external node.js app for third party stuff like AAA, billing engine tasks, notifications and so on. I followed and took some ideas from the rtjson and evapi tutorial found here( http://kb.asipto.com/kamailio:k43-async-sip-routing-nodejs) to build the node.js app consuming events. When I stress tested the scenario using SIPp and tried sending a lot of events at 300-350cps from Kamailio, I noticed that at times the client is receiving 2-3 events in a single message together although I do event_sync_relay once per SIP message received and have netstrings enabled. I believe this is a typical behavior of TCP and needs to be handled by the client using some kind of Netstring handler. Please correct me if I'm wrong. And hence I'd like to know what particularly needs to be taken care of while writing a client that is listening for events on raw tcp socket and how does kamailio handle this situation while receiving messages over TCP socket?? Does kamailio recognize the end of netstring properly on evapi:message-received and give exactly one message to take care of on every "message-received" event or should that be handled in the script somewhere !! I also referred cgrates client over evapi example which is written in GO, but I couldnt find them handling TCP streams clearly either. I'd really appreciate some expert suggestion here to make an informed decision on using the evapi module for a large scale solution.
Thanks,
- Jayesh
Hello,
tcb is stream protocol and several messages can be queued on the pipe at the same time. That is the reason for netstring format, to be able to easily detect the boundaries of each message. If netstring format is enabled and kamailio receives several messages at once, it splits them and for each is executing the event route.
If netstring format is not used, the kamailio is executing the event route with the entire content that was read at once from the tcp connection.
Cheers, Daniel
On 09/09/15 22:01, Jayesh Nambiar wrote:
Hello, I'm exploring the evapi module for my kamailio to interface with an external node.js app for third party stuff like AAA, billing engine tasks, notifications and so on. I followed and took some ideas from the rtjson and evapi tutorial found here(http://kb.asipto.com/kamailio:k43-async-sip-routing-nodejs) to build the node.js app consuming events. When I stress tested the scenario using SIPp and tried sending a lot of events at 300-350cps from Kamailio, I noticed that at times the client is receiving 2-3 events in a single message together although I do event_sync_relay once per SIP message received and have netstrings enabled. I believe this is a typical behavior of TCP and needs to be handled by the client using some kind of Netstring handler. Please correct me if I'm wrong. And hence I'd like to know what particularly needs to be taken care of while writing a client that is listening for events on raw tcp socket and how does kamailio handle this situation while receiving messages over TCP socket?? Does kamailio recognize the end of netstring properly on evapi:message-received and give exactly one message to take care of on every "message-received" event or should that be handled in the script somewhere !! I also referred cgrates client over evapi example which is written in GO, but I couldnt find them handling TCP streams clearly either. I'd really appreciate some expert suggestion here to make an informed decision on using the evapi module for a large scale solution.
Thanks,
- Jayesh
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
Hi Daniel, Thanks for the quick response. So if I do not use Netstrings, does Kamailio allow me to create a custom logic in the script. For eg. if I decide to use newline as a delimiter, can I keep buffering the message until I encounter the delimiter from the event route and then execute whatever I have to within the script??
Thanks,
- Jayesh
On Thu, Sep 10, 2015 at 1:29 PM, Daniel-Constantin Mierla <miconda@gmail.com
wrote:
Hello,
tcb is stream protocol and several messages can be queued on the pipe at the same time. That is the reason for netstring format, to be able to easily detect the boundaries of each message. If netstring format is enabled and kamailio receives several messages at once, it splits them and for each is executing the event route.
If netstring format is not used, the kamailio is executing the event route with the entire content that was read at once from the tcp connection.
Cheers, Daniel
On 09/09/15 22:01, Jayesh Nambiar wrote:
Hello, I'm exploring the evapi module for my kamailio to interface with an external node.js app for third party stuff like AAA, billing engine tasks, notifications and so on. I followed and took some ideas from the rtjson and evapi tutorial found here( http://kb.asipto.com/kamailio:k43-async-sip-routing-nodejs http://kb.asipto.com/kamailio:k43-async-sip-routing-nodejs) to build the node.js app consuming events. When I stress tested the scenario using SIPp and tried sending a lot of events at 300-350cps from Kamailio, I noticed that at times the client is receiving 2-3 events in a single message together although I do event_sync_relay once per SIP message received and have netstrings enabled. I believe this is a typical behavior of TCP and needs to be handled by the client using some kind of Netstring handler. Please correct me if I'm wrong. And hence I'd like to know what particularly needs to be taken care of while writing a client that is listening for events on raw tcp socket and how does kamailio handle this situation while receiving messages over TCP socket?? Does kamailio recognize the end of netstring properly on evapi:message-received and give exactly one message to take care of on every "message-received" event or should that be handled in the script somewhere !! I also referred cgrates client over evapi example which is written in GO, but I couldnt find them handling TCP streams clearly either. I'd really appreciate some expert suggestion here to make an informed decision on using the evapi module for a large scale solution.
Thanks,
- Jayesh
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing listsr-users@lists.sip-router.orghttp://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
-- Daniel-Constantin Mierlahttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
Hello Daniel, After further testing with evapi module, I figured that when Netstrings are used, an event route is invoked individually for each message even if if multiple netstring messages are received in a single TCP packet. But this doesn't work effectively when a single proper message is split-up in two packets. For Example, if a message arrives as: 12:Hello World!, 12:Hello World!, 12:Hello World! in a single packet, kamailio properly invokes the event route "evapi:message-received" thrice for every individual proper netstring message. But if the first packet contains: 12:Hello World!, 12:Hello And Second Packet contains: World!, 12:Hello World! the event route is invoked only once !!
The above pattern is very much possible while sending and receiving packets over TCP Socket. Our tests for receiving an approximately 150 byte message over evapi socket at the rate of roughly 1000cps causes a lot of real events to be missed because of the above problem. You can never be sure when TCP will split messages in different chunks. This definitely looks like a bug which makes it not very reliable at large scale deployments. Would really appreciate your inputs on this. Thanks;
- Jayesh
On Thu, Sep 10, 2015 at 4:01 PM, Jayesh Nambiar jayesh1017@gmail.com wrote:
Hi Daniel, Thanks for the quick response. So if I do not use Netstrings, does Kamailio allow me to create a custom logic in the script. For eg. if I decide to use newline as a delimiter, can I keep buffering the message until I encounter the delimiter from the event route and then execute whatever I have to within the script??
Thanks,
- Jayesh
On Thu, Sep 10, 2015 at 1:29 PM, Daniel-Constantin Mierla < miconda@gmail.com> wrote:
Hello,
tcb is stream protocol and several messages can be queued on the pipe at the same time. That is the reason for netstring format, to be able to easily detect the boundaries of each message. If netstring format is enabled and kamailio receives several messages at once, it splits them and for each is executing the event route.
If netstring format is not used, the kamailio is executing the event route with the entire content that was read at once from the tcp connection.
Cheers, Daniel
On 09/09/15 22:01, Jayesh Nambiar wrote:
Hello, I'm exploring the evapi module for my kamailio to interface with an external node.js app for third party stuff like AAA, billing engine tasks, notifications and so on. I followed and took some ideas from the rtjson and evapi tutorial found here( http://kb.asipto.com/kamailio:k43-async-sip-routing-nodejs http://kb.asipto.com/kamailio:k43-async-sip-routing-nodejs) to build the node.js app consuming events. When I stress tested the scenario using SIPp and tried sending a lot of events at 300-350cps from Kamailio, I noticed that at times the client is receiving 2-3 events in a single message together although I do event_sync_relay once per SIP message received and have netstrings enabled. I believe this is a typical behavior of TCP and needs to be handled by the client using some kind of Netstring handler. Please correct me if I'm wrong. And hence I'd like to know what particularly needs to be taken care of while writing a client that is listening for events on raw tcp socket and how does kamailio handle this situation while receiving messages over TCP socket?? Does kamailio recognize the end of netstring properly on evapi:message-received and give exactly one message to take care of on every "message-received" event or should that be handled in the script somewhere !! I also referred cgrates client over evapi example which is written in GO, but I couldnt find them handling TCP streams clearly either. I'd really appreciate some expert suggestion here to make an informed decision on using the evapi module for a large scale solution.
Thanks,
- Jayesh
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing listsr-users@lists.sip-router.orghttp://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
-- Daniel-Constantin Mierlahttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
Do let me know if I should run some specific tests for you all to get a little more clarity and information and I'd be happy to do those test and update you with results. I eagerly want to take a positive decision on going ahead with evapi module as it looks very promising. Only if we can solve these minor issues.
Thanks,
- Jayesh
On Tue, Sep 15, 2015 at 2:30 AM, Jayesh Nambiar jayesh1017@gmail.com wrote:
Hello Daniel, After further testing with evapi module, I figured that when Netstrings are used, an event route is invoked individually for each message even if if multiple netstring messages are received in a single TCP packet. But this doesn't work effectively when a single proper message is split-up in two packets. For Example, if a message arrives as: 12:Hello World!, 12:Hello World!, 12:Hello World! in a single packet, kamailio properly invokes the event route "evapi:message-received" thrice for every individual proper netstring message. But if the first packet contains: 12:Hello World!, 12:Hello And Second Packet contains: World!, 12:Hello World! the event route is invoked only once !!
The above pattern is very much possible while sending and receiving packets over TCP Socket. Our tests for receiving an approximately 150 byte message over evapi socket at the rate of roughly 1000cps causes a lot of real events to be missed because of the above problem. You can never be sure when TCP will split messages in different chunks. This definitely looks like a bug which makes it not very reliable at large scale deployments. Would really appreciate your inputs on this. Thanks;
- Jayesh
On Thu, Sep 10, 2015 at 4:01 PM, Jayesh Nambiar jayesh1017@gmail.com wrote:
Hi Daniel, Thanks for the quick response. So if I do not use Netstrings, does Kamailio allow me to create a custom logic in the script. For eg. if I decide to use newline as a delimiter, can I keep buffering the message until I encounter the delimiter from the event route and then execute whatever I have to within the script??
Thanks,
- Jayesh
On Thu, Sep 10, 2015 at 1:29 PM, Daniel-Constantin Mierla < miconda@gmail.com> wrote:
Hello,
tcb is stream protocol and several messages can be queued on the pipe at the same time. That is the reason for netstring format, to be able to easily detect the boundaries of each message. If netstring format is enabled and kamailio receives several messages at once, it splits them and for each is executing the event route.
If netstring format is not used, the kamailio is executing the event route with the entire content that was read at once from the tcp connection.
Cheers, Daniel
On 09/09/15 22:01, Jayesh Nambiar wrote:
Hello, I'm exploring the evapi module for my kamailio to interface with an external node.js app for third party stuff like AAA, billing engine tasks, notifications and so on. I followed and took some ideas from the rtjson and evapi tutorial found here( http://kb.asipto.com/kamailio:k43-async-sip-routing-nodejs http://kb.asipto.com/kamailio:k43-async-sip-routing-nodejs) to build the node.js app consuming events. When I stress tested the scenario using SIPp and tried sending a lot of events at 300-350cps from Kamailio, I noticed that at times the client is receiving 2-3 events in a single message together although I do event_sync_relay once per SIP message received and have netstrings enabled. I believe this is a typical behavior of TCP and needs to be handled by the client using some kind of Netstring handler. Please correct me if I'm wrong. And hence I'd like to know what particularly needs to be taken care of while writing a client that is listening for events on raw tcp socket and how does kamailio handle this situation while receiving messages over TCP socket?? Does kamailio recognize the end of netstring properly on evapi:message-received and give exactly one message to take care of on every "message-received" event or should that be handled in the script somewhere !! I also referred cgrates client over evapi example which is written in GO, but I couldnt find them handling TCP streams clearly either. I'd really appreciate some expert suggestion here to make an informed decision on using the evapi module for a large scale solution.
Thanks,
- Jayesh
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing listsr-users@lists.sip-router.orghttp://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
-- Daniel-Constantin Mierlahttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
Hello,
I will look if there are options in libev to buffer data or try to implement a buffering mechanism locally for such cases.
Cheers, Daniel
On 14/09/15 23:00, Jayesh Nambiar wrote:
Hello Daniel, After further testing with evapi module, I figured that when Netstrings are used, an event route is invoked individually for each message even if if multiple netstring messages are received in a single TCP packet. But this doesn't work effectively when a single proper message is split-up in two packets. For Example, if a message arrives as: 12:Hello World!, 12:Hello World!, 12:Hello World! in a single packet, kamailio properly invokes the event route "evapi:message-received" thrice for every individual proper netstring message. But if the first packet contains: 12:Hello World!, 12:Hello And Second Packet contains: World!, 12:Hello World! the event route is invoked only once !!
The above pattern is very much possible while sending and receiving packets over TCP Socket. Our tests for receiving an approximately 150 byte message over evapi socket at the rate of roughly 1000cps causes a lot of real events to be missed because of the above problem. You can never be sure when TCP will split messages in different chunks. This definitely looks like a bug which makes it not very reliable at large scale deployments. Would really appreciate your inputs on this. Thanks;
- Jayesh
On Thu, Sep 10, 2015 at 4:01 PM, Jayesh Nambiar <jayesh1017@gmail.com mailto:jayesh1017@gmail.com> wrote:
Hi Daniel, Thanks for the quick response. So if I do not use Netstrings, does Kamailio allow me to create a custom logic in the script. For eg. if I decide to use newline as a delimiter, can I keep buffering the message until I encounter the delimiter from the event route and then execute whatever I have to within the script?? Thanks, - Jayesh On Thu, Sep 10, 2015 at 1:29 PM, Daniel-Constantin Mierla <miconda@gmail.com <mailto:miconda@gmail.com>> wrote: Hello, tcb is stream protocol and several messages can be queued on the pipe at the same time. That is the reason for netstring format, to be able to easily detect the boundaries of each message. If netstring format is enabled and kamailio receives several messages at once, it splits them and for each is executing the event route. If netstring format is not used, the kamailio is executing the event route with the entire content that was read at once from the tcp connection. Cheers, Daniel On 09/09/15 22:01, Jayesh Nambiar wrote:Hello, I'm exploring the evapi module for my kamailio to interface with an external node.js app for third party stuff like AAA, billing engine tasks, notifications and so on. I followed and took some ideas from the rtjson and evapi tutorial found here(http://kb.asipto.com/kamailio:k43-async-sip-routing-nodejs) to build the node.js app consuming events. When I stress tested the scenario using SIPp and tried sending a lot of events at 300-350cps from Kamailio, I noticed that at times the client is receiving 2-3 events in a single message together although I do event_sync_relay once per SIP message received and have netstrings enabled. I believe this is a typical behavior of TCP and needs to be handled by the client using some kind of Netstring handler. Please correct me if I'm wrong. And hence I'd like to know what particularly needs to be taken care of while writing a client that is listening for events on raw tcp socket and how does kamailio handle this situation while receiving messages over TCP socket?? Does kamailio recognize the end of netstring properly on evapi:message-received and give exactly one message to take care of on every "message-received" event or should that be handled in the script somewhere !! I also referred cgrates client over evapi example which is written in GO, but I couldnt find them handling TCP streams clearly either. I'd really appreciate some expert suggestion here to make an informed decision on using the evapi module for a large scale solution. Thanks, - Jayesh _______________________________________________ SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org <mailto:sr-users@lists.sip-router.org> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users-- Daniel-Constantin Mierla http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat _______________________________________________ SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org <mailto:sr-users@lists.sip-router.org> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
Hello,
I just pushed a patch to master branch that should cope with partial data received on tcp connection. No time to test at all, therefore any feedback will be appreciated.
Cheers, Daniel
On 15/09/15 14:52, Daniel-Constantin Mierla wrote:
Hello,
I will look if there are options in libev to buffer data or try to implement a buffering mechanism locally for such cases.
Cheers, Daniel
On 14/09/15 23:00, Jayesh Nambiar wrote:
Hello Daniel, After further testing with evapi module, I figured that when Netstrings are used, an event route is invoked individually for each message even if if multiple netstring messages are received in a single TCP packet. But this doesn't work effectively when a single proper message is split-up in two packets. For Example, if a message arrives as: 12:Hello World!, 12:Hello World!, 12:Hello World! in a single packet, kamailio properly invokes the event route "evapi:message-received" thrice for every individual proper netstring message. But if the first packet contains: 12:Hello World!, 12:Hello And Second Packet contains: World!, 12:Hello World! the event route is invoked only once !!
The above pattern is very much possible while sending and receiving packets over TCP Socket. Our tests for receiving an approximately 150 byte message over evapi socket at the rate of roughly 1000cps causes a lot of real events to be missed because of the above problem. You can never be sure when TCP will split messages in different chunks. This definitely looks like a bug which makes it not very reliable at large scale deployments. Would really appreciate your inputs on this. Thanks;
- Jayesh
On Thu, Sep 10, 2015 at 4:01 PM, Jayesh Nambiar <jayesh1017@gmail.com mailto:jayesh1017@gmail.com> wrote:
Hi Daniel, Thanks for the quick response. So if I do not use Netstrings, does Kamailio allow me to create a custom logic in the script. For eg. if I decide to use newline as a delimiter, can I keep buffering the message until I encounter the delimiter from the event route and then execute whatever I have to within the script?? Thanks, - Jayesh On Thu, Sep 10, 2015 at 1:29 PM, Daniel-Constantin Mierla <miconda@gmail.com> wrote: Hello, tcb is stream protocol and several messages can be queued on the pipe at the same time. That is the reason for netstring format, to be able to easily detect the boundaries of each message. If netstring format is enabled and kamailio receives several messages at once, it splits them and for each is executing the event route. If netstring format is not used, the kamailio is executing the event route with the entire content that was read at once from the tcp connection. Cheers, Daniel On 09/09/15 22:01, Jayesh Nambiar wrote:Hello, I'm exploring the evapi module for my kamailio to interface with an external node.js app for third party stuff like AAA, billing engine tasks, notifications and so on. I followed and took some ideas from the rtjson and evapi tutorial found here(http://kb.asipto.com/kamailio:k43-async-sip-routing-nodejs) to build the node.js app consuming events. When I stress tested the scenario using SIPp and tried sending a lot of events at 300-350cps from Kamailio, I noticed that at times the client is receiving 2-3 events in a single message together although I do event_sync_relay once per SIP message received and have netstrings enabled. I believe this is a typical behavior of TCP and needs to be handled by the client using some kind of Netstring handler. Please correct me if I'm wrong. And hence I'd like to know what particularly needs to be taken care of while writing a client that is listening for events on raw tcp socket and how does kamailio handle this situation while receiving messages over TCP socket?? Does kamailio recognize the end of netstring properly on evapi:message-received and give exactly one message to take care of on every "message-received" event or should that be handled in the script somewhere !! I also referred cgrates client over evapi example which is written in GO, but I couldnt find them handling TCP streams clearly either. I'd really appreciate some expert suggestion here to make an informed decision on using the evapi module for a large scale solution. Thanks, - Jayesh _______________________________________________ SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org <mailto:sr-users@lists.sip-router.org> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users-- Daniel-Constantin Mierla http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat _______________________________________________ SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org <mailto:sr-users@lists.sip-router.org> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users-- Daniel-Constantin Mierla http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
Will have this tested by tomorrow and will get back to you. Thanks.
- Jayesh
On Fri, Sep 18, 2015 at 1:41 PM, Daniel-Constantin Mierla <miconda@gmail.com
wrote:
Hello,
I just pushed a patch to master branch that should cope with partial data received on tcp connection. No time to test at all, therefore any feedback will be appreciated.
Cheers, Daniel
On 15/09/15 14:52, Daniel-Constantin Mierla wrote:
Hello,
I will look if there are options in libev to buffer data or try to implement a buffering mechanism locally for such cases.
Cheers, Daniel
On 14/09/15 23:00, Jayesh Nambiar wrote:
Hello Daniel, After further testing with evapi module, I figured that when Netstrings are used, an event route is invoked individually for each message even if if multiple netstring messages are received in a single TCP packet. But this doesn't work effectively when a single proper message is split-up in two packets. For Example, if a message arrives as: 12:Hello World!, 12:Hello World!, 12:Hello World! in a single packet, kamailio properly invokes the event route "evapi:message-received" thrice for every individual proper netstring message. But if the first packet contains: 12:Hello World!, 12:Hello And Second Packet contains: World!, 12:Hello World! the event route is invoked only once !!
The above pattern is very much possible while sending and receiving packets over TCP Socket. Our tests for receiving an approximately 150 byte message over evapi socket at the rate of roughly 1000cps causes a lot of real events to be missed because of the above problem. You can never be sure when TCP will split messages in different chunks. This definitely looks like a bug which makes it not very reliable at large scale deployments. Would really appreciate your inputs on this. Thanks;
- Jayesh
On Thu, Sep 10, 2015 at 4:01 PM, Jayesh Nambiar < jayesh1017@gmail.com jayesh1017@gmail.com> wrote:
Hi Daniel, Thanks for the quick response. So if I do not use Netstrings, does Kamailio allow me to create a custom logic in the script. For eg. if I decide to use newline as a delimiter, can I keep buffering the message until I encounter the delimiter from the event route and then execute whatever I have to within the script??
Thanks,
- Jayesh
On Thu, Sep 10, 2015 at 1:29 PM, Daniel-Constantin Mierla < miconda@gmail.commiconda@gmail.com> wrote:
Hello,
tcb is stream protocol and several messages can be queued on the pipe at the same time. That is the reason for netstring format, to be able to easily detect the boundaries of each message. If netstring format is enabled and kamailio receives several messages at once, it splits them and for each is executing the event route.
If netstring format is not used, the kamailio is executing the event route with the entire content that was read at once from the tcp connection.
Cheers, Daniel
On 09/09/15 22:01, Jayesh Nambiar wrote:
Hello, I'm exploring the evapi module for my kamailio to interface with an external node.js app for third party stuff like AAA, billing engine tasks, notifications and so on. I followed and took some ideas from the rtjson and evapi tutorial found here( http://kb.asipto.com/kamailio:k43-async-sip-routing-nodejs) to build the node.js app consuming events. When I stress tested the scenario using SIPp and tried sending a lot of events at 300-350cps from Kamailio, I noticed that at times the client is receiving 2-3 events in a single message together although I do event_sync_relay once per SIP message received and have netstrings enabled. I believe this is a typical behavior of TCP and needs to be handled by the client using some kind of Netstring handler. Please correct me if I'm wrong. And hence I'd like to know what particularly needs to be taken care of while writing a client that is listening for events on raw tcp socket and how does kamailio handle this situation while receiving messages over TCP socket?? Does kamailio recognize the end of netstring properly on evapi:message-received and give exactly one message to take care of on every "message-received" event or should that be handled in the script somewhere !! I also referred cgrates client over evapi example which is written in GO, but I couldnt find them handling TCP streams clearly either. I'd really appreciate some expert suggestion here to make an informed decision on using the evapi module for a large scale solution.
Thanks,
- Jayesh
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing listsr-users@lists.sip-router.orghttp://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
-- Daniel-Constantin Mierlahttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
-- Daniel-Constantin Mierlahttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
-- Daniel-Constantin Mierlahttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
Here are the tests that I did:
With the patch applied, I see that Kamailio is invoking event individually for each netstring even when they come in different chunks. But I did see instances where when there were complete netstrings in a single chunk; kamailio did not raise an event for them. Here's the illustration:
T 1.2.3.4:48873 -> 5.6.7.8:3927 [AP] 146:{"event":"REGISTER","tindex":"16916","tlabel":"1753048277","PhoneNumber":"20708","DeviceId":"abcd1234abcd1234","CallId":" 21220-3848@5.6.7.8 "},144:{"event":"REGISTER","tindex":"7954","tlabel":"254315075","PhoneNumber":"20709","DeviceId":"abcd1234abcd1234","CallId":" 21223-3848@5.6.7.8 "},145:{"event":"REGISTER","tindex":"64529","tlabel":"599481568","PhoneNumber":"20709","DeviceId":"abcd1234abcd1234","CallId":" 21222-3848@5.6.7.8"},145:{"event":"REGISTER","tindex":"46605","tlabel":"112015324","PhoneNumber":"20710","DeviceId"
T 1.2.3.4:48873 -> 5.6.7.8:3927 [AP] :"abcd1234abcd1234","CallId":"21224-3848@5.6.7.8"},
*The above two chunks contain 4 proper netstrings where the second chunk contains part of the 4th netstring. In this case Evapi properly raised 4 individual events.*
1.2.3.4:48873 -> 5.6.7.8:3927 [AP]
145:{"event":"REGISTER","tindex":"55567","tlabel":"627458699","PhoneNumber":"20711","DeviceId":"abcd1234abcd1234","CallId":" 21225-3848@5.6.7.8 "},143:{"event":"REGISTER","tindex":"28682","tlabel":"9676691","PhoneNumber":"20712","DeviceId":"abcd1234abcd1234","CallId":" 21226-3848@5.6.7.8"},
*This above chunk contains two complete netstrings but Kamailio never raised events for these two netstrings.*
For the events that were not raised I see proper Kamailio Logs which is: evapi_recv_client(): {0} [1.2.3.4:48873] - received [145:{"event":"REGISTER","tindex":"55567","tlabel":"627458699","PhoneNumber":"20711","DeviceId":"abcd1234abcd1234","CallId":" 21225-3848@5.6.7.8 "},143:{"event":"REGISTER","tindex":"28682","tlabel":"9676691","PhoneNumber":"20712","DeviceId":"abcd1234abcd1234","CallId":" 21226-3848@5.6.7.8"},]
But I don't see the logs that I've written in the script after the event was raised, which means Kamailio did not invoke events for these two netstrings.
So in my tests, out of 27232 messages sent, there were approximately 27 messages for which the events were not raised by Evapi. The rate of messages started at 500cps and I stopped after I saw missed events at around 1200cps.
Thanks, and do let me know for any further tests or information required about the same.
- Jayesh
On Fri, Sep 18, 2015 at 2:55 PM, Jayesh Nambiar jayesh1017@gmail.com wrote:
Will have this tested by tomorrow and will get back to you. Thanks.
- Jayesh
On Fri, Sep 18, 2015 at 1:41 PM, Daniel-Constantin Mierla < miconda@gmail.com> wrote:
Hello,
I just pushed a patch to master branch that should cope with partial data received on tcp connection. No time to test at all, therefore any feedback will be appreciated.
Cheers, Daniel
On 15/09/15 14:52, Daniel-Constantin Mierla wrote:
Hello,
I will look if there are options in libev to buffer data or try to implement a buffering mechanism locally for such cases.
Cheers, Daniel
On 14/09/15 23:00, Jayesh Nambiar wrote:
Hello Daniel, After further testing with evapi module, I figured that when Netstrings are used, an event route is invoked individually for each message even if if multiple netstring messages are received in a single TCP packet. But this doesn't work effectively when a single proper message is split-up in two packets. For Example, if a message arrives as: 12:Hello World!, 12:Hello World!, 12:Hello World! in a single packet, kamailio properly invokes the event route "evapi:message-received" thrice for every individual proper netstring message. But if the first packet contains: 12:Hello World!, 12:Hello And Second Packet contains: World!, 12:Hello World! the event route is invoked only once !!
The above pattern is very much possible while sending and receiving packets over TCP Socket. Our tests for receiving an approximately 150 byte message over evapi socket at the rate of roughly 1000cps causes a lot of real events to be missed because of the above problem. You can never be sure when TCP will split messages in different chunks. This definitely looks like a bug which makes it not very reliable at large scale deployments. Would really appreciate your inputs on this. Thanks;
- Jayesh
On Thu, Sep 10, 2015 at 4:01 PM, Jayesh Nambiar < jayesh1017@gmail.com jayesh1017@gmail.com> wrote:
Hi Daniel, Thanks for the quick response. So if I do not use Netstrings, does Kamailio allow me to create a custom logic in the script. For eg. if I decide to use newline as a delimiter, can I keep buffering the message until I encounter the delimiter from the event route and then execute whatever I have to within the script??
Thanks,
- Jayesh
On Thu, Sep 10, 2015 at 1:29 PM, Daniel-Constantin Mierla < miconda@gmail.commiconda@gmail.com> wrote:
Hello,
tcb is stream protocol and several messages can be queued on the pipe at the same time. That is the reason for netstring format, to be able to easily detect the boundaries of each message. If netstring format is enabled and kamailio receives several messages at once, it splits them and for each is executing the event route.
If netstring format is not used, the kamailio is executing the event route with the entire content that was read at once from the tcp connection.
Cheers, Daniel
On 09/09/15 22:01, Jayesh Nambiar wrote:
Hello, I'm exploring the evapi module for my kamailio to interface with an external node.js app for third party stuff like AAA, billing engine tasks, notifications and so on. I followed and took some ideas from the rtjson and evapi tutorial found here( http://kb.asipto.com/kamailio:k43-async-sip-routing-nodejs) to build the node.js app consuming events. When I stress tested the scenario using SIPp and tried sending a lot of events at 300-350cps from Kamailio, I noticed that at times the client is receiving 2-3 events in a single message together although I do event_sync_relay once per SIP message received and have netstrings enabled. I believe this is a typical behavior of TCP and needs to be handled by the client using some kind of Netstring handler. Please correct me if I'm wrong. And hence I'd like to know what particularly needs to be taken care of while writing a client that is listening for events on raw tcp socket and how does kamailio handle this situation while receiving messages over TCP socket?? Does kamailio recognize the end of netstring properly on evapi:message-received and give exactly one message to take care of on every "message-received" event or should that be handled in the script somewhere !! I also referred cgrates client over evapi example which is written in GO, but I couldnt find them handling TCP streams clearly either. I'd really appreciate some expert suggestion here to make an informed decision on using the evapi module for a large scale solution.
Thanks,
- Jayesh
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing listsr-users@lists.sip-router.orghttp://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
-- Daniel-Constantin Mierlahttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
-- Daniel-Constantin Mierlahttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
-- Daniel-Constantin Mierlahttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
I pushed a commit to add more debug message while processing received data. You can use debugger module to set a higher debug level for evapi module in order to see what happens.
I checked the netstring packet size and it is invalid (unless email stripped some white chars there) -- for example in:
145:{"event":"REGISTER","tindex":"55567","tlabel":"627458699","PhoneNumber":"20711","DeviceId":"abcd1234abcd1234","CallId":"21225-3848@5.6.7.8"},
the size is 141, not 145 -- it looks like the size includes the size itself plus the delimiters ':,'. The size is only the effective data, see:
https://en.wikipedia.org/wiki/Netstring
If evapi gets a packet with an invalid size, then it discards the buffer content.
See if the app on the other side of evapi connection builds netstrings with wrong size.
Cheers, Daniel
On 18/09/15 23:19, Jayesh Nambiar wrote:
Here are the tests that I did:
With the patch applied, I see that Kamailio is invoking event individually for each netstring even when they come in different chunks. But I did see instances where when there were complete netstrings in a single chunk; kamailio did not raise an event for them. Here's the illustration:
T 1.2.3.4:48873 http://1.2.3.4:48873 -> 5.6.7.8:3927 http://5.6.7.8:3927 [AP] 146:{"event":"REGISTER","tindex":"16916","tlabel":"1753048277","PhoneNumber":"20708","DeviceId":"abcd1234abcd1234","CallId":"21220-3848@5.6.7.8 mailto:21220-3848@5.6.7.8"},144:{"event":"REGISTER","tindex":"7954","tlabel":"254315075","PhoneNumber":"20709","DeviceId":"abcd1234abcd1234","CallId":"21223-3848@5.6.7.8 mailto:21223-3848@5.6.7.8"},145:{"event":"REGISTER","tindex":"64529","tlabel":"599481568","PhoneNumber":"20709","DeviceId":"abcd1234abcd1234","CallId":"21222-3848@5.6.7.8 mailto:21222-3848@5.6.7.8"},145:{"event":"REGISTER","tindex":"46605","tlabel":"112015324","PhoneNumber":"20710","DeviceId"
T 1.2.3.4:48873 http://1.2.3.4:48873 -> 5.6.7.8:3927 http://5.6.7.8:3927 [AP] :"abcd1234abcd1234","CallId":"21224-3848@5.6.7.8 mailto:21224-3848@5.6.7.8"},
*The above two chunks contain 4 proper netstrings where the second chunk contains part of the 4th netstring. In this case Evapi properly raised 4 individual events.*
1.2.3.4:48873 http://1.2.3.4:48873 -> 5.6.7.8:3927 http://5.6.7.8:3927 [AP]
145:{"event":"REGISTER","tindex":"55567","tlabel":"627458699","PhoneNumber":"20711","DeviceId":"abcd1234abcd1234","CallId":"21225-3848@5.6.7.8 mailto:21225-3848@5.6.7.8"},143:{"event":"REGISTER","tindex":"28682","tlabel":"9676691","PhoneNumber":"20712","DeviceId":"abcd1234abcd1234","CallId":"21226-3848@5.6.7.8 mailto:21226-3848@5.6.7.8"},
*This above chunk contains two complete netstrings but Kamailio never raised events for these two netstrings.*
For the events that were not raised I see proper Kamailio Logs which is: evapi_recv_client(): {0} [1.2.3.4:48873 http://1.2.3.4:48873] - received [145:{"event":"REGISTER","tindex":"55567","tlabel":"627458699","PhoneNumber":"20711","DeviceId":"abcd1234abcd1234","CallId":"21225-3848@5.6.7.8 mailto:21225-3848@5.6.7.8"},143:{"event":"REGISTER","tindex":"28682","tlabel":"9676691","PhoneNumber":"20712","DeviceId":"abcd1234abcd1234","CallId":"21226-3848@5.6.7.8 mailto:21226-3848@5.6.7.8"},]
But I don't see the logs that I've written in the script after the event was raised, which means Kamailio did not invoke events for these two netstrings.
So in my tests, out of 27232 messages sent, there were approximately 27 messages for which the events were not raised by Evapi. The rate of messages started at 500cps and I stopped after I saw missed events at around 1200cps.
Thanks, and do let me know for any further tests or information required about the same.
- Jayesh
On Fri, Sep 18, 2015 at 2:55 PM, Jayesh Nambiar <jayesh1017@gmail.com mailto:jayesh1017@gmail.com> wrote:
Will have this tested by tomorrow and will get back to you. Thanks. - Jayesh On Fri, Sep 18, 2015 at 1:41 PM, Daniel-Constantin Mierla <miconda@gmail.com <mailto:miconda@gmail.com>> wrote: Hello, I just pushed a patch to master branch that should cope with partial data received on tcp connection. No time to test at all, therefore any feedback will be appreciated. Cheers, Daniel On 15/09/15 14:52, Daniel-Constantin Mierla wrote:Hello, I will look if there are options in libev to buffer data or try to implement a buffering mechanism locally for such cases. Cheers, Daniel On 14/09/15 23:00, Jayesh Nambiar wrote:Hello Daniel, After further testing with evapi module, I figured that when Netstrings are used, an event route is invoked individually for each message even if if multiple netstring messages are received in a single TCP packet. But this doesn't work effectively when a single proper message is split-up in two packets. For Example, if a message arrives as: 12:Hello World!, 12:Hello World!, 12:Hello World! in a single packet, kamailio properly invokes the event route "evapi:message-received" thrice for every individual proper netstring message. But if the first packet contains: 12:Hello World!, 12:Hello And Second Packet contains: World!, 12:Hello World! the event route is invoked only once !! The above pattern is very much possible while sending and receiving packets over TCP Socket. Our tests for receiving an approximately 150 byte message over evapi socket at the rate of roughly 1000cps causes a lot of real events to be missed because of the above problem. You can never be sure when TCP will split messages in different chunks. This definitely looks like a bug which makes it not very reliable at large scale deployments. Would really appreciate your inputs on this. Thanks; - Jayesh On Thu, Sep 10, 2015 at 4:01 PM, Jayesh Nambiar <jayesh1017@gmail.com <mailto:jayesh1017@gmail.com>> wrote: Hi Daniel, Thanks for the quick response. So if I do not use Netstrings, does Kamailio allow me to create a custom logic in the script. For eg. if I decide to use newline as a delimiter, can I keep buffering the message until I encounter the delimiter from the event route and then execute whatever I have to within the script?? Thanks, - Jayesh On Thu, Sep 10, 2015 at 1:29 PM, Daniel-Constantin Mierla <miconda@gmail.com <mailto:miconda@gmail.com>> wrote: Hello, tcb is stream protocol and several messages can be queued on the pipe at the same time. That is the reason for netstring format, to be able to easily detect the boundaries of each message. If netstring format is enabled and kamailio receives several messages at once, it splits them and for each is executing the event route. If netstring format is not used, the kamailio is executing the event route with the entire content that was read at once from the tcp connection. Cheers, Daniel On 09/09/15 22:01, Jayesh Nambiar wrote:Hello, I'm exploring the evapi module for my kamailio to interface with an external node.js app for third party stuff like AAA, billing engine tasks, notifications and so on. I followed and took some ideas from the rtjson and evapi tutorial found here(http://kb.asipto.com/kamailio:k43-async-sip-routing-nodejs) to build the node.js app consuming events. When I stress tested the scenario using SIPp and tried sending a lot of events at 300-350cps from Kamailio, I noticed that at times the client is receiving 2-3 events in a single message together although I do event_sync_relay once per SIP message received and have netstrings enabled. I believe this is a typical behavior of TCP and needs to be handled by the client using some kind of Netstring handler. Please correct me if I'm wrong. And hence I'd like to know what particularly needs to be taken care of while writing a client that is listening for events on raw tcp socket and how does kamailio handle this situation while receiving messages over TCP socket?? Does kamailio recognize the end of netstring properly on evapi:message-received and give exactly one message to take care of on every "message-received" event or should that be handled in the script somewhere !! I also referred cgrates client over evapi example which is written in GO, but I couldnt find them handling TCP streams clearly either. I'd really appreciate some expert suggestion here to make an informed decision on using the evapi module for a large scale solution. Thanks, - Jayesh _______________________________________________ SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org <mailto:sr-users@lists.sip-router.org> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users-- Daniel-Constantin Mierla http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat _______________________________________________ SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org <mailto:sr-users@lists.sip-router.org> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users-- Daniel-Constantin Mierla http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat-- Daniel-Constantin Mierla http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
Hi Daniel, I have captured debug messages after testing with your latest commit on master. Once again, here's what I'm doing exactly for my test: 1) Use SIPp to send OPTIONS towards Kamailio. 2) On getting options send event as follows: evapi_async_relay("{"event":"REGISTER","tindex":"$T(id_index)","tlabel":"$T(id_label)","PhoneNumber":"$avp(phone_number)","DeviceId":"$avp(device_id)","CallId":"$ci"}"); 3) There's a client connected which listens for messages on this socket, parses the netstring, and sends same data back as netstring to Kamailio. 4) On the event_route[evapi:message-received, I do the following: xlog("L_INFO", "GOT [$evapi(msg)] from $evapi(srcaddr):$evapi(srcport)\n"); if($evapi(msg)=~"REGISTER" && $evapi(msg)=~"tindex") { jansson_get_field("$evapi(msg)", "tlabel", "$var(tlabel)"); jansson_get_field("$evapi(msg)", "tindex", "$var(tindex)"); $var(t_index) = $(var(tindex){s.int}); $var(t_label) = $(var(tlabel){s.int}); t_continue('$var(t_index)', '$var(t_label)', 'REGISTER_RESPONSE'); exit; } 5) On route[REGISTER_RESPONSE], I send 200 OK to SIPp
Now here are detailed logs for which neither SIPp didnt get a response nor the GOT [$evapi(msg)] from $evapi(srcaddr):$evapi(srcport)\n"); got logged in the syslog:
First Event: DEBUG: evapi [evapi_dispatch.c:598]: evapi_relay(): relaying event data [{"event":"REGISTER","tindex":"26266","tlabel":"587925078","PhoneNumber":"21956","DeviceId":"abcd1234abcd1234","CallId":" 23021-12910@198.24.63.39"}] (146) DEBUG: evapi [evapi_dispatch.c:623]: evapi_relay(): sending [0x7f132588de68] [146:{"event":"REGISTER","tindex":"26266","tlabel":"587925078","PhoneNumber":"21956","DeviceId":"abcd1234abcd1234","CallId":" 23021-12910@198.24.63.39"},] (151)
DEBUG: evapi [evapi_dispatch.c:488]: evapi_recv_notify(): received [0x7f132588de68] [146:{"event":"REGISTER","tindex":"26266","tlabel":"587925078","PhoneNumber":"21956","DeviceId":"abcd1234abcd1234","CallId":" 23021-12910@198.24.63.39"},] (151) NOTICE: evapi [evapi_dispatch.c:290]: evapi_recv_client(): {0} [ 198.24.63.45:48881] - received [146:{"event":"REGISTER","tindex":"26266","tlabel":"587925078","PhoneNumber":"21956","DeviceId":"abcd1234abcd1234","CallId":" 23021-12910@198.24.63.39"},] (151) (18)
Second Event: DEBUG: evapi [evapi_dispatch.c:598]: evapi_relay(): relaying event data [{"event":"REGISTER","tindex":"32244","tlabel":"1637923412","PhoneNumber":"25597","DeviceId":"abcd1234abcd1234","CallId":" 27182-12910@198.24.63.39"}] (147) DEBUG: evapi [evapi_dispatch.c:623]: evapi_relay(): sending [0x7f1325884410] [147:{"event":"REGISTER","tindex":"32244","tlabel":"1637923412","PhoneNumber":"25597","DeviceId":"abcd1234abcd1234","CallId":" 27182-12910@198.24.63.39"},] (152)
DEBUG: evapi [evapi_dispatch.c:488]: evapi_recv_notify(): received [0x7f1325884410] [147:{"event":"REGISTER","tindex":"32244","tlabel":"1637923412","PhoneNumber":"25597","DeviceId":"abcd1234abcd1234","CallId":" 27182-12910@198.24.63.39"},] (152) NOTICE: evapi [evapi_dispatch.c:290]: evapi_recv_client(): {0} [ 198.24.63.45:48881] - received [147:{"event":"REGISTER","tindex":"32244","tlabel":"1637923412","PhoneNumber":"25597","DeviceId":"abcd1234abcd1234","CallId":" 27182-12910@198.24.63.39"},] (152) (85)
I had three such events which got missed out of some 25000 odd messages sent from SIPp. Other major difference that I see in the debug logs for problematic events are that there is a positive number in the second parentheses of evapi_recv_client() function. For the events that were invoked successfully the value of second parentheses is 0 for evapi_recv_client() function. Hope the debugging helps.
Debug log for a successful event: DEBUG: evapi [evapi_dispatch.c:598]: evapi_relay(): relaying event data [{"event":"REGISTER","tindex":"22127","tlabel":"1896682192","PhoneNumber":"73168","DeviceId":"abcd1234abcd1234","CallId":" 49957-13056@198.24.63.39"}] (147) DEBUG: evapi [evapi_dispatch.c:623]: evapi_relay(): sending [0x7f132568e850] [147:{"event":"REGISTER","tindex":"22127","tlabel":"1896682192","PhoneNumber":"73168","DeviceId":"abcd1234abcd1234","CallId":" 49957-13056@198.24.63.39"},] (152)
DEBUG: evapi [evapi_dispatch.c:488]: evapi_recv_notify(): received [0x7f132568e850] [147:{"event":"REGISTER","tindex":"22127","tlabel":"1896682192","PhoneNumber":"73168","DeviceId":"abcd1234abcd1234","CallId":" 49957-13056@198.24.63.39"},] (152) NOTICE: evapi [evapi_dispatch.c:290]: evapi_recv_client(): {0} [ 198.24.63.45:48881] - received [147:{"event":"REGISTER","tindex":"22127","tlabel":"1896682192","PhoneNumber":"73168","DeviceId":"abcd1234abcd1234","CallId":" 49957-13056@198.24.63.39"},] (152) (0)
Thanks,
- Jayesh On Sat, Sep 19, 2015 at 3:55 AM Daniel-Constantin Mierla miconda@gmail.com wrote:
I pushed a commit to add more debug message while processing received data. You can use debugger module to set a higher debug level for evapi module in order to see what happens.
I checked the netstring packet size and it is invalid (unless email stripped some white chars there) -- for example in:
145:{"event":"REGISTER","tindex":"55567","tlabel":"627458699","PhoneNumber":"20711","DeviceId":"abcd1234abcd1234","CallId": "21225-3848@5.6.7.8" 21225-3848@5.6.7.8},
the size is 141, not 145 -- it looks like the size includes the size itself plus the delimiters ':,'. The size is only the effective data, see:
https://en.wikipedia.org/wiki/Netstring
If evapi gets a packet with an invalid size, then it discards the buffer content.
See if the app on the other side of evapi connection builds netstrings with wrong size.
Cheers, Daniel
On 18/09/15 23:19, Jayesh Nambiar wrote:
Here are the tests that I did:
With the patch applied, I see that Kamailio is invoking event individually for each netstring even when they come in different chunks. But I did see instances where when there were complete netstrings in a single chunk; kamailio did not raise an event for them. Here's the illustration:
T 1.2.3.4:48873 -> 5.6.7.8:3927 [AP]
146:{"event":"REGISTER","tindex":"16916","tlabel":"1753048277","PhoneNumber":"20708","DeviceId":"abcd1234abcd1234","CallId":" 21220-3848@5.6.7.821220-3848@5.6.7.8 "},144:{"event":"REGISTER","tindex":"7954","tlabel":"254315075","PhoneNumber":"20709","DeviceId":"abcd1234abcd1234","CallId":" 21223-3848@5.6.7.821223-3848@5.6.7.8 "},145:{"event":"REGISTER","tindex":"64529","tlabel":"599481568","PhoneNumber":"20709","DeviceId":"abcd1234abcd1234","CallId":" 21222-3848@5.6.7.821222-3848@5.6.7.8"},145:{"event":"REGISTER","tindex":"46605","tlabel":"112015324","PhoneNumber":"20710","DeviceId"
T 1.2.3.4:48873 -> 5.6.7.8:3927 [AP] :"abcd1234abcd1234","CallId":"21224-3848@5.6.7.8"},
*The above two chunks contain 4 proper netstrings where the second chunk contains part of the 4th netstring. In this case Evapi properly raised 4 individual events.*
1.2.3.4:48873 -> 5.6.7.8:3927 [AP]
145:{"event":"REGISTER","tindex":"55567","tlabel":"627458699","PhoneNumber":"20711","DeviceId":"abcd1234abcd1234","CallId":" 21225-3848@5.6.7.821225-3848@5.6.7.8 "},143:{"event":"REGISTER","tindex":"28682","tlabel":"9676691","PhoneNumber":"20712","DeviceId":"abcd1234abcd1234","CallId":" 21226-3848@5.6.7.821226-3848@5.6.7.8"},
*This above chunk contains two complete netstrings but Kamailio never raised events for these two netstrings.*
For the events that were not raised I see proper Kamailio Logs which is: evapi_recv_client(): {0} [1.2.3.4:48873] - received [145:{"event":"REGISTER","tindex":"55567","tlabel":"627458699","PhoneNumber":"20711","DeviceId":"abcd1234abcd1234","CallId":" 21225-3848@5.6.7.821225-3848@5.6.7.8 "},143:{"event":"REGISTER","tindex":"28682","tlabel":"9676691","PhoneNumber":"20712","DeviceId":"abcd1234abcd1234","CallId":" 21226-3848@5.6.7.821226-3848@5.6.7.8"},]
But I don't see the logs that I've written in the script after the event was raised, which means Kamailio did not invoke events for these two netstrings.
So in my tests, out of 27232 messages sent, there were approximately 27 messages for which the events were not raised by Evapi. The rate of messages started at 500cps and I stopped after I saw missed events at around 1200cps.
Thanks, and do let me know for any further tests or information required about the same.
- Jayesh
On Fri, Sep 18, 2015 at 2:55 PM, Jayesh Nambiar jayesh1017@gmail.com wrote:
Will have this tested by tomorrow and will get back to you. Thanks.
- Jayesh
On Fri, Sep 18, 2015 at 1:41 PM, Daniel-Constantin Mierla < miconda@gmail.commiconda@gmail.com> wrote:
Hello,
I just pushed a patch to master branch that should cope with partial data received on tcp connection. No time to test at all, therefore any feedback will be appreciated.
Cheers, Daniel
On 15/09/15 14:52, Daniel-Constantin Mierla wrote:
Hello,
I will look if there are options in libev to buffer data or try to implement a buffering mechanism locally for such cases.
Cheers, Daniel
On 14/09/15 23:00, Jayesh Nambiar wrote:
Hello Daniel, After further testing with evapi module, I figured that when Netstrings are used, an event route is invoked individually for each message even if if multiple netstring messages are received in a single TCP packet. But this doesn't work effectively when a single proper message is split-up in two packets. For Example, if a message arrives as: 12:Hello World!, 12:Hello World!, 12:Hello World! in a single packet, kamailio properly invokes the event route "evapi:message-received" thrice for every individual proper netstring message. But if the first packet contains: 12:Hello World!, 12:Hello And Second Packet contains: World!, 12:Hello World! the event route is invoked only once !!
The above pattern is very much possible while sending and receiving packets over TCP Socket. Our tests for receiving an approximately 150 byte message over evapi socket at the rate of roughly 1000cps causes a lot of real events to be missed because of the above problem. You can never be sure when TCP will split messages in different chunks. This definitely looks like a bug which makes it not very reliable at large scale deployments. Would really appreciate your inputs on this. Thanks;
- Jayesh
On Thu, Sep 10, 2015 at 4:01 PM, Jayesh Nambiar < jayesh1017@gmail.com jayesh1017@gmail.com> wrote:
Hi Daniel, Thanks for the quick response. So if I do not use Netstrings, does Kamailio allow me to create a custom logic in the script. For eg. if I decide to use newline as a delimiter, can I keep buffering the message until I encounter the delimiter from the event route and then execute whatever I have to within the script??
Thanks,
- Jayesh
On Thu, Sep 10, 2015 at 1:29 PM, Daniel-Constantin Mierla < miconda@gmail.commiconda@gmail.com> wrote:
Hello,
tcb is stream protocol and several messages can be queued on the pipe at the same time. That is the reason for netstring format, to be able to easily detect the boundaries of each message. If netstring format is enabled and kamailio receives several messages at once, it splits them and for each is executing the event route.
If netstring format is not used, the kamailio is executing the event route with the entire content that was read at once from the tcp connection.
Cheers, Daniel
On 09/09/15 22:01, Jayesh Nambiar wrote:
Hello, I'm exploring the evapi module for my kamailio to interface with an external node.js app for third party stuff like AAA, billing engine tasks, notifications and so on. I followed and took some ideas from the rtjson and evapi tutorial found here( http://kb.asipto.com/kamailio:k43-async-sip-routing-nodejs http://kb.asipto.com/kamailio:k43-async-sip-routing-nodejs) to build the node.js app consuming events. When I stress tested the scenario using SIPp and tried sending a lot of events at 300-350cps from Kamailio, I noticed that at times the client is receiving 2-3 events in a single message together although I do event_sync_relay once per SIP message received and have netstrings enabled. I believe this is a typical behavior of TCP and needs to be handled by the client using some kind of Netstring handler. Please correct me if I'm wrong. And hence I'd like to know what particularly needs to be taken care of while writing a client that is listening for events on raw tcp socket and how does kamailio handle this situation while receiving messages over TCP socket?? Does kamailio recognize the end of netstring properly on evapi:message-received and give exactly one message to take care of on every "message-received" event or should that be handled in the script somewhere !! I also referred cgrates client over evapi example which is written in GO, but I couldnt find them handling TCP streams clearly either. I'd really appreciate some expert suggestion here to make an informed decision on using the evapi module for a large scale solution.
Thanks,
- Jayesh
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing listsr-users@lists.sip-router.orghttp://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
-- Daniel-Constantin Mierlahttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.orgsr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
-- Daniel-Constantin Mierlahttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
-- Daniel-Constantin Mierlahttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
-- Daniel-Constantin Mierlahttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
Hello,
I pushed another commit to add more debug messages to see if the event route is supposed to be executed or not. Can you run the tests again and give again the log messages for missing event route executions?
Cheers, Daniel
On 21/09/15 09:04, Jayesh Nambiar wrote:
Hi Daniel, I have captured debug messages after testing with your latest commit on master. Once again, here's what I'm doing exactly for my test:
- Use SIPp to send OPTIONS towards Kamailio.
- On getting options send event as follows:
evapi_async_relay("{"event":"REGISTER","tindex":"$T(id_index)","tlabel":"$T(id_label)","PhoneNumber":"$avp(phone_number)","DeviceId":"$avp(device_id)","CallId":"$ci"}"); 3) There's a client connected which listens for messages on this socket, parses the netstring, and sends same data back as netstring to Kamailio. 4) On the event_route[evapi:message-received, I do the following: xlog("L_INFO", "GOT [$evapi(msg)] from $evapi(srcaddr):$evapi(srcport)\n"); if($evapi(msg)=~"REGISTER" && $evapi(msg)=~"tindex") { jansson_get_field("$evapi(msg)", "tlabel", "$var(tlabel)"); jansson_get_field("$evapi(msg)", "tindex", "$var(tindex)"); $var(t_index) = $(var(tindex){s.int http://s.int}); $var(t_label) = $(var(tlabel){s.int http://s.int}); t_continue('$var(t_index)', '$var(t_label)', 'REGISTER_RESPONSE'); exit; } 5) On route[REGISTER_RESPONSE], I send 200 OK to SIPp
Now here are detailed logs for which neither SIPp didnt get a response nor the GOT [$evapi(msg)] from $evapi(srcaddr):$evapi(srcport)\n"); got logged in the syslog:
First Event: DEBUG: evapi [evapi_dispatch.c:598]: evapi_relay(): relaying event data [{"event":"REGISTER","tindex":"26266","tlabel":"587925078","PhoneNumber":"21956","DeviceId":"abcd1234abcd1234","CallId":"23021-12910@198.24.63.39 mailto:23021-12910@198.24.63.39"}] (146) DEBUG: evapi [evapi_dispatch.c:623]: evapi_relay(): sending [0x7f132588de68] [146:{"event":"REGISTER","tindex":"26266","tlabel":"587925078","PhoneNumber":"21956","DeviceId":"abcd1234abcd1234","CallId":"23021-12910@198.24.63.39 mailto:23021-12910@198.24.63.39"},] (151)
DEBUG: evapi [evapi_dispatch.c:488]: evapi_recv_notify(): received [0x7f132588de68] [146:{"event":"REGISTER","tindex":"26266","tlabel":"587925078","PhoneNumber":"21956","DeviceId":"abcd1234abcd1234","CallId":"23021-12910@198.24.63.39 mailto:23021-12910@198.24.63.39"},] (151) NOTICE: evapi [evapi_dispatch.c:290]: evapi_recv_client(): {0} [198.24.63.45:48881 http://198.24.63.45:48881] - received [146:{"event":"REGISTER","tindex":"26266","tlabel":"587925078","PhoneNumber":"21956","DeviceId":"abcd1234abcd1234","CallId":"23021-12910@198.24.63.39 mailto:23021-12910@198.24.63.39"},] (151) (18)
Second Event: DEBUG: evapi [evapi_dispatch.c:598]: evapi_relay(): relaying event data [{"event":"REGISTER","tindex":"32244","tlabel":"1637923412","PhoneNumber":"25597","DeviceId":"abcd1234abcd1234","CallId":"27182-12910@198.24.63.39 mailto:27182-12910@198.24.63.39"}] (147) DEBUG: evapi [evapi_dispatch.c:623]: evapi_relay(): sending [0x7f1325884410] [147:{"event":"REGISTER","tindex":"32244","tlabel":"1637923412","PhoneNumber":"25597","DeviceId":"abcd1234abcd1234","CallId":"27182-12910@198.24.63.39 mailto:27182-12910@198.24.63.39"},] (152)
DEBUG: evapi [evapi_dispatch.c:488]: evapi_recv_notify(): received [0x7f1325884410] [147:{"event":"REGISTER","tindex":"32244","tlabel":"1637923412","PhoneNumber":"25597","DeviceId":"abcd1234abcd1234","CallId":"27182-12910@198.24.63.39 mailto:27182-12910@198.24.63.39"},] (152) NOTICE: evapi [evapi_dispatch.c:290]: evapi_recv_client(): {0} [198.24.63.45:48881 http://198.24.63.45:48881] - received [147:{"event":"REGISTER","tindex":"32244","tlabel":"1637923412","PhoneNumber":"25597","DeviceId":"abcd1234abcd1234","CallId":"27182-12910@198.24.63.39 mailto:27182-12910@198.24.63.39"},] (152) (85)
I had three such events which got missed out of some 25000 odd messages sent from SIPp. Other major difference that I see in the debug logs for problematic events are that there is a positive number in the second parentheses of evapi_recv_client() function. For the events that were invoked successfully the value of second parentheses is 0 for evapi_recv_client() function. Hope the debugging helps.
Debug log for a successful event: DEBUG: evapi [evapi_dispatch.c:598]: evapi_relay(): relaying event data [{"event":"REGISTER","tindex":"22127","tlabel":"1896682192","PhoneNumber":"73168","DeviceId":"abcd1234abcd1234","CallId":"49957-13056@198.24.63.39 mailto:49957-13056@198.24.63.39"}] (147) DEBUG: evapi [evapi_dispatch.c:623]: evapi_relay(): sending [0x7f132568e850] [147:{"event":"REGISTER","tindex":"22127","tlabel":"1896682192","PhoneNumber":"73168","DeviceId":"abcd1234abcd1234","CallId":"49957-13056@198.24.63.39 mailto:49957-13056@198.24.63.39"},] (152)
DEBUG: evapi [evapi_dispatch.c:488]: evapi_recv_notify(): received [0x7f132568e850] [147:{"event":"REGISTER","tindex":"22127","tlabel":"1896682192","PhoneNumber":"73168","DeviceId":"abcd1234abcd1234","CallId":"49957-13056@198.24.63.39 mailto:49957-13056@198.24.63.39"},] (152) NOTICE: evapi [evapi_dispatch.c:290]: evapi_recv_client(): {0} [198.24.63.45:48881 http://198.24.63.45:48881] - received [147:{"event":"REGISTER","tindex":"22127","tlabel":"1896682192","PhoneNumber":"73168","DeviceId":"abcd1234abcd1234","CallId":"49957-13056@198.24.63.39 mailto:49957-13056@198.24.63.39"},] (152) (0)
Thanks,
- Jayesh
On Sat, Sep 19, 2015 at 3:55 AM Daniel-Constantin Mierla <miconda@gmail.com mailto:miconda@gmail.com> wrote:
I pushed a commit to add more debug message while processing received data. You can use debugger module to set a higher debug level for evapi module in order to see what happens. I checked the netstring packet size and it is invalid (unless email stripped some white chars there) -- for example in: 145:{"event":"REGISTER","tindex":"55567","tlabel":"627458699","PhoneNumber":"20711","DeviceId":"abcd1234abcd1234","CallId":"21225-3848@5.6.7.8" <mailto:21225-3848@5.6.7.8>}, the size is 141, not 145 -- it looks like the size includes the size itself plus the delimiters ':,'. The size is only the effective data, see: https://en.wikipedia.org/wiki/Netstring If evapi gets a packet with an invalid size, then it discards the buffer content. See if the app on the other side of evapi connection builds netstrings with wrong size. Cheers, Daniel On 18/09/15 23:19, Jayesh Nambiar wrote:Here are the tests that I did: With the patch applied, I see that Kamailio is invoking event individually for each netstring even when they come in different chunks. But I did see instances where when there were complete netstrings in a single chunk; kamailio did not raise an event for them. Here's the illustration: T 1.2.3.4:48873 <http://1.2.3.4:48873> -> 5.6.7.8:3927 <http://5.6.7.8:3927> [AP] 146:{"event":"REGISTER","tindex":"16916","tlabel":"1753048277","PhoneNumber":"20708","DeviceId":"abcd1234abcd1234","CallId":"21220-3848@5.6.7.8 <mailto:21220-3848@5.6.7.8>"},144:{"event":"REGISTER","tindex":"7954","tlabel":"254315075","PhoneNumber":"20709","DeviceId":"abcd1234abcd1234","CallId":"21223-3848@5.6.7.8 <mailto:21223-3848@5.6.7.8>"},145:{"event":"REGISTER","tindex":"64529","tlabel":"599481568","PhoneNumber":"20709","DeviceId":"abcd1234abcd1234","CallId":"21222-3848@5.6.7.8 <mailto:21222-3848@5.6.7.8>"},145:{"event":"REGISTER","tindex":"46605","tlabel":"112015324","PhoneNumber":"20710","DeviceId" T 1.2.3.4:48873 <http://1.2.3.4:48873> -> 5.6.7.8:3927 <http://5.6.7.8:3927> [AP] :"abcd1234abcd1234","CallId":"21224-3848@5.6.7.8 <mailto:21224-3848@5.6.7.8>"}, *The above two chunks contain 4 proper netstrings where the second chunk contains part of the 4th netstring. In this case Evapi properly raised 4 individual events.* 1.2.3.4:48873 <http://1.2.3.4:48873> -> 5.6.7.8:3927 <http://5.6.7.8:3927> [AP] 145:{"event":"REGISTER","tindex":"55567","tlabel":"627458699","PhoneNumber":"20711","DeviceId":"abcd1234abcd1234","CallId":"21225-3848@5.6.7.8 <mailto:21225-3848@5.6.7.8>"},143:{"event":"REGISTER","tindex":"28682","tlabel":"9676691","PhoneNumber":"20712","DeviceId":"abcd1234abcd1234","CallId":"21226-3848@5.6.7.8 <mailto:21226-3848@5.6.7.8>"}, *This above chunk contains two complete netstrings but Kamailio never raised events for these two netstrings.* * * For the events that were not raised I see proper Kamailio Logs which is: evapi_recv_client(): {0} [1.2.3.4:48873 <http://1.2.3.4:48873>] - received [145:{"event":"REGISTER","tindex":"55567","tlabel":"627458699","PhoneNumber":"20711","DeviceId":"abcd1234abcd1234","CallId":"21225-3848@5.6.7.8 <mailto:21225-3848@5.6.7.8>"},143:{"event":"REGISTER","tindex":"28682","tlabel":"9676691","PhoneNumber":"20712","DeviceId":"abcd1234abcd1234","CallId":"21226-3848@5.6.7.8 <mailto:21226-3848@5.6.7.8>"},] * * But I don't see the logs that I've written in the script after the event was raised, which means Kamailio did not invoke events for these two netstrings. So in my tests, out of 27232 messages sent, there were approximately 27 messages for which the events were not raised by Evapi. The rate of messages started at 500cps and I stopped after I saw missed events at around 1200cps. Thanks, and do let me know for any further tests or information required about the same. - Jayesh * * On Fri, Sep 18, 2015 at 2:55 PM, Jayesh Nambiar <jayesh1017@gmail.com <mailto:jayesh1017@gmail.com>> wrote: Will have this tested by tomorrow and will get back to you. Thanks. - Jayesh On Fri, Sep 18, 2015 at 1:41 PM, Daniel-Constantin Mierla <miconda@gmail.com <mailto:miconda@gmail.com>> wrote: Hello, I just pushed a patch to master branch that should cope with partial data received on tcp connection. No time to test at all, therefore any feedback will be appreciated. Cheers, Daniel On 15/09/15 14:52, Daniel-Constantin Mierla wrote:Hello, I will look if there are options in libev to buffer data or try to implement a buffering mechanism locally for such cases. Cheers, Daniel On 14/09/15 23:00, Jayesh Nambiar wrote:Hello Daniel, After further testing with evapi module, I figured that when Netstrings are used, an event route is invoked individually for each message even if if multiple netstring messages are received in a single TCP packet. But this doesn't work effectively when a single proper message is split-up in two packets. For Example, if a message arrives as: 12:Hello World!, 12:Hello World!, 12:Hello World! in a single packet, kamailio properly invokes the event route "evapi:message-received" thrice for every individual proper netstring message. But if the first packet contains: 12:Hello World!, 12:Hello And Second Packet contains: World!, 12:Hello World! the event route is invoked only once !! The above pattern is very much possible while sending and receiving packets over TCP Socket. Our tests for receiving an approximately 150 byte message over evapi socket at the rate of roughly 1000cps causes a lot of real events to be missed because of the above problem. You can never be sure when TCP will split messages in different chunks. This definitely looks like a bug which makes it not very reliable at large scale deployments. Would really appreciate your inputs on this. Thanks; - Jayesh On Thu, Sep 10, 2015 at 4:01 PM, Jayesh Nambiar <jayesh1017@gmail.com <mailto:jayesh1017@gmail.com>> wrote: Hi Daniel, Thanks for the quick response. So if I do not use Netstrings, does Kamailio allow me to create a custom logic in the script. For eg. if I decide to use newline as a delimiter, can I keep buffering the message until I encounter the delimiter from the event route and then execute whatever I have to within the script?? Thanks, - Jayesh On Thu, Sep 10, 2015 at 1:29 PM, Daniel-Constantin Mierla <miconda@gmail.com <mailto:miconda@gmail.com>> wrote: Hello, tcb is stream protocol and several messages can be queued on the pipe at the same time. That is the reason for netstring format, to be able to easily detect the boundaries of each message. If netstring format is enabled and kamailio receives several messages at once, it splits them and for each is executing the event route. If netstring format is not used, the kamailio is executing the event route with the entire content that was read at once from the tcp connection. Cheers, Daniel On 09/09/15 22:01, Jayesh Nambiar wrote:Hello, I'm exploring the evapi module for my kamailio to interface with an external node.js app for third party stuff like AAA, billing engine tasks, notifications and so on. I followed and took some ideas from the rtjson and evapi tutorial found here(http://kb.asipto.com/kamailio:k43-async-sip-routing-nodejs) to build the node.js app consuming events. When I stress tested the scenario using SIPp and tried sending a lot of events at 300-350cps from Kamailio, I noticed that at times the client is receiving 2-3 events in a single message together although I do event_sync_relay once per SIP message received and have netstrings enabled. I believe this is a typical behavior of TCP and needs to be handled by the client using some kind of Netstring handler. Please correct me if I'm wrong. And hence I'd like to know what particularly needs to be taken care of while writing a client that is listening for events on raw tcp socket and how does kamailio handle this situation while receiving messages over TCP socket?? Does kamailio recognize the end of netstring properly on evapi:message-received and give exactly one message to take care of on every "message-received" event or should that be handled in the script somewhere !! I also referred cgrates client over evapi example which is written in GO, but I couldnt find them handling TCP streams clearly either. I'd really appreciate some expert suggestion here to make an informed decision on using the evapi module for a large scale solution. Thanks, - Jayesh _______________________________________________ SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org <mailto:sr-users@lists.sip-router.org> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users-- Daniel-Constantin Mierla http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat _______________________________________________ SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org <mailto:sr-users@lists.sip-router.org> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users-- Daniel-Constantin Mierla http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat-- Daniel-Constantin Mierla http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat-- Daniel-Constantin Mierla http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
Hi, Took time to get back with the test results as I was trying to analyze too much myself before getting back on the list. Also I was stuck with weird error of module version mismatch, but I figured and solved that issue. So I do see the logs "frame size mismatch" when I miss the events. Since the logs are too huge, I am sending it in pastebin. Here it is: http://pastebin.com/MzTpYsrk
In Kamailio Logs, I see: DEBUG: evapi [evapi_dispatch.c:361]: evapi_recv_client(): frame size mismatch the ending char (h):[{"event":"REGISTER","tindex":"12603","tlabel":"626010915 ","PhoneNumber":"42000","D146:"event":"REGISTER","tindex":"3639","tlabel":" 1889788252","P] (146)
Whereas the messages in ngrep looks good where the messages are split in two TCP packets. The first packet ends at D and the second packet is the proper continuation which send the rest. But it feels Kamailio skipped the beginning of the message and started parsing at colon again and it thinks the netstring started with D146 which is incorrect. Hence, it never emitted any event for the messages in the first chunk.
Hoping this to be helpful !! Thanks.
- Jayesh
On Mon, Sep 21, 2015 at 4:37 PM Daniel-Constantin Mierla miconda@gmail.com wrote:
Hello,
I pushed another commit to add more debug messages to see if the event route is supposed to be executed or not. Can you run the tests again and give again the log messages for missing event route executions?
Cheers, Daniel
On 21/09/15 09:04, Jayesh Nambiar wrote:
Hi Daniel, I have captured debug messages after testing with your latest commit on master. Once again, here's what I'm doing exactly for my test:
- Use SIPp to send OPTIONS towards Kamailio.
- On getting options send event as follows:
evapi_async_relay("{"event":"REGISTER","tindex":"$T(id_index)","tlabel":"$T(id_label)","PhoneNumber":"$avp(phone_number)","DeviceId":"$avp(device_id)","CallId":"$ci"}"); 3) There's a client connected which listens for messages on this socket, parses the netstring, and sends same data back as netstring to Kamailio. 4) On the event_route[evapi:message-received, I do the following: xlog("L_INFO", "GOT [$evapi(msg)] from $evapi(srcaddr):$evapi(srcport)\n"); if($evapi(msg)=~"REGISTER" && $evapi(msg)=~"tindex") { jansson_get_field("$evapi(msg)", "tlabel", "$var(tlabel)"); jansson_get_field("$evapi(msg)", "tindex", "$var(tindex)"); $var(t_index) = $(var(tindex){s.int}); $var(t_label) = $(var(tlabel){s.int}); t_continue('$var(t_index)', '$var(t_label)', 'REGISTER_RESPONSE'); exit; } 5) On route[REGISTER_RESPONSE], I send 200 OK to SIPp
Now here are detailed logs for which neither SIPp didnt get a response nor the GOT [$evapi(msg)] from $evapi(srcaddr):$evapi(srcport)\n"); got logged in the syslog:
First Event: DEBUG: evapi [evapi_dispatch.c:598]: evapi_relay(): relaying event data [{"event":"REGISTER","tindex":"26266","tlabel":"587925078","PhoneNumber":"21956","DeviceId":"abcd1234abcd1234","CallId":" 23021-12910@198.24.63.3923021-12910@198.24.63.39"}] (146) DEBUG: evapi [evapi_dispatch.c:623]: evapi_relay(): sending [0x7f132588de68] [146:{"event":"REGISTER","tindex":"26266","tlabel":"587925078","PhoneNumber":"21956","DeviceId":"abcd1234abcd1234","CallId":" 23021-12910@198.24.63.3923021-12910@198.24.63.39"},] (151)
DEBUG: evapi [evapi_dispatch.c:488]: evapi_recv_notify(): received [0x7f132588de68] [146:{"event":"REGISTER","tindex":"26266","tlabel":"587925078","PhoneNumber":"21956","DeviceId":"abcd1234abcd1234","CallId":" 23021-12910@198.24.63.3923021-12910@198.24.63.39"},] (151) NOTICE: evapi [evapi_dispatch.c:290]: evapi_recv_client(): {0} [ 198.24.63.45:48881] - received [146:{"event":"REGISTER","tindex":"26266","tlabel":"587925078","PhoneNumber":"21956","DeviceId":"abcd1234abcd1234","CallId":" 23021-12910@198.24.63.3923021-12910@198.24.63.39"},] (151) (18)
Second Event: DEBUG: evapi [evapi_dispatch.c:598]: evapi_relay(): relaying event data [{"event":"REGISTER","tindex":"32244","tlabel":"1637923412","PhoneNumber":"25597","DeviceId":"abcd1234abcd1234","CallId":" 27182-12910@198.24.63.3927182-12910@198.24.63.39"}] (147) DEBUG: evapi [evapi_dispatch.c:623]: evapi_relay(): sending [0x7f1325884410] [147:{"event":"REGISTER","tindex":"32244","tlabel":"1637923412","PhoneNumber":"25597","DeviceId":"abcd1234abcd1234","CallId":" 27182-12910@198.24.63.3927182-12910@198.24.63.39"},] (152)
DEBUG: evapi [evapi_dispatch.c:488]: evapi_recv_notify(): received [0x7f1325884410] [147:{"event":"REGISTER","tindex":"32244","tlabel":"1637923412","PhoneNumber":"25597","DeviceId":"abcd1234abcd1234","CallId":" 27182-12910@198.24.63.3927182-12910@198.24.63.39"},] (152) NOTICE: evapi [evapi_dispatch.c:290]: evapi_recv_client(): {0} [ 198.24.63.45:48881] - received [147:{"event":"REGISTER","tindex":"32244","tlabel":"1637923412","PhoneNumber":"25597","DeviceId":"abcd1234abcd1234","CallId":" 27182-12910@198.24.63.3927182-12910@198.24.63.39"},] (152) (85)
I had three such events which got missed out of some 25000 odd messages sent from SIPp. Other major difference that I see in the debug logs for problematic events are that there is a positive number in the second parentheses of evapi_recv_client() function. For the events that were invoked successfully the value of second parentheses is 0 for evapi_recv_client() function. Hope the debugging helps.
Debug log for a successful event: DEBUG: evapi [evapi_dispatch.c:598]: evapi_relay(): relaying event data [{"event":"REGISTER","tindex":"22127","tlabel":"1896682192","PhoneNumber":"73168","DeviceId":"abcd1234abcd1234","CallId":" 49957-13056@198.24.63.3949957-13056@198.24.63.39"}] (147) DEBUG: evapi [evapi_dispatch.c:623]: evapi_relay(): sending [0x7f132568e850] [147:{"event":"REGISTER","tindex":"22127","tlabel":"1896682192","PhoneNumber":"73168","DeviceId":"abcd1234abcd1234","CallId":" 49957-13056@198.24.63.3949957-13056@198.24.63.39"},] (152)
DEBUG: evapi [evapi_dispatch.c:488]: evapi_recv_notify(): received [0x7f132568e850] [147:{"event":"REGISTER","tindex":"22127","tlabel":"1896682192","PhoneNumber":"73168","DeviceId":"abcd1234abcd1234","CallId":" 49957-13056@198.24.63.3949957-13056@198.24.63.39"},] (152) NOTICE: evapi [evapi_dispatch.c:290]: evapi_recv_client(): {0} [ 198.24.63.45:48881] - received [147:{"event":"REGISTER","tindex":"22127","tlabel":"1896682192","PhoneNumber":"73168","DeviceId":"abcd1234abcd1234","CallId":" 49957-13056@198.24.63.3949957-13056@198.24.63.39"},] (152) (0)
Thanks,
- Jayesh
On Sat, Sep 19, 2015 at 3:55 AM Daniel-Constantin Mierla < miconda@gmail.com> wrote:
I pushed a commit to add more debug message while processing received data. You can use debugger module to set a higher debug level for evapi module in order to see what happens.
I checked the netstring packet size and it is invalid (unless email stripped some white chars there) -- for example in:
145:{"event":"REGISTER","tindex":"55567","tlabel":"627458699","PhoneNumber":"20711","DeviceId":"abcd1234abcd1234","CallId": 21225-3848@5.6.7.8"21225-3848@5.6.7.8" 21225-3848@5.6.7.8},
the size is 141, not 145 -- it looks like the size includes the size itself plus the delimiters ':,'. The size is only the effective data, see:
https://en.wikipedia.org/wiki/Netstring
If evapi gets a packet with an invalid size, then it discards the buffer content.
See if the app on the other side of evapi connection builds netstrings with wrong size.
Cheers, Daniel
On 18/09/15 23:19, Jayesh Nambiar wrote:
Here are the tests that I did:
With the patch applied, I see that Kamailio is invoking event individually for each netstring even when they come in different chunks. But I did see instances where when there were complete netstrings in a single chunk; kamailio did not raise an event for them. Here's the illustration:
T 1.2.3.4:48873 -> 5.6.7.8:3927 [AP]
146:{"event":"REGISTER","tindex":"16916","tlabel":"1753048277","PhoneNumber":"20708","DeviceId":"abcd1234abcd1234","CallId":" 21220-3848@5.6.7.821220-3848@5.6.7.8 "},144:{"event":"REGISTER","tindex":"7954","tlabel":"254315075","PhoneNumber":"20709","DeviceId":"abcd1234abcd1234","CallId":" 21223-3848@5.6.7.821223-3848@5.6.7.8 "},145:{"event":"REGISTER","tindex":"64529","tlabel":"599481568","PhoneNumber":"20709","DeviceId":"abcd1234abcd1234","CallId":" 21222-3848@5.6.7.821222-3848@5.6.7.8"},145:{"event":"REGISTER","tindex":"46605","tlabel":"112015324","PhoneNumber":"20710","DeviceId"
T 1.2.3.4:48873 -> 5.6.7.8:3927 [AP] :"abcd1234abcd1234","CallId":" 21224-3848@5.6.7.821224-3848@5.6.7.8 "},
*The above two chunks contain 4 proper netstrings where the second chunk contains part of the 4th netstring. In this case Evapi properly raised 4 individual events.*
1.2.3.4:48873 -> 5.6.7.8:3927 [AP]
145:{"event":"REGISTER","tindex":"55567","tlabel":"627458699","PhoneNumber":"20711","DeviceId":"abcd1234abcd1234","CallId":" 21225-3848@5.6.7.821225-3848@5.6.7.8 "},143:{"event":"REGISTER","tindex":"28682","tlabel":"9676691","PhoneNumber":"20712","DeviceId":"abcd1234abcd1234","CallId":" 21226-3848@5.6.7.821226-3848@5.6.7.8"},
*This above chunk contains two complete netstrings but Kamailio never raised events for these two netstrings.*
For the events that were not raised I see proper Kamailio Logs which is: evapi_recv_client(): {0} [1.2.3.4:48873] - received [145:{"event":"REGISTER","tindex":"55567","tlabel":"627458699","PhoneNumber":"20711","DeviceId":"abcd1234abcd1234","CallId":" 21225-3848@5.6.7.821225-3848@5.6.7.8 "},143:{"event":"REGISTER","tindex":"28682","tlabel":"9676691","PhoneNumber":"20712","DeviceId":"abcd1234abcd1234","CallId":" 21226-3848@5.6.7.821226-3848@5.6.7.8"},]
But I don't see the logs that I've written in the script after the event was raised, which means Kamailio did not invoke events for these two netstrings.
So in my tests, out of 27232 messages sent, there were approximately 27 messages for which the events were not raised by Evapi. The rate of messages started at 500cps and I stopped after I saw missed events at around 1200cps.
Thanks, and do let me know for any further tests or information required about the same.
- Jayesh
On Fri, Sep 18, 2015 at 2:55 PM, Jayesh Nambiar < jayesh1017@gmail.com jayesh1017@gmail.com> wrote:
Will have this tested by tomorrow and will get back to you. Thanks.
- Jayesh
On Fri, Sep 18, 2015 at 1:41 PM, Daniel-Constantin Mierla < miconda@gmail.commiconda@gmail.com> wrote:
Hello,
I just pushed a patch to master branch that should cope with partial data received on tcp connection. No time to test at all, therefore any feedback will be appreciated.
Cheers, Daniel
On 15/09/15 14:52, Daniel-Constantin Mierla wrote:
Hello,
I will look if there are options in libev to buffer data or try to implement a buffering mechanism locally for such cases.
Cheers, Daniel
On 14/09/15 23:00, Jayesh Nambiar wrote:
Hello Daniel, After further testing with evapi module, I figured that when Netstrings are used, an event route is invoked individually for each message even if if multiple netstring messages are received in a single TCP packet. But this doesn't work effectively when a single proper message is split-up in two packets. For Example, if a message arrives as: 12:Hello World!, 12:Hello World!, 12:Hello World! in a single packet, kamailio properly invokes the event route "evapi:message-received" thrice for every individual proper netstring message. But if the first packet contains: 12:Hello World!, 12:Hello And Second Packet contains: World!, 12:Hello World! the event route is invoked only once !!
The above pattern is very much possible while sending and receiving packets over TCP Socket. Our tests for receiving an approximately 150 byte message over evapi socket at the rate of roughly 1000cps causes a lot of real events to be missed because of the above problem. You can never be sure when TCP will split messages in different chunks. This definitely looks like a bug which makes it not very reliable at large scale deployments. Would really appreciate your inputs on this. Thanks;
- Jayesh
On Thu, Sep 10, 2015 at 4:01 PM, Jayesh Nambiar < jayesh1017@gmail.comjayesh1017@gmail.com> wrote:
Hi Daniel, Thanks for the quick response. So if I do not use Netstrings, does Kamailio allow me to create a custom logic in the script. For eg. if I decide to use newline as a delimiter, can I keep buffering the message until I encounter the delimiter from the event route and then execute whatever I have to within the script??
Thanks,
- Jayesh
On Thu, Sep 10, 2015 at 1:29 PM, Daniel-Constantin Mierla < miconda@gmail.commiconda@gmail.com> wrote:
Hello,
tcb is stream protocol and several messages can be queued on the pipe at the same time. That is the reason for netstring format, to be able to easily detect the boundaries of each message. If netstring format is enabled and kamailio receives several messages at once, it splits them and for each is executing the event route.
If netstring format is not used, the kamailio is executing the event route with the entire content that was read at once from the tcp connection.
Cheers, Daniel
On 09/09/15 22:01, Jayesh Nambiar wrote:
Hello, I'm exploring the evapi module for my kamailio to interface with an external node.js app for third party stuff like AAA, billing engine tasks, notifications and so on. I followed and took some ideas from the rtjson and evapi tutorial found here( http://kb.asipto.com/kamailio:k43-async-sip-routing-nodejs http://kb.asipto.com/kamailio:k43-async-sip-routing-nodejs) to build the node.js app consuming events. When I stress tested the scenario using SIPp and tried sending a lot of events at 300-350cps from Kamailio, I noticed that at times the client is receiving 2-3 events in a single message together although I do event_sync_relay once per SIP message received and have netstrings enabled. I believe this is a typical behavior of TCP and needs to be handled by the client using some kind of Netstring handler. Please correct me if I'm wrong. And hence I'd like to know what particularly needs to be taken care of while writing a client that is listening for events on raw tcp socket and how does kamailio handle this situation while receiving messages over TCP socket?? Does kamailio recognize the end of netstring properly on evapi:message-received and give exactly one message to take care of on every "message-received" event or should that be handled in the script somewhere !! I also referred cgrates client over evapi example which is written in GO, but I couldnt find them handling TCP streams clearly either. I'd really appreciate some expert suggestion here to make an informed decision on using the evapi module for a large scale solution.
Thanks,
- Jayesh
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing listsr-users@lists.sip-router.orghttp://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
-- Daniel-Constantin Mierlahttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.orgsr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
-- Daniel-Constantin Mierlahttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
-- Daniel-Constantin Mierlahttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
-- Daniel-Constantin Mierlahttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
-- Daniel-Constantin Mierlahttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
Also forgot to mention, this starts happening at around 1200cps. And using SIPp I accelerate from 500cps to 1200cps very quick. So it might be related to concurrency is what I'm assuming. Thanks.
- Jayesh
On Tue, Sep 22, 2015 at 12:40 PM Jayesh Nambiar jayesh1017@gmail.com wrote:
Hi, Took time to get back with the test results as I was trying to analyze too much myself before getting back on the list. Also I was stuck with weird error of module version mismatch, but I figured and solved that issue. So I do see the logs "frame size mismatch" when I miss the events. Since the logs are too huge, I am sending it in pastebin. Here it is: http://pastebin.com/MzTpYsrk
In Kamailio Logs, I see: DEBUG: evapi [evapi_dispatch.c:361]: evapi_recv_client(): frame size mismatch the ending char (h):[{"event":"REGISTER","tindex":"12603","tlabel":"626010915 ","PhoneNumber":"42000","D146:"event":"REGISTER","tindex":"3639","tlabel":" 1889788252","P] (146)
Whereas the messages in ngrep looks good where the messages are split in two TCP packets. The first packet ends at D and the second packet is the proper continuation which send the rest. But it feels Kamailio skipped the beginning of the message and started parsing at colon again and it thinks the netstring started with D146 which is incorrect. Hence, it never emitted any event for the messages in the first chunk.
Hoping this to be helpful !! Thanks.
- Jayesh
On Mon, Sep 21, 2015 at 4:37 PM Daniel-Constantin Mierla < miconda@gmail.com> wrote:
Hello,
I pushed another commit to add more debug messages to see if the event route is supposed to be executed or not. Can you run the tests again and give again the log messages for missing event route executions?
Cheers, Daniel
On 21/09/15 09:04, Jayesh Nambiar wrote:
Hi Daniel, I have captured debug messages after testing with your latest commit on master. Once again, here's what I'm doing exactly for my test:
- Use SIPp to send OPTIONS towards Kamailio.
- On getting options send event as follows:
evapi_async_relay("{"event":"REGISTER","tindex":"$T(id_index)","tlabel":"$T(id_label)","PhoneNumber":"$avp(phone_number)","DeviceId":"$avp(device_id)","CallId":"$ci"}"); 3) There's a client connected which listens for messages on this socket, parses the netstring, and sends same data back as netstring to Kamailio. 4) On the event_route[evapi:message-received, I do the following: xlog("L_INFO", "GOT [$evapi(msg)] from $evapi(srcaddr):$evapi(srcport)\n"); if($evapi(msg)=~"REGISTER" && $evapi(msg)=~"tindex") { jansson_get_field("$evapi(msg)", "tlabel", "$var(tlabel)"); jansson_get_field("$evapi(msg)", "tindex", "$var(tindex)"); $var(t_index) = $(var(tindex){s.int}); $var(t_label) = $(var(tlabel){s.int}); t_continue('$var(t_index)', '$var(t_label)', 'REGISTER_RESPONSE'); exit; } 5) On route[REGISTER_RESPONSE], I send 200 OK to SIPp
Now here are detailed logs for which neither SIPp didnt get a response nor the GOT [$evapi(msg)] from $evapi(srcaddr):$evapi(srcport)\n"); got logged in the syslog:
First Event: DEBUG: evapi [evapi_dispatch.c:598]: evapi_relay(): relaying event data [{"event":"REGISTER","tindex":"26266","tlabel":"587925078","PhoneNumber":"21956","DeviceId":"abcd1234abcd1234","CallId":" 23021-12910@198.24.63.3923021-12910@198.24.63.39"}] (146) DEBUG: evapi [evapi_dispatch.c:623]: evapi_relay(): sending [0x7f132588de68] [146:{"event":"REGISTER","tindex":"26266","tlabel":"587925078","PhoneNumber":"21956","DeviceId":"abcd1234abcd1234","CallId":" 23021-12910@198.24.63.3923021-12910@198.24.63.39"},] (151)
DEBUG: evapi [evapi_dispatch.c:488]: evapi_recv_notify(): received [0x7f132588de68] [146:{"event":"REGISTER","tindex":"26266","tlabel":"587925078","PhoneNumber":"21956","DeviceId":"abcd1234abcd1234","CallId":" 23021-12910@198.24.63.3923021-12910@198.24.63.39"},] (151) NOTICE: evapi [evapi_dispatch.c:290]: evapi_recv_client(): {0} [ 198.24.63.45:48881] - received [146:{"event":"REGISTER","tindex":"26266","tlabel":"587925078","PhoneNumber":"21956","DeviceId":"abcd1234abcd1234","CallId":" 23021-12910@198.24.63.3923021-12910@198.24.63.39"},] (151) (18)
Second Event: DEBUG: evapi [evapi_dispatch.c:598]: evapi_relay(): relaying event data [{"event":"REGISTER","tindex":"32244","tlabel":"1637923412","PhoneNumber":"25597","DeviceId":"abcd1234abcd1234","CallId":" 27182-12910@198.24.63.3927182-12910@198.24.63.39"}] (147) DEBUG: evapi [evapi_dispatch.c:623]: evapi_relay(): sending [0x7f1325884410] [147:{"event":"REGISTER","tindex":"32244","tlabel":"1637923412","PhoneNumber":"25597","DeviceId":"abcd1234abcd1234","CallId":" 27182-12910@198.24.63.3927182-12910@198.24.63.39"},] (152)
DEBUG: evapi [evapi_dispatch.c:488]: evapi_recv_notify(): received [0x7f1325884410] [147:{"event":"REGISTER","tindex":"32244","tlabel":"1637923412","PhoneNumber":"25597","DeviceId":"abcd1234abcd1234","CallId":" 27182-12910@198.24.63.3927182-12910@198.24.63.39"},] (152) NOTICE: evapi [evapi_dispatch.c:290]: evapi_recv_client(): {0} [ 198.24.63.45:48881] - received [147:{"event":"REGISTER","tindex":"32244","tlabel":"1637923412","PhoneNumber":"25597","DeviceId":"abcd1234abcd1234","CallId":" 27182-12910@198.24.63.3927182-12910@198.24.63.39"},] (152) (85)
I had three such events which got missed out of some 25000 odd messages sent from SIPp. Other major difference that I see in the debug logs for problematic events are that there is a positive number in the second parentheses of evapi_recv_client() function. For the events that were invoked successfully the value of second parentheses is 0 for evapi_recv_client() function. Hope the debugging helps.
Debug log for a successful event: DEBUG: evapi [evapi_dispatch.c:598]: evapi_relay(): relaying event data [{"event":"REGISTER","tindex":"22127","tlabel":"1896682192","PhoneNumber":"73168","DeviceId":"abcd1234abcd1234","CallId":" 49957-13056@198.24.63.3949957-13056@198.24.63.39"}] (147) DEBUG: evapi [evapi_dispatch.c:623]: evapi_relay(): sending [0x7f132568e850] [147:{"event":"REGISTER","tindex":"22127","tlabel":"1896682192","PhoneNumber":"73168","DeviceId":"abcd1234abcd1234","CallId":" 49957-13056@198.24.63.3949957-13056@198.24.63.39"},] (152)
DEBUG: evapi [evapi_dispatch.c:488]: evapi_recv_notify(): received [0x7f132568e850] [147:{"event":"REGISTER","tindex":"22127","tlabel":"1896682192","PhoneNumber":"73168","DeviceId":"abcd1234abcd1234","CallId":" 49957-13056@198.24.63.3949957-13056@198.24.63.39"},] (152) NOTICE: evapi [evapi_dispatch.c:290]: evapi_recv_client(): {0} [ 198.24.63.45:48881] - received [147:{"event":"REGISTER","tindex":"22127","tlabel":"1896682192","PhoneNumber":"73168","DeviceId":"abcd1234abcd1234","CallId":" 49957-13056@198.24.63.3949957-13056@198.24.63.39"},] (152) (0)
Thanks,
- Jayesh
On Sat, Sep 19, 2015 at 3:55 AM Daniel-Constantin Mierla < miconda@gmail.com> wrote:
I pushed a commit to add more debug message while processing received data. You can use debugger module to set a higher debug level for evapi module in order to see what happens.
I checked the netstring packet size and it is invalid (unless email stripped some white chars there) -- for example in:
145:{"event":"REGISTER","tindex":"55567","tlabel":"627458699","PhoneNumber":"20711","DeviceId":"abcd1234abcd1234","CallId": 21225-3848@5.6.7.8"21225-3848@5.6.7.8" 21225-3848@5.6.7.8},
the size is 141, not 145 -- it looks like the size includes the size itself plus the delimiters ':,'. The size is only the effective data, see:
https://en.wikipedia.org/wiki/Netstring
If evapi gets a packet with an invalid size, then it discards the buffer content.
See if the app on the other side of evapi connection builds netstrings with wrong size.
Cheers, Daniel
On 18/09/15 23:19, Jayesh Nambiar wrote:
Here are the tests that I did:
With the patch applied, I see that Kamailio is invoking event individually for each netstring even when they come in different chunks. But I did see instances where when there were complete netstrings in a single chunk; kamailio did not raise an event for them. Here's the illustration:
T 1.2.3.4:48873 -> 5.6.7.8:3927 [AP]
146:{"event":"REGISTER","tindex":"16916","tlabel":"1753048277","PhoneNumber":"20708","DeviceId":"abcd1234abcd1234","CallId":" 21220-3848@5.6.7.821220-3848@5.6.7.8 "},144:{"event":"REGISTER","tindex":"7954","tlabel":"254315075","PhoneNumber":"20709","DeviceId":"abcd1234abcd1234","CallId":" 21223-3848@5.6.7.821223-3848@5.6.7.8 "},145:{"event":"REGISTER","tindex":"64529","tlabel":"599481568","PhoneNumber":"20709","DeviceId":"abcd1234abcd1234","CallId":" 21222-3848@5.6.7.821222-3848@5.6.7.8"},145:{"event":"REGISTER","tindex":"46605","tlabel":"112015324","PhoneNumber":"20710","DeviceId"
T 1.2.3.4:48873 -> 5.6.7.8:3927 [AP] :"abcd1234abcd1234","CallId":" 21224-3848@5.6.7.821224-3848@5.6.7.8 "},
*The above two chunks contain 4 proper netstrings where the second chunk contains part of the 4th netstring. In this case Evapi properly raised 4 individual events.*
1.2.3.4:48873 -> 5.6.7.8:3927 [AP]
145:{"event":"REGISTER","tindex":"55567","tlabel":"627458699","PhoneNumber":"20711","DeviceId":"abcd1234abcd1234","CallId":" 21225-3848@5.6.7.821225-3848@5.6.7.8 "},143:{"event":"REGISTER","tindex":"28682","tlabel":"9676691","PhoneNumber":"20712","DeviceId":"abcd1234abcd1234","CallId":" 21226-3848@5.6.7.821226-3848@5.6.7.8"},
*This above chunk contains two complete netstrings but Kamailio never raised events for these two netstrings.*
For the events that were not raised I see proper Kamailio Logs which is: evapi_recv_client(): {0} [1.2.3.4:48873] - received [145:{"event":"REGISTER","tindex":"55567","tlabel":"627458699","PhoneNumber":"20711","DeviceId":"abcd1234abcd1234","CallId":" 21225-3848@5.6.7.821225-3848@5.6.7.8 "},143:{"event":"REGISTER","tindex":"28682","tlabel":"9676691","PhoneNumber":"20712","DeviceId":"abcd1234abcd1234","CallId":" 21226-3848@5.6.7.821226-3848@5.6.7.8"},]
But I don't see the logs that I've written in the script after the event was raised, which means Kamailio did not invoke events for these two netstrings.
So in my tests, out of 27232 messages sent, there were approximately 27 messages for which the events were not raised by Evapi. The rate of messages started at 500cps and I stopped after I saw missed events at around 1200cps.
Thanks, and do let me know for any further tests or information required about the same.
- Jayesh
On Fri, Sep 18, 2015 at 2:55 PM, Jayesh Nambiar < jayesh1017@gmail.com jayesh1017@gmail.com> wrote:
Will have this tested by tomorrow and will get back to you. Thanks.
- Jayesh
On Fri, Sep 18, 2015 at 1:41 PM, Daniel-Constantin Mierla < miconda@gmail.commiconda@gmail.com> wrote:
Hello,
I just pushed a patch to master branch that should cope with partial data received on tcp connection. No time to test at all, therefore any feedback will be appreciated.
Cheers, Daniel
On 15/09/15 14:52, Daniel-Constantin Mierla wrote:
Hello,
I will look if there are options in libev to buffer data or try to implement a buffering mechanism locally for such cases.
Cheers, Daniel
On 14/09/15 23:00, Jayesh Nambiar wrote:
Hello Daniel, After further testing with evapi module, I figured that when Netstrings are used, an event route is invoked individually for each message even if if multiple netstring messages are received in a single TCP packet. But this doesn't work effectively when a single proper message is split-up in two packets. For Example, if a message arrives as: 12:Hello World!, 12:Hello World!, 12:Hello World! in a single packet, kamailio properly invokes the event route "evapi:message-received" thrice for every individual proper netstring message. But if the first packet contains: 12:Hello World!, 12:Hello And Second Packet contains: World!, 12:Hello World! the event route is invoked only once !!
The above pattern is very much possible while sending and receiving packets over TCP Socket. Our tests for receiving an approximately 150 byte message over evapi socket at the rate of roughly 1000cps causes a lot of real events to be missed because of the above problem. You can never be sure when TCP will split messages in different chunks. This definitely looks like a bug which makes it not very reliable at large scale deployments. Would really appreciate your inputs on this. Thanks;
- Jayesh
On Thu, Sep 10, 2015 at 4:01 PM, Jayesh Nambiar < jayesh1017@gmail.comjayesh1017@gmail.com> wrote:
Hi Daniel, Thanks for the quick response. So if I do not use Netstrings, does Kamailio allow me to create a custom logic in the script. For eg. if I decide to use newline as a delimiter, can I keep buffering the message until I encounter the delimiter from the event route and then execute whatever I have to within the script??
Thanks,
- Jayesh
On Thu, Sep 10, 2015 at 1:29 PM, Daniel-Constantin Mierla < miconda@gmail.commiconda@gmail.com> wrote:
> Hello, > > tcb is stream protocol and several messages can be queued on the > pipe at the same time. That is the reason for netstring format, to be able > to easily detect the boundaries of each message. If netstring format is > enabled and kamailio receives several messages at once, it splits them and > for each is executing the event route. > > If netstring format is not used, the kamailio is executing the event > route with the entire content that was read at once from the tcp connection. > > Cheers, > Daniel > > > On 09/09/15 22:01, Jayesh Nambiar wrote: > > Hello, > I'm exploring the evapi module for my kamailio to interface with an > external node.js app for third party stuff like AAA, billing engine tasks, > notifications and so on. I followed and took some ideas from the rtjson and > evapi tutorial found here( > http://kb.asipto.com/kamailio:k43-async-sip-routing-nodejs > http://kb.asipto.com/kamailio:k43-async-sip-routing-nodejs) to > build the node.js app consuming events. > When I stress tested the scenario using SIPp and tried sending a lot > of events at 300-350cps from Kamailio, I noticed that at times the client > is receiving 2-3 events in a single message together although I do > event_sync_relay once per SIP message received and have netstrings enabled. > I believe this is a typical behavior of TCP and needs to be handled by the > client using some kind of Netstring handler. Please correct me if I'm wrong. > And hence I'd like to know what particularly needs to be taken care > of while writing a client that is listening for events on raw tcp socket > and how does kamailio handle this situation while receiving messages over > TCP socket?? Does kamailio recognize the end of netstring properly on > evapi:message-received and give exactly one message to take care of on > every "message-received" event or should that be handled in the script > somewhere !! > I also referred cgrates client over evapi example which is written > in GO, but I couldnt find them handling TCP streams clearly either. > I'd really appreciate some expert suggestion here to make an > informed decision on using the evapi module for a large scale solution. > > Thanks, > > - Jayesh > > > > > _______________________________________________ > SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing listsr-users@lists.sip-router.orghttp://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users > > > -- > Daniel-Constantin Mierlahttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda > Book: SIP Routing With Kamailio - http://www.asipto.com > Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat > > > _______________________________________________ > SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing > list > sr-users@lists.sip-router.orgsr-users@lists.sip-router.org > http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users > http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users > >
-- Daniel-Constantin Mierlahttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
-- Daniel-Constantin Mierlahttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
-- Daniel-Constantin Mierlahttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
-- Daniel-Constantin Mierlahttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
Hello,
I haven't seen in the ngrep when the data containing next line is received:
{"event":"REGISTER","tindex":"3639","tlabel":"1889788252" ...
Do you still have it? Is it after:
{"event":"REGISTER","tindex":"12603","tlabel":"626010915", ...
Cheers, Daniel
On 22/09/15 09:09, Jayesh Nambiar wrote:
Hi, Took time to get back with the test results as I was trying to analyze too much myself before getting back on the list. Also I was stuck with weird error of module version mismatch, but I figured and solved that issue. So I do see the logs "frame size mismatch" when I miss the events. Since the logs are too huge, I am sending it in pastebin. Here it is: http://pastebin.com/MzTpYsrk
In Kamailio Logs, I see: DEBUG: evapi [evapi_dispatch.c:361]: evapi_recv_client(): frame size mismatch the ending char (h):[{"event":"REGISTER","tindex":"12603","tlabel":"626010915","PhoneNumber":"42000","D146:"event":"REGISTER","tindex":"3639","tlabel":"1889788252","P] (146)
Whereas the messages in ngrep looks good where the messages are split in two TCP packets. The first packet ends at D and the second packet is the proper continuation which send the rest. But it feels Kamailio skipped the beginning of the message and started parsing at colon again and it thinks the netstring started with D146 which is incorrect. Hence, it never emitted any event for the messages in the first chunk.
Hoping this to be helpful !! Thanks.
- Jayesh
On Mon, Sep 21, 2015 at 4:37 PM Daniel-Constantin Mierla <miconda@gmail.com mailto:miconda@gmail.com> wrote:
Hello, I pushed another commit to add more debug messages to see if the event route is supposed to be executed or not. Can you run the tests again and give again the log messages for missing event route executions? Cheers, Daniel On 21/09/15 09:04, Jayesh Nambiar wrote:Hi Daniel, I have captured debug messages after testing with your latest commit on master. Once again, here's what I'm doing exactly for my test: 1) Use SIPp to send OPTIONS towards Kamailio. 2) On getting options send event as follows: evapi_async_relay("{\"event\":\"REGISTER\",\"tindex\":\"$T(id_index)\",\"tlabel\":\"$T(id_label)\",\"PhoneNumber\":\"$avp(phone_number)\",\"DeviceId\":\"$avp(device_id)\",\"CallId\":\"$ci\"}"); 3) There's a client connected which listens for messages on this socket, parses the netstring, and sends same data back as netstring to Kamailio. 4) On the event_route[evapi:message-received, I do the following: xlog("L_INFO", "GOT [$evapi(msg)] from $evapi(srcaddr):$evapi(srcport)\n"); if($evapi(msg)=~"REGISTER" && $evapi(msg)=~"tindex") { jansson_get_field("$evapi(msg)", "tlabel", "$var(tlabel)"); jansson_get_field("$evapi(msg)", "tindex", "$var(tindex)"); $var(t_index) = $(var(tindex){s.int <http://s.int>}); $var(t_label) = $(var(tlabel){s.int <http://s.int>}); t_continue('$var(t_index)', '$var(t_label)', 'REGISTER_RESPONSE'); exit; } 5) On route[REGISTER_RESPONSE], I send 200 OK to SIPp Now here are detailed logs for which neither SIPp didnt get a response nor the GOT [$evapi(msg)] from $evapi(srcaddr):$evapi(srcport)\n"); got logged in the syslog: First Event: DEBUG: evapi [evapi_dispatch.c:598]: evapi_relay(): relaying event data [{"event":"REGISTER","tindex":"26266","tlabel":"587925078","PhoneNumber":"21956","DeviceId":"abcd1234abcd1234","CallId":"23021-12910@198.24.63.39 <mailto:23021-12910@198.24.63.39>"}] (146) DEBUG: evapi [evapi_dispatch.c:623]: evapi_relay(): sending [0x7f132588de68] [146:{"event":"REGISTER","tindex":"26266","tlabel":"587925078","PhoneNumber":"21956","DeviceId":"abcd1234abcd1234","CallId":"23021-12910@198.24.63.39 <mailto:23021-12910@198.24.63.39>"},] (151) DEBUG: evapi [evapi_dispatch.c:488]: evapi_recv_notify(): received [0x7f132588de68] [146:{"event":"REGISTER","tindex":"26266","tlabel":"587925078","PhoneNumber":"21956","DeviceId":"abcd1234abcd1234","CallId":"23021-12910@198.24.63.39 <mailto:23021-12910@198.24.63.39>"},] (151) NOTICE: evapi [evapi_dispatch.c:290]: evapi_recv_client(): {0} [198.24.63.45:48881 <http://198.24.63.45:48881>] - received [146:{"event":"REGISTER","tindex":"26266","tlabel":"587925078","PhoneNumber":"21956","DeviceId":"abcd1234abcd1234","CallId":"23021-12910@198.24.63.39 <mailto:23021-12910@198.24.63.39>"},] (151) (18) Second Event: DEBUG: evapi [evapi_dispatch.c:598]: evapi_relay(): relaying event data [{"event":"REGISTER","tindex":"32244","tlabel":"1637923412","PhoneNumber":"25597","DeviceId":"abcd1234abcd1234","CallId":"27182-12910@198.24.63.39 <mailto:27182-12910@198.24.63.39>"}] (147) DEBUG: evapi [evapi_dispatch.c:623]: evapi_relay(): sending [0x7f1325884410] [147:{"event":"REGISTER","tindex":"32244","tlabel":"1637923412","PhoneNumber":"25597","DeviceId":"abcd1234abcd1234","CallId":"27182-12910@198.24.63.39 <mailto:27182-12910@198.24.63.39>"},] (152) DEBUG: evapi [evapi_dispatch.c:488]: evapi_recv_notify(): received [0x7f1325884410] [147:{"event":"REGISTER","tindex":"32244","tlabel":"1637923412","PhoneNumber":"25597","DeviceId":"abcd1234abcd1234","CallId":"27182-12910@198.24.63.39 <mailto:27182-12910@198.24.63.39>"},] (152) NOTICE: evapi [evapi_dispatch.c:290]: evapi_recv_client(): {0} [198.24.63.45:48881 <http://198.24.63.45:48881>] - received [147:{"event":"REGISTER","tindex":"32244","tlabel":"1637923412","PhoneNumber":"25597","DeviceId":"abcd1234abcd1234","CallId":"27182-12910@198.24.63.39 <mailto:27182-12910@198.24.63.39>"},] (152) (85) I had three such events which got missed out of some 25000 odd messages sent from SIPp. Other major difference that I see in the debug logs for problematic events are that there is a positive number in the second parentheses of evapi_recv_client() function. For the events that were invoked successfully the value of second parentheses is 0 for evapi_recv_client() function. Hope the debugging helps. Debug log for a successful event: DEBUG: evapi [evapi_dispatch.c:598]: evapi_relay(): relaying event data [{"event":"REGISTER","tindex":"22127","tlabel":"1896682192","PhoneNumber":"73168","DeviceId":"abcd1234abcd1234","CallId":"49957-13056@198.24.63.39 <mailto:49957-13056@198.24.63.39>"}] (147) DEBUG: evapi [evapi_dispatch.c:623]: evapi_relay(): sending [0x7f132568e850] [147:{"event":"REGISTER","tindex":"22127","tlabel":"1896682192","PhoneNumber":"73168","DeviceId":"abcd1234abcd1234","CallId":"49957-13056@198.24.63.39 <mailto:49957-13056@198.24.63.39>"},] (152) DEBUG: evapi [evapi_dispatch.c:488]: evapi_recv_notify(): received [0x7f132568e850] [147:{"event":"REGISTER","tindex":"22127","tlabel":"1896682192","PhoneNumber":"73168","DeviceId":"abcd1234abcd1234","CallId":"49957-13056@198.24.63.39 <mailto:49957-13056@198.24.63.39>"},] (152) NOTICE: evapi [evapi_dispatch.c:290]: evapi_recv_client(): {0} [198.24.63.45:48881 <http://198.24.63.45:48881>] - received [147:{"event":"REGISTER","tindex":"22127","tlabel":"1896682192","PhoneNumber":"73168","DeviceId":"abcd1234abcd1234","CallId":"49957-13056@198.24.63.39 <mailto:49957-13056@198.24.63.39>"},] (152) (0) Thanks, - Jayesh On Sat, Sep 19, 2015 at 3:55 AM Daniel-Constantin Mierla <miconda@gmail.com <mailto:miconda@gmail.com>> wrote: I pushed a commit to add more debug message while processing received data. You can use debugger module to set a higher debug level for evapi module in order to see what happens. I checked the netstring packet size and it is invalid (unless email stripped some white chars there) -- for example in: 145:{"event":"REGISTER","tindex":"55567","tlabel":"627458699","PhoneNumber":"20711","DeviceId":"abcd1234abcd1234","CallId":"21225-3848@5.6.7.8" <mailto:21225-3848@5.6.7.8>}, the size is 141, not 145 -- it looks like the size includes the size itself plus the delimiters ':,'. The size is only the effective data, see: https://en.wikipedia.org/wiki/Netstring If evapi gets a packet with an invalid size, then it discards the buffer content. See if the app on the other side of evapi connection builds netstrings with wrong size. Cheers, Daniel On 18/09/15 23:19, Jayesh Nambiar wrote:Here are the tests that I did: With the patch applied, I see that Kamailio is invoking event individually for each netstring even when they come in different chunks. But I did see instances where when there were complete netstrings in a single chunk; kamailio did not raise an event for them. Here's the illustration: T 1.2.3.4:48873 <http://1.2.3.4:48873> -> 5.6.7.8:3927 <http://5.6.7.8:3927> [AP] 146:{"event":"REGISTER","tindex":"16916","tlabel":"1753048277","PhoneNumber":"20708","DeviceId":"abcd1234abcd1234","CallId":"21220-3848@5.6.7.8 <mailto:21220-3848@5.6.7.8>"},144:{"event":"REGISTER","tindex":"7954","tlabel":"254315075","PhoneNumber":"20709","DeviceId":"abcd1234abcd1234","CallId":"21223-3848@5.6.7.8 <mailto:21223-3848@5.6.7.8>"},145:{"event":"REGISTER","tindex":"64529","tlabel":"599481568","PhoneNumber":"20709","DeviceId":"abcd1234abcd1234","CallId":"21222-3848@5.6.7.8 <mailto:21222-3848@5.6.7.8>"},145:{"event":"REGISTER","tindex":"46605","tlabel":"112015324","PhoneNumber":"20710","DeviceId" T 1.2.3.4:48873 <http://1.2.3.4:48873> -> 5.6.7.8:3927 <http://5.6.7.8:3927> [AP] :"abcd1234abcd1234","CallId":"21224-3848@5.6.7.8 <mailto:21224-3848@5.6.7.8>"}, *The above two chunks contain 4 proper netstrings where the second chunk contains part of the 4th netstring. In this case Evapi properly raised 4 individual events.* 1.2.3.4:48873 <http://1.2.3.4:48873> -> 5.6.7.8:3927 <http://5.6.7.8:3927> [AP] 145:{"event":"REGISTER","tindex":"55567","tlabel":"627458699","PhoneNumber":"20711","DeviceId":"abcd1234abcd1234","CallId":"21225-3848@5.6.7.8 <mailto:21225-3848@5.6.7.8>"},143:{"event":"REGISTER","tindex":"28682","tlabel":"9676691","PhoneNumber":"20712","DeviceId":"abcd1234abcd1234","CallId":"21226-3848@5.6.7.8 <mailto:21226-3848@5.6.7.8>"}, *This above chunk contains two complete netstrings but Kamailio never raised events for these two netstrings.* * * For the events that were not raised I see proper Kamailio Logs which is: evapi_recv_client(): {0} [1.2.3.4:48873 <http://1.2.3.4:48873>] - received [145:{"event":"REGISTER","tindex":"55567","tlabel":"627458699","PhoneNumber":"20711","DeviceId":"abcd1234abcd1234","CallId":"21225-3848@5.6.7.8 <mailto:21225-3848@5.6.7.8>"},143:{"event":"REGISTER","tindex":"28682","tlabel":"9676691","PhoneNumber":"20712","DeviceId":"abcd1234abcd1234","CallId":"21226-3848@5.6.7.8 <mailto:21226-3848@5.6.7.8>"},] * * But I don't see the logs that I've written in the script after the event was raised, which means Kamailio did not invoke events for these two netstrings. So in my tests, out of 27232 messages sent, there were approximately 27 messages for which the events were not raised by Evapi. The rate of messages started at 500cps and I stopped after I saw missed events at around 1200cps. Thanks, and do let me know for any further tests or information required about the same. - Jayesh * * On Fri, Sep 18, 2015 at 2:55 PM, Jayesh Nambiar <jayesh1017@gmail.com <mailto:jayesh1017@gmail.com>> wrote: Will have this tested by tomorrow and will get back to you. Thanks. - Jayesh On Fri, Sep 18, 2015 at 1:41 PM, Daniel-Constantin Mierla <miconda@gmail.com <mailto:miconda@gmail.com>> wrote: Hello, I just pushed a patch to master branch that should cope with partial data received on tcp connection. No time to test at all, therefore any feedback will be appreciated. Cheers, Daniel On 15/09/15 14:52, Daniel-Constantin Mierla wrote:Hello, I will look if there are options in libev to buffer data or try to implement a buffering mechanism locally for such cases. Cheers, Daniel On 14/09/15 23:00, Jayesh Nambiar wrote:Hello Daniel, After further testing with evapi module, I figured that when Netstrings are used, an event route is invoked individually for each message even if if multiple netstring messages are received in a single TCP packet. But this doesn't work effectively when a single proper message is split-up in two packets. For Example, if a message arrives as: 12:Hello World!, 12:Hello World!, 12:Hello World! in a single packet, kamailio properly invokes the event route "evapi:message-received" thrice for every individual proper netstring message. But if the first packet contains: 12:Hello World!, 12:Hello And Second Packet contains: World!, 12:Hello World! the event route is invoked only once !! The above pattern is very much possible while sending and receiving packets over TCP Socket. Our tests for receiving an approximately 150 byte message over evapi socket at the rate of roughly 1000cps causes a lot of real events to be missed because of the above problem. You can never be sure when TCP will split messages in different chunks. This definitely looks like a bug which makes it not very reliable at large scale deployments. Would really appreciate your inputs on this. Thanks; - Jayesh On Thu, Sep 10, 2015 at 4:01 PM, Jayesh Nambiar <jayesh1017@gmail.com <mailto:jayesh1017@gmail.com>> wrote: Hi Daniel, Thanks for the quick response. So if I do not use Netstrings, does Kamailio allow me to create a custom logic in the script. For eg. if I decide to use newline as a delimiter, can I keep buffering the message until I encounter the delimiter from the event route and then execute whatever I have to within the script?? Thanks, - Jayesh On Thu, Sep 10, 2015 at 1:29 PM, Daniel-Constantin Mierla <miconda@gmail.com <mailto:miconda@gmail.com>> wrote: Hello, tcb is stream protocol and several messages can be queued on the pipe at the same time. That is the reason for netstring format, to be able to easily detect the boundaries of each message. If netstring format is enabled and kamailio receives several messages at once, it splits them and for each is executing the event route. If netstring format is not used, the kamailio is executing the event route with the entire content that was read at once from the tcp connection. Cheers, Daniel On 09/09/15 22:01, Jayesh Nambiar wrote:Hello, I'm exploring the evapi module for my kamailio to interface with an external node.js app for third party stuff like AAA, billing engine tasks, notifications and so on. I followed and took some ideas from the rtjson and evapi tutorial found here(http://kb.asipto.com/kamailio:k43-async-sip-routing-nodejs) to build the node.js app consuming events. When I stress tested the scenario using SIPp and tried sending a lot of events at 300-350cps from Kamailio, I noticed that at times the client is receiving 2-3 events in a single message together although I do event_sync_relay once per SIP message received and have netstrings enabled. I believe this is a typical behavior of TCP and needs to be handled by the client using some kind of Netstring handler. Please correct me if I'm wrong. And hence I'd like to know what particularly needs to be taken care of while writing a client that is listening for events on raw tcp socket and how does kamailio handle this situation while receiving messages over TCP socket?? Does kamailio recognize the end of netstring properly on evapi:message-received and give exactly one message to take care of on every "message-received" event or should that be handled in the script somewhere !! I also referred cgrates client over evapi example which is written in GO, but I couldnt find them handling TCP streams clearly either. I'd really appreciate some expert suggestion here to make an informed decision on using the evapi module for a large scale solution. Thanks, - Jayesh _______________________________________________ SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org <mailto:sr-users@lists.sip-router.org> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users-- Daniel-Constantin Mierla http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat _______________________________________________ SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org <mailto:sr-users@lists.sip-router.org> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users-- Daniel-Constantin Mierla http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat-- Daniel-Constantin Mierla http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat-- Daniel-Constantin Mierla http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat-- Daniel-Constantin Mierla http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
Hi, Here's the NGREP trace where {"event":"REGISTER","tindex":"3639","tlabel":"1889788252" ... is present on the first line itself.
198.24.63.45:48886 -> 198.24.63.39:3927 [AP]
146:{"event":"REGISTER","tindex":"3639","tlabel":"1889788252","PhoneNumber":"42008","DeviceId":"abcd1234abcd1234","CallId":" 11542-19872@198.24.63.39 "},147:{"event":"REGISTER","tindex":"12600","tlabel":"1916422130","PhoneNumber":"42009","DeviceId":"abcd1234abcd1234","CallId":" 11543-19872@198.24.63.39 "},147:{"event":"REGISTER","tindex":"51251","tlabel":"1549871937","PhoneNumber":"42010","DeviceId":"abcd1234abcd 1234","CallId":"11544-19872@198.24.63.39 "},147:{"event":"REGISTER","tindex":"60213","tlabel":"2100336684","PhoneNumber":"42011","DeviceId":"abcd1234abcd1234","CallId":" 11545-19872@198.24.63.39"},146:{"event
":"REGISTER","tindex":"33328","tlabel":"622251552","PhoneNumber":"42012","DeviceId":"abcd1234abcd1234","CallId":" 11546-19872@198.24.63.39 "},147:{"event":"REGISTER","tindex":"42289","tlabel":"1360864805","PhoneNumber":"42013","DeviceId":"abcd1234abcd1234","CallId":" 11547-19872@198.24.63.39 "},146:{"event":"REGISTER","tindex":"48204","tlabel":"315887672","PhoneNumber":"42014","DeviceId":"abcd1234abcd1234","CallId":" 11548-19872@198.24.63.39 "},145:{"event":"REGISTER","tindex":"36682","tlabel":"18660440","PhoneNumber":"42015","DeviceId":"abcd1234abcd1234","CallId":" 11549-19872@198.24.63.39"},
And yes, this packet is right after the second packet in the pastebin trace.
Thanks,
- Jayesh
On Tue, Sep 22, 2015 at 5:03 PM Daniel-Constantin Mierla miconda@gmail.com wrote:
Hello,
I haven't seen in the ngrep when the data containing next line is received:
{"event":"REGISTER","tindex":"3639","tlabel":"1889788252" ...
Do you still have it? Is it after:
{"event":"REGISTER","tindex":"12603","tlabel":"626010915", ...
Cheers, Daniel
On 22/09/15 09:09, Jayesh Nambiar wrote:
Hi, Took time to get back with the test results as I was trying to analyze too much myself before getting back on the list. Also I was stuck with weird error of module version mismatch, but I figured and solved that issue. So I do see the logs "frame size mismatch" when I miss the events. Since the logs are too huge, I am sending it in pastebin. Here it is: http://pastebin.com/MzTpYsrk
In Kamailio Logs, I see: DEBUG: evapi [evapi_dispatch.c:361]: evapi_recv_client(): frame size mismatch the ending char (h):[{"event":"REGISTER","tindex":"12603","tlabel":"626010915 ","PhoneNumber":"42000","D146:"event":"REGISTER","tindex":"3639","tlabel":" 1889788252","P] (146)
Whereas the messages in ngrep looks good where the messages are split in two TCP packets. The first packet ends at D and the second packet is the proper continuation which send the rest. But it feels Kamailio skipped the beginning of the message and started parsing at colon again and it thinks the netstring started with D146 which is incorrect. Hence, it never emitted any event for the messages in the first chunk.
Hoping this to be helpful !! Thanks.
- Jayesh
On Mon, Sep 21, 2015 at 4:37 PM Daniel-Constantin Mierla < miconda@gmail.com> wrote:
Hello,
I pushed another commit to add more debug messages to see if the event route is supposed to be executed or not. Can you run the tests again and give again the log messages for missing event route executions?
Cheers, Daniel
On 21/09/15 09:04, Jayesh Nambiar wrote:
Hi Daniel, I have captured debug messages after testing with your latest commit on master. Once again, here's what I'm doing exactly for my test:
- Use SIPp to send OPTIONS towards Kamailio.
- On getting options send event as follows:
evapi_async_relay("{"event":"REGISTER","tindex":"$T(id_index)","tlabel":"$T(id_label)","PhoneNumber":"$avp(phone_number)","DeviceId":"$avp(device_id)","CallId":"$ci"}"); 3) There's a client connected which listens for messages on this socket, parses the netstring, and sends same data back as netstring to Kamailio. 4) On the event_route[evapi:message-received, I do the following: xlog("L_INFO", "GOT [$evapi(msg)] from $evapi(srcaddr):$evapi(srcport)\n"); if($evapi(msg)=~"REGISTER" && $evapi(msg)=~"tindex") { jansson_get_field("$evapi(msg)", "tlabel", "$var(tlabel)"); jansson_get_field("$evapi(msg)", "tindex", "$var(tindex)"); $var(t_index) = $(var(tindex){s.int}); $var(t_label) = $(var(tlabel){s.int}); t_continue('$var(t_index)', '$var(t_label)', 'REGISTER_RESPONSE'); exit; } 5) On route[REGISTER_RESPONSE], I send 200 OK to SIPp
Now here are detailed logs for which neither SIPp didnt get a response nor the GOT [$evapi(msg)] from $evapi(srcaddr):$evapi(srcport)\n"); got logged in the syslog:
First Event: DEBUG: evapi [evapi_dispatch.c:598]: evapi_relay(): relaying event data [{"event":"REGISTER","tindex":"26266","tlabel":"587925078","PhoneNumber":"21956","DeviceId":"abcd1234abcd1234","CallId":" 23021-12910@198.24.63.39"}] (146) DEBUG: evapi [evapi_dispatch.c:623]: evapi_relay(): sending [0x7f132588de68] [146:{"event":"REGISTER","tindex":"26266","tlabel":"587925078","PhoneNumber":"21956","DeviceId":"abcd1234abcd1234","CallId":" 23021-12910@198.24.63.39"},] (151)
DEBUG: evapi [evapi_dispatch.c:488]: evapi_recv_notify(): received [0x7f132588de68] [146:{"event":"REGISTER","tindex":"26266","tlabel":"587925078","PhoneNumber":"21956","DeviceId":"abcd1234abcd1234","CallId":" 23021-12910@198.24.63.39"},] (151) NOTICE: evapi [evapi_dispatch.c:290]: evapi_recv_client(): {0} [ 198.24.63.45:48881] - received [146:{"event":"REGISTER","tindex":"26266","tlabel":"587925078","PhoneNumber":"21956","DeviceId":"abcd1234abcd1234","CallId":" 23021-12910@198.24.63.39"},] (151) (18)
Second Event: DEBUG: evapi [evapi_dispatch.c:598]: evapi_relay(): relaying event data [{"event":"REGISTER","tindex":"32244","tlabel":"1637923412","PhoneNumber":"25597","DeviceId":"abcd1234abcd1234","CallId":" 27182-12910@198.24.63.39"}] (147) DEBUG: evapi [evapi_dispatch.c:623]: evapi_relay(): sending [0x7f1325884410] [147:{"event":"REGISTER","tindex":"32244","tlabel":"1637923412","PhoneNumber":"25597","DeviceId":"abcd1234abcd1234","CallId":" 27182-12910@198.24.63.39"},] (152)
DEBUG: evapi [evapi_dispatch.c:488]: evapi_recv_notify(): received [0x7f1325884410] [147:{"event":"REGISTER","tindex":"32244","tlabel":"1637923412","PhoneNumber":"25597","DeviceId":"abcd1234abcd1234","CallId":" 27182-12910@198.24.63.39"},] (152) NOTICE: evapi [evapi_dispatch.c:290]: evapi_recv_client(): {0} [ 198.24.63.45:48881] - received [147:{"event":"REGISTER","tindex":"32244","tlabel":"1637923412","PhoneNumber":"25597","DeviceId":"abcd1234abcd1234","CallId":" 27182-12910@198.24.63.39"},] (152) (85)
I had three such events which got missed out of some 25000 odd messages sent from SIPp. Other major difference that I see in the debug logs for problematic events are that there is a positive number in the second parentheses of evapi_recv_client() function. For the events that were invoked successfully the value of second parentheses is 0 for evapi_recv_client() function. Hope the debugging helps.
Debug log for a successful event: DEBUG: evapi [evapi_dispatch.c:598]: evapi_relay(): relaying event data [{"event":"REGISTER","tindex":"22127","tlabel":"1896682192","PhoneNumber":"73168","DeviceId":"abcd1234abcd1234","CallId":" 49957-13056@198.24.63.39"}] (147) DEBUG: evapi [evapi_dispatch.c:623]: evapi_relay(): sending [0x7f132568e850] [147:{"event":"REGISTER","tindex":"22127","tlabel":"1896682192","PhoneNumber":"73168","DeviceId":"abcd1234abcd1234","CallId":" 49957-13056@198.24.63.39"},] (152)
DEBUG: evapi [evapi_dispatch.c:488]: evapi_recv_notify(): received [0x7f132568e850] [147:{"event":"REGISTER","tindex":"22127","tlabel":"1896682192","PhoneNumber":"73168","DeviceId":"abcd1234abcd1234","CallId":" 49957-13056@198.24.63.39"},] (152) NOTICE: evapi [evapi_dispatch.c:290]: evapi_recv_client(): {0} [ 198.24.63.45:48881] - received [147:{"event":"REGISTER","tindex":"22127","tlabel":"1896682192","PhoneNumber":"73168","DeviceId":"abcd1234abcd1234","CallId":" 49957-13056@198.24.63.39"},] (152) (0)
Thanks,
- Jayesh
On Sat, Sep 19, 2015 at 3:55 AM Daniel-Constantin Mierla < miconda@gmail.com> wrote:
I pushed a commit to add more debug message while processing received data. You can use debugger module to set a higher debug level for evapi module in order to see what happens.
I checked the netstring packet size and it is invalid (unless email stripped some white chars there) -- for example in:
145:{"event":"REGISTER","tindex":"55567","tlabel":"627458699","PhoneNumber":"20711","DeviceId":"abcd1234abcd1234","CallId": "21225-3848@5.6.7.8" 21225-3848@5.6.7.8 21225-3848@5.6.7.8},
the size is 141, not 145 -- it looks like the size includes the size itself plus the delimiters ':,'. The size is only the effective data, see:
https://en.wikipedia.org/wiki/Netstring
If evapi gets a packet with an invalid size, then it discards the buffer content.
See if the app on the other side of evapi connection builds netstrings with wrong size.
Cheers, Daniel
On 18/09/15 23:19, Jayesh Nambiar wrote:
Here are the tests that I did:
With the patch applied, I see that Kamailio is invoking event individually for each netstring even when they come in different chunks. But I did see instances where when there were complete netstrings in a single chunk; kamailio did not raise an event for them. Here's the illustration:
T 1.2.3.4:48873 -> 5.6.7.8:3927 [AP]
146:{"event":"REGISTER","tindex":"16916","tlabel":"1753048277","PhoneNumber":"20708","DeviceId":"abcd1234abcd1234","CallId":" 21220-3848@5.6.7.8 "},144:{"event":"REGISTER","tindex":"7954","tlabel":"254315075","PhoneNumber":"20709","DeviceId":"abcd1234abcd1234","CallId":" 21223-3848@5.6.7.8 "},145:{"event":"REGISTER","tindex":"64529","tlabel":"599481568","PhoneNumber":"20709","DeviceId":"abcd1234abcd1234","CallId":" 21222-3848@5.6.7.8"},145:{"event":"REGISTER","tindex":"46605","tlabel":"112015324","PhoneNumber":"20710","DeviceId"
T 1.2.3.4:48873 -> 5.6.7.8:3927 [AP] :"abcd1234abcd1234","CallId":"21224-3848@5.6.7.8"},
*The above two chunks contain 4 proper netstrings where the second chunk contains part of the 4th netstring. In this case Evapi properly raised 4 individual events.*
1.2.3.4:48873 -> 5.6.7.8:3927 [AP]
145:{"event":"REGISTER","tindex":"55567","tlabel":"627458699","PhoneNumber":"20711","DeviceId":"abcd1234abcd1234","CallId":" 21225-3848@5.6.7.8 "},143:{"event":"REGISTER","tindex":"28682","tlabel":"9676691","PhoneNumber":"20712","DeviceId":"abcd1234abcd1234","CallId":" 21226-3848@5.6.7.8"},
*This above chunk contains two complete netstrings but Kamailio never raised events for these two netstrings.*
For the events that were not raised I see proper Kamailio Logs which is: evapi_recv_client(): {0} [1.2.3.4:48873] - received [145:{"event":"REGISTER","tindex":"55567","tlabel":"627458699","PhoneNumber":"20711","DeviceId":"abcd1234abcd1234","CallId":" 21225-3848@5.6.7.8 "},143:{"event":"REGISTER","tindex":"28682","tlabel":"9676691","PhoneNumber":"20712","DeviceId":"abcd1234abcd1234","CallId":" 21226-3848@5.6.7.8"},]
But I don't see the logs that I've written in the script after the event was raised, which means Kamailio did not invoke events for these two netstrings.
So in my tests, out of 27232 messages sent, there were approximately 27 messages for which the events were not raised by Evapi. The rate of messages started at 500cps and I stopped after I saw missed events at around 1200cps.
Thanks, and do let me know for any further tests or information required about the same.
- Jayesh
On Fri, Sep 18, 2015 at 2:55 PM, Jayesh Nambiar jayesh1017@gmail.com wrote:
Will have this tested by tomorrow and will get back to you. Thanks.
- Jayesh
On Fri, Sep 18, 2015 at 1:41 PM, Daniel-Constantin Mierla < miconda@gmail.com> wrote:
Hello,
I just pushed a patch to master branch that should cope with partial data received on tcp connection. No time to test at all, therefore any feedback will be appreciated.
Cheers, Daniel
On 15/09/15 14:52, Daniel-Constantin Mierla wrote:
Hello,
I will look if there are options in libev to buffer data or try to implement a buffering mechanism locally for such cases.
Cheers, Daniel
On 14/09/15 23:00, Jayesh Nambiar wrote:
Hello Daniel, After further testing with evapi module, I figured that when Netstrings are used, an event route is invoked individually for each message even if if multiple netstring messages are received in a single TCP packet. But this doesn't work effectively when a single proper message is split-up in two packets. For Example, if a message arrives as: 12:Hello World!, 12:Hello World!, 12:Hello World! in a single packet, kamailio properly invokes the event route "evapi:message-received" thrice for every individual proper netstring message. But if the first packet contains: 12:Hello World!, 12:Hello And Second Packet contains: World!, 12:Hello World! the event route is invoked only once !!
The above pattern is very much possible while sending and receiving packets over TCP Socket. Our tests for receiving an approximately 150 byte message over evapi socket at the rate of roughly 1000cps causes a lot of real events to be missed because of the above problem. You can never be sure when TCP will split messages in different chunks. This definitely looks like a bug which makes it not very reliable at large scale deployments. Would really appreciate your inputs on this. Thanks;
- Jayesh
On Thu, Sep 10, 2015 at 4:01 PM, Jayesh Nambiar jayesh1017@gmail.com wrote:
Hi Daniel, Thanks for the quick response. So if I do not use Netstrings, does Kamailio allow me to create a custom logic in the script. For eg. if I decide to use newline as a delimiter, can I keep buffering the message until I encounter the delimiter from the event route and then execute whatever I have to within the script??
Thanks,
- Jayesh
On Thu, Sep 10, 2015 at 1:29 PM, Daniel-Constantin Mierla < miconda@gmail.com> wrote:
> Hello, > > tcb is stream protocol and several messages can be queued on the > pipe at the same time. That is the reason for netstring format, to be able > to easily detect the boundaries of each message. If netstring format is > enabled and kamailio receives several messages at once, it splits them and > for each is executing the event route. > > If netstring format is not used, the kamailio is executing the event > route with the entire content that was read at once from the tcp connection. > > Cheers, > Daniel > > > On 09/09/15 22:01, Jayesh Nambiar wrote: > > Hello, > I'm exploring the evapi module for my kamailio to interface with an > external node.js app for third party stuff like AAA, billing engine tasks, > notifications and so on. I followed and took some ideas from the rtjson and > evapi tutorial found here( > http://kb.asipto.com/kamailio:k43-async-sip-routing-nodejs) to > build the node.js app consuming events. > When I stress tested the scenario using SIPp and tried sending a lot > of events at 300-350cps from Kamailio, I noticed that at times the client > is receiving 2-3 events in a single message together although I do > event_sync_relay once per SIP message received and have netstrings enabled. > I believe this is a typical behavior of TCP and needs to be handled by the > client using some kind of Netstring handler. Please correct me if I'm wrong. > And hence I'd like to know what particularly needs to be taken care > of while writing a client that is listening for events on raw tcp socket > and how does kamailio handle this situation while receiving messages over > TCP socket?? Does kamailio recognize the end of netstring properly on > evapi:message-received and give exactly one message to take care of on > every "message-received" event or should that be handled in the script > somewhere !! > I also referred cgrates client over evapi example which is written > in GO, but I couldnt find them handling TCP streams clearly either. > I'd really appreciate some expert suggestion here to make an > informed decision on using the evapi module for a large scale solution. > > Thanks, > > - Jayesh > > > > > _______________________________________________ > SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing listsr-users@lists.sip-router.orghttp://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users > > > -- > Daniel-Constantin Mierlahttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda > Book: SIP Routing With Kamailio - http://www.asipto.com > Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat > > > _______________________________________________ > SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing > list > sr-users@lists.sip-router.org > http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users > >
-- Daniel-Constantin Mierlahttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
-- Daniel-Constantin Mierlahttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
-- Daniel-Constantin Mierlahttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
-- Daniel-Constantin Mierlahttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
-- Daniel-Constantin Mierlahttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
Hello Daniel, Just checking if you got what you were looking for. Do let me know if you need more info on the same.
Thanks,
- Jayesh
On Tue, Sep 22, 2015 at 8:23 PM Jayesh Nambiar jayesh1017@gmail.com wrote:
Hi, Here's the NGREP trace where {"event":"REGISTER","tindex":"3639","tlabel":"1889788252" ... is present on the first line itself.
198.24.63.45:48886 -> 198.24.63.39:3927 [AP]
146:{"event":"REGISTER","tindex":"3639","tlabel":"1889788252","PhoneNumber":"42008","DeviceId":"abcd1234abcd1234","CallId":" 11542-19872@198.24.63.39 "},147:{"event":"REGISTER","tindex":"12600","tlabel":"1916422130","PhoneNumber":"42009","DeviceId":"abcd1234abcd1234","CallId":" 11543-19872@198.24.63.39 "},147:{"event":"REGISTER","tindex":"51251","tlabel":"1549871937","PhoneNumber":"42010","DeviceId":"abcd1234abcd 1234","CallId":"11544-19872@198.24.63.39 "},147:{"event":"REGISTER","tindex":"60213","tlabel":"2100336684","PhoneNumber":"42011","DeviceId":"abcd1234abcd1234","CallId":" 11545-19872@198.24.63.39"},146:{"event
":"REGISTER","tindex":"33328","tlabel":"622251552","PhoneNumber":"42012","DeviceId":"abcd1234abcd1234","CallId":" 11546-19872@198.24.63.39 "},147:{"event":"REGISTER","tindex":"42289","tlabel":"1360864805","PhoneNumber":"42013","DeviceId":"abcd1234abcd1234","CallId":" 11547-19872@198.24.63.39 "},146:{"event":"REGISTER","tindex":"48204","tlabel":"315887672","PhoneNumber":"42014","DeviceId":"abcd1234abcd1234","CallId":" 11548-19872@198.24.63.39 "},145:{"event":"REGISTER","tindex":"36682","tlabel":"18660440","PhoneNumber":"42015","DeviceId":"abcd1234abcd1234","CallId":" 11549-19872@198.24.63.39"},
And yes, this packet is right after the second packet in the pastebin trace.
Thanks,
- Jayesh
On Tue, Sep 22, 2015 at 5:03 PM Daniel-Constantin Mierla < miconda@gmail.com> wrote:
Hello,
I haven't seen in the ngrep when the data containing next line is received:
{"event":"REGISTER","tindex":"3639","tlabel":"1889788252" ...
Do you still have it? Is it after:
{"event":"REGISTER","tindex":"12603","tlabel":"626010915", ...
Cheers, Daniel
On 22/09/15 09:09, Jayesh Nambiar wrote:
Hi, Took time to get back with the test results as I was trying to analyze too much myself before getting back on the list. Also I was stuck with weird error of module version mismatch, but I figured and solved that issue. So I do see the logs "frame size mismatch" when I miss the events. Since the logs are too huge, I am sending it in pastebin. Here it is: http://pastebin.com/MzTpYsrk
In Kamailio Logs, I see: DEBUG: evapi [evapi_dispatch.c:361]: evapi_recv_client(): frame size mismatch the ending char (h):[{"event":"REGISTER","tindex":"12603","tlabel":"626010915 ","PhoneNumber":"42000","D146:"event":"REGISTER","tindex":"3639","tlabel":" 1889788252","P] (146)
Whereas the messages in ngrep looks good where the messages are split in two TCP packets. The first packet ends at D and the second packet is the proper continuation which send the rest. But it feels Kamailio skipped the beginning of the message and started parsing at colon again and it thinks the netstring started with D146 which is incorrect. Hence, it never emitted any event for the messages in the first chunk.
Hoping this to be helpful !! Thanks.
- Jayesh
On Mon, Sep 21, 2015 at 4:37 PM Daniel-Constantin Mierla < miconda@gmail.com> wrote:
Hello,
I pushed another commit to add more debug messages to see if the event route is supposed to be executed or not. Can you run the tests again and give again the log messages for missing event route executions?
Cheers, Daniel
On 21/09/15 09:04, Jayesh Nambiar wrote:
Hi Daniel, I have captured debug messages after testing with your latest commit on master. Once again, here's what I'm doing exactly for my test:
- Use SIPp to send OPTIONS towards Kamailio.
- On getting options send event as follows:
evapi_async_relay("{"event":"REGISTER","tindex":"$T(id_index)","tlabel":"$T(id_label)","PhoneNumber":"$avp(phone_number)","DeviceId":"$avp(device_id)","CallId":"$ci"}"); 3) There's a client connected which listens for messages on this socket, parses the netstring, and sends same data back as netstring to Kamailio. 4) On the event_route[evapi:message-received, I do the following: xlog("L_INFO", "GOT [$evapi(msg)] from $evapi(srcaddr):$evapi(srcport)\n"); if($evapi(msg)=~"REGISTER" && $evapi(msg)=~"tindex") { jansson_get_field("$evapi(msg)", "tlabel", "$var(tlabel)"); jansson_get_field("$evapi(msg)", "tindex", "$var(tindex)"); $var(t_index) = $(var(tindex){s.int}); $var(t_label) = $(var(tlabel){s.int}); t_continue('$var(t_index)', '$var(t_label)', 'REGISTER_RESPONSE'); exit; } 5) On route[REGISTER_RESPONSE], I send 200 OK to SIPp
Now here are detailed logs for which neither SIPp didnt get a response nor the GOT [$evapi(msg)] from $evapi(srcaddr):$evapi(srcport)\n"); got logged in the syslog:
First Event: DEBUG: evapi [evapi_dispatch.c:598]: evapi_relay(): relaying event data [{"event":"REGISTER","tindex":"26266","tlabel":"587925078","PhoneNumber":"21956","DeviceId":"abcd1234abcd1234","CallId":" 23021-12910@198.24.63.39"}] (146) DEBUG: evapi [evapi_dispatch.c:623]: evapi_relay(): sending [0x7f132588de68] [146:{"event":"REGISTER","tindex":"26266","tlabel":"587925078","PhoneNumber":"21956","DeviceId":"abcd1234abcd1234","CallId":" 23021-12910@198.24.63.39"},] (151)
DEBUG: evapi [evapi_dispatch.c:488]: evapi_recv_notify(): received [0x7f132588de68] [146:{"event":"REGISTER","tindex":"26266","tlabel":"587925078","PhoneNumber":"21956","DeviceId":"abcd1234abcd1234","CallId":" 23021-12910@198.24.63.39"},] (151) NOTICE: evapi [evapi_dispatch.c:290]: evapi_recv_client(): {0} [ 198.24.63.45:48881] - received [146:{"event":"REGISTER","tindex":"26266","tlabel":"587925078","PhoneNumber":"21956","DeviceId":"abcd1234abcd1234","CallId":" 23021-12910@198.24.63.39"},] (151) (18)
Second Event: DEBUG: evapi [evapi_dispatch.c:598]: evapi_relay(): relaying event data [{"event":"REGISTER","tindex":"32244","tlabel":"1637923412","PhoneNumber":"25597","DeviceId":"abcd1234abcd1234","CallId":" 27182-12910@198.24.63.39"}] (147) DEBUG: evapi [evapi_dispatch.c:623]: evapi_relay(): sending [0x7f1325884410] [147:{"event":"REGISTER","tindex":"32244","tlabel":"1637923412","PhoneNumber":"25597","DeviceId":"abcd1234abcd1234","CallId":" 27182-12910@198.24.63.39"},] (152)
DEBUG: evapi [evapi_dispatch.c:488]: evapi_recv_notify(): received [0x7f1325884410] [147:{"event":"REGISTER","tindex":"32244","tlabel":"1637923412","PhoneNumber":"25597","DeviceId":"abcd1234abcd1234","CallId":" 27182-12910@198.24.63.39"},] (152) NOTICE: evapi [evapi_dispatch.c:290]: evapi_recv_client(): {0} [ 198.24.63.45:48881] - received [147:{"event":"REGISTER","tindex":"32244","tlabel":"1637923412","PhoneNumber":"25597","DeviceId":"abcd1234abcd1234","CallId":" 27182-12910@198.24.63.39"},] (152) (85)
I had three such events which got missed out of some 25000 odd messages sent from SIPp. Other major difference that I see in the debug logs for problematic events are that there is a positive number in the second parentheses of evapi_recv_client() function. For the events that were invoked successfully the value of second parentheses is 0 for evapi_recv_client() function. Hope the debugging helps.
Debug log for a successful event: DEBUG: evapi [evapi_dispatch.c:598]: evapi_relay(): relaying event data [{"event":"REGISTER","tindex":"22127","tlabel":"1896682192","PhoneNumber":"73168","DeviceId":"abcd1234abcd1234","CallId":" 49957-13056@198.24.63.39"}] (147) DEBUG: evapi [evapi_dispatch.c:623]: evapi_relay(): sending [0x7f132568e850] [147:{"event":"REGISTER","tindex":"22127","tlabel":"1896682192","PhoneNumber":"73168","DeviceId":"abcd1234abcd1234","CallId":" 49957-13056@198.24.63.39"},] (152)
DEBUG: evapi [evapi_dispatch.c:488]: evapi_recv_notify(): received [0x7f132568e850] [147:{"event":"REGISTER","tindex":"22127","tlabel":"1896682192","PhoneNumber":"73168","DeviceId":"abcd1234abcd1234","CallId":" 49957-13056@198.24.63.39"},] (152) NOTICE: evapi [evapi_dispatch.c:290]: evapi_recv_client(): {0} [ 198.24.63.45:48881] - received [147:{"event":"REGISTER","tindex":"22127","tlabel":"1896682192","PhoneNumber":"73168","DeviceId":"abcd1234abcd1234","CallId":" 49957-13056@198.24.63.39"},] (152) (0)
Thanks,
- Jayesh
On Sat, Sep 19, 2015 at 3:55 AM Daniel-Constantin Mierla < miconda@gmail.com> wrote:
I pushed a commit to add more debug message while processing received data. You can use debugger module to set a higher debug level for evapi module in order to see what happens.
I checked the netstring packet size and it is invalid (unless email stripped some white chars there) -- for example in:
145:{"event":"REGISTER","tindex":"55567","tlabel":"627458699","PhoneNumber":"20711","DeviceId":"abcd1234abcd1234","CallId": "21225-3848@5.6.7.8" 21225-3848@5.6.7.8 21225-3848@5.6.7.8},
the size is 141, not 145 -- it looks like the size includes the size itself plus the delimiters ':,'. The size is only the effective data, see:
https://en.wikipedia.org/wiki/Netstring
If evapi gets a packet with an invalid size, then it discards the buffer content.
See if the app on the other side of evapi connection builds netstrings with wrong size.
Cheers, Daniel
On 18/09/15 23:19, Jayesh Nambiar wrote:
Here are the tests that I did:
With the patch applied, I see that Kamailio is invoking event individually for each netstring even when they come in different chunks. But I did see instances where when there were complete netstrings in a single chunk; kamailio did not raise an event for them. Here's the illustration:
T 1.2.3.4:48873 -> 5.6.7.8:3927 [AP]
146:{"event":"REGISTER","tindex":"16916","tlabel":"1753048277","PhoneNumber":"20708","DeviceId":"abcd1234abcd1234","CallId":" 21220-3848@5.6.7.8 "},144:{"event":"REGISTER","tindex":"7954","tlabel":"254315075","PhoneNumber":"20709","DeviceId":"abcd1234abcd1234","CallId":" 21223-3848@5.6.7.8 "},145:{"event":"REGISTER","tindex":"64529","tlabel":"599481568","PhoneNumber":"20709","DeviceId":"abcd1234abcd1234","CallId":" 21222-3848@5.6.7.8"},145:{"event":"REGISTER","tindex":"46605","tlabel":"112015324","PhoneNumber":"20710","DeviceId"
T 1.2.3.4:48873 -> 5.6.7.8:3927 [AP] :"abcd1234abcd1234","CallId":"21224-3848@5.6.7.8"},
*The above two chunks contain 4 proper netstrings where the second chunk contains part of the 4th netstring. In this case Evapi properly raised 4 individual events.*
1.2.3.4:48873 -> 5.6.7.8:3927 [AP]
145:{"event":"REGISTER","tindex":"55567","tlabel":"627458699","PhoneNumber":"20711","DeviceId":"abcd1234abcd1234","CallId":" 21225-3848@5.6.7.8 "},143:{"event":"REGISTER","tindex":"28682","tlabel":"9676691","PhoneNumber":"20712","DeviceId":"abcd1234abcd1234","CallId":" 21226-3848@5.6.7.8"},
*This above chunk contains two complete netstrings but Kamailio never raised events for these two netstrings.*
For the events that were not raised I see proper Kamailio Logs which is: evapi_recv_client(): {0} [1.2.3.4:48873] - received [145:{"event":"REGISTER","tindex":"55567","tlabel":"627458699","PhoneNumber":"20711","DeviceId":"abcd1234abcd1234","CallId":" 21225-3848@5.6.7.8 "},143:{"event":"REGISTER","tindex":"28682","tlabel":"9676691","PhoneNumber":"20712","DeviceId":"abcd1234abcd1234","CallId":" 21226-3848@5.6.7.8"},]
But I don't see the logs that I've written in the script after the event was raised, which means Kamailio did not invoke events for these two netstrings.
So in my tests, out of 27232 messages sent, there were approximately 27 messages for which the events were not raised by Evapi. The rate of messages started at 500cps and I stopped after I saw missed events at around 1200cps.
Thanks, and do let me know for any further tests or information required about the same.
- Jayesh
On Fri, Sep 18, 2015 at 2:55 PM, Jayesh Nambiar jayesh1017@gmail.com wrote:
Will have this tested by tomorrow and will get back to you. Thanks.
- Jayesh
On Fri, Sep 18, 2015 at 1:41 PM, Daniel-Constantin Mierla < miconda@gmail.com> wrote:
Hello,
I just pushed a patch to master branch that should cope with partial data received on tcp connection. No time to test at all, therefore any feedback will be appreciated.
Cheers, Daniel
On 15/09/15 14:52, Daniel-Constantin Mierla wrote:
Hello,
I will look if there are options in libev to buffer data or try to implement a buffering mechanism locally for such cases.
Cheers, Daniel
On 14/09/15 23:00, Jayesh Nambiar wrote:
Hello Daniel, After further testing with evapi module, I figured that when Netstrings are used, an event route is invoked individually for each message even if if multiple netstring messages are received in a single TCP packet. But this doesn't work effectively when a single proper message is split-up in two packets. For Example, if a message arrives as: 12:Hello World!, 12:Hello World!, 12:Hello World! in a single packet, kamailio properly invokes the event route "evapi:message-received" thrice for every individual proper netstring message. But if the first packet contains: 12:Hello World!, 12:Hello And Second Packet contains: World!, 12:Hello World! the event route is invoked only once !!
The above pattern is very much possible while sending and receiving packets over TCP Socket. Our tests for receiving an approximately 150 byte message over evapi socket at the rate of roughly 1000cps causes a lot of real events to be missed because of the above problem. You can never be sure when TCP will split messages in different chunks. This definitely looks like a bug which makes it not very reliable at large scale deployments. Would really appreciate your inputs on this. Thanks;
- Jayesh
On Thu, Sep 10, 2015 at 4:01 PM, Jayesh Nambiar <jayesh1017@gmail.com > wrote:
> Hi Daniel, > Thanks for the quick response. So if I do not use Netstrings, does > Kamailio allow me to create a custom logic in the script. For eg. if I > decide to use newline as a delimiter, can I keep buffering the message > until I encounter the delimiter from the event route and then execute > whatever I have to within the script?? > > Thanks, > > - Jayesh > > On Thu, Sep 10, 2015 at 1:29 PM, Daniel-Constantin Mierla < > miconda@gmail.com> wrote: > >> Hello, >> >> tcb is stream protocol and several messages can be queued on the >> pipe at the same time. That is the reason for netstring format, to be able >> to easily detect the boundaries of each message. If netstring format is >> enabled and kamailio receives several messages at once, it splits them and >> for each is executing the event route. >> >> If netstring format is not used, the kamailio is executing the >> event route with the entire content that was read at once from the tcp >> connection. >> >> Cheers, >> Daniel >> >> >> On 09/09/15 22:01, Jayesh Nambiar wrote: >> >> Hello, >> I'm exploring the evapi module for my kamailio to interface with an >> external node.js app for third party stuff like AAA, billing engine tasks, >> notifications and so on. I followed and took some ideas from the rtjson and >> evapi tutorial found here( >> http://kb.asipto.com/kamailio:k43-async-sip-routing-nodejs) to >> build the node.js app consuming events. >> When I stress tested the scenario using SIPp and tried sending a >> lot of events at 300-350cps from Kamailio, I noticed that at times the >> client is receiving 2-3 events in a single message together although I do >> event_sync_relay once per SIP message received and have netstrings enabled. >> I believe this is a typical behavior of TCP and needs to be handled by the >> client using some kind of Netstring handler. Please correct me if I'm wrong. >> And hence I'd like to know what particularly needs to be taken care >> of while writing a client that is listening for events on raw tcp socket >> and how does kamailio handle this situation while receiving messages over >> TCP socket?? Does kamailio recognize the end of netstring properly on >> evapi:message-received and give exactly one message to take care of on >> every "message-received" event or should that be handled in the script >> somewhere !! >> I also referred cgrates client over evapi example which is written >> in GO, but I couldnt find them handling TCP streams clearly either. >> I'd really appreciate some expert suggestion here to make an >> informed decision on using the evapi module for a large scale solution. >> >> Thanks, >> >> - Jayesh >> >> >> >> >> _______________________________________________ >> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing listsr-users@lists.sip-router.orghttp://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users >> >> >> -- >> Daniel-Constantin Mierlahttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda >> Book: SIP Routing With Kamailio - http://www.asipto.com >> Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat >> >> >> _______________________________________________ >> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing >> list >> sr-users@lists.sip-router.org >> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users >> >> >
-- Daniel-Constantin Mierlahttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
-- Daniel-Constantin Mierlahttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
-- Daniel-Constantin Mierlahttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
-- Daniel-Constantin Mierlahttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
-- Daniel-Constantin Mierlahttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
Hi Daniel, I think this is not solved completely after you applied the patch and my testing shows that there are some problems left out to solve. Please do let me know if you are looking at more inputs which I might be able to test and provide. Thanks,
- Jayesh
On Wed, Sep 23, 2015 at 8:24 PM Jayesh Nambiar jayesh1017@gmail.com wrote:
Hello Daniel, Just checking if you got what you were looking for. Do let me know if you need more info on the same.
Thanks,
- Jayesh
On Tue, Sep 22, 2015 at 8:23 PM Jayesh Nambiar jayesh1017@gmail.com wrote:
Hi, Here's the NGREP trace where {"event":"REGISTER","tindex":"3639","tlabel":"1889788252" ... is present on the first line itself.
198.24.63.45:48886 -> 198.24.63.39:3927 [AP]
146:{"event":"REGISTER","tindex":"3639","tlabel":"1889788252","PhoneNumber":"42008","DeviceId":"abcd1234abcd1234","CallId":" 11542-19872@198.24.63.39 "},147:{"event":"REGISTER","tindex":"12600","tlabel":"1916422130","PhoneNumber":"42009","DeviceId":"abcd1234abcd1234","CallId":" 11543-19872@198.24.63.39 "},147:{"event":"REGISTER","tindex":"51251","tlabel":"1549871937","PhoneNumber":"42010","DeviceId":"abcd1234abcd 1234","CallId":"11544-19872@198.24.63.39 "},147:{"event":"REGISTER","tindex":"60213","tlabel":"2100336684","PhoneNumber":"42011","DeviceId":"abcd1234abcd1234","CallId":" 11545-19872@198.24.63.39"},146:{"event
":"REGISTER","tindex":"33328","tlabel":"622251552","PhoneNumber":"42012","DeviceId":"abcd1234abcd1234","CallId":" 11546-19872@198.24.63.39 "},147:{"event":"REGISTER","tindex":"42289","tlabel":"1360864805","PhoneNumber":"42013","DeviceId":"abcd1234abcd1234","CallId":" 11547-19872@198.24.63.39 "},146:{"event":"REGISTER","tindex":"48204","tlabel":"315887672","PhoneNumber":"42014","DeviceId":"abcd1234abcd1234","CallId":" 11548-19872@198.24.63.39 "},145:{"event":"REGISTER","tindex":"36682","tlabel":"18660440","PhoneNumber":"42015","DeviceId":"abcd1234abcd1234","CallId":" 11549-19872@198.24.63.39"},
And yes, this packet is right after the second packet in the pastebin trace.
Thanks,
- Jayesh
On Tue, Sep 22, 2015 at 5:03 PM Daniel-Constantin Mierla < miconda@gmail.com> wrote:
Hello,
I haven't seen in the ngrep when the data containing next line is received:
{"event":"REGISTER","tindex":"3639","tlabel":"1889788252" ...
Do you still have it? Is it after:
{"event":"REGISTER","tindex":"12603","tlabel":"626010915", ...
Cheers, Daniel
On 22/09/15 09:09, Jayesh Nambiar wrote:
Hi, Took time to get back with the test results as I was trying to analyze too much myself before getting back on the list. Also I was stuck with weird error of module version mismatch, but I figured and solved that issue. So I do see the logs "frame size mismatch" when I miss the events. Since the logs are too huge, I am sending it in pastebin. Here it is: http://pastebin.com/MzTpYsrk
In Kamailio Logs, I see: DEBUG: evapi [evapi_dispatch.c:361]: evapi_recv_client(): frame size mismatch the ending char (h):[{"event":"REGISTER","tindex":"12603","tlabel":"626010915 ","PhoneNumber":"42000","D146:"event":"REGISTER","tindex":"3639","tlabel":" 1889788252","P] (146)
Whereas the messages in ngrep looks good where the messages are split in two TCP packets. The first packet ends at D and the second packet is the proper continuation which send the rest. But it feels Kamailio skipped the beginning of the message and started parsing at colon again and it thinks the netstring started with D146 which is incorrect. Hence, it never emitted any event for the messages in the first chunk.
Hoping this to be helpful !! Thanks.
- Jayesh
On Mon, Sep 21, 2015 at 4:37 PM Daniel-Constantin Mierla < miconda@gmail.com> wrote:
Hello,
I pushed another commit to add more debug messages to see if the event route is supposed to be executed or not. Can you run the tests again and give again the log messages for missing event route executions?
Cheers, Daniel
On 21/09/15 09:04, Jayesh Nambiar wrote:
Hi Daniel, I have captured debug messages after testing with your latest commit on master. Once again, here's what I'm doing exactly for my test:
- Use SIPp to send OPTIONS towards Kamailio.
- On getting options send event as follows:
evapi_async_relay("{"event":"REGISTER","tindex":"$T(id_index)","tlabel":"$T(id_label)","PhoneNumber":"$avp(phone_number)","DeviceId":"$avp(device_id)","CallId":"$ci"}"); 3) There's a client connected which listens for messages on this socket, parses the netstring, and sends same data back as netstring to Kamailio. 4) On the event_route[evapi:message-received, I do the following: xlog("L_INFO", "GOT [$evapi(msg)] from $evapi(srcaddr):$evapi(srcport)\n"); if($evapi(msg)=~"REGISTER" && $evapi(msg)=~"tindex") { jansson_get_field("$evapi(msg)", "tlabel", "$var(tlabel)"); jansson_get_field("$evapi(msg)", "tindex", "$var(tindex)"); $var(t_index) = $(var(tindex){s.int}); $var(t_label) = $(var(tlabel){s.int}); t_continue('$var(t_index)', '$var(t_label)', 'REGISTER_RESPONSE'); exit; } 5) On route[REGISTER_RESPONSE], I send 200 OK to SIPp
Now here are detailed logs for which neither SIPp didnt get a response nor the GOT [$evapi(msg)] from $evapi(srcaddr):$evapi(srcport)\n"); got logged in the syslog:
First Event: DEBUG: evapi [evapi_dispatch.c:598]: evapi_relay(): relaying event data [{"event":"REGISTER","tindex":"26266","tlabel":"587925078","PhoneNumber":"21956","DeviceId":"abcd1234abcd1234","CallId":" 23021-12910@198.24.63.39"}] (146) DEBUG: evapi [evapi_dispatch.c:623]: evapi_relay(): sending [0x7f132588de68] [146:{"event":"REGISTER","tindex":"26266","tlabel":"587925078","PhoneNumber":"21956","DeviceId":"abcd1234abcd1234","CallId":" 23021-12910@198.24.63.39"},] (151)
DEBUG: evapi [evapi_dispatch.c:488]: evapi_recv_notify(): received [0x7f132588de68] [146:{"event":"REGISTER","tindex":"26266","tlabel":"587925078","PhoneNumber":"21956","DeviceId":"abcd1234abcd1234","CallId":" 23021-12910@198.24.63.39"},] (151) NOTICE: evapi [evapi_dispatch.c:290]: evapi_recv_client(): {0} [ 198.24.63.45:48881] - received [146:{"event":"REGISTER","tindex":"26266","tlabel":"587925078","PhoneNumber":"21956","DeviceId":"abcd1234abcd1234","CallId":" 23021-12910@198.24.63.39"},] (151) (18)
Second Event: DEBUG: evapi [evapi_dispatch.c:598]: evapi_relay(): relaying event data [{"event":"REGISTER","tindex":"32244","tlabel":"1637923412","PhoneNumber":"25597","DeviceId":"abcd1234abcd1234","CallId":" 27182-12910@198.24.63.39"}] (147) DEBUG: evapi [evapi_dispatch.c:623]: evapi_relay(): sending [0x7f1325884410] [147:{"event":"REGISTER","tindex":"32244","tlabel":"1637923412","PhoneNumber":"25597","DeviceId":"abcd1234abcd1234","CallId":" 27182-12910@198.24.63.39"},] (152)
DEBUG: evapi [evapi_dispatch.c:488]: evapi_recv_notify(): received [0x7f1325884410] [147:{"event":"REGISTER","tindex":"32244","tlabel":"1637923412","PhoneNumber":"25597","DeviceId":"abcd1234abcd1234","CallId":" 27182-12910@198.24.63.39"},] (152) NOTICE: evapi [evapi_dispatch.c:290]: evapi_recv_client(): {0} [ 198.24.63.45:48881] - received [147:{"event":"REGISTER","tindex":"32244","tlabel":"1637923412","PhoneNumber":"25597","DeviceId":"abcd1234abcd1234","CallId":" 27182-12910@198.24.63.39"},] (152) (85)
I had three such events which got missed out of some 25000 odd messages sent from SIPp. Other major difference that I see in the debug logs for problematic events are that there is a positive number in the second parentheses of evapi_recv_client() function. For the events that were invoked successfully the value of second parentheses is 0 for evapi_recv_client() function. Hope the debugging helps.
Debug log for a successful event: DEBUG: evapi [evapi_dispatch.c:598]: evapi_relay(): relaying event data [{"event":"REGISTER","tindex":"22127","tlabel":"1896682192","PhoneNumber":"73168","DeviceId":"abcd1234abcd1234","CallId":" 49957-13056@198.24.63.39"}] (147) DEBUG: evapi [evapi_dispatch.c:623]: evapi_relay(): sending [0x7f132568e850] [147:{"event":"REGISTER","tindex":"22127","tlabel":"1896682192","PhoneNumber":"73168","DeviceId":"abcd1234abcd1234","CallId":" 49957-13056@198.24.63.39"},] (152)
DEBUG: evapi [evapi_dispatch.c:488]: evapi_recv_notify(): received [0x7f132568e850] [147:{"event":"REGISTER","tindex":"22127","tlabel":"1896682192","PhoneNumber":"73168","DeviceId":"abcd1234abcd1234","CallId":" 49957-13056@198.24.63.39"},] (152) NOTICE: evapi [evapi_dispatch.c:290]: evapi_recv_client(): {0} [ 198.24.63.45:48881] - received [147:{"event":"REGISTER","tindex":"22127","tlabel":"1896682192","PhoneNumber":"73168","DeviceId":"abcd1234abcd1234","CallId":" 49957-13056@198.24.63.39"},] (152) (0)
Thanks,
- Jayesh
On Sat, Sep 19, 2015 at 3:55 AM Daniel-Constantin Mierla < miconda@gmail.com> wrote:
I pushed a commit to add more debug message while processing received data. You can use debugger module to set a higher debug level for evapi module in order to see what happens.
I checked the netstring packet size and it is invalid (unless email stripped some white chars there) -- for example in:
145:{"event":"REGISTER","tindex":"55567","tlabel":"627458699","PhoneNumber":"20711","DeviceId":"abcd1234abcd1234","CallId": "21225-3848@5.6.7.8" 21225-3848@5.6.7.8 21225-3848@5.6.7.8},
the size is 141, not 145 -- it looks like the size includes the size itself plus the delimiters ':,'. The size is only the effective data, see:
https://en.wikipedia.org/wiki/Netstring
If evapi gets a packet with an invalid size, then it discards the buffer content.
See if the app on the other side of evapi connection builds netstrings with wrong size.
Cheers, Daniel
On 18/09/15 23:19, Jayesh Nambiar wrote:
Here are the tests that I did:
With the patch applied, I see that Kamailio is invoking event individually for each netstring even when they come in different chunks. But I did see instances where when there were complete netstrings in a single chunk; kamailio did not raise an event for them. Here's the illustration:
T 1.2.3.4:48873 -> 5.6.7.8:3927 [AP]
146:{"event":"REGISTER","tindex":"16916","tlabel":"1753048277","PhoneNumber":"20708","DeviceId":"abcd1234abcd1234","CallId":" 21220-3848@5.6.7.8 "},144:{"event":"REGISTER","tindex":"7954","tlabel":"254315075","PhoneNumber":"20709","DeviceId":"abcd1234abcd1234","CallId":" 21223-3848@5.6.7.8 "},145:{"event":"REGISTER","tindex":"64529","tlabel":"599481568","PhoneNumber":"20709","DeviceId":"abcd1234abcd1234","CallId":" 21222-3848@5.6.7.8"},145:{"event":"REGISTER","tindex":"46605","tlabel":"112015324","PhoneNumber":"20710","DeviceId"
T 1.2.3.4:48873 -> 5.6.7.8:3927 [AP] :"abcd1234abcd1234","CallId":"21224-3848@5.6.7.8"},
*The above two chunks contain 4 proper netstrings where the second chunk contains part of the 4th netstring. In this case Evapi properly raised 4 individual events.*
1.2.3.4:48873 -> 5.6.7.8:3927 [AP]
145:{"event":"REGISTER","tindex":"55567","tlabel":"627458699","PhoneNumber":"20711","DeviceId":"abcd1234abcd1234","CallId":" 21225-3848@5.6.7.8 "},143:{"event":"REGISTER","tindex":"28682","tlabel":"9676691","PhoneNumber":"20712","DeviceId":"abcd1234abcd1234","CallId":" 21226-3848@5.6.7.8"},
*This above chunk contains two complete netstrings but Kamailio never raised events for these two netstrings.*
For the events that were not raised I see proper Kamailio Logs which is: evapi_recv_client(): {0} [1.2.3.4:48873] - received [145:{"event":"REGISTER","tindex":"55567","tlabel":"627458699","PhoneNumber":"20711","DeviceId":"abcd1234abcd1234","CallId":" 21225-3848@5.6.7.8 "},143:{"event":"REGISTER","tindex":"28682","tlabel":"9676691","PhoneNumber":"20712","DeviceId":"abcd1234abcd1234","CallId":" 21226-3848@5.6.7.8"},]
But I don't see the logs that I've written in the script after the event was raised, which means Kamailio did not invoke events for these two netstrings.
So in my tests, out of 27232 messages sent, there were approximately 27 messages for which the events were not raised by Evapi. The rate of messages started at 500cps and I stopped after I saw missed events at around 1200cps.
Thanks, and do let me know for any further tests or information required about the same.
- Jayesh
On Fri, Sep 18, 2015 at 2:55 PM, Jayesh Nambiar jayesh1017@gmail.com wrote:
Will have this tested by tomorrow and will get back to you. Thanks.
- Jayesh
On Fri, Sep 18, 2015 at 1:41 PM, Daniel-Constantin Mierla < miconda@gmail.com> wrote:
> Hello, > > I just pushed a patch to master branch that should cope with partial > data received on tcp connection. No time to test at all, therefore any > feedback will be appreciated. > > Cheers, > Daniel > > > On 15/09/15 14:52, Daniel-Constantin Mierla wrote: > > Hello, > > I will look if there are options in libev to buffer data or try to > implement a buffering mechanism locally for such cases. > > Cheers, > Daniel > > On 14/09/15 23:00, Jayesh Nambiar wrote: > > Hello Daniel, > After further testing with evapi module, I figured that when > Netstrings are used, an event route is invoked individually for each > message even if if multiple netstring messages are received in a single TCP > packet. But this doesn't work effectively when a single proper message is > split-up in two packets. For Example, if a message arrives as: > 12:Hello World!, 12:Hello World!, 12:Hello World! in a single > packet, kamailio properly invokes the event route "evapi:message-received" > thrice for every individual proper netstring message. > But if the first packet contains: > 12:Hello World!, 12:Hello > And Second Packet contains: > World!, 12:Hello World! > the event route is invoked only once !! > > The above pattern is very much possible while sending and receiving > packets over TCP Socket. Our tests for receiving an approximately 150 byte > message over evapi socket at the rate of roughly 1000cps causes a lot of > real events to be missed because of the above problem. You can never be > sure when TCP will split messages in different chunks. > This definitely looks like a bug which makes it not very reliable at > large scale deployments. Would really appreciate your inputs on this. > Thanks; > > - Jayesh > > > > On Thu, Sep 10, 2015 at 4:01 PM, Jayesh Nambiar < > jayesh1017@gmail.com> wrote: > >> Hi Daniel, >> Thanks for the quick response. So if I do not use Netstrings, does >> Kamailio allow me to create a custom logic in the script. For eg. if I >> decide to use newline as a delimiter, can I keep buffering the message >> until I encounter the delimiter from the event route and then execute >> whatever I have to within the script?? >> >> Thanks, >> >> - Jayesh >> >> On Thu, Sep 10, 2015 at 1:29 PM, Daniel-Constantin Mierla < >> miconda@gmail.com> wrote: >> >>> Hello, >>> >>> tcb is stream protocol and several messages can be queued on the >>> pipe at the same time. That is the reason for netstring format, to be able >>> to easily detect the boundaries of each message. If netstring format is >>> enabled and kamailio receives several messages at once, it splits them and >>> for each is executing the event route. >>> >>> If netstring format is not used, the kamailio is executing the >>> event route with the entire content that was read at once from the tcp >>> connection. >>> >>> Cheers, >>> Daniel >>> >>> >>> On 09/09/15 22:01, Jayesh Nambiar wrote: >>> >>> Hello, >>> I'm exploring the evapi module for my kamailio to interface with >>> an external node.js app for third party stuff like AAA, billing engine >>> tasks, notifications and so on. I followed and took some ideas from the >>> rtjson and evapi tutorial found here( >>> http://kb.asipto.com/kamailio:k43-async-sip-routing-nodejs) to >>> build the node.js app consuming events. >>> When I stress tested the scenario using SIPp and tried sending a >>> lot of events at 300-350cps from Kamailio, I noticed that at times the >>> client is receiving 2-3 events in a single message together although I do >>> event_sync_relay once per SIP message received and have netstrings enabled. >>> I believe this is a typical behavior of TCP and needs to be handled by the >>> client using some kind of Netstring handler. Please correct me if I'm wrong. >>> And hence I'd like to know what particularly needs to be taken >>> care of while writing a client that is listening for events on raw tcp >>> socket and how does kamailio handle this situation while receiving messages >>> over TCP socket?? Does kamailio recognize the end of netstring properly on >>> evapi:message-received and give exactly one message to take care of on >>> every "message-received" event or should that be handled in the script >>> somewhere !! >>> I also referred cgrates client over evapi example which is written >>> in GO, but I couldnt find them handling TCP streams clearly either. >>> I'd really appreciate some expert suggestion here to make an >>> informed decision on using the evapi module for a large scale solution. >>> >>> Thanks, >>> >>> - Jayesh >>> >>> >>> >>> >>> _______________________________________________ >>> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing listsr-users@lists.sip-router.orghttp://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users >>> >>> >>> -- >>> Daniel-Constantin Mierlahttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda >>> Book: SIP Routing With Kamailio - http://www.asipto.com >>> Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat >>> >>> >>> _______________________________________________ >>> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing >>> list >>> sr-users@lists.sip-router.org >>> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users >>> >>> >> > > -- > Daniel-Constantin Mierlahttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda > Book: SIP Routing With Kamailio - http://www.asipto.com > Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat > > > -- > Daniel-Constantin Mierlahttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda > Book: SIP Routing With Kamailio - http://www.asipto.com > Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat > >
-- Daniel-Constantin Mierlahttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
-- Daniel-Constantin Mierlahttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
-- Daniel-Constantin Mierlahttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
Hello,
I will look over it during next days.
One more thing, are the packages sent by you printed by evapi module, even not they are not processed (but discarded)?
Cheers, Daniel
On 25/09/15 13:55, Jayesh Nambiar wrote:
Hi Daniel, I think this is not solved completely after you applied the patch and my testing shows that there are some problems left out to solve. Please do let me know if you are looking at more inputs which I might be able to test and provide. Thanks,
- Jayesh
On Wed, Sep 23, 2015 at 8:24 PM Jayesh Nambiar <jayesh1017@gmail.com mailto:jayesh1017@gmail.com> wrote:
Hello Daniel, Just checking if you got what you were looking for. Do let me know if you need more info on the same. Thanks, - Jayesh On Tue, Sep 22, 2015 at 8:23 PM Jayesh Nambiar <jayesh1017@gmail.com <mailto:jayesh1017@gmail.com>> wrote: Hi, Here's the NGREP trace where {"event":"REGISTER","tindex":"3639","tlabel":"1889788252" ... is present on the first line itself. 198.24.63.45:48886 <http://198.24.63.45:48886> -> 198.24.63.39:3927 <http://198.24.63.39:3927> [AP] 146:{"event":"REGISTER","tindex":"3639","tlabel":"1889788252","PhoneNumber":"42008","DeviceId":"abcd1234abcd1234","CallId":"11542-19872@198.24.63.39 <mailto:11542-19872@198.24.63.39>"},147:{"event":"REGISTER","tindex":"12600","tlabel":"1916422130","PhoneNumber":"42009","DeviceId":"abcd1234abcd1234","CallId":"11543-19872@198.24.63.39 <mailto:11543-19872@198.24.63.39>"},147:{"event":"REGISTER","tindex":"51251","tlabel":"1549871937","PhoneNumber":"42010","DeviceId":"abcd1234abcd 1234","CallId":"11544-19872@198.24.63.39 <mailto:11544-19872@198.24.63.39>"},147:{"event":"REGISTER","tindex":"60213","tlabel":"2100336684","PhoneNumber":"42011","DeviceId":"abcd1234abcd1234","CallId":"11545-19872@198.24.63.39 <mailto:11545-19872@198.24.63.39>"},146:{"event ":"REGISTER","tindex":"33328","tlabel":"622251552","PhoneNumber":"42012","DeviceId":"abcd1234abcd1234","CallId":"11546-19872@198.24.63.39 <mailto:11546-19872@198.24.63.39>"},147:{"event":"REGISTER","tindex":"42289","tlabel":"1360864805","PhoneNumber":"42013","DeviceId":"abcd1234abcd1234","CallId":"11547-19872@198.24.63.39 <mailto:11547-19872@198.24.63.39>"},146:{"event":"REGISTER","tindex":"48204","tlabel":"315887672","PhoneNumber":"42014","DeviceId":"abcd1234abcd1234","CallId":"11548-19872@198.24.63.39 <mailto:11548-19872@198.24.63.39>"},145:{"event":"REGISTER","tindex":"36682","tlabel":"18660440","PhoneNumber":"42015","DeviceId":"abcd1234abcd1234","CallId":"11549-19872@198.24.63.39 <mailto:11549-19872@198.24.63.39>"}, And yes, this packet is right after the second packet in the pastebin trace. Thanks, - Jayesh On Tue, Sep 22, 2015 at 5:03 PM Daniel-Constantin Mierla <miconda@gmail.com <mailto:miconda@gmail.com>> wrote: Hello, I haven't seen in the ngrep when the data containing next line is received: {"event":"REGISTER","tindex":"3639","tlabel":"1889788252" ... Do you still have it? Is it after: {"event":"REGISTER","tindex":"12603","tlabel":"626010915", ... Cheers, Daniel On 22/09/15 09:09, Jayesh Nambiar wrote:Hi, Took time to get back with the test results as I was trying to analyze too much myself before getting back on the list. Also I was stuck with weird error of module version mismatch, but I figured and solved that issue. So I do see the logs "frame size mismatch" when I miss the events. Since the logs are too huge, I am sending it in pastebin. Here it is: http://pastebin.com/MzTpYsrk In Kamailio Logs, I see: DEBUG: evapi [evapi_dispatch.c:361]: evapi_recv_client(): frame size mismatch the ending char (h):[{"event":"REGISTER","tindex":"12603","tlabel":"626010915","PhoneNumber":"42000","D146:"event":"REGISTER","tindex":"3639","tlabel":"1889788252","P] (146) Whereas the messages in ngrep looks good where the messages are split in two TCP packets. The first packet ends at D and the second packet is the proper continuation which send the rest. But it feels Kamailio skipped the beginning of the message and started parsing at colon again and it thinks the netstring started with D146 which is incorrect. Hence, it never emitted any event for the messages in the first chunk. Hoping this to be helpful !! Thanks. - Jayesh On Mon, Sep 21, 2015 at 4:37 PM Daniel-Constantin Mierla <miconda@gmail.com <mailto:miconda@gmail.com>> wrote: Hello, I pushed another commit to add more debug messages to see if the event route is supposed to be executed or not. Can you run the tests again and give again the log messages for missing event route executions? Cheers, Daniel On 21/09/15 09:04, Jayesh Nambiar wrote:Hi Daniel, I have captured debug messages after testing with your latest commit on master. Once again, here's what I'm doing exactly for my test: 1) Use SIPp to send OPTIONS towards Kamailio. 2) On getting options send event as follows: evapi_async_relay("{\"event\":\"REGISTER\",\"tindex\":\"$T(id_index)\",\"tlabel\":\"$T(id_label)\",\"PhoneNumber\":\"$avp(phone_number)\",\"DeviceId\":\"$avp(device_id)\",\"CallId\":\"$ci\"}"); 3) There's a client connected which listens for messages on this socket, parses the netstring, and sends same data back as netstring to Kamailio. 4) On the event_route[evapi:message-received, I do the following: xlog("L_INFO", "GOT [$evapi(msg)] from $evapi(srcaddr):$evapi(srcport)\n"); if($evapi(msg)=~"REGISTER" && $evapi(msg)=~"tindex") { jansson_get_field("$evapi(msg)", "tlabel", "$var(tlabel)"); jansson_get_field("$evapi(msg)", "tindex", "$var(tindex)"); $var(t_index) = $(var(tindex){s.int <http://s.int>}); $var(t_label) = $(var(tlabel){s.int <http://s.int>}); t_continue('$var(t_index)', '$var(t_label)', 'REGISTER_RESPONSE'); exit; } 5) On route[REGISTER_RESPONSE], I send 200 OK to SIPp Now here are detailed logs for which neither SIPp didnt get a response nor the GOT [$evapi(msg)] from $evapi(srcaddr):$evapi(srcport)\n"); got logged in the syslog: First Event: DEBUG: evapi [evapi_dispatch.c:598]: evapi_relay(): relaying event data [{"event":"REGISTER","tindex":"26266","tlabel":"587925078","PhoneNumber":"21956","DeviceId":"abcd1234abcd1234","CallId":"23021-12910@198.24.63.39 <mailto:23021-12910@198.24.63.39>"}] (146) DEBUG: evapi [evapi_dispatch.c:623]: evapi_relay(): sending [0x7f132588de68] [146:{"event":"REGISTER","tindex":"26266","tlabel":"587925078","PhoneNumber":"21956","DeviceId":"abcd1234abcd1234","CallId":"23021-12910@198.24.63.39 <mailto:23021-12910@198.24.63.39>"},] (151) DEBUG: evapi [evapi_dispatch.c:488]: evapi_recv_notify(): received [0x7f132588de68] [146:{"event":"REGISTER","tindex":"26266","tlabel":"587925078","PhoneNumber":"21956","DeviceId":"abcd1234abcd1234","CallId":"23021-12910@198.24.63.39 <mailto:23021-12910@198.24.63.39>"},] (151) NOTICE: evapi [evapi_dispatch.c:290]: evapi_recv_client(): {0} [198.24.63.45:48881 <http://198.24.63.45:48881>] - received [146:{"event":"REGISTER","tindex":"26266","tlabel":"587925078","PhoneNumber":"21956","DeviceId":"abcd1234abcd1234","CallId":"23021-12910@198.24.63.39 <mailto:23021-12910@198.24.63.39>"},] (151) (18) Second Event: DEBUG: evapi [evapi_dispatch.c:598]: evapi_relay(): relaying event data [{"event":"REGISTER","tindex":"32244","tlabel":"1637923412","PhoneNumber":"25597","DeviceId":"abcd1234abcd1234","CallId":"27182-12910@198.24.63.39 <mailto:27182-12910@198.24.63.39>"}] (147) DEBUG: evapi [evapi_dispatch.c:623]: evapi_relay(): sending [0x7f1325884410] [147:{"event":"REGISTER","tindex":"32244","tlabel":"1637923412","PhoneNumber":"25597","DeviceId":"abcd1234abcd1234","CallId":"27182-12910@198.24.63.39 <mailto:27182-12910@198.24.63.39>"},] (152) DEBUG: evapi [evapi_dispatch.c:488]: evapi_recv_notify(): received [0x7f1325884410] [147:{"event":"REGISTER","tindex":"32244","tlabel":"1637923412","PhoneNumber":"25597","DeviceId":"abcd1234abcd1234","CallId":"27182-12910@198.24.63.39 <mailto:27182-12910@198.24.63.39>"},] (152) NOTICE: evapi [evapi_dispatch.c:290]: evapi_recv_client(): {0} [198.24.63.45:48881 <http://198.24.63.45:48881>] - received [147:{"event":"REGISTER","tindex":"32244","tlabel":"1637923412","PhoneNumber":"25597","DeviceId":"abcd1234abcd1234","CallId":"27182-12910@198.24.63.39 <mailto:27182-12910@198.24.63.39>"},] (152) (85) I had three such events which got missed out of some 25000 odd messages sent from SIPp. Other major difference that I see in the debug logs for problematic events are that there is a positive number in the second parentheses of evapi_recv_client() function. For the events that were invoked successfully the value of second parentheses is 0 for evapi_recv_client() function. Hope the debugging helps. Debug log for a successful event: DEBUG: evapi [evapi_dispatch.c:598]: evapi_relay(): relaying event data [{"event":"REGISTER","tindex":"22127","tlabel":"1896682192","PhoneNumber":"73168","DeviceId":"abcd1234abcd1234","CallId":"49957-13056@198.24.63.39 <mailto:49957-13056@198.24.63.39>"}] (147) DEBUG: evapi [evapi_dispatch.c:623]: evapi_relay(): sending [0x7f132568e850] [147:{"event":"REGISTER","tindex":"22127","tlabel":"1896682192","PhoneNumber":"73168","DeviceId":"abcd1234abcd1234","CallId":"49957-13056@198.24.63.39 <mailto:49957-13056@198.24.63.39>"},] (152) DEBUG: evapi [evapi_dispatch.c:488]: evapi_recv_notify(): received [0x7f132568e850] [147:{"event":"REGISTER","tindex":"22127","tlabel":"1896682192","PhoneNumber":"73168","DeviceId":"abcd1234abcd1234","CallId":"49957-13056@198.24.63.39 <mailto:49957-13056@198.24.63.39>"},] (152) NOTICE: evapi [evapi_dispatch.c:290]: evapi_recv_client(): {0} [198.24.63.45:48881 <http://198.24.63.45:48881>] - received [147:{"event":"REGISTER","tindex":"22127","tlabel":"1896682192","PhoneNumber":"73168","DeviceId":"abcd1234abcd1234","CallId":"49957-13056@198.24.63.39 <mailto:49957-13056@198.24.63.39>"},] (152) (0) Thanks, - Jayesh On Sat, Sep 19, 2015 at 3:55 AM Daniel-Constantin Mierla <miconda@gmail.com <mailto:miconda@gmail.com>> wrote: I pushed a commit to add more debug message while processing received data. You can use debugger module to set a higher debug level for evapi module in order to see what happens. I checked the netstring packet size and it is invalid (unless email stripped some white chars there) -- for example in: 145:{"event":"REGISTER","tindex":"55567","tlabel":"627458699","PhoneNumber":"20711","DeviceId":"abcd1234abcd1234","CallId":"21225-3848@5.6.7.8" <mailto:21225-3848@5.6.7.8>}, the size is 141, not 145 -- it looks like the size includes the size itself plus the delimiters ':,'. The size is only the effective data, see: https://en.wikipedia.org/wiki/Netstring If evapi gets a packet with an invalid size, then it discards the buffer content. See if the app on the other side of evapi connection builds netstrings with wrong size. Cheers, Daniel On 18/09/15 23:19, Jayesh Nambiar wrote:Here are the tests that I did: With the patch applied, I see that Kamailio is invoking event individually for each netstring even when they come in different chunks. But I did see instances where when there were complete netstrings in a single chunk; kamailio did not raise an event for them. Here's the illustration: T 1.2.3.4:48873 <http://1.2.3.4:48873> -> 5.6.7.8:3927 <http://5.6.7.8:3927> [AP] 146:{"event":"REGISTER","tindex":"16916","tlabel":"1753048277","PhoneNumber":"20708","DeviceId":"abcd1234abcd1234","CallId":"21220-3848@5.6.7.8 <mailto:21220-3848@5.6.7.8>"},144:{"event":"REGISTER","tindex":"7954","tlabel":"254315075","PhoneNumber":"20709","DeviceId":"abcd1234abcd1234","CallId":"21223-3848@5.6.7.8 <mailto:21223-3848@5.6.7.8>"},145:{"event":"REGISTER","tindex":"64529","tlabel":"599481568","PhoneNumber":"20709","DeviceId":"abcd1234abcd1234","CallId":"21222-3848@5.6.7.8 <mailto:21222-3848@5.6.7.8>"},145:{"event":"REGISTER","tindex":"46605","tlabel":"112015324","PhoneNumber":"20710","DeviceId" T 1.2.3.4:48873 <http://1.2.3.4:48873> -> 5.6.7.8:3927 <http://5.6.7.8:3927> [AP] :"abcd1234abcd1234","CallId":"21224-3848@5.6.7.8 <mailto:21224-3848@5.6.7.8>"}, *The above two chunks contain 4 proper netstrings where the second chunk contains part of the 4th netstring. In this case Evapi properly raised 4 individual events.* 1.2.3.4:48873 <http://1.2.3.4:48873> -> 5.6.7.8:3927 <http://5.6.7.8:3927> [AP] 145:{"event":"REGISTER","tindex":"55567","tlabel":"627458699","PhoneNumber":"20711","DeviceId":"abcd1234abcd1234","CallId":"21225-3848@5.6.7.8 <mailto:21225-3848@5.6.7.8>"},143:{"event":"REGISTER","tindex":"28682","tlabel":"9676691","PhoneNumber":"20712","DeviceId":"abcd1234abcd1234","CallId":"21226-3848@5.6.7.8 <mailto:21226-3848@5.6.7.8>"}, *This above chunk contains two complete netstrings but Kamailio never raised events for these two netstrings.* * * For the events that were not raised I see proper Kamailio Logs which is: evapi_recv_client(): {0} [1.2.3.4:48873 <http://1.2.3.4:48873>] - received [145:{"event":"REGISTER","tindex":"55567","tlabel":"627458699","PhoneNumber":"20711","DeviceId":"abcd1234abcd1234","CallId":"21225-3848@5.6.7.8 <mailto:21225-3848@5.6.7.8>"},143:{"event":"REGISTER","tindex":"28682","tlabel":"9676691","PhoneNumber":"20712","DeviceId":"abcd1234abcd1234","CallId":"21226-3848@5.6.7.8 <mailto:21226-3848@5.6.7.8>"},] * * But I don't see the logs that I've written in the script after the event was raised, which means Kamailio did not invoke events for these two netstrings. So in my tests, out of 27232 messages sent, there were approximately 27 messages for which the events were not raised by Evapi. The rate of messages started at 500cps and I stopped after I saw missed events at around 1200cps. Thanks, and do let me know for any further tests or information required about the same. - Jayesh * * On Fri, Sep 18, 2015 at 2:55 PM, Jayesh Nambiar <jayesh1017@gmail.com <mailto:jayesh1017@gmail.com>> wrote: Will have this tested by tomorrow and will get back to you. Thanks. - Jayesh On Fri, Sep 18, 2015 at 1:41 PM, Daniel-Constantin Mierla <miconda@gmail.com <mailto:miconda@gmail.com>> wrote: Hello, I just pushed a patch to master branch that should cope with partial data received on tcp connection. No time to test at all, therefore any feedback will be appreciated. Cheers, Daniel On 15/09/15 14:52, Daniel-Constantin Mierla wrote:Hello, I will look if there are options in libev to buffer data or try to implement a buffering mechanism locally for such cases. Cheers, Daniel On 14/09/15 23:00, Jayesh Nambiar wrote:Hello Daniel, After further testing with evapi module, I figured that when Netstrings are used, an event route is invoked individually for each message even if if multiple netstring messages are received in a single TCP packet. But this doesn't work effectively when a single proper message is split-up in two packets. For Example, if a message arrives as: 12:Hello World!, 12:Hello World!, 12:Hello World! in a single packet, kamailio properly invokes the event route "evapi:message-received" thrice for every individual proper netstring message. But if the first packet contains: 12:Hello World!, 12:Hello And Second Packet contains: World!, 12:Hello World! the event route is invoked only once !! The above pattern is very much possible while sending and receiving packets over TCP Socket. Our tests for receiving an approximately 150 byte message over evapi socket at the rate of roughly 1000cps causes a lot of real events to be missed because of the above problem. You can never be sure when TCP will split messages in different chunks. This definitely looks like a bug which makes it not very reliable at large scale deployments. Would really appreciate your inputs on this. Thanks; - Jayesh On Thu, Sep 10, 2015 at 4:01 PM, Jayesh Nambiar <jayesh1017@gmail.com <mailto:jayesh1017@gmail.com>> wrote: Hi Daniel, Thanks for the quick response. So if I do not use Netstrings, does Kamailio allow me to create a custom logic in the script. For eg. if I decide to use newline as a delimiter, can I keep buffering the message until I encounter the delimiter from the event route and then execute whatever I have to within the script?? Thanks, - Jayesh On Thu, Sep 10, 2015 at 1:29 PM, Daniel-Constantin Mierla <miconda@gmail.com <mailto:miconda@gmail.com>> wrote: Hello, tcb is stream protocol and several messages can be queued on the pipe at the same time. That is the reason for netstring format, to be able to easily detect the boundaries of each message. If netstring format is enabled and kamailio receives several messages at once, it splits them and for each is executing the event route. If netstring format is not used, the kamailio is executing the event route with the entire content that was read at once from the tcp connection. Cheers, Daniel On 09/09/15 22:01, Jayesh Nambiar wrote:> Hello, > I'm exploring the evapi > module for my kamailio to > interface with an external > node.js app for third party > stuff like AAA, billing > engine tasks, notifications > and so on. I followed and > took some ideas from the > rtjson and evapi tutorial > found > here(http://kb.asipto.com/kamailio:k43-async-sip-routing-nodejs) > to build the node.js app > consuming events. > When I stress tested the > scenario using SIPp and > tried sending a lot of > events at 300-350cps from > Kamailio, I noticed that at > times the client is > receiving 2-3 events in a > single message together > although I do > event_sync_relay once per > SIP message received and > have netstrings enabled. I > believe this is a typical > behavior of TCP and needs to > be handled by the client > using some kind of Netstring > handler. Please correct me > if I'm wrong. > And hence I'd like to know > what particularly needs to > be taken care of while > writing a client that is > listening for events on raw > tcp socket and how does > kamailio handle this > situation while receiving > messages over TCP socket?? > Does kamailio recognize the > end of netstring properly on > evapi:message-received and > give exactly one message to > take care of on every > "message-received" event or > should that be handled in > the script somewhere !! > I also referred cgrates > client over evapi example > which is written in GO, but > I couldnt find them handling > TCP streams clearly either. > I'd really appreciate some > expert suggestion here to > make an informed decision on > using the evapi module for a > large scale solution. > > Thanks, > > - Jayesh > > > > > _______________________________________________ > SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list > sr-users@lists.sip-router.org mailto:sr-users@lists.sip-router.org > http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
-- Daniel-Constantin Mierla http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat _______________________________________________ SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org <mailto:sr-users@lists.sip-router.org> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users-- Daniel-Constantin Mierla http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat-- Daniel-Constantin Mierla http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat-- Daniel-Constantin Mierla http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat-- Daniel-Constantin Mierla http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat-- Daniel-Constantin Mierla http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
Thank you daniel. Yes, they all get printed by the evapi module even when they get discarded. If you once again check the paste-bin, the logs pasted as Kamailio Logs are all discarded packets but show up as received by evapi module. I believe they got discarded because a packet that came before them was partial and evapi discarded the remaining part of the message in the new packet and encountered error. - Jayesh
On Mon, Sep 28, 2015 at 11:13 AM Daniel-Constantin Mierla miconda@gmail.com wrote:
Hello,
I will look over it during next days.
One more thing, are the packages sent by you printed by evapi module, even not they are not processed (but discarded)?
Cheers, Daniel
On 25/09/15 13:55, Jayesh Nambiar wrote:
Hi Daniel, I think this is not solved completely after you applied the patch and my testing shows that there are some problems left out to solve. Please do let me know if you are looking at more inputs which I might be able to test and provide. Thanks,
- Jayesh
On Wed, Sep 23, 2015 at 8:24 PM Jayesh Nambiar jayesh1017@gmail.com wrote:
Hello Daniel, Just checking if you got what you were looking for. Do let me know if you need more info on the same.
Thanks,
- Jayesh
On Tue, Sep 22, 2015 at 8:23 PM Jayesh Nambiar jayesh1017@gmail.com wrote:
Hi, Here's the NGREP trace where {"event":"REGISTER","tindex":"3639","tlabel":"1889788252" ... is present on the first line itself.
198.24.63.45:48886 -> 198.24.63.39:3927 [AP]
146:{"event":"REGISTER","tindex":"3639","tlabel":"1889788252","PhoneNumber":"42008","DeviceId":"abcd1234abcd1234","CallId":" 11542-19872@198.24.63.3911542-19872@198.24.63.39 "},147:{"event":"REGISTER","tindex":"12600","tlabel":"1916422130","PhoneNumber":"42009","DeviceId":"abcd1234abcd1234","CallId":" 11543-19872@198.24.63.3911543-19872@198.24.63.39 "},147:{"event":"REGISTER","tindex":"51251","tlabel":"1549871937","PhoneNumber":"42010","DeviceId":"abcd1234abcd 1234","CallId":"11544-19872@198.24.63.39 "},147:{"event":"REGISTER","tindex":"60213","tlabel":"2100336684","PhoneNumber":"42011","DeviceId":"abcd1234abcd1234","CallId":" 11545-19872@198.24.63.3911545-19872@198.24.63.39"},146:{"event
":"REGISTER","tindex":"33328","tlabel":"622251552","PhoneNumber":"42012","DeviceId":"abcd1234abcd1234","CallId":" 11546-19872@198.24.63.3911546-19872@198.24.63.39 "},147:{"event":"REGISTER","tindex":"42289","tlabel":"1360864805","PhoneNumber":"42013","DeviceId":"abcd1234abcd1234","CallId":" 11547-19872@198.24.63.3911547-19872@198.24.63.39 "},146:{"event":"REGISTER","tindex":"48204","tlabel":"315887672","PhoneNumber":"42014","DeviceId":"abcd1234abcd1234","CallId":" 11548-19872@198.24.63.3911548-19872@198.24.63.39 "},145:{"event":"REGISTER","tindex":"36682","tlabel":"18660440","PhoneNumber":"42015","DeviceId":"abcd1234abcd1234","CallId":" 11549-19872@198.24.63.3911549-19872@198.24.63.39"},
And yes, this packet is right after the second packet in the pastebin trace.
Thanks,
- Jayesh
On Tue, Sep 22, 2015 at 5:03 PM Daniel-Constantin Mierla < miconda@gmail.com> wrote:
Hello,
I haven't seen in the ngrep when the data containing next line is received:
{"event":"REGISTER","tindex":"3639","tlabel":"1889788252" ...
Do you still have it? Is it after:
{"event":"REGISTER","tindex":"12603","tlabel":"626010915", ...
Cheers, Daniel
On 22/09/15 09:09, Jayesh Nambiar wrote:
Hi, Took time to get back with the test results as I was trying to analyze too much myself before getting back on the list. Also I was stuck with weird error of module version mismatch, but I figured and solved that issue. So I do see the logs "frame size mismatch" when I miss the events. Since the logs are too huge, I am sending it in pastebin. Here it is: http://pastebin.com/MzTpYsrk
In Kamailio Logs, I see: DEBUG: evapi [evapi_dispatch.c:361]: evapi_recv_client(): frame size mismatch the ending char (h):[{"event":"REGISTER","tindex":"12603","tlabel":"626010915 ","PhoneNumber":"42000","D146:"event":"REGISTER","tindex":"3639","tlabel":" 1889788252","P] (146)
Whereas the messages in ngrep looks good where the messages are split in two TCP packets. The first packet ends at D and the second packet is the proper continuation which send the rest. But it feels Kamailio skipped the beginning of the message and started parsing at colon again and it thinks the netstring started with D146 which is incorrect. Hence, it never emitted any event for the messages in the first chunk.
Hoping this to be helpful !! Thanks.
- Jayesh
On Mon, Sep 21, 2015 at 4:37 PM Daniel-Constantin Mierla < miconda@gmail.commiconda@gmail.com> wrote:
Hello,
I pushed another commit to add more debug messages to see if the event route is supposed to be executed or not. Can you run the tests again and give again the log messages for missing event route executions?
Cheers, Daniel
On 21/09/15 09:04, Jayesh Nambiar wrote:
Hi Daniel, I have captured debug messages after testing with your latest commit on master. Once again, here's what I'm doing exactly for my test:
- Use SIPp to send OPTIONS towards Kamailio.
- On getting options send event as follows:
evapi_async_relay("{"event":"REGISTER","tindex":"$T(id_index)","tlabel":"$T(id_label)","PhoneNumber":"$avp(phone_number)","DeviceId":"$avp(device_id)","CallId":"$ci"}"); 3) There's a client connected which listens for messages on this socket, parses the netstring, and sends same data back as netstring to Kamailio. 4) On the event_route[evapi:message-received, I do the following: xlog("L_INFO", "GOT [$evapi(msg)] from $evapi(srcaddr):$evapi(srcport)\n"); if($evapi(msg)=~"REGISTER" && $evapi(msg)=~"tindex") { jansson_get_field("$evapi(msg)", "tlabel", "$var(tlabel)"); jansson_get_field("$evapi(msg)", "tindex", "$var(tindex)"); $var(t_index) = $(var(tindex){s.int}); $var(t_label) = $(var(tlabel){s.int}); t_continue('$var(t_index)', '$var(t_label)', 'REGISTER_RESPONSE'); exit; } 5) On route[REGISTER_RESPONSE], I send 200 OK to SIPp
Now here are detailed logs for which neither SIPp didnt get a response nor the GOT [$evapi(msg)] from $evapi(srcaddr):$evapi(srcport)\n"); got logged in the syslog:
First Event: DEBUG: evapi [evapi_dispatch.c:598]: evapi_relay(): relaying event data [{"event":"REGISTER","tindex":"26266","tlabel":"587925078","PhoneNumber":"21956","DeviceId":"abcd1234abcd1234","CallId":" 23021-12910@198.24.63.3923021-12910@198.24.63.39"}] (146) DEBUG: evapi [evapi_dispatch.c:623]: evapi_relay(): sending [0x7f132588de68] [146:{"event":"REGISTER","tindex":"26266","tlabel":"587925078","PhoneNumber":"21956","DeviceId":"abcd1234abcd1234","CallId":" 23021-12910@198.24.63.3923021-12910@198.24.63.39"},] (151)
DEBUG: evapi [evapi_dispatch.c:488]: evapi_recv_notify(): received [0x7f132588de68] [146:{"event":"REGISTER","tindex":"26266","tlabel":"587925078","PhoneNumber":"21956","DeviceId":"abcd1234abcd1234","CallId":" 23021-12910@198.24.63.3923021-12910@198.24.63.39"},] (151) NOTICE: evapi [evapi_dispatch.c:290]: evapi_recv_client(): {0} [ 198.24.63.45:48881] - received [146:{"event":"REGISTER","tindex":"26266","tlabel":"587925078","PhoneNumber":"21956","DeviceId":"abcd1234abcd1234","CallId":" 23021-12910@198.24.63.3923021-12910@198.24.63.39"},] (151) (18)
Second Event: DEBUG: evapi [evapi_dispatch.c:598]: evapi_relay(): relaying event data [{"event":"REGISTER","tindex":"32244","tlabel":"1637923412","PhoneNumber":"25597","DeviceId":"abcd1234abcd1234","CallId":" 27182-12910@198.24.63.3927182-12910@198.24.63.39"}] (147) DEBUG: evapi [evapi_dispatch.c:623]: evapi_relay(): sending [0x7f1325884410] [147:{"event":"REGISTER","tindex":"32244","tlabel":"1637923412","PhoneNumber":"25597","DeviceId":"abcd1234abcd1234","CallId":" 27182-12910@198.24.63.3927182-12910@198.24.63.39"},] (152)
DEBUG: evapi [evapi_dispatch.c:488]: evapi_recv_notify(): received [0x7f1325884410] [147:{"event":"REGISTER","tindex":"32244","tlabel":"1637923412","PhoneNumber":"25597","DeviceId":"abcd1234abcd1234","CallId":" 27182-12910@198.24.63.3927182-12910@198.24.63.39"},] (152) NOTICE: evapi [evapi_dispatch.c:290]: evapi_recv_client(): {0} [ 198.24.63.45:48881] - received [147:{"event":"REGISTER","tindex":"32244","tlabel":"1637923412","PhoneNumber":"25597","DeviceId":"abcd1234abcd1234","CallId":" 27182-12910@198.24.63.3927182-12910@198.24.63.39"},] (152) (85)
I had three such events which got missed out of some 25000 odd messages sent from SIPp. Other major difference that I see in the debug logs for problematic events are that there is a positive number in the second parentheses of evapi_recv_client() function. For the events that were invoked successfully the value of second parentheses is 0 for evapi_recv_client() function. Hope the debugging helps.
Debug log for a successful event: DEBUG: evapi [evapi_dispatch.c:598]: evapi_relay(): relaying event data [{"event":"REGISTER","tindex":"22127","tlabel":"1896682192","PhoneNumber":"73168","DeviceId":"abcd1234abcd1234","CallId":" 49957-13056@198.24.63.3949957-13056@198.24.63.39"}] (147) DEBUG: evapi [evapi_dispatch.c:623]: evapi_relay(): sending [0x7f132568e850] [147:{"event":"REGISTER","tindex":"22127","tlabel":"1896682192","PhoneNumber":"73168","DeviceId":"abcd1234abcd1234","CallId":" 49957-13056@198.24.63.3949957-13056@198.24.63.39"},] (152)
DEBUG: evapi [evapi_dispatch.c:488]: evapi_recv_notify(): received [0x7f132568e850] [147:{"event":"REGISTER","tindex":"22127","tlabel":"1896682192","PhoneNumber":"73168","DeviceId":"abcd1234abcd1234","CallId":" 49957-13056@198.24.63.3949957-13056@198.24.63.39"},] (152) NOTICE: evapi [evapi_dispatch.c:290]: evapi_recv_client(): {0} [ 198.24.63.45:48881] - received [147:{"event":"REGISTER","tindex":"22127","tlabel":"1896682192","PhoneNumber":"73168","DeviceId":"abcd1234abcd1234","CallId":" 49957-13056@198.24.63.3949957-13056@198.24.63.39"},] (152) (0)
Thanks,
- Jayesh
On Sat, Sep 19, 2015 at 3:55 AM Daniel-Constantin Mierla < miconda@gmail.commiconda@gmail.com> wrote:
I pushed a commit to add more debug message while processing received data. You can use debugger module to set a higher debug level for evapi module in order to see what happens.
I checked the netstring packet size and it is invalid (unless email stripped some white chars there) -- for example in:
145:{"event":"REGISTER","tindex":"55567","tlabel":"627458699","PhoneNumber":"20711","DeviceId":"abcd1234abcd1234","CallId": 21225-3848@5.6.7.8"21225-3848@5.6.7.8" 21225-3848@5.6.7.8},
the size is 141, not 145 -- it looks like the size includes the size itself plus the delimiters ':,'. The size is only the effective data, see:
https://en.wikipedia.org/wiki/Netstring
If evapi gets a packet with an invalid size, then it discards the buffer content.
See if the app on the other side of evapi connection builds netstrings with wrong size.
Cheers, Daniel
On 18/09/15 23:19, Jayesh Nambiar wrote:
Here are the tests that I did:
With the patch applied, I see that Kamailio is invoking event individually for each netstring even when they come in different chunks. But I did see instances where when there were complete netstrings in a single chunk; kamailio did not raise an event for them. Here's the illustration:
T 1.2.3.4:48873 -> 5.6.7.8:3927 [AP]
146:{"event":"REGISTER","tindex":"16916","tlabel":"1753048277","PhoneNumber":"20708","DeviceId":"abcd1234abcd1234","CallId":" 21220-3848@5.6.7.821220-3848@5.6.7.8 "},144:{"event":"REGISTER","tindex":"7954","tlabel":"254315075","PhoneNumber":"20709","DeviceId":"abcd1234abcd1234","CallId":" 21223-3848@5.6.7.821223-3848@5.6.7.8 "},145:{"event":"REGISTER","tindex":"64529","tlabel":"599481568","PhoneNumber":"20709","DeviceId":"abcd1234abcd1234","CallId":" 21222-3848@5.6.7.821222-3848@5.6.7.8"},145:{"event":"REGISTER","tindex":"46605","tlabel":"112015324","PhoneNumber":"20710","DeviceId"
T 1.2.3.4:48873 -> 5.6.7.8:3927 [AP] :"abcd1234abcd1234","CallId":" 21224-3848@5.6.7.8 21224-3848@5.6.7.8"},
*The above two chunks contain 4 proper netstrings where the second chunk contains part of the 4th netstring. In this case Evapi properly raised 4 individual events.*
1.2.3.4:48873 -> 5.6.7.8:3927 [AP]
145:{"event":"REGISTER","tindex":"55567","tlabel":"627458699","PhoneNumber":"20711","DeviceId":"abcd1234abcd1234","CallId":" 21225-3848@5.6.7.821225-3848@5.6.7.8 "},143:{"event":"REGISTER","tindex":"28682","tlabel":"9676691","PhoneNumber":"20712","DeviceId":"abcd1234abcd1234","CallId":" 21226-3848@5.6.7.821226-3848@5.6.7.8"},
*This above chunk contains two complete netstrings but Kamailio never raised events for these two netstrings.*
For the events that were not raised I see proper Kamailio Logs which is: evapi_recv_client(): {0} [1.2.3.4:48873] - received [145:{"event":"REGISTER","tindex":"55567","tlabel":"627458699","PhoneNumber":"20711","DeviceId":"abcd1234abcd1234","CallId":" 21225-3848@5.6.7.821225-3848@5.6.7.8 "},143:{"event":"REGISTER","tindex":"28682","tlabel":"9676691","PhoneNumber":"20712","DeviceId":"abcd1234abcd1234","CallId":" 21226-3848@5.6.7.821226-3848@5.6.7.8"},]
But I don't see the logs that I've written in the script after the event was raised, which means Kamailio did not invoke events for these two netstrings.
So in my tests, out of 27232 messages sent, there were approximately 27 messages for which the events were not raised by Evapi. The rate of messages started at 500cps and I stopped after I saw missed events at around 1200cps.
Thanks, and do let me know for any further tests or information required about the same.
- Jayesh
On Fri, Sep 18, 2015 at 2:55 PM, Jayesh Nambiar < jayesh1017@gmail.comjayesh1017@gmail.com> wrote:
> Will have this tested by tomorrow and will get back to you. Thanks. > > - Jayesh > > On Fri, Sep 18, 2015 at 1:41 PM, Daniel-Constantin Mierla < > miconda@gmail.commiconda@gmail.com> wrote: > >> Hello, >> >> I just pushed a patch to master branch that should cope with >> partial data received on tcp connection. No time to test at all, therefore >> any feedback will be appreciated. >> >> Cheers, >> Daniel >> >> >> On 15/09/15 14:52, Daniel-Constantin Mierla wrote: >> >> Hello, >> >> I will look if there are options in libev to buffer data or try to >> implement a buffering mechanism locally for such cases. >> >> Cheers, >> Daniel >> >> On 14/09/15 23:00, Jayesh Nambiar wrote: >> >> Hello Daniel, >> After further testing with evapi module, I figured that when >> Netstrings are used, an event route is invoked individually for each >> message even if if multiple netstring messages are received in a single TCP >> packet. But this doesn't work effectively when a single proper message is >> split-up in two packets. For Example, if a message arrives as: >> 12:Hello World!, 12:Hello World!, 12:Hello World! in a single >> packet, kamailio properly invokes the event route "evapi:message-received" >> thrice for every individual proper netstring message. >> But if the first packet contains: >> 12:Hello World!, 12:Hello >> And Second Packet contains: >> World!, 12:Hello World! >> the event route is invoked only once !! >> >> The above pattern is very much possible while sending and receiving >> packets over TCP Socket. Our tests for receiving an approximately 150 byte >> message over evapi socket at the rate of roughly 1000cps causes a lot of >> real events to be missed because of the above problem. You can never be >> sure when TCP will split messages in different chunks. >> This definitely looks like a bug which makes it not very reliable >> at large scale deployments. Would really appreciate your inputs on this. >> Thanks; >> >> - Jayesh >> >> >> >> On Thu, Sep 10, 2015 at 4:01 PM, Jayesh Nambiar < >> jayesh1017@gmail.comjayesh1017@gmail.com> wrote: >> >>> Hi Daniel, >>> Thanks for the quick response. So if I do not use Netstrings, does >>> Kamailio allow me to create a custom logic in the script. For eg. if I >>> decide to use newline as a delimiter, can I keep buffering the message >>> until I encounter the delimiter from the event route and then execute >>> whatever I have to within the script?? >>> >>> Thanks, >>> >>> - Jayesh >>> >>> On Thu, Sep 10, 2015 at 1:29 PM, Daniel-Constantin Mierla < >>> miconda@gmail.commiconda@gmail.com> wrote: >>> >>>> Hello, >>>> >>>> tcb is stream protocol and several messages can be queued on the >>>> pipe at the same time. That is the reason for netstring format, to be able >>>> to easily detect the boundaries of each message. If netstring format is >>>> enabled and kamailio receives several messages at once, it splits them and >>>> for each is executing the event route. >>>> >>>> If netstring format is not used, the kamailio is executing the >>>> event route with the entire content that was read at once from the tcp >>>> connection. >>>> >>>> Cheers, >>>> Daniel >>>> >>>> >>>> On 09/09/15 22:01, Jayesh Nambiar wrote: >>>> >>>> Hello, >>>> I'm exploring the evapi module for my kamailio to interface with >>>> an external node.js app for third party stuff like AAA, billing engine >>>> tasks, notifications and so on. I followed and took some ideas from the >>>> rtjson and evapi tutorial found here( >>>> http://kb.asipto.com/kamailio:k43-async-sip-routing-nodejs >>>> http://kb.asipto.com/kamailio:k43-async-sip-routing-nodejs) to >>>> build the node.js app consuming events. >>>> When I stress tested the scenario using SIPp and tried sending a >>>> lot of events at 300-350cps from Kamailio, I noticed that at times the >>>> client is receiving 2-3 events in a single message together although I do >>>> event_sync_relay once per SIP message received and have netstrings enabled. >>>> I believe this is a typical behavior of TCP and needs to be handled by the >>>> client using some kind of Netstring handler. Please correct me if I'm wrong. >>>> And hence I'd like to know what particularly needs to be taken >>>> care of while writing a client that is listening for events on raw tcp >>>> socket and how does kamailio handle this situation while receiving messages >>>> over TCP socket?? Does kamailio recognize the end of netstring properly on >>>> evapi:message-received and give exactly one message to take care of on >>>> every "message-received" event or should that be handled in the script >>>> somewhere !! >>>> I also referred cgrates client over evapi example which is >>>> written in GO, but I couldnt find them handling TCP streams clearly either. >>>> I'd really appreciate some expert suggestion here to make an >>>> informed decision on using the evapi module for a large scale solution. >>>> >>>> Thanks, >>>> >>>> - Jayesh >>>> >>>> >>>> >>>> >>>> _______________________________________________ >>>> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing listsr-users@lists.sip-router.orghttp://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users >>>> >>>> >>>> -- >>>> Daniel-Constantin Mierlahttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda >>>> Book: SIP Routing With Kamailio - http://www.asipto.com >>>> Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat >>>> >>>> >>>> _______________________________________________ >>>> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users >>>> mailing list >>>> sr-users@lists.sip-router.orgsr-users@lists.sip-router.org >>>> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users >>>> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users >>>> >>>> >>> >> >> -- >> Daniel-Constantin Mierlahttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda >> Book: SIP Routing With Kamailio - http://www.asipto.com >> Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat >> >> >> -- >> Daniel-Constantin Mierlahttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda >> Book: SIP Routing With Kamailio - http://www.asipto.com >> Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat >> >> >
-- Daniel-Constantin Mierlahttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
-- Daniel-Constantin Mierlahttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
-- Daniel-Constantin Mierlahttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
-- Daniel-Constantin Mierlahttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
Hello,
can you send here the log messages printed by evapi that contain "] - received [" which have:
- the previous bunch of json documents that were processed ok (just before the one that was discarded) - the bunch of json documents that were discarded - the next bunch of json documents that were processed ok
The logs were sent on could of emails and pastebin, so it is hard to track them -- if you have the logs, just locate the log messages with the discarded json and send it here along with the previous and next logs.
Cheers, Daniel
On 28/09/15 17:04, Jayesh Nambiar wrote:
Thank you daniel. Yes, they all get printed by the evapi module even when they get discarded. If you once again check the paste-bin, the logs pasted as Kamailio Logs are all discarded packets but show up as received by evapi module. I believe they got discarded because a packet that came before them was partial and evapi discarded the remaining part of the message in the new packet and encountered error.
- Jayesh
On Mon, Sep 28, 2015 at 11:13 AM Daniel-Constantin Mierla <miconda@gmail.com mailto:miconda@gmail.com> wrote:
Hello, I will look over it during next days. One more thing, are the packages sent by you printed by evapi module, even not they are not processed (but discarded)? Cheers, Daniel
Hi Daniel: I did a new test where I stopped as soon as I saw one message that got discarded. Basically the following message got discarded: {"event":"REGISTER","tindex":"52171","tlabel":"748030241","PhoneNumber":"11161","DeviceId":"abcd1234abcd1234","CallId":" 10943-15480@198.24.63.39"} The CallId parameter in the above json is auto incremented, so I tried gathering logs for messages with following CallIds: "CallId":"10940-15480@198.24.63.39" "CallId":"10941-15480@198.24.63.39" "CallId":"10942-15480@198.24.63.39" "CallId":"10943-15480@198.24.63.39" "CallId":"10944-15480@198.24.63.39" "CallId":"10945-15480@198.24.63.39"
And the message with CallID "CallId":"10943-15480@198.24.63.39" got discarded.
Here's the bunch of kamailio logs printed by the evapi module: *Logs for processed messages just before the discarded message:* DEBUG: evapi [evapi_dispatch.c:492]: evapi_recv_notify(): received [0x7f43da494ad0] [146:{"event":"REGISTER","tindex":"61133","tlabel":"450961398","PhoneNumber":"11158","DeviceId":"abcd1234abcd1234","CallId":" 10940-15480@198.24.63.39"},] (151)
NOTICE: evapi_recv_client(): {0} [198.24.63.45:48905] - received [147:{"event":"REGISTER","tindex":"25291","tlabel":"2047024302","PhoneNumber":"11132","DeviceId":"abcd1234abcd1234","CallId":" 10914-15480@198.24.63.39 "},146:{"event":"REGISTER","tindex":"61136","tlabel":"839735223","PhoneNumber":"11148","DeviceId":"abcd1234abcd1234","CallId":" 10930-15480@198.24.63.39 "},143:{"event":"REGISTER","tindex":"4562","tlabel":"6586867","PhoneNumber":"11149","DeviceId":"abcd1234abcd1234","CallId":" 10931-15480@198.24.63.39 "},146:{"event":"REGISTER","tindex":"43213","tlabel":"220429340","PhoneNumber":"11151","DeviceId":"abcd1234abcd1234","CallId":" 10932-15480@198.24.63.39 "},146:{"event":"REGISTER","tindex":"52174","tlabel":"708781916","PhoneNumber":"11150","DeviceId":"abcd1234abcd1234","CallId":" 10933-15480@198.24.63.39 "},147:{"event":"REGISTER","tindex":"34251","tlabel":"1354642017","PhoneNumber":"11153","DeviceId":"abcd1234abcd1234","CallId":" 10935-15480@198.24.63.39 "},146:{"event":"REGISTER","tindex":"7366","tlabel":"1368396808","PhoneNumber":"11154","DeviceId":"abcd1234abcd1234","CallId":" 10936-15480@198.24.63.39 "},146:{"event":"REGISTER","tindex":"25289","tlabel":"119788881","PhoneNumber":"11152","DeviceId":"abcd1234abcd1234","CallId":" 10934-15480@198.24.63.39 "},146:{"event":"REGISTER","tindex":"16327","tlabel":"971937029","PhoneNumber":"11155","DeviceId":"abcd1234abcd1234","CallId":" 10937-15480@198.24.63.39 "},146:{"event":"REGISTER","tindex":"22242","tlabel":"609737074","PhoneNumber":"11156","DeviceId":"abcd1234abcd1234","CallId":" 10938-15480@198.24.63.39 "},146:{"event":"REGISTER","tindex":"10720","tlabel":"930036340","PhoneNumber":"11157","DeviceId":"abcd1234abcd1234","CallId":" 10939-15480@198.24.63.39 "},146:{"event":"REGISTER","tindex":"61133","tlabel":"450961398","PhoneNumber":"11158","DeviceId":"abcd1234abcd1234","CallId":" 10940-15480@198.24.63.39"},146:{"event":"REGISTER","tindex":"43210","tlabel":"639090568","PhoneNumber":"11160","Dev] (1899) (0)
evapi_recv_client(): executing event route for frame: [{"event":"REGISTER","tindex":"61133","tlabel":"450961398","PhoneNumber":"11158","DeviceId":"abcd1234abcd1234","CallId":" 10940-15480@198.24.63.39"}] (146)
Sep 29 05:35:14 v39 /usr/local/kamailio-dev/sbin/kamailio[15387]: DEBUG: evapi [evapi_dispatch.c:492]: evapi_recv_notify(): received [0x7f43da4b67e0] [146:{"event":"REGISTER","tindex":"4559","tlabel":"1701553747","PhoneNumber":"11159","DeviceId":"abcd1234abcd1234","CallId":" 10941-15480@198.24.63.39"},] (151)
NOTICE: evapi [evapi_dispatch.c:290]: evapi_recv_client(): {0} [ 198.24.63.45:48905] - received [iceId":"abcd1234abcd1234","CallId":" 10942-15480@198.24.63.39 "},146:{"event":"REGISTER","tindex":"4559","tlabel":"1701553747","PhoneNumber":"11159","DeviceId":"abcd1234abcd1234","CallId":" 10941-15480@198.24.63.39"},] (214) (88)
DEBUG: evapi [evapi_dispatch.c:370]: evapi_recv_client(): executing event route for frame: [{"event":"REGISTER","tindex":"4559","tlabel":"1701553747","PhoneNumber":"11159","DeviceId":"abcd1234abcd1234","CallId":" 10941-15480@198.24.63.39"}] (146)
DEBUG: evapi [evapi_dispatch.c:492]: evapi_recv_notify(): received [0x7f43d93c2a70] [146:{"event":"REGISTER","tindex":"43210","tlabel":"639090568","PhoneNumber":"11160","DeviceId":"abcd1234abcd1234","CallId":" 10942-15480@198.24.63.39"},] (151)
DEBUG: evapi [evapi_dispatch.c:370]: evapi_recv_client(): executing event route for frame: [{"event":"REGISTER","tindex":"43210","tlabel":"639090568","PhoneNumber":"11160","DeviceId":"abcd1234abcd1234","CallId":" 10942-15480@198.24.63.39"}] (146)
DEBUG: evapi [evapi_dispatch.c:353]: evapi_recv_client(): residual data [146:{"event":"REGISTER","tindex":"43210","tlabel":"639090568","PhoneNumber":"11160","Dev] (88)
DEBUG: evapi [evapi_dispatch.c:361]: evapi_recv_client(): frame size mismatch the ending char ("): [{"event":"REGISTER","tindex":"43210","tlabel":"639090568","PhoneNumber":"11160","Dev146:{"event":"REGISTER","tindex":"52171","tlabel":"748030241",] (146)
*Logs for discarded message:* DEBUG: evapi [evapi_dispatch.c:492]: evapi_recv_notify(): received [0x7f43da3e69a0] [146:{"event":"REGISTER","tindex":"52171","tlabel":"748030241","PhoneNumber":"11161","DeviceId":"abcd1234abcd1234","CallId":" 10943-15480@198.24.63.39"},] (151)
NOTICE: evapi [evapi_dispatch.c:290]: evapi_recv_client(): {0} [ 198.24.63.45:48905] - received [146:{"event":"REGISTER","tindex":"52171","tlabel":"748030241","PhoneNumber":"11161","DeviceId":"abcd1234abcd1234","CallId":" 10943-15480@198.24.63.39"},] (151) (88)
*Logs for messages after the discarded message:* DEBUG: evapi [evapi_dispatch.c:492]: evapi_recv_notify(): received [0x7f43da3e6aa0] [146:{"event":"REGISTER","tindex":"25286","tlabel":"809034258","PhoneNumber":"11163","DeviceId":"abcd1234abcd1234","CallId":" 10944-15480@198.24.63.39"},] (151)
NOTICE: evapi [evapi_dispatch.c:290]: evapi_recv_client(): {0} [ 198.24.63.45:48905] - received [146:{"event":"REGISTER","tindex":"25286","tlabel":"809034258","PhoneNumber":"11163","DeviceId":"abcd1234abcd1234","CallId":" 10944-15480@198.24.63.39"},] (151) (0)
DEBUG: evapi [evapi_dispatch.c:370]: evapi_recv_client(): executing event route for frame: [{"event":"REGISTER","tindex":"25286","tlabel":"809034258","PhoneNumber":"11163","DeviceId":"abcd1234abcd1234","CallId":" 10944-15480@198.24.63.39"}] (146)
DEBUG: evapi [evapi_dispatch.c:492]: evapi_recv_notify(): received [0x7f43d9598778] [146:{"event":"REGISTER","tindex":"34248","tlabel":"187029249","PhoneNumber":"11162","DeviceId":"abcd1234abcd1234","CallId":" 10945-15480@198.24.63.39"},] (151)
NOTICE: evapi [evapi_dispatch.c:290]: evapi_recv_client(): {0} [ 198.24.63.45:48905] - received [146:{"event":"REGISTER","tindex":"34248","tlabel":"187029249","PhoneNumber":"11162","DeviceId":"abcd1234abcd1234","CallId":" 10945-15480@198.24.63.39"},] (151) (0)
DEBUG: evapi [evapi_dispatch.c:370]: evapi_recv_client(): executing event route for frame: [{"event":"REGISTER","tindex":"34248","tlabel":"187029249","PhoneNumber":"11162","DeviceId":"abcd1234abcd1234","CallId":" 10945-15480@198.24.63.39"}] (146)
DEBUG: evapi [evapi_dispatch.c:492]: evapi_recv_notify(): received [0x7f43d9598878] [145:{"event":"REGISTER","tindex":"7363","tlabel":"949153805","PhoneNumber":"11164","DeviceId":"abcd1234abcd1234","CallId":" 10946-15480@198.24.63.39"},] (150)
NOTICE: evapi [evapi_dispatch.c:290]: evapi_recv_client(): {0} [ 198.24.63.45:48905] - received [145:{"event":"REGISTER","tindex":"7363","tlabel":"949153805","PhoneNumber":"11164","DeviceId":"abcd1234abcd1234","CallId":" 10946-15480@198.24.63.39"},] (150) (0)
DEBUG: evapi [evapi_dispatch.c:370]: evapi_recv_client(): executing event route for frame: [{"event":"REGISTER","tindex":"7363","tlabel":"949153805","PhoneNumber":"11164","DeviceId":"abcd1234abcd1234","CallId":" 10946-15480@198.24.63.39"}] (145)
Just in case if you need to see the NGREP trace for the above messages, it is here: http://pastebin.com/4zTKmBJX
Thanks,
- Jayesh
On Tue, Sep 29, 2015 at 12:16 PM Daniel-Constantin Mierla miconda@gmail.com wrote:
Hello,
can you send here the log messages printed by evapi that contain "] - received [" which have:
- the previous bunch of json documents that were processed ok (just before
the one that was discarded)
- the bunch of json documents that were discarded
- the next bunch of json documents that were processed ok
The logs were sent on could of emails and pastebin, so it is hard to track them -- if you have the logs, just locate the log messages with the discarded json and send it here along with the previous and next logs.
Cheers, Daniel
On 28/09/15 17:04, Jayesh Nambiar wrote:
Thank you daniel. Yes, they all get printed by the evapi module even when they get discarded. If you once again check the paste-bin, the logs pasted as Kamailio Logs are all discarded packets but show up as received by evapi module. I believe they got discarded because a packet that came before them was partial and evapi discarded the remaining part of the message in the new packet and encountered error.
- Jayesh
On Mon, Sep 28, 2015 at 11:13 AM Daniel-Constantin Mierla < miconda@gmail.com> wrote:
Hello,
I will look over it during next days.
One more thing, are the packages sent by you printed by evapi module, even not they are not processed (but discarded)?
Cheers, Daniel
-- Daniel-Constantin Mierlahttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com
Hi Daniel, Taking a second look at the NGREP Trace I think I can see what is going wrong. The message in the third packet was discarded by Evapi module. If you look at the first two packets carefully, the first packet ends at Dev and the second packet is the continuation of first packet which starts with "iceId". The first two packets are processed perfectly fine. Now the third packet is a new message, but Kamailio is trying to continue the third packet again with the first packet. So even when the third packet starts with '146:' Kamailio considers it to be starting with 'Dev146'. This causes the evapi to discard this message since it does not remain a valid netstring then. Here the three packets that I'm talking about:
T 198.24.63.45:48905 -> 198.24.63.39:3927 [A] 146:{"event":"REGISTER","tindex":"43213","tlabel":"220429340","PhoneNumber":"11151","DeviceId":"abcd1234abcd1234","CallId":"10932-15480@198.24.63.39"},146:{"event":"REGISTER","tindex":"52174","tlabel":"708781916","PhoneNumber":"11150","DeviceId":"abcd1234abcd1234","CallId":"10933-15480@198.24.63.39"},147:{"event":"REGISTER","tindex":"34251","tlabel":"1354642017","PhoneNumber":"11153","DeviceId":"abcd1234abcd1234","CallId":"10935-15480@198.24.63.39"},146:{"event":"REGISTER","tindex":"7366","tlabel":"1368396808","PhoneNumber":"11154","DeviceId":"abcd1234abcd1234","CallId":"10936-15480@198.24.63.39"},146:{"event":"REGISTER","tindex":"25289","tlabel":"119788881","PhoneNumber":"11152","DeviceId":"abcd1234abcd1234","CallId":"10934-15480@198.24.63.39"},146:{"event":"REGISTER","tindex":"16327","tlabel":"971937029","PhoneNumber":"11155","DeviceId":"abcd1234abcd1234","CallId":"10937-15480@198.24.63.39"},146:{"event":"REGISTER","tindex":"22242","tlabel":"609737074","PhoneNumber":"11156","DeviceId":"abcd1234abcd1234","CallId":"10938-15480@198.24.63.39"},146:{"event":"REGISTER","tindex":"10720","tlabel":"930036340","PhoneNumber":"11157","DeviceId":"abcd1234abcd1234","CallId":"10939-15480@198.24.63.39"},146:{"event":"REGISTER","tindex":"61133","tlabel":"450961398","PhoneNumber":"11158","DeviceId":"abcd1234abcd1234","CallId":"10940-15480@198.24.63.39"},146:{"event":"REGISTER","tindex":"43210","tlabel":"639090568","PhoneNumber":"11160","Dev
T 198.24.63.45:48905 -> 198.24.63.39:3927 [AP] iceId":"abcd1234abcd1234","CallId":"10942-15480@198.24.63.39"},146:{"event":"REGISTER","tindex":"4559","tlabel":"1701553747","PhoneNumber":"11159","DeviceId":"abcd1234abcd1234","CallId":"10941-15480@198.24.63.39"},
T 198.24.63.45:48905 -> 198.24.63.39:3927 [AP] 146:{"event":"REGISTER","tindex":"52171","tlabel":"748030241","PhoneNumber":"11161","DeviceId":"abcd1234abcd1234","CallId":"10943-15480@198.24.63.39"},
On Tue, Sep 29, 2015 at 4:43 PM Jayesh Nambiar jayesh1017@gmail.com wrote:
Hi Daniel: I did a new test where I stopped as soon as I saw one message that got discarded. Basically the following message got discarded:
{"event":"REGISTER","tindex":"52171","tlabel":"748030241","PhoneNumber":"11161","DeviceId":"abcd1234abcd1234","CallId":" 10943-15480@198.24.63.39"} The CallId parameter in the above json is auto incremented, so I tried gathering logs for messages with following CallIds: "CallId":"10940-15480@198.24.63.39" "CallId":"10941-15480@198.24.63.39" "CallId":"10942-15480@198.24.63.39" "CallId":"10943-15480@198.24.63.39" "CallId":"10944-15480@198.24.63.39" "CallId":"10945-15480@198.24.63.39"
And the message with CallID "CallId":"10943-15480@198.24.63.39" got discarded.
Here's the bunch of kamailio logs printed by the evapi module: *Logs for processed messages just before the discarded message:* DEBUG: evapi [evapi_dispatch.c:492]: evapi_recv_notify(): received [0x7f43da494ad0] [146:{"event":"REGISTER","tindex":"61133","tlabel":"450961398","PhoneNumber":"11158","DeviceId":"abcd1234abcd1234","CallId":" 10940-15480@198.24.63.39"},] (151)
NOTICE: evapi_recv_client(): {0} [198.24.63.45:48905] - received [147:{"event":"REGISTER","tindex":"25291","tlabel":"2047024302","PhoneNumber":"11132","DeviceId":"abcd1234abcd1234","CallId":" 10914-15480@198.24.63.39 "},146:{"event":"REGISTER","tindex":"61136","tlabel":"839735223","PhoneNumber":"11148","DeviceId":"abcd1234abcd1234","CallId":" 10930-15480@198.24.63.39 "},143:{"event":"REGISTER","tindex":"4562","tlabel":"6586867","PhoneNumber":"11149","DeviceId":"abcd1234abcd1234","CallId":" 10931-15480@198.24.63.39 "},146:{"event":"REGISTER","tindex":"43213","tlabel":"220429340","PhoneNumber":"11151","DeviceId":"abcd1234abcd1234","CallId":" 10932-15480@198.24.63.39 "},146:{"event":"REGISTER","tindex":"52174","tlabel":"708781916","PhoneNumber":"11150","DeviceId":"abcd1234abcd1234","CallId":" 10933-15480@198.24.63.39 "},147:{"event":"REGISTER","tindex":"34251","tlabel":"1354642017","PhoneNumber":"11153","DeviceId":"abcd1234abcd1234","CallId":" 10935-15480@198.24.63.39 "},146:{"event":"REGISTER","tindex":"7366","tlabel":"1368396808","PhoneNumber":"11154","DeviceId":"abcd1234abcd1234","CallId":" 10936-15480@198.24.63.39 "},146:{"event":"REGISTER","tindex":"25289","tlabel":"119788881","PhoneNumber":"11152","DeviceId":"abcd1234abcd1234","CallId":" 10934-15480@198.24.63.39 "},146:{"event":"REGISTER","tindex":"16327","tlabel":"971937029","PhoneNumber":"11155","DeviceId":"abcd1234abcd1234","CallId":" 10937-15480@198.24.63.39 "},146:{"event":"REGISTER","tindex":"22242","tlabel":"609737074","PhoneNumber":"11156","DeviceId":"abcd1234abcd1234","CallId":" 10938-15480@198.24.63.39 "},146:{"event":"REGISTER","tindex":"10720","tlabel":"930036340","PhoneNumber":"11157","DeviceId":"abcd1234abcd1234","CallId":" 10939-15480@198.24.63.39 "},146:{"event":"REGISTER","tindex":"61133","tlabel":"450961398","PhoneNumber":"11158","DeviceId":"abcd1234abcd1234","CallId":" 10940-15480@198.24.63.39"},146:{"event":"REGISTER","tindex":"43210","tlabel":"639090568","PhoneNumber":"11160","Dev] (1899) (0)
evapi_recv_client(): executing event route for frame: [{"event":"REGISTER","tindex":"61133","tlabel":"450961398","PhoneNumber":"11158","DeviceId":"abcd1234abcd1234","CallId":" 10940-15480@198.24.63.39"}] (146)
Sep 29 05:35:14 v39 /usr/local/kamailio-dev/sbin/kamailio[15387]: DEBUG: evapi [evapi_dispatch.c:492]: evapi_recv_notify(): received [0x7f43da4b67e0] [146:{"event":"REGISTER","tindex":"4559","tlabel":"1701553747","PhoneNumber":"11159","DeviceId":"abcd1234abcd1234","CallId":" 10941-15480@198.24.63.39"},] (151)
NOTICE: evapi [evapi_dispatch.c:290]: evapi_recv_client(): {0} [ 198.24.63.45:48905] - received [iceId":"abcd1234abcd1234","CallId":" 10942-15480@198.24.63.39 "},146:{"event":"REGISTER","tindex":"4559","tlabel":"1701553747","PhoneNumber":"11159","DeviceId":"abcd1234abcd1234","CallId":" 10941-15480@198.24.63.39"},] (214) (88)
DEBUG: evapi [evapi_dispatch.c:370]: evapi_recv_client(): executing event route for frame: [{"event":"REGISTER","tindex":"4559","tlabel":"1701553747","PhoneNumber":"11159","DeviceId":"abcd1234abcd1234","CallId":" 10941-15480@198.24.63.39"}] (146)
DEBUG: evapi [evapi_dispatch.c:492]: evapi_recv_notify(): received [0x7f43d93c2a70] [146:{"event":"REGISTER","tindex":"43210","tlabel":"639090568","PhoneNumber":"11160","DeviceId":"abcd1234abcd1234","CallId":" 10942-15480@198.24.63.39"},] (151)
DEBUG: evapi [evapi_dispatch.c:370]: evapi_recv_client(): executing event route for frame: [{"event":"REGISTER","tindex":"43210","tlabel":"639090568","PhoneNumber":"11160","DeviceId":"abcd1234abcd1234","CallId":" 10942-15480@198.24.63.39"}] (146)
DEBUG: evapi [evapi_dispatch.c:353]: evapi_recv_client(): residual data [146:{"event":"REGISTER","tindex":"43210","tlabel":"639090568","PhoneNumber":"11160","Dev] (88)
DEBUG: evapi [evapi_dispatch.c:361]: evapi_recv_client(): frame size mismatch the ending char ("): [{"event":"REGISTER","tindex":"43210","tlabel":"639090568","PhoneNumber":"11160","Dev146:{"event":"REGISTER","tindex":"52171","tlabel":"748030241",] (146)
*Logs for discarded message:* DEBUG: evapi [evapi_dispatch.c:492]: evapi_recv_notify(): received [0x7f43da3e69a0] [146:{"event":"REGISTER","tindex":"52171","tlabel":"748030241","PhoneNumber":"11161","DeviceId":"abcd1234abcd1234","CallId":" 10943-15480@198.24.63.39"},] (151)
NOTICE: evapi [evapi_dispatch.c:290]: evapi_recv_client(): {0} [ 198.24.63.45:48905] - received [146:{"event":"REGISTER","tindex":"52171","tlabel":"748030241","PhoneNumber":"11161","DeviceId":"abcd1234abcd1234","CallId":" 10943-15480@198.24.63.39"},] (151) (88)
*Logs for messages after the discarded message:* DEBUG: evapi [evapi_dispatch.c:492]: evapi_recv_notify(): received [0x7f43da3e6aa0] [146:{"event":"REGISTER","tindex":"25286","tlabel":"809034258","PhoneNumber":"11163","DeviceId":"abcd1234abcd1234","CallId":" 10944-15480@198.24.63.39"},] (151)
NOTICE: evapi [evapi_dispatch.c:290]: evapi_recv_client(): {0} [ 198.24.63.45:48905] - received [146:{"event":"REGISTER","tindex":"25286","tlabel":"809034258","PhoneNumber":"11163","DeviceId":"abcd1234abcd1234","CallId":" 10944-15480@198.24.63.39"},] (151) (0)
DEBUG: evapi [evapi_dispatch.c:370]: evapi_recv_client(): executing event route for frame: [{"event":"REGISTER","tindex":"25286","tlabel":"809034258","PhoneNumber":"11163","DeviceId":"abcd1234abcd1234","CallId":" 10944-15480@198.24.63.39"}] (146)
DEBUG: evapi [evapi_dispatch.c:492]: evapi_recv_notify(): received [0x7f43d9598778] [146:{"event":"REGISTER","tindex":"34248","tlabel":"187029249","PhoneNumber":"11162","DeviceId":"abcd1234abcd1234","CallId":" 10945-15480@198.24.63.39"},] (151)
NOTICE: evapi [evapi_dispatch.c:290]: evapi_recv_client(): {0} [ 198.24.63.45:48905] - received [146:{"event":"REGISTER","tindex":"34248","tlabel":"187029249","PhoneNumber":"11162","DeviceId":"abcd1234abcd1234","CallId":" 10945-15480@198.24.63.39"},] (151) (0)
DEBUG: evapi [evapi_dispatch.c:370]: evapi_recv_client(): executing event route for frame: [{"event":"REGISTER","tindex":"34248","tlabel":"187029249","PhoneNumber":"11162","DeviceId":"abcd1234abcd1234","CallId":" 10945-15480@198.24.63.39"}] (146)
DEBUG: evapi [evapi_dispatch.c:492]: evapi_recv_notify(): received [0x7f43d9598878] [145:{"event":"REGISTER","tindex":"7363","tlabel":"949153805","PhoneNumber":"11164","DeviceId":"abcd1234abcd1234","CallId":" 10946-15480@198.24.63.39"},] (150)
NOTICE: evapi [evapi_dispatch.c:290]: evapi_recv_client(): {0} [ 198.24.63.45:48905] - received [145:{"event":"REGISTER","tindex":"7363","tlabel":"949153805","PhoneNumber":"11164","DeviceId":"abcd1234abcd1234","CallId":" 10946-15480@198.24.63.39"},] (150) (0)
DEBUG: evapi [evapi_dispatch.c:370]: evapi_recv_client(): executing event route for frame: [{"event":"REGISTER","tindex":"7363","tlabel":"949153805","PhoneNumber":"11164","DeviceId":"abcd1234abcd1234","CallId":" 10946-15480@198.24.63.39"}] (145)
Just in case if you need to see the NGREP trace for the above messages, it is here: http://pastebin.com/4zTKmBJX
Thanks,
- Jayesh
On Tue, Sep 29, 2015 at 12:16 PM Daniel-Constantin Mierla < miconda@gmail.com> wrote:
Hello,
can you send here the log messages printed by evapi that contain "] - received [" which have:
- the previous bunch of json documents that were processed ok (just
before the one that was discarded)
- the bunch of json documents that were discarded
- the next bunch of json documents that were processed ok
The logs were sent on could of emails and pastebin, so it is hard to track them -- if you have the logs, just locate the log messages with the discarded json and send it here along with the previous and next logs.
Cheers, Daniel
On 28/09/15 17:04, Jayesh Nambiar wrote:
Thank you daniel. Yes, they all get printed by the evapi module even when they get discarded. If you once again check the paste-bin, the logs pasted as Kamailio Logs are all discarded packets but show up as received by evapi module. I believe they got discarded because a packet that came before them was partial and evapi discarded the remaining part of the message in the new packet and encountered error.
- Jayesh
On Mon, Sep 28, 2015 at 11:13 AM Daniel-Constantin Mierla < miconda@gmail.com> wrote:
Hello,
I will look over it during next days.
One more thing, are the packages sent by you printed by evapi module, even not they are not processed (but discarded)?
Cheers, Daniel
-- Daniel-Constantin Mierlahttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com
Hello,
thanks for troubleshooting further and pointing to the issue. I pushed a patch last night, hopefully fixes it. Try with latest master branch and let me know the results.
Cheers, Daniel
On 29/09/15 14:15, Jayesh Nambiar wrote:
Hi Daniel, Taking a second look at the NGREP Trace I think I can see what is going wrong. The message in the third packet was discarded by Evapi module. If you look at the first two packets carefully, the first packet ends at Dev and the second packet is the continuation of first packet which starts with "iceId". The first two packets are processed perfectly fine. Now the third packet is a new message, but Kamailio is trying to continue the third packet again with the first packet. So even when the third packet starts with '146:' Kamailio considers it to be starting with 'Dev146'. This causes the evapi to discard this message since it does not remain a valid netstring then. Here the three packets that I'm talking about:
T 198.24.63.45:48905 http://198.24.63.45:48905 -> 198.24.63.39:3927 http://198.24.63.39:3927 [A] 146:{"event":"REGISTER","tindex":"43213","tlabel":"220429340","PhoneNumber":"11151","DeviceId":"abcd1234abcd1234","CallId":"10932-15480@198.24.63.39 mailto:10932-15480@198.24.63.39"},146:{"event":"REGISTER","tindex":"52174","tlabel":"708781916","PhoneNumber":"11150","DeviceId":"abcd1234abcd1234","CallId":"10933-15480@198.24.63.39 mailto:10933-15480@198.24.63.39"},147:{"event":"REGISTER","tindex":"34251","tlabel":"1354642017","PhoneNumber":"11153","DeviceId":"abcd1234abcd1234","CallId":"10935-15480@198.24.63.39 mailto:10935-15480@198.24.63.39"},146:{"event":"REGISTER","tindex":"7366","tlabel":"1368396808","PhoneNumber":"11154","DeviceId":"abcd1234abcd1234","CallId":"10936-15480@198.24.63.39 mailto:10936-15480@198.24.63.39"},146:{"event":"REGISTER","tindex":"25289","tlabel":"119788881","PhoneNumber":"11152","DeviceId":"abcd1234abcd1234","CallId":"10934-15480@198.24.63.39 mailto:10934-15480@198.24.63.39"},146:{"event":"REGISTER","tindex":"16327","tlabel":"971937029","PhoneNumber":"11155","DeviceId":"abcd1234abcd1234","CallId":"10937-15480@198.24.63.39 mailto:10937-15480@198.24.63.39"},146:{"event":"REGISTER","tindex":"22242","tlabel":"609737074","PhoneNumber":"11156","DeviceId":"abcd1234abcd1234","CallId":"10938-15480@198.24.63.39 mailto:10938-15480@198.24.63.39"},146:{"event":"REGISTER","tindex":"10720","tlabel":"930036340","PhoneNumber":"11157","DeviceId":"abcd1234abcd1234","CallId":"10939-15480@198.24.63.39 mailto:10939-15480@198.24.63.39"},146:{"event":"REGISTER","tindex":"61133","tlabel":"450961398","PhoneNumber":"11158","DeviceId":"abcd1234abcd1234","CallId":"10940-15480@198.24.63.39 mailto:10940-15480@198.24.63.39"},146:{"event":"REGISTER","tindex":"43210","tlabel":"639090568","PhoneNumber":"11160","Dev
T 198.24.63.45:48905 http://198.24.63.45:48905 -> 198.24.63.39:3927 http://198.24.63.39:3927 [AP] iceId":"abcd1234abcd1234","CallId":"10942-15480@198.24.63.39 mailto:10942-15480@198.24.63.39"},146:{"event":"REGISTER","tindex":"4559","tlabel":"1701553747","PhoneNumber":"11159","DeviceId":"abcd1234abcd1234","CallId":"10941-15480@198.24.63.39 mailto:10941-15480@198.24.63.39"},
T 198.24.63.45:48905 http://198.24.63.45:48905 -> 198.24.63.39:3927 http://198.24.63.39:3927 [AP] 146:{"event":"REGISTER","tindex":"52171","tlabel":"748030241","PhoneNumber":"11161","DeviceId":"abcd1234abcd1234","CallId":"10943-15480@198.24.63.39 mailto:10943-15480@198.24.63.39"},
On Tue, Sep 29, 2015 at 4:43 PM Jayesh Nambiar <jayesh1017@gmail.com mailto:jayesh1017@gmail.com> wrote:
Hi Daniel: I did a new test where I stopped as soon as I saw one message that got discarded. Basically the following message got discarded: {"event":"REGISTER","tindex":"52171","tlabel":"748030241","PhoneNumber":"11161","DeviceId":"abcd1234abcd1234","CallId":"10943-15480@198.24.63.39 <mailto:10943-15480@198.24.63.39>"} The CallId parameter in the above json is auto incremented, so I tried gathering logs for messages with following CallIds: "CallId":"10940-15480@198.24.63.39 <mailto:10940-15480@198.24.63.39>" "CallId":"10941-15480@198.24.63.39 <mailto:10941-15480@198.24.63.39>" "CallId":"10942-15480@198.24.63.39 <mailto:10942-15480@198.24.63.39>" "CallId":"10943-15480@198.24.63.39 <mailto:10943-15480@198.24.63.39>" "CallId":"10944-15480@198.24.63.39 <mailto:10944-15480@198.24.63.39>" "CallId":"10945-15480@198.24.63.39 <mailto:10945-15480@198.24.63.39>" And the message with CallID "CallId":"10943-15480@198.24.63.39 <http://198.24.63.39>" got discarded. Here's the bunch of kamailio logs printed by the evapi module: *Logs for processed messages just before the discarded message:* DEBUG: evapi [evapi_dispatch.c:492]: evapi_recv_notify(): received [0x7f43da494ad0] [146:{"event":"REGISTER","tindex":"61133","tlabel":"450961398","PhoneNumber":"11158","DeviceId":"abcd1234abcd1234","CallId":"10940-15480@198.24.63.39 <mailto:10940-15480@198.24.63.39>"},] (151) NOTICE: evapi_recv_client(): {0} [198.24.63.45:48905 <http://198.24.63.45:48905>] - received [147:{"event":"REGISTER","tindex":"25291","tlabel":"2047024302","PhoneNumber":"11132","DeviceId":"abcd1234abcd1234","CallId":"10914-15480@198.24.63.39 <mailto:10914-15480@198.24.63.39>"},146:{"event":"REGISTER","tindex":"61136","tlabel":"839735223","PhoneNumber":"11148","DeviceId":"abcd1234abcd1234","CallId":"10930-15480@198.24.63.39 <mailto:10930-15480@198.24.63.39>"},143:{"event":"REGISTER","tindex":"4562","tlabel":"6586867","PhoneNumber":"11149","DeviceId":"abcd1234abcd1234","CallId":"10931-15480@198.24.63.39 <mailto:10931-15480@198.24.63.39>"},146:{"event":"REGISTER","tindex":"43213","tlabel":"220429340","PhoneNumber":"11151","DeviceId":"abcd1234abcd1234","CallId":"10932-15480@198.24.63.39 <mailto:10932-15480@198.24.63.39>"},146:{"event":"REGISTER","tindex":"52174","tlabel":"708781916","PhoneNumber":"11150","DeviceId":"abcd1234abcd1234","CallId":"10933-15480@198.24.63.39 <mailto:10933-15480@198.24.63.39>"},147:{"event":"REGISTER","tindex":"34251","tlabel":"1354642017","PhoneNumber":"11153","DeviceId":"abcd1234abcd1234","CallId":"10935-15480@198.24.63.39 <mailto:10935-15480@198.24.63.39>"},146:{"event":"REGISTER","tindex":"7366","tlabel":"1368396808","PhoneNumber":"11154","DeviceId":"abcd1234abcd1234","CallId":"10936-15480@198.24.63.39 <mailto:10936-15480@198.24.63.39>"},146:{"event":"REGISTER","tindex":"25289","tlabel":"119788881","PhoneNumber":"11152","DeviceId":"abcd1234abcd1234","CallId":"10934-15480@198.24.63.39 <mailto:10934-15480@198.24.63.39>"},146:{"event":"REGISTER","tindex":"16327","tlabel":"971937029","PhoneNumber":"11155","DeviceId":"abcd1234abcd1234","CallId":"10937-15480@198.24.63.39 <mailto:10937-15480@198.24.63.39>"},146:{"event":"REGISTER","tindex":"22242","tlabel":"609737074","PhoneNumber":"11156","DeviceId":"abcd1234abcd1234","CallId":"10938-15480@198.24.63.39 <mailto:10938-15480@198.24.63.39>"},146:{"event":"REGISTER","tindex":"10720","tlabel":"930036340","PhoneNumber":"11157","DeviceId":"abcd1234abcd1234","CallId":"10939-15480@198.24.63.39 <mailto:10939-15480@198.24.63.39>"},146:{"event":"REGISTER","tindex":"61133","tlabel":"450961398","PhoneNumber":"11158","DeviceId":"abcd1234abcd1234","CallId":"10940-15480@198.24.63.39 <mailto:10940-15480@198.24.63.39>"},146:{"event":"REGISTER","tindex":"43210","tlabel":"639090568","PhoneNumber":"11160","Dev] (1899) (0) evapi_recv_client(): executing event route for frame: [{"event":"REGISTER","tindex":"61133","tlabel":"450961398","PhoneNumber":"11158","DeviceId":"abcd1234abcd1234","CallId":"10940-15480@198.24.63.39 <mailto:10940-15480@198.24.63.39>"}] (146) Sep 29 05:35:14 v39 /usr/local/kamailio-dev/sbin/kamailio[15387]: DEBUG: evapi [evapi_dispatch.c:492]: evapi_recv_notify(): received [0x7f43da4b67e0] [146:{"event":"REGISTER","tindex":"4559","tlabel":"1701553747","PhoneNumber":"11159","DeviceId":"abcd1234abcd1234","CallId":"10941-15480@198.24.63.39 <mailto:10941-15480@198.24.63.39>"},] (151) NOTICE: evapi [evapi_dispatch.c:290]: evapi_recv_client(): {0} [198.24.63.45:48905 <http://198.24.63.45:48905>] - received [iceId":"abcd1234abcd1234","CallId":"10942-15480@198.24.63.39 <mailto:10942-15480@198.24.63.39>"},146:{"event":"REGISTER","tindex":"4559","tlabel":"1701553747","PhoneNumber":"11159","DeviceId":"abcd1234abcd1234","CallId":"10941-15480@198.24.63.39 <mailto:10941-15480@198.24.63.39>"},] (214) (88) DEBUG: evapi [evapi_dispatch.c:370]: evapi_recv_client(): executing event route for frame: [{"event":"REGISTER","tindex":"4559","tlabel":"1701553747","PhoneNumber":"11159","DeviceId":"abcd1234abcd1234","CallId":"10941-15480@198.24.63.39 <mailto:10941-15480@198.24.63.39>"}] (146) DEBUG: evapi [evapi_dispatch.c:492]: evapi_recv_notify(): received [0x7f43d93c2a70] [146:{"event":"REGISTER","tindex":"43210","tlabel":"639090568","PhoneNumber":"11160","DeviceId":"abcd1234abcd1234","CallId":"10942-15480@198.24.63.39 <mailto:10942-15480@198.24.63.39>"},] (151) DEBUG: evapi [evapi_dispatch.c:370]: evapi_recv_client(): executing event route for frame: [{"event":"REGISTER","tindex":"43210","tlabel":"639090568","PhoneNumber":"11160","DeviceId":"abcd1234abcd1234","CallId":"10942-15480@198.24.63.39 <mailto:10942-15480@198.24.63.39>"}] (146) DEBUG: evapi [evapi_dispatch.c:353]: evapi_recv_client(): residual data [146:{"event":"REGISTER","tindex":"43210","tlabel":"639090568","PhoneNumber":"11160","Dev] (88) DEBUG: evapi [evapi_dispatch.c:361]: evapi_recv_client(): frame size mismatch the ending char ("): [{"event":"REGISTER","tindex":"43210","tlabel":"639090568","PhoneNumber":"11160","Dev146:{"event":"REGISTER","tindex":"52171","tlabel":"748030241",] (146) *Logs for discarded message:* DEBUG: evapi [evapi_dispatch.c:492]: evapi_recv_notify(): received [0x7f43da3e69a0] [146:{"event":"REGISTER","tindex":"52171","tlabel":"748030241","PhoneNumber":"11161","DeviceId":"abcd1234abcd1234","CallId":"10943-15480@198.24.63.39 <mailto:10943-15480@198.24.63.39>"},] (151) NOTICE: evapi [evapi_dispatch.c:290]: evapi_recv_client(): {0} [198.24.63.45:48905 <http://198.24.63.45:48905>] - received [146:{"event":"REGISTER","tindex":"52171","tlabel":"748030241","PhoneNumber":"11161","DeviceId":"abcd1234abcd1234","CallId":"10943-15480@198.24.63.39 <mailto:10943-15480@198.24.63.39>"},] (151) (88) *Logs for messages after the discarded message:* DEBUG: evapi [evapi_dispatch.c:492]: evapi_recv_notify(): received [0x7f43da3e6aa0] [146:{"event":"REGISTER","tindex":"25286","tlabel":"809034258","PhoneNumber":"11163","DeviceId":"abcd1234abcd1234","CallId":"10944-15480@198.24.63.39 <mailto:10944-15480@198.24.63.39>"},] (151) NOTICE: evapi [evapi_dispatch.c:290]: evapi_recv_client(): {0} [198.24.63.45:48905 <http://198.24.63.45:48905>] - received [146:{"event":"REGISTER","tindex":"25286","tlabel":"809034258","PhoneNumber":"11163","DeviceId":"abcd1234abcd1234","CallId":"10944-15480@198.24.63.39 <mailto:10944-15480@198.24.63.39>"},] (151) (0) DEBUG: evapi [evapi_dispatch.c:370]: evapi_recv_client(): executing event route for frame: [{"event":"REGISTER","tindex":"25286","tlabel":"809034258","PhoneNumber":"11163","DeviceId":"abcd1234abcd1234","CallId":"10944-15480@198.24.63.39 <mailto:10944-15480@198.24.63.39>"}] (146) DEBUG: evapi [evapi_dispatch.c:492]: evapi_recv_notify(): received [0x7f43d9598778] [146:{"event":"REGISTER","tindex":"34248","tlabel":"187029249","PhoneNumber":"11162","DeviceId":"abcd1234abcd1234","CallId":"10945-15480@198.24.63.39 <mailto:10945-15480@198.24.63.39>"},] (151) NOTICE: evapi [evapi_dispatch.c:290]: evapi_recv_client(): {0} [198.24.63.45:48905 <http://198.24.63.45:48905>] - received [146:{"event":"REGISTER","tindex":"34248","tlabel":"187029249","PhoneNumber":"11162","DeviceId":"abcd1234abcd1234","CallId":"10945-15480@198.24.63.39 <mailto:10945-15480@198.24.63.39>"},] (151) (0) DEBUG: evapi [evapi_dispatch.c:370]: evapi_recv_client(): executing event route for frame: [{"event":"REGISTER","tindex":"34248","tlabel":"187029249","PhoneNumber":"11162","DeviceId":"abcd1234abcd1234","CallId":"10945-15480@198.24.63.39 <mailto:10945-15480@198.24.63.39>"}] (146) DEBUG: evapi [evapi_dispatch.c:492]: evapi_recv_notify(): received [0x7f43d9598878] [145:{"event":"REGISTER","tindex":"7363","tlabel":"949153805","PhoneNumber":"11164","DeviceId":"abcd1234abcd1234","CallId":"10946-15480@198.24.63.39 <mailto:10946-15480@198.24.63.39>"},] (150) NOTICE: evapi [evapi_dispatch.c:290]: evapi_recv_client(): {0} [198.24.63.45:48905 <http://198.24.63.45:48905>] - received [145:{"event":"REGISTER","tindex":"7363","tlabel":"949153805","PhoneNumber":"11164","DeviceId":"abcd1234abcd1234","CallId":"10946-15480@198.24.63.39 <mailto:10946-15480@198.24.63.39>"},] (150) (0) DEBUG: evapi [evapi_dispatch.c:370]: evapi_recv_client(): executing event route for frame: [{"event":"REGISTER","tindex":"7363","tlabel":"949153805","PhoneNumber":"11164","DeviceId":"abcd1234abcd1234","CallId":"10946-15480@198.24.63.39 <mailto:10946-15480@198.24.63.39>"}] (145) Just in case if you need to see the NGREP trace for the above messages, it is here: http://pastebin.com/4zTKmBJX Thanks, - Jayesh On Tue, Sep 29, 2015 at 12:16 PM Daniel-Constantin Mierla <miconda@gmail.com <mailto:miconda@gmail.com>> wrote: Hello, can you send here the log messages printed by evapi that contain "] - received [" which have: - the previous bunch of json documents that were processed ok (just before the one that was discarded) - the bunch of json documents that were discarded - the next bunch of json documents that were processed ok The logs were sent on could of emails and pastebin, so it is hard to track them -- if you have the logs, just locate the log messages with the discarded json and send it here along with the previous and next logs. Cheers, Daniel On 28/09/15 17:04, Jayesh Nambiar wrote:Thank you daniel. Yes, they all get printed by the evapi module even when they get discarded. If you once again check the paste-bin, the logs pasted as Kamailio Logs are all discarded packets but show up as received by evapi module. I believe they got discarded because a packet that came before them was partial and evapi discarded the remaining part of the message in the new packet and encountered error. - Jayesh On Mon, Sep 28, 2015 at 11:13 AM Daniel-Constantin Mierla <miconda@gmail.com <mailto:miconda@gmail.com>> wrote: Hello, I will look over it during next days. One more thing, are the packages sent by you printed by evapi module, even not they are not processed (but discarded)? Cheers, Daniel-- Daniel-Constantin Mierla http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com
Hello,
wondering if you had the time to test and see if the issue has been fixed...
Cheers, Daniel
On 30/09/15 08:31, Daniel-Constantin Mierla wrote:
Hello,
thanks for troubleshooting further and pointing to the issue. I pushed a patch last night, hopefully fixes it. Try with latest master branch and let me know the results.
Cheers, Daniel
On 29/09/15 14:15, Jayesh Nambiar wrote:
Hi Daniel, Taking a second look at the NGREP Trace I think I can see what is going wrong. The message in the third packet was discarded by Evapi module. If you look at the first two packets carefully, the first packet ends at Dev and the second packet is the continuation of first packet which starts with "iceId". The first two packets are processed perfectly fine. Now the third packet is a new message, but Kamailio is trying to continue the third packet again with the first packet. So even when the third packet starts with '146:' Kamailio considers it to be starting with 'Dev146'. This causes the evapi to discard this message since it does not remain a valid netstring then. Here the three packets that I'm talking about:
T 198.24.63.45:48905 http://198.24.63.45:48905 -> 198.24.63.39:3927 http://198.24.63.39:3927 [A] 146:{"event":"REGISTER","tindex":"43213","tlabel":"220429340","PhoneNumber":"11151","DeviceId":"abcd1234abcd1234","CallId":"10932-15480@198.24.63.39 mailto:10932-15480@198.24.63.39"},146:{"event":"REGISTER","tindex":"52174","tlabel":"708781916","PhoneNumber":"11150","DeviceId":"abcd1234abcd1234","CallId":"10933-15480@198.24.63.39 mailto:10933-15480@198.24.63.39"},147:{"event":"REGISTER","tindex":"34251","tlabel":"1354642017","PhoneNumber":"11153","DeviceId":"abcd1234abcd1234","CallId":"10935-15480@198.24.63.39 mailto:10935-15480@198.24.63.39"},146:{"event":"REGISTER","tindex":"7366","tlabel":"1368396808","PhoneNumber":"11154","DeviceId":"abcd1234abcd1234","CallId":"10936-15480@198.24.63.39 mailto:10936-15480@198.24.63.39"},146:{"event":"REGISTER","tindex":"25289","tlabel":"119788881","PhoneNumber":"11152","DeviceId":"abcd1234abcd1234","C allId":"10934-15480@198.24.63.39 mailto:10934-15480@198.24.63.39"},146:{"event":"REGISTER","tindex":"16327","tlabel":"971937029","PhoneNumber":"11155","DeviceId":"abcd1234abcd1234","CallId":"10937-15480@198.24.63.39 mailto:10937-15480@198.24.63.39"},146:{"event":"REGISTER","tindex":"22242","tlabel":"609737074","PhoneNumber":"11156","DeviceId":"abcd1234abcd1234","CallId":"10938-15480@198.24.63.39 mailto:10938-15480@198.24.63.39"},146:{"event":"REGISTER","tindex":"10720","tlabel":"930036340","PhoneNumber":"11157","DeviceId":"abcd1234abcd1234","CallId":"10939-15480@198.24.63.39 mailto:10939-15480@198.24.63.39"},146:{"event":"REGISTER","tindex":"61133","tlabel":"450961398","PhoneNumber":"11158","DeviceId":"abcd1234abcd1234","CallId":"10940-15480@198.24.63.39 mailto:10940-15480@198.24.63.39"},146:{"event":"R EGISTER","tindex":"43210","tlabel":"639090568","PhoneNumber":"11160","Dev
T 198.24.63.45:48905 http://198.24.63.45:48905 -> 198.24.63.39:3927 http://198.24.63.39:3927 [AP] iceId":"abcd1234abcd1234","CallId":"10942-15480@198.24.63.39 mailto:10942-15480@198.24.63.39"},146:{"event":"REGISTER","tindex":"4559","tlabel":"1701553747","PhoneNumber":"11159","DeviceId":"abcd1234abcd1234","CallId":"10941-15480@198.24.63.39 mailto:10941-15480@198.24.63.39"},
T 198.24.63.45:48905 http://198.24.63.45:48905 -> 198.24.63.39:3927 http://198.24.63.39:3927 [AP] 146:{"event":"REGISTER","tindex":"52171","tlabel":"748030241","PhoneNumber":"11161","DeviceId":"abcd1234abcd1234","CallId":"10943-15480@198.24.63.39 mailto:10943-15480@198.24.63.39"},
On Tue, Sep 29, 2015 at 4:43 PM Jayesh Nambiar <jayesh1017@gmail.com mailto:jayesh1017@gmail.com> wrote:
Hi Daniel: I did a new test where I stopped as soon as I saw one message that got discarded. Basically the following message got discarded: {"event":"REGISTER","tindex":"52171","tlabel":"748030241","PhoneNumber":"11161","DeviceId":"abcd1234abcd1234","CallId":"10943-15480@198.24.63.39"} The CallId parameter in the above json is auto incremented, so I tried gathering logs for messages with following CallIds: "CallId":"10940-15480@198.24.63.39 <mailto:10940-15480@198.24.63.39>" "CallId":"10941-15480@198.24.63.39 <mailto:10941-15480@198.24.63.39>" "CallId":"10942-15480@198.24.63.39 <mailto:10942-15480@198.24.63.39>" "CallId":"10943-15480@198.24.63.39 <mailto:10943-15480@198.24.63.39>" "CallId":"10944-15480@198.24.63.39 <mailto:10944-15480@198.24.63.39>" "CallId":"10945-15480@198.24.63.39 <mailto:10945-15480@198.24.63.39>" And the message with CallID "CallId":"10943-15480@198.24.63.39 <http://198.24.63.39>" got discarded. Here's the bunch of kamailio logs printed by the evapi module: *Logs for processed messages just before the discarded message:* DEBUG: evapi [evapi_dispatch.c:492]: evapi_recv_notify(): received [0x7f43da494ad0] [146:{"event":"REGISTER","tindex":"61133","tlabel":"450961398","PhoneNumber":"11158","DeviceId":"abcd1234abcd1234","CallId":"10940-15480@198.24.63.39"},] (151) NOTICE: evapi_recv_client(): {0} [198.24.63.45:48905 <http://198.24.63.45:48905>] - received [147:{"event":"REGISTER","tindex":"25291","tlabel":"2047024302","PhoneNumber":"11132","DeviceId":"abcd1234abcd1234","CallId":"10914-15480@198.24.63.39"},146:{"event":"REGISTER","tindex":"61136","tlabel":"839735223","PhoneNumber":"11148","DeviceId":"abcd1234abcd1234","CallId":"10930-15480@198.24.63.39"},143:{"event":"REGISTER","tindex":"4562","tlabel":"6586867","PhoneNumber":"11149","DeviceId":"abcd1234abcd1234","CallId":"10931-15480@198.24.63.39"},146:{"event":"REGISTER","tindex":"43213","tlabel":"220429340","PhoneNumber":"11151","DeviceId":"abcd1234abcd1234","CallId":"10932-15480@198.24.63.39"},146:{"event":"REGISTER","tindex":"52174","tlabel":"708781916","PhoneNumber":"11150","DeviceId":"abcd1234abcd1234","CallId":"10933-15480@198.24.63.39"},147:{"event":"REGISTER","tindex":"34251","tlabel":"1354642017","PhoneNumber":"11153","DeviceId":"abcd1234abcd1234","CallId":"10935-15480@198.24.63.39"},146:{"event":"REGISTER","tindex":"7366","tlabel":"1368396808","PhoneNumber":"11154","DeviceId":"abcd1234abcd1234","CallId":"10936-15480@198.24.63.39"},146:{"event":"REGISTER","tindex":"25289","tlabel":"119788881","PhoneNumber":"11152","DeviceId":"abcd1234abcd1234","CallId":"10934-15480@198.24.63.39"},146:{"event":"REGISTER","tindex":"16327","tlabel":"971937029","PhoneNumber":"11155","DeviceId":"abcd1234abcd1234","CallId":"10937-15480@198.24.63.39"},146:{"event":"REGISTER","tindex":"22242","tlabel":"609737074","PhoneNumber":"11156","DeviceId":"abcd1234abcd1234","CallId":"10938-15480@198.24.63.39"},146:{"event":"REGISTER","tindex":"10720","tlabel":"930036340","PhoneNumber":"11157","DeviceId":"abcd1234abcd1234","CallId":"10939-15480@198.24.63.39"},146:{"event":"REGISTER","tindex":"61133","tlabel":"450961398","PhoneNumber":"11158","DeviceId":"abcd1234abcd1234","CallId":"10940-15480@198.24.63.39"},146:{"event":"REGISTER","tindex":"43210","tlabel":"639090568","PhoneNumber":"11160","Dev] (1899) (0) evapi_recv_client(): executing event route for frame: [{"event":"REGISTER","tindex":"61133","tlabel":"450961398","PhoneNumber":"11158","DeviceId":"abcd1234abcd1234","CallId":"10940-15480@198.24.63.39"}] (146) Sep 29 05:35:14 v39 /usr/local/kamailio-dev/sbin/kamailio[15387]: DEBUG: evapi [evapi_dispatch.c:492]: evapi_recv_notify(): received [0x7f43da4b67e0] [146:{"event":"REGISTER","tindex":"4559","tlabel":"1701553747","PhoneNumber":"11159","DeviceId":"abcd1234abcd1234","CallId":"10941-15480@198.24.63.39"},] (151) NOTICE: evapi [evapi_dispatch.c:290]: evapi_recv_client(): {0} [198.24.63.45:48905 <http://198.24.63.45:48905>] - received [iceId":"abcd1234abcd1234","CallId":"10942-15480@198.24.63.39"},146:{"event":"REGISTER","tindex":"4559","tlabel":"1701553747","PhoneNumber":"11159","DeviceId":"abcd1234abcd1234","CallId":"10941-15480@198.24.63.39"},] (214) (88) DEBUG: evapi [evapi_dispatch.c:370]: evapi_recv_client(): executing event route for frame: [{"event":"REGISTER","tindex":"4559","tlabel":"1701553747","PhoneNumber":"11159","DeviceId":"abcd1234abcd1234","CallId":"10941-15480@198.24.63.39"}] (146) DEBUG: evapi [evapi_dispatch.c:492]: evapi_recv_notify(): received [0x7f43d93c2a70] [146:{"event":"REGISTER","tindex":"43210","tlabel":"639090568","PhoneNumber":"11160","DeviceId":"abcd1234abcd1234","CallId":"10942-15480@198.24.63.39"},] (151) DEBUG: evapi [evapi_dispatch.c:370]: evapi_recv_client(): executing event route for frame: [{"event":"REGISTER","tindex":"43210","tlabel":"639090568","PhoneNumber":"11160","DeviceId":"abcd1234abcd1234","CallId":"10942-15480@198.24.63.39"}] (146) DEBUG: evapi [evapi_dispatch.c:353]: evapi_recv_client(): residual data [146:{"event":"REGISTER","tindex":"43210","tlabel":"639090568","PhoneNumber":"11160","Dev] (88) DEBUG: evapi [evapi_dispatch.c:361]: evapi_recv_client(): frame size mismatch the ending char ("): [{"event":"REGISTER","tindex":"43210","tlabel":"639090568","PhoneNumber":"11160","Dev146:{"event":"REGISTER","tindex":"52171","tlabel":"748030241",] (146) *Logs for discarded message:* DEBUG: evapi [evapi_dispatch.c:492]: evapi_recv_notify(): received [0x7f43da3e69a0] [146:{"event":"REGISTER","tindex":"52171","tlabel":"748030241","PhoneNumber":"11161","DeviceId":"abcd1234abcd1234","CallId":"10943-15480@198.24.63.39"},] (151) NOTICE: evapi [evapi_dispatch.c:290]: evapi_recv_client(): {0} [198.24.63.45:48905 <http://198.24.63.45:48905>] - received [146:{"event":"REGISTER","tindex":"52171","tlabel":"748030241","PhoneNumber":"11161","DeviceId":"abcd1234abcd1234","CallId":"10943-15480@198.24.63.39"},] (151) (88) *Logs for messages after the discarded message:* DEBUG: evapi [evapi_dispatch.c:492]: evapi_recv_notify(): received [0x7f43da3e6aa0] [146:{"event":"REGISTER","tindex":"25286","tlabel":"809034258","PhoneNumber":"11163","DeviceId":"abcd1234abcd1234","CallId":"10944-15480@198.24.63.39"},] (151) NOTICE: evapi [evapi_dispatch.c:290]: evapi_recv_client(): {0} [198.24.63.45:48905 <http://198.24.63.45:48905>] - received [146:{"event":"REGISTER","tindex":"25286","tlabel":"809034258","PhoneNumber":"11163","DeviceId":"abcd1234abcd1234","CallId":"10944-15480@198.24.63.39"},] (151) (0) DEBUG: evapi [evapi_dispatch.c:370]: evapi_recv_client(): executing event route for frame: [{"event":"REGISTER","tindex":"25286","tlabel":"809034258","PhoneNumber":"11163","DeviceId":"abcd1234abcd1234","CallId":"10944-15480@198.24.63.39"}] (146) DEBUG: evapi [evapi_dispatch.c:492]: evapi_recv_notify(): received [0x7f43d9598778] [146:{"event":"REGISTER","tindex":"34248","tlabel":"187029249","PhoneNumber":"11162","DeviceId":"abcd1234abcd1234","CallId":"10945-15480@198.24.63.39"},] (151) NOTICE: evapi [evapi_dispatch.c:290]: evapi_recv_client(): {0} [198.24.63.45:48905 <http://198.24.63.45:48905>] - received [146:{"event":"REGISTER","tindex":"34248","tlabel":"187029249","PhoneNumber":"11162","DeviceId":"abcd1234abcd1234","CallId":"10945-15480@198.24.63.39"},] (151) (0) DEBUG: evapi [evapi_dispatch.c:370]: evapi_recv_client(): executing event route for frame: [{"event":"REGISTER","tindex":"34248","tlabel":"187029249","PhoneNumber":"11162","DeviceId":"abcd1234abcd1234","CallId":"10945-15480@198.24.63.39"}] (146) DEBUG: evapi [evapi_dispatch.c:492]: evapi_recv_notify(): received [0x7f43d9598878] [145:{"event":"REGISTER","tindex":"7363","tlabel":"949153805","PhoneNumber":"11164","DeviceId":"abcd1234abcd1234","CallId":"10946-15480@198.24.63.39"},] (150) NOTICE: evapi [evapi_dispatch.c:290]: evapi_recv_client(): {0} [198.24.63.45:48905 <http://198.24.63.45:48905>] - received [145:{"event":"REGISTER","tindex":"7363","tlabel":"949153805","PhoneNumber":"11164","DeviceId":"abcd1234abcd1234","CallId":"10946-15480@198.24.63.39"},] (150) (0) DEBUG: evapi [evapi_dispatch.c:370]: evapi_recv_client(): executing event route for frame: [{"event":"REGISTER","tindex":"7363","tlabel":"949153805","PhoneNumber":"11164","DeviceId":"abcd1234abcd1234","CallId":"10946-15480@198.24.63.39"}] (145) Just in case if you need to see the NGREP trace for the above messages, it is here: http://pastebin.com/4zTKmBJX Thanks, - Jayesh On Tue, Sep 29, 2015 at 12:16 PM Daniel-Constantin Mierla <miconda@gmail.com <mailto:miconda@gmail.com>> wrote: Hello, can you send here the log messages printed by evapi that contain "] - received [" which have: - the previous bunch of json documents that were processed ok (just before the one that was discarded) - the bunch of json documents that were discarded - the next bunch of json documents that were processed ok The logs were sent on could of emails and pastebin, so it is hard to track them -- if you have the logs, just locate the log messages with the discarded json and send it here along with the previous and next logs. Cheers, Daniel On 28/09/15 17:04, Jayesh Nambiar wrote:Thank you daniel. Yes, they all get printed by the evapi module even when they get discarded. If you once again check the paste-bin, the logs pasted as Kamailio Logs are all discarded packets but show up as received by evapi module. I believe they got discarded because a packet that came before them was partial and evapi discarded the remaining part of the message in the new packet and encountered error. - Jayesh On Mon, Sep 28, 2015 at 11:13 AM Daniel-Constantin Mierla <miconda@gmail.com> wrote: Hello, I will look over it during next days. One more thing, are the packages sent by you printed by evapi module, even not they are not processed (but discarded)? Cheers, Daniel-- Daniel-Constantin Mierla http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com-- Daniel-Constantin Mierla http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
Yes that problem is definitely solved. Sorry to not post the response. I'm having some other issues in scaling and still trying to figure where the problem must be, but I can scale upto 2000 cps easily now.
Thanks for the efforts and the credits :)
- Jayesh
On Sun, Oct 11, 2015 at 10:59 PM Daniel-Constantin Mierla miconda@gmail.com wrote:
Hello,
wondering if you had the time to test and see if the issue has been fixed...
Cheers, Daniel
On 30/09/15 08:31, Daniel-Constantin Mierla wrote:
Hello,
thanks for troubleshooting further and pointing to the issue. I pushed a patch last night, hopefully fixes it. Try with latest master branch and let me know the results.
Cheers, Daniel
On 29/09/15 14:15, Jayesh Nambiar wrote:
Hi Daniel, Taking a second look at the NGREP Trace I think I can see what is going wrong. The message in the third packet was discarded by Evapi module. If you look at the first two packets carefully, the first packet ends at Dev and the second packet is the continuation of first packet which starts with "iceId". The first two packets are processed perfectly fine. Now the third packet is a new message, but Kamailio is trying to continue the third packet again with the first packet. So even when the third packet starts with '146:' Kamailio considers it to be starting with 'Dev146'. This causes the evapi to discard this message since it does not remain a valid netstring then. Here the three packets that I'm talking about:
T 198.24.63.45:48905 -> 198.24.63.39:3927 [A] 146:{"event":"REGISTER","tindex":"43213","tlabel":"220429340","PhoneNumber":"11151","DeviceId":"abcd1234abcd1234","CallId":"10932-15480@198.24.63.39"},146:{"event":"REGISTER","tindex":"52174","tlabel":"708781916","PhoneNumber":"11150","DeviceId":"abcd1234abcd1234","CallId":"10933-15480@198.24.63.39"},147:{"event":"REGISTER","tindex":"34251","tlabel":"1354642017","PhoneNumber":"11153","DeviceId":"abcd1234abcd1234","CallId":"10935-15480@198.24.63.39"},146:{"event":"REGISTER","tindex":"7366","tlabel":"1368396808","PhoneNumber":"11154","DeviceId":"abcd1234abcd1234","CallId":"10936-15480@198.24.63.39"},146:{"event":"REGISTER","tindex":"25289","tlabel":"119788881","PhoneNumber":"11152","DeviceId":"abcd1234abcd1234","C
allId":"10934-15480@198.24.63.39"},146:{"event":"REGISTER","tindex":"16327","tlabel":"971937029","PhoneNumber":"11155","DeviceId":"abcd1234abcd1234","CallId":"10937-15480@198.24.63.39"},146:{"event":"REGISTER","tindex":"22242","tlabel":"609737074","PhoneNumber":"11156","DeviceId":"abcd1234abcd1234","CallId":"10938-15480@198.24.63.39"},146:{"event":"REGISTER","tindex":"10720","tlabel":"930036340","PhoneNumber":"11157","DeviceId":"abcd1234abcd1234","CallId":"10939-15480@198.24.63.39"},146:{"event":"REGISTER","tindex":"61133","tlabel":"450961398","PhoneNumber":"11158","DeviceId":"abcd1234abcd1234","CallId":"10940-15480@198.24.63.39"},146:{"event":"R
EGISTER","tindex":"43210","tlabel":"639090568","PhoneNumber":"11160","Dev
T 198.24.63.45:48905 -> 198.24.63.39:3927 [AP] iceId":"abcd1234abcd1234","CallId":"10942-15480@198.24.63.39"},146:{"event":"REGISTER","tindex":"4559","tlabel":"1701553747","PhoneNumber":"11159","DeviceId":"abcd1234abcd1234","CallId":"10941-15480@198.24.63.39"},
T 198.24.63.45:48905 -> 198.24.63.39:3927 [AP] 146:{"event":"REGISTER","tindex":"52171","tlabel":"748030241","PhoneNumber":"11161","DeviceId":"abcd1234abcd1234","CallId":"10943-15480@198.24.63.39"},
On Tue, Sep 29, 2015 at 4:43 PM Jayesh Nambiar jayesh1017@gmail.com wrote:
Hi Daniel: I did a new test where I stopped as soon as I saw one message that got discarded. Basically the following message got discarded:
{"event":"REGISTER","tindex":"52171","tlabel":"748030241","PhoneNumber":"11161","DeviceId":"abcd1234abcd1234","CallId":" 10943-15480@198.24.63.39"} The CallId parameter in the above json is auto incremented, so I tried gathering logs for messages with following CallIds: "CallId":"10940-15480@198.24.63.39" "CallId":"10941-15480@198.24.63.39" "CallId":"10942-15480@198.24.63.39" "CallId":"10943-15480@198.24.63.39" "CallId":"10944-15480@198.24.63.39" "CallId":"10945-15480@198.24.63.39"
And the message with CallID "CallId":"10943-15480@198.24.63.39" got discarded.
Here's the bunch of kamailio logs printed by the evapi module: *Logs for processed messages just before the discarded message:* DEBUG: evapi [evapi_dispatch.c:492]: evapi_recv_notify(): received [0x7f43da494ad0] [146:{"event":"REGISTER","tindex":"61133","tlabel":"450961398","PhoneNumber":"11158","DeviceId":"abcd1234abcd1234","CallId":" 10940-15480@198.24.63.39"},] (151)
NOTICE: evapi_recv_client(): {0} [198.24.63.45:48905] - received [147:{"event":"REGISTER","tindex":"25291","tlabel":"2047024302","PhoneNumber":"11132","DeviceId":"abcd1234abcd1234","CallId":" 10914-15480@198.24.63.39 "},146:{"event":"REGISTER","tindex":"61136","tlabel":"839735223","PhoneNumber":"11148","DeviceId":"abcd1234abcd1234","CallId":" 10930-15480@198.24.63.39 "},143:{"event":"REGISTER","tindex":"4562","tlabel":"6586867","PhoneNumber":"11149","DeviceId":"abcd1234abcd1234","CallId":" 10931-15480@198.24.63.39 "},146:{"event":"REGISTER","tindex":"43213","tlabel":"220429340","PhoneNumber":"11151","DeviceId":"abcd1234abcd1234","CallId":" 10932-15480@198.24.63.39 "},146:{"event":"REGISTER","tindex":"52174","tlabel":"708781916","PhoneNumber":"11150","DeviceId":"abcd1234abcd1234","CallId":" 10933-15480@198.24.63.39 "},147:{"event":"REGISTER","tindex":"34251","tlabel":"1354642017","PhoneNumber":"11153","DeviceId":"abcd1234abcd1234","CallId":" 10935-15480@198.24.63.39 "},146:{"event":"REGISTER","tindex":"7366","tlabel":"1368396808","PhoneNumber":"11154","DeviceId":"abcd1234abcd1234","CallId":" 10936-15480@198.24.63.39 "},146:{"event":"REGISTER","tindex":"25289","tlabel":"119788881","PhoneNumber":"11152","DeviceId":"abcd1234abcd1234","CallId":" 10934-15480@198.24.63.39 "},146:{"event":"REGISTER","tindex":"16327","tlabel":"971937029","PhoneNumber":"11155","DeviceId":"abcd1234abcd1234","CallId":" 10937-15480@198.24.63.39 "},146:{"event":"REGISTER","tindex":"22242","tlabel":"609737074","PhoneNumber":"11156","DeviceId":"abcd1234abcd1234","CallId":" 10938-15480@198.24.63.39 "},146:{"event":"REGISTER","tindex":"10720","tlabel":"930036340","PhoneNumber":"11157","DeviceId":"abcd1234abcd1234","CallId":" 10939-15480@198.24.63.39 "},146:{"event":"REGISTER","tindex":"61133","tlabel":"450961398","PhoneNumber":"11158","DeviceId":"abcd1234abcd1234","CallId":" 10940-15480@198.24.63.39"},146:{"event":"REGISTER","tindex":"43210","tlabel":"639090568","PhoneNumber":"11160","Dev] (1899) (0)
evapi_recv_client(): executing event route for frame: [{"event":"REGISTER","tindex":"61133","tlabel":"450961398","PhoneNumber":"11158","DeviceId":"abcd1234abcd1234","CallId":" 10940-15480@198.24.63.39"}] (146)
Sep 29 05:35:14 v39 /usr/local/kamailio-dev/sbin/kamailio[15387]: DEBUG: evapi [evapi_dispatch.c:492]: evapi_recv_notify(): received [0x7f43da4b67e0] [146:{"event":"REGISTER","tindex":"4559","tlabel":"1701553747","PhoneNumber":"11159","DeviceId":"abcd1234abcd1234","CallId":" 10941-15480@198.24.63.39"},] (151)
NOTICE: evapi [evapi_dispatch.c:290]: evapi_recv_client(): {0} [ 198.24.63.45:48905] - received [iceId":"abcd1234abcd1234","CallId":" 10942-15480@198.24.63.39 "},146:{"event":"REGISTER","tindex":"4559","tlabel":"1701553747","PhoneNumber":"11159","DeviceId":"abcd1234abcd1234","CallId":" 10941-15480@198.24.63.39"},] (214) (88)
DEBUG: evapi [evapi_dispatch.c:370]: evapi_recv_client(): executing event route for frame: [{"event":"REGISTER","tindex":"4559","tlabel":"1701553747","PhoneNumber":"11159","DeviceId":"abcd1234abcd1234","CallId":" 10941-15480@198.24.63.39"}] (146)
DEBUG: evapi [evapi_dispatch.c:492]: evapi_recv_notify(): received [0x7f43d93c2a70] [146:{"event":"REGISTER","tindex":"43210","tlabel":"639090568","PhoneNumber":"11160","DeviceId":"abcd1234abcd1234","CallId":" 10942-15480@198.24.63.39"},] (151)
DEBUG: evapi [evapi_dispatch.c:370]: evapi_recv_client(): executing event route for frame: [{"event":"REGISTER","tindex":"43210","tlabel":"639090568","PhoneNumber":"11160","DeviceId":"abcd1234abcd1234","CallId":" 10942-15480@198.24.63.39"}] (146)
DEBUG: evapi [evapi_dispatch.c:353]: evapi_recv_client(): residual data [146:{"event":"REGISTER","tindex":"43210","tlabel":"639090568","PhoneNumber":"11160","Dev] (88)
DEBUG: evapi [evapi_dispatch.c:361]: evapi_recv_client(): frame size mismatch the ending char ("): [{"event":"REGISTER","tindex":"43210","tlabel":"639090568","PhoneNumber":"11160","Dev146:{"event":"REGISTER","tindex":"52171","tlabel":"748030241",] (146)
*Logs for discarded message:* DEBUG: evapi [evapi_dispatch.c:492]: evapi_recv_notify(): received [0x7f43da3e69a0] [146:{"event":"REGISTER","tindex":"52171","tlabel":"748030241","PhoneNumber":"11161","DeviceId":"abcd1234abcd1234","CallId":" 10943-15480@198.24.63.39"},] (151)
NOTICE: evapi [evapi_dispatch.c:290]: evapi_recv_client(): {0} [ 198.24.63.45:48905] - received [146:{"event":"REGISTER","tindex":"52171","tlabel":"748030241","PhoneNumber":"11161","DeviceId":"abcd1234abcd1234","CallId":" 10943-15480@198.24.63.39"},] (151) (88)
*Logs for messages after the discarded message:* DEBUG: evapi [evapi_dispatch.c:492]: evapi_recv_notify(): received [0x7f43da3e6aa0] [146:{"event":"REGISTER","tindex":"25286","tlabel":"809034258","PhoneNumber":"11163","DeviceId":"abcd1234abcd1234","CallId":" 10944-15480@198.24.63.39"},] (151)
NOTICE: evapi [evapi_dispatch.c:290]: evapi_recv_client(): {0} [ 198.24.63.45:48905] - received [146:{"event":"REGISTER","tindex":"25286","tlabel":"809034258","PhoneNumber":"11163","DeviceId":"abcd1234abcd1234","CallId":" 10944-15480@198.24.63.39"},] (151) (0)
DEBUG: evapi [evapi_dispatch.c:370]: evapi_recv_client(): executing event route for frame: [{"event":"REGISTER","tindex":"25286","tlabel":"809034258","PhoneNumber":"11163","DeviceId":"abcd1234abcd1234","CallId":" 10944-15480@198.24.63.39"}] (146)
DEBUG: evapi [evapi_dispatch.c:492]: evapi_recv_notify(): received [0x7f43d9598778] [146:{"event":"REGISTER","tindex":"34248","tlabel":"187029249","PhoneNumber":"11162","DeviceId":"abcd1234abcd1234","CallId":" 10945-15480@198.24.63.39"},] (151)
NOTICE: evapi [evapi_dispatch.c:290]: evapi_recv_client(): {0} [ 198.24.63.45:48905] - received [146:{"event":"REGISTER","tindex":"34248","tlabel":"187029249","PhoneNumber":"11162","DeviceId":"abcd1234abcd1234","CallId":" 10945-15480@198.24.63.39"},] (151) (0)
DEBUG: evapi [evapi_dispatch.c:370]: evapi_recv_client(): executing event route for frame: [{"event":"REGISTER","tindex":"34248","tlabel":"187029249","PhoneNumber":"11162","DeviceId":"abcd1234abcd1234","CallId":" 10945-15480@198.24.63.39"}] (146)
DEBUG: evapi [evapi_dispatch.c:492]: evapi_recv_notify(): received [0x7f43d9598878] [145:{"event":"REGISTER","tindex":"7363","tlabel":"949153805","PhoneNumber":"11164","DeviceId":"abcd1234abcd1234","CallId":" 10946-15480@198.24.63.39"},] (150)
NOTICE: evapi [evapi_dispatch.c:290]: evapi_recv_client(): {0} [ 198.24.63.45:48905] - received [145:{"event":"REGISTER","tindex":"7363","tlabel":"949153805","PhoneNumber":"11164","DeviceId":"abcd1234abcd1234","CallId":" 10946-15480@198.24.63.39"},] (150) (0)
DEBUG: evapi [evapi_dispatch.c:370]: evapi_recv_client(): executing event route for frame: [{"event":"REGISTER","tindex":"7363","tlabel":"949153805","PhoneNumber":"11164","DeviceId":"abcd1234abcd1234","CallId":" 10946-15480@198.24.63.39"}] (145)
Just in case if you need to see the NGREP trace for the above messages, it is here: http://pastebin.com/4zTKmBJX
Thanks,
- Jayesh
On Tue, Sep 29, 2015 at 12:16 PM Daniel-Constantin Mierla < miconda@gmail.com> wrote:
Hello,
can you send here the log messages printed by evapi that contain "] - received [" which have:
- the previous bunch of json documents that were processed ok (just
before the one that was discarded)
- the bunch of json documents that were discarded
- the next bunch of json documents that were processed ok
The logs were sent on could of emails and pastebin, so it is hard to track them -- if you have the logs, just locate the log messages with the discarded json and send it here along with the previous and next logs.
Cheers, Daniel
On 28/09/15 17:04, Jayesh Nambiar wrote:
Thank you daniel. Yes, they all get printed by the evapi module even when they get discarded. If you once again check the paste-bin, the logs pasted as Kamailio Logs are all discarded packets but show up as received by evapi module. I believe they got discarded because a packet that came before them was partial and evapi discarded the remaining part of the message in the new packet and encountered error.
- Jayesh
On Mon, Sep 28, 2015 at 11:13 AM Daniel-Constantin Mierla < miconda@gmail.com> wrote:
Hello,
I will look over it during next days.
One more thing, are the packages sent by you printed by evapi module, even not they are not processed (but discarded)?
Cheers, Daniel
-- Daniel-Constantin Mierlahttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com
-- Daniel-Constantin Mierlahttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
-- Daniel-Constantin Mierlahttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com
Thanks for testing and feedback -- good to know it was fixed, not to keep it open in my side anymore.
Cheers, Daniel
On 13/10/15 12:37, Jayesh Nambiar wrote:
Yes that problem is definitely solved. Sorry to not post the response. I'm having some other issues in scaling and still trying to figure where the problem must be, but I can scale upto 2000 cps easily now.
Thanks for the efforts and the credits :)
- Jayesh
On Sun, Oct 11, 2015 at 10:59 PM Daniel-Constantin Mierla <miconda@gmail.com mailto:miconda@gmail.com> wrote:
Hello, wondering if you had the time to test and see if the issue has been fixed... Cheers, Daniel On 30/09/15 08:31, Daniel-Constantin Mierla wrote:Hello, thanks for troubleshooting further and pointing to the issue. I pushed a patch last night, hopefully fixes it. Try with latest master branch and let me know the results. Cheers, Daniel On 29/09/15 14:15, Jayesh Nambiar wrote:Hi Daniel, Taking a second look at the NGREP Trace I think I can see what is going wrong. The message in the third packet was discarded by Evapi module. If you look at the first two packets carefully, the first packet ends at Dev and the second packet is the continuation of first packet which starts with "iceId". The first two packets are processed perfectly fine. Now the third packet is a new message, but Kamailio is trying to continue the third packet again with the first packet. So even when the third packet starts with '146:' Kamailio considers it to be starting with 'Dev146'. This causes the evapi to discard this message since it does not remain a valid netstring then. Here the three packets that I'm talking about: T 198.24.63.45:48905 <http://198.24.63.45:48905> -> 198.24.63.39:3927 <http://198.24.63.39:3927> [A] 146:{"event":"REGISTER","tindex":"43213","tlabel":"220429340","PhoneNumber":"11151","DeviceId":"abcd1234abcd1234","CallId":"10932-15480@198.24.63.39 <mailto:10932-15480@198.24.63.39>"},146:{"event":"REGISTER","tindex":"52174","tlabel":"708781916","PhoneNumber":"11150","DeviceId":"abcd1234abcd1234","CallId":"10933-15480@198.24.63.39 <mailto:10933-15480@198.24.63.39>"},147:{"event":"REGISTER","tindex":"34251","tlabel":"1354642017","PhoneNumber":"11153","DeviceId":"abcd1234abcd1234","CallId":"10935-15480@198.24.63.39 <mailto:10935-15480@198.24.63.39>"},146:{"event":"REGISTER","tindex":"7366","tlabel":"1368396808","PhoneNumber":"11154","DeviceId":"abcd1234abcd1234","CallId":"10936-15480@198.24.63.39 <mailto:10936-15480@198.24.63.39>"},146:{"event":"REGISTER","tindex":"25289","tlabel":"119788881","PhoneNumber":"11152","DeviceId":"abcd1234abcd1234","C allId":"10934-15480@198.24.63.39 <mailto:10934-15480@198.24.63.39>"},146:{"event":"REGISTER","tindex":"16327","tlabel":"971937029","PhoneNumber":"11155","DeviceId":"abcd1234abcd1234","CallId":"10937-15480@198.24.63.39 <mailto:10937-15480@198.24.63.39>"},146:{"event":"REGISTER","tindex":"22242","tlabel":"609737074","PhoneNumber":"11156","DeviceId":"abcd1234abcd1234","CallId":"10938-15480@198.24.63.39 <mailto:10938-15480@198.24.63.39>"},146:{"event":"REGISTER","tindex":"10720","tlabel":"930036340","PhoneNumber":"11157","DeviceId":"abcd1234abcd1234","CallId":"10939-15480@198.24.63.39 <mailto:10939-15480@198.24.63.39>"},146:{"event":"REGISTER","tindex":"61133","tlabel":"450961398","PhoneNumber":"11158","DeviceId":"abcd1234abcd1234","CallId":"10940-15480@198.24.63.39 <mailto:10940-15480@198.24.63.39>"},146:{"event":"R EGISTER","tindex":"43210","tlabel":"639090568","PhoneNumber":"11160","Dev T 198.24.63.45:48905 <http://198.24.63.45:48905> -> 198.24.63.39:3927 <http://198.24.63.39:3927> [AP] iceId":"abcd1234abcd1234","CallId":"10942-15480@198.24.63.39 <mailto:10942-15480@198.24.63.39>"},146:{"event":"REGISTER","tindex":"4559","tlabel":"1701553747","PhoneNumber":"11159","DeviceId":"abcd1234abcd1234","CallId":"10941-15480@198.24.63.39 <mailto:10941-15480@198.24.63.39>"}, T 198.24.63.45:48905 <http://198.24.63.45:48905> -> 198.24.63.39:3927 <http://198.24.63.39:3927> [AP] 146:{"event":"REGISTER","tindex":"52171","tlabel":"748030241","PhoneNumber":"11161","DeviceId":"abcd1234abcd1234","CallId":"10943-15480@198.24.63.39 <mailto:10943-15480@198.24.63.39>"}, On Tue, Sep 29, 2015 at 4:43 PM Jayesh Nambiar <jayesh1017@gmail.com <mailto:jayesh1017@gmail.com>> wrote: Hi Daniel: I did a new test where I stopped as soon as I saw one message that got discarded. Basically the following message got discarded: {"event":"REGISTER","tindex":"52171","tlabel":"748030241","PhoneNumber":"11161","DeviceId":"abcd1234abcd1234","CallId":"10943-15480@198.24.63.39 <mailto:10943-15480@198.24.63.39>"} The CallId parameter in the above json is auto incremented, so I tried gathering logs for messages with following CallIds: "CallId":"10940-15480@198.24.63.39 <mailto:10940-15480@198.24.63.39>" "CallId":"10941-15480@198.24.63.39 <mailto:10941-15480@198.24.63.39>" "CallId":"10942-15480@198.24.63.39 <mailto:10942-15480@198.24.63.39>" "CallId":"10943-15480@198.24.63.39 <mailto:10943-15480@198.24.63.39>" "CallId":"10944-15480@198.24.63.39 <mailto:10944-15480@198.24.63.39>" "CallId":"10945-15480@198.24.63.39 <mailto:10945-15480@198.24.63.39>" And the message with CallID "CallId":"10943-15480@198.24.63.39 <http://198.24.63.39>" got discarded. Here's the bunch of kamailio logs printed by the evapi module: *Logs for processed messages just before the discarded message:* DEBUG: evapi [evapi_dispatch.c:492]: evapi_recv_notify(): received [0x7f43da494ad0] [146:{"event":"REGISTER","tindex":"61133","tlabel":"450961398","PhoneNumber":"11158","DeviceId":"abcd1234abcd1234","CallId":"10940-15480@198.24.63.39 <mailto:10940-15480@198.24.63.39>"},] (151) NOTICE: evapi_recv_client(): {0} [198.24.63.45:48905 <http://198.24.63.45:48905>] - received [147:{"event":"REGISTER","tindex":"25291","tlabel":"2047024302","PhoneNumber":"11132","DeviceId":"abcd1234abcd1234","CallId":"10914-15480@198.24.63.39 <mailto:10914-15480@198.24.63.39>"},146:{"event":"REGISTER","tindex":"61136","tlabel":"839735223","PhoneNumber":"11148","DeviceId":"abcd1234abcd1234","CallId":"10930-15480@198.24.63.39 <mailto:10930-15480@198.24.63.39>"},143:{"event":"REGISTER","tindex":"4562","tlabel":"6586867","PhoneNumber":"11149","DeviceId":"abcd1234abcd1234","CallId":"10931-15480@198.24.63.39 <mailto:10931-15480@198.24.63.39>"},146:{"event":"REGISTER","tindex":"43213","tlabel":"220429340","PhoneNumber":"11151","DeviceId":"abcd1234abcd1234","CallId":"10932-15480@198.24.63.39 <mailto:10932-15480@198.24.63.39>"},146:{"event":"REGISTER","tindex":"52174","tlabel":"708781916","PhoneNumber":"11150","DeviceId":"abcd1234abcd1234","CallId":"10933-15480@198.24.63.39 <mailto:10933-15480@198.24.63.39>"},147:{"event":"REGISTER","tindex":"34251","tlabel":"1354642017","PhoneNumber":"11153","DeviceId":"abcd1234abcd1234","CallId":"10935-15480@198.24.63.39 <mailto:10935-15480@198.24.63.39>"},146:{"event":"REGISTER","tindex":"7366","tlabel":"1368396808","PhoneNumber":"11154","DeviceId":"abcd1234abcd1234","CallId":"10936-15480@198.24.63.39 <mailto:10936-15480@198.24.63.39>"},146:{"event":"REGISTER","tindex":"25289","tlabel":"119788881","PhoneNumber":"11152","DeviceId":"abcd1234abcd1234","CallId":"10934-15480@198.24.63.39 <mailto:10934-15480@198.24.63.39>"},146:{"event":"REGISTER","tindex":"16327","tlabel":"971937029","PhoneNumber":"11155","DeviceId":"abcd1234abcd1234","CallId":"10937-15480@198.24.63.39 <mailto:10937-15480@198.24.63.39>"},146:{"event":"REGISTER","tindex":"22242","tlabel":"609737074","PhoneNumber":"11156","DeviceId":"abcd1234abcd1234","CallId":"10938-15480@198.24.63.39 <mailto:10938-15480@198.24.63.39>"},146:{"event":"REGISTER","tindex":"10720","tlabel":"930036340","PhoneNumber":"11157","DeviceId":"abcd1234abcd1234","CallId":"10939-15480@198.24.63.39 <mailto:10939-15480@198.24.63.39>"},146:{"event":"REGISTER","tindex":"61133","tlabel":"450961398","PhoneNumber":"11158","DeviceId":"abcd1234abcd1234","CallId":"10940-15480@198.24.63.39 <mailto:10940-15480@198.24.63.39>"},146:{"event":"REGISTER","tindex":"43210","tlabel":"639090568","PhoneNumber":"11160","Dev] (1899) (0) evapi_recv_client(): executing event route for frame: [{"event":"REGISTER","tindex":"61133","tlabel":"450961398","PhoneNumber":"11158","DeviceId":"abcd1234abcd1234","CallId":"10940-15480@198.24.63.39 <mailto:10940-15480@198.24.63.39>"}] (146) Sep 29 05:35:14 v39 /usr/local/kamailio-dev/sbin/kamailio[15387]: DEBUG: evapi [evapi_dispatch.c:492]: evapi_recv_notify(): received [0x7f43da4b67e0] [146:{"event":"REGISTER","tindex":"4559","tlabel":"1701553747","PhoneNumber":"11159","DeviceId":"abcd1234abcd1234","CallId":"10941-15480@198.24.63.39 <mailto:10941-15480@198.24.63.39>"},] (151) NOTICE: evapi [evapi_dispatch.c:290]: evapi_recv_client(): {0} [198.24.63.45:48905 <http://198.24.63.45:48905>] - received [iceId":"abcd1234abcd1234","CallId":"10942-15480@198.24.63.39 <mailto:10942-15480@198.24.63.39>"},146:{"event":"REGISTER","tindex":"4559","tlabel":"1701553747","PhoneNumber":"11159","DeviceId":"abcd1234abcd1234","CallId":"10941-15480@198.24.63.39 <mailto:10941-15480@198.24.63.39>"},] (214) (88) DEBUG: evapi [evapi_dispatch.c:370]: evapi_recv_client(): executing event route for frame: [{"event":"REGISTER","tindex":"4559","tlabel":"1701553747","PhoneNumber":"11159","DeviceId":"abcd1234abcd1234","CallId":"10941-15480@198.24.63.39 <mailto:10941-15480@198.24.63.39>"}] (146) DEBUG: evapi [evapi_dispatch.c:492]: evapi_recv_notify(): received [0x7f43d93c2a70] [146:{"event":"REGISTER","tindex":"43210","tlabel":"639090568","PhoneNumber":"11160","DeviceId":"abcd1234abcd1234","CallId":"10942-15480@198.24.63.39 <mailto:10942-15480@198.24.63.39>"},] (151) DEBUG: evapi [evapi_dispatch.c:370]: evapi_recv_client(): executing event route for frame: [{"event":"REGISTER","tindex":"43210","tlabel":"639090568","PhoneNumber":"11160","DeviceId":"abcd1234abcd1234","CallId":"10942-15480@198.24.63.39 <mailto:10942-15480@198.24.63.39>"}] (146) DEBUG: evapi [evapi_dispatch.c:353]: evapi_recv_client(): residual data [146:{"event":"REGISTER","tindex":"43210","tlabel":"639090568","PhoneNumber":"11160","Dev] (88) DEBUG: evapi [evapi_dispatch.c:361]: evapi_recv_client(): frame size mismatch the ending char ("): [{"event":"REGISTER","tindex":"43210","tlabel":"639090568","PhoneNumber":"11160","Dev146:{"event":"REGISTER","tindex":"52171","tlabel":"748030241",] (146) *Logs for discarded message:* DEBUG: evapi [evapi_dispatch.c:492]: evapi_recv_notify(): received [0x7f43da3e69a0] [146:{"event":"REGISTER","tindex":"52171","tlabel":"748030241","PhoneNumber":"11161","DeviceId":"abcd1234abcd1234","CallId":"10943-15480@198.24.63.39 <mailto:10943-15480@198.24.63.39>"},] (151) NOTICE: evapi [evapi_dispatch.c:290]: evapi_recv_client(): {0} [198.24.63.45:48905 <http://198.24.63.45:48905>] - received [146:{"event":"REGISTER","tindex":"52171","tlabel":"748030241","PhoneNumber":"11161","DeviceId":"abcd1234abcd1234","CallId":"10943-15480@198.24.63.39 <mailto:10943-15480@198.24.63.39>"},] (151) (88) *Logs for messages after the discarded message:* DEBUG: evapi [evapi_dispatch.c:492]: evapi_recv_notify(): received [0x7f43da3e6aa0] [146:{"event":"REGISTER","tindex":"25286","tlabel":"809034258","PhoneNumber":"11163","DeviceId":"abcd1234abcd1234","CallId":"10944-15480@198.24.63.39 <mailto:10944-15480@198.24.63.39>"},] (151) NOTICE: evapi [evapi_dispatch.c:290]: evapi_recv_client(): {0} [198.24.63.45:48905 <http://198.24.63.45:48905>] - received [146:{"event":"REGISTER","tindex":"25286","tlabel":"809034258","PhoneNumber":"11163","DeviceId":"abcd1234abcd1234","CallId":"10944-15480@198.24.63.39 <mailto:10944-15480@198.24.63.39>"},] (151) (0) DEBUG: evapi [evapi_dispatch.c:370]: evapi_recv_client(): executing event route for frame: [{"event":"REGISTER","tindex":"25286","tlabel":"809034258","PhoneNumber":"11163","DeviceId":"abcd1234abcd1234","CallId":"10944-15480@198.24.63.39 <mailto:10944-15480@198.24.63.39>"}] (146) DEBUG: evapi [evapi_dispatch.c:492]: evapi_recv_notify(): received [0x7f43d9598778] [146:{"event":"REGISTER","tindex":"34248","tlabel":"187029249","PhoneNumber":"11162","DeviceId":"abcd1234abcd1234","CallId":"10945-15480@198.24.63.39 <mailto:10945-15480@198.24.63.39>"},] (151) NOTICE: evapi [evapi_dispatch.c:290]: evapi_recv_client(): {0} [198.24.63.45:48905 <http://198.24.63.45:48905>] - received [146:{"event":"REGISTER","tindex":"34248","tlabel":"187029249","PhoneNumber":"11162","DeviceId":"abcd1234abcd1234","CallId":"10945-15480@198.24.63.39 <mailto:10945-15480@198.24.63.39>"},] (151) (0) DEBUG: evapi [evapi_dispatch.c:370]: evapi_recv_client(): executing event route for frame: [{"event":"REGISTER","tindex":"34248","tlabel":"187029249","PhoneNumber":"11162","DeviceId":"abcd1234abcd1234","CallId":"10945-15480@198.24.63.39 <mailto:10945-15480@198.24.63.39>"}] (146) DEBUG: evapi [evapi_dispatch.c:492]: evapi_recv_notify(): received [0x7f43d9598878] [145:{"event":"REGISTER","tindex":"7363","tlabel":"949153805","PhoneNumber":"11164","DeviceId":"abcd1234abcd1234","CallId":"10946-15480@198.24.63.39 <mailto:10946-15480@198.24.63.39>"},] (150) NOTICE: evapi [evapi_dispatch.c:290]: evapi_recv_client(): {0} [198.24.63.45:48905 <http://198.24.63.45:48905>] - received [145:{"event":"REGISTER","tindex":"7363","tlabel":"949153805","PhoneNumber":"11164","DeviceId":"abcd1234abcd1234","CallId":"10946-15480@198.24.63.39 <mailto:10946-15480@198.24.63.39>"},] (150) (0) DEBUG: evapi [evapi_dispatch.c:370]: evapi_recv_client(): executing event route for frame: [{"event":"REGISTER","tindex":"7363","tlabel":"949153805","PhoneNumber":"11164","DeviceId":"abcd1234abcd1234","CallId":"10946-15480@198.24.63.39 <mailto:10946-15480@198.24.63.39>"}] (145) Just in case if you need to see the NGREP trace for the above messages, it is here: http://pastebin.com/4zTKmBJX Thanks, - Jayesh On Tue, Sep 29, 2015 at 12:16 PM Daniel-Constantin Mierla <miconda@gmail.com <mailto:miconda@gmail.com>> wrote: Hello, can you send here the log messages printed by evapi that contain "] - received [" which have: - the previous bunch of json documents that were processed ok (just before the one that was discarded) - the bunch of json documents that were discarded - the next bunch of json documents that were processed ok The logs were sent on could of emails and pastebin, so it is hard to track them -- if you have the logs, just locate the log messages with the discarded json and send it here along with the previous and next logs. Cheers, Daniel On 28/09/15 17:04, Jayesh Nambiar wrote:Thank you daniel. Yes, they all get printed by the evapi module even when they get discarded. If you once again check the paste-bin, the logs pasted as Kamailio Logs are all discarded packets but show up as received by evapi module. I believe they got discarded because a packet that came before them was partial and evapi discarded the remaining part of the message in the new packet and encountered error. - Jayesh On Mon, Sep 28, 2015 at 11:13 AM Daniel-Constantin Mierla <miconda@gmail.com <mailto:miconda@gmail.com>> wrote: Hello, I will look over it during next days. One more thing, are the packages sent by you printed by evapi module, even not they are not processed (but discarded)? Cheers, Daniel-- Daniel-Constantin Mierla http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com-- Daniel-Constantin Mierla http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat-- Daniel-Constantin Mierla http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com
-
Daniel-Constantin Mierla -
Jayesh Nambiar -
Jayesh Nambiar