9 Sep
2015
9 Sep
'15
4:01 p.m.
Hello,
I'm exploring the evapi module for my kamailio to interface with an
external node.js app for third party stuff like AAA, billing engine tasks,
notifications and so on. I followed and took some ideas from the rtjson and
evapi tutorial found here(
http://kb.asipto.com/kamailio:k43-async-sip-routing-nodejs) to build the
node.js app consuming events.
When I stress tested the scenario using SIPp and tried sending a lot of
events at 300-350cps from Kamailio, I noticed that at times the client is
receiving 2-3 events in a single message together although I do
event_sync_relay once per SIP message received and have netstrings enabled.
I believe this is a typical behavior of TCP and needs to be handled by the
client using some kind of Netstring handler. Please correct me if I'm wrong.
And hence I'd like to know what particularly needs to be taken care of
while writing a client that is listening for events on raw tcp socket and
how does kamailio handle this situation while receiving messages over TCP
socket?? Does kamailio recognize the end of netstring properly on
evapi:message-received and give exactly one message to take care of on
every "message-received" event or should that be handled in the script
somewhere !!
I also referred cgrates client over evapi example which is written in GO,
but I couldnt find them handling TCP streams clearly either.
I'd really appreciate some expert suggestion here to make an informed
decision on using the evapi module for a large scale solution.
Thanks,
- Jayesh
10 Sep
10 Sep
3:59 a.m.
Hello,
tcb is stream protocol and several messages can be queued on the pipe at
the same time. That is the reason for netstring format, to be able to
easily detect the boundaries of each message. If netstring format is
enabled and kamailio receives several messages at once, it splits them
and for each is executing the event route.
If netstring format is not used, the kamailio is executing the event
route with the entire content that was read at once from the tcp connection.
Cheers,
Daniel
On 09/09/15 22:01, Jayesh Nambiar wrote:
Hello,
I'm exploring the evapi module for my kamailio to interface with an
external node.js app for third party stuff like AAA, billing engine
tasks, notifications and so on. I followed and took some ideas from
the rtjson and evapi tutorial found
here(http://kb.asipto.com/kamailio:k43-async-sip-routing-nodejs) to
build the node.js app consuming events.
When I stress tested the scenario using SIPp and tried sending a lot
of events at 300-350cps from Kamailio, I noticed that at times the
client is receiving 2-3 events in a single message together although I
do event_sync_relay once per SIP message received and have netstrings
enabled. I believe this is a typical behavior of TCP and needs to be
handled by the client using some kind of Netstring handler. Please
correct me if I'm wrong.
And hence I'd like to know what particularly needs to be taken care of
while writing a client that is listening for events on raw tcp socket
and how does kamailio handle this situation while receiving messages
over TCP socket?? Does kamailio recognize the end of netstring
properly on evapi:message-received and give exactly one message to
take care of on every "message-received" event or should that be
handled in the script somewhere !!
I also referred cgrates client over evapi example which is written in
GO, but I couldnt find them handling TCP streams clearly either.
I'd really appreciate some expert suggestion here to make an informed
decision on using the evapi module for a large scale solution.
Thanks,
- Jayesh
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users(a)lists.sip-router.org
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Book: SIP Routing With Kamailio - http://www.asipto.com
Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
6:31 a.m.
Hi Daniel,
Thanks for the quick response. So if I do not use Netstrings, does Kamailio
allow me to create a custom logic in the script. For eg. if I decide to use
newline as a delimiter, can I keep buffering the message until I encounter
the delimiter from the event route and then execute whatever I have to
within the script??
Thanks,
- Jayesh
On Thu, Sep 10, 2015 at 1:29 PM, Daniel-Constantin Mierla <miconda(a)gmail.com
wrote:
Hello,
tcb is stream protocol and several messages can be queued on the pipe at
the same time. That is the reason for netstring format, to be able to
easily detect the boundaries of each message. If netstring format is
enabled and kamailio receives several messages at once, it splits them and
for each is executing the event route.
If netstring format is not used, the kamailio is executing the event route
with the entire content that was read at once from the tcp connection.
Cheers,
Daniel
On 09/09/15 22:01, Jayesh Nambiar wrote:
Hello,
I'm exploring the evapi module for my kamailio to interface with an
external node.js app for third party stuff like AAA, billing engine tasks,
notifications and so on. I followed and took some ideas from the rtjson and
evapi tutorial found here(
<http://kb.asipto.com/kamailio:k43-async-sip-routing-nodejs>
http://kb.asipto.com/kamailio:k43-async-sip-routing-nodejs) to build the
node.js app consuming events.
When I stress tested the scenario using SIPp and tried sending a lot of
events at 300-350cps from Kamailio, I noticed that at times the client is
receiving 2-3 events in a single message together although I do
event_sync_relay once per SIP message received and have netstrings enabled.
I believe this is a typical behavior of TCP and needs to be handled by the
client using some kind of Netstring handler. Please correct me if I'm wrong.
And hence I'd like to know what particularly needs to be taken care of
while writing a client that is listening for events on raw tcp socket and
how does kamailio handle this situation while receiving messages over TCP
socket?? Does kamailio recognize the end of netstring properly on
evapi:message-received and give exactly one message to take care of on
every "message-received" event or should that be handled in the script
somewhere !!
I also referred cgrates client over evapi example which is written in GO,
but I couldnt find them handling TCP streams clearly either.
I'd really appreciate some expert suggestion here to make an informed
decision on using the evapi module for a large scale solution.
Thanks,
- Jayesh
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing
listsr-users@lists.sip-router.orghttp://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
--
Daniel-Constantin Mierlahttp://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda
Book: SIP Routing With Kamailio - http://www.asipto.com
Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users(a)lists.sip-router.org
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
14 Sep
14 Sep
5 p.m.
Hello Daniel,
After further testing with evapi module, I figured that when Netstrings are
used, an event route is invoked individually for each message even if if
multiple netstring messages are received in a single TCP packet. But this
doesn't work effectively when a single proper message is split-up in two
packets. For Example, if a message arrives as:
12:Hello World!, 12:Hello World!, 12:Hello World! in a single packet,
kamailio properly invokes the event route "evapi:message-received" thrice
for every individual proper netstring message.
But if the first packet contains:
12:Hello World!, 12:Hello
And Second Packet contains:
World!, 12:Hello World!
the event route is invoked only once !!
The above pattern is very much possible while sending and receiving packets
over TCP Socket. Our tests for receiving an approximately 150 byte message
over evapi socket at the rate of roughly 1000cps causes a lot of real
events to be missed because of the above problem. You can never be sure
when TCP will split messages in different chunks.
This definitely looks like a bug which makes it not very reliable at large
scale deployments. Would really appreciate your inputs on this. Thanks;
- Jayesh
On Thu, Sep 10, 2015 at 4:01 PM, Jayesh Nambiar <jayesh1017(a)gmail.com>
wrote:
Hi Daniel,
Thanks for the quick response. So if I do not use Netstrings, does
Kamailio allow me to create a custom logic in the script. For eg. if I
decide to use newline as a delimiter, can I keep buffering the message
until I encounter the delimiter from the event route and then execute
whatever I have to within the script??
Thanks,
- Jayesh
On Thu, Sep 10, 2015 at 1:29 PM, Daniel-Constantin Mierla <
miconda(a)gmail.com> wrote:
Hello,
tcb is stream protocol and several messages can be queued on the pipe at
the same time. That is the reason for netstring format, to be able to
easily detect the boundaries of each message. If netstring format is
enabled and kamailio receives several messages at once, it splits them and
for each is executing the event route.
If netstring format is not used, the kamailio is executing the event
route with the entire content that was read at once from the tcp connection.
Cheers,
Daniel
On 09/09/15 22:01, Jayesh Nambiar wrote:
Hello,
I'm exploring the evapi module for my kamailio to interface with an
external node.js app for third party stuff like AAA, billing engine tasks,
notifications and so on. I followed and took some ideas from the rtjson and
evapi tutorial found here(
<http://kb.asipto.com/kamailio:k43-async-sip-routing-nodejs>
http://kb.asipto.com/kamailio:k43-async-sip-routing-nodejs) to build the
node.js app consuming events.
When I stress tested the scenario using SIPp and tried sending a lot of
events at 300-350cps from Kamailio, I noticed that at times the client is
receiving 2-3 events in a single message together although I do
event_sync_relay once per SIP message received and have netstrings enabled.
I believe this is a typical behavior of TCP and needs to be handled by the
client using some kind of Netstring handler. Please correct me if I'm wrong.
And hence I'd like to know what particularly needs to be taken care of
while writing a client that is listening for events on raw tcp socket and
how does kamailio handle this situation while receiving messages over TCP
socket?? Does kamailio recognize the end of netstring properly on
evapi:message-received and give exactly one message to take care of on
every "message-received" event or should that be handled in the script
somewhere !!
I also referred cgrates client over evapi example which is written in GO,
but I couldnt find them handling TCP streams clearly either.
I'd really appreciate some expert suggestion here to make an informed
decision on using the evapi module for a large scale solution.
Thanks,
- Jayesh
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing
listsr-users@lists.sip-router.orghttp://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
--
Daniel-Constantin Mierlahttp://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda
Book: SIP Routing With Kamailio - http://www.asipto.com
Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users(a)lists.sip-router.org
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
15 Sep
15 Sep
7:09 a.m.
Do let me know if I should run some specific tests for you all to get a
little more clarity and information and I'd be happy to do those test and
update you with results. I eagerly want to take a positive decision on
going ahead with evapi module as it looks very promising. Only if we can
solve these minor issues.
Thanks,
- Jayesh
On Tue, Sep 15, 2015 at 2:30 AM, Jayesh Nambiar <jayesh1017(a)gmail.com>
wrote:
Hello Daniel,
After further testing with evapi module, I figured that when Netstrings
are used, an event route is invoked individually for each message even if
if multiple netstring messages are received in a single TCP packet. But
this doesn't work effectively when a single proper message is split-up in
two packets. For Example, if a message arrives as:
12:Hello World!, 12:Hello World!, 12:Hello World! in a single packet,
kamailio properly invokes the event route "evapi:message-received" thrice
for every individual proper netstring message.
But if the first packet contains:
12:Hello World!, 12:Hello
And Second Packet contains:
World!, 12:Hello World!
the event route is invoked only once !!
The above pattern is very much possible while sending and receiving
packets over TCP Socket. Our tests for receiving an approximately 150 byte
message over evapi socket at the rate of roughly 1000cps causes a lot of
real events to be missed because of the above problem. You can never be
sure when TCP will split messages in different chunks.
This definitely looks like a bug which makes it not very reliable at large
scale deployments. Would really appreciate your inputs on this. Thanks;
- Jayesh
On Thu, Sep 10, 2015 at 4:01 PM, Jayesh Nambiar <jayesh1017(a)gmail.com>
wrote:
Hi Daniel,
Thanks for the quick response. So if I do not use Netstrings, does
Kamailio allow me to create a custom logic in the script. For eg. if I
decide to use newline as a delimiter, can I keep buffering the message
until I encounter the delimiter from the event route and then execute
whatever I have to within the script??
Thanks,
- Jayesh
On Thu, Sep 10, 2015 at 1:29 PM, Daniel-Constantin Mierla <
miconda(a)gmail.com> wrote:
Hello,
tcb is stream protocol and several messages can be queued on the pipe at
the same time. That is the reason for netstring format, to be able to
easily detect the boundaries of each message. If netstring format is
enabled and kamailio receives several messages at once, it splits them and
for each is executing the event route.
If netstring format is not used, the kamailio is executing the event
route with the entire content that was read at once from the tcp connection.
Cheers,
Daniel
On 09/09/15 22:01, Jayesh Nambiar wrote:
Hello,
I'm exploring the evapi module for my kamailio to interface with an
external node.js app for third party stuff like AAA, billing engine tasks,
notifications and so on. I followed and took some ideas from the rtjson and
evapi tutorial found here(
<http://kb.asipto.com/kamailio:k43-async-sip-routing-nodejs>
http://kb.asipto.com/kamailio:k43-async-sip-routing-nodejs) to build
the node.js app consuming events.
When I stress tested the scenario using SIPp and tried sending a lot of
events at 300-350cps from Kamailio, I noticed that at times the client is
receiving 2-3 events in a single message together although I do
event_sync_relay once per SIP message received and have netstrings enabled.
I believe this is a typical behavior of TCP and needs to be handled by the
client using some kind of Netstring handler. Please correct me if I'm wrong.
And hence I'd like to know what particularly needs to be taken care of
while writing a client that is listening for events on raw tcp socket and
how does kamailio handle this situation while receiving messages over TCP
socket?? Does kamailio recognize the end of netstring properly on
evapi:message-received and give exactly one message to take care of on
every "message-received" event or should that be handled in the script
somewhere !!
I also referred cgrates client over evapi example which is written in
GO, but I couldnt find them handling TCP streams clearly either.
I'd really appreciate some expert suggestion here to make an informed
decision on using the evapi module for a large scale solution.
Thanks,
- Jayesh
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing
listsr-users@lists.sip-router.orghttp://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
--
Daniel-Constantin Mierlahttp://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda
Book: SIP Routing With Kamailio - http://www.asipto.com
Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users(a)lists.sip-router.org
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
8:52 a.m.
Hello,
I will look if there are options in libev to buffer data or try to
implement a buffering mechanism locally for such cases.
Cheers,
Daniel
On 14/09/15 23:00, Jayesh Nambiar wrote:
Hello Daniel,
After further testing with evapi module, I figured that when
Netstrings are used, an event route is invoked individually for each
message even if if multiple netstring messages are received in a
single TCP packet. But this doesn't work effectively when a single
proper message is split-up in two packets. For Example, if a message
arrives as:
12:Hello World!, 12:Hello World!, 12:Hello World! in a single packet,
kamailio properly invokes the event route "evapi:message-received"
thrice for every individual proper netstring message.
But if the first packet contains:
12:Hello World!, 12:Hello
And Second Packet contains:
World!, 12:Hello World!
the event route is invoked only once !!
The above pattern is very much possible while sending and receiving
packets over TCP Socket. Our tests for receiving an approximately 150
byte message over evapi socket at the rate of roughly 1000cps causes a
lot of real events to be missed because of the above problem. You can
never be sure when TCP will split messages in different chunks.
This definitely looks like a bug which makes it not very reliable at
large scale deployments. Would really appreciate your inputs on this.
Thanks;
- Jayesh
On Thu, Sep 10, 2015 at 4:01 PM, Jayesh Nambiar <jayesh1017(a)gmail.com
<mailto:jayesh1017@gmail.com>> wrote:
Hi Daniel,
Thanks for the quick response. So if I do not use Netstrings, does
Kamailio allow me to create a custom logic in the script. For eg.
if I decide to use newline as a delimiter, can I keep buffering
the message until I encounter the delimiter from the event route
and then execute whatever I have to within the script??
Thanks,
- Jayesh
On Thu, Sep 10, 2015 at 1:29 PM, Daniel-Constantin Mierla
<miconda(a)gmail.com <mailto:miconda@gmail.com>> wrote:
Hello,
tcb is stream protocol and several messages can be queued on
the pipe at the same time. That is the reason for netstring
format, to be able to easily detect the boundaries of each
message. If netstring format is enabled and kamailio receives
several messages at once, it splits them and for each is
executing the event route.
If netstring format is not used, the kamailio is executing the
event route with the entire content that was read at once from
the tcp connection.
Cheers,
Daniel
On 09/09/15 22:01, Jayesh Nambiar wrote:
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Book: SIP Routing With Kamailio - http://www.asipto.com
Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
Hello,
I'm exploring the evapi module for my kamailio to interface
with an external node.js app for third party stuff like AAA,
billing engine tasks, notifications and so on. I followed and
took some ideas from the rtjson and evapi tutorial found
here(http://kb.asipto.com/kamailio:k43-async-sip-routing-nodejs)
to build the node.js app consuming events.
When I stress tested the scenario using SIPp and tried
sending a lot of events at 300-350cps from Kamailio, I
noticed that at times the client is receiving 2-3 events in a
single message together although I do event_sync_relay once
per SIP message received and have netstrings enabled. I
believe this is a typical behavior of TCP and needs to be
handled by the client using some kind of Netstring handler.
Please correct me if I'm wrong.
And hence I'd like to know what particularly needs to be
taken care of while writing a client that is listening for
events on raw tcp socket and how does kamailio handle this
situation while receiving messages over TCP socket?? Does
kamailio recognize the end of netstring properly on
evapi:message-received and give exactly one message to take
care of on every "message-received" event or should that be
handled in the script somewhere !!
I also referred cgrates client over evapi example which is
written in GO, but I couldnt find them handling TCP streams
clearly either.
I'd really appreciate some expert suggestion here to make an
informed decision on using the evapi module for a large scale
solution.
Thanks,
- Jayesh
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users(a)lists.sip-router.org
<mailto:sr-users@lists.sip-router.org>
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> -
http://www.linkedin.com/in/miconda
Book: SIP Routing With Kamailio - http://www.asipto.com
Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users
mailing list
sr-users(a)lists.sip-router.org
<mailto:sr-users@lists.sip-router.org>
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
18 Sep
18 Sep
4:11 a.m.
Hello,
I just pushed a patch to master branch that should cope with partial
data received on tcp connection. No time to test at all, therefore any
feedback will be appreciated.
Cheers,
Daniel
On 15/09/15 14:52, Daniel-Constantin Mierla wrote:
Hello,
I will look if there are options in libev to buffer data or try to
implement a buffering mechanism locally for such cases.
Cheers,
Daniel
On 14/09/15 23:00, Jayesh Nambiar wrote:
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Book: SIP Routing With Kamailio - http://www.asipto.com
Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
Hello Daniel,
After further testing with evapi module, I figured that when
Netstrings are used, an event route is invoked individually for each
message even if if multiple netstring messages are received in a
single TCP packet. But this doesn't work effectively when a single
proper message is split-up in two packets. For Example, if a message
arrives as:
12:Hello World!, 12:Hello World!, 12:Hello World! in a single packet,
kamailio properly invokes the event route "evapi:message-received"
thrice for every individual proper netstring message.
But if the first packet contains:
12:Hello World!, 12:Hello
And Second Packet contains:
World!, 12:Hello World!
the event route is invoked only once !!
The above pattern is very much possible while sending and receiving
packets over TCP Socket. Our tests for receiving an approximately 150
byte message over evapi socket at the rate of roughly 1000cps causes
a lot of real events to be missed because of the above problem. You
can never be sure when TCP will split messages in different chunks.
This definitely looks like a bug which makes it not very reliable at
large scale deployments. Would really appreciate your inputs on this.
Thanks;
- Jayesh
On Thu, Sep 10, 2015 at 4:01 PM, Jayesh Nambiar <jayesh1017(a)gmail.com
<mailto:jayesh1017@gmail.com>> wrote:
Hi Daniel,
Thanks for the quick response. So if I do not use Netstrings,
does Kamailio allow me to create a custom logic in the script.
For eg. if I decide to use newline as a delimiter, can I keep
buffering the message until I encounter the delimiter from the
event route and then execute whatever I have to within the script??
Thanks,
- Jayesh
On Thu, Sep 10, 2015 at 1:29 PM, Daniel-Constantin Mierla
<miconda(a)gmail.com> wrote:
Hello,
tcb is stream protocol and several messages can be queued on
the pipe at the same time. That is the reason for netstring
format, to be able to easily detect the boundaries of each
message. If netstring format is enabled and kamailio receives
several messages at once, it splits them and for each is
executing the event route.
If netstring format is not used, the kamailio is executing
the event route with the entire content that was read at once
from the tcp connection.
Cheers,
Daniel
On 09/09/15 22:01, Jayesh Nambiar wrote:
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Book: SIP Routing With Kamailio - http://www.asipto.com
Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat Hello,
I'm exploring the evapi module for my kamailio to interface
with an external node.js app for third party stuff like AAA,
billing engine tasks, notifications and so on. I followed
and took some ideas from the rtjson and evapi tutorial found
here(http://kb.asipto.com/kamailio:k43-async-sip-routing-nodejs)
to build the node.js app consuming events.
When I stress tested the scenario using SIPp and tried
sending a lot of events at 300-350cps from Kamailio, I
noticed that at times the client is receiving 2-3 events in
a single message together although I do event_sync_relay
once per SIP message received and have netstrings enabled. I
believe this is a typical behavior of TCP and needs to be
handled by the client using some kind of Netstring handler.
Please correct me if I'm wrong.
And hence I'd like to know what particularly needs to be
taken care of while writing a client that is listening for
events on raw tcp socket and how does kamailio handle this
situation while receiving messages over TCP socket?? Does
kamailio recognize the end of netstring properly on
evapi:message-received and give exactly one message to take
care of on every "message-received" event or should that be
handled in the script somewhere !!
I also referred cgrates client over evapi example which is
written in GO, but I couldnt find them handling TCP streams
clearly either.
I'd really appreciate some expert suggestion here to make an
informed decision on using the evapi module for a large
scale solution.
Thanks,
- Jayesh
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users(a)lists.sip-router.org
<mailto:sr-users@lists.sip-router.org>
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> -
http://www.linkedin.com/in/miconda
Book: SIP Routing With Kamailio - http://www.asipto.com
Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users
mailing list
sr-users(a)lists.sip-router.org
<mailto:sr-users@lists.sip-router.org>
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
5:25 a.m.
Will have this tested by tomorrow and will get back to you. Thanks.
- Jayesh
On Fri, Sep 18, 2015 at 1:41 PM, Daniel-Constantin Mierla <miconda(a)gmail.com
wrote:
> Hello,
>
> I just pushed a patch to master branch that should cope with partial data
> received on tcp connection. No time to test at all, therefore any feedback
> will be appreciated.
>
> Cheers,
> Daniel
>
>
> On 15/09/15 14:52, Daniel-Constantin Mierla wrote:
>
> Hello,
>
> I will look if there are options in libev to buffer data or try to
> implement a buffering mechanism locally for such cases.
>
> Cheers,
> Daniel
>
> On 14/09/15 23:00, Jayesh Nambiar wrote:
>
> Hello Daniel,
> After further testing with evapi module, I figured that when Netstrings
> are used, an event route is invoked individually for each message even if
> if multiple netstring messages are received in a single TCP packet. But
> this doesn't work effectively when a single proper message is split-up in
> two packets. For Example, if a message arrives as:
> 12:Hello World!, 12:Hello World!, 12:Hello World! in a single packet,
> kamailio properly invokes the event route "evapi:message-received" thrice
> for every individual proper netstring message.
> But if the first packet contains:
> 12:Hello World!, 12:Hello
> And Second Packet contains:
> World!, 12:Hello World!
> the event route is invoked only once !!
>
> The above pattern is very much possible while sending and receiving
> packets over TCP Socket. Our tests for receiving an approximately 150 byte
> message over evapi socket at the rate of roughly 1000cps causes a lot of
> real events to be missed because of the above problem. You can never be
> sure when TCP will split messages in different chunks.
> This definitely looks like a bug which makes it not very reliable at large
> scale deployments. Would really appreciate your inputs on this. Thanks;
>
> - Jayesh
>
>
>
> On Thu, Sep 10, 2015 at 4:01 PM, Jayesh Nambiar < <jayesh1017(a)gmail.com>
> jayesh1017(a)gmail.com wrote:
>
>> Hi Daniel,
>> Thanks for the quick response. So if I do not use Netstrings, does
>> Kamailio allow me to create a custom logic in the script. For eg. if I
>> decide to use newline as a delimiter, can I keep buffering the message
>> until I encounter the delimiter from the event route and then execute
>> whatever I have to within the script??
>>
>> Thanks,
>>
>> - Jayesh
>>
>> On Thu, Sep 10, 2015 at 1:29 PM, Daniel-Constantin Mierla <
>> <miconda@gmail.com>miconda@gmail.com
wrote:
>>
>>> Hello,
>>>
>>> tcb is stream protocol and several messages can be queued on the pipe at
>>> the same time. That is the reason for netstring format, to be able to
>>> easily detect the boundaries of each message. If netstring format is
>>> enabled and kamailio receives several messages at once, it splits them and
>>> for each is executing the event route.
>>>
>>> If netstring format is not used, the kamailio is executing the event
>>> route with the entire content that was read at once from the tcp connection.
>>>
>>> Cheers,
>>> Daniel
>>>
>>>
>>> On 09/09/15 22:01, Jayesh Nambiar wrote:
>>>
>>> Hello,
>>> I'm exploring the evapi module for my kamailio to interface with an
>>> external node.js app for third party stuff like AAA, billing engine tasks,
>>> notifications and so on. I followed and took some ideas from the rtjson and
>>> evapi tutorial found here(
>>> http://kb.asipto.com/kamailio:k43-async-sip-routing-nodejs) to build
>>> the node.js app consuming events.
>>> When I stress tested the scenario using SIPp and tried sending a lot of
>>> events at 300-350cps from Kamailio, I noticed that at times the client is
>>> receiving 2-3 events in a single message together although I do
>>> event_sync_relay once per SIP message received and have netstrings enabled.
>>> I believe this is a typical behavior of TCP and needs to be handled by the
>>> client using some kind of Netstring handler. Please correct me if I'm
wrong.
>>> And hence I'd like to know what particularly needs to be taken care of
>>> while writing a client that is listening for events on raw tcp socket and
>>> how does kamailio handle this situation while receiving messages over TCP
>>> socket?? Does kamailio recognize the end of netstring properly on
>>> evapi:message-received and give exactly one message to take care of on
>>> every "message-received" event or should that be handled in the
script
>>> somewhere !!
>>> I also referred cgrates client over evapi example which is written in
>>> GO, but I couldnt find them handling TCP streams clearly either.
>>> I'd really appreciate some expert suggestion here to make an informed
>>> decision on using the evapi module for a large scale solution.
>>>
>>> Thanks,
>>>
>>> - Jayesh
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing
listsr-users@lists.sip-router.orghttp://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
>>>
>>>
>>> --
>>> Daniel-Constantin Mierlahttp://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda
>>> Book: SIP Routing With Kamailio - http://www.asipto.com
>>> Kamailio Advanced Training, Sep 28-30, 2015, in Berlin -
http://asipto.com/u/kat
>>>
>>>
>>> _______________________________________________
>>> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
>>> sr-users(a)lists.sip-router.org
>>> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
>>>
>>>
>>
>
> --
> Daniel-Constantin Mierlahttp://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda
> Book: SIP Routing With Kamailio - http://www.asipto.com
> Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
>
>
> --
> Daniel-Constantin Mierlahttp://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda
> Book: SIP Routing With Kamailio - http://www.asipto.com
> Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
>
>
5:19 p.m.
Here are the tests that I did:
With the patch applied, I see that Kamailio is invoking event individually
for each netstring even when they come in different chunks. But I did see
instances where when there were complete netstrings in a single chunk;
kamailio did not raise an event for them. Here's the illustration:
T 1.2.3.4:48873 -> 5.6.7.8:3927 [AP]
146:{"event":"REGISTER","tindex":"16916","tlabel":"1753048277","PhoneNumber":"20708","DeviceId":"abcd1234abcd1234","CallId":"
21220-3848(a)5.6.7.8
"},144:{"event":"REGISTER","tindex":"7954","tlabel":"254315075","PhoneNumber":"20709","DeviceId":"abcd1234abcd1234","CallId":"
21223-3848(a)5.6.7.8
"},145:{"event":"REGISTER","tindex":"64529","tlabel":"599481568","PhoneNumber":"20709","DeviceId":"abcd1234abcd1234","CallId":"
21222-3848@5.6.7.8"},145:{"event":"REGISTER","tindex":"46605","tlabel":"112015324","PhoneNumber":"20710","DeviceId"
T 1.2.3.4:48873 -> 5.6.7.8:3927 [AP]
:"abcd1234abcd1234","CallId":"21224-3848@5.6.7.8"},
*The above two chunks contain 4 proper netstrings where the second chunk
contains part of the 4th netstring. In this case Evapi properly raised 4
individual events.*
1.2.3.4:48873 -> 5.6.7.8:3927 [AP]
145:{"event":"REGISTER","tindex":"55567","tlabel":"627458699","PhoneNumber":"20711","DeviceId":"abcd1234abcd1234","CallId":"
21225-3848(a)5.6.7.8
"},143:{"event":"REGISTER","tindex":"28682","tlabel":"9676691","PhoneNumber":"20712","DeviceId":"abcd1234abcd1234","CallId":"
21226-3848(a)5.6.7.8"}uot;},
*This above chunk contains two complete netstrings but Kamailio never
raised events for these two netstrings.*
For the events that were not raised I see proper Kamailio Logs which is:
evapi_recv_client(): {0} [1.2.3.4:48873] - received
[145:{"event":"REGISTER","tindex":"55567","tlabel":"627458699","PhoneNumber":"20711","DeviceId":"abcd1234abcd1234","CallId":"
21225-3848(a)5.6.7.8
"},143:{"event":"REGISTER","tindex":"28682","tlabel":"9676691","PhoneNumber":"20712","DeviceId":"abcd1234abcd1234","CallId":"
21226-3848(a)5.6.7.8"}uot;},]
But I don't see the logs that I've written in the script after the event
was raised, which means Kamailio did not invoke events for these two
netstrings.
So in my tests, out of 27232 messages sent, there were approximately 27
messages for which the events were not raised by Evapi. The rate of
messages started at 500cps and I stopped after I saw missed events at
around 1200cps.
Thanks, and do let me know for any further tests or information required
about the same.
- Jayesh
On Fri, Sep 18, 2015 at 2:55 PM, Jayesh Nambiar <jayesh1017(a)gmail.com>
wrote:
Will have this tested by tomorrow and will get back to
you. Thanks.
- Jayesh
On Fri, Sep 18, 2015 at 1:41 PM, Daniel-Constantin Mierla <
miconda(a)gmail.com> wrote:
Hello,
I just pushed a patch to master branch that should cope with partial data
received on tcp connection. No time to test at all, therefore any feedback
will be appreciated.
Cheers,
Daniel
On 15/09/15 14:52, Daniel-Constantin Mierla wrote:
Hello,
I will look if there are options in libev to buffer data or try to
implement a buffering mechanism locally for such cases.
Cheers,
Daniel
On 14/09/15 23:00, Jayesh Nambiar wrote:
Hello Daniel,
After further testing with evapi module, I figured that when Netstrings
are used, an event route is invoked individually for each message even if
if multiple netstring messages are received in a single TCP packet. But
this doesn't work effectively when a single proper message is split-up in
two packets. For Example, if a message arrives as:
12:Hello World!, 12:Hello World!, 12:Hello World! in a single packet,
kamailio properly invokes the event route "evapi:message-received" thrice
for every individual proper netstring message.
But if the first packet contains:
12:Hello World!, 12:Hello
And Second Packet contains:
World!, 12:Hello World!
the event route is invoked only once !!
The above pattern is very much possible while sending and receiving
packets over TCP Socket. Our tests for receiving an approximately 150 byte
message over evapi socket at the rate of roughly 1000cps causes a lot of
real events to be missed because of the above problem. You can never be
sure when TCP will split messages in different chunks.
This definitely looks like a bug which makes it not very reliable at
large scale deployments. Would really appreciate your inputs on this.
Thanks;
- Jayesh
On Thu, Sep 10, 2015 at 4:01 PM, Jayesh Nambiar < <jayesh1017(a)gmail.com>
jayesh1017(a)gmail.com> wrote:
Hi Daniel,
Thanks for the quick response. So if I do not use Netstrings, does
Kamailio allow me to create a custom logic in the script. For eg. if I
decide to use newline as a delimiter, can I keep buffering the message
until I encounter the delimiter from the event route and then execute
whatever I have to within the script??
Thanks,
- Jayesh
On Thu, Sep 10, 2015 at 1:29 PM, Daniel-Constantin Mierla <
<miconda@gmail.com>miconda@gmail.com> wrote:
--
Daniel-Constantin Mierlahttp://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda
Book: SIP Routing With Kamailio - http://www.asipto.com
Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
--
Daniel-Constantin Mierlahttp://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda
Book: SIP Routing With Kamailio - http://www.asipto.com
Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
Hello,
tcb is stream protocol and several messages can be queued on the pipe
at the same time. That is the reason for netstring format, to be able to
easily detect the boundaries of each message. If netstring format is
enabled and kamailio receives several messages at once, it splits them and
for each is executing the event route.
If netstring format is not used, the kamailio is executing the event
route with the entire content that was read at once from the tcp connection.
Cheers,
Daniel
On 09/09/15 22:01, Jayesh Nambiar wrote:
Hello,
I'm exploring the evapi module for my kamailio to interface with an
external node.js app for third party stuff like AAA, billing engine tasks,
notifications and so on. I followed and took some ideas from the rtjson and
evapi tutorial found here(
http://kb.asipto.com/kamailio:k43-async-sip-routing-nodejs) to build
the node.js app consuming events.
When I stress tested the scenario using SIPp and tried sending a lot of
events at 300-350cps from Kamailio, I noticed that at times the client is
receiving 2-3 events in a single message together although I do
event_sync_relay once per SIP message received and have netstrings enabled.
I believe this is a typical behavior of TCP and needs to be handled by the
client using some kind of Netstring handler. Please correct me if I'm wrong.
And hence I'd like to know what particularly needs to be taken care of
while writing a client that is listening for events on raw tcp socket and
how does kamailio handle this situation while receiving messages over TCP
socket?? Does kamailio recognize the end of netstring properly on
evapi:message-received and give exactly one message to take care of on
every "message-received" event or should that be handled in the script
somewhere !!
I also referred cgrates client over evapi example which is written in
GO, but I couldnt find them handling TCP streams clearly either.
I'd really appreciate some expert suggestion here to make an informed
decision on using the evapi module for a large scale solution.
Thanks,
- Jayesh
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing
listsr-users@lists.sip-router.orghttp://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
--
Daniel-Constantin Mierlahttp://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda
Book: SIP Routing With Kamailio - http://www.asipto.com
Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users(a)lists.sip-router.org
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
6:25 p.m.
I pushed a commit to add more debug message while processing received
data. You can use debugger module to set a higher debug level for evapi
module in order to see what happens.
I checked the netstring packet size and it is invalid (unless email
stripped some white chars there) -- for example in:
145:{"event":"REGISTER","tindex":"55567","tlabel":"627458699","PhoneNumber":"20711","DeviceId":"abcd1234abcd1234","CallId":"21225-3848@5.6.7.8"},
the size is 141, not 145 -- it looks like the size includes the size
itself plus the delimiters ':,'. The size is only the effective data, see:
https://en.wikipedia.org/wiki/Netstring
If evapi gets a packet with an invalid size, then it discards the buffer
content.
See if the app on the other side of evapi connection builds netstrings
with wrong size.
Cheers,
Daniel
On 18/09/15 23:19, Jayesh Nambiar wrote:
Here are the tests that I did:
With the patch applied, I see that Kamailio is invoking event
individually for each netstring even when they come in different
chunks. But I did see instances where when there were complete
netstrings in a single chunk; kamailio did not raise an event for
them. Here's the illustration:
T 1.2.3.4:48873 <http://1.2.3.4:48873> -> 5.6.7.8:3927
<http://5.6.7.8:3927> [AP]
146:{"event":"REGISTER","tindex":"16916","tlabel":"1753048277","PhoneNumber":"20708","DeviceId":"abcd1234abcd1234","CallId":"21220-3848@5.6.7.8
<mailto:21220-3848@5.6.7.8>"},144:{"event":"REGISTER","tindex":"7954","tlabel":"254315075","PhoneNumber":"20709","DeviceId":"abcd1234abcd1234","CallId":"21223-3848@5.6.7.8
<mailto:21223-3848@5.6.7.8>"},145:{"event":"REGISTER","tindex":"64529","tlabel":"599481568","PhoneNumber":"20709","DeviceId":"abcd1234abcd1234","CallId":"21222-3848@5.6.7.8
<mailto:21222-3848@5.6.7.8>"},145:{"event":"REGISTER","tindex":"46605","tlabel":"112015324","PhoneNumber":"20710","DeviceId"
T 1.2.3.4:48873 <http://1.2.3.4:48873> -> 5.6.7.8:3927
<http://5.6.7.8:3927> [AP]
:"abcd1234abcd1234","CallId":"21224-3848@5.6.7.8
<mailto:21224-3848@5.6.7.8>"},
*The above two chunks contain 4 proper netstrings where the second
chunk contains part of the 4th netstring. In this case Evapi properly
raised 4 individual events.*
1.2.3.4:48873 <http://1.2.3.4:48873> -> 5.6.7.8:3927
<http://5.6.7.8:3927> [AP]
145:{"event":"REGISTER","tindex":"55567","tlabel":"627458699","PhoneNumber":"20711","DeviceId":"abcd1234abcd1234","CallId":"21225-3848@5.6.7.8
<mailto:21225-3848@5.6.7.8>"},143:{"event":"REGISTER","tindex":"28682","tlabel":"9676691","PhoneNumber":"20712","DeviceId":"abcd1234abcd1234","CallId":"21226-3848@5.6.7.8
<mailto:21226-3848@5.6.7.8>"},
*This above chunk contains two complete netstrings but Kamailio never
raised events for these two netstrings.*
*
*
For the events that were not raised I see proper Kamailio Logs which is:
evapi_recv_client(): {0} [1.2.3.4:48873 <http://1.2.3.4:48873>] -
received
[145:{"event":"REGISTER","tindex":"55567","tlabel":"627458699","PhoneNumber":"20711","DeviceId":"abcd1234abcd1234","CallId":"21225-3848@5.6.7.8
<mailto:21225-3848@5.6.7.8>"},143:{"event":"REGISTER","tindex":"28682","tlabel":"9676691","PhoneNumber":"20712","DeviceId":"abcd1234abcd1234","CallId":"21226-3848@5.6.7.8
<mailto:21226-3848@5.6.7.8>"},]
*
*
But I don't see the logs that I've written in the script after the
event was raised, which means Kamailio did not invoke events for these
two netstrings.
So in my tests, out of 27232 messages sent, there were approximately
27 messages for which the events were not raised by Evapi. The rate of
messages started at 500cps and I stopped after I saw missed events at
around 1200cps.
Thanks, and do let me know for any further tests or information
required about the same.
- Jayesh
*
*
On Fri, Sep 18, 2015 at 2:55 PM, Jayesh Nambiar <jayesh1017(a)gmail.com
<mailto:jayesh1017@gmail.com>> wrote:
Will have this tested by tomorrow and will get back to you. Thanks.
- Jayesh
On Fri, Sep 18, 2015 at 1:41 PM, Daniel-Constantin Mierla
<miconda(a)gmail.com <mailto:miconda@gmail.com>> wrote:
Hello,
I just pushed a patch to master branch that should cope with
partial data received on tcp connection. No time to test at
all, therefore any feedback will be appreciated.
Cheers,
Daniel
On 15/09/15 14:52, Daniel-Constantin Mierla wrote:
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Book: SIP Routing With Kamailio - http://www.asipto.com
Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
Hello,
I will look if there are options in libev to buffer data or
try to implement a buffering mechanism locally for such cases.
Cheers,
Daniel
On 14/09/15 23:00, Jayesh Nambiar wrote:
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> -
http://www.linkedin.com/in/miconda
Book: SIP Routing With Kamailio - http://www.asipto.com
Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
Hello Daniel,
After further testing with evapi module, I figured that when
Netstrings are used, an event route is invoked individually
for each message even if if multiple netstring messages are
received in a single TCP packet. But this doesn't work
effectively when a single proper message is split-up in two
packets. For Example, if a message arrives as:
12:Hello World!, 12:Hello World!, 12:Hello World! in a
single packet, kamailio properly invokes the event route
"evapi:message-received" thrice for every individual proper
netstring message.
But if the first packet contains:
12:Hello World!, 12:Hello
And Second Packet contains:
World!, 12:Hello World!
the event route is invoked only once !!
The above pattern is very much possible while sending and
receiving packets over TCP Socket. Our tests for receiving
an approximately 150 byte message over evapi socket at the
rate of roughly 1000cps causes a lot of real events to be
missed because of the above problem. You can never be sure
when TCP will split messages in different chunks.
This definitely looks like a bug which makes it not very
reliable at large scale deployments. Would really appreciate
your inputs on this. Thanks;
- Jayesh
On Thu, Sep 10, 2015 at 4:01 PM, Jayesh Nambiar
<jayesh1017(a)gmail.com <mailto:jayesh1017@gmail.com>> wrote:
Hi Daniel,
Thanks for the quick response. So if I do not use
Netstrings, does Kamailio allow me to create a custom
logic in the script. For eg. if I decide to use newline
as a delimiter, can I keep buffering the message until I
encounter the delimiter from the event route and then
execute whatever I have to within the script??
Thanks,
- Jayesh
On Thu, Sep 10, 2015 at 1:29 PM, Daniel-Constantin
Mierla <miconda(a)gmail.com <mailto:miconda@gmail.com>> wrote:
Hello,
tcb is stream protocol and several messages can be
queued on the pipe at the same time. That is the
reason for netstring format, to be able to easily
detect the boundaries of each message. If netstring
format is enabled and kamailio receives several
messages at once, it splits them and for each is
executing the event route.
If netstring format is not used, the kamailio is
executing the event route with the entire content
that was read at once from the tcp connection.
Cheers,
Daniel
On 09/09/15 22:01, Jayesh Nambiar wrote:
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> -
http://www.linkedin.com/in/miconda
Book: SIP Routing With Kamailio - http://www.asipto.com
Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
Hello,
I'm exploring the evapi module for my kamailio to
interface with an external node.js app for third
party stuff like AAA, billing engine tasks,
notifications and so on. I followed and took some
ideas from the rtjson and evapi tutorial found
here(http://kb.asipto.com/kamailio:k43-async-sip-routing-nodejs)
to build the node.js app consuming events.
When I stress tested the scenario using SIPp and
tried sending a lot of events at 300-350cps from
Kamailio, I noticed that at times the client is
receiving 2-3 events in a single message together
although I do event_sync_relay once per SIP message
received and have netstrings enabled. I believe
this is a typical behavior of TCP and needs to be
handled by the client using some kind of Netstring
handler. Please correct me if I'm wrong.
And hence I'd like to know what particularly needs
to be taken care of while writing a client that is
listening for events on raw tcp socket and how does
kamailio handle this situation while receiving
messages over TCP socket?? Does kamailio recognize
the end of netstring properly on
evapi:message-received and give exactly one message
to take care of on every "message-received" event
or should that be handled in the script somewhere !!
I also referred cgrates client over evapi example
which is written in GO, but I couldnt find them
handling TCP streams clearly either.
I'd really appreciate some expert suggestion here
to make an informed decision on using the evapi
module for a large scale solution.
Thanks,
- Jayesh
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users(a)lists.sip-router.org
<mailto:sr-users@lists.sip-router.org>
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda
<http://twitter.com/#%21/miconda> -
http://www.linkedin.com/in/miconda
Book: SIP Routing With Kamailio - http://www.asipto.com
Kamailio Advanced Training, Sep 28-30, 2015, in Berlin -
http://asipto.com/u/kat
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) -
sr-users mailing list
sr-users(a)lists.sip-router.org
<mailto:sr-users@lists.sip-router.org>
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
21 Sep
21 Sep
3:04 a.m.
Hi Daniel,
I have captured debug messages after testing with your latest commit on
master. Once again, here's what I'm doing exactly for my test:
1) Use SIPp to send OPTIONS towards Kamailio.
2) On getting options send event as follows:
evapi_async_relay("{\"event\":\"REGISTER\",\"tindex\":\"$T(id_index)\",\"tlabel\":\"$T(id_label)\",\"PhoneNumber\":\"$avp(phone_number)\",\"DeviceId\":\"$avp(device_id)\",\"CallId\":\"$ci\"}");
3) There's a client connected which listens for messages on this socket,
parses the netstring, and sends same data back as netstring to Kamailio.
4) On the event_route[evapi:message-received, I do the following:
xlog("L_INFO", "GOT [$evapi(msg)] from
$evapi(srcaddr):$evapi(srcport)\n");
if($evapi(msg)=~"REGISTER" && $evapi(msg)=~"tindex") {
jansson_get_field("$evapi(msg)", "tlabel",
"$var(tlabel)");
jansson_get_field("$evapi(msg)", "tindex",
"$var(tindex)");
$var(t_index) = $(var(tindex){s.int});
$var(t_label) = $(var(tlabel){s.int});
t_continue('$var(t_index)', '$var(t_label)',
'REGISTER_RESPONSE');
exit;
}
5) On route[REGISTER_RESPONSE], I send 200 OK to SIPp
Now here are detailed logs for which neither SIPp didnt get a response nor
the GOT [$evapi(msg)] from $evapi(srcaddr):$evapi(srcport)\n"); got logged
in the syslog:
First Event:
DEBUG: evapi [evapi_dispatch.c:598]: evapi_relay(): relaying event data
[{"event":"REGISTER","tindex":"26266","tlabel":"587925078","PhoneNumber":"21956","DeviceId":"abcd1234abcd1234","CallId":"
23021-12910(a)198.24.63.39"}] (146)
DEBUG: evapi [evapi_dispatch.c:623]: evapi_relay(): sending
[0x7f132588de68]
[146:{"event":"REGISTER","tindex":"26266","tlabel":"587925078","PhoneNumber":"21956","DeviceId":"abcd1234abcd1234","CallId":"
23021-12910(a)198.24.63.39"}uot;},] (151)
DEBUG: evapi [evapi_dispatch.c:488]: evapi_recv_notify(): received
[0x7f132588de68]
[146:{"event":"REGISTER","tindex":"26266","tlabel":"587925078","PhoneNumber":"21956","DeviceId":"abcd1234abcd1234","CallId":"
23021-12910(a)198.24.63.39"}uot;},] (151)
NOTICE: evapi [evapi_dispatch.c:290]: evapi_recv_client(): {0} [
198.24.63.45:48881] - received
[146:{"event":"REGISTER","tindex":"26266","tlabel":"587925078","PhoneNumber":"21956","DeviceId":"abcd1234abcd1234","CallId":"
23021-12910(a)198.24.63.39"}uot;},] (151) (18)
Second Event:
DEBUG: evapi [evapi_dispatch.c:598]: evapi_relay(): relaying event data
[{"event":"REGISTER","tindex":"32244","tlabel":"1637923412","PhoneNumber":"25597","DeviceId":"abcd1234abcd1234","CallId":"
27182-12910(a)198.24.63.39"}] (147)
DEBUG: evapi [evapi_dispatch.c:623]: evapi_relay(): sending
[0x7f1325884410]
[147:{"event":"REGISTER","tindex":"32244","tlabel":"1637923412","PhoneNumber":"25597","DeviceId":"abcd1234abcd1234","CallId":"
27182-12910(a)198.24.63.39"}uot;},] (152)
DEBUG: evapi [evapi_dispatch.c:488]: evapi_recv_notify(): received
[0x7f1325884410]
[147:{"event":"REGISTER","tindex":"32244","tlabel":"1637923412","PhoneNumber":"25597","DeviceId":"abcd1234abcd1234","CallId":"
27182-12910(a)198.24.63.39"}uot;},] (152)
NOTICE: evapi [evapi_dispatch.c:290]: evapi_recv_client(): {0} [
198.24.63.45:48881] - received
[147:{"event":"REGISTER","tindex":"32244","tlabel":"1637923412","PhoneNumber":"25597","DeviceId":"abcd1234abcd1234","CallId":"
27182-12910(a)198.24.63.39"}uot;},] (152) (85)
I had three such events which got missed out of some 25000 odd messages
sent from SIPp. Other major difference that I see in the debug logs for
problematic events are that there is a positive number in the second
parentheses of evapi_recv_client() function. For the events that were
invoked successfully the value of second parentheses is 0 for
evapi_recv_client() function. Hope the debugging helps.
Debug log for a successful event:
DEBUG: evapi [evapi_dispatch.c:598]: evapi_relay(): relaying event data
[{"event":"REGISTER","tindex":"22127","tlabel":"1896682192","PhoneNumber":"73168","DeviceId":"abcd1234abcd1234","CallId":"
49957-13056(a)198.24.63.39"}] (147)
DEBUG: evapi [evapi_dispatch.c:623]: evapi_relay(): sending
[0x7f132568e850]
[147:{"event":"REGISTER","tindex":"22127","tlabel":"1896682192","PhoneNumber":"73168","DeviceId":"abcd1234abcd1234","CallId":"
49957-13056(a)198.24.63.39"}uot;},] (152)
DEBUG: evapi [evapi_dispatch.c:488]: evapi_recv_notify(): received
[0x7f132568e850]
[147:{"event":"REGISTER","tindex":"22127","tlabel":"1896682192","PhoneNumber":"73168","DeviceId":"abcd1234abcd1234","CallId":"
49957-13056(a)198.24.63.39"}uot;},] (152)
NOTICE: evapi [evapi_dispatch.c:290]: evapi_recv_client(): {0} [
198.24.63.45:48881] - received
[147:{"event":"REGISTER","tindex":"22127","tlabel":"1896682192","PhoneNumber":"73168","DeviceId":"abcd1234abcd1234","CallId":"
49957-13056(a)198.24.63.39"}uot;},] (152) (0)
Thanks,
- Jayesh
On Sat, Sep 19, 2015 at 3:55 AM Daniel-Constantin Mierla <miconda(a)gmail.com>
wrote:
I pushed a commit to add more debug message while
processing received
data. You can use debugger module to set a higher debug level for evapi
module in order to see what happens.
I checked the netstring packet size and it is invalid (unless email
stripped some white chars there) -- for example in:
145:{"event":"REGISTER","tindex":"55567","tlabel":"627458699","PhoneNumber":"20711","DeviceId":"abcd1234abcd1234","CallId":
"21225-3848(a)5.6.7.8" <21225-3848(a)5.6.7.8>}8>},
the size is 141, not 145 -- it looks like the size includes the size
itself plus the delimiters ':,'. The size is only the effective data, see:
https://en.wikipedia.org/wiki/Netstring
If evapi gets a packet with an invalid size, then it discards the buffer
content.
See if the app on the other side of evapi connection builds netstrings
with wrong size.
Cheers,
Daniel
On 18/09/15 23:19, Jayesh Nambiar wrote:
Here are the tests that I did:
With the patch applied, I see that Kamailio is invoking event individually
for each netstring even when they come in different chunks. But I did see
instances where when there were complete netstrings in a single chunk;
kamailio did not raise an event for them. Here's the illustration:
T 1.2.3.4:48873 -> 5.6.7.8:3927 [AP]
146:{"event":"REGISTER","tindex":"16916","tlabel":"1753048277","PhoneNumber":"20708","DeviceId":"abcd1234abcd1234","CallId":"
<21220-3848@5.6.7.8>21220-3848@5.6.7.8
"},144:{"event":"REGISTER","tindex":"7954","tlabel":"254315075","PhoneNumber":"20709","DeviceId":"abcd1234abcd1234","CallId":"
<21223-3848@5.6.7.8>21223-3848@5.6.7.8
"},145:{"event":"REGISTER","tindex":"64529","tlabel":"599481568","PhoneNumber":"20709","DeviceId":"abcd1234abcd1234","CallId":"
<21222-3848@5.6.7.8>21222-3848@5.6.7.8"},145:{"event":"REGISTER","tindex":"46605","tlabel":"112015324","PhoneNumber":"20710","DeviceId"
T 1.2.3.4:48873 -> 5.6.7.8:3927 [AP]
:"abcd1234abcd1234","CallId":"21224-3848@5.6.7.8"},
*The above two chunks contain 4 proper netstrings where the second chunk
contains part of the 4th netstring. In this case Evapi properly raised 4
individual events.*
1.2.3.4:48873 -> 5.6.7.8:3927 [AP]
145:{"event":"REGISTER","tindex":"55567","tlabel":"627458699","PhoneNumber":"20711","DeviceId":"abcd1234abcd1234","CallId":"
<21225-3848@5.6.7.8>21225-3848@5.6.7.8
"},143:{"event":"REGISTER","tindex":"28682","tlabel":"9676691","PhoneNumber":"20712","DeviceId":"abcd1234abcd1234","CallId":"
<21226-3848@5.6.7.8>21226-3848@5.6.7.8"},
*This above chunk contains two complete netstrings but Kamailio never
raised events for these two netstrings.*
For the events that were not raised I see proper Kamailio Logs which is:
evapi_recv_client(): {0} [1.2.3.4:48873] - received
[145:{"event":"REGISTER","tindex":"55567","tlabel":"627458699","PhoneNumber":"20711","DeviceId":"abcd1234abcd1234","CallId":"
<21225-3848@5.6.7.8>21225-3848@5.6.7.8
"},143:{"event":"REGISTER","tindex":"28682","tlabel":"9676691","PhoneNumber":"20712","DeviceId":"abcd1234abcd1234","CallId":"
<21226-3848@5.6.7.8>21226-3848@5.6.7.8"},]
But I don't see the logs that I've written in the script after the event
was raised, which means Kamailio did not invoke events for these two
netstrings.
So in my tests, out of 27232 messages sent, there were approximately 27
messages for which the events were not raised by Evapi. The rate of
messages started at 500cps and I stopped after I saw missed events at
around 1200cps.
Thanks, and do let me know for any further tests or information required
about the same.
- Jayesh
On Fri, Sep 18, 2015 at 2:55 PM, Jayesh Nambiar <jayesh1017(a)gmail.com>
wrote:
Will have this tested by tomorrow and will get
back to you. Thanks.
- Jayesh
On Fri, Sep 18, 2015 at 1:41 PM, Daniel-Constantin Mierla <
<miconda@gmail.com>miconda@gmail.com> wrote:
--
Daniel-Constantin Mierlahttp://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda
Book: SIP Routing With Kamailio - http://www.asipto.com
Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
Hello,
I just pushed a patch to master branch that should cope with partial
data received on tcp connection. No time to test at all, therefore any
feedback will be appreciated.
Cheers,
Daniel
On 15/09/15 14:52, Daniel-Constantin Mierla wrote:
Hello,
I will look if there are options in libev to buffer data or try to
implement a buffering mechanism locally for such cases.
Cheers,
Daniel
On 14/09/15 23:00, Jayesh Nambiar wrote:
Hello Daniel,
After further testing with evapi module, I figured that when Netstrings
are used, an event route is invoked individually for each message even if
if multiple netstring messages are received in a single TCP packet. But
this doesn't work effectively when a single proper message is split-up in
two packets. For Example, if a message arrives as:
12:Hello World!, 12:Hello World!, 12:Hello World! in a single packet,
kamailio properly invokes the event route "evapi:message-received" thrice
for every individual proper netstring message.
But if the first packet contains:
12:Hello World!, 12:Hello
And Second Packet contains:
World!, 12:Hello World!
the event route is invoked only once !!
The above pattern is very much possible while sending and receiving
packets over TCP Socket. Our tests for receiving an approximately 150 byte
message over evapi socket at the rate of roughly 1000cps causes a lot of
real events to be missed because of the above problem. You can never be
sure when TCP will split messages in different chunks.
This definitely looks like a bug which makes it not very reliable at
large scale deployments. Would really appreciate your inputs on this.
Thanks;
- Jayesh
On Thu, Sep 10, 2015 at 4:01 PM, Jayesh Nambiar < <jayesh1017(a)gmail.com>
jayesh1017(a)gmail.com> wrote:
Hi Daniel,
Thanks for the quick response. So if I do not use Netstrings, does
Kamailio allow me to create a custom logic in the script. For eg. if I
decide to use newline as a delimiter, can I keep buffering the message
until I encounter the delimiter from the event route and then execute
whatever I have to within the script??
Thanks,
- Jayesh
On Thu, Sep 10, 2015 at 1:29 PM, Daniel-Constantin Mierla <
<miconda@gmail.com>miconda@gmail.com> wrote:
> Hello,
>
> tcb is stream protocol and several messages can be queued on the pipe
> at the same time. That is the reason for netstring format, to be able to
> easily detect the boundaries of each message. If netstring format is
> enabled and kamailio receives several messages at once, it splits them and
> for each is executing the event route.
>
> If netstring format is not used, the kamailio is executing the event
> route with the entire content that was read at once from the tcp connection.
>
> Cheers,
> Daniel
>
>
> On 09/09/15 22:01, Jayesh Nambiar wrote:
>
> Hello,
> I'm exploring the evapi module for my kamailio to interface with an
> external node.js app for third party stuff like AAA, billing engine tasks,
> notifications and so on. I followed and took some ideas from the rtjson and
> evapi tutorial found here(
> <http://kb.asipto.com/kamailio:k43-async-sip-routing-nodejs>
> http://kb.asipto.com/kamailio:k43-async-sip-routing-nodejs) to build
> the node.js app consuming events.
> When I stress tested the scenario using SIPp and tried sending a lot
> of events at 300-350cps from Kamailio, I noticed that at times the client
> is receiving 2-3 events in a single message together although I do
> event_sync_relay once per SIP message received and have netstrings enabled.
> I believe this is a typical behavior of TCP and needs to be handled by the
> client using some kind of Netstring handler. Please correct me if I'm wrong.
> And hence I'd like to know what particularly needs to be taken care of
> while writing a client that is listening for events on raw tcp socket and
> how does kamailio handle this situation while receiving messages over TCP
> socket?? Does kamailio recognize the end of netstring properly on
> evapi:message-received and give exactly one message to take care of on
> every "message-received" event or should that be handled in the script
> somewhere !!
> I also referred cgrates client over evapi example which is written in
> GO, but I couldnt find them handling TCP streams clearly either.
> I'd really appreciate some expert suggestion here to make an informed
> decision on using the evapi module for a large scale solution.
>
> Thanks,
>
> - Jayesh
>
>
>
>
> _______________________________________________
> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing
listsr-users@lists.sip-router.orghttp://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
>
>
> --
> Daniel-Constantin Mierlahttp://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda
> Book: SIP Routing With Kamailio - http://www.asipto.com
> Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
>
>
> _______________________________________________
> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
> <sr-users@lists.sip-router.org>sr-users@lists.sip-router.org
> <http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users>
> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
>
>
--
Daniel-Constantin Mierlahttp://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda
Book: SIP Routing With Kamailio - http://www.asipto.com
Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
--
Daniel-Constantin Mierlahttp://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda
Book: SIP Routing With Kamailio - http://www.asipto.com
Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
7:07 a.m.
Hello,
I pushed another commit to add more debug messages to see if the event
route is supposed to be executed or not. Can you run the tests again and
give again the log messages for missing event route executions?
Cheers,
Daniel
On 21/09/15 09:04, Jayesh Nambiar wrote:
Hi Daniel,
I have captured debug messages after testing with your latest commit
on master. Once again, here's what I'm doing exactly for my test:
1) Use SIPp to send OPTIONS towards Kamailio.
2) On getting options send event as follows:
evapi_async_relay("{\"event\":\"REGISTER\",\"tindex\":\"$T(id_index)\",\"tlabel\":\"$T(id_label)\",\"PhoneNumber\":\"$avp(phone_number)\",\"DeviceId\":\"$avp(device_id)\",\"CallId\":\"$ci\"}");
3) There's a client connected which listens for messages on this
socket, parses the netstring, and sends same data back as netstring to
Kamailio.
4) On the event_route[evapi:message-received, I do the following:
xlog("L_INFO", "GOT [$evapi(msg)] from
$evapi(srcaddr):$evapi(srcport)\n");
if($evapi(msg)=~"REGISTER" && $evapi(msg)=~"tindex") {
jansson_get_field("$evapi(msg)", "tlabel",
"$var(tlabel)");
jansson_get_field("$evapi(msg)", "tindex",
"$var(tindex)");
$var(t_index) = $(var(tindex){s.int <http://s.int>});
$var(t_label) = $(var(tlabel){s.int <http://s.int>});
t_continue('$var(t_index)', '$var(t_label)',
'REGISTER_RESPONSE');
exit;
}
5) On route[REGISTER_RESPONSE], I send 200 OK to SIPp
Now here are detailed logs for which neither SIPp didnt get a response
nor the GOT [$evapi(msg)] from $evapi(srcaddr):$evapi(srcport)\n");
got logged in the syslog:
First Event:
DEBUG: evapi [evapi_dispatch.c:598]: evapi_relay(): relaying event
data
[{"event":"REGISTER","tindex":"26266","tlabel":"587925078","PhoneNumber":"21956","DeviceId":"abcd1234abcd1234","CallId":"23021-12910@198.24.63.39
<mailto:23021-12910@198.24.63.39>"}] (146)
DEBUG: evapi [evapi_dispatch.c:623]: evapi_relay(): sending
[0x7f132588de68]
[146:{"event":"REGISTER","tindex":"26266","tlabel":"587925078","PhoneNumber":"21956","DeviceId":"abcd1234abcd1234","CallId":"23021-12910@198.24.63.39
<mailto:23021-12910@198.24.63.39>"},] (151)
DEBUG: evapi [evapi_dispatch.c:488]: evapi_recv_notify(): received
[0x7f132588de68]
[146:{"event":"REGISTER","tindex":"26266","tlabel":"587925078","PhoneNumber":"21956","DeviceId":"abcd1234abcd1234","CallId":"23021-12910@198.24.63.39
<mailto:23021-12910@198.24.63.39>"},] (151)
NOTICE: evapi [evapi_dispatch.c:290]: evapi_recv_client(): {0}
[198.24.63.45:48881 <http://198.24.63.45:48881>] - received
[146:{"event":"REGISTER","tindex":"26266","tlabel":"587925078","PhoneNumber":"21956","DeviceId":"abcd1234abcd1234","CallId":"23021-12910@198.24.63.39
<mailto:23021-12910@198.24.63.39>"},] (151) (18)
Second Event:
DEBUG: evapi [evapi_dispatch.c:598]: evapi_relay(): relaying event
data
[{"event":"REGISTER","tindex":"32244","tlabel":"1637923412","PhoneNumber":"25597","DeviceId":"abcd1234abcd1234","CallId":"27182-12910@198.24.63.39
<mailto:27182-12910@198.24.63.39>"}] (147)
DEBUG: evapi [evapi_dispatch.c:623]: evapi_relay(): sending
[0x7f1325884410]
[147:{"event":"REGISTER","tindex":"32244","tlabel":"1637923412","PhoneNumber":"25597","DeviceId":"abcd1234abcd1234","CallId":"27182-12910@198.24.63.39
<mailto:27182-12910@198.24.63.39>"},] (152)
DEBUG: evapi [evapi_dispatch.c:488]: evapi_recv_notify(): received
[0x7f1325884410]
[147:{"event":"REGISTER","tindex":"32244","tlabel":"1637923412","PhoneNumber":"25597","DeviceId":"abcd1234abcd1234","CallId":"27182-12910@198.24.63.39
<mailto:27182-12910@198.24.63.39>"},] (152)
NOTICE: evapi [evapi_dispatch.c:290]: evapi_recv_client(): {0}
[198.24.63.45:48881 <http://198.24.63.45:48881>] - received
[147:{"event":"REGISTER","tindex":"32244","tlabel":"1637923412","PhoneNumber":"25597","DeviceId":"abcd1234abcd1234","CallId":"27182-12910@198.24.63.39
<mailto:27182-12910@198.24.63.39>"},] (152) (85)
I had three such events which got missed out of some 25000 odd
messages sent from SIPp. Other major difference that I see in the
debug logs for problematic events are that there is a positive number
in the second parentheses of evapi_recv_client() function. For the
events that were invoked successfully the value of second parentheses
is 0 for evapi_recv_client() function. Hope the debugging helps.
Debug log for a successful event:
DEBUG: evapi [evapi_dispatch.c:598]: evapi_relay(): relaying event
data
[{"event":"REGISTER","tindex":"22127","tlabel":"1896682192","PhoneNumber":"73168","DeviceId":"abcd1234abcd1234","CallId":"49957-13056@198.24.63.39
<mailto:49957-13056@198.24.63.39>"}] (147)
DEBUG: evapi [evapi_dispatch.c:623]: evapi_relay(): sending
[0x7f132568e850]
[147:{"event":"REGISTER","tindex":"22127","tlabel":"1896682192","PhoneNumber":"73168","DeviceId":"abcd1234abcd1234","CallId":"49957-13056@198.24.63.39
<mailto:49957-13056@198.24.63.39>"},] (152)
DEBUG: evapi [evapi_dispatch.c:488]: evapi_recv_notify(): received
[0x7f132568e850]
[147:{"event":"REGISTER","tindex":"22127","tlabel":"1896682192","PhoneNumber":"73168","DeviceId":"abcd1234abcd1234","CallId":"49957-13056@198.24.63.39
<mailto:49957-13056@198.24.63.39>"},] (152)
NOTICE: evapi [evapi_dispatch.c:290]: evapi_recv_client(): {0}
[198.24.63.45:48881 <http://198.24.63.45:48881>] - received
[147:{"event":"REGISTER","tindex":"22127","tlabel":"1896682192","PhoneNumber":"73168","DeviceId":"abcd1234abcd1234","CallId":"49957-13056@198.24.63.39
<mailto:49957-13056@198.24.63.39>"},] (152) (0)
Thanks,
- Jayesh
On Sat, Sep 19, 2015 at 3:55 AM Daniel-Constantin Mierla
<miconda(a)gmail.com <mailto:miconda@gmail.com>> wrote:
I pushed a commit to add more debug message while processing
received data. You can use debugger module to set a higher debug
level for evapi module in order to see what happens.
I checked the netstring packet size and it is invalid (unless
email stripped some white chars there) -- for example in:
145:{"event":"REGISTER","tindex":"55567","tlabel":"627458699","PhoneNumber":"20711","DeviceId":"abcd1234abcd1234","CallId":"21225-3848@5.6.7.8"
<mailto:21225-3848@5.6.7.8>},
the size is 141, not 145 -- it looks like the size includes the
size itself plus the delimiters ':,'. The size is only the
effective data, see:
https://en.wikipedia.org/wiki/Netstring
If evapi gets a packet with an invalid size, then it discards the
buffer content.
See if the app on the other side of evapi connection builds
netstrings with wrong size.
Cheers,
Daniel
On 18/09/15 23:19, Jayesh Nambiar wrote:
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Book: SIP Routing With Kamailio - http://www.asipto.com
Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
Here are the tests that I did:
With the patch applied, I see that Kamailio is invoking event
individually for each netstring even when they come in different
chunks. But I did see instances where when there were complete
netstrings in a single chunk; kamailio did not raise an event for
them. Here's the illustration:
T 1.2.3.4:48873 <http://1.2.3.4:48873> -> 5.6.7.8:3927
<http://5.6.7.8:3927> [AP]
146:{"event":"REGISTER","tindex":"16916","tlabel":"1753048277","PhoneNumber":"20708","DeviceId":"abcd1234abcd1234","CallId":"21220-3848@5.6.7.8
<mailto:21220-3848@5.6.7.8>"},144:{"event":"REGISTER","tindex":"7954","tlabel":"254315075","PhoneNumber":"20709","DeviceId":"abcd1234abcd1234","CallId":"21223-3848@5.6.7.8
<mailto:21223-3848@5.6.7.8>"},145:{"event":"REGISTER","tindex":"64529","tlabel":"599481568","PhoneNumber":"20709","DeviceId":"abcd1234abcd1234","CallId":"21222-3848@5.6.7.8
<mailto:21222-3848@5.6.7.8>"},145:{"event":"REGISTER","tindex":"46605","tlabel":"112015324","PhoneNumber":"20710","DeviceId"
T 1.2.3.4:48873 <http://1.2.3.4:48873> -> 5.6.7.8:3927
<http://5.6.7.8:3927> [AP]
:"abcd1234abcd1234","CallId":"21224-3848@5.6.7.8
<mailto:21224-3848@5.6.7.8>"},
*The above two chunks contain 4 proper netstrings where the
second chunk contains part of the 4th netstring. In this case
Evapi properly raised 4 individual events.*
1.2.3.4:48873 <http://1.2.3.4:48873> -> 5.6.7.8:3927
<http://5.6.7.8:3927> [AP]
145:{"event":"REGISTER","tindex":"55567","tlabel":"627458699","PhoneNumber":"20711","DeviceId":"abcd1234abcd1234","CallId":"21225-3848@5.6.7.8
<mailto:21225-3848@5.6.7.8>"},143:{"event":"REGISTER","tindex":"28682","tlabel":"9676691","PhoneNumber":"20712","DeviceId":"abcd1234abcd1234","CallId":"21226-3848@5.6.7.8
<mailto:21226-3848@5.6.7.8>"},
*This above chunk contains two complete netstrings but Kamailio
never raised events for these two netstrings.*
*
*
For the events that were not raised I see proper Kamailio Logs
which is:
evapi_recv_client(): {0} [1.2.3.4:48873 <http://1.2.3.4:48873>] -
received
[145:{"event":"REGISTER","tindex":"55567","tlabel":"627458699","PhoneNumber":"20711","DeviceId":"abcd1234abcd1234","CallId":"21225-3848@5.6.7.8
<mailto:21225-3848@5.6.7.8>"},143:{"event":"REGISTER","tindex":"28682","tlabel":"9676691","PhoneNumber":"20712","DeviceId":"abcd1234abcd1234","CallId":"21226-3848@5.6.7.8
<mailto:21226-3848@5.6.7.8>"},]
*
*
But I don't see the logs that I've written in the script after
the event was raised, which means Kamailio did not invoke events
for these two netstrings.
So in my tests, out of 27232 messages sent, there were
approximately 27 messages for which the events were not raised by
Evapi. The rate of messages started at 500cps and I stopped after
I saw missed events at around 1200cps.
Thanks, and do let me know for any further tests or information
required about the same.
- Jayesh
*
*
On Fri, Sep 18, 2015 at 2:55 PM, Jayesh Nambiar
<jayesh1017(a)gmail.com <mailto:jayesh1017@gmail.com>> wrote:
Will have this tested by tomorrow and will get back to you.
Thanks.
- Jayesh
On Fri, Sep 18, 2015 at 1:41 PM, Daniel-Constantin Mierla
<miconda(a)gmail.com <mailto:miconda@gmail.com>> wrote:
Hello,
I just pushed a patch to master branch that should cope
with partial data received on tcp connection. No time to
test at all, therefore any feedback will be appreciated.
Cheers,
Daniel
On 15/09/15 14:52, Daniel-Constantin Mierla wrote:
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> -
http://www.linkedin.com/in/miconda
Book: SIP Routing With Kamailio - http://www.asipto.com
Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
Hello,
I will look if there are options in libev to buffer data
or try to implement a buffering mechanism locally for
such cases.
Cheers,
Daniel
On 14/09/15 23:00, Jayesh Nambiar wrote:
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda
<http://twitter.com/#%21/miconda> - http://www.linkedin.com/in/miconda
Book: SIP Routing With Kamailio - http://www.asipto.com
Kamailio Advanced Training, Sep 28-30, 2015, in Berlin -
http://asipto.com/u/kat
Hello Daniel,
After further testing with evapi module, I figured that
when Netstrings are used, an event route is invoked
individually for each message even if if multiple
netstring messages are received in a single TCP packet.
But this doesn't work effectively when a single proper
message is split-up in two packets. For Example, if a
message arrives as:
12:Hello World!, 12:Hello World!, 12:Hello World! in a
single packet, kamailio properly invokes the event
route "evapi:message-received" thrice for every
individual proper netstring message.
But if the first packet contains:
12:Hello World!, 12:Hello
And Second Packet contains:
World!, 12:Hello World!
the event route is invoked only once !!
The above pattern is very much possible while sending
and receiving packets over TCP Socket. Our tests for
receiving an approximately 150 byte message over evapi
socket at the rate of roughly 1000cps causes a lot of
real events to be missed because of the above problem.
You can never be sure when TCP will split messages in
different chunks.
This definitely looks like a bug which makes it not
very reliable at large scale deployments. Would really
appreciate your inputs on this. Thanks;
- Jayesh
On Thu, Sep 10, 2015 at 4:01 PM, Jayesh Nambiar
<jayesh1017(a)gmail.com <mailto:jayesh1017@gmail.com>> wrote:
Hi Daniel,
Thanks for the quick response. So if I do not use
Netstrings, does Kamailio allow me to create a
custom logic in the script. For eg. if I decide to
use newline as a delimiter, can I keep buffering
the message until I encounter the delimiter from
the event route and then execute whatever I have to
within the script??
Thanks,
- Jayesh
On Thu, Sep 10, 2015 at 1:29 PM, Daniel-Constantin
Mierla <miconda(a)gmail.com
<mailto:miconda@gmail.com>> wrote:
Hello,
tcb is stream protocol and several messages can
be queued on the pipe at the same time. That is
the reason for netstring format, to be able to
easily detect the boundaries of each message.
If netstring format is enabled and kamailio
receives several messages at once, it splits
them and for each is executing the event route.
If netstring format is not used, the kamailio
is executing the event route with the entire
content that was read at once from the tcp
connection.
Cheers,
Daniel
On 09/09/15 22:01, Jayesh Nambiar wrote:
> Hello,
> I'm exploring the evapi module for my kamailio
> to interface with an external node.js app for
> third party stuff like AAA, billing engine
> tasks, notifications and so on. I followed and
> took some ideas from the rtjson and evapi
> tutorial found
> here(http://kb.asipto.com/kamailio:k43-async-sip-routing-nodejs)
> to build the node.js app consuming events.
> When I stress tested the scenario using SIPp
> and tried sending a lot of events at
> 300-350cps from Kamailio, I noticed that at
> times the client is receiving 2-3 events in a
> single message together although I do
> event_sync_relay once per SIP message received
> and have netstrings enabled. I believe this is
> a typical behavior of TCP and needs to be
> handled by the client using some kind of
> Netstring handler. Please correct me if I'm wrong.
> And hence I'd like to know what particularly
> needs to be taken care of while writing a
> client that is listening for events on raw tcp
> socket and how does kamailio handle this
> situation while receiving messages over TCP
> socket?? Does kamailio recognize the end of
> netstring properly on evapi:message-received
> and give exactly one message to take care of
> on every "message-received" event or should
> that be handled in the script somewhere !!
> I also referred cgrates client over evapi
> example which is written in GO, but I couldnt
> find them handling TCP streams clearly either.
> I'd really appreciate some expert suggestion
> here to make an informed decision on using the
> evapi module for a large scale solution.
>
> Thanks,
>
> - Jayesh
>
>
>
>
> _______________________________________________
> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users
mailing list
> sr-users(a)lists.sip-router.org
> <mailto:sr-users@lists.sip-router.org>
> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda
<http://twitter.com/#%21/miconda> -
http://www.linkedin.com/in/miconda
Book: SIP Routing With Kamailio - http://www.asipto.com
Kamailio Advanced Training, Sep 28-30, 2015, in Berlin -
http://asipto.com/u/kat
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER)
- sr-users mailing list
sr-users(a)lists.sip-router.org
<mailto:sr-users@lists.sip-router.org>
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda
<http://twitter.com/#%21/miconda> - http://www.linkedin.com/in/miconda
Book: SIP Routing With Kamailio - http://www.asipto.com
Kamailio Advanced Training, Sep 28-30, 2015, in Berlin -
http://asipto.com/u/kat
22 Sep
22 Sep
3:09 a.m.
Hi,
Took time to get back with the test results as I was trying to analyze too
much myself before getting back on the list. Also I was stuck with weird
error of module version mismatch, but I figured and solved that issue.
So I do see the logs "frame size mismatch" when I miss the events. Since
the logs are too huge, I am sending it in pastebin. Here it is:
http://pastebin.com/MzTpYsrk
In Kamailio Logs, I see:
DEBUG: evapi [evapi_dispatch.c:361]: evapi_recv_client(): frame size
mismatch the ending char
(h):[{"event":"REGISTER","tindex":"12603","tlabel":"626010915
","PhoneNumber":"42000","D146:"event":"REGISTER","tindex":"3639","tlabel":"
1889788252","P] (146)
Whereas the messages in ngrep looks good where the messages are split in
two TCP packets. The first packet ends at D and the second packet is the
proper continuation which send the rest. But it feels Kamailio skipped the
beginning of the message and started parsing at colon again and it thinks
the netstring started with D146 which is incorrect. Hence, it never emitted
any event for the messages in the first chunk.
Hoping this to be helpful !! Thanks.
- Jayesh
On Mon, Sep 21, 2015 at 4:37 PM Daniel-Constantin Mierla <miconda(a)gmail.com>
wrote:
Hello,
I pushed another commit to add more debug messages to see if the event
route is supposed to be executed or not. Can you run the tests again and
give again the log messages for missing event route executions?
Cheers,
Daniel
On 21/09/15 09:04, Jayesh Nambiar wrote:
Hi Daniel,
I have captured debug messages after testing with your latest commit on
master. Once again, here's what I'm doing exactly for my test:
1) Use SIPp to send OPTIONS towards Kamailio.
2) On getting options send event as follows:
evapi_async_relay("{\"event\":\"REGISTER\",\"tindex\":\"$T(id_index)\",\"tlabel\":\"$T(id_label)\",\"PhoneNumber\":\"$avp(phone_number)\",\"DeviceId\":\"$avp(device_id)\",\"CallId\":\"$ci\"}");
3) There's a client connected which listens for messages on this socket,
parses the netstring, and sends same data back as netstring to Kamailio.
4) On the event_route[evapi:message-received, I do the following:
xlog("L_INFO", "GOT [$evapi(msg)] from
$evapi(srcaddr):$evapi(srcport)\n");
if($evapi(msg)=~"REGISTER" && $evapi(msg)=~"tindex") {
jansson_get_field("$evapi(msg)", "tlabel",
"$var(tlabel)");
jansson_get_field("$evapi(msg)", "tindex",
"$var(tindex)");
$var(t_index) = $(var(tindex){s.int});
$var(t_label) = $(var(tlabel){s.int});
t_continue('$var(t_index)', '$var(t_label)',
'REGISTER_RESPONSE');
exit;
}
5) On route[REGISTER_RESPONSE], I send 200 OK to SIPp
Now here are detailed logs for which neither SIPp didnt get a response nor
the GOT [$evapi(msg)] from $evapi(srcaddr):$evapi(srcport)\n"); got logged
in the syslog:
First Event:
DEBUG: evapi [evapi_dispatch.c:598]: evapi_relay(): relaying event data
[{"event":"REGISTER","tindex":"26266","tlabel":"587925078","PhoneNumber":"21956","DeviceId":"abcd1234abcd1234","CallId":"
<23021-12910@198.24.63.39>23021-12910@198.24.63.39"}] (146)
DEBUG: evapi [evapi_dispatch.c:623]: evapi_relay(): sending
[0x7f132588de68]
[146:{"event":"REGISTER","tindex":"26266","tlabel":"587925078","PhoneNumber":"21956","DeviceId":"abcd1234abcd1234","CallId":"
<23021-12910@198.24.63.39>23021-12910@198.24.63.39"},] (151)
DEBUG: evapi [evapi_dispatch.c:488]: evapi_recv_notify(): received
[0x7f132588de68]
[146:{"event":"REGISTER","tindex":"26266","tlabel":"587925078","PhoneNumber":"21956","DeviceId":"abcd1234abcd1234","CallId":"
<23021-12910@198.24.63.39>23021-12910@198.24.63.39"},] (151)
NOTICE: evapi [evapi_dispatch.c:290]: evapi_recv_client(): {0} [
198.24.63.45:48881] - received
[146:{"event":"REGISTER","tindex":"26266","tlabel":"587925078","PhoneNumber":"21956","DeviceId":"abcd1234abcd1234","CallId":"
<23021-12910@198.24.63.39>23021-12910@198.24.63.39"},] (151) (18)
Second Event:
DEBUG: evapi [evapi_dispatch.c:598]: evapi_relay(): relaying event data
[{"event":"REGISTER","tindex":"32244","tlabel":"1637923412","PhoneNumber":"25597","DeviceId":"abcd1234abcd1234","CallId":"
<27182-12910@198.24.63.39>27182-12910@198.24.63.39"}] (147)
DEBUG: evapi [evapi_dispatch.c:623]: evapi_relay(): sending
[0x7f1325884410]
[147:{"event":"REGISTER","tindex":"32244","tlabel":"1637923412","PhoneNumber":"25597","DeviceId":"abcd1234abcd1234","CallId":"
<27182-12910@198.24.63.39>27182-12910@198.24.63.39"},] (152)
DEBUG: evapi [evapi_dispatch.c:488]: evapi_recv_notify(): received
[0x7f1325884410]
[147:{"event":"REGISTER","tindex":"32244","tlabel":"1637923412","PhoneNumber":"25597","DeviceId":"abcd1234abcd1234","CallId":"
<27182-12910@198.24.63.39>27182-12910@198.24.63.39"},] (152)
NOTICE: evapi [evapi_dispatch.c:290]: evapi_recv_client(): {0} [
198.24.63.45:48881] - received
[147:{"event":"REGISTER","tindex":"32244","tlabel":"1637923412","PhoneNumber":"25597","DeviceId":"abcd1234abcd1234","CallId":"
<27182-12910@198.24.63.39>27182-12910@198.24.63.39"},] (152) (85)
I had three such events which got missed out of some 25000 odd messages
sent from SIPp. Other major difference that I see in the debug logs for
problematic events are that there is a positive number in the second
parentheses of evapi_recv_client() function. For the events that were
invoked successfully the value of second parentheses is 0 for
evapi_recv_client() function. Hope the debugging helps.
Debug log for a successful event:
DEBUG: evapi [evapi_dispatch.c:598]: evapi_relay(): relaying event data
[{"event":"REGISTER","tindex":"22127","tlabel":"1896682192","PhoneNumber":"73168","DeviceId":"abcd1234abcd1234","CallId":"
<49957-13056@198.24.63.39>49957-13056@198.24.63.39"}] (147)
DEBUG: evapi [evapi_dispatch.c:623]: evapi_relay(): sending
[0x7f132568e850]
[147:{"event":"REGISTER","tindex":"22127","tlabel":"1896682192","PhoneNumber":"73168","DeviceId":"abcd1234abcd1234","CallId":"
<49957-13056@198.24.63.39>49957-13056@198.24.63.39"},] (152)
DEBUG: evapi [evapi_dispatch.c:488]: evapi_recv_notify(): received
[0x7f132568e850]
[147:{"event":"REGISTER","tindex":"22127","tlabel":"1896682192","PhoneNumber":"73168","DeviceId":"abcd1234abcd1234","CallId":"
<49957-13056@198.24.63.39>49957-13056@198.24.63.39"},] (152)
NOTICE: evapi [evapi_dispatch.c:290]: evapi_recv_client(): {0} [
198.24.63.45:48881] - received
[147:{"event":"REGISTER","tindex":"22127","tlabel":"1896682192","PhoneNumber":"73168","DeviceId":"abcd1234abcd1234","CallId":"
<49957-13056@198.24.63.39>49957-13056@198.24.63.39"},] (152) (0)
Thanks,
- Jayesh
On Sat, Sep 19, 2015 at 3:55 AM Daniel-Constantin Mierla <
miconda(a)gmail.com> wrote:
I pushed a commit to add more debug message while
processing received
data. You can use debugger module to set a higher debug level for evapi
module in order to see what happens.
I checked the netstring packet size and it is invalid (unless email
stripped some white chars there) -- for example in:
145:{"event":"REGISTER","tindex":"55567","tlabel":"627458699","PhoneNumber":"20711","DeviceId":"abcd1234abcd1234","CallId":
<21225-3848@5.6.7.8>"21225-3848@5.6.7.8" <21225-3848(a)5.6.7.8>}8>},
the size is 141, not 145 -- it looks like the size includes the size
itself plus the delimiters ':,'. The size is only the effective data, see:
https://en.wikipedia.org/wiki/Netstring
If evapi gets a packet with an invalid size, then it discards the buffer
content.
See if the app on the other side of evapi connection builds netstrings
with wrong size.
Cheers,
Daniel
On 18/09/15 23:19, Jayesh Nambiar wrote:
Here are the tests that I did:
With the patch applied, I see that Kamailio is invoking event
individually for each netstring even when they come in different chunks.
But I did see instances where when there were complete netstrings in a
single chunk; kamailio did not raise an event for them. Here's the
illustration:
T 1.2.3.4:48873 -> 5.6.7.8:3927 [AP]
146:{"event":"REGISTER","tindex":"16916","tlabel":"1753048277","PhoneNumber":"20708","DeviceId":"abcd1234abcd1234","CallId":"
<21220-3848@5.6.7.8>21220-3848@5.6.7.8
"},144:{"event":"REGISTER","tindex":"7954","tlabel":"254315075","PhoneNumber":"20709","DeviceId":"abcd1234abcd1234","CallId":"
<21223-3848@5.6.7.8>21223-3848@5.6.7.8
"},145:{"event":"REGISTER","tindex":"64529","tlabel":"599481568","PhoneNumber":"20709","DeviceId":"abcd1234abcd1234","CallId":"
<21222-3848@5.6.7.8>21222-3848@5.6.7.8"},145:{"event":"REGISTER","tindex":"46605","tlabel":"112015324","PhoneNumber":"20710","DeviceId"
T 1.2.3.4:48873 -> 5.6.7.8:3927 [AP]
:"abcd1234abcd1234","CallId":"
<21224-3848@5.6.7.8>21224-3848@5.6.7.8
"},
*The above two chunks contain 4 proper netstrings where the second chunk
contains part of the 4th netstring. In this case Evapi properly raised 4
individual events.*
1.2.3.4:48873 -> 5.6.7.8:3927 [AP]
145:{"event":"REGISTER","tindex":"55567","tlabel":"627458699","PhoneNumber":"20711","DeviceId":"abcd1234abcd1234","CallId":"
<21225-3848@5.6.7.8>21225-3848@5.6.7.8
"},143:{"event":"REGISTER","tindex":"28682","tlabel":"9676691","PhoneNumber":"20712","DeviceId":"abcd1234abcd1234","CallId":"
<21226-3848@5.6.7.8>21226-3848@5.6.7.8"},
*This above chunk contains two complete netstrings but Kamailio never
raised events for these two netstrings.*
For the events that were not raised I see proper Kamailio Logs which is:
evapi_recv_client(): {0} [1.2.3.4:48873] - received
[145:{"event":"REGISTER","tindex":"55567","tlabel":"627458699","PhoneNumber":"20711","DeviceId":"abcd1234abcd1234","CallId":"
<21225-3848@5.6.7.8>21225-3848@5.6.7.8
"},143:{"event":"REGISTER","tindex":"28682","tlabel":"9676691","PhoneNumber":"20712","DeviceId":"abcd1234abcd1234","CallId":"
<21226-3848@5.6.7.8>21226-3848@5.6.7.8"},]
But I don't see the logs that I've written in the script after the event
was raised, which means Kamailio did not invoke events for these two
netstrings.
So in my tests, out of 27232 messages sent, there were approximately 27
messages for which the events were not raised by Evapi. The rate of
messages started at 500cps and I stopped after I saw missed events at
around 1200cps.
Thanks, and do let me know for any further tests or information required
about the same.
- Jayesh
On Fri, Sep 18, 2015 at 2:55 PM, Jayesh Nambiar < <jayesh1017(a)gmail.com>
jayesh1017(a)gmail.com> wrote:
--
Daniel-Constantin Mierlahttp://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda
Book: SIP Routing With Kamailio - http://www.asipto.com
Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
Will have this tested by tomorrow and will get
back to you. Thanks.
- Jayesh
On Fri, Sep 18, 2015 at 1:41 PM, Daniel-Constantin Mierla <
<miconda@gmail.com>miconda@gmail.com> wrote:
--
Daniel-Constantin Mierlahttp://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda
Book: SIP Routing With Kamailio - http://www.asipto.com
Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
Hello,
I just pushed a patch to master branch that should cope with partial
data received on tcp connection. No time to test at all, therefore any
feedback will be appreciated.
Cheers,
Daniel
On 15/09/15 14:52, Daniel-Constantin Mierla wrote:
Hello,
I will look if there are options in libev to buffer data or try to
implement a buffering mechanism locally for such cases.
Cheers,
Daniel
On 14/09/15 23:00, Jayesh Nambiar wrote:
Hello Daniel,
After further testing with evapi module, I figured that when Netstrings
are used, an event route is invoked individually for each message even if
if multiple netstring messages are received in a single TCP packet. But
this doesn't work effectively when a single proper message is split-up in
two packets. For Example, if a message arrives as:
12:Hello World!, 12:Hello World!, 12:Hello World! in a single packet,
kamailio properly invokes the event route "evapi:message-received" thrice
for every individual proper netstring message.
But if the first packet contains:
12:Hello World!, 12:Hello
And Second Packet contains:
World!, 12:Hello World!
the event route is invoked only once !!
The above pattern is very much possible while sending and receiving
packets over TCP Socket. Our tests for receiving an approximately 150 byte
message over evapi socket at the rate of roughly 1000cps causes a lot of
real events to be missed because of the above problem. You can never be
sure when TCP will split messages in different chunks.
This definitely looks like a bug which makes it not very reliable at
large scale deployments. Would really appreciate your inputs on this.
Thanks;
- Jayesh
On Thu, Sep 10, 2015 at 4:01 PM, Jayesh Nambiar <
<jayesh1017@gmail.com>jayesh1017@gmail.com> wrote:
> Hi Daniel,
> Thanks for the quick response. So if I do not use Netstrings, does
> Kamailio allow me to create a custom logic in the script. For eg. if I
> decide to use newline as a delimiter, can I keep buffering the message
> until I encounter the delimiter from the event route and then execute
> whatever I have to within the script??
>
> Thanks,
>
> - Jayesh
>
> On Thu, Sep 10, 2015 at 1:29 PM, Daniel-Constantin Mierla <
> <miconda@gmail.com>miconda@gmail.com> wrote:
>
>> Hello,
>>
>> tcb is stream protocol and several messages can be queued on the pipe
>> at the same time. That is the reason for netstring format, to be able to
>> easily detect the boundaries of each message. If netstring format is
>> enabled and kamailio receives several messages at once, it splits them and
>> for each is executing the event route.
>>
>> If netstring format is not used, the kamailio is executing the event
>> route with the entire content that was read at once from the tcp connection.
>>
>> Cheers,
>> Daniel
>>
>>
>> On 09/09/15 22:01, Jayesh Nambiar wrote:
>>
>> Hello,
>> I'm exploring the evapi module for my kamailio to interface with an
>> external node.js app for third party stuff like AAA, billing engine tasks,
>> notifications and so on. I followed and took some ideas from the rtjson and
>> evapi tutorial found here(
>> <http://kb.asipto.com/kamailio:k43-async-sip-routing-nodejs>
>> http://kb.asipto.com/kamailio:k43-async-sip-routing-nodejs) to build
>> the node.js app consuming events.
>> When I stress tested the scenario using SIPp and tried sending a lot
>> of events at 300-350cps from Kamailio, I noticed that at times the client
>> is receiving 2-3 events in a single message together although I do
>> event_sync_relay once per SIP message received and have netstrings enabled.
>> I believe this is a typical behavior of TCP and needs to be handled by the
>> client using some kind of Netstring handler. Please correct me if I'm wrong.
>> And hence I'd like to know what particularly needs to be taken care
>> of while writing a client that is listening for events on raw tcp socket
>> and how does kamailio handle this situation while receiving messages over
>> TCP socket?? Does kamailio recognize the end of netstring properly on
>> evapi:message-received and give exactly one message to take care of on
>> every "message-received" event or should that be handled in the script
>> somewhere !!
>> I also referred cgrates client over evapi example which is written in
>> GO, but I couldnt find them handling TCP streams clearly either.
>> I'd really appreciate some expert suggestion here to make an informed
>> decision on using the evapi module for a large scale solution.
>>
>> Thanks,
>>
>> - Jayesh
>>
>>
>>
>>
>> _______________________________________________
>> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing
listsr-users@lists.sip-router.orghttp://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
>>
>>
>> --
>> Daniel-Constantin Mierlahttp://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda
>> Book: SIP Routing With Kamailio - http://www.asipto.com
>> Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
>>
>>
>> _______________________________________________
>> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing
>> list
>> <sr-users@lists.sip-router.org>sr-users@lists.sip-router.org
>> <http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users>
>> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
>>
>>
>
--
Daniel-Constantin Mierlahttp://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda
Book: SIP Routing With Kamailio - http://www.asipto.com
Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
--
Daniel-Constantin Mierlahttp://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda
Book: SIP Routing With Kamailio - http://www.asipto.com
Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
3:12 a.m.
Also forgot to mention, this starts happening at around 1200cps. And using
SIPp I accelerate from 500cps to 1200cps very quick. So it might be related
to concurrency is what I'm assuming. Thanks.
- Jayesh
On Tue, Sep 22, 2015 at 12:40 PM Jayesh Nambiar <jayesh1017(a)gmail.com>
wrote:
Hi,
Took time to get back with the test results as I was trying to analyze too
much myself before getting back on the list. Also I was stuck with weird
error of module version mismatch, but I figured and solved that issue.
So I do see the logs "frame size mismatch" when I miss the events. Since
the logs are too huge, I am sending it in pastebin. Here it is:
http://pastebin.com/MzTpYsrk
In Kamailio Logs, I see:
DEBUG: evapi [evapi_dispatch.c:361]: evapi_recv_client(): frame size
mismatch the ending char
(h):[{"event":"REGISTER","tindex":"12603","tlabel":"626010915
","PhoneNumber":"42000","D146:"event":"REGISTER","tindex":"3639","tlabel":"
1889788252","P] (146)
Whereas the messages in ngrep looks good where the messages are split in
two TCP packets. The first packet ends at D and the second packet is the
proper continuation which send the rest. But it feels Kamailio skipped the
beginning of the message and started parsing at colon again and it thinks
the netstring started with D146 which is incorrect. Hence, it never emitted
any event for the messages in the first chunk.
Hoping this to be helpful !! Thanks.
- Jayesh
On Mon, Sep 21, 2015 at 4:37 PM Daniel-Constantin Mierla <
miconda(a)gmail.com> wrote:
> Hello,
>
> I pushed another commit to add more debug messages to see if the event
> route is supposed to be executed or not. Can you run the tests again and
> give again the log messages for missing event route executions?
>
> Cheers,
> Daniel
>
>
> On 21/09/15 09:04, Jayesh Nambiar wrote:
>
> Hi Daniel,
> I have captured debug messages after testing with your latest commit on
> master. Once again, here's what I'm doing exactly for my test:
> 1) Use SIPp to send OPTIONS towards Kamailio.
> 2) On getting options send event as follows:
>
>
evapi_async_relay("{\"event\":\"REGISTER\",\"tindex\":\"$T(id_index)\",\"tlabel\":\"$T(id_label)\",\"PhoneNumber\":\"$avp(phone_number)\",\"DeviceId\":\"$avp(device_id)\",\"CallId\":\"$ci\"}");
> 3) There's a client connected which listens for messages on this socket,
> parses the netstring, and sends same data back as netstring to Kamailio.
> 4) On the event_route[evapi:message-received, I do the following:
> xlog("L_INFO", "GOT [$evapi(msg)] from
> $evapi(srcaddr):$evapi(srcport)\n");
> if($evapi(msg)=~"REGISTER" && $evapi(msg)=~"tindex")
{
> jansson_get_field("$evapi(msg)", "tlabel",
"$var(tlabel)");
> jansson_get_field("$evapi(msg)", "tindex",
"$var(tindex)");
> $var(t_index) = $(var(tindex){s.int});
> $var(t_label) = $(var(tlabel){s.int});
> t_continue('$var(t_index)', '$var(t_label)',
'REGISTER_RESPONSE');
> exit;
> }
> 5) On route[REGISTER_RESPONSE], I send 200 OK to SIPp
>
> Now here are detailed logs for which neither SIPp didnt get a response
> nor the GOT [$evapi(msg)] from $evapi(srcaddr):$evapi(srcport)\n"); got
> logged in the syslog:
>
> First Event:
> DEBUG: evapi [evapi_dispatch.c:598]: evapi_relay(): relaying event data
>
[{"event":"REGISTER","tindex":"26266","tlabel":"587925078","PhoneNumber":"21956","DeviceId":"abcd1234abcd1234","CallId":"
> <23021-12910@198.24.63.39>23021-12910@198.24.63.39"}] (146)
> DEBUG: evapi [evapi_dispatch.c:623]: evapi_relay(): sending
> [0x7f132588de68]
>
[146:{"event":"REGISTER","tindex":"26266","tlabel":"587925078","PhoneNumber":"21956","DeviceId":"abcd1234abcd1234","CallId":"
> <23021-12910@198.24.63.39>23021-12910@198.24.63.39"},] (151)
>
> DEBUG: evapi [evapi_dispatch.c:488]: evapi_recv_notify(): received
> [0x7f132588de68]
>
[146:{"event":"REGISTER","tindex":"26266","tlabel":"587925078","PhoneNumber":"21956","DeviceId":"abcd1234abcd1234","CallId":"
> <23021-12910@198.24.63.39>23021-12910@198.24.63.39"},] (151)
> NOTICE: evapi [evapi_dispatch.c:290]: evapi_recv_client(): {0} [
> 198.24.63.45:48881] - received
>
[146:{"event":"REGISTER","tindex":"26266","tlabel":"587925078","PhoneNumber":"21956","DeviceId":"abcd1234abcd1234","CallId":"
> <23021-12910@198.24.63.39>23021-12910@198.24.63.39"},] (151) (18)
>
> Second Event:
> DEBUG: evapi [evapi_dispatch.c:598]: evapi_relay(): relaying event data
>
[{"event":"REGISTER","tindex":"32244","tlabel":"1637923412","PhoneNumber":"25597","DeviceId":"abcd1234abcd1234","CallId":"
> <27182-12910@198.24.63.39>27182-12910@198.24.63.39"}] (147)
> DEBUG: evapi [evapi_dispatch.c:623]: evapi_relay(): sending
> [0x7f1325884410]
>
[147:{"event":"REGISTER","tindex":"32244","tlabel":"1637923412","PhoneNumber":"25597","DeviceId":"abcd1234abcd1234","CallId":"
> <27182-12910@198.24.63.39>27182-12910@198.24.63.39"},] (152)
>
> DEBUG: evapi [evapi_dispatch.c:488]: evapi_recv_notify(): received
> [0x7f1325884410]
>
[147:{"event":"REGISTER","tindex":"32244","tlabel":"1637923412","PhoneNumber":"25597","DeviceId":"abcd1234abcd1234","CallId":"
> <27182-12910@198.24.63.39>27182-12910@198.24.63.39"},] (152)
> NOTICE: evapi [evapi_dispatch.c:290]: evapi_recv_client(): {0} [
> 198.24.63.45:48881] - received
>
[147:{"event":"REGISTER","tindex":"32244","tlabel":"1637923412","PhoneNumber":"25597","DeviceId":"abcd1234abcd1234","CallId":"
> <27182-12910@198.24.63.39>27182-12910@198.24.63.39"},] (152) (85)
>
> I had three such events which got missed out of some 25000 odd messages
> sent from SIPp. Other major difference that I see in the debug logs for
> problematic events are that there is a positive number in the second
> parentheses of evapi_recv_client() function. For the events that were
> invoked successfully the value of second parentheses is 0 for
> evapi_recv_client() function. Hope the debugging helps.
>
> Debug log for a successful event:
> DEBUG: evapi [evapi_dispatch.c:598]: evapi_relay(): relaying event data
>
[{"event":"REGISTER","tindex":"22127","tlabel":"1896682192","PhoneNumber":"73168","DeviceId":"abcd1234abcd1234","CallId":"
> <49957-13056@198.24.63.39>49957-13056@198.24.63.39"}] (147)
> DEBUG: evapi [evapi_dispatch.c:623]: evapi_relay(): sending
> [0x7f132568e850]
>
[147:{"event":"REGISTER","tindex":"22127","tlabel":"1896682192","PhoneNumber":"73168","DeviceId":"abcd1234abcd1234","CallId":"
> <49957-13056@198.24.63.39>49957-13056@198.24.63.39"},] (152)
>
> DEBUG: evapi [evapi_dispatch.c:488]: evapi_recv_notify(): received
> [0x7f132568e850]
>
[147:{"event":"REGISTER","tindex":"22127","tlabel":"1896682192","PhoneNumber":"73168","DeviceId":"abcd1234abcd1234","CallId":"
> <49957-13056@198.24.63.39>49957-13056@198.24.63.39"},] (152)
> NOTICE: evapi [evapi_dispatch.c:290]: evapi_recv_client(): {0} [
> 198.24.63.45:48881] - received
>
[147:{"event":"REGISTER","tindex":"22127","tlabel":"1896682192","PhoneNumber":"73168","DeviceId":"abcd1234abcd1234","CallId":"
> <49957-13056@198.24.63.39>49957-13056@198.24.63.39"},] (152) (0)
>
> Thanks,
>
> - Jayesh
> On Sat, Sep 19, 2015 at 3:55 AM Daniel-Constantin Mierla <
> miconda(a)gmail.com> wrote:
>
>> I pushed a commit to add more debug message while processing received
>> data. You can use debugger module to set a higher debug level for evapi
>> module in order to see what happens.
>>
>> I checked the netstring packet size and it is invalid (unless email
>> stripped some white chars there) -- for example in:
>>
>>
>>
>>
145:{"event":"REGISTER","tindex":"55567","tlabel":"627458699","PhoneNumber":"20711","DeviceId":"abcd1234abcd1234","CallId":
>> <21225-3848@5.6.7.8>"21225-3848@5.6.7.8"
<21225-3848(a)5.6.7.8>}8>},
>>
>> the size is 141, not 145 -- it looks like the size includes the size
>> itself plus the delimiters ':,'. The size is only the effective data,
see:
>>
>> https://en.wikipedia.org/wiki/Netstring
>>
>> If evapi gets a packet with an invalid size, then it discards the buffer
>> content.
>>
>> See if the app on the other side of evapi connection builds netstrings
>> with wrong size.
>>
>> Cheers,
>> Daniel
>>
>>
>> On 18/09/15 23:19, Jayesh Nambiar wrote:
>>
>> Here are the tests that I did:
>>
>> With the patch applied, I see that Kamailio is invoking event
>> individually for each netstring even when they come in different chunks.
>> But I did see instances where when there were complete netstrings in a
>> single chunk; kamailio did not raise an event for them. Here's the
>> illustration:
>>
>> T 1.2.3.4:48873 -> 5.6.7.8:3927 [AP]
>>
>>
146:{"event":"REGISTER","tindex":"16916","tlabel":"1753048277","PhoneNumber":"20708","DeviceId":"abcd1234abcd1234","CallId":"
>> <21220-3848@5.6.7.8>21220-3848@5.6.7.8
>>
"},144:{"event":"REGISTER","tindex":"7954","tlabel":"254315075","PhoneNumber":"20709","DeviceId":"abcd1234abcd1234","CallId":"
>> <21223-3848@5.6.7.8>21223-3848@5.6.7.8
>>
"},145:{"event":"REGISTER","tindex":"64529","tlabel":"599481568","PhoneNumber":"20709","DeviceId":"abcd1234abcd1234","CallId":"
>>
<21222-3848@5.6.7.8>21222-3848@5.6.7.8"},145:{"event":"REGISTER","tindex":"46605","tlabel":"112015324","PhoneNumber":"20710","DeviceId"
>>
>>
>>
>>
>> T 1.2.3.4:48873 -> 5.6.7.8:3927 [AP]
>> :"abcd1234abcd1234","CallId":"
<21224-3848@5.6.7.8>21224-3848@5.6.7.8
>> "},
>>
>> *The above two chunks contain 4 proper netstrings where the second chunk
>> contains part of the 4th netstring. In this case Evapi properly raised 4
>> individual events.*
>>
>> 1.2.3.4:48873 -> 5.6.7.8:3927 [AP]
>>
>>
145:{"event":"REGISTER","tindex":"55567","tlabel":"627458699","PhoneNumber":"20711","DeviceId":"abcd1234abcd1234","CallId":"
>> <21225-3848@5.6.7.8>21225-3848@5.6.7.8
>>
"},143:{"event":"REGISTER","tindex":"28682","tlabel":"9676691","PhoneNumber":"20712","DeviceId":"abcd1234abcd1234","CallId":"
>> <21226-3848@5.6.7.8>21226-3848@5.6.7.8"},
>>
>> *This above chunk contains two complete netstrings but Kamailio never
>> raised events for these two netstrings.*
>>
>> For the events that were not raised I see proper Kamailio Logs which is:
>> evapi_recv_client(): {0} [1.2.3.4:48873] - received
>>
[145:{"event":"REGISTER","tindex":"55567","tlabel":"627458699","PhoneNumber":"20711","DeviceId":"abcd1234abcd1234","CallId":"
>> <21225-3848@5.6.7.8>21225-3848@5.6.7.8
>>
"},143:{"event":"REGISTER","tindex":"28682","tlabel":"9676691","PhoneNumber":"20712","DeviceId":"abcd1234abcd1234","CallId":"
>> <21226-3848@5.6.7.8>21226-3848@5.6.7.8"},]
>>
>> But I don't see the logs that I've written in the script after the event
>> was raised, which means Kamailio did not invoke events for these two
>> netstrings.
>>
>> So in my tests, out of 27232 messages sent, there were approximately 27
>> messages for which the events were not raised by Evapi. The rate of
>> messages started at 500cps and I stopped after I saw missed events at
>> around 1200cps.
>>
>> Thanks, and do let me know for any further tests or information required
>> about the same.
>>
>> - Jayesh
>>
>>
>>
>> On Fri, Sep 18, 2015 at 2:55 PM, Jayesh Nambiar <
<jayesh1017(a)gmail.com>
>> jayesh1017(a)gmail.com> wrote:
>>
>>> Will have this tested by tomorrow and will get back to you. Thanks.
>>>
>>> - Jayesh
>>>
>>> On Fri, Sep 18, 2015 at 1:41 PM, Daniel-Constantin Mierla <
>>> <miconda@gmail.com>miconda@gmail.com> wrote:
>>>
>>>> Hello,
>>>>
>>>> I just pushed a patch to master branch that should cope with partial
>>>> data received on tcp connection. No time to test at all, therefore any
>>>> feedback will be appreciated.
>>>>
>>>> Cheers,
>>>> Daniel
>>>>
>>>>
>>>> On 15/09/15 14:52, Daniel-Constantin Mierla wrote:
>>>>
>>>> Hello,
>>>>
>>>> I will look if there are options in libev to buffer data or try to
>>>> implement a buffering mechanism locally for such cases.
>>>>
>>>> Cheers,
>>>> Daniel
>>>>
>>>> On 14/09/15 23:00, Jayesh Nambiar wrote:
>>>>
>>>> Hello Daniel,
>>>> After further testing with evapi module, I figured that when
>>>> Netstrings are used, an event route is invoked individually for each
>>>> message even if if multiple netstring messages are received in a single
TCP
>>>> packet. But this doesn't work effectively when a single proper
message is
>>>> split-up in two packets. For Example, if a message arrives as:
>>>> 12:Hello World!, 12:Hello World!, 12:Hello World! in a single packet,
>>>> kamailio properly invokes the event route
"evapi:message-received" thrice
>>>> for every individual proper netstring message.
>>>> But if the first packet contains:
>>>> 12:Hello World!, 12:Hello
>>>> And Second Packet contains:
>>>> World!, 12:Hello World!
>>>> the event route is invoked only once !!
>>>>
>>>> The above pattern is very much possible while sending and receiving
>>>> packets over TCP Socket. Our tests for receiving an approximately 150
byte
>>>> message over evapi socket at the rate of roughly 1000cps causes a lot of
>>>> real events to be missed because of the above problem. You can never be
>>>> sure when TCP will split messages in different chunks.
>>>> This definitely looks like a bug which makes it not very reliable at
>>>> large scale deployments. Would really appreciate your inputs on this.
>>>> Thanks;
>>>>
>>>> - Jayesh
>>>>
>>>>
>>>>
>>>> On Thu, Sep 10, 2015 at 4:01 PM, Jayesh Nambiar <
>>>> <jayesh1017@gmail.com>jayesh1017@gmail.com> wrote:
>>>>
>>>>> Hi Daniel,
>>>>> Thanks for the quick response. So if I do not use Netstrings, does
>>>>> Kamailio allow me to create a custom logic in the script. For eg. if
I
>>>>> decide to use newline as a delimiter, can I keep buffering the
message
>>>>> until I encounter the delimiter from the event route and then
execute
>>>>> whatever I have to within the script??
>>>>>
>>>>> Thanks,
>>>>>
>>>>> - Jayesh
>>>>>
>>>>> On Thu, Sep 10, 2015 at 1:29 PM, Daniel-Constantin Mierla <
>>>>> <miconda@gmail.com>miconda@gmail.com> wrote:
>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> tcb is stream protocol and several messages can be queued on the
>>>>>> pipe at the same time. That is the reason for netstring format,
to be able
>>>>>> to easily detect the boundaries of each message. If netstring
format is
>>>>>> enabled and kamailio receives several messages at once, it splits
them and
>>>>>> for each is executing the event route.
>>>>>>
>>>>>> If netstring format is not used, the kamailio is executing the
event
>>>>>> route with the entire content that was read at once from the tcp
connection.
>>>>>>
>>>>>> Cheers,
>>>>>> Daniel
>>>>>>
>>>>>>
>>>>>> On 09/09/15 22:01, Jayesh Nambiar wrote:
>>>>>>
>>>>>> Hello,
>>>>>> I'm exploring the evapi module for my kamailio to interface
with an
>>>>>> external node.js app for third party stuff like AAA, billing
engine tasks,
>>>>>> notifications and so on. I followed and took some ideas from the
rtjson and
>>>>>> evapi tutorial found here(
>>>>>>
<http://kb.asipto.com/kamailio:k43-async-sip-routing-nodejs>
>>>>>> http://kb.asipto.com/kamailio:k43-async-sip-routing-nodejs) to
>>>>>> build the node.js app consuming events.
>>>>>> When I stress tested the scenario using SIPp and tried sending a
lot
>>>>>> of events at 300-350cps from Kamailio, I noticed that at times
the client
>>>>>> is receiving 2-3 events in a single message together although I
do
>>>>>> event_sync_relay once per SIP message received and have
netstrings enabled.
>>>>>> I believe this is a typical behavior of TCP and needs to be
handled by the
>>>>>> client using some kind of Netstring handler. Please correct me if
I'm wrong.
>>>>>> And hence I'd like to know what particularly needs to be
taken care
>>>>>> of while writing a client that is listening for events on raw tcp
socket
>>>>>> and how does kamailio handle this situation while receiving
messages over
>>>>>> TCP socket?? Does kamailio recognize the end of netstring
properly on
>>>>>> evapi:message-received and give exactly one message to take care
of on
>>>>>> every "message-received" event or should that be
handled in the script
>>>>>> somewhere !!
>>>>>> I also referred cgrates client over evapi example which is
written
>>>>>> in GO, but I couldnt find them handling TCP streams clearly
either.
>>>>>> I'd really appreciate some expert suggestion here to make an
>>>>>> informed decision on using the evapi module for a large scale
solution.
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> - Jayesh
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users
mailing
listsr-users@lists.sip-router.orghttp://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Daniel-Constantin Mierlahttp://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda
>>>>>> Book: SIP Routing With Kamailio - http://www.asipto.com
>>>>>> Kamailio Advanced Training, Sep 28-30, 2015, in Berlin -
http://asipto.com/u/kat
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users
mailing
>>>>>> list
>>>>>>
<sr-users@lists.sip-router.org>sr-users@lists.sip-router.org
>>>>>>
<http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users>
>>>>>> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
>>>>>>
>>>>>>
>>>>>
>>>>
>>>> --
>>>> Daniel-Constantin Mierlahttp://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda
>>>> Book: SIP Routing With Kamailio - http://www.asipto.com
>>>> Kamailio Advanced Training, Sep 28-30, 2015, in Berlin -
http://asipto.com/u/kat
>>>>
>>>>
>>>> --
>>>> Daniel-Constantin Mierlahttp://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda
>>>> Book: SIP Routing With Kamailio - http://www.asipto.com
>>>> Kamailio Advanced Training, Sep 28-30, 2015, in Berlin -
http://asipto.com/u/kat
>>>>
>>>>
>>>
>>
>> --
>> Daniel-Constantin Mierlahttp://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda
>> Book: SIP Routing With Kamailio - http://www.asipto.com
>> Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
>>
>>
> --
> Daniel-Constantin Mierlahttp://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda
> Book: SIP Routing With Kamailio - http://www.asipto.com
> Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
>
>
7:33 a.m.
Hello,
I haven't seen in the ngrep when the data containing next line is received:
{"event":"REGISTER","tindex":"3639","tlabel":"1889788252"
...
Do you still have it? Is it after:
{"event":"REGISTER","tindex":"12603","tlabel":"626010915",
...
Cheers,
Daniel
On 22/09/15 09:09, Jayesh Nambiar wrote:
Hi,
Took time to get back with the test results as I was trying to analyze
too much myself before getting back on the list. Also I was stuck with
weird error of module version mismatch, but I figured and solved that
issue.
So I do see the logs "frame size mismatch" when I miss the events.
Since the logs are too huge, I am sending it in pastebin. Here it is:
http://pastebin.com/MzTpYsrk
In Kamailio Logs, I see:
DEBUG: evapi [evapi_dispatch.c:361]: evapi_recv_client(): frame size
mismatch the ending char
(h):[{"event":"REGISTER","tindex":"12603","tlabel":"626010915","PhoneNumber":"42000","D146:"event":"REGISTER","tindex":"3639","tlabel":"1889788252","P]
(146)
Whereas the messages in ngrep looks good where the messages are split
in two TCP packets. The first packet ends at D and the second packet
is the proper continuation which send the rest. But it feels Kamailio
skipped the beginning of the message and started parsing at colon
again and it thinks the netstring started with D146 which is
incorrect. Hence, it never emitted any event for the messages in the
first chunk.
Hoping this to be helpful !! Thanks.
- Jayesh
On Mon, Sep 21, 2015 at 4:37 PM Daniel-Constantin Mierla
<miconda(a)gmail.com <mailto:miconda@gmail.com>> wrote:
Hello,
I pushed another commit to add more debug messages to see if the
event route is supposed to be executed or not. Can you run the
tests again and give again the log messages for missing event
route executions?
Cheers,
Daniel
On 21/09/15 09:04, Jayesh Nambiar wrote:
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Book: SIP Routing With Kamailio - http://www.asipto.com
Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
Hi Daniel,
I have captured debug messages after testing with your latest
commit on master. Once again, here's what I'm doing exactly for
my test:
1) Use SIPp to send OPTIONS towards Kamailio.
2) On getting options send event as follows:
evapi_async_relay("{\"event\":\"REGISTER\",\"tindex\":\"$T(id_index)\",\"tlabel\":\"$T(id_label)\",\"PhoneNumber\":\"$avp(phone_number)\",\"DeviceId\":\"$avp(device_id)\",\"CallId\":\"$ci\"}");
3) There's a client connected which listens for messages on this
socket, parses the netstring, and sends same data back as
netstring to Kamailio.
4) On the event_route[evapi:message-received, I do the following:
xlog("L_INFO", "GOT [$evapi(msg)] from
$evapi(srcaddr):$evapi(srcport)\n");
if($evapi(msg)=~"REGISTER" && $evapi(msg)=~"tindex")
{
jansson_get_field("$evapi(msg)", "tlabel",
"$var(tlabel)");
jansson_get_field("$evapi(msg)", "tindex",
"$var(tindex)");
$var(t_index) = $(var(tindex){s.int <http://s.int>});
$var(t_label) = $(var(tlabel){s.int <http://s.int>});
t_continue('$var(t_index)', '$var(t_label)',
'REGISTER_RESPONSE');
exit;
}
5) On route[REGISTER_RESPONSE], I send 200 OK to SIPp
Now here are detailed logs for which neither SIPp didnt get a
response nor the GOT [$evapi(msg)] from
$evapi(srcaddr):$evapi(srcport)\n"); got logged in the syslog:
First Event:
DEBUG: evapi [evapi_dispatch.c:598]: evapi_relay(): relaying
event data
[{"event":"REGISTER","tindex":"26266","tlabel":"587925078","PhoneNumber":"21956","DeviceId":"abcd1234abcd1234","CallId":"23021-12910@198.24.63.39
<mailto:23021-12910@198.24.63.39>"}] (146)
DEBUG: evapi [evapi_dispatch.c:623]: evapi_relay(): sending
[0x7f132588de68]
[146:{"event":"REGISTER","tindex":"26266","tlabel":"587925078","PhoneNumber":"21956","DeviceId":"abcd1234abcd1234","CallId":"23021-12910@198.24.63.39
<mailto:23021-12910@198.24.63.39>"},] (151)
DEBUG: evapi [evapi_dispatch.c:488]: evapi_recv_notify():
received [0x7f132588de68]
[146:{"event":"REGISTER","tindex":"26266","tlabel":"587925078","PhoneNumber":"21956","DeviceId":"abcd1234abcd1234","CallId":"23021-12910@198.24.63.39
<mailto:23021-12910@198.24.63.39>"},] (151)
NOTICE: evapi [evapi_dispatch.c:290]: evapi_recv_client(): {0}
[198.24.63.45:48881 <http://198.24.63.45:48881>] - received
[146:{"event":"REGISTER","tindex":"26266","tlabel":"587925078","PhoneNumber":"21956","DeviceId":"abcd1234abcd1234","CallId":"23021-12910@198.24.63.39
<mailto:23021-12910@198.24.63.39>"},] (151) (18)
Second Event:
DEBUG: evapi [evapi_dispatch.c:598]: evapi_relay(): relaying
event data
[{"event":"REGISTER","tindex":"32244","tlabel":"1637923412","PhoneNumber":"25597","DeviceId":"abcd1234abcd1234","CallId":"27182-12910@198.24.63.39
<mailto:27182-12910@198.24.63.39>"}] (147)
DEBUG: evapi [evapi_dispatch.c:623]: evapi_relay(): sending
[0x7f1325884410]
[147:{"event":"REGISTER","tindex":"32244","tlabel":"1637923412","PhoneNumber":"25597","DeviceId":"abcd1234abcd1234","CallId":"27182-12910@198.24.63.39
<mailto:27182-12910@198.24.63.39>"},] (152)
DEBUG: evapi [evapi_dispatch.c:488]: evapi_recv_notify():
received [0x7f1325884410]
[147:{"event":"REGISTER","tindex":"32244","tlabel":"1637923412","PhoneNumber":"25597","DeviceId":"abcd1234abcd1234","CallId":"27182-12910@198.24.63.39
<mailto:27182-12910@198.24.63.39>"},] (152)
NOTICE: evapi [evapi_dispatch.c:290]: evapi_recv_client(): {0}
[198.24.63.45:48881 <http://198.24.63.45:48881>] - received
[147:{"event":"REGISTER","tindex":"32244","tlabel":"1637923412","PhoneNumber":"25597","DeviceId":"abcd1234abcd1234","CallId":"27182-12910@198.24.63.39
<mailto:27182-12910@198.24.63.39>"},] (152) (85)
I had three such events which got missed out of some 25000 odd
messages sent from SIPp. Other major difference that I see in the
debug logs for problematic events are that there is a positive
number in the second parentheses of evapi_recv_client() function.
For the events that were invoked successfully the value of second
parentheses is 0 for evapi_recv_client() function. Hope the
debugging helps.
Debug log for a successful event:
DEBUG: evapi [evapi_dispatch.c:598]: evapi_relay(): relaying
event data
[{"event":"REGISTER","tindex":"22127","tlabel":"1896682192","PhoneNumber":"73168","DeviceId":"abcd1234abcd1234","CallId":"49957-13056@198.24.63.39
<mailto:49957-13056@198.24.63.39>"}] (147)
DEBUG: evapi [evapi_dispatch.c:623]: evapi_relay(): sending
[0x7f132568e850]
[147:{"event":"REGISTER","tindex":"22127","tlabel":"1896682192","PhoneNumber":"73168","DeviceId":"abcd1234abcd1234","CallId":"49957-13056@198.24.63.39
<mailto:49957-13056@198.24.63.39>"},] (152)
DEBUG: evapi [evapi_dispatch.c:488]: evapi_recv_notify():
received [0x7f132568e850]
[147:{"event":"REGISTER","tindex":"22127","tlabel":"1896682192","PhoneNumber":"73168","DeviceId":"abcd1234abcd1234","CallId":"49957-13056@198.24.63.39
<mailto:49957-13056@198.24.63.39>"},] (152)
NOTICE: evapi [evapi_dispatch.c:290]: evapi_recv_client(): {0}
[198.24.63.45:48881 <http://198.24.63.45:48881>] - received
[147:{"event":"REGISTER","tindex":"22127","tlabel":"1896682192","PhoneNumber":"73168","DeviceId":"abcd1234abcd1234","CallId":"49957-13056@198.24.63.39
<mailto:49957-13056@198.24.63.39>"},] (152) (0)
Thanks,
- Jayesh
On Sat, Sep 19, 2015 at 3:55 AM Daniel-Constantin Mierla
<miconda(a)gmail.com <mailto:miconda@gmail.com>> wrote:
I pushed a commit to add more debug message while processing
received data. You can use debugger module to set a higher
debug level for evapi module in order to see what happens.
I checked the netstring packet size and it is invalid (unless
email stripped some white chars there) -- for example in:
145:{"event":"REGISTER","tindex":"55567","tlabel":"627458699","PhoneNumber":"20711","DeviceId":"abcd1234abcd1234","CallId":"21225-3848@5.6.7.8"
<mailto:21225-3848@5.6.7.8>},
the size is 141, not 145 -- it looks like the size includes
the size itself plus the delimiters ':,'. The size is only
the effective data, see:
https://en.wikipedia.org/wiki/Netstring
If evapi gets a packet with an invalid size, then it discards
the buffer content.
See if the app on the other side of evapi connection builds
netstrings with wrong size.
Cheers,
Daniel
On 18/09/15 23:19, Jayesh Nambiar wrote:
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> -
http://www.linkedin.com/in/miconda
Book: SIP Routing With Kamailio - http://www.asipto.com
Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
Here are the tests that I did:
With the patch applied, I see that Kamailio is invoking
event individually for each netstring even when they come in
different chunks. But I did see instances where when there
were complete netstrings in a single chunk; kamailio did not
raise an event for them. Here's the illustration:
T 1.2.3.4:48873 <http://1.2.3.4:48873> -> 5.6.7.8:3927
<http://5.6.7.8:3927> [AP]
146:{"event":"REGISTER","tindex":"16916","tlabel":"1753048277","PhoneNumber":"20708","DeviceId":"abcd1234abcd1234","CallId":"21220-3848@5.6.7.8
<mailto:21220-3848@5.6.7.8>"},144:{"event":"REGISTER","tindex":"7954","tlabel":"254315075","PhoneNumber":"20709","DeviceId":"abcd1234abcd1234","CallId":"21223-3848@5.6.7.8
<mailto:21223-3848@5.6.7.8>"},145:{"event":"REGISTER","tindex":"64529","tlabel":"599481568","PhoneNumber":"20709","DeviceId":"abcd1234abcd1234","CallId":"21222-3848@5.6.7.8
<mailto:21222-3848@5.6.7.8>"},145:{"event":"REGISTER","tindex":"46605","tlabel":"112015324","PhoneNumber":"20710","DeviceId"
T 1.2.3.4:48873 <http://1.2.3.4:48873> -> 5.6.7.8:3927
<http://5.6.7.8:3927> [AP]
:"abcd1234abcd1234","CallId":"21224-3848@5.6.7.8
<mailto:21224-3848@5.6.7.8>"},
*The above two chunks contain 4 proper netstrings where the
second chunk contains part of the 4th netstring. In this
case Evapi properly raised 4 individual events.*
1.2.3.4:48873 <http://1.2.3.4:48873> -> 5.6.7.8:3927
<http://5.6.7.8:3927> [AP]
145:{"event":"REGISTER","tindex":"55567","tlabel":"627458699","PhoneNumber":"20711","DeviceId":"abcd1234abcd1234","CallId":"21225-3848@5.6.7.8
<mailto:21225-3848@5.6.7.8>"},143:{"event":"REGISTER","tindex":"28682","tlabel":"9676691","PhoneNumber":"20712","DeviceId":"abcd1234abcd1234","CallId":"21226-3848@5.6.7.8
<mailto:21226-3848@5.6.7.8>"},
*This above chunk contains two complete netstrings but
Kamailio never raised events for these two netstrings.*
*
*
For the events that were not raised I see proper Kamailio
Logs which is:
evapi_recv_client(): {0} [1.2.3.4:48873
<http://1.2.3.4:48873>] - received
[145:{"event":"REGISTER","tindex":"55567","tlabel":"627458699","PhoneNumber":"20711","DeviceId":"abcd1234abcd1234","CallId":"21225-3848@5.6.7.8
<mailto:21225-3848@5.6.7.8>"},143:{"event":"REGISTER","tindex":"28682","tlabel":"9676691","PhoneNumber":"20712","DeviceId":"abcd1234abcd1234","CallId":"21226-3848@5.6.7.8
<mailto:21226-3848@5.6.7.8>"},]
*
*
But I don't see the logs that I've written in the script
after the event was raised, which means Kamailio did not
invoke events for these two netstrings.
So in my tests, out of 27232 messages sent, there were
approximately 27 messages for which the events were not
raised by Evapi. The rate of messages started at 500cps and
I stopped after I saw missed events at around 1200cps.
Thanks, and do let me know for any further tests or
information required about the same.
- Jayesh
*
*
On Fri, Sep 18, 2015 at 2:55 PM, Jayesh Nambiar
<jayesh1017(a)gmail.com <mailto:jayesh1017@gmail.com>> wrote:
Will have this tested by tomorrow and will get back to
you. Thanks.
- Jayesh
On Fri, Sep 18, 2015 at 1:41 PM, Daniel-Constantin
Mierla <miconda(a)gmail.com <mailto:miconda@gmail.com>> wrote:
Hello,
I just pushed a patch to master branch that should
cope with partial data received on tcp connection.
No time to test at all, therefore any feedback will
be appreciated.
Cheers,
Daniel
On 15/09/15 14:52, Daniel-Constantin Mierla wrote:
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> -
http://www.linkedin.com/in/miconda
Book: SIP Routing With Kamailio - http://www.asipto.com
Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
Hello,
I will look if there are options in libev to buffer
data or try to implement a buffering mechanism
locally for such cases.
Cheers,
Daniel
On 14/09/15 23:00, Jayesh Nambiar wrote:
> Hello Daniel,
> After further testing with evapi module, I figured
> that when Netstrings are used, an event route is
> invoked individually for each message even if if
> multiple netstring messages are received in a
> single TCP packet. But this doesn't work
> effectively when a single proper message is
> split-up in two packets. For Example, if a message
> arrives as:
> 12:Hello World!, 12:Hello World!, 12:Hello World!
> in a single packet, kamailio properly invokes the
> event route "evapi:message-received" thrice for
> every individual proper netstring message.
> But if the first packet contains:
> 12:Hello World!, 12:Hello
> And Second Packet contains:
> World!, 12:Hello World!
> the event route is invoked only once !!
>
> The above pattern is very much possible while
> sending and receiving packets over TCP Socket. Our
> tests for receiving an approximately 150 byte
> message over evapi socket at the rate of roughly
> 1000cps causes a lot of real events to be missed
> because of the above problem. You can never be
> sure when TCP will split messages in different chunks.
> This definitely looks like a bug which makes it
> not very reliable at large scale deployments.
> Would really appreciate your inputs on this. Thanks;
>
> - Jayesh
>
>
>
> On Thu, Sep 10, 2015 at 4:01 PM, Jayesh Nambiar
> <jayesh1017(a)gmail.com
> <mailto:jayesh1017@gmail.com>> wrote:
>
> Hi Daniel,
> Thanks for the quick response. So if I do not
> use Netstrings, does Kamailio allow me to
> create a custom logic in the script. For eg.
> if I decide to use newline as a delimiter, can
> I keep buffering the message until I encounter
> the delimiter from the event route and then
> execute whatever I have to within the script??
>
> Thanks,
>
> - Jayesh
>
> On Thu, Sep 10, 2015 at 1:29 PM,
> Daniel-Constantin Mierla <miconda(a)gmail.com
> <mailto:miconda@gmail.com>> wrote:
>
> Hello,
>
> tcb is stream protocol and several
> messages can be queued on the pipe at the
> same time. That is the reason for
> netstring format, to be able to easily
> detect the boundaries of each message. If
> netstring format is enabled and kamailio
> receives several messages at once, it
> splits them and for each is executing the
> event route.
>
> If netstring format is not used, the
> kamailio is executing the event route with
> the entire content that was read at once
> from the tcp connection.
>
> Cheers,
> Daniel
>
>
> On 09/09/15 22:01, Jayesh Nambiar wrote:
>> Hello,
>> I'm exploring the evapi module for my
>> kamailio to interface with an external
>> node.js app for third party stuff like
>> AAA, billing engine tasks, notifications
>> and so on. I followed and took some ideas
>> from the rtjson and evapi tutorial found
>>
here(http://kb.asipto.com/kamailio:k43-async-sip-routing-nodejs)
>> to build the node.js app consuming events.
>> When I stress tested the scenario using
>> SIPp and tried sending a lot of events at
>> 300-350cps from Kamailio, I noticed that
>> at times the client is receiving 2-3
>> events in a single message together
>> although I do event_sync_relay once per
>> SIP message received and have netstrings
>> enabled. I believe this is a typical
>> behavior of TCP and needs to be handled
>> by the client using some kind of
>> Netstring handler. Please correct me if
>> I'm wrong.
>> And hence I'd like to know what
>> particularly needs to be taken care of
>> while writing a client that is listening
>> for events on raw tcp socket and how does
>> kamailio handle this situation while
>> receiving messages over TCP socket?? Does
>> kamailio recognize the end of netstring
>> properly on evapi:message-received and
>> give exactly one message to take care of
>> on every "message-received" event or
>> should that be handled in the script
>> somewhere !!
>> I also referred cgrates client over evapi
>> example which is written in GO, but I
>> couldnt find them handling TCP streams
>> clearly either.
>> I'd really appreciate some expert
>> suggestion here to make an informed
>> decision on using the evapi module for a
>> large scale solution.
>>
>> Thanks,
>>
>> - Jayesh
>>
>>
>>
>>
>> _______________________________________________
>> SIP Express Router (SER) and Kamailio (OpenSER) -
sr-users mailing list
>> sr-users(a)lists.sip-router.org
>> <mailto:sr-users@lists.sip-router.org>
>>
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
>
> --
> Daniel-Constantin Mierla
> http://twitter.com/#!/miconda
> <http://twitter.com/#%21/miconda> -
http://www.linkedin.com/in/miconda
> Book: SIP Routing With Kamailio - http://www.asipto.com
> Kamailio Advanced Training, Sep 28-30, 2015, in Berlin -
http://asipto.com/u/kat
>
>
> _______________________________________________
> SIP Express Router (SER) and Kamailio
> (OpenSER) - sr-users mailing list
> sr-users(a)lists.sip-router.org
> <mailto:sr-users@lists.sip-router.org>
>
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
>
>
>
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda
<http://twitter.com/#%21/miconda> -
http://www.linkedin.com/in/miconda
Book: SIP Routing With Kamailio - http://www.asipto.com
Kamailio Advanced Training, Sep 28-30, 2015, in Berlin -
http://asipto.com/u/kat
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda
<http://twitter.com/#%21/miconda> -
http://www.linkedin.com/in/miconda
Book: SIP Routing With Kamailio - http://www.asipto.com
Kamailio Advanced Training, Sep 28-30, 2015, in Berlin -
http://asipto.com/u/kat
10:53 a.m.
Hi,
Here's the NGREP trace where
{"event":"REGISTER","tindex":"3639","tlabel":"1889788252"
... is present on the first line itself.
198.24.63.45:48886 -> 198.24.63.39:3927 [AP]
146:{"event":"REGISTER","tindex":"3639","tlabel":"1889788252","PhoneNumber":"42008","DeviceId":"abcd1234abcd1234","CallId":"
11542-19872(a)198.24.63.39
"},147:{"event":"REGISTER","tindex":"12600","tlabel":"1916422130","PhoneNumber":"42009","DeviceId":"abcd1234abcd1234","CallId":"
11543-19872(a)198.24.63.39
"},147:{"event":"REGISTER","tindex":"51251","tlabel":"1549871937","PhoneNumber":"42010","DeviceId":"abcd1234abcd
1234","CallId":"11544-19872@198.24.63.39
"},147:{"event":"REGISTER","tindex":"60213","tlabel":"2100336684","PhoneNumber":"42011","DeviceId":"abcd1234abcd1234","CallId":"
11545-19872@198.24.63.39"},146:{"event
":"REGISTER","tindex":"33328","tlabel":"622251552","PhoneNumber":"42012","DeviceId":"abcd1234abcd1234","CallId":"
11546-19872(a)198.24.63.39
"},147:{"event":"REGISTER","tindex":"42289","tlabel":"1360864805","PhoneNumber":"42013","DeviceId":"abcd1234abcd1234","CallId":"
11547-19872(a)198.24.63.39
"},146:{"event":"REGISTER","tindex":"48204","tlabel":"315887672","PhoneNumber":"42014","DeviceId":"abcd1234abcd1234","CallId":"
11548-19872(a)198.24.63.39
"},145:{"event":"REGISTER","tindex":"36682","tlabel":"18660440","PhoneNumber":"42015","DeviceId":"abcd1234abcd1234","CallId":"
11549-19872(a)198.24.63.39"}uot;},
And yes, this packet is right after the second packet in the pastebin trace.
Thanks,
- Jayesh
On Tue, Sep 22, 2015 at 5:03 PM Daniel-Constantin Mierla <miconda(a)gmail.com>
wrote:
Hello,
I haven't seen in the ngrep when the data containing next line is received:
{"event":"REGISTER","tindex":"3639","tlabel":"1889788252"
...
Do you still have it? Is it after:
{"event":"REGISTER","tindex":"12603","tlabel":"626010915",
...
Cheers,
Daniel
On 22/09/15 09:09, Jayesh Nambiar wrote:
Hi,
Took time to get back with the test results as I was trying to analyze too
much myself before getting back on the list. Also I was stuck with weird
error of module version mismatch, but I figured and solved that issue.
So I do see the logs "frame size mismatch" when I miss the events. Since
the logs are too huge, I am sending it in pastebin. Here it is:
http://pastebin.com/MzTpYsrk
In Kamailio Logs, I see:
DEBUG: evapi [evapi_dispatch.c:361]: evapi_recv_client(): frame size
mismatch the ending char
(h):[{"event":"REGISTER","tindex":"12603","tlabel":"626010915
","PhoneNumber":"42000","D146:"event":"REGISTER","tindex":"3639","tlabel":"
1889788252","P] (146)
Whereas the messages in ngrep looks good where the messages are split in
two TCP packets. The first packet ends at D and the second packet is the
proper continuation which send the rest. But it feels Kamailio skipped the
beginning of the message and started parsing at colon again and it thinks
the netstring started with D146 which is incorrect. Hence, it never emitted
any event for the messages in the first chunk.
Hoping this to be helpful !! Thanks.
- Jayesh
On Mon, Sep 21, 2015 at 4:37 PM Daniel-Constantin Mierla <
miconda(a)gmail.com> wrote:
Hello,
I pushed another commit to add more debug messages to see if the event
route is supposed to be executed or not. Can you run the tests again and
give again the log messages for missing event route executions?
Cheers,
Daniel
On 21/09/15 09:04, Jayesh Nambiar wrote:
Hi Daniel,
I have captured debug messages after testing with your latest commit on
master. Once again, here's what I'm doing exactly for my test:
1) Use SIPp to send OPTIONS towards Kamailio.
2) On getting options send event as follows:
evapi_async_relay("{\"event\":\"REGISTER\",\"tindex\":\"$T(id_index)\",\"tlabel\":\"$T(id_label)\",\"PhoneNumber\":\"$avp(phone_number)\",\"DeviceId\":\"$avp(device_id)\",\"CallId\":\"$ci\"}");
3) There's a client connected which listens for messages on this socket,
parses the netstring, and sends same data back as netstring to Kamailio.
4) On the event_route[evapi:message-received, I do the following:
xlog("L_INFO", "GOT [$evapi(msg)] from
$evapi(srcaddr):$evapi(srcport)\n");
if($evapi(msg)=~"REGISTER" && $evapi(msg)=~"tindex") {
jansson_get_field("$evapi(msg)", "tlabel",
"$var(tlabel)");
jansson_get_field("$evapi(msg)", "tindex",
"$var(tindex)");
$var(t_index) = $(var(tindex){s.int});
$var(t_label) = $(var(tlabel){s.int});
t_continue('$var(t_index)', '$var(t_label)',
'REGISTER_RESPONSE');
exit;
}
5) On route[REGISTER_RESPONSE], I send 200 OK to SIPp
Now here are detailed logs for which neither SIPp didnt get a response
nor the GOT [$evapi(msg)] from $evapi(srcaddr):$evapi(srcport)\n"); got
logged in the syslog:
First Event:
DEBUG: evapi [evapi_dispatch.c:598]: evapi_relay(): relaying event data
[{"event":"REGISTER","tindex":"26266","tlabel":"587925078","PhoneNumber":"21956","DeviceId":"abcd1234abcd1234","CallId":"
23021-12910(a)198.24.63.39"}] (146)
DEBUG: evapi [evapi_dispatch.c:623]: evapi_relay(): sending
[0x7f132588de68]
[146:{"event":"REGISTER","tindex":"26266","tlabel":"587925078","PhoneNumber":"21956","DeviceId":"abcd1234abcd1234","CallId":"
23021-12910(a)198.24.63.39"}uot;},] (151)
DEBUG: evapi [evapi_dispatch.c:488]: evapi_recv_notify(): received
[0x7f132588de68]
[146:{"event":"REGISTER","tindex":"26266","tlabel":"587925078","PhoneNumber":"21956","DeviceId":"abcd1234abcd1234","CallId":"
23021-12910(a)198.24.63.39"}uot;},] (151)
NOTICE: evapi [evapi_dispatch.c:290]: evapi_recv_client(): {0} [
198.24.63.45:48881] - received
[146:{"event":"REGISTER","tindex":"26266","tlabel":"587925078","PhoneNumber":"21956","DeviceId":"abcd1234abcd1234","CallId":"
23021-12910(a)198.24.63.39"}uot;},] (151) (18)
Second Event:
DEBUG: evapi [evapi_dispatch.c:598]: evapi_relay(): relaying event data
[{"event":"REGISTER","tindex":"32244","tlabel":"1637923412","PhoneNumber":"25597","DeviceId":"abcd1234abcd1234","CallId":"
27182-12910(a)198.24.63.39"}] (147)
DEBUG: evapi [evapi_dispatch.c:623]: evapi_relay(): sending
[0x7f1325884410]
[147:{"event":"REGISTER","tindex":"32244","tlabel":"1637923412","PhoneNumber":"25597","DeviceId":"abcd1234abcd1234","CallId":"
27182-12910(a)198.24.63.39"}uot;},] (152)
DEBUG: evapi [evapi_dispatch.c:488]: evapi_recv_notify(): received
[0x7f1325884410]
[147:{"event":"REGISTER","tindex":"32244","tlabel":"1637923412","PhoneNumber":"25597","DeviceId":"abcd1234abcd1234","CallId":"
27182-12910(a)198.24.63.39"}uot;},] (152)
NOTICE: evapi [evapi_dispatch.c:290]: evapi_recv_client(): {0} [
198.24.63.45:48881] - received
[147:{"event":"REGISTER","tindex":"32244","tlabel":"1637923412","PhoneNumber":"25597","DeviceId":"abcd1234abcd1234","CallId":"
27182-12910(a)198.24.63.39"}uot;},] (152) (85)
I had three such events which got missed out of some 25000 odd messages
sent from SIPp. Other major difference that I see in the debug logs for
problematic events are that there is a positive number in the second
parentheses of evapi_recv_client() function. For the events that were
invoked successfully the value of second parentheses is 0 for
evapi_recv_client() function. Hope the debugging helps.
Debug log for a successful event:
DEBUG: evapi [evapi_dispatch.c:598]: evapi_relay(): relaying event data
[{"event":"REGISTER","tindex":"22127","tlabel":"1896682192","PhoneNumber":"73168","DeviceId":"abcd1234abcd1234","CallId":"
49957-13056(a)198.24.63.39"}] (147)
DEBUG: evapi [evapi_dispatch.c:623]: evapi_relay(): sending
[0x7f132568e850]
[147:{"event":"REGISTER","tindex":"22127","tlabel":"1896682192","PhoneNumber":"73168","DeviceId":"abcd1234abcd1234","CallId":"
49957-13056(a)198.24.63.39"}uot;},] (152)
DEBUG: evapi [evapi_dispatch.c:488]: evapi_recv_notify(): received
[0x7f132568e850]
[147:{"event":"REGISTER","tindex":"22127","tlabel":"1896682192","PhoneNumber":"73168","DeviceId":"abcd1234abcd1234","CallId":"
49957-13056(a)198.24.63.39"}uot;},] (152)
NOTICE: evapi [evapi_dispatch.c:290]: evapi_recv_client(): {0} [
198.24.63.45:48881] - received
[147:{"event":"REGISTER","tindex":"22127","tlabel":"1896682192","PhoneNumber":"73168","DeviceId":"abcd1234abcd1234","CallId":"
49957-13056(a)198.24.63.39"}uot;},] (152) (0)
Thanks,
- Jayesh
On Sat, Sep 19, 2015 at 3:55 AM Daniel-Constantin Mierla <
miconda(a)gmail.com> wrote:
--
Daniel-Constantin Mierlahttp://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda
Book: SIP Routing With Kamailio - http://www.asipto.com
Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
I pushed a commit to add more debug message while
processing received
data. You can use debugger module to set a higher debug level for evapi
module in order to see what happens.
I checked the netstring packet size and it is invalid (unless email
stripped some white chars there) -- for example in:
145:{"event":"REGISTER","tindex":"55567","tlabel":"627458699","PhoneNumber":"20711","DeviceId":"abcd1234abcd1234","CallId":
"21225-3848(a)5.6.7.8" <21225-3848(a)5.6.7.8> <21225-3848(a)5.6.7.8>}8>},
the size is 141, not 145 -- it looks like the size includes the size
itself plus the delimiters ':,'. The size is only the effective data, see:
https://en.wikipedia.org/wiki/Netstring
If evapi gets a packet with an invalid size, then it discards the buffer
content.
See if the app on the other side of evapi connection builds netstrings
with wrong size.
Cheers,
Daniel
On 18/09/15 23:19, Jayesh Nambiar wrote:
Here are the tests that I did:
With the patch applied, I see that Kamailio is invoking event
individually for each netstring even when they come in different chunks.
But I did see instances where when there were complete netstrings in a
single chunk; kamailio did not raise an event for them. Here's the
illustration:
T 1.2.3.4:48873 -> 5.6.7.8:3927 [AP]
146:{"event":"REGISTER","tindex":"16916","tlabel":"1753048277","PhoneNumber":"20708","DeviceId":"abcd1234abcd1234","CallId":"
21220-3848(a)5.6.7.8
"},144:{"event":"REGISTER","tindex":"7954","tlabel":"254315075","PhoneNumber":"20709","DeviceId":"abcd1234abcd1234","CallId":"
21223-3848(a)5.6.7.8
"},145:{"event":"REGISTER","tindex":"64529","tlabel":"599481568","PhoneNumber":"20709","DeviceId":"abcd1234abcd1234","CallId":"
21222-3848@5.6.7.8"},145:{"event":"REGISTER","tindex":"46605","tlabel":"112015324","PhoneNumber":"20710","DeviceId"
T 1.2.3.4:48873 -> 5.6.7.8:3927 [AP]
:"abcd1234abcd1234","CallId":"21224-3848@5.6.7.8"},
*The above two chunks contain 4 proper netstrings where the second chunk
contains part of the 4th netstring. In this case Evapi properly raised 4
individual events.*
1.2.3.4:48873 -> 5.6.7.8:3927 [AP]
145:{"event":"REGISTER","tindex":"55567","tlabel":"627458699","PhoneNumber":"20711","DeviceId":"abcd1234abcd1234","CallId":"
21225-3848(a)5.6.7.8
"},143:{"event":"REGISTER","tindex":"28682","tlabel":"9676691","PhoneNumber":"20712","DeviceId":"abcd1234abcd1234","CallId":"
21226-3848(a)5.6.7.8"}uot;},
*This above chunk contains two complete netstrings but Kamailio never
raised events for these two netstrings.*
For the events that were not raised I see proper Kamailio Logs which is:
evapi_recv_client(): {0} [1.2.3.4:48873] - received
[145:{"event":"REGISTER","tindex":"55567","tlabel":"627458699","PhoneNumber":"20711","DeviceId":"abcd1234abcd1234","CallId":"
21225-3848(a)5.6.7.8
"},143:{"event":"REGISTER","tindex":"28682","tlabel":"9676691","PhoneNumber":"20712","DeviceId":"abcd1234abcd1234","CallId":"
21226-3848(a)5.6.7.8"}uot;},]
But I don't see the logs that I've written in the script after the event
was raised, which means Kamailio did not invoke events for these two
netstrings.
So in my tests, out of 27232 messages sent, there were approximately 27
messages for which the events were not raised by Evapi. The rate of
messages started at 500cps and I stopped after I saw missed events at
around 1200cps.
Thanks, and do let me know for any further tests or information required
about the same.
- Jayesh
On Fri, Sep 18, 2015 at 2:55 PM, Jayesh Nambiar <jayesh1017(a)gmail.com>
wrote:
--
Daniel-Constantin Mierlahttp://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda
Book: SIP Routing With Kamailio - http://www.asipto.com
Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
Will have this tested by tomorrow and will get
back to you. Thanks.
- Jayesh
On Fri, Sep 18, 2015 at 1:41 PM, Daniel-Constantin Mierla <
miconda(a)gmail.com> wrote:
> Hello,
>
> I just pushed a patch to master branch that should cope with partial
> data received on tcp connection. No time to test at all, therefore any
> feedback will be appreciated.
>
> Cheers,
> Daniel
>
>
> On 15/09/15 14:52, Daniel-Constantin Mierla wrote:
>
> Hello,
>
> I will look if there are options in libev to buffer data or try to
> implement a buffering mechanism locally for such cases.
>
> Cheers,
> Daniel
>
> On 14/09/15 23:00, Jayesh Nambiar wrote:
>
> Hello Daniel,
> After further testing with evapi module, I figured that when
> Netstrings are used, an event route is invoked individually for each
> message even if if multiple netstring messages are received in a single TCP
> packet. But this doesn't work effectively when a single proper message is
> split-up in two packets. For Example, if a message arrives as:
> 12:Hello World!, 12:Hello World!, 12:Hello World! in a single packet,
> kamailio properly invokes the event route "evapi:message-received" thrice
> for every individual proper netstring message.
> But if the first packet contains:
> 12:Hello World!, 12:Hello
> And Second Packet contains:
> World!, 12:Hello World!
> the event route is invoked only once !!
>
> The above pattern is very much possible while sending and receiving
> packets over TCP Socket. Our tests for receiving an approximately 150 byte
> message over evapi socket at the rate of roughly 1000cps causes a lot of
> real events to be missed because of the above problem. You can never be
> sure when TCP will split messages in different chunks.
> This definitely looks like a bug which makes it not very reliable at
> large scale deployments. Would really appreciate your inputs on this.
> Thanks;
>
> - Jayesh
>
>
>
> On Thu, Sep 10, 2015 at 4:01 PM, Jayesh Nambiar <jayesh1017(a)gmail.com>
> wrote:
>
>> Hi Daniel,
>> Thanks for the quick response. So if I do not use Netstrings, does
>> Kamailio allow me to create a custom logic in the script. For eg. if I
>> decide to use newline as a delimiter, can I keep buffering the message
>> until I encounter the delimiter from the event route and then execute
>> whatever I have to within the script??
>>
>> Thanks,
>>
>> - Jayesh
>>
>> On Thu, Sep 10, 2015 at 1:29 PM, Daniel-Constantin Mierla <
>> miconda(a)gmail.com> wrote:
>>
>>> Hello,
>>>
>>> tcb is stream protocol and several messages can be queued on the
>>> pipe at the same time. That is the reason for netstring format, to be able
>>> to easily detect the boundaries of each message. If netstring format is
>>> enabled and kamailio receives several messages at once, it splits them and
>>> for each is executing the event route.
>>>
>>> If netstring format is not used, the kamailio is executing the event
>>> route with the entire content that was read at once from the tcp connection.
>>>
>>> Cheers,
>>> Daniel
>>>
>>>
>>> On 09/09/15 22:01, Jayesh Nambiar wrote:
>>>
>>> Hello,
>>> I'm exploring the evapi module for my kamailio to interface with an
>>> external node.js app for third party stuff like AAA, billing engine tasks,
>>> notifications and so on. I followed and took some ideas from the rtjson and
>>> evapi tutorial found here(
>>> http://kb.asipto.com/kamailio:k43-async-sip-routing-nodejs) to
>>> build the node.js app consuming events.
>>> When I stress tested the scenario using SIPp and tried sending a lot
>>> of events at 300-350cps from Kamailio, I noticed that at times the client
>>> is receiving 2-3 events in a single message together although I do
>>> event_sync_relay once per SIP message received and have netstrings enabled.
>>> I believe this is a typical behavior of TCP and needs to be handled by the
>>> client using some kind of Netstring handler. Please correct me if I'm
wrong.
>>> And hence I'd like to know what particularly needs to be taken care
>>> of while writing a client that is listening for events on raw tcp socket
>>> and how does kamailio handle this situation while receiving messages over
>>> TCP socket?? Does kamailio recognize the end of netstring properly on
>>> evapi:message-received and give exactly one message to take care of on
>>> every "message-received" event or should that be handled in the
script
>>> somewhere !!
>>> I also referred cgrates client over evapi example which is written
>>> in GO, but I couldnt find them handling TCP streams clearly either.
>>> I'd really appreciate some expert suggestion here to make an
>>> informed decision on using the evapi module for a large scale solution.
>>>
>>> Thanks,
>>>
>>> - Jayesh
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing
listsr-users@lists.sip-router.orghttp://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
>>>
>>>
>>> --
>>> Daniel-Constantin Mierlahttp://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda
>>> Book: SIP Routing With Kamailio - http://www.asipto.com
>>> Kamailio Advanced Training, Sep 28-30, 2015, in Berlin -
http://asipto.com/u/kat
>>>
>>>
>>> _______________________________________________
>>> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing
>>> list
>>> sr-users(a)lists.sip-router.org
>>> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
>>>
>>>
>>
>
> --
> Daniel-Constantin Mierlahttp://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda
> Book: SIP Routing With Kamailio - http://www.asipto.com
> Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
>
>
> --
> Daniel-Constantin Mierlahttp://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda
> Book: SIP Routing With Kamailio - http://www.asipto.com
> Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
>
>
--
Daniel-Constantin Mierlahttp://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda
Book: SIP Routing With Kamailio - http://www.asipto.com
Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
23 Sep
23 Sep
10:54 a.m.
Hello Daniel,
Just checking if you got what you were looking for. Do let me know if you
need more info on the same.
Thanks,
- Jayesh
On Tue, Sep 22, 2015 at 8:23 PM Jayesh Nambiar <jayesh1017(a)gmail.com> wrote:
Hi,
Here's the NGREP trace where
{"event":"REGISTER","tindex":"3639","tlabel":"1889788252"
... is present on the first line itself.
198.24.63.45:48886 -> 198.24.63.39:3927 [AP]
146:{"event":"REGISTER","tindex":"3639","tlabel":"1889788252","PhoneNumber":"42008","DeviceId":"abcd1234abcd1234","CallId":"
11542-19872(a)198.24.63.39
"},147:{"event":"REGISTER","tindex":"12600","tlabel":"1916422130","PhoneNumber":"42009","DeviceId":"abcd1234abcd1234","CallId":"
11543-19872(a)198.24.63.39
"},147:{"event":"REGISTER","tindex":"51251","tlabel":"1549871937","PhoneNumber":"42010","DeviceId":"abcd1234abcd
1234","CallId":"11544-19872@198.24.63.39
"},147:{"event":"REGISTER","tindex":"60213","tlabel":"2100336684","PhoneNumber":"42011","DeviceId":"abcd1234abcd1234","CallId":"
11545-19872@198.24.63.39"},146:{"event
":"REGISTER","tindex":"33328","tlabel":"622251552","PhoneNumber":"42012","DeviceId":"abcd1234abcd1234","CallId":"
11546-19872(a)198.24.63.39
"},147:{"event":"REGISTER","tindex":"42289","tlabel":"1360864805","PhoneNumber":"42013","DeviceId":"abcd1234abcd1234","CallId":"
11547-19872(a)198.24.63.39
"},146:{"event":"REGISTER","tindex":"48204","tlabel":"315887672","PhoneNumber":"42014","DeviceId":"abcd1234abcd1234","CallId":"
11548-19872(a)198.24.63.39
"},145:{"event":"REGISTER","tindex":"36682","tlabel":"18660440","PhoneNumber":"42015","DeviceId":"abcd1234abcd1234","CallId":"
11549-19872(a)198.24.63.39"}uot;},
And yes, this packet is right after the second packet in the pastebin
trace.
Thanks,
- Jayesh
On Tue, Sep 22, 2015 at 5:03 PM Daniel-Constantin Mierla <
miconda(a)gmail.com> wrote:
> Hello,
>
> I haven't seen in the ngrep when the data containing next line is
> received:
>
>
{"event":"REGISTER","tindex":"3639","tlabel":"1889788252"
...
>
> Do you still have it? Is it after:
>
>
{"event":"REGISTER","tindex":"12603","tlabel":"626010915",
...
>
> Cheers,
> Daniel
>
>
> On 22/09/15 09:09, Jayesh Nambiar wrote:
>
> Hi,
> Took time to get back with the test results as I was trying to analyze
> too much myself before getting back on the list. Also I was stuck with
> weird error of module version mismatch, but I figured and solved that issue.
> So I do see the logs "frame size mismatch" when I miss the events. Since
> the logs are too huge, I am sending it in pastebin. Here it is:
> http://pastebin.com/MzTpYsrk
>
> In Kamailio Logs, I see:
> DEBUG: evapi [evapi_dispatch.c:361]: evapi_recv_client(): frame size
> mismatch the ending char
>
(h):[{"event":"REGISTER","tindex":"12603","tlabel":"626010915
>
","PhoneNumber":"42000","D146:"event":"REGISTER","tindex":"3639","tlabel":"
> 1889788252","P] (146)
>
> Whereas the messages in ngrep looks good where the messages are split in
> two TCP packets. The first packet ends at D and the second packet is the
> proper continuation which send the rest. But it feels Kamailio skipped the
> beginning of the message and started parsing at colon again and it thinks
> the netstring started with D146 which is incorrect. Hence, it never emitted
> any event for the messages in the first chunk.
>
> Hoping this to be helpful !! Thanks.
>
> - Jayesh
>
>
> On Mon, Sep 21, 2015 at 4:37 PM Daniel-Constantin Mierla <
> miconda(a)gmail.com> wrote:
>
>> Hello,
>>
>> I pushed another commit to add more debug messages to see if the event
>> route is supposed to be executed or not. Can you run the tests again and
>> give again the log messages for missing event route executions?
>>
>> Cheers,
>> Daniel
>>
>>
>> On 21/09/15 09:04, Jayesh Nambiar wrote:
>>
>> Hi Daniel,
>> I have captured debug messages after testing with your latest commit on
>> master. Once again, here's what I'm doing exactly for my test:
>> 1) Use SIPp to send OPTIONS towards Kamailio.
>> 2) On getting options send event as follows:
>>
>>
evapi_async_relay("{\"event\":\"REGISTER\",\"tindex\":\"$T(id_index)\",\"tlabel\":\"$T(id_label)\",\"PhoneNumber\":\"$avp(phone_number)\",\"DeviceId\":\"$avp(device_id)\",\"CallId\":\"$ci\"}");
>> 3) There's a client connected which listens for messages on this socket,
>> parses the netstring, and sends same data back as netstring to Kamailio.
>> 4) On the event_route[evapi:message-received, I do the following:
>> xlog("L_INFO", "GOT [$evapi(msg)] from
>> $evapi(srcaddr):$evapi(srcport)\n");
>> if($evapi(msg)=~"REGISTER" &&
$evapi(msg)=~"tindex") {
>> jansson_get_field("$evapi(msg)", "tlabel",
"$var(tlabel)");
>> jansson_get_field("$evapi(msg)", "tindex",
"$var(tindex)");
>> $var(t_index) = $(var(tindex){s.int});
>> $var(t_label) = $(var(tlabel){s.int});
>> t_continue('$var(t_index)', '$var(t_label)',
>> 'REGISTER_RESPONSE');
>> exit;
>> }
>> 5) On route[REGISTER_RESPONSE], I send 200 OK to SIPp
>>
>> Now here are detailed logs for which neither SIPp didnt get a response
>> nor the GOT [$evapi(msg)] from $evapi(srcaddr):$evapi(srcport)\n"); got
>> logged in the syslog:
>>
>> First Event:
>> DEBUG: evapi [evapi_dispatch.c:598]: evapi_relay(): relaying event data
>>
[{"event":"REGISTER","tindex":"26266","tlabel":"587925078","PhoneNumber":"21956","DeviceId":"abcd1234abcd1234","CallId":"
>> 23021-12910(a)198.24.63.39"}] (146)
>> DEBUG: evapi [evapi_dispatch.c:623]: evapi_relay(): sending
>> [0x7f132588de68]
>>
[146:{"event":"REGISTER","tindex":"26266","tlabel":"587925078","PhoneNumber":"21956","DeviceId":"abcd1234abcd1234","CallId":"
>> 23021-12910(a)198.24.63.39"}uot;},] (151)
>>
>> DEBUG: evapi [evapi_dispatch.c:488]: evapi_recv_notify(): received
>> [0x7f132588de68]
>>
[146:{"event":"REGISTER","tindex":"26266","tlabel":"587925078","PhoneNumber":"21956","DeviceId":"abcd1234abcd1234","CallId":"
>> 23021-12910(a)198.24.63.39"}uot;},] (151)
>> NOTICE: evapi [evapi_dispatch.c:290]: evapi_recv_client(): {0} [
>> 198.24.63.45:48881] - received
>>
[146:{"event":"REGISTER","tindex":"26266","tlabel":"587925078","PhoneNumber":"21956","DeviceId":"abcd1234abcd1234","CallId":"
>> 23021-12910(a)198.24.63.39"}uot;},] (151) (18)
>>
>> Second Event:
>> DEBUG: evapi [evapi_dispatch.c:598]: evapi_relay(): relaying event data
>>
[{"event":"REGISTER","tindex":"32244","tlabel":"1637923412","PhoneNumber":"25597","DeviceId":"abcd1234abcd1234","CallId":"
>> 27182-12910(a)198.24.63.39"}] (147)
>> DEBUG: evapi [evapi_dispatch.c:623]: evapi_relay(): sending
>> [0x7f1325884410]
>>
[147:{"event":"REGISTER","tindex":"32244","tlabel":"1637923412","PhoneNumber":"25597","DeviceId":"abcd1234abcd1234","CallId":"
>> 27182-12910(a)198.24.63.39"}uot;},] (152)
>>
>> DEBUG: evapi [evapi_dispatch.c:488]: evapi_recv_notify(): received
>> [0x7f1325884410]
>>
[147:{"event":"REGISTER","tindex":"32244","tlabel":"1637923412","PhoneNumber":"25597","DeviceId":"abcd1234abcd1234","CallId":"
>> 27182-12910(a)198.24.63.39"}uot;},] (152)
>> NOTICE: evapi [evapi_dispatch.c:290]: evapi_recv_client(): {0} [
>> 198.24.63.45:48881] - received
>>
[147:{"event":"REGISTER","tindex":"32244","tlabel":"1637923412","PhoneNumber":"25597","DeviceId":"abcd1234abcd1234","CallId":"
>> 27182-12910(a)198.24.63.39"}uot;},] (152) (85)
>>
>> I had three such events which got missed out of some 25000 odd messages
>> sent from SIPp. Other major difference that I see in the debug logs for
>> problematic events are that there is a positive number in the second
>> parentheses of evapi_recv_client() function. For the events that were
>> invoked successfully the value of second parentheses is 0 for
>> evapi_recv_client() function. Hope the debugging helps.
>>
>> Debug log for a successful event:
>> DEBUG: evapi [evapi_dispatch.c:598]: evapi_relay(): relaying event data
>>
[{"event":"REGISTER","tindex":"22127","tlabel":"1896682192","PhoneNumber":"73168","DeviceId":"abcd1234abcd1234","CallId":"
>> 49957-13056(a)198.24.63.39"}] (147)
>> DEBUG: evapi [evapi_dispatch.c:623]: evapi_relay(): sending
>> [0x7f132568e850]
>>
[147:{"event":"REGISTER","tindex":"22127","tlabel":"1896682192","PhoneNumber":"73168","DeviceId":"abcd1234abcd1234","CallId":"
>> 49957-13056(a)198.24.63.39"}uot;},] (152)
>>
>> DEBUG: evapi [evapi_dispatch.c:488]: evapi_recv_notify(): received
>> [0x7f132568e850]
>>
[147:{"event":"REGISTER","tindex":"22127","tlabel":"1896682192","PhoneNumber":"73168","DeviceId":"abcd1234abcd1234","CallId":"
>> 49957-13056(a)198.24.63.39"}uot;},] (152)
>> NOTICE: evapi [evapi_dispatch.c:290]: evapi_recv_client(): {0} [
>> 198.24.63.45:48881] - received
>>
[147:{"event":"REGISTER","tindex":"22127","tlabel":"1896682192","PhoneNumber":"73168","DeviceId":"abcd1234abcd1234","CallId":"
>> 49957-13056(a)198.24.63.39"}uot;},] (152) (0)
>>
>> Thanks,
>>
>> - Jayesh
>> On Sat, Sep 19, 2015 at 3:55 AM Daniel-Constantin Mierla <
>> miconda(a)gmail.com> wrote:
>>
>>> I pushed a commit to add more debug message while processing received
>>> data. You can use debugger module to set a higher debug level for evapi
>>> module in order to see what happens.
>>>
>>> I checked the netstring packet size and it is invalid (unless email
>>> stripped some white chars there) -- for example in:
>>>
>>>
>>>
>>>
145:{"event":"REGISTER","tindex":"55567","tlabel":"627458699","PhoneNumber":"20711","DeviceId":"abcd1234abcd1234","CallId":
>>> "21225-3848(a)5.6.7.8" <21225-3848(a)5.6.7.8>
<21225-3848(a)5.6.7.8>}8>},
>>>
>>> the size is 141, not 145 -- it looks like the size includes the size
>>> itself plus the delimiters ':,'. The size is only the effective data,
see:
>>>
>>> https://en.wikipedia.org/wiki/Netstring
>>>
>>> If evapi gets a packet with an invalid size, then it discards the
>>> buffer content.
>>>
>>> See if the app on the other side of evapi connection builds netstrings
>>> with wrong size.
>>>
>>> Cheers,
>>> Daniel
>>>
>>>
>>> On 18/09/15 23:19, Jayesh Nambiar wrote:
>>>
>>> Here are the tests that I did:
>>>
>>> With the patch applied, I see that Kamailio is invoking event
>>> individually for each netstring even when they come in different chunks.
>>> But I did see instances where when there were complete netstrings in a
>>> single chunk; kamailio did not raise an event for them. Here's the
>>> illustration:
>>>
>>> T 1.2.3.4:48873 -> 5.6.7.8:3927 [AP]
>>>
>>>
146:{"event":"REGISTER","tindex":"16916","tlabel":"1753048277","PhoneNumber":"20708","DeviceId":"abcd1234abcd1234","CallId":"
>>> 21220-3848(a)5.6.7.8
>>>
"},144:{"event":"REGISTER","tindex":"7954","tlabel":"254315075","PhoneNumber":"20709","DeviceId":"abcd1234abcd1234","CallId":"
>>> 21223-3848(a)5.6.7.8
>>>
"},145:{"event":"REGISTER","tindex":"64529","tlabel":"599481568","PhoneNumber":"20709","DeviceId":"abcd1234abcd1234","CallId":"
>>>
21222-3848@5.6.7.8"},145:{"event":"REGISTER","tindex":"46605","tlabel":"112015324","PhoneNumber":"20710","DeviceId"
>>>
>>>
>>>
>>>
>>> T 1.2.3.4:48873 -> 5.6.7.8:3927 [AP]
>>>
:"abcd1234abcd1234","CallId":"21224-3848@5.6.7.8"},
>>>
>>> *The above two chunks contain 4 proper netstrings where the second
>>> chunk contains part of the 4th netstring. In this case Evapi properly
>>> raised 4 individual events.*
>>>
>>> 1.2.3.4:48873 -> 5.6.7.8:3927 [AP]
>>>
>>>
145:{"event":"REGISTER","tindex":"55567","tlabel":"627458699","PhoneNumber":"20711","DeviceId":"abcd1234abcd1234","CallId":"
>>> 21225-3848(a)5.6.7.8
>>>
"},143:{"event":"REGISTER","tindex":"28682","tlabel":"9676691","PhoneNumber":"20712","DeviceId":"abcd1234abcd1234","CallId":"
>>> 21226-3848(a)5.6.7.8"}uot;},
>>>
>>> *This above chunk contains two complete netstrings but Kamailio never
>>> raised events for these two netstrings.*
>>>
>>> For the events that were not raised I see proper Kamailio Logs which is:
>>> evapi_recv_client(): {0} [1.2.3.4:48873] - received
>>>
[145:{"event":"REGISTER","tindex":"55567","tlabel":"627458699","PhoneNumber":"20711","DeviceId":"abcd1234abcd1234","CallId":"
>>> 21225-3848(a)5.6.7.8
>>>
"},143:{"event":"REGISTER","tindex":"28682","tlabel":"9676691","PhoneNumber":"20712","DeviceId":"abcd1234abcd1234","CallId":"
>>> 21226-3848(a)5.6.7.8"}uot;},]
>>>
>>> But I don't see the logs that I've written in the script after the
>>> event was raised, which means Kamailio did not invoke events for these two
>>> netstrings.
>>>
>>> So in my tests, out of 27232 messages sent, there were approximately 27
>>> messages for which the events were not raised by Evapi. The rate of
>>> messages started at 500cps and I stopped after I saw missed events at
>>> around 1200cps.
>>>
>>> Thanks, and do let me know for any further tests or information
>>> required about the same.
>>>
>>> - Jayesh
>>>
>>>
>>>
>>> On Fri, Sep 18, 2015 at 2:55 PM, Jayesh Nambiar <jayesh1017(a)gmail.com>
>>> wrote:
>>>
>>>> Will have this tested by tomorrow and will get back to you. Thanks.
>>>>
>>>> - Jayesh
>>>>
>>>> On Fri, Sep 18, 2015 at 1:41 PM, Daniel-Constantin Mierla <
>>>> miconda(a)gmail.com> wrote:
>>>>
>>>>> Hello,
>>>>>
>>>>> I just pushed a patch to master branch that should cope with partial
>>>>> data received on tcp connection. No time to test at all, therefore
any
>>>>> feedback will be appreciated.
>>>>>
>>>>> Cheers,
>>>>> Daniel
>>>>>
>>>>>
>>>>> On 15/09/15 14:52, Daniel-Constantin Mierla wrote:
>>>>>
>>>>> Hello,
>>>>>
>>>>> I will look if there are options in libev to buffer data or try to
>>>>> implement a buffering mechanism locally for such cases.
>>>>>
>>>>> Cheers,
>>>>> Daniel
>>>>>
>>>>> On 14/09/15 23:00, Jayesh Nambiar wrote:
>>>>>
>>>>> Hello Daniel,
>>>>> After further testing with evapi module, I figured that when
>>>>> Netstrings are used, an event route is invoked individually for each
>>>>> message even if if multiple netstring messages are received in a
single TCP
>>>>> packet. But this doesn't work effectively when a single proper
message is
>>>>> split-up in two packets. For Example, if a message arrives as:
>>>>> 12:Hello World!, 12:Hello World!, 12:Hello World! in a single
packet,
>>>>> kamailio properly invokes the event route
"evapi:message-received" thrice
>>>>> for every individual proper netstring message.
>>>>> But if the first packet contains:
>>>>> 12:Hello World!, 12:Hello
>>>>> And Second Packet contains:
>>>>> World!, 12:Hello World!
>>>>> the event route is invoked only once !!
>>>>>
>>>>> The above pattern is very much possible while sending and receiving
>>>>> packets over TCP Socket. Our tests for receiving an approximately 150
byte
>>>>> message over evapi socket at the rate of roughly 1000cps causes a lot
of
>>>>> real events to be missed because of the above problem. You can never
be
>>>>> sure when TCP will split messages in different chunks.
>>>>> This definitely looks like a bug which makes it not very reliable at
>>>>> large scale deployments. Would really appreciate your inputs on
this.
>>>>> Thanks;
>>>>>
>>>>> - Jayesh
>>>>>
>>>>>
>>>>>
>>>>> On Thu, Sep 10, 2015 at 4:01 PM, Jayesh Nambiar
<jayesh1017(a)gmail.com
>>>>> > wrote:
>>>>>
>>>>>> Hi Daniel,
>>>>>> Thanks for the quick response. So if I do not use Netstrings,
does
>>>>>> Kamailio allow me to create a custom logic in the script. For eg.
if I
>>>>>> decide to use newline as a delimiter, can I keep buffering the
message
>>>>>> until I encounter the delimiter from the event route and then
execute
>>>>>> whatever I have to within the script??
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> - Jayesh
>>>>>>
>>>>>> On Thu, Sep 10, 2015 at 1:29 PM, Daniel-Constantin Mierla <
>>>>>> miconda(a)gmail.com> wrote:
>>>>>>
>>>>>>> Hello,
>>>>>>>
>>>>>>> tcb is stream protocol and several messages can be queued on
the
>>>>>>> pipe at the same time. That is the reason for netstring
format, to be able
>>>>>>> to easily detect the boundaries of each message. If netstring
format is
>>>>>>> enabled and kamailio receives several messages at once, it
splits them and
>>>>>>> for each is executing the event route.
>>>>>>>
>>>>>>> If netstring format is not used, the kamailio is executing
the
>>>>>>> event route with the entire content that was read at once
from the tcp
>>>>>>> connection.
>>>>>>>
>>>>>>> Cheers,
>>>>>>> Daniel
>>>>>>>
>>>>>>>
>>>>>>> On 09/09/15 22:01, Jayesh Nambiar wrote:
>>>>>>>
>>>>>>> Hello,
>>>>>>> I'm exploring the evapi module for my kamailio to
interface with an
>>>>>>> external node.js app for third party stuff like AAA, billing
engine tasks,
>>>>>>> notifications and so on. I followed and took some ideas from
the rtjson and
>>>>>>> evapi tutorial found here(
>>>>>>> http://kb.asipto.com/kamailio:k43-async-sip-routing-nodejs)
to
>>>>>>> build the node.js app consuming events.
>>>>>>> When I stress tested the scenario using SIPp and tried
sending a
>>>>>>> lot of events at 300-350cps from Kamailio, I noticed that at
times the
>>>>>>> client is receiving 2-3 events in a single message together
although I do
>>>>>>> event_sync_relay once per SIP message received and have
netstrings enabled.
>>>>>>> I believe this is a typical behavior of TCP and needs to be
handled by the
>>>>>>> client using some kind of Netstring handler. Please correct
me if I'm wrong.
>>>>>>> And hence I'd like to know what particularly needs to be
taken care
>>>>>>> of while writing a client that is listening for events on raw
tcp socket
>>>>>>> and how does kamailio handle this situation while receiving
messages over
>>>>>>> TCP socket?? Does kamailio recognize the end of netstring
properly on
>>>>>>> evapi:message-received and give exactly one message to take
care of on
>>>>>>> every "message-received" event or should that be
handled in the script
>>>>>>> somewhere !!
>>>>>>> I also referred cgrates client over evapi example which is
written
>>>>>>> in GO, but I couldnt find them handling TCP streams clearly
either.
>>>>>>> I'd really appreciate some expert suggestion here to make
an
>>>>>>> informed decision on using the evapi module for a large scale
solution.
>>>>>>>
>>>>>>> Thanks,
>>>>>>>
>>>>>>> - Jayesh
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users
mailing
listsr-users@lists.sip-router.orghttp://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Daniel-Constantin Mierlahttp://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda
>>>>>>> Book: SIP Routing With Kamailio - http://www.asipto.com
>>>>>>> Kamailio Advanced Training, Sep 28-30, 2015, in Berlin -
http://asipto.com/u/kat
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users
mailing
>>>>>>> list
>>>>>>> sr-users(a)lists.sip-router.org
>>>>>>>
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> Daniel-Constantin Mierlahttp://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda
>>>>> Book: SIP Routing With Kamailio - http://www.asipto.com
>>>>> Kamailio Advanced Training, Sep 28-30, 2015, in Berlin -
http://asipto.com/u/kat
>>>>>
>>>>>
>>>>> --
>>>>> Daniel-Constantin Mierlahttp://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda
>>>>> Book: SIP Routing With Kamailio - http://www.asipto.com
>>>>> Kamailio Advanced Training, Sep 28-30, 2015, in Berlin -
http://asipto.com/u/kat
>>>>>
>>>>>
>>>>
>>>
>>> --
>>> Daniel-Constantin Mierlahttp://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda
>>> Book: SIP Routing With Kamailio - http://www.asipto.com
>>> Kamailio Advanced Training, Sep 28-30, 2015, in Berlin -
http://asipto.com/u/kat
>>>
>>>
>> --
>> Daniel-Constantin Mierlahttp://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda
>> Book: SIP Routing With Kamailio - http://www.asipto.com
>> Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
>>
>>
> --
> Daniel-Constantin Mierlahttp://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda
> Book: SIP Routing With Kamailio - http://www.asipto.com
> Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
>
>
25 Sep
25 Sep
7:55 a.m.
Hi Daniel,
I think this is not solved completely after you applied the patch and my
testing shows that there are some problems left out to solve. Please do let
me know if you are looking at more inputs which I might be able to test and
provide. Thanks,
- Jayesh
On Wed, Sep 23, 2015 at 8:24 PM Jayesh Nambiar <jayesh1017(a)gmail.com> wrote:
Hello Daniel,
Just checking if you got what you were looking for. Do let me know if you
need more info on the same.
Thanks,
- Jayesh
On Tue, Sep 22, 2015 at 8:23 PM Jayesh Nambiar <jayesh1017(a)gmail.com>
wrote:
> Hi,
> Here's the NGREP trace where
{"event":"REGISTER","tindex":"3639","tlabel":"1889788252"
> ... is present on the first line itself.
>
> 198.24.63.45:48886 -> 198.24.63.39:3927 [AP]
>
>
146:{"event":"REGISTER","tindex":"3639","tlabel":"1889788252","PhoneNumber":"42008","DeviceId":"abcd1234abcd1234","CallId":"
> 11542-19872(a)198.24.63.39
>
"},147:{"event":"REGISTER","tindex":"12600","tlabel":"1916422130","PhoneNumber":"42009","DeviceId":"abcd1234abcd1234","CallId":"
> 11543-19872(a)198.24.63.39
>
"},147:{"event":"REGISTER","tindex":"51251","tlabel":"1549871937","PhoneNumber":"42010","DeviceId":"abcd1234abcd
> 1234","CallId":"11544-19872@198.24.63.39
>
"},147:{"event":"REGISTER","tindex":"60213","tlabel":"2100336684","PhoneNumber":"42011","DeviceId":"abcd1234abcd1234","CallId":"
> 11545-19872@198.24.63.39"},146:{"event
>
>
":"REGISTER","tindex":"33328","tlabel":"622251552","PhoneNumber":"42012","DeviceId":"abcd1234abcd1234","CallId":"
> 11546-19872(a)198.24.63.39
>
"},147:{"event":"REGISTER","tindex":"42289","tlabel":"1360864805","PhoneNumber":"42013","DeviceId":"abcd1234abcd1234","CallId":"
> 11547-19872(a)198.24.63.39
>
"},146:{"event":"REGISTER","tindex":"48204","tlabel":"315887672","PhoneNumber":"42014","DeviceId":"abcd1234abcd1234","CallId":"
> 11548-19872(a)198.24.63.39
>
"},145:{"event":"REGISTER","tindex":"36682","tlabel":"18660440","PhoneNumber":"42015","DeviceId":"abcd1234abcd1234","CallId":"
> 11549-19872(a)198.24.63.39"}uot;},
>
> And yes, this packet is right after the second packet in the pastebin
> trace.
>
> Thanks,
>
> - Jayesh
>
>
>
> On Tue, Sep 22, 2015 at 5:03 PM Daniel-Constantin Mierla <
> miconda(a)gmail.com> wrote:
>
>> Hello,
>>
>> I haven't seen in the ngrep when the data containing next line is
>> received:
>>
>>
{"event":"REGISTER","tindex":"3639","tlabel":"1889788252"
...
>>
>> Do you still have it? Is it after:
>>
>>
{"event":"REGISTER","tindex":"12603","tlabel":"626010915",
...
>>
>> Cheers,
>> Daniel
>>
>>
>> On 22/09/15 09:09, Jayesh Nambiar wrote:
>>
>> Hi,
>> Took time to get back with the test results as I was trying to analyze
>> too much myself before getting back on the list. Also I was stuck with
>> weird error of module version mismatch, but I figured and solved that issue.
>> So I do see the logs "frame size mismatch" when I miss the events.
Since
>> the logs are too huge, I am sending it in pastebin. Here it is:
>> http://pastebin.com/MzTpYsrk
>>
>> In Kamailio Logs, I see:
>> DEBUG: evapi [evapi_dispatch.c:361]: evapi_recv_client(): frame size
>> mismatch the ending char
>>
(h):[{"event":"REGISTER","tindex":"12603","tlabel":"626010915
>>
","PhoneNumber":"42000","D146:"event":"REGISTER","tindex":"3639","tlabel":"
>> 1889788252","P] (146)
>>
>> Whereas the messages in ngrep looks good where the messages are split in
>> two TCP packets. The first packet ends at D and the second packet is the
>> proper continuation which send the rest. But it feels Kamailio skipped the
>> beginning of the message and started parsing at colon again and it thinks
>> the netstring started with D146 which is incorrect. Hence, it never emitted
>> any event for the messages in the first chunk.
>>
>> Hoping this to be helpful !! Thanks.
>>
>> - Jayesh
>>
>>
>> On Mon, Sep 21, 2015 at 4:37 PM Daniel-Constantin Mierla <
>> miconda(a)gmail.com> wrote:
>>
>>> Hello,
>>>
>>> I pushed another commit to add more debug messages to see if the event
>>> route is supposed to be executed or not. Can you run the tests again and
>>> give again the log messages for missing event route executions?
>>>
>>> Cheers,
>>> Daniel
>>>
>>>
>>> On 21/09/15 09:04, Jayesh Nambiar wrote:
>>>
>>> Hi Daniel,
>>> I have captured debug messages after testing with your latest commit on
>>> master. Once again, here's what I'm doing exactly for my test:
>>> 1) Use SIPp to send OPTIONS towards Kamailio.
>>> 2) On getting options send event as follows:
>>>
>>>
evapi_async_relay("{\"event\":\"REGISTER\",\"tindex\":\"$T(id_index)\",\"tlabel\":\"$T(id_label)\",\"PhoneNumber\":\"$avp(phone_number)\",\"DeviceId\":\"$avp(device_id)\",\"CallId\":\"$ci\"}");
>>> 3) There's a client connected which listens for messages on this
>>> socket, parses the netstring, and sends same data back as netstring to
>>> Kamailio.
>>> 4) On the event_route[evapi:message-received, I do the following:
>>> xlog("L_INFO", "GOT [$evapi(msg)] from
>>> $evapi(srcaddr):$evapi(srcport)\n");
>>> if($evapi(msg)=~"REGISTER" &&
$evapi(msg)=~"tindex") {
>>> jansson_get_field("$evapi(msg)", "tlabel",
"$var(tlabel)");
>>> jansson_get_field("$evapi(msg)", "tindex",
"$var(tindex)");
>>> $var(t_index) = $(var(tindex){s.int});
>>> $var(t_label) = $(var(tlabel){s.int});
>>> t_continue('$var(t_index)', '$var(t_label)',
>>> 'REGISTER_RESPONSE');
>>> exit;
>>> }
>>> 5) On route[REGISTER_RESPONSE], I send 200 OK to SIPp
>>>
>>> Now here are detailed logs for which neither SIPp didnt get a response
>>> nor the GOT [$evapi(msg)] from $evapi(srcaddr):$evapi(srcport)\n"); got
>>> logged in the syslog:
>>>
>>> First Event:
>>> DEBUG: evapi [evapi_dispatch.c:598]: evapi_relay(): relaying event data
>>>
[{"event":"REGISTER","tindex":"26266","tlabel":"587925078","PhoneNumber":"21956","DeviceId":"abcd1234abcd1234","CallId":"
>>> 23021-12910(a)198.24.63.39"}] (146)
>>> DEBUG: evapi [evapi_dispatch.c:623]: evapi_relay(): sending
>>> [0x7f132588de68]
>>>
[146:{"event":"REGISTER","tindex":"26266","tlabel":"587925078","PhoneNumber":"21956","DeviceId":"abcd1234abcd1234","CallId":"
>>> 23021-12910(a)198.24.63.39"}uot;},] (151)
>>>
>>> DEBUG: evapi [evapi_dispatch.c:488]: evapi_recv_notify(): received
>>> [0x7f132588de68]
>>>
[146:{"event":"REGISTER","tindex":"26266","tlabel":"587925078","PhoneNumber":"21956","DeviceId":"abcd1234abcd1234","CallId":"
>>> 23021-12910(a)198.24.63.39"}uot;},] (151)
>>> NOTICE: evapi [evapi_dispatch.c:290]: evapi_recv_client(): {0} [
>>> 198.24.63.45:48881] - received
>>>
[146:{"event":"REGISTER","tindex":"26266","tlabel":"587925078","PhoneNumber":"21956","DeviceId":"abcd1234abcd1234","CallId":"
>>> 23021-12910(a)198.24.63.39"}uot;},] (151) (18)
>>>
>>> Second Event:
>>> DEBUG: evapi [evapi_dispatch.c:598]: evapi_relay(): relaying event data
>>>
[{"event":"REGISTER","tindex":"32244","tlabel":"1637923412","PhoneNumber":"25597","DeviceId":"abcd1234abcd1234","CallId":"
>>> 27182-12910(a)198.24.63.39"}] (147)
>>> DEBUG: evapi [evapi_dispatch.c:623]: evapi_relay(): sending
>>> [0x7f1325884410]
>>>
[147:{"event":"REGISTER","tindex":"32244","tlabel":"1637923412","PhoneNumber":"25597","DeviceId":"abcd1234abcd1234","CallId":"
>>> 27182-12910(a)198.24.63.39"}uot;},] (152)
>>>
>>> DEBUG: evapi [evapi_dispatch.c:488]: evapi_recv_notify(): received
>>> [0x7f1325884410]
>>>
[147:{"event":"REGISTER","tindex":"32244","tlabel":"1637923412","PhoneNumber":"25597","DeviceId":"abcd1234abcd1234","CallId":"
>>> 27182-12910(a)198.24.63.39"}uot;},] (152)
>>> NOTICE: evapi [evapi_dispatch.c:290]: evapi_recv_client(): {0} [
>>> 198.24.63.45:48881] - received
>>>
[147:{"event":"REGISTER","tindex":"32244","tlabel":"1637923412","PhoneNumber":"25597","DeviceId":"abcd1234abcd1234","CallId":"
>>> 27182-12910(a)198.24.63.39"}uot;},] (152) (85)
>>>
>>> I had three such events which got missed out of some 25000 odd messages
>>> sent from SIPp. Other major difference that I see in the debug logs for
>>> problematic events are that there is a positive number in the second
>>> parentheses of evapi_recv_client() function. For the events that were
>>> invoked successfully the value of second parentheses is 0 for
>>> evapi_recv_client() function. Hope the debugging helps.
>>>
>>> Debug log for a successful event:
>>> DEBUG: evapi [evapi_dispatch.c:598]: evapi_relay(): relaying event data
>>>
[{"event":"REGISTER","tindex":"22127","tlabel":"1896682192","PhoneNumber":"73168","DeviceId":"abcd1234abcd1234","CallId":"
>>> 49957-13056(a)198.24.63.39"}] (147)
>>> DEBUG: evapi [evapi_dispatch.c:623]: evapi_relay(): sending
>>> [0x7f132568e850]
>>>
[147:{"event":"REGISTER","tindex":"22127","tlabel":"1896682192","PhoneNumber":"73168","DeviceId":"abcd1234abcd1234","CallId":"
>>> 49957-13056(a)198.24.63.39"}uot;},] (152)
>>>
>>> DEBUG: evapi [evapi_dispatch.c:488]: evapi_recv_notify(): received
>>> [0x7f132568e850]
>>>
[147:{"event":"REGISTER","tindex":"22127","tlabel":"1896682192","PhoneNumber":"73168","DeviceId":"abcd1234abcd1234","CallId":"
>>> 49957-13056(a)198.24.63.39"}uot;},] (152)
>>> NOTICE: evapi [evapi_dispatch.c:290]: evapi_recv_client(): {0} [
>>> 198.24.63.45:48881] - received
>>>
[147:{"event":"REGISTER","tindex":"22127","tlabel":"1896682192","PhoneNumber":"73168","DeviceId":"abcd1234abcd1234","CallId":"
>>> 49957-13056(a)198.24.63.39"}uot;},] (152) (0)
>>>
>>> Thanks,
>>>
>>> - Jayesh
>>> On Sat, Sep 19, 2015 at 3:55 AM Daniel-Constantin Mierla <
>>> miconda(a)gmail.com> wrote:
>>>
>>>> I pushed a commit to add more debug message while processing received
>>>> data. You can use debugger module to set a higher debug level for evapi
>>>> module in order to see what happens.
>>>>
>>>> I checked the netstring packet size and it is invalid (unless email
>>>> stripped some white chars there) -- for example in:
>>>>
>>>>
>>>>
>>>>
145:{"event":"REGISTER","tindex":"55567","tlabel":"627458699","PhoneNumber":"20711","DeviceId":"abcd1234abcd1234","CallId":
>>>> "21225-3848(a)5.6.7.8" <21225-3848(a)5.6.7.8>
<21225-3848(a)5.6.7.8>}8>},
>>>>
>>>> the size is 141, not 145 -- it looks like the size includes the size
>>>> itself plus the delimiters ':,'. The size is only the effective
data, see:
>>>>
>>>> https://en.wikipedia.org/wiki/Netstring
>>>>
>>>> If evapi gets a packet with an invalid size, then it discards the
>>>> buffer content.
>>>>
>>>> See if the app on the other side of evapi connection builds netstrings
>>>> with wrong size.
>>>>
>>>> Cheers,
>>>> Daniel
>>>>
>>>>
>>>> On 18/09/15 23:19, Jayesh Nambiar wrote:
>>>>
>>>> Here are the tests that I did:
>>>>
>>>> With the patch applied, I see that Kamailio is invoking event
>>>> individually for each netstring even when they come in different chunks.
>>>> But I did see instances where when there were complete netstrings in a
>>>> single chunk; kamailio did not raise an event for them. Here's the
>>>> illustration:
>>>>
>>>> T 1.2.3.4:48873 -> 5.6.7.8:3927 [AP]
>>>>
>>>>
146:{"event":"REGISTER","tindex":"16916","tlabel":"1753048277","PhoneNumber":"20708","DeviceId":"abcd1234abcd1234","CallId":"
>>>> 21220-3848(a)5.6.7.8
>>>>
"},144:{"event":"REGISTER","tindex":"7954","tlabel":"254315075","PhoneNumber":"20709","DeviceId":"abcd1234abcd1234","CallId":"
>>>> 21223-3848(a)5.6.7.8
>>>>
"},145:{"event":"REGISTER","tindex":"64529","tlabel":"599481568","PhoneNumber":"20709","DeviceId":"abcd1234abcd1234","CallId":"
>>>>
21222-3848@5.6.7.8"},145:{"event":"REGISTER","tindex":"46605","tlabel":"112015324","PhoneNumber":"20710","DeviceId"
>>>>
>>>>
>>>>
>>>>
>>>> T 1.2.3.4:48873 -> 5.6.7.8:3927 [AP]
>>>>
:"abcd1234abcd1234","CallId":"21224-3848@5.6.7.8"},
>>>>
>>>> *The above two chunks contain 4 proper netstrings where the second
>>>> chunk contains part of the 4th netstring. In this case Evapi properly
>>>> raised 4 individual events.*
>>>>
>>>> 1.2.3.4:48873 -> 5.6.7.8:3927 [AP]
>>>>
>>>>
145:{"event":"REGISTER","tindex":"55567","tlabel":"627458699","PhoneNumber":"20711","DeviceId":"abcd1234abcd1234","CallId":"
>>>> 21225-3848(a)5.6.7.8
>>>>
"},143:{"event":"REGISTER","tindex":"28682","tlabel":"9676691","PhoneNumber":"20712","DeviceId":"abcd1234abcd1234","CallId":"
>>>> 21226-3848(a)5.6.7.8"}uot;},
>>>>
>>>> *This above chunk contains two complete netstrings but Kamailio never
>>>> raised events for these two netstrings.*
>>>>
>>>> For the events that were not raised I see proper Kamailio Logs which
>>>> is:
>>>> evapi_recv_client(): {0} [1.2.3.4:48873] - received
>>>>
[145:{"event":"REGISTER","tindex":"55567","tlabel":"627458699","PhoneNumber":"20711","DeviceId":"abcd1234abcd1234","CallId":"
>>>> 21225-3848(a)5.6.7.8
>>>>
"},143:{"event":"REGISTER","tindex":"28682","tlabel":"9676691","PhoneNumber":"20712","DeviceId":"abcd1234abcd1234","CallId":"
>>>> 21226-3848(a)5.6.7.8"}uot;},]
>>>>
>>>> But I don't see the logs that I've written in the script after
the
>>>> event was raised, which means Kamailio did not invoke events for these
two
>>>> netstrings.
>>>>
>>>> So in my tests, out of 27232 messages sent, there were approximately
>>>> 27 messages for which the events were not raised by Evapi. The rate of
>>>> messages started at 500cps and I stopped after I saw missed events at
>>>> around 1200cps.
>>>>
>>>> Thanks, and do let me know for any further tests or information
>>>> required about the same.
>>>>
>>>> - Jayesh
>>>>
>>>>
>>>>
>>>> On Fri, Sep 18, 2015 at 2:55 PM, Jayesh Nambiar
<jayesh1017(a)gmail.com>
>>>> wrote:
>>>>
>>>>> Will have this tested by tomorrow and will get back to you. Thanks.
>>>>>
>>>>> - Jayesh
>>>>>
>>>>> On Fri, Sep 18, 2015 at 1:41 PM, Daniel-Constantin Mierla <
>>>>> miconda(a)gmail.com> wrote:
>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> I just pushed a patch to master branch that should cope with
partial
>>>>>> data received on tcp connection. No time to test at all,
therefore any
>>>>>> feedback will be appreciated.
>>>>>>
>>>>>> Cheers,
>>>>>> Daniel
>>>>>>
>>>>>>
>>>>>> On 15/09/15 14:52, Daniel-Constantin Mierla wrote:
>>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> I will look if there are options in libev to buffer data or try
to
>>>>>> implement a buffering mechanism locally for such cases.
>>>>>>
>>>>>> Cheers,
>>>>>> Daniel
>>>>>>
>>>>>> On 14/09/15 23:00, Jayesh Nambiar wrote:
>>>>>>
>>>>>> Hello Daniel,
>>>>>> After further testing with evapi module, I figured that when
>>>>>> Netstrings are used, an event route is invoked individually for
each
>>>>>> message even if if multiple netstring messages are received in a
single TCP
>>>>>> packet. But this doesn't work effectively when a single
proper message is
>>>>>> split-up in two packets. For Example, if a message arrives as:
>>>>>> 12:Hello World!, 12:Hello World!, 12:Hello World! in a single
>>>>>> packet, kamailio properly invokes the event route
"evapi:message-received"
>>>>>> thrice for every individual proper netstring message.
>>>>>> But if the first packet contains:
>>>>>> 12:Hello World!, 12:Hello
>>>>>> And Second Packet contains:
>>>>>> World!, 12:Hello World!
>>>>>> the event route is invoked only once !!
>>>>>>
>>>>>> The above pattern is very much possible while sending and
receiving
>>>>>> packets over TCP Socket. Our tests for receiving an approximately
150 byte
>>>>>> message over evapi socket at the rate of roughly 1000cps causes a
lot of
>>>>>> real events to be missed because of the above problem. You can
never be
>>>>>> sure when TCP will split messages in different chunks.
>>>>>> This definitely looks like a bug which makes it not very reliable
at
>>>>>> large scale deployments. Would really appreciate your inputs on
this.
>>>>>> Thanks;
>>>>>>
>>>>>> - Jayesh
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Thu, Sep 10, 2015 at 4:01 PM, Jayesh Nambiar <
>>>>>> jayesh1017(a)gmail.com> wrote:
>>>>>>
>>>>>>> Hi Daniel,
>>>>>>> Thanks for the quick response. So if I do not use Netstrings,
does
>>>>>>> Kamailio allow me to create a custom logic in the script. For
eg. if I
>>>>>>> decide to use newline as a delimiter, can I keep buffering
the message
>>>>>>> until I encounter the delimiter from the event route and then
execute
>>>>>>> whatever I have to within the script??
>>>>>>>
>>>>>>> Thanks,
>>>>>>>
>>>>>>> - Jayesh
>>>>>>>
>>>>>>> On Thu, Sep 10, 2015 at 1:29 PM, Daniel-Constantin Mierla
<
>>>>>>> miconda(a)gmail.com> wrote:
>>>>>>>
>>>>>>>> Hello,
>>>>>>>>
>>>>>>>> tcb is stream protocol and several messages can be queued
on the
>>>>>>>> pipe at the same time. That is the reason for netstring
format, to be able
>>>>>>>> to easily detect the boundaries of each message. If
netstring format is
>>>>>>>> enabled and kamailio receives several messages at once,
it splits them and
>>>>>>>> for each is executing the event route.
>>>>>>>>
>>>>>>>> If netstring format is not used, the kamailio is
executing the
>>>>>>>> event route with the entire content that was read at once
from the tcp
>>>>>>>> connection.
>>>>>>>>
>>>>>>>> Cheers,
>>>>>>>> Daniel
>>>>>>>>
>>>>>>>>
>>>>>>>> On 09/09/15 22:01, Jayesh Nambiar wrote:
>>>>>>>>
>>>>>>>> Hello,
>>>>>>>> I'm exploring the evapi module for my kamailio to
interface with
>>>>>>>> an external node.js app for third party stuff like AAA,
billing engine
>>>>>>>> tasks, notifications and so on. I followed and took some
ideas from the
>>>>>>>> rtjson and evapi tutorial found here(
>>>>>>>>
http://kb.asipto.com/kamailio:k43-async-sip-routing-nodejs) to
>>>>>>>> build the node.js app consuming events.
>>>>>>>> When I stress tested the scenario using SIPp and tried
sending a
>>>>>>>> lot of events at 300-350cps from Kamailio, I noticed that
at times the
>>>>>>>> client is receiving 2-3 events in a single message
together although I do
>>>>>>>> event_sync_relay once per SIP message received and have
netstrings enabled.
>>>>>>>> I believe this is a typical behavior of TCP and needs to
be handled by the
>>>>>>>> client using some kind of Netstring handler. Please
correct me if I'm wrong.
>>>>>>>> And hence I'd like to know what particularly needs to
be taken
>>>>>>>> care of while writing a client that is listening for
events on raw tcp
>>>>>>>> socket and how does kamailio handle this situation while
receiving messages
>>>>>>>> over TCP socket?? Does kamailio recognize the end of
netstring properly on
>>>>>>>> evapi:message-received and give exactly one message to
take care of on
>>>>>>>> every "message-received" event or should that
be handled in the script
>>>>>>>> somewhere !!
>>>>>>>> I also referred cgrates client over evapi example which
is written
>>>>>>>> in GO, but I couldnt find them handling TCP streams
clearly either.
>>>>>>>> I'd really appreciate some expert suggestion here to
make an
>>>>>>>> informed decision on using the evapi module for a large
scale solution.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>>
>>>>>>>> - Jayesh
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> SIP Express Router (SER) and Kamailio (OpenSER) -
sr-users mailing
listsr-users@lists.sip-router.orghttp://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Daniel-Constantin Mierlahttp://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda
>>>>>>>> Book: SIP Routing With Kamailio - http://www.asipto.com
>>>>>>>> Kamailio Advanced Training, Sep 28-30, 2015, in Berlin -
http://asipto.com/u/kat
>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> SIP Express Router (SER) and Kamailio (OpenSER) -
sr-users mailing
>>>>>>>> list
>>>>>>>> sr-users(a)lists.sip-router.org
>>>>>>>>
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> Daniel-Constantin Mierlahttp://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda
>>>>>> Book: SIP Routing With Kamailio - http://www.asipto.com
>>>>>> Kamailio Advanced Training, Sep 28-30, 2015, in Berlin -
http://asipto.com/u/kat
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Daniel-Constantin Mierlahttp://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda
>>>>>> Book: SIP Routing With Kamailio - http://www.asipto.com
>>>>>> Kamailio Advanced Training, Sep 28-30, 2015, in Berlin -
http://asipto.com/u/kat
>>>>>>
>>>>>>
>>>>>
>>>>
>>>> --
>>>> Daniel-Constantin Mierlahttp://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda
>>>> Book: SIP Routing With Kamailio - http://www.asipto.com
>>>> Kamailio Advanced Training, Sep 28-30, 2015, in Berlin -
http://asipto.com/u/kat
>>>>
>>>>
>>> --
>>> Daniel-Constantin Mierlahttp://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda
>>> Book: SIP Routing With Kamailio - http://www.asipto.com
>>> Kamailio Advanced Training, Sep 28-30, 2015, in Berlin -
http://asipto.com/u/kat
>>>
>>>
>> --
>> Daniel-Constantin Mierlahttp://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda
>> Book: SIP Routing With Kamailio - http://www.asipto.com
>> Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
>>
>>
28 Sep
28 Sep
1:43 a.m.
Hello,
I will look over it during next days.
One more thing, are the packages sent by you printed by evapi module,
even not they are not processed (but discarded)?
Cheers,
Daniel
On 25/09/15 13:55, Jayesh Nambiar wrote:
Hi Daniel,
I think this is not solved completely after you applied the patch and
my testing shows that there are some problems left out to solve.
Please do let me know if you are looking at more inputs which I might
be able to test and provide. Thanks,
- Jayesh
On Wed, Sep 23, 2015 at 8:24 PM Jayesh Nambiar <jayesh1017(a)gmail.com
<mailto:jayesh1017@gmail.com>> wrote:
Hello Daniel,
Just checking if you got what you were looking for. Do let me know
if you need more info on the same.
Thanks,
- Jayesh
On Tue, Sep 22, 2015 at 8:23 PM Jayesh Nambiar
<jayesh1017(a)gmail.com <mailto:jayesh1017@gmail.com>> wrote:
Hi,
Here's the NGREP trace
where
{"event":"REGISTER","tindex":"3639","tlabel":"1889788252"
...
is present on the first line itself.
198.24.63.45:48886 <http://198.24.63.45:48886> ->
198.24.63.39:3927 <http://198.24.63.39:3927> [AP]
146:{"event":"REGISTER","tindex":"3639","tlabel":"1889788252","PhoneNumber":"42008","DeviceId":"abcd1234abcd1234","CallId":"11542-19872@198.24.63.39
<mailto:11542-19872@198.24.63.39>"},147:{"event":"REGISTER","tindex":"12600","tlabel":"1916422130","PhoneNumber":"42009","DeviceId":"abcd1234abcd1234","CallId":"11543-19872@198.24.63.39
<mailto:11543-19872@198.24.63.39>"},147:{"event":"REGISTER","tindex":"51251","tlabel":"1549871937","PhoneNumber":"42010","DeviceId":"abcd1234abcd
1234","CallId":"11544-19872@198.24.63.39
<mailto:11544-19872@198.24.63.39>"},147:{"event":"REGISTER","tindex":"60213","tlabel":"2100336684","PhoneNumber":"42011","DeviceId":"abcd1234abcd1234","CallId":"11545-19872@198.24.63.39
<mailto:11545-19872@198.24.63.39>"},146:{"event
":"REGISTER","tindex":"33328","tlabel":"622251552","PhoneNumber":"42012","DeviceId":"abcd1234abcd1234","CallId":"11546-19872@198.24.63.39
<mailto:11546-19872@198.24.63.39>"},147:{"event":"REGISTER","tindex":"42289","tlabel":"1360864805","PhoneNumber":"42013","DeviceId":"abcd1234abcd1234","CallId":"11547-19872@198.24.63.39
<mailto:11547-19872@198.24.63.39>"},146:{"event":"REGISTER","tindex":"48204","tlabel":"315887672","PhoneNumber":"42014","DeviceId":"abcd1234abcd1234","CallId":"11548-19872@198.24.63.39
<mailto:11548-19872@198.24.63.39>"},145:{"event":"REGISTER","tindex":"36682","tlabel":"18660440","PhoneNumber":"42015","DeviceId":"abcd1234abcd1234","CallId":"11549-19872@198.24.63.39
<mailto:11549-19872@198.24.63.39>"},
And yes, this packet is right after the second packet in the
pastebin trace.
Thanks,
- Jayesh
On Tue, Sep 22, 2015 at 5:03 PM Daniel-Constantin Mierla
<miconda(a)gmail.com <mailto:miconda@gmail.com>> wrote:
Hello,
I haven't seen in the ngrep when the data containing next
line is received:
{"event":"REGISTER","tindex":"3639","tlabel":"1889788252"
...
Do you still have it? Is it after:
{"event":"REGISTER","tindex":"12603","tlabel":"626010915",
...
Cheers,
Daniel
On 22/09/15 09:09, Jayesh Nambiar wrote:
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Book: SIP Routing With Kamailio - http://www.asipto.com
Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
Hi,
Took time to get back with the test results as I was
trying to analyze too much myself before getting back on
the list. Also I was stuck with weird error of module
version mismatch, but I figured and solved that issue.
So I do see the logs "frame size mismatch" when I miss
the events. Since the logs are too huge, I am sending it
in pastebin. Here it is:
http://pastebin.com/MzTpYsrk
In Kamailio Logs, I see:
DEBUG: evapi [evapi_dispatch.c:361]: evapi_recv_client():
frame size mismatch the ending char
(h):[{"event":"REGISTER","tindex":"12603","tlabel":"626010915","PhoneNumber":"42000","D146:"event":"REGISTER","tindex":"3639","tlabel":"1889788252","P]
(146)
Whereas the messages in ngrep looks good where the
messages are split in two TCP packets. The first packet
ends at D and the second packet is the proper
continuation which send the rest. But it feels Kamailio
skipped the beginning of the message and started parsing
at colon again and it thinks the netstring started with
D146 which is incorrect. Hence, it never emitted any
event for the messages in the first chunk.
Hoping this to be helpful !! Thanks.
- Jayesh
On Mon, Sep 21, 2015 at 4:37 PM Daniel-Constantin Mierla
<miconda(a)gmail.com <mailto:miconda@gmail.com>> wrote:
Hello,
I pushed another commit to add more debug messages to
see if the event route is supposed to be executed or
not. Can you run the tests again and give again the
log messages for missing event route executions?
Cheers,
Daniel
On 21/09/15 09:04, Jayesh Nambiar wrote:
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda
<http://twitter.com/#%21/miconda> - http://www.linkedin.com/in/miconda
Book: SIP Routing With Kamailio - http://www.asipto.com
Kamailio Advanced Training, Sep 28-30, 2015, in Berlin -
http://asipto.com/u/kat
Hi Daniel,
I have captured debug messages after testing with
your latest commit on master. Once again, here's
what I'm doing exactly for my test:
1) Use SIPp to send OPTIONS towards Kamailio.
2) On getting options send event as follows:
evapi_async_relay("{\"event\":\"REGISTER\",\"tindex\":\"$T(id_index)\",\"tlabel\":\"$T(id_label)\",\"PhoneNumber\":\"$avp(phone_number)\",\"DeviceId\":\"$avp(device_id)\",\"CallId\":\"$ci\"}");
3) There's a client connected which listens for
messages on this socket, parses the netstring, and
sends same data back as netstring to Kamailio.
4) On the event_route[evapi:message-received, I do
the following:
xlog("L_INFO", "GOT [$evapi(msg)] from
$evapi(srcaddr):$evapi(srcport)\n");
if($evapi(msg)=~"REGISTER" &&
$evapi(msg)=~"tindex") {
jansson_get_field("$evapi(msg)", "tlabel",
"$var(tlabel)");
jansson_get_field("$evapi(msg)", "tindex",
"$var(tindex)");
$var(t_index) = $(var(tindex){s.int
<http://s.int>});
$var(t_label) = $(var(tlabel){s.int
<http://s.int>});
t_continue('$var(t_index)', '$var(t_label)',
'REGISTER_RESPONSE');
exit;
}
5) On route[REGISTER_RESPONSE], I send 200 OK to SIPp
Now here are detailed logs for which neither SIPp
didnt get a response nor the GOT [$evapi(msg)] from
$evapi(srcaddr):$evapi(srcport)\n"); got logged in
the syslog:
First Event:
DEBUG: evapi [evapi_dispatch.c:598]: evapi_relay():
relaying event data
[{"event":"REGISTER","tindex":"26266","tlabel":"587925078","PhoneNumber":"21956","DeviceId":"abcd1234abcd1234","CallId":"23021-12910@198.24.63.39
<mailto:23021-12910@198.24.63.39>"}] (146)
DEBUG: evapi [evapi_dispatch.c:623]: evapi_relay():
sending [0x7f132588de68]
[146:{"event":"REGISTER","tindex":"26266","tlabel":"587925078","PhoneNumber":"21956","DeviceId":"abcd1234abcd1234","CallId":"23021-12910@198.24.63.39
<mailto:23021-12910@198.24.63.39>"},] (151)
DEBUG: evapi [evapi_dispatch.c:488]:
evapi_recv_notify(): received [0x7f132588de68]
[146:{"event":"REGISTER","tindex":"26266","tlabel":"587925078","PhoneNumber":"21956","DeviceId":"abcd1234abcd1234","CallId":"23021-12910@198.24.63.39
<mailto:23021-12910@198.24.63.39>"},] (151)
NOTICE: evapi [evapi_dispatch.c:290]:
evapi_recv_client(): {0} [198.24.63.45:48881
<http://198.24.63.45:48881>] - received
[146:{"event":"REGISTER","tindex":"26266","tlabel":"587925078","PhoneNumber":"21956","DeviceId":"abcd1234abcd1234","CallId":"23021-12910@198.24.63.39
<mailto:23021-12910@198.24.63.39>"},] (151) (18)
Second Event:
DEBUG: evapi [evapi_dispatch.c:598]: evapi_relay():
relaying event data
[{"event":"REGISTER","tindex":"32244","tlabel":"1637923412","PhoneNumber":"25597","DeviceId":"abcd1234abcd1234","CallId":"27182-12910@198.24.63.39
<mailto:27182-12910@198.24.63.39>"}] (147)
DEBUG: evapi [evapi_dispatch.c:623]:
evapi_relay(): sending [0x7f1325884410]
[147:{"event":"REGISTER","tindex":"32244","tlabel":"1637923412","PhoneNumber":"25597","DeviceId":"abcd1234abcd1234","CallId":"27182-12910@198.24.63.39
<mailto:27182-12910@198.24.63.39>"},] (152)
DEBUG: evapi [evapi_dispatch.c:488]:
evapi_recv_notify(): received [0x7f1325884410]
[147:{"event":"REGISTER","tindex":"32244","tlabel":"1637923412","PhoneNumber":"25597","DeviceId":"abcd1234abcd1234","CallId":"27182-12910@198.24.63.39
<mailto:27182-12910@198.24.63.39>"},] (152)
NOTICE: evapi [evapi_dispatch.c:290]:
evapi_recv_client(): {0} [198.24.63.45:48881
<http://198.24.63.45:48881>] - received
[147:{"event":"REGISTER","tindex":"32244","tlabel":"1637923412","PhoneNumber":"25597","DeviceId":"abcd1234abcd1234","CallId":"27182-12910@198.24.63.39
<mailto:27182-12910@198.24.63.39>"},] (152) (85)
I had three such events which got missed out of some
25000 odd messages sent from SIPp. Other major
difference that I see in the debug logs for
problematic events are that there is a positive
number in the second parentheses
of evapi_recv_client() function. For the events that
were invoked successfully the value of second
parentheses is 0 for evapi_recv_client() function.
Hope the debugging helps.
Debug log for a successful event:
DEBUG: evapi [evapi_dispatch.c:598]: evapi_relay():
relaying event data
[{"event":"REGISTER","tindex":"22127","tlabel":"1896682192","PhoneNumber":"73168","DeviceId":"abcd1234abcd1234","CallId":"49957-13056@198.24.63.39
<mailto:49957-13056@198.24.63.39>"}] (147)
DEBUG: evapi [evapi_dispatch.c:623]:
evapi_relay(): sending [0x7f132568e850]
[147:{"event":"REGISTER","tindex":"22127","tlabel":"1896682192","PhoneNumber":"73168","DeviceId":"abcd1234abcd1234","CallId":"49957-13056@198.24.63.39
<mailto:49957-13056@198.24.63.39>"},] (152)
DEBUG: evapi [evapi_dispatch.c:488]:
evapi_recv_notify(): received [0x7f132568e850]
[147:{"event":"REGISTER","tindex":"22127","tlabel":"1896682192","PhoneNumber":"73168","DeviceId":"abcd1234abcd1234","CallId":"49957-13056@198.24.63.39
<mailto:49957-13056@198.24.63.39>"},] (152)
NOTICE: evapi [evapi_dispatch.c:290]:
evapi_recv_client(): {0} [198.24.63.45:48881
<http://198.24.63.45:48881>] - received
[147:{"event":"REGISTER","tindex":"22127","tlabel":"1896682192","PhoneNumber":"73168","DeviceId":"abcd1234abcd1234","CallId":"49957-13056@198.24.63.39
<mailto:49957-13056@198.24.63.39>"},] (152) (0)
Thanks,
- Jayesh
On Sat, Sep 19, 2015 at 3:55 AM Daniel-Constantin
Mierla <miconda(a)gmail.com
<mailto:miconda@gmail.com>> wrote:
I pushed a commit to add more debug message
while processing received data. You can use
debugger module to set a higher debug level for
evapi module in order to see what happens.
I checked the netstring packet size and it is
invalid (unless email stripped some white chars
there) -- for example in:
145:{"event":"REGISTER","tindex":"55567","tlabel":"627458699","PhoneNumber":"20711","DeviceId":"abcd1234abcd1234","CallId":"21225-3848@5.6.7.8"
<mailto:21225-3848@5.6.7.8>},
the size is 141, not 145 -- it looks like the
size includes the size itself plus the
delimiters ':,'. The size is only the effective
data, see:
https://en.wikipedia.org/wiki/Netstring
If evapi gets a packet with an invalid size,
then it discards the buffer content.
See if the app on the other side of evapi
connection builds netstrings with wrong size.
Cheers,
Daniel
On 18/09/15 23:19, Jayesh Nambiar wrote:
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda
<http://twitter.com/#%21/miconda> -
http://www.linkedin.com/in/miconda
Book: SIP Routing With Kamailio - http://www.asipto.com
Kamailio Advanced Training, Sep 28-30, 2015, in Berlin -
http://asipto.com/u/kat
Here are the tests that I
did:
With the patch applied, I see that Kamailio is
invoking event individually for each netstring
even when they come in different chunks. But I
did see instances where when there were
complete netstrings in a single chunk; kamailio
did not raise an event for them. Here's the
illustration:
T 1.2.3.4:48873 <http://1.2.3.4:48873> ->
5.6.7.8:3927 <http://5.6.7.8:3927> [AP]
146:{"event":"REGISTER","tindex":"16916","tlabel":"1753048277","PhoneNumber":"20708","DeviceId":"abcd1234abcd1234","CallId":"21220-3848@5.6.7.8
<mailto:21220-3848@5.6.7.8>"},144:{"event":"REGISTER","tindex":"7954","tlabel":"254315075","PhoneNumber":"20709","DeviceId":"abcd1234abcd1234","CallId":"21223-3848@5.6.7.8
<mailto:21223-3848@5.6.7.8>"},145:{"event":"REGISTER","tindex":"64529","tlabel":"599481568","PhoneNumber":"20709","DeviceId":"abcd1234abcd1234","CallId":"21222-3848@5.6.7.8
<mailto:21222-3848@5.6.7.8>"},145:{"event":"REGISTER","tindex":"46605","tlabel":"112015324","PhoneNumber":"20710","DeviceId"
T 1.2.3.4:48873 <http://1.2.3.4:48873> ->
5.6.7.8:3927 <http://5.6.7.8:3927> [AP]
:"abcd1234abcd1234","CallId":"21224-3848@5.6.7.8
<mailto:21224-3848@5.6.7.8>"},
*The above two chunks contain 4 proper
netstrings where the second chunk contains part
of the 4th netstring. In this case Evapi
properly raised 4 individual events.*
1.2.3.4:48873 <http://1.2.3.4:48873> ->
5.6.7.8:3927 <http://5.6.7.8:3927> [AP]
145:{"event":"REGISTER","tindex":"55567","tlabel":"627458699","PhoneNumber":"20711","DeviceId":"abcd1234abcd1234","CallId":"21225-3848@5.6.7.8
<mailto:21225-3848@5.6.7.8>"},143:{"event":"REGISTER","tindex":"28682","tlabel":"9676691","PhoneNumber":"20712","DeviceId":"abcd1234abcd1234","CallId":"21226-3848@5.6.7.8
<mailto:21226-3848@5.6.7.8>"},
*This above chunk contains two complete
netstrings but Kamailio never raised events for
these two netstrings.*
*
*
For the events that were not raised I see
proper Kamailio Logs which is:
evapi_recv_client(): {0} [1.2.3.4:48873
<http://1.2.3.4:48873>] - received
[145:{"event":"REGISTER","tindex":"55567","tlabel":"627458699","PhoneNumber":"20711","DeviceId":"abcd1234abcd1234","CallId":"21225-3848@5.6.7.8
<mailto:21225-3848@5.6.7.8>"},143:{"event":"REGISTER","tindex":"28682","tlabel":"9676691","PhoneNumber":"20712","DeviceId":"abcd1234abcd1234","CallId":"21226-3848@5.6.7.8
<mailto:21226-3848@5.6.7.8>"},]
*
*
But I don't see the logs that I've written in
the script after the event was raised, which
means Kamailio did not invoke events for these
two netstrings.
So in my tests, out of 27232 messages sent,
there were approximately 27 messages for which
the events were not raised by Evapi. The rate
of messages started at 500cps and I stopped
after I saw missed events at around 1200cps.
Thanks, and do let me know for any further
tests or information required about the same.
- Jayesh
*
*
On Fri, Sep 18, 2015 at 2:55 PM, Jayesh Nambiar
<jayesh1017(a)gmail.com
<mailto:jayesh1017@gmail.com>> wrote:
Will have this tested by tomorrow and will
get back to you. Thanks.
- Jayesh
On Fri, Sep 18, 2015 at 1:41 PM,
Daniel-Constantin Mierla <miconda(a)gmail.com
<mailto:miconda@gmail.com>> wrote:
Hello,
I just pushed a patch to master branch
that should cope with partial data
received on tcp connection. No time to
test at all, therefore any feedback
will be appreciated.
Cheers,
Daniel
On 15/09/15 14:52, Daniel-Constantin
Mierla wrote:
> Hello,
>
> I will look if there are options in
> libev to buffer data or try to
> implement a buffering mechanism
> locally for such cases.
>
> Cheers,
> Daniel
>
> On 14/09/15 23:00, Jayesh Nambiar wrote:
>> Hello Daniel,
>> After further testing with evapi
>> module, I figured that when
>> Netstrings are used, an event route
>> is invoked individually for each
>> message even if if multiple netstring
>> messages are received in a single TCP
>> packet. But this doesn't work
>> effectively when a single proper
>> message is split-up in two packets.
>> For Example, if a message arrives as:
>> 12:Hello World!, 12:Hello World!,
>> 12:Hello World! in a single packet,
>> kamailio properly invokes the event
>> route "evapi:message-received" thrice
>> for every individual proper netstring
>> message.
>> But if the first packet contains:
>> 12:Hello World!, 12:Hello
>> And Second Packet contains:
>> World!, 12:Hello World!
>> the event route is invoked only once !!
>>
>> The above pattern is very much
>> possible while sending and receiving
>> packets over TCP Socket. Our tests
>> for receiving an approximately 150
>> byte message over evapi socket at the
>> rate of roughly 1000cps causes a lot
>> of real events to be missed because
>> of the above problem. You can never
>> be sure when TCP will split messages
>> in different chunks.
>> This definitely looks like a bug
>> which makes it not very reliable at
>> large scale deployments. Would really
>> appreciate your inputs on this. Thanks;
>>
>> - Jayesh
>>
>>
>>
>> On Thu, Sep 10, 2015 at 4:01 PM,
>> Jayesh Nambiar <jayesh1017(a)gmail.com
>> <mailto:jayesh1017@gmail.com>> wrote:
>>
>> Hi Daniel,
>> Thanks for the quick response. So
>> if I do not use Netstrings, does
>> Kamailio allow me to create a
>> custom logic in the script. For
>> eg. if I decide to use newline as
>> a delimiter, can I keep buffering
>> the message until I encounter the
>> delimiter from the event route
>> and then execute whatever I have
>> to within the script??
>>
>> Thanks,
>>
>> - Jayesh
>>
>> On Thu, Sep 10, 2015 at 1:29 PM,
>> Daniel-Constantin Mierla
>> <miconda(a)gmail.com
>> <mailto:miconda@gmail.com>> wrote:
>>
>> Hello,
>>
>> tcb is stream protocol and
>> several messages can be
>> queued on the pipe at the
>> same time. That is the reason
>> for netstring format, to be
>> able to easily detect the
>> boundaries of each message.
>> If netstring format is
>> enabled and kamailio receives
>> several messages at once, it
>> splits them and for each is
>> executing the event route.
>>
>> If netstring format is not
>> used, the kamailio is
>> executing the event route
>> with the entire content that
>> was read at once from the tcp
>> connection.
>>
>> Cheers,
>> Daniel
>>
>>
>> On 09/09/15 22:01, Jayesh
>> Nambiar wrote:
>>> Hello,
>>> I'm exploring the evapi
>>> module for my kamailio to
>>> interface with an external
>>> node.js app for third party
>>> stuff like AAA, billing
>>> engine tasks, notifications
>>> and so on. I followed and
>>> took some ideas from the
>>> rtjson and evapi tutorial
>>> found
>>>
here(http://kb.asipto.com/kamailio:k43-async-sip-routing-nodejs)
>>> to build the node.js app
>>> consuming events.
>>> When I stress tested the
>>> scenario using SIPp and
>>> tried sending a lot of
>>> events at 300-350cps from
>>> Kamailio, I noticed that at
>>> times the client is
>>> receiving 2-3 events in a
>>> single message together
>>> although I do
>>> event_sync_relay once per
>>> SIP message received and
>>> have netstrings enabled. I
>>> believe this is a typical
>>> behavior of TCP and needs to
>>> be handled by the client
>>> using some kind of Netstring
>>> handler. Please correct me
>>> if I'm wrong.
>>> And hence I'd like to know
>>> what particularly needs to
>>> be taken care of while
>>> writing a client that is
>>> listening for events on raw
>>> tcp socket and how does
>>> kamailio handle this
>>> situation while receiving
>>> messages over TCP socket??
>>> Does kamailio recognize the
>>> end of netstring properly on
>>> evapi:message-received and
>>> give exactly one message to
>>> take care of on every
>>> "message-received" event or
>>> should that be handled in
>>> the script somewhere !!
>>> I also referred cgrates
>>> client over evapi example
>>> which is written in GO, but
>>> I couldnt find them handling
>>> TCP streams clearly either.
>>> I'd really appreciate some
>>> expert suggestion here to
>>> make an informed decision on
>>> using the evapi module for a
>>> large scale solution.
>>>
>>> Thanks,
>>>
>>> - Jayesh
>>>
>>>
>>>
>>>
>>>
_______________________________________________
>>> SIP Express Router (SER) and Kamailio
(OpenSER) - sr-users mailing list
>>> sr-users(a)lists.sip-router.org
<mailto:sr-users@lists.sip-router.org>
>>>
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
>>
>> --
>> Daniel-Constantin Mierla
>> http://twitter.com/#!/miconda
>> <http://twitter.com/#%21/miconda> -
http://www.linkedin.com/in/miconda
>> Book: SIP Routing With Kamailio -
http://www.asipto.com
>> Kamailio Advanced Training, Sep 28-30, 2015,
in Berlin - http://asipto.com/u/kat
>>
>>
>>
_______________________________________________
>> SIP Express Router (SER) and
>> Kamailio (OpenSER) - sr-users
>> mailing list
>> sr-users(a)lists.sip-router.org
>> <mailto:sr-users@lists.sip-router.org>
>>
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
>>
>>
>>
>
> --
> Daniel-Constantin Mierla
> http://twitter.com/#!/miconda
> <http://twitter.com/#%21/miconda> -
http://www.linkedin.com/in/miconda
> Book: SIP Routing With Kamailio - http://www.asipto.com
> Kamailio Advanced Training, Sep 28-30, 2015, in Berlin -
http://asipto.com/u/kat
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda
<http://twitter.com/#%21/miconda> -
http://www.linkedin.com/in/miconda
Book: SIP Routing With Kamailio - http://www.asipto.com
Kamailio Advanced Training, Sep 28-30, 2015, in Berlin -
http://asipto.com/u/kat
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda
<http://twitter.com/#%21/miconda> -
http://www.linkedin.com/in/miconda
Book: SIP Routing With Kamailio - http://www.asipto.com
Kamailio Advanced Training, Sep 28-30, 2015, in Berlin -
http://asipto.com/u/kat
11:04 a.m.
Thank you daniel.
Yes, they all get printed by the evapi module even when they get discarded.
If you once again check the paste-bin, the logs pasted as Kamailio Logs are
all discarded packets but show up as received by evapi module.
I believe they got discarded because a packet that came before them was
partial and evapi discarded the remaining part of the message in the new
packet and encountered error.
- Jayesh
On Mon, Sep 28, 2015 at 11:13 AM Daniel-Constantin Mierla <miconda(a)gmail.com>
wrote:
Hello,
I will look over it during next days.
One more thing, are the packages sent by you printed by evapi module, even
not they are not processed (but discarded)?
Cheers,
Daniel
On 25/09/15 13:55, Jayesh Nambiar wrote:
Hi Daniel,
I think this is not solved completely after you applied the patch and my
testing shows that there are some problems left out to solve. Please do let
me know if you are looking at more inputs which I might be able to test and
provide. Thanks,
- Jayesh
On Wed, Sep 23, 2015 at 8:24 PM Jayesh Nambiar <jayesh1017(a)gmail.com>
wrote:
Hello Daniel,
Just checking if you got what you were looking for. Do let me know if you
need more info on the same.
Thanks,
- Jayesh
On Tue, Sep 22, 2015 at 8:23 PM Jayesh Nambiar <jayesh1017(a)gmail.com>
wrote:
> Hi,
> Here's the NGREP trace where
{"event":"REGISTER","tindex":"3639","tlabel":"1889788252"
> ... is present on the first line itself.
>
> 198.24.63.45:48886 -> 198.24.63.39:3927 [AP]
>
>
146:{"event":"REGISTER","tindex":"3639","tlabel":"1889788252","PhoneNumber":"42008","DeviceId":"abcd1234abcd1234","CallId":"
> <11542-19872@198.24.63.39>11542-19872@198.24.63.39
>
"},147:{"event":"REGISTER","tindex":"12600","tlabel":"1916422130","PhoneNumber":"42009","DeviceId":"abcd1234abcd1234","CallId":"
> <11543-19872@198.24.63.39>11543-19872@198.24.63.39
>
"},147:{"event":"REGISTER","tindex":"51251","tlabel":"1549871937","PhoneNumber":"42010","DeviceId":"abcd1234abcd
> 1234","CallId":"11544-19872@198.24.63.39
>
"},147:{"event":"REGISTER","tindex":"60213","tlabel":"2100336684","PhoneNumber":"42011","DeviceId":"abcd1234abcd1234","CallId":"
> <11545-19872@198.24.63.39>11545-19872@198.24.63.39"},146:{"event
>
>
":"REGISTER","tindex":"33328","tlabel":"622251552","PhoneNumber":"42012","DeviceId":"abcd1234abcd1234","CallId":"
> <11546-19872@198.24.63.39>11546-19872@198.24.63.39
>
"},147:{"event":"REGISTER","tindex":"42289","tlabel":"1360864805","PhoneNumber":"42013","DeviceId":"abcd1234abcd1234","CallId":"
> <11547-19872@198.24.63.39>11547-19872@198.24.63.39
>
"},146:{"event":"REGISTER","tindex":"48204","tlabel":"315887672","PhoneNumber":"42014","DeviceId":"abcd1234abcd1234","CallId":"
> <11548-19872@198.24.63.39>11548-19872@198.24.63.39
>
"},145:{"event":"REGISTER","tindex":"36682","tlabel":"18660440","PhoneNumber":"42015","DeviceId":"abcd1234abcd1234","CallId":"
> <11549-19872@198.24.63.39>11549-19872@198.24.63.39"},
>
> And yes, this packet is right after the second packet in the pastebin
> trace.
>
> Thanks,
>
> - Jayesh
>
>
>
> On Tue, Sep 22, 2015 at 5:03 PM Daniel-Constantin Mierla <
> miconda(a)gmail.com> wrote:
>
>> Hello,
>>
>> I haven't seen in the ngrep when the data containing next line is
>> received:
>>
>>
{"event":"REGISTER","tindex":"3639","tlabel":"1889788252"
...
>>
>> Do you still have it? Is it after:
>>
>>
{"event":"REGISTER","tindex":"12603","tlabel":"626010915",
...
>>
>> Cheers,
>> Daniel
>>
>>
>> On 22/09/15 09:09, Jayesh Nambiar wrote:
>>
>> Hi,
>> Took time to get back with the test results as I was trying to analyze
>> too much myself before getting back on the list. Also I was stuck with
>> weird error of module version mismatch, but I figured and solved that issue.
>> So I do see the logs "frame size mismatch" when I miss the events.
>> Since the logs are too huge, I am sending it in pastebin. Here it is:
>> http://pastebin.com/MzTpYsrk
>>
>> In Kamailio Logs, I see:
>> DEBUG: evapi [evapi_dispatch.c:361]: evapi_recv_client(): frame size
>> mismatch the ending char
>>
(h):[{"event":"REGISTER","tindex":"12603","tlabel":"626010915
>>
","PhoneNumber":"42000","D146:"event":"REGISTER","tindex":"3639","tlabel":"
>> 1889788252","P] (146)
>>
>> Whereas the messages in ngrep looks good where the messages are split
>> in two TCP packets. The first packet ends at D and the second packet is the
>> proper continuation which send the rest. But it feels Kamailio skipped the
>> beginning of the message and started parsing at colon again and it thinks
>> the netstring started with D146 which is incorrect. Hence, it never emitted
>> any event for the messages in the first chunk.
>>
>> Hoping this to be helpful !! Thanks.
>>
>> - Jayesh
>>
>>
>> On Mon, Sep 21, 2015 at 4:37 PM Daniel-Constantin Mierla <
>> <miconda@gmail.com>miconda@gmail.com> wrote:
>>
>>> Hello,
>>>
>>> I pushed another commit to add more debug messages to see if the event
>>> route is supposed to be executed or not. Can you run the tests again and
>>> give again the log messages for missing event route executions?
>>>
>>> Cheers,
>>> Daniel
>>>
>>>
>>> On 21/09/15 09:04, Jayesh Nambiar wrote:
>>>
>>> Hi Daniel,
>>> I have captured debug messages after testing with your latest commit
>>> on master. Once again, here's what I'm doing exactly for my test:
>>> 1) Use SIPp to send OPTIONS towards Kamailio.
>>> 2) On getting options send event as follows:
>>>
>>>
evapi_async_relay("{\"event\":\"REGISTER\",\"tindex\":\"$T(id_index)\",\"tlabel\":\"$T(id_label)\",\"PhoneNumber\":\"$avp(phone_number)\",\"DeviceId\":\"$avp(device_id)\",\"CallId\":\"$ci\"}");
>>> 3) There's a client connected which listens for messages on this
>>> socket, parses the netstring, and sends same data back as netstring to
>>> Kamailio.
>>> 4) On the event_route[evapi:message-received, I do the following:
>>> xlog("L_INFO", "GOT [$evapi(msg)] from
>>> $evapi(srcaddr):$evapi(srcport)\n");
>>> if($evapi(msg)=~"REGISTER" &&
$evapi(msg)=~"tindex") {
>>> jansson_get_field("$evapi(msg)", "tlabel",
"$var(tlabel)");
>>> jansson_get_field("$evapi(msg)", "tindex",
"$var(tindex)");
>>> $var(t_index) = $(var(tindex){s.int});
>>> $var(t_label) = $(var(tlabel){s.int});
>>> t_continue('$var(t_index)', '$var(t_label)',
>>> 'REGISTER_RESPONSE');
>>> exit;
>>> }
>>> 5) On route[REGISTER_RESPONSE], I send 200 OK to SIPp
>>>
>>> Now here are detailed logs for which neither SIPp didnt get a response
>>> nor the GOT [$evapi(msg)] from $evapi(srcaddr):$evapi(srcport)\n"); got
>>> logged in the syslog:
>>>
>>> First Event:
>>> DEBUG: evapi [evapi_dispatch.c:598]: evapi_relay(): relaying event
>>> data
>>>
[{"event":"REGISTER","tindex":"26266","tlabel":"587925078","PhoneNumber":"21956","DeviceId":"abcd1234abcd1234","CallId":"
>>> <23021-12910@198.24.63.39>23021-12910@198.24.63.39"}] (146)
>>> DEBUG: evapi [evapi_dispatch.c:623]: evapi_relay(): sending
>>> [0x7f132588de68]
>>>
[146:{"event":"REGISTER","tindex":"26266","tlabel":"587925078","PhoneNumber":"21956","DeviceId":"abcd1234abcd1234","CallId":"
>>> <23021-12910@198.24.63.39>23021-12910@198.24.63.39"},] (151)
>>>
>>> DEBUG: evapi [evapi_dispatch.c:488]: evapi_recv_notify(): received
>>> [0x7f132588de68]
>>>
[146:{"event":"REGISTER","tindex":"26266","tlabel":"587925078","PhoneNumber":"21956","DeviceId":"abcd1234abcd1234","CallId":"
>>> <23021-12910@198.24.63.39>23021-12910@198.24.63.39"},] (151)
>>> NOTICE: evapi [evapi_dispatch.c:290]: evapi_recv_client(): {0} [
>>> 198.24.63.45:48881] - received
>>>
[146:{"event":"REGISTER","tindex":"26266","tlabel":"587925078","PhoneNumber":"21956","DeviceId":"abcd1234abcd1234","CallId":"
>>> <23021-12910@198.24.63.39>23021-12910@198.24.63.39"},] (151) (18)
>>>
>>> Second Event:
>>> DEBUG: evapi [evapi_dispatch.c:598]: evapi_relay(): relaying event
>>> data
>>>
[{"event":"REGISTER","tindex":"32244","tlabel":"1637923412","PhoneNumber":"25597","DeviceId":"abcd1234abcd1234","CallId":"
>>> <27182-12910@198.24.63.39>27182-12910@198.24.63.39"}] (147)
>>> DEBUG: evapi [evapi_dispatch.c:623]: evapi_relay(): sending
>>> [0x7f1325884410]
>>>
[147:{"event":"REGISTER","tindex":"32244","tlabel":"1637923412","PhoneNumber":"25597","DeviceId":"abcd1234abcd1234","CallId":"
>>> <27182-12910@198.24.63.39>27182-12910@198.24.63.39"},] (152)
>>>
>>> DEBUG: evapi [evapi_dispatch.c:488]: evapi_recv_notify(): received
>>> [0x7f1325884410]
>>>
[147:{"event":"REGISTER","tindex":"32244","tlabel":"1637923412","PhoneNumber":"25597","DeviceId":"abcd1234abcd1234","CallId":"
>>> <27182-12910@198.24.63.39>27182-12910@198.24.63.39"},] (152)
>>> NOTICE: evapi [evapi_dispatch.c:290]: evapi_recv_client(): {0} [
>>> 198.24.63.45:48881] - received
>>>
[147:{"event":"REGISTER","tindex":"32244","tlabel":"1637923412","PhoneNumber":"25597","DeviceId":"abcd1234abcd1234","CallId":"
>>> <27182-12910@198.24.63.39>27182-12910@198.24.63.39"},] (152) (85)
>>>
>>> I had three such events which got missed out of some 25000 odd
>>> messages sent from SIPp. Other major difference that I see in the debug
>>> logs for problematic events are that there is a positive number in the
>>> second parentheses of evapi_recv_client() function. For the events that
>>> were invoked successfully the value of second parentheses is 0 for
>>> evapi_recv_client() function. Hope the debugging helps.
>>>
>>> Debug log for a successful event:
>>> DEBUG: evapi [evapi_dispatch.c:598]: evapi_relay(): relaying event
>>> data
>>>
[{"event":"REGISTER","tindex":"22127","tlabel":"1896682192","PhoneNumber":"73168","DeviceId":"abcd1234abcd1234","CallId":"
>>> <49957-13056@198.24.63.39>49957-13056@198.24.63.39"}] (147)
>>> DEBUG: evapi [evapi_dispatch.c:623]: evapi_relay(): sending
>>> [0x7f132568e850]
>>>
[147:{"event":"REGISTER","tindex":"22127","tlabel":"1896682192","PhoneNumber":"73168","DeviceId":"abcd1234abcd1234","CallId":"
>>> <49957-13056@198.24.63.39>49957-13056@198.24.63.39"},] (152)
>>>
>>> DEBUG: evapi [evapi_dispatch.c:488]: evapi_recv_notify(): received
>>> [0x7f132568e850]
>>>
[147:{"event":"REGISTER","tindex":"22127","tlabel":"1896682192","PhoneNumber":"73168","DeviceId":"abcd1234abcd1234","CallId":"
>>> <49957-13056@198.24.63.39>49957-13056@198.24.63.39"},] (152)
>>> NOTICE: evapi [evapi_dispatch.c:290]: evapi_recv_client(): {0} [
>>> 198.24.63.45:48881] - received
>>>
[147:{"event":"REGISTER","tindex":"22127","tlabel":"1896682192","PhoneNumber":"73168","DeviceId":"abcd1234abcd1234","CallId":"
>>> <49957-13056@198.24.63.39>49957-13056@198.24.63.39"},] (152) (0)
>>>
>>> Thanks,
>>>
>>> - Jayesh
>>> On Sat, Sep 19, 2015 at 3:55 AM Daniel-Constantin Mierla <
>>> <miconda@gmail.com>miconda@gmail.com> wrote:
>>>
>>>> I pushed a commit to add more debug message while processing received
>>>> data. You can use debugger module to set a higher debug level for evapi
>>>> module in order to see what happens.
>>>>
>>>> I checked the netstring packet size and it is invalid (unless email
>>>> stripped some white chars there) -- for example in:
>>>>
>>>>
>>>>
>>>>
145:{"event":"REGISTER","tindex":"55567","tlabel":"627458699","PhoneNumber":"20711","DeviceId":"abcd1234abcd1234","CallId":
>>>> <21225-3848@5.6.7.8>"21225-3848@5.6.7.8"
<21225-3848(a)5.6.7.8>}8>},
>>>>
>>>> the size is 141, not 145 -- it looks like the size includes the size
>>>> itself plus the delimiters ':,'. The size is only the effective
data, see:
>>>>
>>>> https://en.wikipedia.org/wiki/Netstring
>>>>
>>>> If evapi gets a packet with an invalid size, then it discards the
>>>> buffer content.
>>>>
>>>> See if the app on the other side of evapi connection builds
>>>> netstrings with wrong size.
>>>>
>>>> Cheers,
>>>> Daniel
>>>>
>>>>
>>>> On 18/09/15 23:19, Jayesh Nambiar wrote:
>>>>
>>>> Here are the tests that I did:
>>>>
>>>> With the patch applied, I see that Kamailio is invoking event
>>>> individually for each netstring even when they come in different chunks.
>>>> But I did see instances where when there were complete netstrings in a
>>>> single chunk; kamailio did not raise an event for them. Here's the
>>>> illustration:
>>>>
>>>> T 1.2.3.4:48873 -> 5.6.7.8:3927 [AP]
>>>>
>>>>
146:{"event":"REGISTER","tindex":"16916","tlabel":"1753048277","PhoneNumber":"20708","DeviceId":"abcd1234abcd1234","CallId":"
>>>> <21220-3848@5.6.7.8>21220-3848@5.6.7.8
>>>>
"},144:{"event":"REGISTER","tindex":"7954","tlabel":"254315075","PhoneNumber":"20709","DeviceId":"abcd1234abcd1234","CallId":"
>>>> <21223-3848@5.6.7.8>21223-3848@5.6.7.8
>>>>
"},145:{"event":"REGISTER","tindex":"64529","tlabel":"599481568","PhoneNumber":"20709","DeviceId":"abcd1234abcd1234","CallId":"
>>>>
<21222-3848@5.6.7.8>21222-3848@5.6.7.8"},145:{"event":"REGISTER","tindex":"46605","tlabel":"112015324","PhoneNumber":"20710","DeviceId"
>>>>
>>>>
>>>>
>>>>
>>>> T 1.2.3.4:48873 -> 5.6.7.8:3927 [AP]
>>>> :"abcd1234abcd1234","CallId":"
<21224-3848(a)5.6.7.8>
>>>> 21224-3848(a)5.6.7.8"}uot;},
>>>>
>>>> *The above two chunks contain 4 proper netstrings where the second
>>>> chunk contains part of the 4th netstring. In this case Evapi properly
>>>> raised 4 individual events.*
>>>>
>>>> 1.2.3.4:48873 -> 5.6.7.8:3927 [AP]
>>>>
>>>>
145:{"event":"REGISTER","tindex":"55567","tlabel":"627458699","PhoneNumber":"20711","DeviceId":"abcd1234abcd1234","CallId":"
>>>> <21225-3848@5.6.7.8>21225-3848@5.6.7.8
>>>>
"},143:{"event":"REGISTER","tindex":"28682","tlabel":"9676691","PhoneNumber":"20712","DeviceId":"abcd1234abcd1234","CallId":"
>>>> <21226-3848@5.6.7.8>21226-3848@5.6.7.8"},
>>>>
>>>> *This above chunk contains two complete netstrings but Kamailio never
>>>> raised events for these two netstrings.*
>>>>
>>>> For the events that were not raised I see proper Kamailio Logs which
>>>> is:
>>>> evapi_recv_client(): {0} [1.2.3.4:48873] - received
>>>>
[145:{"event":"REGISTER","tindex":"55567","tlabel":"627458699","PhoneNumber":"20711","DeviceId":"abcd1234abcd1234","CallId":"
>>>> <21225-3848@5.6.7.8>21225-3848@5.6.7.8
>>>>
"},143:{"event":"REGISTER","tindex":"28682","tlabel":"9676691","PhoneNumber":"20712","DeviceId":"abcd1234abcd1234","CallId":"
>>>> <21226-3848@5.6.7.8>21226-3848@5.6.7.8"},]
>>>>
>>>> But I don't see the logs that I've written in the script after
the
>>>> event was raised, which means Kamailio did not invoke events for these
two
>>>> netstrings.
>>>>
>>>> So in my tests, out of 27232 messages sent, there were approximately
>>>> 27 messages for which the events were not raised by Evapi. The rate of
>>>> messages started at 500cps and I stopped after I saw missed events at
>>>> around 1200cps.
>>>>
>>>> Thanks, and do let me know for any further tests or information
>>>> required about the same.
>>>>
>>>> - Jayesh
>>>>
>>>>
>>>>
>>>> On Fri, Sep 18, 2015 at 2:55 PM, Jayesh Nambiar <
>>>> <jayesh1017@gmail.com>jayesh1017@gmail.com> wrote:
>>>>
>>>>> Will have this tested by tomorrow and will get back to you. Thanks.
>>>>>
>>>>> - Jayesh
>>>>>
>>>>> On Fri, Sep 18, 2015 at 1:41 PM, Daniel-Constantin Mierla <
>>>>> <miconda@gmail.com>miconda@gmail.com> wrote:
>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> I just pushed a patch to master branch that should cope with
>>>>>> partial data received on tcp connection. No time to test at all,
therefore
>>>>>> any feedback will be appreciated.
>>>>>>
>>>>>> Cheers,
>>>>>> Daniel
>>>>>>
>>>>>>
>>>>>> On 15/09/15 14:52, Daniel-Constantin Mierla wrote:
>>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> I will look if there are options in libev to buffer data or try
to
>>>>>> implement a buffering mechanism locally for such cases.
>>>>>>
>>>>>> Cheers,
>>>>>> Daniel
>>>>>>
>>>>>> On 14/09/15 23:00, Jayesh Nambiar wrote:
>>>>>>
>>>>>> Hello Daniel,
>>>>>> After further testing with evapi module, I figured that when
>>>>>> Netstrings are used, an event route is invoked individually for
each
>>>>>> message even if if multiple netstring messages are received in a
single TCP
>>>>>> packet. But this doesn't work effectively when a single
proper message is
>>>>>> split-up in two packets. For Example, if a message arrives as:
>>>>>> 12:Hello World!, 12:Hello World!, 12:Hello World! in a single
>>>>>> packet, kamailio properly invokes the event route
"evapi:message-received"
>>>>>> thrice for every individual proper netstring message.
>>>>>> But if the first packet contains:
>>>>>> 12:Hello World!, 12:Hello
>>>>>> And Second Packet contains:
>>>>>> World!, 12:Hello World!
>>>>>> the event route is invoked only once !!
>>>>>>
>>>>>> The above pattern is very much possible while sending and
receiving
>>>>>> packets over TCP Socket. Our tests for receiving an approximately
150 byte
>>>>>> message over evapi socket at the rate of roughly 1000cps causes a
lot of
>>>>>> real events to be missed because of the above problem. You can
never be
>>>>>> sure when TCP will split messages in different chunks.
>>>>>> This definitely looks like a bug which makes it not very
reliable
>>>>>> at large scale deployments. Would really appreciate your inputs
on this.
>>>>>> Thanks;
>>>>>>
>>>>>> - Jayesh
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Thu, Sep 10, 2015 at 4:01 PM, Jayesh Nambiar <
>>>>>> <jayesh1017@gmail.com>jayesh1017@gmail.com> wrote:
>>>>>>
>>>>>>> Hi Daniel,
>>>>>>> Thanks for the quick response. So if I do not use Netstrings,
does
>>>>>>> Kamailio allow me to create a custom logic in the script. For
eg. if I
>>>>>>> decide to use newline as a delimiter, can I keep buffering
the message
>>>>>>> until I encounter the delimiter from the event route and then
execute
>>>>>>> whatever I have to within the script??
>>>>>>>
>>>>>>> Thanks,
>>>>>>>
>>>>>>> - Jayesh
>>>>>>>
>>>>>>> On Thu, Sep 10, 2015 at 1:29 PM, Daniel-Constantin Mierla
<
>>>>>>> <miconda@gmail.com>miconda@gmail.com> wrote:
>>>>>>>
>>>>>>>> Hello,
>>>>>>>>
>>>>>>>> tcb is stream protocol and several messages can be queued
on the
>>>>>>>> pipe at the same time. That is the reason for netstring
format, to be able
>>>>>>>> to easily detect the boundaries of each message. If
netstring format is
>>>>>>>> enabled and kamailio receives several messages at once,
it splits them and
>>>>>>>> for each is executing the event route.
>>>>>>>>
>>>>>>>> If netstring format is not used, the kamailio is
executing the
>>>>>>>> event route with the entire content that was read at once
from the tcp
>>>>>>>> connection.
>>>>>>>>
>>>>>>>> Cheers,
>>>>>>>> Daniel
>>>>>>>>
>>>>>>>>
>>>>>>>> On 09/09/15 22:01, Jayesh Nambiar wrote:
>>>>>>>>
>>>>>>>> Hello,
>>>>>>>> I'm exploring the evapi module for my kamailio to
interface with
>>>>>>>> an external node.js app for third party stuff like AAA,
billing engine
>>>>>>>> tasks, notifications and so on. I followed and took some
ideas from the
>>>>>>>> rtjson and evapi tutorial found here(
>>>>>>>>
<http://kb.asipto.com/kamailio:k43-async-sip-routing-nodejs>
>>>>>>>>
http://kb.asipto.com/kamailio:k43-async-sip-routing-nodejs) to
>>>>>>>> build the node.js app consuming events.
>>>>>>>> When I stress tested the scenario using SIPp and tried
sending a
>>>>>>>> lot of events at 300-350cps from Kamailio, I noticed that
at times the
>>>>>>>> client is receiving 2-3 events in a single message
together although I do
>>>>>>>> event_sync_relay once per SIP message received and have
netstrings enabled.
>>>>>>>> I believe this is a typical behavior of TCP and needs to
be handled by the
>>>>>>>> client using some kind of Netstring handler. Please
correct me if I'm wrong.
>>>>>>>> And hence I'd like to know what particularly needs to
be taken
>>>>>>>> care of while writing a client that is listening for
events on raw tcp
>>>>>>>> socket and how does kamailio handle this situation while
receiving messages
>>>>>>>> over TCP socket?? Does kamailio recognize the end of
netstring properly on
>>>>>>>> evapi:message-received and give exactly one message to
take care of on
>>>>>>>> every "message-received" event or should that
be handled in the script
>>>>>>>> somewhere !!
>>>>>>>> I also referred cgrates client over evapi example which
is
>>>>>>>> written in GO, but I couldnt find them handling TCP
streams clearly either.
>>>>>>>> I'd really appreciate some expert suggestion here to
make an
>>>>>>>> informed decision on using the evapi module for a large
scale solution.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>>
>>>>>>>> - Jayesh
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> SIP Express Router (SER) and Kamailio (OpenSER) -
sr-users mailing
listsr-users@lists.sip-router.orghttp://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Daniel-Constantin Mierlahttp://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda
>>>>>>>> Book: SIP Routing With Kamailio - http://www.asipto.com
>>>>>>>> Kamailio Advanced Training, Sep 28-30, 2015, in Berlin -
http://asipto.com/u/kat
>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> SIP Express Router (SER) and Kamailio (OpenSER) -
sr-users
>>>>>>>> mailing list
>>>>>>>>
<sr-users@lists.sip-router.org>sr-users@lists.sip-router.org
>>>>>>>>
<http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users>
>>>>>>>>
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> Daniel-Constantin Mierlahttp://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda
>>>>>> Book: SIP Routing With Kamailio - http://www.asipto.com
>>>>>> Kamailio Advanced Training, Sep 28-30, 2015, in Berlin -
http://asipto.com/u/kat
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Daniel-Constantin Mierlahttp://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda
>>>>>> Book: SIP Routing With Kamailio - http://www.asipto.com
>>>>>> Kamailio Advanced Training, Sep 28-30, 2015, in Berlin -
http://asipto.com/u/kat
>>>>>>
>>>>>>
>>>>>
>>>>
>>>> --
>>>> Daniel-Constantin Mierlahttp://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda
>>>> Book: SIP Routing With Kamailio - http://www.asipto.com
>>>> Kamailio Advanced Training, Sep 28-30, 2015, in Berlin -
http://asipto.com/u/kat
>>>>
>>>>
>>> --
>>> Daniel-Constantin Mierlahttp://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda
>>> Book: SIP Routing With Kamailio - http://www.asipto.com
>>> Kamailio Advanced Training, Sep 28-30, 2015, in Berlin -
http://asipto.com/u/kat
>>>
>>>
>> --
>> Daniel-Constantin Mierlahttp://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda
>> Book: SIP Routing With Kamailio - http://www.asipto.com
>> Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
>>
>>
--
Daniel-Constantin Mierlahttp://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda
Book: SIP Routing With Kamailio - http://www.asipto.com
Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
29 Sep
29 Sep
2:46 a.m.
Hello,
can you send here the log messages printed by evapi that contain "] -
received [" which have:
- the previous bunch of json documents that were processed ok (just
before the one that was discarded)
- the bunch of json documents that were discarded
- the next bunch of json documents that were processed ok
The logs were sent on could of emails and pastebin, so it is hard to
track them -- if you have the logs, just locate the log messages with
the discarded json and send it here along with the previous and next logs.
Cheers,
Daniel
On 28/09/15 17:04, Jayesh Nambiar wrote:
Thank you daniel.
Yes, they all get printed by the evapi module even when they get
discarded. If you once again check the paste-bin, the logs pasted as
Kamailio Logs are all discarded packets but show up as received by
evapi module.
I believe they got discarded because a packet that came before them
was partial and evapi discarded the remaining part of the message in
the new packet and encountered error.
- Jayesh
On Mon, Sep 28, 2015 at 11:13 AM Daniel-Constantin Mierla
<miconda(a)gmail.com <mailto:miconda@gmail.com>> wrote:
Hello,
I will look over it during next days.
One more thing, are the packages sent by you printed by evapi
module, even not they are not processed (but discarded)?
Cheers,
Daniel
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Book: SIP Routing With Kamailio - http://www.asipto.com
7:13 a.m.
Hi Daniel:
I did a new test where I stopped as soon as I saw one message that got
discarded. Basically the following message got discarded:
{"event":"REGISTER","tindex":"52171","tlabel":"748030241","PhoneNumber":"11161","DeviceId":"abcd1234abcd1234","CallId":"
10943-15480(a)198.24.63.39"}
The CallId parameter in the above json is auto incremented, so I tried
gathering logs for messages with following CallIds:
"CallId":"10940-15480@198.24.63.39"
"CallId":"10941-15480@198.24.63.39"
"CallId":"10942-15480@198.24.63.39"
"CallId":"10943-15480@198.24.63.39"
"CallId":"10944-15480@198.24.63.39"
"CallId":"10945-15480@198.24.63.39"
And the message with CallID "CallId":"10943-15480@198.24.63.39" got
discarded.
Here's the bunch of kamailio logs printed by the evapi module:
*Logs for processed messages just before the discarded message:*
DEBUG: evapi [evapi_dispatch.c:492]: evapi_recv_notify(): received
[0x7f43da494ad0]
[146:{"event":"REGISTER","tindex":"61133","tlabel":"450961398","PhoneNumber":"11158","DeviceId":"abcd1234abcd1234","CallId":"
10940-15480(a)198.24.63.39"}uot;},] (151)
NOTICE: evapi_recv_client(): {0} [198.24.63.45:48905] - received
[147:{"event":"REGISTER","tindex":"25291","tlabel":"2047024302","PhoneNumber":"11132","DeviceId":"abcd1234abcd1234","CallId":"
10914-15480(a)198.24.63.39
"},146:{"event":"REGISTER","tindex":"61136","tlabel":"839735223","PhoneNumber":"11148","DeviceId":"abcd1234abcd1234","CallId":"
10930-15480(a)198.24.63.39
"},143:{"event":"REGISTER","tindex":"4562","tlabel":"6586867","PhoneNumber":"11149","DeviceId":"abcd1234abcd1234","CallId":"
10931-15480(a)198.24.63.39
"},146:{"event":"REGISTER","tindex":"43213","tlabel":"220429340","PhoneNumber":"11151","DeviceId":"abcd1234abcd1234","CallId":"
10932-15480(a)198.24.63.39
"},146:{"event":"REGISTER","tindex":"52174","tlabel":"708781916","PhoneNumber":"11150","DeviceId":"abcd1234abcd1234","CallId":"
10933-15480(a)198.24.63.39
"},147:{"event":"REGISTER","tindex":"34251","tlabel":"1354642017","PhoneNumber":"11153","DeviceId":"abcd1234abcd1234","CallId":"
10935-15480(a)198.24.63.39
"},146:{"event":"REGISTER","tindex":"7366","tlabel":"1368396808","PhoneNumber":"11154","DeviceId":"abcd1234abcd1234","CallId":"
10936-15480(a)198.24.63.39
"},146:{"event":"REGISTER","tindex":"25289","tlabel":"119788881","PhoneNumber":"11152","DeviceId":"abcd1234abcd1234","CallId":"
10934-15480(a)198.24.63.39
"},146:{"event":"REGISTER","tindex":"16327","tlabel":"971937029","PhoneNumber":"11155","DeviceId":"abcd1234abcd1234","CallId":"
10937-15480(a)198.24.63.39
"},146:{"event":"REGISTER","tindex":"22242","tlabel":"609737074","PhoneNumber":"11156","DeviceId":"abcd1234abcd1234","CallId":"
10938-15480(a)198.24.63.39
"},146:{"event":"REGISTER","tindex":"10720","tlabel":"930036340","PhoneNumber":"11157","DeviceId":"abcd1234abcd1234","CallId":"
10939-15480(a)198.24.63.39
"},146:{"event":"REGISTER","tindex":"61133","tlabel":"450961398","PhoneNumber":"11158","DeviceId":"abcd1234abcd1234","CallId":"
10940-15480@198.24.63.39"},146:{"event":"REGISTER","tindex":"43210","tlabel":"639090568","PhoneNumber":"11160","Dev]
(1899) (0)
evapi_recv_client(): executing event route for frame:
[{"event":"REGISTER","tindex":"61133","tlabel":"450961398","PhoneNumber":"11158","DeviceId":"abcd1234abcd1234","CallId":"
10940-15480(a)198.24.63.39"}] (146)
Sep 29 05:35:14 v39 /usr/local/kamailio-dev/sbin/kamailio[15387]: DEBUG:
evapi [evapi_dispatch.c:492]: evapi_recv_notify(): received
[0x7f43da4b67e0]
[146:{"event":"REGISTER","tindex":"4559","tlabel":"1701553747","PhoneNumber":"11159","DeviceId":"abcd1234abcd1234","CallId":"
10941-15480(a)198.24.63.39"}uot;},] (151)
NOTICE: evapi [evapi_dispatch.c:290]: evapi_recv_client(): {0} [
198.24.63.45:48905] - received
[iceId":"abcd1234abcd1234","CallId":"
10942-15480(a)198.24.63.39
"},146:{"event":"REGISTER","tindex":"4559","tlabel":"1701553747","PhoneNumber":"11159","DeviceId":"abcd1234abcd1234","CallId":"
10941-15480(a)198.24.63.39"}uot;},] (214) (88)
DEBUG: evapi [evapi_dispatch.c:370]: evapi_recv_client(): executing event
route for frame:
[{"event":"REGISTER","tindex":"4559","tlabel":"1701553747","PhoneNumber":"11159","DeviceId":"abcd1234abcd1234","CallId":"
10941-15480(a)198.24.63.39"}] (146)
DEBUG: evapi [evapi_dispatch.c:492]: evapi_recv_notify(): received
[0x7f43d93c2a70]
[146:{"event":"REGISTER","tindex":"43210","tlabel":"639090568","PhoneNumber":"11160","DeviceId":"abcd1234abcd1234","CallId":"
10942-15480(a)198.24.63.39"}uot;},] (151)
DEBUG: evapi [evapi_dispatch.c:370]: evapi_recv_client(): executing event
route for frame:
[{"event":"REGISTER","tindex":"43210","tlabel":"639090568","PhoneNumber":"11160","DeviceId":"abcd1234abcd1234","CallId":"
10942-15480(a)198.24.63.39"}] (146)
DEBUG: evapi [evapi_dispatch.c:353]: evapi_recv_client(): residual data
[146:{"event":"REGISTER","tindex":"43210","tlabel":"639090568","PhoneNumber":"11160","Dev]
(88)
DEBUG: evapi [evapi_dispatch.c:361]: evapi_recv_client(): frame size
mismatch the ending char ("):
[{"event":"REGISTER","tindex":"43210","tlabel":"639090568","PhoneNumber":"11160","Dev146:{"event":"REGISTER","tindex":"52171","tlabel":"748030241",]
(146)
*Logs for discarded message:*
DEBUG: evapi [evapi_dispatch.c:492]: evapi_recv_notify(): received
[0x7f43da3e69a0]
[146:{"event":"REGISTER","tindex":"52171","tlabel":"748030241","PhoneNumber":"11161","DeviceId":"abcd1234abcd1234","CallId":"
10943-15480(a)198.24.63.39"}uot;},] (151)
NOTICE: evapi [evapi_dispatch.c:290]: evapi_recv_client(): {0} [
198.24.63.45:48905] - received
[146:{"event":"REGISTER","tindex":"52171","tlabel":"748030241","PhoneNumber":"11161","DeviceId":"abcd1234abcd1234","CallId":"
10943-15480(a)198.24.63.39"}uot;},] (151) (88)
*Logs for messages after the discarded message:*
DEBUG: evapi [evapi_dispatch.c:492]: evapi_recv_notify(): received
[0x7f43da3e6aa0]
[146:{"event":"REGISTER","tindex":"25286","tlabel":"809034258","PhoneNumber":"11163","DeviceId":"abcd1234abcd1234","CallId":"
10944-15480(a)198.24.63.39"}uot;},] (151)
NOTICE: evapi [evapi_dispatch.c:290]: evapi_recv_client(): {0} [
198.24.63.45:48905] - received
[146:{"event":"REGISTER","tindex":"25286","tlabel":"809034258","PhoneNumber":"11163","DeviceId":"abcd1234abcd1234","CallId":"
10944-15480(a)198.24.63.39"}uot;},] (151) (0)
DEBUG: evapi [evapi_dispatch.c:370]: evapi_recv_client(): executing event
route for frame:
[{"event":"REGISTER","tindex":"25286","tlabel":"809034258","PhoneNumber":"11163","DeviceId":"abcd1234abcd1234","CallId":"
10944-15480(a)198.24.63.39"}] (146)
DEBUG: evapi [evapi_dispatch.c:492]: evapi_recv_notify(): received
[0x7f43d9598778]
[146:{"event":"REGISTER","tindex":"34248","tlabel":"187029249","PhoneNumber":"11162","DeviceId":"abcd1234abcd1234","CallId":"
10945-15480(a)198.24.63.39"}uot;},] (151)
NOTICE: evapi [evapi_dispatch.c:290]: evapi_recv_client(): {0} [
198.24.63.45:48905] - received
[146:{"event":"REGISTER","tindex":"34248","tlabel":"187029249","PhoneNumber":"11162","DeviceId":"abcd1234abcd1234","CallId":"
10945-15480(a)198.24.63.39"}uot;},] (151) (0)
DEBUG: evapi [evapi_dispatch.c:370]: evapi_recv_client(): executing event
route for frame:
[{"event":"REGISTER","tindex":"34248","tlabel":"187029249","PhoneNumber":"11162","DeviceId":"abcd1234abcd1234","CallId":"
10945-15480(a)198.24.63.39"}] (146)
DEBUG: evapi [evapi_dispatch.c:492]: evapi_recv_notify(): received
[0x7f43d9598878]
[145:{"event":"REGISTER","tindex":"7363","tlabel":"949153805","PhoneNumber":"11164","DeviceId":"abcd1234abcd1234","CallId":"
10946-15480(a)198.24.63.39"}uot;},] (150)
NOTICE: evapi [evapi_dispatch.c:290]: evapi_recv_client(): {0} [
198.24.63.45:48905] - received
[145:{"event":"REGISTER","tindex":"7363","tlabel":"949153805","PhoneNumber":"11164","DeviceId":"abcd1234abcd1234","CallId":"
10946-15480(a)198.24.63.39"}uot;},] (150) (0)
DEBUG: evapi [evapi_dispatch.c:370]: evapi_recv_client(): executing event
route for frame:
[{"event":"REGISTER","tindex":"7363","tlabel":"949153805","PhoneNumber":"11164","DeviceId":"abcd1234abcd1234","CallId":"
10946-15480(a)198.24.63.39"}] (145)
Just in case if you need to see the NGREP trace for the above messages, it
is here:
http://pastebin.com/4zTKmBJX
Thanks,
- Jayesh
On Tue, Sep 29, 2015 at 12:16 PM Daniel-Constantin Mierla <miconda(a)gmail.com>
wrote:
Hello,
can you send here the log messages printed by evapi that contain "] -
received [" which have:
- the previous bunch of json documents that were processed ok (just before
the one that was discarded)
- the bunch of json documents that were discarded
- the next bunch of json documents that were processed ok
The logs were sent on could of emails and pastebin, so it is hard to track
them -- if you have the logs, just locate the log messages with the
discarded json and send it here along with the previous and next logs.
Cheers,
Daniel
On 28/09/15 17:04, Jayesh Nambiar wrote:
Thank you daniel.
Yes, they all get printed by the evapi module even when they get
discarded. If you once again check the paste-bin, the logs pasted as
Kamailio Logs are all discarded packets but show up as received by evapi
module.
I believe they got discarded because a packet that came before them was
partial and evapi discarded the remaining part of the message in the new
packet and encountered error.
- Jayesh
On Mon, Sep 28, 2015 at 11:13 AM Daniel-Constantin Mierla <
miconda(a)gmail.com> wrote:
Hello,
I will look over it during next days.
One more thing, are the packages sent by you printed by evapi module,
even not they are not processed (but discarded)?
Cheers,
Daniel
--
Daniel-Constantin Mierlahttp://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda
Book: SIP Routing With Kamailio - http://www.asipto.com
8:15 a.m.
Hi Daniel,
Taking a second look at the NGREP Trace I think I can see what is going
wrong. The message in the third packet was discarded by Evapi module. If
you look at the first two packets carefully, the first packet ends at Dev
and the second packet is the continuation of first packet which starts with
"iceId". The first two packets are processed perfectly fine.
Now the third packet is a new message, but Kamailio is trying to continue
the third packet again with the first packet. So even when the third packet
starts with '146:' Kamailio considers it to be starting with 'Dev146'.
This
causes the evapi to discard this message since it does not remain a valid
netstring then. Here the three packets that I'm talking about:
T 198.24.63.45:48905 -> 198.24.63.39:3927 [A]
146:{"event":"REGISTER","tindex":"43213","tlabel":"220429340","PhoneNumber":"11151","DeviceId":"abcd1234abcd1234","CallId":"10932-15480@198.24.63.39"},146:{"event":"REGISTER","tindex":"52174","tlabel":"708781916","PhoneNumber":"11150","DeviceId":"abcd1234abcd1234","CallId":"10933-15480@198.24.63.39"},147:{"event":"REGISTER","tindex":"34251","tlabel":"1354642017","PhoneNumber":"11153","DeviceId":"abcd1234abcd1234","CallId":"10935-15480@198.24.63.39"},146:{"event":"REGISTER","tindex":"7366","tlabel":"1368396808","PhoneNumber":"11154","DeviceId":"abcd1234abcd1234","CallId":"10936-15480@198.24.63.39"},146:{"event":"REGISTER","tindex":"25289","tlabel":"119788881","PhoneNumber":"11152","DeviceId":"abcd1234abcd1234","CallId":"10934-15480@198.24.63.39"},146:{"event":"REGISTER","tindex":"16327","tlabel":"971937029","PhoneNumber":"11155","DeviceId":"abcd1234abcd1234","CallId":"10937-15480@198.24.63.39"},146:{"event":"REGISTER","tindex":"22242","tlabel":"609737074","PhoneNumber":"11156","DeviceId":"abcd1234abcd1234","CallId":"10938-15480@198.24.63.39"},146:{"event":"REGISTER","tindex":"10720","tlabel":"930036340","PhoneNumber":"11157","DeviceId":"abcd1234abcd1234","CallId":"10939-15480@198.24.63.39"},146:{"event":"REGISTER","tindex":"61133","tlabel":"450961398","PhoneNumber":"11158","DeviceId":"abcd1234abcd1234","CallId":"10940-15480@198.24.63.39"},146:{"event":"REGISTER","tindex":"43210","tlabel":"639090568","PhoneNumber":"11160","Dev
T 198.24.63.45:48905 -> 198.24.63.39:3927 [AP]
iceId":"abcd1234abcd1234","CallId":"10942-15480@198.24.63.39"},146:{"event":"REGISTER","tindex":"4559","tlabel":"1701553747","PhoneNumber":"11159","DeviceId":"abcd1234abcd1234","CallId":"10941-15480@198.24.63.39"},
T 198.24.63.45:48905 -> 198.24.63.39:3927 [AP]
146:{"event":"REGISTER","tindex":"52171","tlabel":"748030241","PhoneNumber":"11161","DeviceId":"abcd1234abcd1234","CallId":"10943-15480@198.24.63.39"},
On Tue, Sep 29, 2015 at 4:43 PM Jayesh Nambiar <jayesh1017(a)gmail.com> wrote:
Hi Daniel:
I did a new test where I stopped as soon as I saw one message that got
discarded. Basically the following message got discarded:
{"event":"REGISTER","tindex":"52171","tlabel":"748030241","PhoneNumber":"11161","DeviceId":"abcd1234abcd1234","CallId":"
10943-15480(a)198.24.63.39"}
The CallId parameter in the above json is auto incremented, so I tried
gathering logs for messages with following CallIds:
"CallId":"10940-15480@198.24.63.39"
"CallId":"10941-15480@198.24.63.39"
"CallId":"10942-15480@198.24.63.39"
"CallId":"10943-15480@198.24.63.39"
"CallId":"10944-15480@198.24.63.39"
"CallId":"10945-15480@198.24.63.39"
And the message with CallID "CallId":"10943-15480@198.24.63.39" got
discarded.
Here's the bunch of kamailio logs printed by the evapi module:
*Logs for processed messages just before the discarded message:*
DEBUG: evapi [evapi_dispatch.c:492]: evapi_recv_notify(): received
[0x7f43da494ad0]
[146:{"event":"REGISTER","tindex":"61133","tlabel":"450961398","PhoneNumber":"11158","DeviceId":"abcd1234abcd1234","CallId":"
10940-15480(a)198.24.63.39"}uot;},] (151)
NOTICE: evapi_recv_client(): {0} [198.24.63.45:48905] - received
[147:{"event":"REGISTER","tindex":"25291","tlabel":"2047024302","PhoneNumber":"11132","DeviceId":"abcd1234abcd1234","CallId":"
10914-15480(a)198.24.63.39
"},146:{"event":"REGISTER","tindex":"61136","tlabel":"839735223","PhoneNumber":"11148","DeviceId":"abcd1234abcd1234","CallId":"
10930-15480(a)198.24.63.39
"},143:{"event":"REGISTER","tindex":"4562","tlabel":"6586867","PhoneNumber":"11149","DeviceId":"abcd1234abcd1234","CallId":"
10931-15480(a)198.24.63.39
"},146:{"event":"REGISTER","tindex":"43213","tlabel":"220429340","PhoneNumber":"11151","DeviceId":"abcd1234abcd1234","CallId":"
10932-15480(a)198.24.63.39
"},146:{"event":"REGISTER","tindex":"52174","tlabel":"708781916","PhoneNumber":"11150","DeviceId":"abcd1234abcd1234","CallId":"
10933-15480(a)198.24.63.39
"},147:{"event":"REGISTER","tindex":"34251","tlabel":"1354642017","PhoneNumber":"11153","DeviceId":"abcd1234abcd1234","CallId":"
10935-15480(a)198.24.63.39
"},146:{"event":"REGISTER","tindex":"7366","tlabel":"1368396808","PhoneNumber":"11154","DeviceId":"abcd1234abcd1234","CallId":"
10936-15480(a)198.24.63.39
"},146:{"event":"REGISTER","tindex":"25289","tlabel":"119788881","PhoneNumber":"11152","DeviceId":"abcd1234abcd1234","CallId":"
10934-15480(a)198.24.63.39
"},146:{"event":"REGISTER","tindex":"16327","tlabel":"971937029","PhoneNumber":"11155","DeviceId":"abcd1234abcd1234","CallId":"
10937-15480(a)198.24.63.39
"},146:{"event":"REGISTER","tindex":"22242","tlabel":"609737074","PhoneNumber":"11156","DeviceId":"abcd1234abcd1234","CallId":"
10938-15480(a)198.24.63.39
"},146:{"event":"REGISTER","tindex":"10720","tlabel":"930036340","PhoneNumber":"11157","DeviceId":"abcd1234abcd1234","CallId":"
10939-15480(a)198.24.63.39
"},146:{"event":"REGISTER","tindex":"61133","tlabel":"450961398","PhoneNumber":"11158","DeviceId":"abcd1234abcd1234","CallId":"
10940-15480@198.24.63.39"},146:{"event":"REGISTER","tindex":"43210","tlabel":"639090568","PhoneNumber":"11160","Dev]
(1899) (0)
evapi_recv_client(): executing event route for frame:
[{"event":"REGISTER","tindex":"61133","tlabel":"450961398","PhoneNumber":"11158","DeviceId":"abcd1234abcd1234","CallId":"
10940-15480(a)198.24.63.39"}] (146)
Sep 29 05:35:14 v39 /usr/local/kamailio-dev/sbin/kamailio[15387]: DEBUG:
evapi [evapi_dispatch.c:492]: evapi_recv_notify(): received
[0x7f43da4b67e0]
[146:{"event":"REGISTER","tindex":"4559","tlabel":"1701553747","PhoneNumber":"11159","DeviceId":"abcd1234abcd1234","CallId":"
10941-15480(a)198.24.63.39"}uot;},] (151)
NOTICE: evapi [evapi_dispatch.c:290]: evapi_recv_client(): {0} [
198.24.63.45:48905] - received
[iceId":"abcd1234abcd1234","CallId":"
10942-15480(a)198.24.63.39
"},146:{"event":"REGISTER","tindex":"4559","tlabel":"1701553747","PhoneNumber":"11159","DeviceId":"abcd1234abcd1234","CallId":"
10941-15480(a)198.24.63.39"}uot;},] (214) (88)
DEBUG: evapi [evapi_dispatch.c:370]: evapi_recv_client(): executing event
route for frame:
[{"event":"REGISTER","tindex":"4559","tlabel":"1701553747","PhoneNumber":"11159","DeviceId":"abcd1234abcd1234","CallId":"
10941-15480(a)198.24.63.39"}] (146)
DEBUG: evapi [evapi_dispatch.c:492]: evapi_recv_notify(): received
[0x7f43d93c2a70]
[146:{"event":"REGISTER","tindex":"43210","tlabel":"639090568","PhoneNumber":"11160","DeviceId":"abcd1234abcd1234","CallId":"
10942-15480(a)198.24.63.39"}uot;},] (151)
DEBUG: evapi [evapi_dispatch.c:370]: evapi_recv_client(): executing event
route for frame:
[{"event":"REGISTER","tindex":"43210","tlabel":"639090568","PhoneNumber":"11160","DeviceId":"abcd1234abcd1234","CallId":"
10942-15480(a)198.24.63.39"}] (146)
DEBUG: evapi [evapi_dispatch.c:353]: evapi_recv_client(): residual data
[146:{"event":"REGISTER","tindex":"43210","tlabel":"639090568","PhoneNumber":"11160","Dev]
(88)
DEBUG: evapi [evapi_dispatch.c:361]: evapi_recv_client(): frame size
mismatch the ending char ("):
[{"event":"REGISTER","tindex":"43210","tlabel":"639090568","PhoneNumber":"11160","Dev146:{"event":"REGISTER","tindex":"52171","tlabel":"748030241",]
(146)
*Logs for discarded message:*
DEBUG: evapi [evapi_dispatch.c:492]: evapi_recv_notify(): received
[0x7f43da3e69a0]
[146:{"event":"REGISTER","tindex":"52171","tlabel":"748030241","PhoneNumber":"11161","DeviceId":"abcd1234abcd1234","CallId":"
10943-15480(a)198.24.63.39"}uot;},] (151)
NOTICE: evapi [evapi_dispatch.c:290]: evapi_recv_client(): {0} [
198.24.63.45:48905] - received
[146:{"event":"REGISTER","tindex":"52171","tlabel":"748030241","PhoneNumber":"11161","DeviceId":"abcd1234abcd1234","CallId":"
10943-15480(a)198.24.63.39"}uot;},] (151) (88)
*Logs for messages after the discarded message:*
DEBUG: evapi [evapi_dispatch.c:492]: evapi_recv_notify(): received
[0x7f43da3e6aa0]
[146:{"event":"REGISTER","tindex":"25286","tlabel":"809034258","PhoneNumber":"11163","DeviceId":"abcd1234abcd1234","CallId":"
10944-15480(a)198.24.63.39"}uot;},] (151)
NOTICE: evapi [evapi_dispatch.c:290]: evapi_recv_client(): {0} [
198.24.63.45:48905] - received
[146:{"event":"REGISTER","tindex":"25286","tlabel":"809034258","PhoneNumber":"11163","DeviceId":"abcd1234abcd1234","CallId":"
10944-15480(a)198.24.63.39"}uot;},] (151) (0)
DEBUG: evapi [evapi_dispatch.c:370]: evapi_recv_client(): executing event
route for frame:
[{"event":"REGISTER","tindex":"25286","tlabel":"809034258","PhoneNumber":"11163","DeviceId":"abcd1234abcd1234","CallId":"
10944-15480(a)198.24.63.39"}] (146)
DEBUG: evapi [evapi_dispatch.c:492]: evapi_recv_notify(): received
[0x7f43d9598778]
[146:{"event":"REGISTER","tindex":"34248","tlabel":"187029249","PhoneNumber":"11162","DeviceId":"abcd1234abcd1234","CallId":"
10945-15480(a)198.24.63.39"}uot;},] (151)
NOTICE: evapi [evapi_dispatch.c:290]: evapi_recv_client(): {0} [
198.24.63.45:48905] - received
[146:{"event":"REGISTER","tindex":"34248","tlabel":"187029249","PhoneNumber":"11162","DeviceId":"abcd1234abcd1234","CallId":"
10945-15480(a)198.24.63.39"}uot;},] (151) (0)
DEBUG: evapi [evapi_dispatch.c:370]: evapi_recv_client(): executing event
route for frame:
[{"event":"REGISTER","tindex":"34248","tlabel":"187029249","PhoneNumber":"11162","DeviceId":"abcd1234abcd1234","CallId":"
10945-15480(a)198.24.63.39"}] (146)
DEBUG: evapi [evapi_dispatch.c:492]: evapi_recv_notify(): received
[0x7f43d9598878]
[145:{"event":"REGISTER","tindex":"7363","tlabel":"949153805","PhoneNumber":"11164","DeviceId":"abcd1234abcd1234","CallId":"
10946-15480(a)198.24.63.39"}uot;},] (150)
NOTICE: evapi [evapi_dispatch.c:290]: evapi_recv_client(): {0} [
198.24.63.45:48905] - received
[145:{"event":"REGISTER","tindex":"7363","tlabel":"949153805","PhoneNumber":"11164","DeviceId":"abcd1234abcd1234","CallId":"
10946-15480(a)198.24.63.39"}uot;},] (150) (0)
DEBUG: evapi [evapi_dispatch.c:370]: evapi_recv_client(): executing event
route for frame:
[{"event":"REGISTER","tindex":"7363","tlabel":"949153805","PhoneNumber":"11164","DeviceId":"abcd1234abcd1234","CallId":"
10946-15480(a)198.24.63.39"}] (145)
Just in case if you need to see the NGREP trace for the above messages, it
is here:
http://pastebin.com/4zTKmBJX
Thanks,
- Jayesh
On Tue, Sep 29, 2015 at 12:16 PM Daniel-Constantin Mierla <
miconda(a)gmail.com> wrote:
> Hello,
>
> can you send here the log messages printed by evapi that contain "] -
> received [" which have:
>
> - the previous bunch of json documents that were processed ok (just
> before the one that was discarded)
> - the bunch of json documents that were discarded
> - the next bunch of json documents that were processed ok
>
> The logs were sent on could of emails and pastebin, so it is hard to
> track them -- if you have the logs, just locate the log messages with the
> discarded json and send it here along with the previous and next logs.
>
> Cheers,
> Daniel
>
>
> On 28/09/15 17:04, Jayesh Nambiar wrote:
>
> Thank you daniel.
> Yes, they all get printed by the evapi module even when they get
> discarded. If you once again check the paste-bin, the logs pasted as
> Kamailio Logs are all discarded packets but show up as received by evapi
> module.
> I believe they got discarded because a packet that came before them was
> partial and evapi discarded the remaining part of the message in the new
> packet and encountered error.
> - Jayesh
>
> On Mon, Sep 28, 2015 at 11:13 AM Daniel-Constantin Mierla <
> miconda(a)gmail.com> wrote:
>
>> Hello,
>>
>> I will look over it during next days.
>>
>> One more thing, are the packages sent by you printed by evapi module,
>> even not they are not processed (but discarded)?
>>
>> Cheers,
>> Daniel
>>
>>
> --
> Daniel-Constantin Mierlahttp://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda
> Book: SIP Routing With Kamailio - http://www.asipto.com
>
>
30 Sep
30 Sep
2:31 a.m.
Hello,
thanks for troubleshooting further and pointing to the issue. I pushed a
patch last night, hopefully fixes it. Try with latest master branch and
let me know the results.
Cheers,
Daniel
On 29/09/15 14:15, Jayesh Nambiar wrote:
Hi Daniel,
Taking a second look at the NGREP Trace I think I can see what is
going wrong. The message in the third packet was discarded by Evapi
module. If you look at the first two packets carefully, the first
packet ends at Dev and the second packet is the continuation of first
packet which starts with "iceId". The first two packets are processed
perfectly fine.
Now the third packet is a new message, but Kamailio is trying to
continue the third packet again with the first packet. So even when
the third packet starts with '146:' Kamailio considers it to be
starting with 'Dev146'. This causes the evapi to discard this message
since it does not remain a valid netstring then. Here the three
packets that I'm talking about:
T 198.24.63.45:48905 <http://198.24.63.45:48905> -> 198.24.63.39:3927
<http://198.24.63.39:3927> [A]
146:{"event":"REGISTER","tindex":"43213","tlabel":"220429340","PhoneNumber":"11151","DeviceId":"abcd1234abcd1234","CallId":"10932-15480@198.24.63.39
<mailto:10932-15480@198.24.63.39>"},146:{"event":"REGISTER","tindex":"52174","tlabel":"708781916","PhoneNumber":"11150","DeviceId":"abcd1234abcd1234","CallId":"10933-15480@198.24.63.39
<mailto:10933-15480@198.24.63.39>"},147:{"event":"REGISTER","tindex":"34251","tlabel":"1354642017","PhoneNumber":"11153","DeviceId":"abcd1234abcd1234","CallId":"10935-15480@198.24.63.39
<mailto:10935-15480@198.24.63.39>"},146:{"event":"REGISTER","tindex":"7366","tlabel":"1368396808","PhoneNumber":"11154","DeviceId":"abcd1234abcd1234","CallId":"10936-15480@198.24.63.39
<mailto:10936-15480@198.24.63.39>"},146:{"event":"REGISTER","tindex":"25289","tlabel":"119788881","PhoneNumber":"11152","DeviceId":"abcd1234abcd1234","CallId":"10934-15480@198.24.63.39
<mailto:10934-15480@198.24.63.39>"},146:{"event":"REGISTER","tindex":"16327","tlabel":"971937029","PhoneNumber":"11155","DeviceId":"abcd1234abcd1234","CallId":"10937-15480@198.24.63.39
<mailto:10937-15480@198.24.63.39>"},146:{"event":"REGISTER","tindex":"22242","tlabel":"609737074","PhoneNumber":"11156","DeviceId":"abcd1234abcd1234","CallId":"10938-15480@198.24.63.39
<mailto:10938-15480@198.24.63.39>"},146:{"event":"REGISTER","tindex":"10720","tlabel":"930036340","PhoneNumber":"11157","DeviceId":"abcd1234abcd1234","CallId":"10939-15480@198.24.63.39
<mailto:10939-15480@198.24.63.39>"},146:{"event":"REGISTER","tindex":"61133","tlabel":"450961398","PhoneNumber":"11158","DeviceId":"abcd1234abcd1234","CallId":"10940-15480@198.24.63.39
<mailto:10940-15480@198.24.63.39>"},146:{"event":"REGISTER","tindex":"43210","tlabel":"639090568","PhoneNumber":"11160","Dev
T 198.24.63.45:48905 <http://198.24.63.45:48905> -> 198.24.63.39:3927
<http://198.24.63.39:3927> [AP]
iceId":"abcd1234abcd1234","CallId":"10942-15480@198.24.63.39
<mailto:10942-15480@198.24.63.39>"},146:{"event":"REGISTER","tindex":"4559","tlabel":"1701553747","PhoneNumber":"11159","DeviceId":"abcd1234abcd1234","CallId":"10941-15480@198.24.63.39
<mailto:10941-15480@198.24.63.39>"},
T 198.24.63.45:48905 <http://198.24.63.45:48905> -> 198.24.63.39:3927
<http://198.24.63.39:3927> [AP]
146:{"event":"REGISTER","tindex":"52171","tlabel":"748030241","PhoneNumber":"11161","DeviceId":"abcd1234abcd1234","CallId":"10943-15480@198.24.63.39
<mailto:10943-15480@198.24.63.39>"},
On Tue, Sep 29, 2015 at 4:43 PM Jayesh Nambiar <jayesh1017(a)gmail.com
<mailto:jayesh1017@gmail.com>> wrote:
Hi Daniel:
I did a new test where I stopped as soon as I saw one message that
got discarded. Basically the following message got discarded:
{"event":"REGISTER","tindex":"52171","tlabel":"748030241","PhoneNumber":"11161","DeviceId":"abcd1234abcd1234","CallId":"10943-15480@198.24.63.39
<mailto:10943-15480@198.24.63.39>"}
The CallId parameter in the above json is auto incremented, so I
tried gathering logs for messages with following CallIds:
"CallId":"10940-15480@198.24.63.39
<mailto:10940-15480@198.24.63.39>"
"CallId":"10941-15480@198.24.63.39
<mailto:10941-15480@198.24.63.39>"
"CallId":"10942-15480@198.24.63.39
<mailto:10942-15480@198.24.63.39>"
"CallId":"10943-15480@198.24.63.39
<mailto:10943-15480@198.24.63.39>"
"CallId":"10944-15480@198.24.63.39
<mailto:10944-15480@198.24.63.39>"
"CallId":"10945-15480@198.24.63.39
<mailto:10945-15480@198.24.63.39>"
And the message with CallID "CallId":"10943-15480@198.24.63.39
<http://198.24.63.39>" got discarded.
Here's the bunch of kamailio logs printed by the evapi module:
*Logs for processed messages just before the discarded message:*
DEBUG: evapi [evapi_dispatch.c:492]: evapi_recv_notify(): received
[0x7f43da494ad0]
[146:{"event":"REGISTER","tindex":"61133","tlabel":"450961398","PhoneNumber":"11158","DeviceId":"abcd1234abcd1234","CallId":"10940-15480@198.24.63.39
<mailto:10940-15480@198.24.63.39>"},] (151)
NOTICE: evapi_recv_client(): {0} [198.24.63.45:48905
<http://198.24.63.45:48905>] - received
[147:{"event":"REGISTER","tindex":"25291","tlabel":"2047024302","PhoneNumber":"11132","DeviceId":"abcd1234abcd1234","CallId":"10914-15480@198.24.63.39
<mailto:10914-15480@198.24.63.39>"},146:{"event":"REGISTER","tindex":"61136","tlabel":"839735223","PhoneNumber":"11148","DeviceId":"abcd1234abcd1234","CallId":"10930-15480@198.24.63.39
<mailto:10930-15480@198.24.63.39>"},143:{"event":"REGISTER","tindex":"4562","tlabel":"6586867","PhoneNumber":"11149","DeviceId":"abcd1234abcd1234","CallId":"10931-15480@198.24.63.39
<mailto:10931-15480@198.24.63.39>"},146:{"event":"REGISTER","tindex":"43213","tlabel":"220429340","PhoneNumber":"11151","DeviceId":"abcd1234abcd1234","CallId":"10932-15480@198.24.63.39
<mailto:10932-15480@198.24.63.39>"},146:{"event":"REGISTER","tindex":"52174","tlabel":"708781916","PhoneNumber":"11150","DeviceId":"abcd1234abcd1234","CallId":"10933-15480@198.24.63.39
<mailto:10933-15480@198.24.63.39>"},147:{"event":"REGISTER","tindex":"34251","tlabel":"1354642017","PhoneNumber":"11153","DeviceId":"abcd1234abcd1234","CallId":"10935-15480@198.24.63.39
<mailto:10935-15480@198.24.63.39>"},146:{"event":"REGISTER","tindex":"7366","tlabel":"1368396808","PhoneNumber":"11154","DeviceId":"abcd1234abcd1234","CallId":"10936-15480@198.24.63.39
<mailto:10936-15480@198.24.63.39>"},146:{"event":"REGISTER","tindex":"25289","tlabel":"119788881","PhoneNumber":"11152","DeviceId":"abcd1234abcd1234","CallId":"10934-15480@198.24.63.39
<mailto:10934-15480@198.24.63.39>"},146:{"event":"REGISTER","tindex":"16327","tlabel":"971937029","PhoneNumber":"11155","DeviceId":"abcd1234abcd1234","CallId":"10937-15480@198.24.63.39
<mailto:10937-15480@198.24.63.39>"},146:{"event":"REGISTER","tindex":"22242","tlabel":"609737074","PhoneNumber":"11156","DeviceId":"abcd1234abcd1234","CallId":"10938-15480@198.24.63.39
<mailto:10938-15480@198.24.63.39>"},146:{"event":"REGISTER","tindex":"10720","tlabel":"930036340","PhoneNumber":"11157","DeviceId":"abcd1234abcd1234","CallId":"10939-15480@198.24.63.39
<mailto:10939-15480@198.24.63.39>"},146:{"event":"REGISTER","tindex":"61133","tlabel":"450961398","PhoneNumber":"11158","DeviceId":"abcd1234abcd1234","CallId":"10940-15480@198.24.63.39
<mailto:10940-15480@198.24.63.39>"},146:{"event":"REGISTER","tindex":"43210","tlabel":"639090568","PhoneNumber":"11160","Dev]
(1899) (0)
evapi_recv_client(): executing event route for frame:
[{"event":"REGISTER","tindex":"61133","tlabel":"450961398","PhoneNumber":"11158","DeviceId":"abcd1234abcd1234","CallId":"10940-15480@198.24.63.39
<mailto:10940-15480@198.24.63.39>"}] (146)
Sep 29 05:35:14 v39 /usr/local/kamailio-dev/sbin/kamailio[15387]:
DEBUG: evapi [evapi_dispatch.c:492]: evapi_recv_notify(): received
[0x7f43da4b67e0]
[146:{"event":"REGISTER","tindex":"4559","tlabel":"1701553747","PhoneNumber":"11159","DeviceId":"abcd1234abcd1234","CallId":"10941-15480@198.24.63.39
<mailto:10941-15480@198.24.63.39>"},] (151)
NOTICE: evapi [evapi_dispatch.c:290]: evapi_recv_client(): {0}
[198.24.63.45:48905 <http://198.24.63.45:48905>] - received
[iceId":"abcd1234abcd1234","CallId":"10942-15480@198.24.63.39
<mailto:10942-15480@198.24.63.39>"},146:{"event":"REGISTER","tindex":"4559","tlabel":"1701553747","PhoneNumber":"11159","DeviceId":"abcd1234abcd1234","CallId":"10941-15480@198.24.63.39
<mailto:10941-15480@198.24.63.39>"},] (214) (88)
DEBUG: evapi [evapi_dispatch.c:370]: evapi_recv_client():
executing event route for frame:
[{"event":"REGISTER","tindex":"4559","tlabel":"1701553747","PhoneNumber":"11159","DeviceId":"abcd1234abcd1234","CallId":"10941-15480@198.24.63.39
<mailto:10941-15480@198.24.63.39>"}] (146)
DEBUG: evapi [evapi_dispatch.c:492]: evapi_recv_notify(): received
[0x7f43d93c2a70]
[146:{"event":"REGISTER","tindex":"43210","tlabel":"639090568","PhoneNumber":"11160","DeviceId":"abcd1234abcd1234","CallId":"10942-15480@198.24.63.39
<mailto:10942-15480@198.24.63.39>"},] (151)
DEBUG: evapi [evapi_dispatch.c:370]: evapi_recv_client():
executing event route for frame:
[{"event":"REGISTER","tindex":"43210","tlabel":"639090568","PhoneNumber":"11160","DeviceId":"abcd1234abcd1234","CallId":"10942-15480@198.24.63.39
<mailto:10942-15480@198.24.63.39>"}] (146)
DEBUG: evapi [evapi_dispatch.c:353]: evapi_recv_client(): residual
data
[146:{"event":"REGISTER","tindex":"43210","tlabel":"639090568","PhoneNumber":"11160","Dev]
(88)
DEBUG: evapi [evapi_dispatch.c:361]: evapi_recv_client(): frame
size mismatch the ending char ("):
[{"event":"REGISTER","tindex":"43210","tlabel":"639090568","PhoneNumber":"11160","Dev146:{"event":"REGISTER","tindex":"52171","tlabel":"748030241",]
(146)
*Logs for discarded message:*
DEBUG: evapi [evapi_dispatch.c:492]: evapi_recv_notify(): received
[0x7f43da3e69a0]
[146:{"event":"REGISTER","tindex":"52171","tlabel":"748030241","PhoneNumber":"11161","DeviceId":"abcd1234abcd1234","CallId":"10943-15480@198.24.63.39
<mailto:10943-15480@198.24.63.39>"},] (151)
NOTICE: evapi [evapi_dispatch.c:290]: evapi_recv_client(): {0}
[198.24.63.45:48905 <http://198.24.63.45:48905>] - received
[146:{"event":"REGISTER","tindex":"52171","tlabel":"748030241","PhoneNumber":"11161","DeviceId":"abcd1234abcd1234","CallId":"10943-15480@198.24.63.39
<mailto:10943-15480@198.24.63.39>"},] (151) (88)
*Logs for messages after the discarded message:*
DEBUG: evapi [evapi_dispatch.c:492]: evapi_recv_notify(): received
[0x7f43da3e6aa0]
[146:{"event":"REGISTER","tindex":"25286","tlabel":"809034258","PhoneNumber":"11163","DeviceId":"abcd1234abcd1234","CallId":"10944-15480@198.24.63.39
<mailto:10944-15480@198.24.63.39>"},] (151)
NOTICE: evapi [evapi_dispatch.c:290]: evapi_recv_client(): {0}
[198.24.63.45:48905 <http://198.24.63.45:48905>] - received
[146:{"event":"REGISTER","tindex":"25286","tlabel":"809034258","PhoneNumber":"11163","DeviceId":"abcd1234abcd1234","CallId":"10944-15480@198.24.63.39
<mailto:10944-15480@198.24.63.39>"},] (151) (0)
DEBUG: evapi [evapi_dispatch.c:370]: evapi_recv_client():
executing event route for frame:
[{"event":"REGISTER","tindex":"25286","tlabel":"809034258","PhoneNumber":"11163","DeviceId":"abcd1234abcd1234","CallId":"10944-15480@198.24.63.39
<mailto:10944-15480@198.24.63.39>"}] (146)
DEBUG: evapi [evapi_dispatch.c:492]: evapi_recv_notify(): received
[0x7f43d9598778]
[146:{"event":"REGISTER","tindex":"34248","tlabel":"187029249","PhoneNumber":"11162","DeviceId":"abcd1234abcd1234","CallId":"10945-15480@198.24.63.39
<mailto:10945-15480@198.24.63.39>"},] (151)
NOTICE: evapi [evapi_dispatch.c:290]: evapi_recv_client(): {0}
[198.24.63.45:48905 <http://198.24.63.45:48905>] - received
[146:{"event":"REGISTER","tindex":"34248","tlabel":"187029249","PhoneNumber":"11162","DeviceId":"abcd1234abcd1234","CallId":"10945-15480@198.24.63.39
<mailto:10945-15480@198.24.63.39>"},] (151) (0)
DEBUG: evapi [evapi_dispatch.c:370]: evapi_recv_client():
executing event route for frame:
[{"event":"REGISTER","tindex":"34248","tlabel":"187029249","PhoneNumber":"11162","DeviceId":"abcd1234abcd1234","CallId":"10945-15480@198.24.63.39
<mailto:10945-15480@198.24.63.39>"}] (146)
DEBUG: evapi [evapi_dispatch.c:492]: evapi_recv_notify(): received
[0x7f43d9598878]
[145:{"event":"REGISTER","tindex":"7363","tlabel":"949153805","PhoneNumber":"11164","DeviceId":"abcd1234abcd1234","CallId":"10946-15480@198.24.63.39
<mailto:10946-15480@198.24.63.39>"},] (150)
NOTICE: evapi [evapi_dispatch.c:290]: evapi_recv_client(): {0}
[198.24.63.45:48905 <http://198.24.63.45:48905>] - received
[145:{"event":"REGISTER","tindex":"7363","tlabel":"949153805","PhoneNumber":"11164","DeviceId":"abcd1234abcd1234","CallId":"10946-15480@198.24.63.39
<mailto:10946-15480@198.24.63.39>"},] (150) (0)
DEBUG: evapi [evapi_dispatch.c:370]: evapi_recv_client():
executing event route for frame:
[{"event":"REGISTER","tindex":"7363","tlabel":"949153805","PhoneNumber":"11164","DeviceId":"abcd1234abcd1234","CallId":"10946-15480@198.24.63.39
<mailto:10946-15480@198.24.63.39>"}] (145)
Just in case if you need to see the NGREP trace for the above
messages, it is here:
http://pastebin.com/4zTKmBJX
Thanks,
- Jayesh
On Tue, Sep 29, 2015 at 12:16 PM Daniel-Constantin Mierla
<miconda(a)gmail.com <mailto:miconda@gmail.com>> wrote:
Hello,
can you send here the log messages printed by evapi that
contain "] - received [" which have:
- the previous bunch of json documents that were processed ok
(just before the one that was discarded)
- the bunch of json documents that were discarded
- the next bunch of json documents that were processed ok
The logs were sent on could of emails and pastebin, so it is
hard to track them -- if you have the logs, just locate the
log messages with the discarded json and send it here along
with the previous and next logs.
Cheers,
Daniel
On 28/09/15 17:04, Jayesh Nambiar wrote:
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Book: SIP Routing With Kamailio - http://www.asipto.com
Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
Thank you daniel.
Yes, they all get printed by the evapi module even when they
get discarded. If you once again check the paste-bin, the
logs pasted as Kamailio Logs are all discarded packets but
show up as received by evapi module.
I believe they got discarded because a packet that came
before them was partial and evapi discarded the remaining
part of the message in the new packet and encountered error.
- Jayesh
On Mon, Sep 28, 2015 at 11:13 AM Daniel-Constantin Mierla
<miconda(a)gmail.com <mailto:miconda@gmail.com>> wrote:
Hello,
I will look over it during next days.
One more thing, are the packages sent by you printed by
evapi module, even not they are not processed (but
discarded)?
Cheers,
Daniel
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> -
http://www.linkedin.com/in/miconda
Book: SIP Routing With Kamailio - http://www.asipto.com
11 Oct
11 Oct
1:29 p.m.
Hello,
wondering if you had the time to test and see if the issue has been fixed...
Cheers,
Daniel
On 30/09/15 08:31, Daniel-Constantin Mierla wrote:
Hello,
thanks for troubleshooting further and pointing to the issue. I pushed
a patch last night, hopefully fixes it. Try with latest master branch
and let me know the results.
Cheers,
Daniel
On 29/09/15 14:15, Jayesh Nambiar wrote:
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Book: SIP Routing With Kamailio - http://www.asipto.com
Hi Daniel,
Taking a second look at the NGREP Trace I think I can see what is
going wrong. The message in the third packet was discarded by Evapi
module. If you look at the first two packets carefully, the first
packet ends at Dev and the second packet is the continuation of first
packet which starts with "iceId". The first two packets are processed
perfectly fine.
Now the third packet is a new message, but Kamailio is trying to
continue the third packet again with the first packet. So even when
the third packet starts with '146:' Kamailio considers it to be
starting with 'Dev146'. This causes the evapi to discard this message
since it does not remain a valid netstring then. Here the three
packets that I'm talking about:
T 198.24.63.45:48905 <http://198.24.63.45:48905> -> 198.24.63.39:3927
<http://198.24.63.39:3927> [A]
146:{"event":"REGISTER","tindex":"43213","tlabel":"220429340","PhoneNumber":"11151","DeviceId":"abcd1234abcd1234","CallId":"10932-15480@198.24.63.39
<mailto:10932-15480@198.24.63.39>"},146:{"event":"REGISTER","tindex":"52174","tlabel":"708781916","PhoneNumber":"11150","DeviceId":"abcd1234abcd1234","CallId":"10933-15480@198.24.63.39
<mailto:10933-15480@198.24.63.39>"},147:{"event":"REGISTER","tindex":"34251","tlabel":"1354642017","PhoneNumber":"11153","DeviceId":"abcd1234abcd1234","CallId":"10935-15480@198.24.63.39
<mailto:10935-15480@198.24.63.39>"},146:{"event":"REGISTER","tindex":"7366","tlabel":"1368396808","PhoneNumber":"11154","DeviceId":"abcd1234abcd1234","CallId":"10936-15480@198.24.63.39
<mailto:10936-15480@198.24.63.39>"},146:{"event":"REGISTER","tindex":"25289","tlabel":"119788881","PhoneNumber":"11152","DeviceId":"abcd1234abcd1234","C
allId":"10934-15480@198.24.63.39
<mailto:10934-15480@198.24.63.39>"},146:{"event":"REGISTER","tindex":"16327","tlabel":"971937029","PhoneNumber":"11155","DeviceId":"abcd1234abcd1234","CallId":"10937-15480@198.24.63.39
<mailto:10937-15480@198.24.63.39>"},146:{"event":"REGISTER","tindex":"22242","tlabel":"609737074","PhoneNumber":"11156","DeviceId":"abcd1234abcd1234","CallId":"10938-15480@198.24.63.39
<mailto:10938-15480@198.24.63.39>"},146:{"event":"REGISTER","tindex":"10720","tlabel":"930036340","PhoneNumber":"11157","DeviceId":"abcd1234abcd1234","CallId":"10939-15480@198.24.63.39
<mailto:10939-15480@198.24.63.39>"},146:{"event":"REGISTER","tindex":"61133","tlabel":"450961398","PhoneNumber":"11158","DeviceId":"abcd1234abcd1234","CallId":"10940-15480@198.24.63.39
<mailto:10940-15480@198.24.63.39>"},146:{"event":"R
EGISTER","tindex":"43210","tlabel":"639090568","PhoneNumber":"11160","Dev
T 198.24.63.45:48905 <http://198.24.63.45:48905> -> 198.24.63.39:3927
<http://198.24.63.39:3927> [AP]
iceId":"abcd1234abcd1234","CallId":"10942-15480@198.24.63.39
<mailto:10942-15480@198.24.63.39>"},146:{"event":"REGISTER","tindex":"4559","tlabel":"1701553747","PhoneNumber":"11159","DeviceId":"abcd1234abcd1234","CallId":"10941-15480@198.24.63.39
<mailto:10941-15480@198.24.63.39>"},
T 198.24.63.45:48905 <http://198.24.63.45:48905> -> 198.24.63.39:3927
<http://198.24.63.39:3927> [AP]
146:{"event":"REGISTER","tindex":"52171","tlabel":"748030241","PhoneNumber":"11161","DeviceId":"abcd1234abcd1234","CallId":"10943-15480@198.24.63.39
<mailto:10943-15480@198.24.63.39>"},
On Tue, Sep 29, 2015 at 4:43 PM Jayesh Nambiar <jayesh1017(a)gmail.com
<mailto:jayesh1017@gmail.com>> wrote:
Hi Daniel:
I did a new test where I stopped as soon as I saw one message
that got discarded. Basically the following message got discarded:
{"event":"REGISTER","tindex":"52171","tlabel":"748030241","PhoneNumber":"11161","DeviceId":"abcd1234abcd1234","CallId":"10943-15480@198.24.63.39"}
The CallId parameter in the above json is auto incremented, so I
tried gathering logs for messages with following CallIds:
"CallId":"10940-15480@198.24.63.39
<mailto:10940-15480@198.24.63.39>"
"CallId":"10941-15480@198.24.63.39
<mailto:10941-15480@198.24.63.39>"
"CallId":"10942-15480@198.24.63.39
<mailto:10942-15480@198.24.63.39>"
"CallId":"10943-15480@198.24.63.39
<mailto:10943-15480@198.24.63.39>"
"CallId":"10944-15480@198.24.63.39
<mailto:10944-15480@198.24.63.39>"
"CallId":"10945-15480@198.24.63.39
<mailto:10945-15480@198.24.63.39>"
And the message with CallID "CallId":"10943-15480@198.24.63.39
<http://198.24.63.39>" got discarded.
Here's the bunch of kamailio logs printed by the evapi module:
*Logs for processed messages just before the discarded message:*
DEBUG: evapi [evapi_dispatch.c:492]: evapi_recv_notify():
received [0x7f43da494ad0]
[146:{"event":"REGISTER","tindex":"61133","tlabel":"450961398","PhoneNumber":"11158","DeviceId":"abcd1234abcd1234","CallId":"10940-15480@198.24.63.39"},]
(151)
NOTICE: evapi_recv_client(): {0} [198.24.63.45:48905
<http://198.24.63.45:48905>] - received
[147:{"event":"REGISTER","tindex":"25291","tlabel":"2047024302","PhoneNumber":"11132","DeviceId":"abcd1234abcd1234","CallId":"10914-15480@198.24.63.39"},146:{"event":"REGISTER","tindex":"61136","tlabel":"839735223","PhoneNumber":"11148","DeviceId":"abcd1234abcd1234","CallId":"10930-15480@198.24.63.39"},143:{"event":"REGISTER","tindex":"4562","tlabel":"6586867","PhoneNumber":"11149","DeviceId":"abcd1234abcd1234","CallId":"10931-15480@198.24.63.39"},146:{"event":"REGISTER","tindex":"43213","tlabel":"220429340","PhoneNumber":"11151","DeviceId":"abcd1234abcd1234","CallId":"10932-15480@198.24.63.39"},146:{"event":"REGISTER","tindex":"52174","tlabel":"708781916","PhoneNumber":"11150","DeviceId":"abcd1234abcd1234","CallId":"10933-15480@198.24.63.39"},147:{"event":"REGISTER","tindex":"34251","tlabel":"1354642017","PhoneNumber":"11153","DeviceId":"abcd1234abcd1234","CallId":"10935-15480@198.24.63.39"},146:{"event":"REGISTER","tindex":"7366","tlabel":"1368396808","PhoneNumber":"11154","DeviceId":"abcd1234abcd1234","CallId":"10936-15480@198.24.63.39"},146:{"event":"REGISTER","tindex":"25289","tlabel":"119788881","PhoneNumber":"11152","DeviceId":"abcd1234abcd1234","CallId":"10934-15480@198.24.63.39"},146:{"event":"REGISTER","tindex":"16327","tlabel":"971937029","PhoneNumber":"11155","DeviceId":"abcd1234abcd1234","CallId":"10937-15480@198.24.63.39"},146:{"event":"REGISTER","tindex":"22242","tlabel":"609737074","PhoneNumber":"11156","DeviceId":"abcd1234abcd1234","CallId":"10938-15480@198.24.63.39"},146:{"event":"REGISTER","tindex":"10720","tlabel":"930036340","PhoneNumber":"11157","DeviceId":"abcd1234abcd1234","CallId":"10939-15480@198.24.63.39"},146:{"event":"REGISTER","tindex":"61133","tlabel":"450961398","PhoneNumber":"11158","DeviceId":"abcd1234abcd1234","CallId":"10940-15480@198.24.63.39"},146:{"event":"REGISTER","tindex":"43210","tlabel":"639090568","PhoneNumber":"11160","Dev]
(1899) (0)
evapi_recv_client(): executing event route for frame:
[{"event":"REGISTER","tindex":"61133","tlabel":"450961398","PhoneNumber":"11158","DeviceId":"abcd1234abcd1234","CallId":"10940-15480@198.24.63.39"}]
(146)
Sep 29 05:35:14 v39 /usr/local/kamailio-dev/sbin/kamailio[15387]:
DEBUG: evapi [evapi_dispatch.c:492]: evapi_recv_notify():
received [0x7f43da4b67e0]
[146:{"event":"REGISTER","tindex":"4559","tlabel":"1701553747","PhoneNumber":"11159","DeviceId":"abcd1234abcd1234","CallId":"10941-15480@198.24.63.39"},]
(151)
NOTICE: evapi [evapi_dispatch.c:290]: evapi_recv_client(): {0}
[198.24.63.45:48905 <http://198.24.63.45:48905>] - received
[iceId":"abcd1234abcd1234","CallId":"10942-15480@198.24.63.39"},146:{"event":"REGISTER","tindex":"4559","tlabel":"1701553747","PhoneNumber":"11159","DeviceId":"abcd1234abcd1234","CallId":"10941-15480@198.24.63.39"},]
(214) (88)
DEBUG: evapi [evapi_dispatch.c:370]: evapi_recv_client():
executing event route for frame:
[{"event":"REGISTER","tindex":"4559","tlabel":"1701553747","PhoneNumber":"11159","DeviceId":"abcd1234abcd1234","CallId":"10941-15480@198.24.63.39"}]
(146)
DEBUG: evapi [evapi_dispatch.c:492]: evapi_recv_notify():
received [0x7f43d93c2a70]
[146:{"event":"REGISTER","tindex":"43210","tlabel":"639090568","PhoneNumber":"11160","DeviceId":"abcd1234abcd1234","CallId":"10942-15480@198.24.63.39"},]
(151)
DEBUG: evapi [evapi_dispatch.c:370]: evapi_recv_client():
executing event route for frame:
[{"event":"REGISTER","tindex":"43210","tlabel":"639090568","PhoneNumber":"11160","DeviceId":"abcd1234abcd1234","CallId":"10942-15480@198.24.63.39"}]
(146)
DEBUG: evapi [evapi_dispatch.c:353]: evapi_recv_client():
residual data
[146:{"event":"REGISTER","tindex":"43210","tlabel":"639090568","PhoneNumber":"11160","Dev]
(88)
DEBUG: evapi [evapi_dispatch.c:361]: evapi_recv_client(): frame
size mismatch the ending char ("):
[{"event":"REGISTER","tindex":"43210","tlabel":"639090568","PhoneNumber":"11160","Dev146:{"event":"REGISTER","tindex":"52171","tlabel":"748030241",]
(146)
*Logs for discarded message:*
DEBUG: evapi [evapi_dispatch.c:492]: evapi_recv_notify():
received [0x7f43da3e69a0]
[146:{"event":"REGISTER","tindex":"52171","tlabel":"748030241","PhoneNumber":"11161","DeviceId":"abcd1234abcd1234","CallId":"10943-15480@198.24.63.39"},]
(151)
NOTICE: evapi [evapi_dispatch.c:290]: evapi_recv_client(): {0}
[198.24.63.45:48905 <http://198.24.63.45:48905>] - received
[146:{"event":"REGISTER","tindex":"52171","tlabel":"748030241","PhoneNumber":"11161","DeviceId":"abcd1234abcd1234","CallId":"10943-15480@198.24.63.39"},]
(151) (88)
*Logs for messages after the discarded message:*
DEBUG: evapi [evapi_dispatch.c:492]: evapi_recv_notify():
received [0x7f43da3e6aa0]
[146:{"event":"REGISTER","tindex":"25286","tlabel":"809034258","PhoneNumber":"11163","DeviceId":"abcd1234abcd1234","CallId":"10944-15480@198.24.63.39"},]
(151)
NOTICE: evapi [evapi_dispatch.c:290]: evapi_recv_client(): {0}
[198.24.63.45:48905 <http://198.24.63.45:48905>] - received
[146:{"event":"REGISTER","tindex":"25286","tlabel":"809034258","PhoneNumber":"11163","DeviceId":"abcd1234abcd1234","CallId":"10944-15480@198.24.63.39"},]
(151) (0)
DEBUG: evapi [evapi_dispatch.c:370]: evapi_recv_client():
executing event route for frame:
[{"event":"REGISTER","tindex":"25286","tlabel":"809034258","PhoneNumber":"11163","DeviceId":"abcd1234abcd1234","CallId":"10944-15480@198.24.63.39"}]
(146)
DEBUG: evapi [evapi_dispatch.c:492]: evapi_recv_notify():
received [0x7f43d9598778]
[146:{"event":"REGISTER","tindex":"34248","tlabel":"187029249","PhoneNumber":"11162","DeviceId":"abcd1234abcd1234","CallId":"10945-15480@198.24.63.39"},]
(151)
NOTICE: evapi [evapi_dispatch.c:290]: evapi_recv_client(): {0}
[198.24.63.45:48905 <http://198.24.63.45:48905>] - received
[146:{"event":"REGISTER","tindex":"34248","tlabel":"187029249","PhoneNumber":"11162","DeviceId":"abcd1234abcd1234","CallId":"10945-15480@198.24.63.39"},]
(151) (0)
DEBUG: evapi [evapi_dispatch.c:370]: evapi_recv_client():
executing event route for frame:
[{"event":"REGISTER","tindex":"34248","tlabel":"187029249","PhoneNumber":"11162","DeviceId":"abcd1234abcd1234","CallId":"10945-15480@198.24.63.39"}]
(146)
DEBUG: evapi [evapi_dispatch.c:492]: evapi_recv_notify():
received [0x7f43d9598878]
[145:{"event":"REGISTER","tindex":"7363","tlabel":"949153805","PhoneNumber":"11164","DeviceId":"abcd1234abcd1234","CallId":"10946-15480@198.24.63.39"},]
(150)
NOTICE: evapi [evapi_dispatch.c:290]: evapi_recv_client(): {0}
[198.24.63.45:48905 <http://198.24.63.45:48905>] - received
[145:{"event":"REGISTER","tindex":"7363","tlabel":"949153805","PhoneNumber":"11164","DeviceId":"abcd1234abcd1234","CallId":"10946-15480@198.24.63.39"},]
(150) (0)
DEBUG: evapi [evapi_dispatch.c:370]: evapi_recv_client():
executing event route for frame:
[{"event":"REGISTER","tindex":"7363","tlabel":"949153805","PhoneNumber":"11164","DeviceId":"abcd1234abcd1234","CallId":"10946-15480@198.24.63.39"}]
(145)
Just in case if you need to see the NGREP trace for the above
messages, it is here:
http://pastebin.com/4zTKmBJX
Thanks,
- Jayesh
On Tue, Sep 29, 2015 at 12:16 PM Daniel-Constantin Mierla
<miconda(a)gmail.com <mailto:miconda@gmail.com>> wrote:
Hello,
can you send here the log messages printed by evapi that
contain "] - received [" which have:
- the previous bunch of json documents that were processed ok
(just before the one that was discarded)
- the bunch of json documents that were discarded
- the next bunch of json documents that were processed ok
The logs were sent on could of emails and pastebin, so it is
hard to track them -- if you have the logs, just locate the
log messages with the discarded json and send it here along
with the previous and next logs.
Cheers,
Daniel
On 28/09/15 17:04, Jayesh Nambiar wrote:
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Book: SIP Routing With Kamailio - http://www.asipto.com
Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat Thank you daniel.
Yes, they all get printed by the evapi module even when they
get discarded. If you once again check the paste-bin, the
logs pasted as Kamailio Logs are all discarded packets but
show up as received by evapi module.
I believe they got discarded because a packet that came
before them was partial and evapi discarded the remaining
part of the message in the new packet and encountered error.
- Jayesh
On Mon, Sep 28, 2015 at 11:13 AM Daniel-Constantin Mierla
<miconda(a)gmail.com> wrote:
Hello,
I will look over it during next days.
One more thing, are the packages sent by you printed by
evapi module, even not they are not processed (but
discarded)?
Cheers,
Daniel
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> -
http://www.linkedin.com/in/miconda
Book: SIP Routing With Kamailio - http://www.asipto.com
13 Oct
13 Oct
6:37 a.m.
Yes that problem is definitely solved. Sorry to not post the response. I'm
having some other issues in scaling and still trying to figure where the
problem must be, but I can scale upto 2000 cps easily now.
Thanks for the efforts and the credits :)
- Jayesh
On Sun, Oct 11, 2015 at 10:59 PM Daniel-Constantin Mierla <miconda(a)gmail.com>
wrote:
Hello,
wondering if you had the time to test and see if the issue has been
fixed...
Cheers,
Daniel
On 30/09/15 08:31, Daniel-Constantin Mierla wrote:
Hello,
thanks for troubleshooting further and pointing to the issue. I pushed a
patch last night, hopefully fixes it. Try with latest master branch and let
me know the results.
Cheers,
Daniel
On 29/09/15 14:15, Jayesh Nambiar wrote:
Hi Daniel,
Taking a second look at the NGREP Trace I think I can see what is going
wrong. The message in the third packet was discarded by Evapi module. If
you look at the first two packets carefully, the first packet ends at Dev
and the second packet is the continuation of first packet which starts with
"iceId". The first two packets are processed perfectly fine.
Now the third packet is a new message, but Kamailio is trying to continue
the third packet again with the first packet. So even when the third packet
starts with '146:' Kamailio considers it to be starting with 'Dev146'.
This
causes the evapi to discard this message since it does not remain a valid
netstring then. Here the three packets that I'm talking about:
T 198.24.63.45:48905 -> 198.24.63.39:3927 [A]
146:{"event":"REGISTER","tindex":"43213","tlabel":"220429340","PhoneNumber":"11151","DeviceId":"abcd1234abcd1234","CallId":"10932-15480@198.24.63.39"},146:{"event":"REGISTER","tindex":"52174","tlabel":"708781916","PhoneNumber":"11150","DeviceId":"abcd1234abcd1234","CallId":"10933-15480@198.24.63.39"},147:{"event":"REGISTER","tindex":"34251","tlabel":"1354642017","PhoneNumber":"11153","DeviceId":"abcd1234abcd1234","CallId":"10935-15480@198.24.63.39"},146:{"event":"REGISTER","tindex":"7366","tlabel":"1368396808","PhoneNumber":"11154","DeviceId":"abcd1234abcd1234","CallId":"10936-15480@198.24.63.39"},146:{"event":"REGISTER","tindex":"25289","tlabel":"119788881","PhoneNumber":"11152","DeviceId":"abcd1234abcd1234","C
allId":"10934-15480@198.24.63.39"},146:{"event":"REGISTER","tindex":"16327","tlabel":"971937029","PhoneNumber":"11155","DeviceId":"abcd1234abcd1234","CallId":"10937-15480@198.24.63.39"},146:{"event":"REGISTER","tindex":"22242","tlabel":"609737074","PhoneNumber":"11156","DeviceId":"abcd1234abcd1234","CallId":"10938-15480@198.24.63.39"},146:{"event":"REGISTER","tindex":"10720","tlabel":"930036340","PhoneNumber":"11157","DeviceId":"abcd1234abcd1234","CallId":"10939-15480@198.24.63.39"},146:{"event":"REGISTER","tindex":"61133","tlabel":"450961398","PhoneNumber":"11158","DeviceId":"abcd1234abcd1234","CallId":"10940-15480@198.24.63.39"},146:{"event":"R
EGISTER","tindex":"43210","tlabel":"639090568","PhoneNumber":"11160","Dev
T 198.24.63.45:48905 -> 198.24.63.39:3927 [AP]
iceId":"abcd1234abcd1234","CallId":"10942-15480@198.24.63.39"},146:{"event":"REGISTER","tindex":"4559","tlabel":"1701553747","PhoneNumber":"11159","DeviceId":"abcd1234abcd1234","CallId":"10941-15480@198.24.63.39"},
T 198.24.63.45:48905 -> 198.24.63.39:3927 [AP]
146:{"event":"REGISTER","tindex":"52171","tlabel":"748030241","PhoneNumber":"11161","DeviceId":"abcd1234abcd1234","CallId":"10943-15480@198.24.63.39"},
On Tue, Sep 29, 2015 at 4:43 PM Jayesh Nambiar <jayesh1017(a)gmail.com>
wrote:
Hi Daniel:
I did a new test where I stopped as soon as I saw one message that got
discarded. Basically the following message got discarded:
{"event":"REGISTER","tindex":"52171","tlabel":"748030241","PhoneNumber":"11161","DeviceId":"abcd1234abcd1234","CallId":"
10943-15480(a)198.24.63.39"}
The CallId parameter in the above json is auto incremented, so I tried
gathering logs for messages with following CallIds:
"CallId":"10940-15480@198.24.63.39"
"CallId":"10941-15480@198.24.63.39"
"CallId":"10942-15480@198.24.63.39"
"CallId":"10943-15480@198.24.63.39"
"CallId":"10944-15480@198.24.63.39"
"CallId":"10945-15480@198.24.63.39"
And the message with CallID "CallId":"10943-15480@198.24.63.39" got
discarded.
Here's the bunch of kamailio logs printed by the evapi module:
*Logs for processed messages just before the discarded message:*
DEBUG: evapi [evapi_dispatch.c:492]: evapi_recv_notify(): received
[0x7f43da494ad0]
[146:{"event":"REGISTER","tindex":"61133","tlabel":"450961398","PhoneNumber":"11158","DeviceId":"abcd1234abcd1234","CallId":"
10940-15480(a)198.24.63.39"}uot;},] (151)
NOTICE: evapi_recv_client(): {0} [198.24.63.45:48905] - received
[147:{"event":"REGISTER","tindex":"25291","tlabel":"2047024302","PhoneNumber":"11132","DeviceId":"abcd1234abcd1234","CallId":"
10914-15480(a)198.24.63.39
"},146:{"event":"REGISTER","tindex":"61136","tlabel":"839735223","PhoneNumber":"11148","DeviceId":"abcd1234abcd1234","CallId":"
10930-15480(a)198.24.63.39
"},143:{"event":"REGISTER","tindex":"4562","tlabel":"6586867","PhoneNumber":"11149","DeviceId":"abcd1234abcd1234","CallId":"
10931-15480(a)198.24.63.39
"},146:{"event":"REGISTER","tindex":"43213","tlabel":"220429340","PhoneNumber":"11151","DeviceId":"abcd1234abcd1234","CallId":"
10932-15480(a)198.24.63.39
"},146:{"event":"REGISTER","tindex":"52174","tlabel":"708781916","PhoneNumber":"11150","DeviceId":"abcd1234abcd1234","CallId":"
10933-15480(a)198.24.63.39
"},147:{"event":"REGISTER","tindex":"34251","tlabel":"1354642017","PhoneNumber":"11153","DeviceId":"abcd1234abcd1234","CallId":"
10935-15480(a)198.24.63.39
"},146:{"event":"REGISTER","tindex":"7366","tlabel":"1368396808","PhoneNumber":"11154","DeviceId":"abcd1234abcd1234","CallId":"
10936-15480(a)198.24.63.39
"},146:{"event":"REGISTER","tindex":"25289","tlabel":"119788881","PhoneNumber":"11152","DeviceId":"abcd1234abcd1234","CallId":"
10934-15480(a)198.24.63.39
"},146:{"event":"REGISTER","tindex":"16327","tlabel":"971937029","PhoneNumber":"11155","DeviceId":"abcd1234abcd1234","CallId":"
10937-15480(a)198.24.63.39
"},146:{"event":"REGISTER","tindex":"22242","tlabel":"609737074","PhoneNumber":"11156","DeviceId":"abcd1234abcd1234","CallId":"
10938-15480(a)198.24.63.39
"},146:{"event":"REGISTER","tindex":"10720","tlabel":"930036340","PhoneNumber":"11157","DeviceId":"abcd1234abcd1234","CallId":"
10939-15480(a)198.24.63.39
"},146:{"event":"REGISTER","tindex":"61133","tlabel":"450961398","PhoneNumber":"11158","DeviceId":"abcd1234abcd1234","CallId":"
10940-15480@198.24.63.39"},146:{"event":"REGISTER","tindex":"43210","tlabel":"639090568","PhoneNumber":"11160","Dev]
(1899) (0)
evapi_recv_client(): executing event route for frame:
[{"event":"REGISTER","tindex":"61133","tlabel":"450961398","PhoneNumber":"11158","DeviceId":"abcd1234abcd1234","CallId":"
10940-15480(a)198.24.63.39"}] (146)
Sep 29 05:35:14 v39 /usr/local/kamailio-dev/sbin/kamailio[15387]: DEBUG:
evapi [evapi_dispatch.c:492]: evapi_recv_notify(): received
[0x7f43da4b67e0]
[146:{"event":"REGISTER","tindex":"4559","tlabel":"1701553747","PhoneNumber":"11159","DeviceId":"abcd1234abcd1234","CallId":"
10941-15480(a)198.24.63.39"}uot;},] (151)
NOTICE: evapi [evapi_dispatch.c:290]: evapi_recv_client(): {0} [
198.24.63.45:48905] - received
[iceId":"abcd1234abcd1234","CallId":"
10942-15480(a)198.24.63.39
"},146:{"event":"REGISTER","tindex":"4559","tlabel":"1701553747","PhoneNumber":"11159","DeviceId":"abcd1234abcd1234","CallId":"
10941-15480(a)198.24.63.39"}uot;},] (214) (88)
DEBUG: evapi [evapi_dispatch.c:370]: evapi_recv_client(): executing event
route for frame:
[{"event":"REGISTER","tindex":"4559","tlabel":"1701553747","PhoneNumber":"11159","DeviceId":"abcd1234abcd1234","CallId":"
10941-15480(a)198.24.63.39"}] (146)
DEBUG: evapi [evapi_dispatch.c:492]: evapi_recv_notify(): received
[0x7f43d93c2a70]
[146:{"event":"REGISTER","tindex":"43210","tlabel":"639090568","PhoneNumber":"11160","DeviceId":"abcd1234abcd1234","CallId":"
10942-15480(a)198.24.63.39"}uot;},] (151)
DEBUG: evapi [evapi_dispatch.c:370]: evapi_recv_client(): executing event
route for frame:
[{"event":"REGISTER","tindex":"43210","tlabel":"639090568","PhoneNumber":"11160","DeviceId":"abcd1234abcd1234","CallId":"
10942-15480(a)198.24.63.39"}] (146)
DEBUG: evapi [evapi_dispatch.c:353]: evapi_recv_client(): residual data
[146:{"event":"REGISTER","tindex":"43210","tlabel":"639090568","PhoneNumber":"11160","Dev]
(88)
DEBUG: evapi [evapi_dispatch.c:361]: evapi_recv_client(): frame size
mismatch the ending char ("):
[{"event":"REGISTER","tindex":"43210","tlabel":"639090568","PhoneNumber":"11160","Dev146:{"event":"REGISTER","tindex":"52171","tlabel":"748030241",]
(146)
*Logs for discarded message:*
DEBUG: evapi [evapi_dispatch.c:492]: evapi_recv_notify(): received
[0x7f43da3e69a0]
[146:{"event":"REGISTER","tindex":"52171","tlabel":"748030241","PhoneNumber":"11161","DeviceId":"abcd1234abcd1234","CallId":"
10943-15480(a)198.24.63.39"}uot;},] (151)
NOTICE: evapi [evapi_dispatch.c:290]: evapi_recv_client(): {0} [
198.24.63.45:48905] - received
[146:{"event":"REGISTER","tindex":"52171","tlabel":"748030241","PhoneNumber":"11161","DeviceId":"abcd1234abcd1234","CallId":"
10943-15480(a)198.24.63.39"}uot;},] (151) (88)
*Logs for messages after the discarded message:*
DEBUG: evapi [evapi_dispatch.c:492]: evapi_recv_notify(): received
[0x7f43da3e6aa0]
[146:{"event":"REGISTER","tindex":"25286","tlabel":"809034258","PhoneNumber":"11163","DeviceId":"abcd1234abcd1234","CallId":"
10944-15480(a)198.24.63.39"}uot;},] (151)
NOTICE: evapi [evapi_dispatch.c:290]: evapi_recv_client(): {0} [
198.24.63.45:48905] - received
[146:{"event":"REGISTER","tindex":"25286","tlabel":"809034258","PhoneNumber":"11163","DeviceId":"abcd1234abcd1234","CallId":"
10944-15480(a)198.24.63.39"}uot;},] (151) (0)
DEBUG: evapi [evapi_dispatch.c:370]: evapi_recv_client(): executing event
route for frame:
[{"event":"REGISTER","tindex":"25286","tlabel":"809034258","PhoneNumber":"11163","DeviceId":"abcd1234abcd1234","CallId":"
10944-15480(a)198.24.63.39"}] (146)
DEBUG: evapi [evapi_dispatch.c:492]: evapi_recv_notify(): received
[0x7f43d9598778]
[146:{"event":"REGISTER","tindex":"34248","tlabel":"187029249","PhoneNumber":"11162","DeviceId":"abcd1234abcd1234","CallId":"
10945-15480(a)198.24.63.39"}uot;},] (151)
NOTICE: evapi [evapi_dispatch.c:290]: evapi_recv_client(): {0} [
198.24.63.45:48905] - received
[146:{"event":"REGISTER","tindex":"34248","tlabel":"187029249","PhoneNumber":"11162","DeviceId":"abcd1234abcd1234","CallId":"
10945-15480(a)198.24.63.39"}uot;},] (151) (0)
DEBUG: evapi [evapi_dispatch.c:370]: evapi_recv_client(): executing event
route for frame:
[{"event":"REGISTER","tindex":"34248","tlabel":"187029249","PhoneNumber":"11162","DeviceId":"abcd1234abcd1234","CallId":"
10945-15480(a)198.24.63.39"}] (146)
DEBUG: evapi [evapi_dispatch.c:492]: evapi_recv_notify(): received
[0x7f43d9598878]
[145:{"event":"REGISTER","tindex":"7363","tlabel":"949153805","PhoneNumber":"11164","DeviceId":"abcd1234abcd1234","CallId":"
10946-15480(a)198.24.63.39"}uot;},] (150)
NOTICE: evapi [evapi_dispatch.c:290]: evapi_recv_client(): {0} [
198.24.63.45:48905] - received
[145:{"event":"REGISTER","tindex":"7363","tlabel":"949153805","PhoneNumber":"11164","DeviceId":"abcd1234abcd1234","CallId":"
10946-15480(a)198.24.63.39"}uot;},] (150) (0)
DEBUG: evapi [evapi_dispatch.c:370]: evapi_recv_client(): executing event
route for frame:
[{"event":"REGISTER","tindex":"7363","tlabel":"949153805","PhoneNumber":"11164","DeviceId":"abcd1234abcd1234","CallId":"
10946-15480(a)198.24.63.39"}] (145)
Just in case if you need to see the NGREP trace for the above messages,
it is here:
http://pastebin.com/4zTKmBJX
Thanks,
- Jayesh
On Tue, Sep 29, 2015 at 12:16 PM Daniel-Constantin Mierla <
miconda(a)gmail.com> wrote:
> Hello,
>
> can you send here the log messages printed by evapi that contain "] -
> received [" which have:
>
> - the previous bunch of json documents that were processed ok (just
> before the one that was discarded)
> - the bunch of json documents that were discarded
> - the next bunch of json documents that were processed ok
>
> The logs were sent on could of emails and pastebin, so it is hard to
> track them -- if you have the logs, just locate the log messages with the
> discarded json and send it here along with the previous and next logs.
>
> Cheers,
> Daniel
>
>
> On 28/09/15 17:04, Jayesh Nambiar wrote:
>
> Thank you daniel.
> Yes, they all get printed by the evapi module even when they get
> discarded. If you once again check the paste-bin, the logs pasted as
> Kamailio Logs are all discarded packets but show up as received by evapi
> module.
> I believe they got discarded because a packet that came before them was
> partial and evapi discarded the remaining part of the message in the new
> packet and encountered error.
> - Jayesh
>
> On Mon, Sep 28, 2015 at 11:13 AM Daniel-Constantin Mierla <
> miconda(a)gmail.com> wrote:
>
>> Hello,
>>
>> I will look over it during next days.
>>
>> One more thing, are the packages sent by you printed by evapi module,
>> even not they are not processed (but discarded)?
>>
>> Cheers,
>> Daniel
>>
>>
> --
> Daniel-Constantin Mierlahttp://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda
> Book: SIP Routing With Kamailio - http://www.asipto.com
>
>
--
Daniel-Constantin Mierlahttp://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda
Book: SIP Routing With Kamailio - http://www.asipto.com
Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
--
Daniel-Constantin Mierlahttp://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda
Book: SIP Routing With Kamailio - http://www.asipto.com
8:09 a.m.
Thanks for testing and feedback -- good to know it was fixed, not to
keep it open in my side anymore.
Cheers,
Daniel
On 13/10/15 12:37, Jayesh Nambiar wrote:
Yes that problem is definitely solved. Sorry to not
post the response.
I'm having some other issues in scaling and still trying to figure
where the problem must be, but I can scale upto 2000 cps easily now.
Thanks for the efforts and the credits :)
- Jayesh
On Sun, Oct 11, 2015 at 10:59 PM Daniel-Constantin Mierla
<miconda(a)gmail.com <mailto:miconda@gmail.com>> wrote:
Hello,
wondering if you had the time to test and see if the issue has
been fixed...
Cheers,
Daniel
On 30/09/15 08:31, Daniel-Constantin Mierla wrote:
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Book: SIP Routing With Kamailio - http://www.asipto.com
Hello,
thanks for troubleshooting further and pointing to the issue. I
pushed a patch last night, hopefully fixes it. Try with latest
master branch and let me know the results.
Cheers,
Daniel
On 29/09/15 14:15, Jayesh Nambiar wrote:
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> -
http://www.linkedin.com/in/miconda
Book: SIP Routing With Kamailio - http://www.asipto.com
Hi Daniel,
Taking a second look at the NGREP Trace I think I can see what
is going wrong. The message in the third packet was discarded by
Evapi module. If you look at the first two packets carefully,
the first packet ends at Dev and the second packet is the
continuation of first packet which starts with "iceId". The
first two packets are processed perfectly fine.
Now the third packet is a new message, but Kamailio is trying to
continue the third packet again with the first packet. So even
when the third packet starts with '146:' Kamailio considers it
to be starting with 'Dev146'. This causes the evapi to discard
this message since it does not remain a valid netstring then.
Here the three packets that I'm talking about:
T 198.24.63.45:48905 <http://198.24.63.45:48905> -> 198.24.63.39:3927
<http://198.24.63.39:3927> [A]
146:{"event":"REGISTER","tindex":"43213","tlabel":"220429340","PhoneNumber":"11151","DeviceId":"abcd1234abcd1234","CallId":"10932-15480@198.24.63.39
<mailto:10932-15480@198.24.63.39>"},146:{"event":"REGISTER","tindex":"52174","tlabel":"708781916","PhoneNumber":"11150","DeviceId":"abcd1234abcd1234","CallId":"10933-15480@198.24.63.39
<mailto:10933-15480@198.24.63.39>"},147:{"event":"REGISTER","tindex":"34251","tlabel":"1354642017","PhoneNumber":"11153","DeviceId":"abcd1234abcd1234","CallId":"10935-15480@198.24.63.39
<mailto:10935-15480@198.24.63.39>"},146:{"event":"REGISTER","tindex":"7366","tlabel":"1368396808","PhoneNumber":"11154","DeviceId":"abcd1234abcd1234","CallId":"10936-15480@198.24.63.39
<mailto:10936-15480@198.24.63.39>"},146:{"event":"REGISTER","tindex":"25289","tlabel":"119788881","PhoneNumber":"11152","DeviceId":"abcd1234abcd1234","C
allId":"10934-15480@198.24.63.39
<mailto:10934-15480@198.24.63.39>"},146:{"event":"REGISTER","tindex":"16327","tlabel":"971937029","PhoneNumber":"11155","DeviceId":"abcd1234abcd1234","CallId":"10937-15480@198.24.63.39
<mailto:10937-15480@198.24.63.39>"},146:{"event":"REGISTER","tindex":"22242","tlabel":"609737074","PhoneNumber":"11156","DeviceId":"abcd1234abcd1234","CallId":"10938-15480@198.24.63.39
<mailto:10938-15480@198.24.63.39>"},146:{"event":"REGISTER","tindex":"10720","tlabel":"930036340","PhoneNumber":"11157","DeviceId":"abcd1234abcd1234","CallId":"10939-15480@198.24.63.39
<mailto:10939-15480@198.24.63.39>"},146:{"event":"REGISTER","tindex":"61133","tlabel":"450961398","PhoneNumber":"11158","DeviceId":"abcd1234abcd1234","CallId":"10940-15480@198.24.63.39
<mailto:10940-15480@198.24.63.39>"},146:{"event":"R
EGISTER","tindex":"43210","tlabel":"639090568","PhoneNumber":"11160","Dev
T 198.24.63.45:48905 <http://198.24.63.45:48905> -> 198.24.63.39:3927
<http://198.24.63.39:3927> [AP]
iceId":"abcd1234abcd1234","CallId":"10942-15480@198.24.63.39
<mailto:10942-15480@198.24.63.39>"},146:{"event":"REGISTER","tindex":"4559","tlabel":"1701553747","PhoneNumber":"11159","DeviceId":"abcd1234abcd1234","CallId":"10941-15480@198.24.63.39
<mailto:10941-15480@198.24.63.39>"},
T 198.24.63.45:48905 <http://198.24.63.45:48905> -> 198.24.63.39:3927
<http://198.24.63.39:3927> [AP]
146:{"event":"REGISTER","tindex":"52171","tlabel":"748030241","PhoneNumber":"11161","DeviceId":"abcd1234abcd1234","CallId":"10943-15480@198.24.63.39
<mailto:10943-15480@198.24.63.39>"},
On Tue, Sep 29, 2015 at 4:43 PM Jayesh Nambiar
<jayesh1017(a)gmail.com <mailto:jayesh1017@gmail.com>> wrote:
Hi Daniel:
I did a new test where I stopped as soon as I saw one
message that got discarded. Basically the following message
got discarded:
{"event":"REGISTER","tindex":"52171","tlabel":"748030241","PhoneNumber":"11161","DeviceId":"abcd1234abcd1234","CallId":"10943-15480@198.24.63.39
<mailto:10943-15480@198.24.63.39>"}
The CallId parameter in the above json is auto incremented,
so I tried gathering logs for messages with following CallIds:
"CallId":"10940-15480@198.24.63.39
<mailto:10940-15480@198.24.63.39>"
"CallId":"10941-15480@198.24.63.39
<mailto:10941-15480@198.24.63.39>"
"CallId":"10942-15480@198.24.63.39
<mailto:10942-15480@198.24.63.39>"
"CallId":"10943-15480@198.24.63.39
<mailto:10943-15480@198.24.63.39>"
"CallId":"10944-15480@198.24.63.39
<mailto:10944-15480@198.24.63.39>"
"CallId":"10945-15480@198.24.63.39
<mailto:10945-15480@198.24.63.39>"
And the message with
CallID "CallId":"10943-15480@198.24.63.39
<http://198.24.63.39>" got discarded.
Here's the bunch of kamailio logs printed by the evapi module:
*Logs for processed messages just before the discarded message:*
DEBUG: evapi [evapi_dispatch.c:492]: evapi_recv_notify():
received [0x7f43da494ad0]
[146:{"event":"REGISTER","tindex":"61133","tlabel":"450961398","PhoneNumber":"11158","DeviceId":"abcd1234abcd1234","CallId":"10940-15480@198.24.63.39
<mailto:10940-15480@198.24.63.39>"},] (151)
NOTICE: evapi_recv_client(): {0} [198.24.63.45:48905
<http://198.24.63.45:48905>] - received
[147:{"event":"REGISTER","tindex":"25291","tlabel":"2047024302","PhoneNumber":"11132","DeviceId":"abcd1234abcd1234","CallId":"10914-15480@198.24.63.39
<mailto:10914-15480@198.24.63.39>"},146:{"event":"REGISTER","tindex":"61136","tlabel":"839735223","PhoneNumber":"11148","DeviceId":"abcd1234abcd1234","CallId":"10930-15480@198.24.63.39
<mailto:10930-15480@198.24.63.39>"},143:{"event":"REGISTER","tindex":"4562","tlabel":"6586867","PhoneNumber":"11149","DeviceId":"abcd1234abcd1234","CallId":"10931-15480@198.24.63.39
<mailto:10931-15480@198.24.63.39>"},146:{"event":"REGISTER","tindex":"43213","tlabel":"220429340","PhoneNumber":"11151","DeviceId":"abcd1234abcd1234","CallId":"10932-15480@198.24.63.39
<mailto:10932-15480@198.24.63.39>"},146:{"event":"REGISTER","tindex":"52174","tlabel":"708781916","PhoneNumber":"11150","DeviceId":"abcd1234abcd1234","CallId":"10933-15480@198.24.63.39
<mailto:10933-15480@198.24.63.39>"},147:{"event":"REGISTER","tindex":"34251","tlabel":"1354642017","PhoneNumber":"11153","DeviceId":"abcd1234abcd1234","CallId":"10935-15480@198.24.63.39
<mailto:10935-15480@198.24.63.39>"},146:{"event":"REGISTER","tindex":"7366","tlabel":"1368396808","PhoneNumber":"11154","DeviceId":"abcd1234abcd1234","CallId":"10936-15480@198.24.63.39
<mailto:10936-15480@198.24.63.39>"},146:{"event":"REGISTER","tindex":"25289","tlabel":"119788881","PhoneNumber":"11152","DeviceId":"abcd1234abcd1234","CallId":"10934-15480@198.24.63.39
<mailto:10934-15480@198.24.63.39>"},146:{"event":"REGISTER","tindex":"16327","tlabel":"971937029","PhoneNumber":"11155","DeviceId":"abcd1234abcd1234","CallId":"10937-15480@198.24.63.39
<mailto:10937-15480@198.24.63.39>"},146:{"event":"REGISTER","tindex":"22242","tlabel":"609737074","PhoneNumber":"11156","DeviceId":"abcd1234abcd1234","CallId":"10938-15480@198.24.63.39
<mailto:10938-15480@198.24.63.39>"},146:{"event":"REGISTER","tindex":"10720","tlabel":"930036340","PhoneNumber":"11157","DeviceId":"abcd1234abcd1234","CallId":"10939-15480@198.24.63.39
<mailto:10939-15480@198.24.63.39>"},146:{"event":"REGISTER","tindex":"61133","tlabel":"450961398","PhoneNumber":"11158","DeviceId":"abcd1234abcd1234","CallId":"10940-15480@198.24.63.39
<mailto:10940-15480@198.24.63.39>"},146:{"event":"REGISTER","tindex":"43210","tlabel":"639090568","PhoneNumber":"11160","Dev]
(1899) (0)
evapi_recv_client(): executing event route for frame:
[{"event":"REGISTER","tindex":"61133","tlabel":"450961398","PhoneNumber":"11158","DeviceId":"abcd1234abcd1234","CallId":"10940-15480@198.24.63.39
<mailto:10940-15480@198.24.63.39>"}] (146)
Sep 29 05:35:14 v39
/usr/local/kamailio-dev/sbin/kamailio[15387]: DEBUG: evapi
[evapi_dispatch.c:492]: evapi_recv_notify(): received
[0x7f43da4b67e0]
[146:{"event":"REGISTER","tindex":"4559","tlabel":"1701553747","PhoneNumber":"11159","DeviceId":"abcd1234abcd1234","CallId":"10941-15480@198.24.63.39
<mailto:10941-15480@198.24.63.39>"},] (151)
NOTICE: evapi [evapi_dispatch.c:290]: evapi_recv_client():
{0} [198.24.63.45:48905 <http://198.24.63.45:48905>] -
received
[iceId":"abcd1234abcd1234","CallId":"10942-15480@198.24.63.39
<mailto:10942-15480@198.24.63.39>"},146:{"event":"REGISTER","tindex":"4559","tlabel":"1701553747","PhoneNumber":"11159","DeviceId":"abcd1234abcd1234","CallId":"10941-15480@198.24.63.39
<mailto:10941-15480@198.24.63.39>"},] (214) (88)
DEBUG: evapi [evapi_dispatch.c:370]: evapi_recv_client():
executing event route for frame:
[{"event":"REGISTER","tindex":"4559","tlabel":"1701553747","PhoneNumber":"11159","DeviceId":"abcd1234abcd1234","CallId":"10941-15480@198.24.63.39
<mailto:10941-15480@198.24.63.39>"}] (146)
DEBUG: evapi [evapi_dispatch.c:492]: evapi_recv_notify():
received [0x7f43d93c2a70]
[146:{"event":"REGISTER","tindex":"43210","tlabel":"639090568","PhoneNumber":"11160","DeviceId":"abcd1234abcd1234","CallId":"10942-15480@198.24.63.39
<mailto:10942-15480@198.24.63.39>"},] (151)
DEBUG: evapi [evapi_dispatch.c:370]: evapi_recv_client():
executing event route for frame:
[{"event":"REGISTER","tindex":"43210","tlabel":"639090568","PhoneNumber":"11160","DeviceId":"abcd1234abcd1234","CallId":"10942-15480@198.24.63.39
<mailto:10942-15480@198.24.63.39>"}] (146)
DEBUG: evapi [evapi_dispatch.c:353]: evapi_recv_client():
residual data
[146:{"event":"REGISTER","tindex":"43210","tlabel":"639090568","PhoneNumber":"11160","Dev]
(88)
DEBUG: evapi [evapi_dispatch.c:361]: evapi_recv_client():
frame size mismatch the ending char ("):
[{"event":"REGISTER","tindex":"43210","tlabel":"639090568","PhoneNumber":"11160","Dev146:{"event":"REGISTER","tindex":"52171","tlabel":"748030241",]
(146)
*Logs for discarded message:*
DEBUG: evapi [evapi_dispatch.c:492]: evapi_recv_notify():
received [0x7f43da3e69a0]
[146:{"event":"REGISTER","tindex":"52171","tlabel":"748030241","PhoneNumber":"11161","DeviceId":"abcd1234abcd1234","CallId":"10943-15480@198.24.63.39
<mailto:10943-15480@198.24.63.39>"},] (151)
NOTICE: evapi [evapi_dispatch.c:290]: evapi_recv_client():
{0} [198.24.63.45:48905 <http://198.24.63.45:48905>] -
received
[146:{"event":"REGISTER","tindex":"52171","tlabel":"748030241","PhoneNumber":"11161","DeviceId":"abcd1234abcd1234","CallId":"10943-15480@198.24.63.39
<mailto:10943-15480@198.24.63.39>"},] (151) (88)
*Logs for messages after the discarded message:*
DEBUG: evapi [evapi_dispatch.c:492]: evapi_recv_notify():
received [0x7f43da3e6aa0]
[146:{"event":"REGISTER","tindex":"25286","tlabel":"809034258","PhoneNumber":"11163","DeviceId":"abcd1234abcd1234","CallId":"10944-15480@198.24.63.39
<mailto:10944-15480@198.24.63.39>"},] (151)
NOTICE: evapi [evapi_dispatch.c:290]: evapi_recv_client():
{0} [198.24.63.45:48905 <http://198.24.63.45:48905>] -
received
[146:{"event":"REGISTER","tindex":"25286","tlabel":"809034258","PhoneNumber":"11163","DeviceId":"abcd1234abcd1234","CallId":"10944-15480@198.24.63.39
<mailto:10944-15480@198.24.63.39>"},] (151) (0)
DEBUG: evapi [evapi_dispatch.c:370]: evapi_recv_client():
executing event route for frame:
[{"event":"REGISTER","tindex":"25286","tlabel":"809034258","PhoneNumber":"11163","DeviceId":"abcd1234abcd1234","CallId":"10944-15480@198.24.63.39
<mailto:10944-15480@198.24.63.39>"}] (146)
DEBUG: evapi [evapi_dispatch.c:492]: evapi_recv_notify():
received [0x7f43d9598778]
[146:{"event":"REGISTER","tindex":"34248","tlabel":"187029249","PhoneNumber":"11162","DeviceId":"abcd1234abcd1234","CallId":"10945-15480@198.24.63.39
<mailto:10945-15480@198.24.63.39>"},] (151)
NOTICE: evapi [evapi_dispatch.c:290]: evapi_recv_client():
{0} [198.24.63.45:48905 <http://198.24.63.45:48905>] -
received
[146:{"event":"REGISTER","tindex":"34248","tlabel":"187029249","PhoneNumber":"11162","DeviceId":"abcd1234abcd1234","CallId":"10945-15480@198.24.63.39
<mailto:10945-15480@198.24.63.39>"},] (151) (0)
DEBUG: evapi [evapi_dispatch.c:370]: evapi_recv_client():
executing event route for frame:
[{"event":"REGISTER","tindex":"34248","tlabel":"187029249","PhoneNumber":"11162","DeviceId":"abcd1234abcd1234","CallId":"10945-15480@198.24.63.39
<mailto:10945-15480@198.24.63.39>"}] (146)
DEBUG: evapi [evapi_dispatch.c:492]: evapi_recv_notify():
received [0x7f43d9598878]
[145:{"event":"REGISTER","tindex":"7363","tlabel":"949153805","PhoneNumber":"11164","DeviceId":"abcd1234abcd1234","CallId":"10946-15480@198.24.63.39
<mailto:10946-15480@198.24.63.39>"},] (150)
NOTICE: evapi [evapi_dispatch.c:290]: evapi_recv_client():
{0} [198.24.63.45:48905 <http://198.24.63.45:48905>] -
received
[145:{"event":"REGISTER","tindex":"7363","tlabel":"949153805","PhoneNumber":"11164","DeviceId":"abcd1234abcd1234","CallId":"10946-15480@198.24.63.39
<mailto:10946-15480@198.24.63.39>"},] (150) (0)
DEBUG: evapi [evapi_dispatch.c:370]: evapi_recv_client():
executing event route for frame:
[{"event":"REGISTER","tindex":"7363","tlabel":"949153805","PhoneNumber":"11164","DeviceId":"abcd1234abcd1234","CallId":"10946-15480@198.24.63.39
<mailto:10946-15480@198.24.63.39>"}] (145)
Just in case if you need to see the NGREP trace for the
above messages, it is here:
http://pastebin.com/4zTKmBJX
Thanks,
- Jayesh
On Tue, Sep 29, 2015 at 12:16 PM Daniel-Constantin Mierla
<miconda(a)gmail.com <mailto:miconda@gmail.com>> wrote:
Hello,
can you send here the log messages printed by evapi that
contain "] - received [" which have:
- the previous bunch of json documents that were
processed ok (just before the one that was discarded)
- the bunch of json documents that were discarded
- the next bunch of json documents that were processed ok
The logs were sent on could of emails and pastebin, so
it is hard to track them -- if you have the logs, just
locate the log messages with the discarded json and send
it here along with the previous and next logs.
Cheers,
Daniel
On 28/09/15 17:04, Jayesh Nambiar wrote:
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> -
http://www.linkedin.com/in/miconda
Book: SIP Routing With Kamailio - http://www.asipto.com
Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
Thank you daniel.
Yes, they all get printed by the evapi module even when
they get discarded. If you once again check the
paste-bin, the logs pasted as Kamailio Logs are all
discarded packets but show up as received by evapi module.
I believe they got discarded because a packet that came
before them was partial and evapi discarded the
remaining part of the message in the new packet and
encountered error.
- Jayesh
On Mon, Sep 28, 2015 at 11:13 AM Daniel-Constantin
Mierla <miconda(a)gmail.com <mailto:miconda@gmail.com>>
wrote:
Hello,
I will look over it during next days.
One more thing, are the packages sent by you
printed by evapi module, even not they are not
processed (but discarded)?
Cheers,
Daniel
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda
<http://twitter.com/#%21/miconda> - http://www.linkedin.com/in/miconda
Book: SIP Routing With Kamailio - http://www.asipto.com
3147
days inactive
3181
days old
26 comments
3 participants
participants (3)
-
Daniel-Constantin Mierla -
Jayesh Nambiar -
Jayesh Nambiar