<div dir="ltr"><div>Hello Daniel</div><div><br></div><div>Thanks a lot for the feedback</div><div>I tried the netstring option but since i had to reparse in the application, I finally didn't use it. <br>But I will consider it since I will test the performance for heavy load during the next days. </div><div><br></div>About the tcp_nodelay option, I remember I saw no frames with more than a message (Json object) relayed to evapi and sent out to the socket :$, but anycase, I removed that option and changed the application side.<div><br></div><div>best regards</div><div>david</div><div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">El vie, 11 dic 2020 a las 13:51, Daniel-Constantin Mierla (<<a href="mailto:miconda@gmail.com">miconda@gmail.com</a>>) escribió:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<p>Hello,</p>
<p>tcp is a streaming protocol, it is the receiving application that
has to parse and split the stream content in logical messages from
its point of view. As you noticed, there can be socket options to
enforce some sending policy (wait or not wait for a specific
amount of data to accumulate, not to send too many small packets,
but if it is too much in very short time, it is no difference).</p>
<p>As a side note, your patch below is setting the TCP_NODELAY to
listen socket, not to the socket associated with the client
connection (returned by accept()). I do not think is inherited by
accept() from listen socket, in such case practically your patch
didn't make any difference in behaviour.</p>
<p>The evapi module has the option (modparam) to serialize the
packets in netstring format, being easier to split the stream in
messages and I would recommend that mode for heavy traffic.</p>
<p>At the end, we can introduce modparams for evapi to set
TCP_NODELAY, it can be useful to reduce delays when sending short
data messages from time to time, but it won't help in your case.</p>
<p>Cheers,<br>
Daniel<br>
</p>
<div>On 11.12.20 09:39, David Escartin
wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Dear all
<div><br>
</div>
<div>seems the issue was not on the module or related to
kamailio, but related to the application we were using to read
from tcp socket.<br>
I saw that some messages sent with evapi_relay were
encapsulated in the same frame, and i even tried to force the
TCP_NODELAY option on the evapi socket by compiling the
kamailio with this</div>
--- a/src/modules/evapi/evapi_dispatch.c<br>
+++ b/src/modules/evapi/evapi_dispatch.c<br>
@@ -30,8 +30,8 @@<br>
#include <netinet/in.h><br>
#include <arpa/inet.h><br>
#include <fcntl.h><br>
-<br>
#include <ev.h><br>
+#include <netinet/tcp.h><br>
<br>
#include "../../core/sr_module.h"<br>
#include "../../core/dprint.h"<br>
@@ -690,6 +691,15 @@ int evapi_run_dispatcher(char *laddr, int
lport)<br>
freeaddrinfo(ai_res);<br>
return -1;<br>
}<br>
+ <br>
+ if(setsockopt(evapi_srv_sock, IPPROTO_TCP, TCP_NODELAY,<br>
+ &yes_true, sizeof(int)) < 0) {<br>
+ LM_INFO("cannot set TCP_NODELAY option on
descriptor\n");<br>
+ close(evapi_srv_sock);<br>
+ freeaddrinfo(ai_res);<br>
+ return -1;<br>
+ }<br>
+<br>
<br>
if (bind(evapi_srv_sock, ai_res->ai_addr,
ai_res->ai_addrlen) < 0) {<br>
<div> LM_ERR("cannot bind to local address and
port [%s:%d]\n", laddr, lport);</div>
<div><br>
</div>
<div>and i saw that with this change we had always a frame for
each message published to evapi, but the issue was still
there. <br>
So no matter if this option was activated or not in Kamailio,
I had to tune the application (in erlang) to delimit the
messages received by converting them to line mode. This way we
could reach up to 1000 processed messages per second.</div>
<div><br>
</div>
<div>best regards<br>
david</div>
<div><br>
</div>
<div> </div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">El lun, 30 nov 2020 a las
11:19, David Escartin (<<a href="mailto:descartin@sonoc.io" target="_blank">descartin@sonoc.io</a>>) escribió:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">Dear all
<div><br>
</div>
<div>we have been testing this module with the following
setup</div>
<div>kamailio 5.3.2<br>
</div>
<div>evapi params</div>
<div>modparam("evapi", "workers", 4)<br>
modparam("evapi", "netstring_format", 0)<br>
modparam("evapi", "bind_addr", "<a href="http://127.0.0.1:8448/" target="_blank">127.0.0.1:8448</a>")<br>
modparam("evapi", "max_clients", 32)<br>
</div>
<div><br>
</div>
<div>then in the configuration we do evapi_relay of avp
including a json data (which can be quite long), like this</div>
<div>{"key" : "<a href="mailto:aarp2q0tcpqhs0cpucuhukjs2ah2j00q@10.18.5.64" target="_blank">aarp2q0tcpqhs0cpucuhukjs2ah2j00q@10.18.5.64</a>"
, "msg" : {"rg_in":"701","ani_init":{"ani_source":"pai",
....... }}}<br>
</div>
<div><br>
</div>
<div>We have an application listening on the tcp socket and
writing those messages to a kafka cluster, and this works
ok, and in the previous manual tests we have done no issue
was found.<br>
But when making some load tests, and passing some live
traffic we see some issues</div>
<div><br>
</div>
<div>seems like some times, when there are messages to be
sent to the tcp socket at the same time, they are sent in
the same message, when normally each data sent using
evapi_relay is sent in 1 message</div>
<div>We do sometimes see something like this on the
application consuming from the tcp socket</div>
<div>2020-11-25 15:20:01.744 UTC [error]
<0.706.0>@evapi_kafka_listener:handle_info:167 body
"{\"key\" : \"<a href="mailto:6142651aa63616c6c04a783cd@72.21.24.130" target="_blank">6142651aa63616c6c04a783cd@72.21.24.130</a>\"
, \"msg\" :
{\"rg_in\":\"677\",\"ani_init\":{\"ani_source\":\"fro\",.......}}}{\"key\"
: \"<a href="mailto:isbc7caT4001915251VabcGhEfHdNiF0i@172.16.120.1" target="_blank">isbc7caT4001915251VabcGhEfHdNiF0i@172.16.120.1</a>\"
, \"msg\" :
{\"rg_in\":\"22\",\"ani_init\":{\"ani_source\":\"pai\",
....... ,\"translate" not valid json; error =
{691,invalid_trailing_data}<br>
2020-11-25 15:20:01.745 UTC [error]
<0.706.0>@evapi_kafka_listener:handle_info:167 body
"dPartition\":\"-1\",......}}}" not valid json; error =
{1,invalid_json}<br>
</div>
<div><br>
</div>
<div>and we do see that the application cannot parse the
json message fine, because we have like 2 json objects
together ......{\"ani_source\":\"fro\",.......}}}{\"key\"
: \"isbc7caT4001915251Vabc............</div>
<div>This happens with 2 different UDP receivers
processing messages and calling evapi_relay at the same
time. But i don't think this happens all the time. Seems
like some issue when several processes try to use evapi
workers at the same time.</div>
<div>We tried to increase evapi workers and it's the same</div>
<div><br>
</div>
<div>We also saw another issue I think. Seems when the avp
sent to evapi socket is bigger than ~1680 char, the json
is also truncated, and also happens when we use the socket
in Lo interface which has an MTU of 65535.</div>
<div><br>
</div>
<div>Could you please take a look to see if there is any
problem or limitation, or if we are using something wrong?</div>
<div><br>
</div>
<div>thanks and best regards </div>
<div>david</div>
<div><br>
</div>
-- <br>
<div dir="ltr">
<div dir="ltr">
<table style="font-family:"Times New Roman"" width="100%" cellspacing="10" cellpadding="0" border="0">
<tbody>
<tr>
<td border="0" cellpadding="0" cellspacing="0" style="border-right:3px dotted rgb(0,155,223)" width="150" valign="center" align="center"><img alt="Logo" src="https://www.sonoc.io/wp-content/uploads/2020/07/Logo_SONOC_emails.png" hspace="5" height="100"></td>
<td style="padding:0px" valign="center" align="left">
<p><span style="color:rgb(16,47,74)"><font style="font-size:14px" face="Open Sans">David
Escartín Almudévar</font></span><br>
<span style="color:rgb(16,47,74)"><font style="font-size:14px" face="Open Sans"><font style="font-size:12px" face="Open Sans">VoIP/Switch
Engineer</font></font></span><br>
<font style="font-size:14px" face="Open Sans"><font style="font-size:12px" face="Open Sans"><a href="mailto:descartin@sonoc.io" style="color:rgb(0,153,204)" target="_blank"><span style="color:rgb(0,155,223)">descartin@sonoc.io</span></a><span style="color:rgb(0,155,223)"></span></font></font></p>
<p><span style="color:rgb(16,47,74)"><font style="font-size:14px" face="Open Sans"><strong><font face="Open Sans">SONOC</font></strong></font></span><br>
<span style="color:rgb(16,47,74)"><font style="font-size:14px" face="Open Sans"><font style="font-size:12px" face="Open Sans">C/
Josefa Amar y Borbón, 10, 4ª · 50001
Zaragoza, España</font><br>
<font size="-1" face="Open Sans">Tlf: +34
917019888 ·</font></font></span><font style="font-size:14px" face="Open Sans" color="#102f4a"><font size="-1" face="Open
Sans" color="#102f4a"> </font></font><font style="font-size:14px" face="Open Sans"><font size="-1" face="Open Sans"><font size="-1" face="Open Sans"><a href="http://www.sonoc.io" style="color:rgb(0,153,204)" target="_blank"><span style="color:rgb(0,155,223)">www.sonoc.io</span></a></font></font></font></p>
</td>
</tr>
</tbody>
</table>
</div>
</div>
</div>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr">
<div dir="ltr">
<table style="font-family:"Times New Roman"" width="100%" cellspacing="10" cellpadding="0" border="0">
<tbody>
<tr>
<td border="0" cellpadding="0" cellspacing="0" style="border-right:3px dotted rgb(0,155,223)" width="150" valign="center" align="center"><img alt="Logo" src="https://www.sonoc.io/wp-content/uploads/2020/07/Logo_SONOC_emails.png" hspace="5" height="100"></td>
<td style="padding:0px" valign="center" align="left">
<p><span style="color:rgb(16,47,74)"><font style="font-size:14px" face="Open Sans">David
Escartín Almudévar</font></span><br>
<span style="color:rgb(16,47,74)"><font style="font-size:14px" face="Open Sans"><font style="font-size:12px" face="Open Sans">VoIP/Switch
Engineer</font></font></span><br>
<font style="font-size:14px" face="Open Sans"><font style="font-size:12px" face="Open Sans"><a href="mailto:descartin@sonoc.io" style="color:rgb(0,153,204)" target="_blank"><span style="color:rgb(0,155,223)">descartin@sonoc.io</span></a><span style="color:rgb(0,155,223)"></span></font></font></p>
<p><span style="color:rgb(16,47,74)"><font style="font-size:14px" face="Open Sans"><strong><font face="Open Sans">SONOC</font></strong></font></span><br>
<span style="color:rgb(16,47,74)"><font style="font-size:14px" face="Open Sans"><font style="font-size:12px" face="Open Sans">C/
Josefa Amar y Borbón, 10, 4ª · 50001 Zaragoza,
España</font><br>
<font size="-1" face="Open Sans">Tlf: +34
917019888 ·</font></font></span><font style="font-size:14px" face="Open Sans" color="#102f4a"><font size="-1" face="Open Sans" color="#102f4a"> </font></font><font style="font-size:14px" face="Open Sans"><font size="-1" face="Open Sans"><font size="-1" face="Open Sans"><a href="http://www.sonoc.io" style="color:rgb(0,153,204)" target="_blank"><span style="color:rgb(0,155,223)">www.sonoc.io</span></a></font></font></font></p>
</td>
</tr>
</tbody>
</table>
</div>
</div>
<br>
<fieldset></fieldset>
<pre>_______________________________________________
Kamailio (SER) - Development Mailing List
<a href="mailto:sr-dev@lists.kamailio.org" target="_blank">sr-dev@lists.kamailio.org</a>
<a href="https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev" target="_blank">https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev</a>
</pre>
</blockquote>
<pre cols="72">--
Daniel-Constantin Mierla -- <a href="http://www.asipto.com" target="_blank">www.asipto.com</a>
<a href="http://www.twitter.com/miconda" target="_blank">www.twitter.com/miconda</a> -- <a href="http://www.linkedin.com/in/miconda" target="_blank">www.linkedin.com/in/miconda</a>
Funding: <a href="https://www.paypal.me/dcmierla" target="_blank">https://www.paypal.me/dcmierla</a></pre>
</div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><table border="0" cellpadding="0" cellspacing="10" width="100%" style="font-family:"Times New Roman""><tbody><tr><td align="center" border="0" cellpadding="0" cellspacing="0" valign="center" width="150" style="border-right:3px dotted rgb(0,155,223)"><img alt="Logo" height="100" hspace="5" src="https://www.sonoc.io/wp-content/uploads/2020/07/Logo_SONOC_emails.png"></td><td align="left" valign="center" style="padding:0px"><p><span style="color:rgb(16,47,74)"><font face="Open Sans" style="font-size:14px">David Escartín Almudévar</font></span><br><span style="color:rgb(16,47,74)"><font face="Open Sans" style="font-size:14px"><font face="Open Sans" style="font-size:12px">VoIP/Switch Engineer</font></font></span><font face="Open Sans" style="font-size:14px"><font face="Open Sans" style="font-size:12px"></font></font><br><font face="Open Sans" style="font-size:14px"><font face="Open Sans" style="font-size:12px"><a href="mailto:descartin@sonoc.io" style="color:rgb(0,153,204)" target="_blank"><span style="color:rgb(0,155,223)">descartin@sonoc.io</span></a><span style="color:rgb(0,155,223)"></span></font></font></p><p><span style="color:rgb(16,47,74)"><font face="Open Sans" style="font-size:14px"><strong><font face="Open Sans">SONOC</font></strong></font></span><br><span style="color:rgb(16,47,74)"><font face="Open Sans" style="font-size:14px"><font face="Open Sans" style="font-size:12px">C/ Josefa Amar y Borbón, 10, 4ª · 50001 Zaragoza, España</font><br><font face="Open Sans" size="-1">Tlf: +34 917019888 ·</font></font></span><font color="#102f4a" face="Open Sans" style="font-size:14px"><font color="#102f4a" face="Open Sans" size="-1"> </font></font><font face="Open Sans" style="font-size:14px"><font face="Open Sans" size="-1"><font face="Open Sans" size="-1"><a href="http://www.sonoc.io" style="color:rgb(0,153,204)" target="_blank"><span style="color:rgb(0,155,223)">www.sonoc.io</span></a></font></font></font></p></td></tr></tbody></table></div></div></div></div>