<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p>Hello,</p>
<p>thanks for the feedback! It is good to know that it works well so
far for you. I don't see any reason not to make the library to
preload as part of the next release.</p>
<p>Just to let everyone know, for now, the built packages are pinned
to link against libssl 1.0.x.</p>
<p>Soon, I will approach the openssl project in order to find a
proper solution for long term.</p>
<p>Cheers,<br>
Daniel<br>
</p>
<div class="moz-cite-prefix">On 13.05.19 10:48, Floimair Florian
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:036000A8-5AAD-427E-ADB9-9E8976944C0F@commend.com">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<meta name="Generator" content="Microsoft Word 15 (filtered
medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]-->
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Vorformatiert Zchn";
margin:0cm;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.HTMLVorformatiertZchn
{mso-style-name:"HTML Vorformatiert Zchn";
mso-style-priority:99;
mso-style-link:"HTML Vorformatiert";
font-family:"Consolas",serif;}
span.E-MailFormatvorlage21
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:70.85pt 70.85pt 2.0cm 70.85pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
<div class="WordSection1">
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"
lang="EN-US">Hi all!<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"
lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"
lang="EN-US">We have used the work-around with the
pre-loaded library and so far this seems to have fixed our
problem (that my colleague Kristijan Vrban reported).<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"
lang="EN-US">At least we did not have a single failure
within the last week, whereas before the issue happened
about once every 2 days.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"
lang="EN-US">Would be nice if this would be part of the next
Kamailio version.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"
lang="EN-US"><o:p> </o:p></span></p>
<div>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="font-size:12.0pt;font-family:"Times New
Roman",serif" lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="font-size:12.0pt;font-family:"Times New
Roman",serif" lang="EN-US"> <o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black"
lang="EN-US">With best regards<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black"
lang="EN-US"><br>
<b>Florian Floimair<br>
</b>Innovation - Software-Development<br>
<br>
<b>COMMEND INTERNATIONAL GMBH<br>
</b>A-5020 Salzburg, Saalachstraße 51<br>
</span><span lang="DE-AT"><a href="http://www.commend.com/"
moz-do-not-send="true"><span
style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#0563C1"
lang="EN-US">http://www.commend.com</span></a></span><span
style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black"
lang="EN-US"><br>
<br>
<b>Security and Communication by Commend<br>
<br>
</b></span><span
style="font-size:8.0pt;font-family:"Arial",sans-serif;color:gray"
lang="EN-US">FN 178618z | LG Salzburg</span><span
style="mso-fareast-language:EN-US" lang="EN-US"><o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"
lang="EN-US"><o:p> </o:p></span></p>
<div style="border:none;border-top:solid #B5C4DF
1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span
style="font-size:12.0pt;color:black">Von: </span></b><span
style="font-size:12.0pt;color:black">sr-users
<a class="moz-txt-link-rfc2396E" href="mailto:sr-users-bounces@lists.kamailio.org"><sr-users-bounces@lists.kamailio.org></a> im Auftrag von
Daniel-Constantin Mierla <a class="moz-txt-link-rfc2396E" href="mailto:miconda@gmail.com"><miconda@gmail.com></a><br>
<b>Antworten an: </b><a class="moz-txt-link-rfc2396E" href="mailto:miconda@gmail.com">"miconda@gmail.com"</a>
<a class="moz-txt-link-rfc2396E" href="mailto:miconda@gmail.com"><miconda@gmail.com></a>, "Kamailio (SER) - Users Mailing
List" <a class="moz-txt-link-rfc2396E" href="mailto:sr-users@lists.kamailio.org"><sr-users@lists.kamailio.org></a><br>
<b>Datum: </b>Montag, 15. April 2019 um 09:07<br>
<b>An: </b>Aymeric Moizard <a class="moz-txt-link-rfc2396E" href="mailto:amoizard@gmail.com"><amoizard@gmail.com></a>,
"Kamailio (SER) - Users Mailing List"
<a class="moz-txt-link-rfc2396E" href="mailto:sr-users@lists.kamailio.org"><sr-users@lists.kamailio.org></a><br>
<b>Betreff: </b>Re: [SR-Users] Kamailio stop to process
incoming SIP traffic via TCP.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p>Hello Aymeric,<o:p></o:p></p>
<p>would you be able to test with tls module compiled against
libssl 1.1 and using the pre-loaded shared object workaround?<o:p></o:p></p>
<p> * <a
href="https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fkamailio%2Fkamailio%2Ftree%2Fmaster%2Fsrc%2Fmodules%2Ftls%2Futils%2Fopenssl_mutex_shared&data=02%7C01%7Cf.floimair%40commend.com%7C4008d49af1b347abe20308d6c1710532%7C13b1ddb756454e7fbe663171548559da%7C0%7C0%7C636909088526480174&sdata=d9E%2Fy4cvdLkGCPUexoCJ7tws3QL4rFqz5ebnMGnsESQ%3D&reserved=0"
moz-do-not-send="true">
https://github.com/kamailio/kamailio/tree/master/src/modules/tls/utils/openssl_mutex_shared</a><o:p></o:p></p>
<p>You should be able to use it with any version, no need to
test with kamailio master branch.<o:p></o:p></p>
<p>Just clone the master branch, then:<o:p></o:p></p>
<p>cd src/modules/tls/utils/openssl_mutex_shared<o:p></o:p></p>
<p>make<o:p></o:p></p>
<p>Either from there or copy openssl_mutex_shared.so to a
location you want, then pre-load it before starting your
version of Kamailio.<o:p></o:p></p>
<p>The README.md in the folder has some more details.<o:p></o:p></p>
<p>I would like to have some validation that it works fine
before approaching this topic with libssl project to allow to
init the locks with shared process option.<o:p></o:p></p>
<p>Thanks,<br>
Daniel<o:p></o:p></p>
<div>
<p class="MsoNormal">On 26.03.19 16:18, Daniel-Constantin
Mierla wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p>Hello,<o:p></o:p></p>
<p>yep, locking there is expected, as listing the tls
connections wait for no other processes to change the
content of internal tls connection structures. So it is a
side effect of libssl/libcrypto getting stuck and the other
processing waiting for it to move one. I have the Kamailio
training in USA these days, so the trip and schedule of the
day didn't allow me to look more at the libsll/libcrypto
code in order to find a solution here. It is a high priority
in my list, as I get time during the next days.<o:p></o:p></p>
<p>Cheers,<br>
Daniel<o:p></o:p></p>
<div>
<p class="MsoNormal">On 26.03.19 15:55, Aymeric Moizard
wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal">Hi All, <o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">I was debugging a TCP
issue (most probably, I may start a thread
for this question).<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">I was trying to get
some info for TCP and TLS.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">I typed:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">$> sudo kamctl rpc
tls.list<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">And waited for a
while.... until... I realized that my
User-Agent, connected with TCP was not
able to register any more. I think the rpc
command has introduced something wrong.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">The device can
successfully "connect", send the REGISTER
over the established TCP connection. The
REGISTER do not appear in the logs any
more, I don't see any traffic for TCP any
more. So the behavior is the same as I had
before: TCP and TLS are both not working
and UDP is still working fine.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">kamctl do not work any
more... so kamctl trap do not work...<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">I have been able to
type.. manually... for (all?) kamailio
threads:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">gdb /usr/sbin/kamailio
16500 -batch --eval-command="bt full"
>> kamailio-trap-tcp-down.txt<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">I'm temporarly puting
the backtrace I have here:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><a
href="https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fsip.antisip.com%2Fkamailio-trap-tcp-down.txt&data=02%7C01%7Cf.floimair%40commend.com%7C4008d49af1b347abe20308d6c1710532%7C13b1ddb756454e7fbe663171548559da%7C0%7C0%7C636909088526490178&sdata=1lfFxvR0m4PVcfnYsrrIO%2FM2nbGK6zfpl2C01O2c7M0%3D&reserved=0"
moz-do-not-send="true">https://sip.antisip.com/kamailio-trap-tcp-down.txt</a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">You can see a thread
stuck on the json command line: "<span
style="color:black">tls_list"</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span
style="color:black">And many other
waiting on CRYPTO_THREAD_write_lock</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span
style="color:black">? might be related
to: <a
href="https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fopenssl%2Fopenssl%2Fissues%2F5376&data=02%7C01%7Cf.floimair%40commend.com%7C4008d49af1b347abe20308d6c1710532%7C13b1ddb756454e7fbe663171548559da%7C0%7C0%7C636909088526490178&sdata=D5Fb4U3trdbRUY7ifMLSc5KE4mAxjK2%2BzOy8nSD1Rks%3D&reserved=0"
moz-do-not-send="true">
https://github.com/openssl/openssl/issues/5376</a></span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">SIDE NOTE:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Right before I was
typing the last gdb command for the last
thread, kamailio<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">has crashed: This was
around 5 minutes after the dead lock
started.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<div>
<p class="MsoNormal">Mar 26 14:47:11 sip
kamailio[16493]: ERROR: <core>
[core/tcp_main.c:2561]:
tcpconn_do_send(): failed to send on
0x7ff8dfc2fdc8 (91.121.30.149:5061-><a
href="https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2F62.210.97.21%3A49351&data=02%7C01%7Cf.floimair%40commend.com%7C4008d49af1b347abe20308d6c1710532%7C13b1ddb756454e7fbe663171548559da%7C0%7C0%7C636909088526500195&sdata=9XqEUKoMwNEvCPFtKfvB0c43yk1GcSzYOiPdY9Pj1uo%3D&reserved=0"
moz-do-not-send="true">62.210.97.21:49351</a>):
Broken pipe (32)<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Mar 26 14:47:11 sip
kamailio[16493]: ERROR: <core>
[core/tcp_read.c:1505]: tcp_read_req():
ERROR: tcp_read_req: error reading - c:
0x7ff8dfc2fdc8 r: 0x7ff8dfc2fe48 (-1)<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Mar 26 14:47:11 sip
kamailio[16493]: WARNING: <core>
[core/tcp_read.c:1848]: handle_io():
F_TCPCONN connection marked as bad:
0x7ff8dfa6a408 id 846 refcnt 3<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Mar 26 14:47:11 sip
kamailio[16371]: ALERT: <core>
[main.c:755]: handle_sigs(): child
process 16374 exited by a signal 11<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Mar 26 14:47:11 sip
kamailio[16371]: ALERT: <core>
[main.c:758]: handle_sigs(): core was
not generated<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Mar 26 14:47:11 sip
kamailio[16371]: INFO: <core>
[main.c:781]: handle_sigs(): terminating
due to SIGCHLD<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Mar 26 14:47:11 sip
kamailio[16493]: INFO: <core>
[main.c:836]: sig_usr(): signal 15
received<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Mar 26 14:47:11 sip
kamailio[16500]: INFO: <core>
[main.c:836]: sig_usr(): signal 15
received<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Mar 26 14:47:11 sip
kamailio[16479]: INFO: <core>
[main.c:836]: sig_usr(): signal 15
received<o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Unfortunalty, even if I
did my best to setup my service to
generate a core on crash, I still have
"core was not generated".... (debian
stretch)<o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">Tks for reading!<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Regards<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Aymeric<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">Le mar. 26 mars 2019 à 14:11,
Kristijan Vrban <<a
href="mailto:vrban.lkml@gmail.com"
moz-do-not-send="true">vrban.lkml@gmail.com</a>>
a écrit :<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC
1.0pt;padding:0cm 0cm 0cm
6.0pt;margin-left:4.8pt;margin-right:0cm">
<p class="MsoNormal">And again one more kamctl trap file
where<br>
<br>
set_reply_no_connect was set.<br>
<br>
Am Di., 26. März 2019 um 08:53 Uhr schrieb Kristijan
Vrban<br>
<<a href="mailto:vrban.lkml@gmail.com"
target="_blank" moz-do-not-send="true">vrban.lkml@gmail.com</a>>:<br>
><br>
> Attached also the output of kamctl trap<br>
><br>
> Am Di., 26. März 2019 um 08:42 Uhr schrieb
Kristijan Vrban<br>
> <<a href="mailto:vrban.lkml@gmail.com"
target="_blank" moz-do-not-send="true">vrban.lkml@gmail.com</a>>:<br>
> ><br>
> > > Have you done a test with tools such as
sipp, or was this happening<br>
> > > after a while, with usual phones
registering?<br>
> ><br>
> > Usual variety of devices registering via
TLS. But i can not exclude<br>
> > that some devices displaying behavioural
problems.<br>
> ><br>
> > > Can you list the tcp connections and
see if they are listed?<br>
> > > kamctl tcp core.tcp_list<br>
> ><br>
> > Need Kex module for that? So i can deliver
next time. But when i do<br>
> > "lsof -u kamailio |grep TCP"<br>
> > i get a long list of more then 2000 lines
with:<br>
> ><br>
> > ...<br>
> > kamailio 37561 kamailio 2105u sock
0,9 0t0<br>
> > 27856287 protocol: TCP<br>
> > kamailio 37561 kamailio 2106u sock
0,9 0t0<br>
> > 27856305 protocol: TCP<br>
> > kamailio 37561 kamailio 2107u sock
0,9 0t0<br>
> > 27856306 protocol: TCP<br>
> > kamailio 37561 kamailio 2108u sock
0,9 0t0<br>
> > 27856914 protocol: TCP<br>
> > ...<br>
> ><br>
> > So about the time Kamailio created a lot of
socket in the TCP domain,<br>
> > but which are not bound to any port (eg via
connect(2) or listen(2) or<br>
> > bind(2))<br>
> > Until we get to the maximum number of 2048
connections.<br>
> ><br>
> > Best<br>
> > Kristijan<br>
> ><br>
> > Am Mo., 25. März 2019 um 14:27 Uhr schrieb
Daniel-Constantin Mierla<br>
> > <<a href="mailto:miconda@gmail.com"
target="_blank" moz-do-not-send="true">miconda@gmail.com</a>>:<br>
> > ><br>
> > > Have you done a test with tools such as
sipp, or was this happening<br>
> > > after a while, with usual phones
registering?<br>
> > ><br>
> > > Can you list the tcp connections and
see if they are listed?<br>
> > ><br>
> > > kamctl tcp core.tcp_list<br>
> > ><br>
> > > Cheers,<br>
> > > Daniel<br>
> > ><br>
> > > On 25.03.19 08:03, Kristijan Vrban
wrote:<br>
> > > >> The solution here is to use
set_reply_no_connect()<br>
> > > > implemented it. Now the issue has
shifted to:<br>
> > > ><br>
> > > > ERROR: <core>
[core/tcp_main.c:3959]: handle_new_connect(): maximum<br>
> > > > number of connections exceeded:
2048/2048<br>
> > > ><br>
> > > > But not a single TCP connection is
active between Kamailio and any<br>
> > > > device. Seems this counter for
maximum number of connections<br>
> > > > now has an issue?<br>
> > > ><br>
> > > > Kristijan<br>
> > > ><br>
> > > > Am Mi., 20. März 2019 um 15:07 Uhr
schrieb Daniel-Constantin Mierla<br>
> > > > <<a
href="mailto:miconda@gmail.com" target="_blank"
moz-do-not-send="true">miconda@gmail.com</a>>:<br>
> > > >> Hello,<br>
> > > >><br>
> > > >> based on the trap output I
think I could figure out what happened there.<br>
> > > >><br>
> > > >> You have tcp_children to very
low value (1 or so), the problem is not<br>
> > > >> actually that one, but the
fact that the connection to upstream (the<br>
> > > >> device/app sending the
request) was closed after receiving the request<br>
> > > >> and routing of the reply gets
stuck in the way of:<br>
> > > >><br>
> > > >> - a reply is received and
has to be forwarded<br>
> > > >> - connection was lost, so
Kamailio tries to establish a new one, but<br>
> > > >> takes time till fails because
the upstream is behind nat or so based on<br>
> > > >> the via header:<br>
> > > >><br>
> > > >> Via: SIP/2.0/TLS<br>
> > > >>
10.1.0.4:10002;rport=55229;received=13.94.188.218;branch=z9hG4bK-3336-7f2927bfd703ae907348edff3611bfc9<br>
> > > >><br>
> > > >> - the reply is retransmitted
and gets to another worker, which tries<br>
> > > >> to forward it again, but
discovers a connection structure for that<br>
> > > >> destination exists (created by
previous reply worker) and now waits for<br>
> > > >> the connection to be released
(or better said, for the mutex on writing<br>
> > > >> buffer to be unlocked)<br>
> > > >><br>
> > > >> - as the second reply waits,
there can be other retransmissions of the<br>
> > > >> reply ending up in other
workers stuck on waiting for the mutex of the<br>
> > > >> connection write buffer<br>
> > > >><br>
> > > >> The solution here is to use
set_reply_no_connect() -- you can put it<br>
> > > >> first in request_route block.
I think this would be a good addition to<br>
> > > >> the default configuration file
as well, IMO, the sip server should not<br>
> > > >> connect for sending replies
and should do it also for requests that go<br>
> > > >> behind nat.<br>
> > > >><br>
> > > >> Cheers,<br>
> > > >> Daniel<br>
> > > >><br>
> > > >> On 19.03.19 10:53, Kristijan
Vrban wrote:<br>
> > > >>> So i had again the
situation. But this time, incoming udp was<br>
> > > >>> affected. Kamailio was
sending out OPTIONS (via dispatcher module) to<br>
> > > >>> a group of asterisk
machines<br>
> > > >>> but the 200 OK reply to
the OPTIONS where not processed, so the<br>
> > > >>> dispatcher module set all
asterisk to inactive, even though they<br>
> > > >>> replied 200 OK<br>
> > > >>><br>
> > > >>> Attached the output of
kamctl trap during the situation. Hope there is<br>
> > > >>> any useful in it. Because
after "kamctl trap" it was working again<br>
> > > >>> without kamailio restart.<br>
> > > >>><br>
> > > >>> Best<br>
> > > >>> Kristijan<br>
> > > >>><br>
> > > >>> Am Mo., 18. März 2019 um
12:27 Uhr schrieb Daniel-Constantin Mierla<br>
> > > >>> <<a
href="mailto:miconda@gmail.com" target="_blank"
moz-do-not-send="true">miconda@gmail.com</a>>:<br>
> > > >>>> Hello,<br>
> > > >>>><br>
> > > >>>> setting tcp_children=1
is not a god option for scallability, practically<br>
> > > >>>> you set kamailio to
process a single tcp message at one time, on high<br>
> > > >>>> traffic, that won't
work well.<br>
> > > >>>><br>
> > > >>>> Maybe try to set
tcp_children to 2 or 4, that should make an eventual<br>
> > > >>>> race appear faster.<br>
> > > >>>><br>
> > > >>>> Regarding the pid, if
it is an outgoing connection, then it can be<br>
> > > >>>> created by any worker
process, including a UDP worker, if that was the<br>
> > > >>>> one receiving the sip
message over udp and sends it out via tcp.<br>
> > > >>>><br>
> > > >>>> Cheers,<br>
> > > >>>> Daniel<br>
> > > >>>><br>
> > > >>>> On 18.03.19 10:09,
Kristijan Vrban wrote:<br>
> > > >>>>> Hi Daniel,<br>
> > > >>>>><br>
> > > >>>>> for testing, i now
had set: "tcp_children=1" and so far this issue did
not occur<br>
> > > >>>>> ever since. So now
value to provide for "kamctl trap" yet.<br>
> > > >>>>><br>
> > > >>>>> "kamctl ps" show
this two process to handle tcp:<br>
> > > >>>>><br>
> > > >>>>> ...<br>
> > > >>>>> }, {<br>
> > > >>>>> "IDX": 25,<br>
> > > >>>>> "PID":
71929,<br>
> > > >>>>> "DSC": "tcp
receiver (generic) child=0"<br>
> > > >>>>> }, {<br>
> > > >>>>> "IDX": 26,<br>
> > > >>>>> "PID":
71933,<br>
> > > >>>>> "DSC": "tcp
main process"<br>
> > > >>>>> }<br>
> > > >>>>> ...<br>
> > > >>>>><br>
> > > >>>>><br>
> > > >>>>> Ok, but then is
was wondering to see a TCP connection on a udp
receiver child:<br>
> > > >>>>><br>
> > > >>>>><br>
> > > >>>>> netstat -ntp |grep
5061<br>
> > > >>>>><br>
> > > >>>>> ...<br>
> > > >>>>> tcp 0
0 <a
href="https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2F172.17.217.10%3A5061&data=02%7C01%7Cf.floimair%40commend.com%7C4008d49af1b347abe20308d6c1710532%7C13b1ddb756454e7fbe663171548559da%7C0%7C0%7C636909088526500195&sdata=wO8Jbw676nbMafjlenNGCrZgNEhZvye4Go7UWk1umF8%3D&reserved=0"
target="_blank" moz-do-not-send="true">
172.17.217.10:5061</a> <a
href="https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2F195.70.114.125%3A18252&data=02%7C01%7Cf.floimair%40commend.com%7C4008d49af1b347abe20308d6c1710532%7C13b1ddb756454e7fbe663171548559da%7C0%7C0%7C636909088526510200&sdata=z%2FJm5HNDlL6xhlui1%2FBcMkvSQDAbp%2FxkM3KY9sAUX0s%3D&reserved=0"
target="_blank" moz-do-not-send="true">
195.70.114.125:18252</a><br>
> > > >>>>> ESTABLISHED
71895/kamailio<br>
> > > >>>>> ...<br>
> > > >>>>><br>
> > > >>>>> An pid 71895 is:<br>
> > > >>>>><br>
> > > >>>>> }, {<br>
> > > >>>>> "IDX": 3,<br>
> > > >>>>> "PID":
71895,<br>
> > > >>>>> "DSC": "udp
receiver child=2 sock=<a
href="https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2F127.0.0.1%3A5060&data=02%7C01%7Cf.floimair%40commend.com%7C4008d49af1b347abe20308d6c1710532%7C13b1ddb756454e7fbe663171548559da%7C0%7C0%7C636909088526520221&sdata=8r55f9MZ2gaw%2B2MA1LY1IfbnWkZDLHdr%2FSwRu7hwnvQ%3D&reserved=0"
target="_blank" moz-do-not-send="true">127.0.0.1:5060</a>"<br>
> > > >>>>> }, {<br>
> > > >>>>><br>
> > > >>>>><br>
> > > >>>>><br>
> > > >>>>> And if i look into
it via "lsof -p 71895" (the udp receiver child)<br>
> > > >>>>><br>
> > > >>>>> ...<br>
> > > >>>>> kamailio 71895
kamailio 14u sock 0,9 0t0<br>
> > > >>>>> 8856085 protocol:
TCP<br>
> > > >>>>> kamailio 71895
kamailio 15u sock 0,9 0t0<br>
> > > >>>>> 8886886 protocol:
TCP<br>
> > > >>>>> kamailio 71895
kamailio 16u sock 0,9 0t0<br>
> > > >>>>> 8854886 protocol:
TCP<br>
> > > >>>>> kamailio 71895
kamailio 17u sock 0,9 0t0<br>
> > > >>>>> 8828915 protocol:
TCP<br>
> > > >>>>> kamailio 71895
kamailio 18u unix 0x000000005f73cb91 0t0<br>
> > > >>>>> 1680314 type=DGRAM<br>
> > > >>>>> kamailio 71895
kamailio 19u IPv4 1846523 0t0<br>
> > > >>>>> TCP
kamailio-preview:sip-tls->XXX:18252 (ESTABLISHED)<br>
> > > >>>>> kamailio 71895
kamailio 20u sock 0,9 0t0<br>
> > > >>>>> 8887192 protocol:
TCP<br>
> > > >>>>> kamailio 71895
kamailio 21u sock 0,9 0t0<br>
> > > >>>>> 8813634 protocol:
TCP<br>
> > > >>>>> kamailio 71895
kamailio 22u unix 0x00000000c19bd102 0t0<br>
> > > >>>>> 1681407
type=STREAM<br>
> > > >>>>> kamailio 71895
kamailio 23u sock 0,9 0t0<br>
> > > >>>>> 8850488 protocol:
TCP<br>
> > > >>>>> ...<br>
> > > >>>>><br>
> > > >>>>> Not only the
ESTABLISHED TCP session. But also this empty sockets<br>
> > > >>>>> "protocol: TCP"<br>
> > > >>>>> What are they
doing there in the udp receiver? Is that how it's
supposed to be?<br>
> > > >>>>><br>
> > > >>>>> Kristijan<br>
> > > >>>>><br>
> > > >>>>> Am Do., 14. März
2019 um 14:48 Uhr schrieb Daniel-Constantin Mierla<br>
> > > >>>>> <<a
href="mailto:miconda@gmail.com" target="_blank"
moz-do-not-send="true">miconda@gmail.com</a>>:<br>
> > > >>>>>> Can you get
file written by `kamctl trap`? It should have the
backtrace<br>
> > > >>>>>> for all
kamailio processes. You need latest kamailio 5.2.<br>
> > > >>>>>><br>
> > > >>>>>> Also, get the
output for: kamctl ps<br>
> > > >>>>>><br>
> > > >>>>>> Cheers,<br>
> > > >>>>>> Daniel<br>
> > > >>>>>><br>
> > > >>>>>> On 14.03.19
13:52, Kristijan Vrban wrote:<br>
> > > >>>>>>> When i
attach via gdb to one of the tcp worker, i see this:<br>
> > > >>>>>>><br>
> > > >>>>>>> (gdb) bt<br>
> > > >>>>>>> #0
0x00007fdaf4d14470 in futex_wait
(private=<optimized out>,<br>
> > > >>>>>>>
expected=1, futex_word=0x7fdaeca92f8c) at<br>
> > > >>>>>>>
../sysdeps/unix/sysv/linux/futex-internal.h:61<br>
> > > >>>>>>> #1
futex_wait_simple (private=<optimized out>,
expected=1,<br>
> > > >>>>>>>
futex_word=0x7fdaeca92f8c) at
../sysdeps/nptl/futex-internal.h:135<br>
> > > >>>>>>> #2
__pthread_rwlock_wrlock_slow (rwlock=0x7fdaeca92f80)
at<br>
> > > >>>>>>>
pthread_rwlock_wrlock.c:67<br>
> > > >>>>>>> #3
0x00007fdaf0912ee9 in CRYPTO_THREAD_write_lock () from<br>
> > > >>>>>>>
/usr/lib/x86_64-linux-gnu/libcrypto.so.1.1<br>
> > > >>>>>>> #4
0x00007fdaf08e1c08 in ?? () from
/usr/lib/x86_64-linux-gnu/libcrypto.so.1.1<br>
> > > >>>>>>> #5
0x00007fdaf08a6f69 in ?? () from
/usr/lib/x86_64-linux-gnu/libcrypto.so.1.1<br>
> > > >>>>>>> #6
0x00007fdaf08b36c7 in EVP_CIPHER_CTX_ctrl () from<br>
> > > >>>>>>>
/usr/lib/x86_64-linux-gnu/libcrypto.so.1.1<br>
> > > >>>>>>> #7
0x00007fdaf0c31144 in ?? () from
/usr/lib/x86_64-linux-gnu/libssl.so.1.1<br>
> > > >>>>>>> #8
0x00007fdaf0c2bddb in ?? () from
/usr/lib/x86_64-linux-gnu/libssl.so.1.1<br>
> > > >>>>>>> #9
0x00007fdaf0c22858 in ?? () from
/usr/lib/x86_64-linux-gnu/libssl.so.1.1<br>
> > > >>>>>>> #10
0x00007fdaf0c1af61 in SSL_do_handshake () from<br>
> > > >>>>>>>
/usr/lib/x86_64-linux-gnu/libssl.so.1.1<br>
> > > >>>>>>> #11
0x00007fdaf0e8d31b in tls_accept (c=0x7fdaed26fa98,<br>
> > > >>>>>>>
error=0x7ffffe2a2df0) at tls_server.c:422<br>
> > > >>>>>>> #12
0x00007fdaf0e96a1b in tls_read_f (c=0x7fdaed26fa98,<br>
> > > >>>>>>>
flags=0x7ffffe2c318c) at tls_server.c:1116<br>
> > > >>>>>>> #13
0x0000556ead5e7c46 in tcp_read_headers
(c=0x7fdaed26fa98,<br>
> > > >>>>>>>
read_flags=0x7ffffe2c318c) at core/tcp_read.c:469<br>
> > > >>>>>>> #14
0x0000556ead5ef9cb in tcp_read_req
(con=0x7fdaed26fa98,<br>
> > > >>>>>>>
bytes_read=0x7ffffe2c3184, read_flags=0x7ffffe2c318c)
at<br>
> > > >>>>>>>
core/tcp_read.c:1496<br>
> > > >>>>>>> #15
0x0000556ead5f575f in handle_io (fm=0x7fdaf597aa98,
events=1,<br>
> > > >>>>>>> idx=-1) at
core/tcp_read.c:1862<br>
> > > >>>>>>> #16
0x0000556ead5e2053 in io_wait_loop_epoll
(h=0x556eadaaeec0 <io_w>,<br>
> > > >>>>>>> t=2,
repeat=0) at core/io_wait.h:1065<br>
> > > >>>>>>> #17
0x0000556ead5f6b35 in tcp_receive_loop (unix_sock=49)
at<br>
> > > >>>>>>>
core/tcp_read.c:1974<br>
> > > >>>>>>> #18
0x0000556ead4c8e24 in tcp_init_children () at
core/tcp_main.c:4853<br>
> > > >>>>>>> #19
0x0000556ead3c352a in main_loop () at main.c:1735<br>
> > > >>>>>>> #20
0x0000556ead3ca5f8 in main (argc=13,
argv=0x7ffffe2c3828) at main.c:2675<br>
> > > >>>>>>><br>
> > > >>>>>>><br>
> > > >>>>>>><br>
> > > >>>>>>><br>
> > > >>>>>>><br>
> > > >>>>>>><br>
> > > >>>>>>><br>
> > > >>>>>>> Am Do.,
14. März 2019 um 13:41 Uhr schrieb Kristijan Vrban<br>
> > > >>>>>>> <<a
href="mailto:vrban.lkml@gmail.com" target="_blank"
moz-do-not-send="true">vrban.lkml@gmail.com</a>>:<br>
> > > >>>>>>>> Hi,
with full debug is see this in log for every incoming
TCP SIP request:<br>
> > > >>>>>>>><br>
> > > >>>>>>>> Mar 14
12:10:15 kamailio-preview /usr/sbin/kamailio[17940]:
DEBUG:<br>
> > > >>>>>>>>
<core> [core/tcp_main.c:3871]: send2child():
WARNING: no free tcp<br>
> > > >>>>>>>>
receiver, connection passed to the least busy one
(105)<br>
> > > >>>>>>>> Mar 14
12:10:15 kamailio-preview /usr/sbin/kamailio[17940]:
DEBUG:<br>
> > > >>>>>>>>
<core> [core/tcp_main.c:3875]: send2child():
selected tcp worker 2<br>
> > > >>>>>>>>
27(17937) for activity on [tls:<a
href="https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2F172.17.217.10%3A5061&data=02%7C01%7Cf.floimair%40commend.com%7C4008d49af1b347abe20308d6c1710532%7C13b1ddb756454e7fbe663171548559da%7C0%7C0%7C636909088526520221&sdata=ad7Txe5RLrnj0CMbQZORXvAdU0NpbrCjP5RzNrbxJdU%3D&reserved=0"
target="_blank" moz-do-not-send="true">172.17.217.10:5061</a>],
0x7fdaeda8f928<br>
> > > >>>>>>>><br>
> > > >>>>>>>> So the
Kamailio TCP process is working, and received TCP
traffic. But<br>
> > > >>>>>>>> the
tcp workers are somehow busy.<br>
> > > >>>>>>>><br>
> > > >>>>>>>> When i
attach via strace to the TCP worker, i do not see any
activity. Just:<br>
> > > >>>>>>>><br>
> > > >>>>>>>>
futex(0x7fdaeca92f8c, FUTEX_WAIT_PRIVATE, 1, NULL<br>
> > > >>>>>>>><br>
> > > >>>>>>>> and
nothing, even when i see the main tcp process choose
this worker process.<br>
> > > >>>>>>>><br>
> > > >>>>>>>>
Kristijan<br>
> > > >>>>>>>><br>
> > > >>>>>>>> Am
Mi., 27. Feb. 2019 um 15:14 Uhr schrieb Kristijan
Vrban<br>
> > > >>>>>>>> <<a
href="mailto:vrban.lkml@gmail.com" target="_blank"
moz-do-not-send="true">vrban.lkml@gmail.com</a>>:<br>
> > > >>>>>>>>>
first of all thanks for the feedback. i prepared our
system now to run<br>
> > > >>>>>>>>>
with debug=3<br>
> > > >>>>>>>>> I
hope to see more then then.<br>
> > > >>>>>>>>><br>
> > > >>>>>>>>> Am
Mi., 27. Feb. 2019 um 11:53 Uhr schrieb Kristijan
Vrban<br>
> > > >>>>>>>>>
<<a href="mailto:vrban.lkml@gmail.com"
target="_blank" moz-do-not-send="true">vrban.lkml@gmail.com</a>>:<br>
> > >
>>>>>>>>>> Hi kamailios,<br>
> > >
>>>>>>>>>><br>
> > >
>>>>>>>>>> i have a
creepy situation with v5.2.1 stable Kamilio. After a
day or<br>
> > >
>>>>>>>>>> so, Kamailio
stop to process incoming SIP traffic via TCP. The<br>
> > >
>>>>>>>>>> incoming TCP
network packages get TCP-ACK from the OS (Debian 9,<br>
> > >
>>>>>>>>>>
4.18.0-15-generic-Linux) but Kamailio does not show
any processing for<br>
> > >
>>>>>>>>>> the
SIP-Traffic incoming via TCP. No logs, nothing. While
traffic via<br>
> > >
>>>>>>>>>> UDP is
working just totally fine.<br>
> > >
>>>>>>>>>><br>
> > >
>>>>>>>>>> When i look
via command "netstat -ntp" is see, that the Recv-Q get<br>
> > >
>>>>>>>>>> bigger and
bigger. e.g.:<br>
> > >
>>>>>>>>>><br>
> > >
>>>>>>>>>> Proto Recv-Q
Send-Q Local Address Foreign Address State PID/Program<br>
> > >
>>>>>>>>>> name tcp 4566
0 <a
href="https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2F172.17.217.12%3A5060&data=02%7C01%7Cf.floimair%40commend.com%7C4008d49af1b347abe20308d6c1710532%7C13b1ddb756454e7fbe663171548559da%7C0%7C0%7C636909088526530221&sdata=TlooUxH53u7tlp54rf3FIyKisAusK00CtjbPKUpXQy8%3D&reserved=0"
target="_blank" moz-do-not-send="true">
172.17.217.12:5060</a> xxx.xxx.xxx.xxx:57252
ESTABLISHED<br>
> > >
>>>>>>>>>>
31347/kamailio<br>
> > >
>>>>>>>>>><br>
> > >
>>>>>>>>>> After
Kamailio restart, all is working fine again for a day.
We have<br>
> > >
>>>>>>>>>> maybe 10-20
devices online via TCP and low call volume (1-2 call
per<br>
> > >
>>>>>>>>>> minute). The
only settings for tcp we have is "tcp_delayed_ack=no"<br>
> > >
>>>>>>>>>><br>
> > >
>>>>>>>>>> How to could
we debug this situation? Again, no error, no warings
in<br>
> > >
>>>>>>>>>> the log. Just
nothing.<br>
> > >
>>>>>>>>>><br>
> > >
>>>>>>>>>> Kristijan<br>
> > > >>>>>>>
_______________________________________________<br>
> > > >>>>>>> Kamailio
(SER) - Users Mailing List<br>
> > > >>>>>>> <a
href="mailto:sr-users@lists.kamailio.org"
target="_blank" moz-do-not-send="true">sr-users@lists.kamailio.org</a><br>
> > > >>>>>>> <a
href="https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.kamailio.org%2Fcgi-bin%2Fmailman%2Flistinfo%2Fsr-users&data=02%7C01%7Cf.floimair%40commend.com%7C4008d49af1b347abe20308d6c1710532%7C13b1ddb756454e7fbe663171548559da%7C0%7C0%7C636909088526530221&sdata=CfDOAJ2wQlJDTfTfwj4Ba0BIT74gZJCiS4XNLW%2F1Dog%3D&reserved=0"
target="_blank" moz-do-not-send="true">
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users</a><br>
> > > >>>>>> --<br>
> > > >>>>>>
Daniel-Constantin Mierla -- <a
href="https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.asipto.com&data=02%7C01%7Cf.floimair%40commend.com%7C4008d49af1b347abe20308d6c1710532%7C13b1ddb756454e7fbe663171548559da%7C0%7C0%7C636909088526540229&sdata=JZ6S53yHAYHGO%2BiQ1IRPYwlaR6H8QeIRiBxjiqLhAqc%3D&reserved=0"
target="_blank" moz-do-not-send="true">
www.asipto.com</a><br>
> > > >>>>>> <a
href="https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.twitter.com%2Fmiconda&data=02%7C01%7Cf.floimair%40commend.com%7C4008d49af1b347abe20308d6c1710532%7C13b1ddb756454e7fbe663171548559da%7C0%7C0%7C636909088526550233&sdata=abtGRZNNELUIh4VJhWIPMjMgRDg4fLT%2F%2B28i2l1IXdE%3D&reserved=0"
target="_blank" moz-do-not-send="true">
www.twitter.com/miconda</a> -- <a
href="https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.linkedin.com%2Fin%2Fmiconda&data=02%7C01%7Cf.floimair%40commend.com%7C4008d49af1b347abe20308d6c1710532%7C13b1ddb756454e7fbe663171548559da%7C0%7C0%7C636909088526550233&sdata=yiFMASKu6gBkuFqifQsZjR%2F%2Fjbxr2z7reCKIrrpNK1s%3D&reserved=0"
target="_blank" moz-do-not-send="true">
www.linkedin.com/in/miconda</a><br>
> > > >>>>>> Kamailio World
Conference - May 6-8, 2019 -- <a
href="https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.kamailioworld.com&data=02%7C01%7Cf.floimair%40commend.com%7C4008d49af1b347abe20308d6c1710532%7C13b1ddb756454e7fbe663171548559da%7C0%7C0%7C636909088526560246&sdata=IqTpYULtc2fymVugRCXgOn3FKigN2eKGjIb2cTYrD0k%3D&reserved=0"
target="_blank" moz-do-not-send="true">
www.kamailioworld.com</a><br>
> > > >>>>>> Kamailio
Advanced Training - Mar 25-27, 2019, in Washington,
DC, USA --
<a
href="https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.asipto.com&data=02%7C01%7Cf.floimair%40commend.com%7C4008d49af1b347abe20308d6c1710532%7C13b1ddb756454e7fbe663171548559da%7C0%7C0%7C636909088526570254&sdata=mSGruspS5Vddgo9VHb2%2FjNvXg28Indn%2FPBXwofox31g%3D&reserved=0"
target="_blank" moz-do-not-send="true">
www.asipto.com</a><br>
> > > >>>>>><br>
> > > >>>>>
_______________________________________________<br>
> > > >>>>> Kamailio (SER) -
Users Mailing List<br>
> > > >>>>> <a
href="mailto:sr-users@lists.kamailio.org"
target="_blank" moz-do-not-send="true">sr-users@lists.kamailio.org</a><br>
> > > >>>>> <a
href="https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.kamailio.org%2Fcgi-bin%2Fmailman%2Flistinfo%2Fsr-users&data=02%7C01%7Cf.floimair%40commend.com%7C4008d49af1b347abe20308d6c1710532%7C13b1ddb756454e7fbe663171548559da%7C0%7C0%7C636909088526570254&sdata=39EnwRbLjbVt5fhJC1qOAxbeOEUlsXirOysqS25zW70%3D&reserved=0"
target="_blank" moz-do-not-send="true">
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users</a><br>
> > > >>>> --<br>
> > > >>>> Daniel-Constantin
Mierla -- <a
href="https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.asipto.com&data=02%7C01%7Cf.floimair%40commend.com%7C4008d49af1b347abe20308d6c1710532%7C13b1ddb756454e7fbe663171548559da%7C0%7C0%7C636909088526580262&sdata=IKSvu7HKThYWiDReXfGd3YOUN%2FkSrCIIA0ZphemHNrk%3D&reserved=0"
target="_blank" moz-do-not-send="true">
www.asipto.com</a><br>
> > > >>>> <a
href="https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.twitter.com%2Fmiconda&data=02%7C01%7Cf.floimair%40commend.com%7C4008d49af1b347abe20308d6c1710532%7C13b1ddb756454e7fbe663171548559da%7C0%7C0%7C636909088526590271&sdata=mNjRxHhnBtOTuWYH0SAjchWFLrJ9ZMEohj8WRf9Q%2B4E%3D&reserved=0"
target="_blank" moz-do-not-send="true">
www.twitter.com/miconda</a> -- <a
href="https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.linkedin.com%2Fin%2Fmiconda&data=02%7C01%7Cf.floimair%40commend.com%7C4008d49af1b347abe20308d6c1710532%7C13b1ddb756454e7fbe663171548559da%7C0%7C0%7C636909088526590271&sdata=FDEKg%2BZZ0rtVhp3N8KE%2FO6Os19S4gcuNFnkowh6Cg4Y%3D&reserved=0"
target="_blank" moz-do-not-send="true">
www.linkedin.com/in/miconda</a><br>
> > > >>>> Kamailio World
Conference - May 6-8, 2019 -- <a
href="https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.kamailioworld.com&data=02%7C01%7Cf.floimair%40commend.com%7C4008d49af1b347abe20308d6c1710532%7C13b1ddb756454e7fbe663171548559da%7C0%7C0%7C636909088526610292&sdata=pRIyX6ukA3fD9uSltzI9oNQS9guiQoLvako%2FYgOpomk%3D&reserved=0"
target="_blank" moz-do-not-send="true">
www.kamailioworld.com</a><br>
> > > >>>> Kamailio Advanced
Training - Mar 25-27, 2019, in Washington, DC, USA --
<a
href="https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.asipto.com&data=02%7C01%7Cf.floimair%40commend.com%7C4008d49af1b347abe20308d6c1710532%7C13b1ddb756454e7fbe663171548559da%7C0%7C0%7C636909088526610292&sdata=VgI9SYl%2FirYVa2j%2FgqxTJ%2F8SlbEafUeyDg9Ej4jXw8s%3D&reserved=0"
target="_blank" moz-do-not-send="true">
www.asipto.com</a><br>
> > > >>>><br>
> > > >> --<br>
> > > >> Daniel-Constantin Mierla -- <a
href="https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.asipto.com&data=02%7C01%7Cf.floimair%40commend.com%7C4008d49af1b347abe20308d6c1710532%7C13b1ddb756454e7fbe663171548559da%7C0%7C0%7C636909088526620296&sdata=wdG9sK2JFMtALBtBpiOdOOrCT2N5rcFU5vTIVgyTkmA%3D&reserved=0"
target="_blank" moz-do-not-send="true">
www.asipto.com</a><br>
> > > >> <a
href="https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.twitter.com%2Fmiconda&data=02%7C01%7Cf.floimair%40commend.com%7C4008d49af1b347abe20308d6c1710532%7C13b1ddb756454e7fbe663171548559da%7C0%7C0%7C636909088526630309&sdata=4XzLDUOq%2FoYdSWdJ2ZZh7sHPFY45w19dvan1m3%2FOgpY%3D&reserved=0"
target="_blank" moz-do-not-send="true">
www.twitter.com/miconda</a> -- <a
href="https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.linkedin.com%2Fin%2Fmiconda&data=02%7C01%7Cf.floimair%40commend.com%7C4008d49af1b347abe20308d6c1710532%7C13b1ddb756454e7fbe663171548559da%7C0%7C0%7C636909088526630309&sdata=QsMOMqcG422FyAKD87NNbDcTAnVELlxwlTsk9qhQY3E%3D&reserved=0"
target="_blank" moz-do-not-send="true">
www.linkedin.com/in/miconda</a><br>
> > > >> Kamailio World Conference -
May 6-8, 2019 -- <a
href="https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.kamailioworld.com&data=02%7C01%7Cf.floimair%40commend.com%7C4008d49af1b347abe20308d6c1710532%7C13b1ddb756454e7fbe663171548559da%7C0%7C0%7C636909088526640317&sdata=f5hJyISlS%2FyhFIbYtOAwGevmaTSuyInBl5QcIIQWxJk%3D&reserved=0"
target="_blank" moz-do-not-send="true">
www.kamailioworld.com</a><br>
> > > >> Kamailio Advanced Training -
Mar 25-27, 2019, in Washington, DC, USA -- <a
href="https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.asipto.com&data=02%7C01%7Cf.floimair%40commend.com%7C4008d49af1b347abe20308d6c1710532%7C13b1ddb756454e7fbe663171548559da%7C0%7C0%7C636909088526650321&sdata=lMfoFBzZ7RV5ZLl9c1phXnxYHrz6%2F78MLZq3ftuiW84%3D&reserved=0"
target="_blank" moz-do-not-send="true">
www.asipto.com</a><br>
> > > >><br>
> > > --<br>
> > > Daniel-Constantin Mierla -- <a
href="https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.asipto.com&data=02%7C01%7Cf.floimair%40commend.com%7C4008d49af1b347abe20308d6c1710532%7C13b1ddb756454e7fbe663171548559da%7C0%7C0%7C636909088526650321&sdata=lMfoFBzZ7RV5ZLl9c1phXnxYHrz6%2F78MLZq3ftuiW84%3D&reserved=0"
target="_blank" moz-do-not-send="true">
www.asipto.com</a><br>
> > > <a
href="https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.twitter.com%2Fmiconda&data=02%7C01%7Cf.floimair%40commend.com%7C4008d49af1b347abe20308d6c1710532%7C13b1ddb756454e7fbe663171548559da%7C0%7C0%7C636909088526660329&sdata=BgWlmOacvqNaaoay9DB%2B6ZLgvwJUycf9CGup97yzS5g%3D&reserved=0"
target="_blank" moz-do-not-send="true">
www.twitter.com/miconda</a> -- <a
href="https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.linkedin.com%2Fin%2Fmiconda&data=02%7C01%7Cf.floimair%40commend.com%7C4008d49af1b347abe20308d6c1710532%7C13b1ddb756454e7fbe663171548559da%7C0%7C0%7C636909088526660329&sdata=ozEAUiWg7FTMwGYijeingJv11ygyhYo4W3GBm3ZJMuM%3D&reserved=0"
target="_blank" moz-do-not-send="true">
www.linkedin.com/in/miconda</a><br>
> > > Kamailio World Conference - May 6-8,
2019 -- <a
href="https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.kamailioworld.com&data=02%7C01%7Cf.floimair%40commend.com%7C4008d49af1b347abe20308d6c1710532%7C13b1ddb756454e7fbe663171548559da%7C0%7C0%7C636909088526670338&sdata=F9zjXHrdLEB6%2FB4tmQvPqwyQnFM9ZjI8MKdSnA9nyTg%3D&reserved=0"
target="_blank" moz-do-not-send="true">
www.kamailioworld.com</a><br>
> > > Kamailio Advanced Training - Mar 25-27,
2019, in Washington, DC, USA -- <a
href="https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.asipto.com&data=02%7C01%7Cf.floimair%40commend.com%7C4008d49af1b347abe20308d6c1710532%7C13b1ddb756454e7fbe663171548559da%7C0%7C0%7C636909088526680346&sdata=biJ1rDk5WXly4BwxPwIDOHZcIxHlX3cL%2BciU4zSna1A%3D&reserved=0"
target="_blank" moz-do-not-send="true">
www.asipto.com</a><br>
> > ><br>
_______________________________________________<br>
Kamailio (SER) - Users Mailing List<br>
<a href="mailto:sr-users@lists.kamailio.org"
target="_blank" moz-do-not-send="true">sr-users@lists.kamailio.org</a><br>
<a
href="https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.kamailio.org%2Fcgi-bin%2Fmailman%2Flistinfo%2Fsr-users&data=02%7C01%7Cf.floimair%40commend.com%7C4008d49af1b347abe20308d6c1710532%7C13b1ddb756454e7fbe663171548559da%7C0%7C0%7C636909088526680346&sdata=RqR%2BxmUOrHlvfRlJv43ZH8LOkd4ZCMu6Xn59E7aK4Ew%3D&reserved=0"
target="_blank" moz-do-not-send="true">https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users</a><o:p></o:p></p>
</blockquote>
</div>
<p class="MsoNormal"><br clear="all">
<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p class="MsoNormal">-- <o:p></o:p></p>
<div>
<p class="MsoNormal"><img id="_x0000_i1025"
src="http://sip.antisip.com/am48.png"
alt="http://sip.antisip.com/am48.png"
moz-do-not-send="true" border="0">Antisip -
<a
href="https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.antisip.com&data=02%7C01%7Cf.floimair%40commend.com%7C4008d49af1b347abe20308d6c1710532%7C13b1ddb756454e7fbe663171548559da%7C0%7C0%7C636909088526690359&sdata=%2F2zOiWbiz2yAKOz4oQKPPiaT%2BE3HoYVETeU1Ktr4PL8%3D&reserved=0"
target="_blank" moz-do-not-send="true">
http://www.antisip.com</a><o:p></o:p></p>
</div>
</blockquote>
<pre>-- <o:p></o:p></pre>
<pre>Daniel-Constantin Mierla -- <a href="https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.asipto.com&data=02%7C01%7Cf.floimair%40commend.com%7C4008d49af1b347abe20308d6c1710532%7C13b1ddb756454e7fbe663171548559da%7C0%7C0%7C636909088526690359&sdata=wYyFvDHHTyQ4715bAnRuSSsCM1f6IvjappwbujOFcKo%3D&reserved=0" moz-do-not-send="true">www.asipto.com</a><o:p></o:p></pre>
<pre><a href="https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.twitter.com%2Fmiconda&data=02%7C01%7Cf.floimair%40commend.com%7C4008d49af1b347abe20308d6c1710532%7C13b1ddb756454e7fbe663171548559da%7C0%7C0%7C636909088526700363&sdata=VqBde6P5Pa4b5%2Bcz0mrUfaQYXwnjK0lPcWe8ZkA5WX8%3D&reserved=0" moz-do-not-send="true">www.twitter.com/miconda</a> -- <a href="https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.linkedin.com%2Fin%2Fmiconda&data=02%7C01%7Cf.floimair%40commend.com%7C4008d49af1b347abe20308d6c1710532%7C13b1ddb756454e7fbe663171548559da%7C0%7C0%7C636909088526700363&sdata=Lhst%2FwKY6WwaZzYCZSWiKfr%2Bk02JSPP1V8PXw5pLJZs%3D&reserved=0" moz-do-not-send="true">www.linkedin.com/in/miconda</a><o:p></o:p></pre>
<pre>Kamailio World Conference - May 6-8, 2019 -- <a href="https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.kamailioworld.com&data=02%7C01%7Cf.floimair%40commend.com%7C4008d49af1b347abe20308d6c1710532%7C13b1ddb756454e7fbe663171548559da%7C0%7C0%7C636909088526710376&sdata=ENTJ2mcgqVDyCPSmNnewFjiqbJs%2BZpIihYnXr20rWmw%3D&reserved=0" moz-do-not-send="true">www.kamailioworld.com</a><o:p></o:p></pre>
<pre>Kamailio Advanced Training - Mar 25-27, 2019, in Washington, DC, USA -- <a href="https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.asipto.com&data=02%7C01%7Cf.floimair%40commend.com%7C4008d49af1b347abe20308d6c1710532%7C13b1ddb756454e7fbe663171548559da%7C0%7C0%7C636909088526720380&sdata=YmCXbGb3X7gNpKgm7TQJJH8sFSC7dlPxgVS0woqphTs%3D&reserved=0" moz-do-not-send="true">www.asipto.com</a><o:p></o:p></pre>
</blockquote>
<pre>-- <o:p></o:p></pre>
<pre>Daniel-Constantin Mierla -- <a href="https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.asipto.com&data=02%7C01%7Cf.floimair%40commend.com%7C4008d49af1b347abe20308d6c1710532%7C13b1ddb756454e7fbe663171548559da%7C0%7C0%7C636909088526720380&sdata=YmCXbGb3X7gNpKgm7TQJJH8sFSC7dlPxgVS0woqphTs%3D&reserved=0" moz-do-not-send="true">www.asipto.com</a><o:p></o:p></pre>
<pre><a href="https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.twitter.com%2Fmiconda&data=02%7C01%7Cf.floimair%40commend.com%7C4008d49af1b347abe20308d6c1710532%7C13b1ddb756454e7fbe663171548559da%7C0%7C0%7C636909088526730388&sdata=2QaiULvJkZ58paz8XPyt%2BimNtHv0RJWgRumA5rpyaUg%3D&reserved=0" moz-do-not-send="true">www.twitter.com/miconda</a> -- <a href="https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.linkedin.com%2Fin%2Fmiconda&data=02%7C01%7Cf.floimair%40commend.com%7C4008d49af1b347abe20308d6c1710532%7C13b1ddb756454e7fbe663171548559da%7C0%7C0%7C636909088526730388&sdata=oZaUpr8gw4Wao0MY7UJQHGd08zPA1XiT8YSma0ItkRg%3D&reserved=0" moz-do-not-send="true">www.linkedin.com/in/miconda</a><o:p></o:p></pre>
<pre>Kamailio World Conference - May 6-8, 2019 -- <a href="https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.kamailioworld.com&data=02%7C01%7Cf.floimair%40commend.com%7C4008d49af1b347abe20308d6c1710532%7C13b1ddb756454e7fbe663171548559da%7C0%7C0%7C636909088526740401&sdata=7OsH4zVH1My373X60M1VYWknIEjUrIe7yj9kclQig3k%3D&reserved=0" moz-do-not-send="true">www.kamailioworld.com</a><o:p></o:p></pre>
</div>
</blockquote>
<pre class="moz-signature" cols="72">--
Daniel-Constantin Mierla -- <a class="moz-txt-link-abbreviated" href="http://www.asipto.com">www.asipto.com</a>
<a class="moz-txt-link-abbreviated" href="http://www.twitter.com/miconda">www.twitter.com/miconda</a> -- <a class="moz-txt-link-abbreviated" href="http://www.linkedin.com/in/miconda">www.linkedin.com/in/miconda</a></pre>
</body>
</html>