<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Vorformatiert Zchn";
margin:0cm;
font-size:10.0pt;
font-family:"Courier New";}
span.HTMLVorformatiertZchn
{mso-style-name:"HTML Vorformatiert Zchn";
mso-style-priority:99;
mso-style-link:"HTML Vorformatiert";
font-family:Consolas;
mso-fareast-language:DE;}
span.E-MailFormatvorlage21
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:70.85pt 70.85pt 2.0cm 70.85pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="DE" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Hello,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="mso-fareast-language:EN-US">thanks for the confirmation. Please create a pull request on our tracker with the fix if your tests were successful.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="mso-fareast-language:EN-US">Cheers,<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="mso-fareast-language:EN-US">Henning<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="mso-fareast-language:EN-US">-- <o:p>
</o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="mso-fareast-language:EN-US">Henning Westerholt –
</span><span style="mso-fareast-language:EN-US"><a href="https://skalatan.de/blog/"><span lang="EN-GB" style="color:#0563C1">https://skalatan.de/blog/</span></a></span><span lang="EN-GB" style="mso-fareast-language:EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="mso-fareast-language:EN-US">Kamailio services –
</span><span style="mso-fareast-language:EN-US"><a href="https://gilawa.com/"><span lang="EN-GB" style="color:#0563C1">https://gilawa.com</span></a></span><span lang="EN-GB" style="mso-fareast-language:EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal" style="margin-left:35.4pt"><b>From:</b> sr-dev <sr-dev-bounces@lists.kamailio.org>
<b>On Behalf Of </b>Andrew Pogrebennyk<br>
<b>Sent:</b> Friday, April 29, 2022 6:27 PM<br>
<b>To:</b> Daniel-Constantin Mierla <miconda@gmail.com><br>
<b>Cc:</b> Kamailio (SER) - Development Mailing List <sr-dev@lists.kamailio.org><br>
<b>Subject:</b> Re: [sr-dev] db_redis shared tcp connection issue<o:p></o:p></p>
</div>
<p class="MsoNormal" style="margin-left:35.4pt"><o:p> </o:p></p>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">Daniel,<o:p></o:p></p>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">I think I found it. Since some historic times the ims_usrloc_scscf and usrloc_pcscf have had connection opened for main process in child init.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">I changed the child init from:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">case WRITE_THROUGH:<br>
/* connect to db only from SIP workers, TIMER and MAIN processes */<br>
if (_rank<=0 && _rank!=PROC_TIMER && _rank!=PROC_MAIN)<br>
return 0;<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"><br>
to<br>
case WRITE_THROUGH:<br>
/* skip child init for non-worker process ranks */<br>
if (_rank==PROC_INIT || _rank==PROC_MAIN || _rank==PROC_TCP_MAIN)<br>
return 0;<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">Testing it.<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal" style="margin-left:35.4pt"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">On Fri, Apr 29, 2022 at 4:18 PM Daniel-Constantin Mierla <<a href="mailto:miconda@gmail.com">miconda@gmail.com</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<div>
<p style="margin-left:35.4pt">No.<o:p></o:p></p>
<p style="margin-left:35.4pt">Connections opened in mod init or child init for rank proc main/init must be closed again there.<o:p></o:p></p>
<p style="margin-left:35.4pt">If a component wants to keep the connection open, has to be done in child init for ranks corresponding to sip workers, rpcs, timers, ...<o:p></o:p></p>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">On 29.04.22 15:25, Andrew Pogrebennyk wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal" style="margin-left:35.4pt">Hi Daniel, <o:p></o:p></p>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">I am not sure if I understood you correctly. Do you mean that child_init should open the connection only when the rank is proc main or proc init?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">For example, in pua module we have <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">static int child_init(int rank)<br>
{<br>
if (rank==PROC_INIT || rank==PROC_MAIN || rank==PROC_TCP_MAIN)<br>
return 0; /* do nothing for the main process */<br>
<br>
if (pua_dbf.init==0)<br>
{<br>
LM_CRIT("database not bound\n");<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">Is that correct? If I have a module which does not connect in child_init for rank PROC_RPC, but the origin of this module (ims_dialog vs dialog), does also establish connection in RPC rank would that be a problem?
No, right? :)<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">Thanks for the pointer, checking it.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">Andrew<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal" style="margin-left:35.4pt"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">On Fri, Apr 29, 2022 at 1:17 PM Daniel-Constantin Mierla <<a href="mailto:miconda@gmail.com" target="_blank">miconda@gmail.com</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<div>
<p style="margin-left:35.4pt">Hello,<o:p></o:p></p>
<p style="margin-left:35.4pt">this sounds like a module does a db operation in mod init opening the connection, but does it close it afterwards there. It should then re-open in child init.<o:p></o:p></p>
<p style="margin-left:35.4pt">It can be also in child_init(), but when the rank is proc main or proc init. In child init db connection has to be left opened only for the other ranks.<o:p></o:p></p>
<p style="margin-left:35.4pt">Try to identify which component makes the first operation.<o:p></o:p></p>
<p style="margin-left:35.4pt">Cheers,<br>
Daniel<o:p></o:p></p>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">On 29.04.22 12:39, Andrew Pogrebennyk wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal" style="margin-left:35.4pt">Dear community,<br>
I've been looking at some weirdness in db_redis behavior when it returns the responses to the queries made by tcp processes in mixed order.<br>
Tested this on various kamailio 5.3 and 5.4 (sipwise spce) and they are showing interesting pattern.<br>
After restart of kamailio I ran lsof to enumerate all the sockets open in kamailio children.<br>
There is a connection to db port 6379 which is held by multiple processes at the same time.<o:p></o:p></p>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<p class="MsoNormal" style="margin-left:35.4pt">for i in $(ps auxww | grep kamailio.proxy | grep -v grep | awk '{print $2}'); do echo "print file descriptors of $i" && sudo lsof -p $i | grep 6379; done > redis_conn.txt<o:p></o:p></p>
</blockquote>
<p class="MsoNormal" style="margin-left:35.4pt">...i see that lsof lists tcp client socket to redis server with same source TCP port and same inode number in several processes:<br>
<br>
14199, "TIMER NH",<br>
14200, "ctl handler",<br>
14205, "Dialog Clean Timer",<br>
14206, "JSONRPCS FIFO",<br>
14210, "JSONRPCS DATAGRAM",<br>
14213, "tcp receiver (generic) child=0",<br>
14214, "tcp receiver (generic) child=1",<br>
14215, "tcp receiver (generic) child=2",<br>
14220, "tcp receiver (generic) child=3",<br>
14224, "tcp receiver (generic) child=4",<br>
14225, "tcp main process" <o:p></o:p></p>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">The UDP processes are safe (and some timer ones too), because in that lsof they have unique TCP client port.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">That's giving me a lot of headache because UA registrations received by any of the TCP workers (or IPSec ones for that matter) are randomly failing, because if two processes made same query to DB in parallel
it is appearing on the wire with same TCP source port and replies can be mixed up.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">This can be some bug in usage of hiredis, impacting all users of db_redis module. Is there any relation to the way kamailio is working its TCP workers, where maybe tcp workers are forked from the main attendant
processes after having opened the DB connection?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">P.S. Why I have the above hypothesis: when I log redis queries with redis-cli monitor at startup of kamailio, I see only that srem_key_lua is executed against redis in runtime only once from that source port,
but then this connection is shared across multiple processes.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">Regards,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">Andrew<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal" style="margin-left:35.4pt"><o:p> </o:p></p>
<pre style="margin-left:35.4pt">_______________________________________________<o:p></o:p></pre>
<pre style="margin-left:35.4pt">Kamailio (SER) - Development Mailing List<o:p></o:p></pre>
<pre style="margin-left:35.4pt"><a href="mailto:sr-dev@lists.kamailio.org" target="_blank">sr-dev@lists.kamailio.org</a><o:p></o:p></pre>
<pre style="margin-left:35.4pt"><a href="https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev" target="_blank">https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev</a><o:p></o:p></pre>
</blockquote>
<pre style="margin-left:35.4pt">-- <o:p></o:p></pre>
<pre style="margin-left:35.4pt">Daniel-Constantin Mierla -- <a href="http://www.asipto.com" target="_blank">www.asipto.com</a><o:p></o:p></pre>
<pre style="margin-left:35.4pt"><a href="http://www.twitter.com/miconda" target="_blank">www.twitter.com/miconda</a> -- <a href="http://www.linkedin.com/in/miconda" target="_blank">www.linkedin.com/in/miconda</a><o:p></o:p></pre>
<pre style="margin-left:35.4pt">Kamailio Advanced Training - Online<o:p></o:p></pre>
<pre style="margin-left:35.4pt"> * <a href="https://www.asipto.com/sw/kamailio-advanced-training-online/" target="_blank">https://www.asipto.com/sw/kamailio-advanced-training-online/</a><o:p></o:p></pre>
</div>
</blockquote>
</div>
</blockquote>
<pre style="margin-left:35.4pt">-- <o:p></o:p></pre>
<pre style="margin-left:35.4pt">Daniel-Constantin Mierla -- <a href="http://www.asipto.com" target="_blank">www.asipto.com</a><o:p></o:p></pre>
<pre style="margin-left:35.4pt"><a href="http://www.twitter.com/miconda" target="_blank">www.twitter.com/miconda</a> -- <a href="http://www.linkedin.com/in/miconda" target="_blank">www.linkedin.com/in/miconda</a><o:p></o:p></pre>
<pre style="margin-left:35.4pt">Kamailio Advanced Training - Online<o:p></o:p></pre>
<pre style="margin-left:35.4pt"> * <a href="https://www.asipto.com/sw/kamailio-advanced-training-online/" target="_blank">https://www.asipto.com/sw/kamailio-advanced-training-online/</a><o:p></o:p></pre>
</div>
</blockquote>
</div>
</div>
</body>
</html>