<div dir="ltr">Daniel,<div>I think I found it. Since some historic times the ims_usrloc_scscf and usrloc_pcscf have had connection opened for main process in child init.</div><div>I changed the child init from:</div><div> case WRITE_THROUGH:<br> /* connect to db only from SIP workers, TIMER and MAIN processes */<br> if (_rank<=0 && _rank!=PROC_TIMER && _rank!=PROC_MAIN)<br> return 0;</div><div><br>to<br> case WRITE_THROUGH:<br> /* skip child init for non-worker process ranks */<br> if (_rank==PROC_INIT || _rank==PROC_MAIN || _rank==PROC_TCP_MAIN)<br> return 0;<br></div><div>Testing it.</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Apr 29, 2022 at 4:18 PM Daniel-Constantin Mierla <<a href="mailto:miconda@gmail.com">miconda@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<p>No.</p>
<p>Connections opened in mod init or child init for rank proc
main/init must be closed again there.</p>
<p>If a component wants to keep the connection open, has to be done
in child init for ranks corresponding to sip workers, rpcs,
timers, ...<br>
</p>
<div>On 29.04.22 15:25, Andrew Pogrebennyk
wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Hi Daniel,
<div>I am not sure if I understood you correctly. Do you mean
that child_init should open the connection only when the rank
is proc main or proc init?</div>
<div><br>
</div>
<div>For example, in pua module we have </div>
<div><br>
</div>
<div>static int child_init(int rank)<br>
{<br>
if (rank==PROC_INIT || rank==PROC_MAIN ||
rank==PROC_TCP_MAIN)<br>
return 0; /* do nothing for the main process
*/<br>
<br>
if (pua_dbf.init==0)<br>
{<br>
LM_CRIT("database not bound\n");<br>
</div>
<div><br>
</div>
<div>Is that correct? If I have a module which does not connect
in child_init for rank PROC_RPC, but the origin of this module
(ims_dialog vs dialog), does also establish connection in RPC
rank would that be a problem? No, right? :)</div>
<div><br>
</div>
<div>Thanks for the pointer, checking it.</div>
<div>Andrew</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Fri, Apr 29, 2022 at 1:17
PM Daniel-Constantin Mierla <<a href="mailto:miconda@gmail.com" target="_blank">miconda@gmail.com</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<p>Hello,</p>
<p>this sounds like a module does a db operation in mod init
opening the connection, but does it close it afterwards
there. It should then re-open in child init.</p>
<p>It can be also in child_init(), but when the rank is proc
main or proc init. In child init db connection has to be
left opened only for the other ranks.<br>
</p>
<p>Try to identify which component makes the first
operation.</p>
<p>Cheers,<br>
Daniel<br>
</p>
<div>On 29.04.22 12:39, Andrew Pogrebennyk wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Dear community,<br>
I've been looking at some weirdness in db_redis behavior
when it returns the responses to the queries made by tcp
processes in mixed order.<br>
Tested this on various kamailio 5.3 and 5.4 (sipwise
spce) and they are showing interesting pattern.<br>
After restart of kamailio I ran lsof to enumerate all
the sockets open in kamailio children.<br>
There is a connection to db port 6379 which is held by
multiple processes at the same time.<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">for i in $(ps auxww
| grep kamailio.proxy | grep -v grep | awk '{print
$2}'); do echo "print file descriptors of $i"
&& sudo lsof -p $i | grep 6379; done >
redis_conn.txt</blockquote>
...i see that lsof lists tcp client socket to redis
server with same source TCP port and same inode number
in several processes:<br>
<br>
14199, "TIMER NH",<br>
14200, "ctl handler",<br>
14205, "Dialog Clean Timer",<br>
14206, "JSONRPCS FIFO",<br>
14210, "JSONRPCS DATAGRAM",<br>
14213, "tcp receiver (generic) child=0",<br>
14214, "tcp receiver (generic) child=1",<br>
14215, "tcp receiver (generic) child=2",<br>
14220, "tcp receiver (generic) child=3",<br>
14224, "tcp receiver (generic) child=4",<br>
14225, "tcp main process"
<div><br>
</div>
<div>The UDP processes are safe (and some timer ones
too), because in that lsof they have unique TCP client
port.<br>
</div>
<div><br>
</div>
<div>That's giving me a lot of headache because UA
registrations received by any of the TCP workers (or
IPSec ones for that matter) are
randomly failing, because if two processes made same
query to DB in parallel it is appearing on the wire
with same TCP source port and replies can be mixed up.</div>
<div><br>
</div>
<div>This can be some bug in usage of hiredis, impacting
all users of db_redis module. Is there any relation to
the way kamailio is working its TCP workers, where
maybe tcp workers are forked from the main attendant
processes after having opened the DB connection?</div>
<div><br>
</div>
<div>P.S. Why I have the above hypothesis: when I log
redis queries with redis-cli monitor at startup of
kamailio, I see only that srem_key_lua is executed
against redis in runtime only once from that source
port, but then this connection is shared across
multiple processes.</div>
<div><br>
</div>
<div>Regards,</div>
<div>Andrew</div>
</div>
<br>
<fieldset></fieldset>
<pre>_______________________________________________
Kamailio (SER) - Development Mailing List
<a href="mailto:sr-dev@lists.kamailio.org" target="_blank">sr-dev@lists.kamailio.org</a>
<a href="https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev" target="_blank">https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev</a>
</pre>
</blockquote>
<pre cols="72">--
Daniel-Constantin Mierla -- <a href="http://www.asipto.com" target="_blank">www.asipto.com</a>
<a href="http://www.twitter.com/miconda" target="_blank">www.twitter.com/miconda</a> -- <a href="http://www.linkedin.com/in/miconda" target="_blank">www.linkedin.com/in/miconda</a>
Kamailio Advanced Training - Online
* <a href="https://www.asipto.com/sw/kamailio-advanced-training-online/" target="_blank">https://www.asipto.com/sw/kamailio-advanced-training-online/</a></pre>
</div>
</blockquote>
</div>
</blockquote>
<pre cols="72">--
Daniel-Constantin Mierla -- <a href="http://www.asipto.com" target="_blank">www.asipto.com</a>
<a href="http://www.twitter.com/miconda" target="_blank">www.twitter.com/miconda</a> -- <a href="http://www.linkedin.com/in/miconda" target="_blank">www.linkedin.com/in/miconda</a>
Kamailio Advanced Training - Online
* <a href="https://www.asipto.com/sw/kamailio-advanced-training-online/" target="_blank">https://www.asipto.com/sw/kamailio-advanced-training-online/</a></pre>
</div>
</blockquote></div>