[sr-dev] db_redis shared tcp connection issue

[Daniel-Constantin Mierla]

Hello,

this sounds like a module does a db operation in mod init opening the
connection, but does it close it afterwards there. It should then
re-open in child init.

It can be also in child_init(), but when the rank is proc main or proc
init. In child init db connection has to be left opened only for the
other ranks.

Try to identify which component makes the first operation.

Cheers,
Daniel

On 29.04.22 12:39, Andrew Pogrebennyk wrote:
> Dear community,
> I've been looking at some weirdness in db_redis behavior when it
> returns the responses to the queries made by tcp processes in mixed order.
> Tested this on various kamailio 5.3 and 5.4 (sipwise spce) and they
> are showing interesting pattern.
> After restart of kamailio I ran lsof to enumerate all the sockets open
> in kamailio children.
> There is a connection to db port 6379 which is held by multiple
> processes at the same time.
>
>     for i in $(ps auxww | grep kamailio.proxy | grep -v grep | awk
>     '{print $2}'); do echo "print file descriptors of $i" && sudo lsof
>     -p $i | grep 6379; done > redis_conn.txt
>
> ...i see that lsof lists tcp client socket to redis server with same
> source TCP port and same inode number in several processes:
>
>   14199,   "TIMER NH",
>   14200,  "ctl handler",
>   14205,  "Dialog Clean Timer",
>   14206,  "JSONRPCS FIFO",
>   14210,  "JSONRPCS DATAGRAM",
>   14213,  "tcp receiver (generic) child=0",
>   14214,  "tcp receiver (generic) child=1",
>   14215,  "tcp receiver (generic) child=2",
>   14220,  "tcp receiver (generic) child=3",
>   14224,  "tcp receiver (generic) child=4",
>   14225,  "tcp main process"
>
> The UDP processes are safe (and some timer ones too), because in
> that lsof they have unique TCP client port.
>
> That's giving me a lot of headache because UA registrations received
> by any of the TCP workers (or IPSec ones for that matter) are
> randomly failing, because  if two processes made same query to DB in
> parallel it is appearing on the wire with same TCP source port and
> replies can be mixed up.
>
> This can be some bug in usage of hiredis, impacting all users of
> db_redis module. Is there any relation to the way kamailio is working
> its TCP workers, where maybe tcp workers are forked from the main
> attendant processes after having opened the DB connection?
>
> P.S. Why I have the above hypothesis: when I log redis queries with
> redis-cli monitor at startup of kamailio, I see only that srem_key_lua
> is executed against redis in runtime only once from that source port,
> but then this connection is shared across multiple processes.
>
> Regards,
> Andrew
>
> _______________________________________________
> Kamailio (SER) - Development Mailing List
> sr-dev at lists.kamailio.org
> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

-- 
Daniel-Constantin Mierla -- www.asipto.com
www.twitter.com/miconda -- www.linkedin.com/in/miconda
Kamailio Advanced Training - Online
  * https://www.asipto.com/sw/kamailio-advanced-training-online/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.kamailio.org/pipermail/sr-dev/attachments/20220429/b511c849/attachment-0001.htm>