[SR-Users] Possible problem with multiple threads handling same SIP message

[Dirk Teurlings - SIGNET B.V.]

Hello,

Running kamailio 4.2.5 for a few weeks now, coming from 4.2.3. Since the 
upgrade we increased the logging substantially in our routing to get 
more insight into what is going on.

We noticed that in a few cases the log showed a SIP message was being 
threated by multiple forks, each considering if it was a non-existing 
call. e.g.

Jul  9 13:19:32 son-sbc1-dc1 /usr/sbin/kamailio[17170]: INFO: <script>: 
28ca767d5524f7ac095efcd02d6995f5 at 1.2.3.4:5060 - CB, Checking service 
restrictions
Jul  9 13:19:32 son-sbc1-dc1 /usr/sbin/kamailio[17170]: INFO: <script>: 
28ca767d5524f7ac095efcd02d6995f5 at 1.2.3.4:5060 - CB, Checking 
international restrictions
Jul  9 13:19:32 son-sbc1-dc1 /usr/sbin/kamailio[17171]: INFO: <script>: 
28ca767d5524f7ac095efcd02d6995f5 at 1.2.3.4:5060 - CB, Checking service 
restrictions
Jul  9 13:19:32 son-sbc1-dc1 /usr/sbin/kamailio[17171]: INFO: <script>: 
28ca767d5524f7ac095efcd02d6995f5 at 1.2.3.4:5060 - CB, Checking 
international restrictions

As you can see this shows different threads handling the exact same 
callid (printed by $ci). And both run the same checks in our routing 
plan. The main problem is, I'm not able to reproduce this on the 
testplatform, but only see it happen on the main systems. Also, it 
doesn't happen that often, in this case there are about maybe 10 similar 
calls like this over the run of a few weeks. This case show the call 
being handled by 2 threads, but we've even seen with one call it being 
handled by 4 threads, all with the same $ci and the same messages, just 
different PIDs.

I've checked and made sure the routing isn't recursively calling itself. 
But even if that was true, I would imagine the same PID for all the log 
entries.

Running production in debug mode isn't an option, as it will interfere 
too much with the day-to-day business. So I'm hoping to get a few 
pointers here maybe, I'm happy to look at the source code more in depth, 
I just need a little help in the right direction to not get lost in 
other code.

Any help is much appreciated.

Cheers,
Dirk