<p></p>
<h3>Description</h3>
<p>While running a simple load with a UAC(Sipp) and two UAS(Both Sipp) , with Kamailio in the middle, acting as a Call Stateful Proxy, I observed that the Dispatcher Module reports one call stuck despite NO calls being reported as Active in the Dialog Module.</p>
<h2>Setup:</h2>
<pre><code> |------> UAS1(sipp)
</code></pre>
<p>UAC(sipp) ----------------------Kamailio(Call Stateful Proxy)------------<br>
|------> UAS2(sipp)</p>
<p>Scenario: The scenario is simple INVITE----- 200OK------ACK , with BYE being sent after 60 secs by UAC.</p>
<h3>Troubleshooting</h3>
<p>Point noteworthy is that this is purely timing issue and happens generally with very long duration, like 24 hours or so, but can crop up withing a few hours also.</p>
<h4>Reproduction</h4>
<p>Just run the calls via sipp as described in the setup.</p>
<h4>Debugging Data</h4>
<p><strong>OUTPUT after cleanly stopping the test:</strong></p>
<blockquote>
<p>Output of "kamctl stats dialog" command<br>
{<br>
"jsonrpc": "2.0",<br>
"result": [<br>
"dialog:active_dialogs = 0",<br>
"dialog:early_dialogs = 0",<br>
"dialog:expired_dialogs = 0",<br>
"dialog:failed_dialogs = 959",<br>
"dialog:processed_dialogs = 1695662"<br>
],<br>
"id": 21282<br>
}</p>
</blockquote>
<blockquote>
<p>Output of "kamcmd dispatcher.list"</p>
</blockquote>
<p>{<br>
NRSETS: 1<br>
RECORDS: {<br>
SET: {<br>
ID: 1<br>
TARGETS: {<br>
DEST: {<br>
URI: sip:10.214.3.20:5060;transport=udp<br>
FLAGS: AP<br>
PRIORITY: 0<br>
ATTRS: {<br>
BODY: duid=sample-cas-1;maxload=1000<br>
DUID: sample-cas-1<br>
MAXLOAD: 1000<br>
WEIGHT: 0<br>
RWEIGHT: 0<br>
SOCKET:<br>
}<br>
LATENCY: {<br>
AVG: 2.409000<br>
STD: 108.999000<br>
EST: 1.124000<br>
MAX: 9503<br>
TIMEOUT: 22<br>
}<br>
RUNTIME: {<br>
DLGLOAD: 997<br>
}<br>
}<br>
DEST: {<br>
URI: sip:10.214.3.19:5060;transport=udp<br>
FLAGS: AP<br>
PRIORITY: 0<br>
ATTRS: {<br>
BODY: duid=sample-cas-0;maxload=1000<br>
DUID: sample-cas-0<br>
MAXLOAD: 1000<br>
WEIGHT: 0<br>
RWEIGHT: 0<br>
SOCKET:<br>
}<br>
LATENCY: {<br>
AVG: 1.429000<br>
STD: 40.279000<br>
EST: 0.999000<br>
MAX: 3502<br>
TIMEOUT: 22<br>
}<br>
RUNTIME: {<br>
DLGLOAD: 985<br>
}<br>
}<br>
}<br>
}<br>
}<br>
}</p>
<p>NOTE: Please see that the dispatcher shows a lot of calls stuck after overnight load, and there by we started seeing 404's sent back by Kamailio dispatcher module for most of the calls.<br>
It is noteworthy that we have kept a close tab on the tests thereafter and caught an iteration where initially only call is stuck.</p>
<h3>Possible Solutions</h3>
<p>The problem most likely seems to be in the dispatcher module, which does not seem to be incrementing/decrementing the load variable for each UAS( 'dload' variable to be precise), due to which the concurrent incrementing/decrementing of the 'dload' parameter makes the dispatcher report spurious values.</p>
<p>FIX: The following needs to be done.</p>
<ol>
<li>
<p>File: <strong>dispatch.c</strong><br>
Function: <strong>ds_load_add()</strong><br>
Just before dset->dlist[dst].dload++; line, we must take a lock and release it thereafter. Like:</p>
<pre><code> **lock_get(&dset->lock)**;
dset->dlist[dst].dload++;
**lock_release(&dset->lock);**
</code></pre>
</li>
</ol>
<p>2: File: <strong>dispatch.c</strong><br>
Function: <strong>ds_load_replace()</strong> , the code needs a lock as well while decrementing. Like:<br>
<strong>lock_get(&idx->lock);</strong><br>
if(idx->dlist[olddst].dload > 0) {<br>
idx->dlist[olddst].dload--;<br>
<strong>lock_release(&idx->lock);</strong><br>
......<br>
} else {<br>
<strong>lock_release(&idx->lock);</strong><br>
......<br>
}</p>
<ol start="3">
<li>
<p>File: <strong>dispatch.c</strong><br>
Function: <strong>ds_load_remove_byid</strong> , the code needs a lock as well while decrementing. Like:</p>
<p><strong>lock_get(&idx->lock);</strong><br>
if(idx->dlist[olddst].dload > 0) {<br>
idx->dlist[olddst].dload--;<br>
<strong>lock_release(&idx->lock);</strong><br>
...........<br>
} else {<br>
<strong>lock_release(&idx->lock);</strong><br>
........<br>
}</p>
</li>
</ol>
<p>NOTE: I believe the lock should be taken also for the the API "ds_get_leastloaded()" in dispatch.c because otherwise, we may get select incorrect destination, especially when we are at just about the same load and that is almost close to the 'maxload' value configured in the dispatcher list file. But I will wait for the community comments for this.</p>
<p>The above fix solved my issue, so please review the same and if you deem suitable, I could go ahead with the fix delivery.</p>
<h3>Additional Information</h3>
<p>Kamailio Version:<br>
[root@CPaaSVM ~]# kamailio -v<br>
version: kamailio 5.3.2 (x86_64/linux) 7ba545<br>
flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES<br>
ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB<br>
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.<br>
id: 7ba545<br>
compiled on 19:01:39 May 1 2020 with gcc 4.8.5</p>
<p>NOTE: I do not think this is version specific and probably that it exists in the later versions also.</p>
<ul>
<li><strong>Operating System</strong>:<br>
Centos 7.7</li>
</ul>
<p style="font-size:small;-webkit-text-size-adjust:none;color:#666;">—<br />You are receiving this because you are subscribed to this thread.<br />Reply to this email directly, <a href="https://github.com/kamailio/kamailio/issues/2322">view it on GitHub</a>, or <a href="https://github.com/notifications/unsubscribe-auth/ABO7UZIYGTX33W2VPW72FT3RQ6XKLANCNFSM4M5V5VYA">unsubscribe</a>.<img src="https://github.com/notifications/beacon/ABO7UZIBYWMT4XWR6XZZRYDRQ6XKLA5CNFSM4M5V5VYKYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4JFSROGA.gif" height="1" width="1" alt="" /></p>
<script type="application/ld+json">[
{
"@context": "http://schema.org",
"@type": "EmailMessage",
"potentialAction": {
"@type": "ViewAction",
"target": "https://github.com/kamailio/kamailio/issues/2322",
"url": "https://github.com/kamailio/kamailio/issues/2322",
"name": "View Issue"
},
"description": "View this Issue on GitHub",
"publisher": {
"@type": "Organization",
"name": "GitHub",
"url": "https://github.com"
}
}
]</script>