[kamailio/kamailio] Kamailio dispatcher module reports incorrect values with Call-Load based dispatching algorithm(No.10) (#2322) - sr-dev

11 May 2020

### Description
While running a simple load with a UAC(Sipp) and two UAS(Both Sipp) , with Kamailio in the
middle, acting as a Call Stateful Proxy, I observed that the Dispatcher Module reports one
call stuck despite NO calls being reported as Active in the Dialog Module.

Setup:
------- 

                   |------> UAS1(sipp)
UAC(sipp) ----------------------Kamailio(Call Stateful Proxy)------------

                   |------> UAS2(sipp)

Scenario: The scenario is simple INVITE----- 200OK------ACK , with BYE being sent after 60
secs by UAC.

### Troubleshooting

Point noteworthy is that this is purely timing issue and happens generally with very long
duration, like 24 hours or so, but can crop up withing a few hours also. 

#### Reproduction

Just run the calls via sipp as described in the setup. 

#### Debugging Data

**OUTPUT after cleanly stopping the test:**

...
  Output of "kamctl stats dialog" command
     {
  "jsonrpc":  "2.0",
  "result": [
    "dialog:active_dialogs = 0",
    "dialog:early_dialogs = 0",
    "dialog:expired_dialogs = 0",
    "dialog:failed_dialogs = 959",
    "dialog:processed_dialogs = 1695662"
  ],
  "id": 21282
 }

...
  Output of "kamcmd dispatcher.list" 
   {
	NRSETS: 1
	RECORDS: {
		SET: {
			ID: 1
			TARGETS: {
				DEST: {
					URI: sip:10.214.3.20:5060;transport=udp
					FLAGS: AP
					PRIORITY: 0
					ATTRS: {
						BODY: duid=sample-cas-1;maxload=1000
						DUID: sample-cas-1
						MAXLOAD: 1000
						WEIGHT: 0
						RWEIGHT: 0
						SOCKET: 
					}
					LATENCY: {
						AVG: 2.409000
						STD: 108.999000
						EST: 1.124000
						MAX: 9503
						TIMEOUT: 22
					}
					RUNTIME: {
						DLGLOAD: 997
					}
				}
				DEST: {
					URI: sip:10.214.3.19:5060;transport=udp
					FLAGS: AP
					PRIORITY: 0
					ATTRS: {
						BODY: duid=sample-cas-0;maxload=1000
						DUID: sample-cas-0
						MAXLOAD: 1000
						WEIGHT: 0
						RWEIGHT: 0
						SOCKET: 
					}
					LATENCY: {
						AVG: 1.429000
						STD: 40.279000
						EST: 0.999000
						MAX: 3502
						TIMEOUT: 22
					}
					RUNTIME: {
						DLGLOAD: 985
					}
				}
			}
		}
	}
}

NOTE: Please see that the dispatcher shows a lot of calls stuck after overnight load, and
there by we started seeing 404's sent back by Kamailio dispatcher module for most of
the calls. 
It is noteworthy that we have kept a close tab on the tests thereafter and caught an
iteration where initially only call is stuck. 

### Possible Solutions

The problem most likely seems to be in the dispatcher module, which does not seem to be
incrementing/decrementing the load variable for each UAS( 'dload' variable to be
precise), due to which the concurrent incrementing/decrementing of the 'dload'
parameter makes the dispatcher report spurious values.

FIX: The following needs to be done.
1.   File: **dispatch.c**
        Function: **ds_load_add()**
        Just before dset->dlist[dst].dload++;  line, we must take a lock and release it
thereafter. Like:

                 **lock_get(&dset->lock)**;
                       dset->dlist[dst].dload++;
                 **lock_release(&dset->lock);**

 2: File: **dispatch.c**
       Function: **ds_load_replace()** , the code needs a lock as well while decrementing.
Like: 
       **lock_get(&idx->lock);**
        if(idx->dlist[olddst].dload > 0) {
                idx->dlist[olddst].dload--;
                **lock_release(&idx->lock);**
                  ......
        } else {
                **lock_release(&idx->lock);**
                ...... 
        }

3. File: **dispatch.c**
       Function: **ds_load_remove_byid** , the code needs a lock as well while
decrementing. Like: 

    **lock_get(&idx->lock);**
        if(idx->dlist[olddst].dload > 0) {
                idx->dlist[olddst].dload--;
                **lock_release(&idx->lock);**
                ...........
        }  else {
                **lock_release(&idx->lock);**
                ........
        }

NOTE: I believe the lock should be taken also for the the API
"ds_get_leastloaded()" in dispatch.c because otherwise, we may get select
incorrect destination, especially when we are at just about the same load and that is
almost close to the 'maxload' value configured in the dispatcher list file. But I
will wait for the community comments for this.

The above fix solved my issue, so please review the same and if you deem suitable, I could
go ahead with the fix delivery.

### Additional Information

Kamailio Version: 
[root@CPaaSVM ~]# kamailio -v
version: kamailio 5.3.2 (x86_64/linux) 7ba545
flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST,
DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY,
USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR,
USE_DST_BLACKLIST, HAVE_RESOLV_RES
ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144, MAX_URI_SIZE 1024, BUF_SIZE 65535,
DEFAULT PKG_SIZE 8MB
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
id: 7ba545
compiled on 19:01:39 May  1 2020 with gcc 4.8.5

NOTE: I do not think this is version specific and probably that it exists in the later
versions also.

* **Operating System**:
Centos 7.7

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/2322