[SR-Users] Dispatcher Confusion (v3.2.0)
Asgaroth
00asgaroth00 at gmail.com
Mon Oct 24 17:30:01 CEST 2011
OK, did a few more tests but have come accross something, which I am not
sure is intended behaviour.
When setting a destingation as probing in failure route (due to
timeout), the destination still gets used in destination selection.
# ./kamailio -V
version: kamailio 3.3.0-dev0 (i386/linux) 25bedc
flags: STATS: Off, USE_IPV6, USE_TCP, USE_TLS, TLS_HOOKS, USE_RAW_SOCKS,
DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC,
DBG_QM_MALLOC, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE,
USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16,
MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 4MB
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
id: 25bedc
compiled on 09:18:41 Oct 21 2011 with gcc 4.1.2
Dispatcher module parameters (in testing) are as follows (SBC_PING_FROM
is #defined previously):
modparam("dispatcher", "flags", 2)
modparam("dispatcher", "dst_avp", "$avp(AVP_DST)")
modparam("dispatcher", "grp_avp", "$avp(AVP_GRP)")
modparam("dispatcher", "cnt_avp", "$avp(AVP_CNT)")
modparam("dispatcher", "ds_ping_method", "OPTIONS")
modparam("dispatcher", "ds_ping_from", SBC_PING_FROM)
modparam("dispatcher", "ds_ping_interval", 10)
modparam("dispatcher", "ds_probing_threshhold", 1)
modparam("dispatcher", "ds_probing_mode", 0)
Main routing logic has following snippet to select destination (hash
table selects dispatcher setid based on request domain):
if(!ds_select_dst("$sht(which_sbc=>$rd)", "0")) {
sl_send_reply("500", "No destination available");
xlog("route[MAIN] : $rm : No destinations
available for $rd");
exit;
}
Failure route has following logic to select next destination based on
timout/failure of destination:
if (t_branch_timeout() && !t_branch_replied())
{
xlog("route[TO_SBC] : $rm : timeout and no reply
($si:$sp->$Ri:$Rp->$du)\n");
xlog("route[TO_SBC] : $rm : setting $du to probing state");
ds_mark_dst("p");
if(ds_next_dst())
{
xlog("route[TO_SBC] : $rm : next destination
select ($du)\n");
t_on_failure("TO_SBC");
t_relay();
exit;
} else {
send_reply("500", "No destination available");
xlog("route[TO_SBC] : $rm : No destinations
available for $rd");
exit;
}
}
According to 3.2 module docs for dispatcher, when a destination is set
into probing state, it will not be used by ds_select_dst:
----
4.6. |ds_mark_dst("s")|
Mark the last used address from destination set as inactive
("i"/"I"/"0"), active ("a"/"A"/"1") or probing ("p"/"P"/"2"). With this
function, an automatic detection of failed gateways can be implemented.
When an address is marked as inactive or probing, it will be ignored by
'ds_select_dst' and 'ds_select_domain'.
possible parameters:
*
/"i", "I" or "0"/ - the last destination should be set to inactive
and will be ignored in future requests.
*
/"a", "A" or "1"/ - the last destination should be set to active and
the error-counter should set to "0".
*
/"p", "P" or "2"/ - the last destination will be set to probing.
Note: You will need to call this function "threshhold"-times, before
it will be actually set to probing.
This function can be used from REQUEST_ROUTE, FAILURE_ROUTE.
---
What happens here, for me, is:
[1] Gateway is in Active mode (state: AX).
[2] Request comes in and times out, destination is set to Active/Probing
(state: AP)
[3] Another request comes in and it selects gateway that is in AP mode,
times out, and then selects next dst in list.
[4] Another request comes in and it selects gateway that is in AP mode,
times out, and then selects next dst in list.
.
.
NOTE: The requests selecting the AP mode gateway may not be right after
each other (algorythm used is hash over callid) but I have stripped
those out in above steps. If I'm not mistaked, if 2 destination in a
set, and 1 destination is marked as AP, then remaining destination
should always be selected as destination to send to. The destination
marked AP (active-probing) should not be selected while in probing state.
When the gateway is set into AP mode at step [2], then, according to
docs, any new request coming in should not have the gateway selected as
it is marked as being in probing state.
Is this the intended behaviour or am I missing something in the
documentation, or is it a bug?
Thanks
On 22/10/2011 15:55, Daniel-Constantin Mierla wrote:
> Hello,
>
> yes, I will do it. Just happens to travel these days, but when I get
> the time for it, I will do it.
>
> Btw, did you test also the load balancing functionality? Was it
> affected or all is ok when you change the states? I mean when you
> disable/make inactive a gateway, is no longer used for routing, right?
> Just double checking, to be sure it was not affected by this change.
>
> Thanks,
> Daniel
>
> On 10/22/11 3:42 PM, Asgaroth wrote:
>> Hi Daniel,
>>
>> Just a reminder for this issue, to backport to 3.2.0 :)
>>
>> Thanks
>>
>> On 21/10/2011 10:53, Asgaroth wrote:
>>> Hi Daniel,
>>>
>>> It appears the change you made has fixed the issue. Below are my tests:
>>>
>>> # kamailio -V
>>> version: kamailio 3.3.0-dev0 (i386/linux) 25bedc
>>> flags: STATS: Off, USE_IPV6, USE_TCP, USE_TLS, TLS_HOOKS,
>>> USE_RAW_SOCKS,
>>> DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC,
>>> DBG_QM_MALLOC, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE,
>>> USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES
>>> ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16,
>>> MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 4MB
>>> poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
>>> id: 25bedc
>>> compiled on 09:18:41 Oct 21 2011 with gcc 4.1.2
>>>
>>> Here we have the dispatcher state as loaded from db on startup in
>>> kamailio memory.
>>>
>>> # kamctl dispatcher dump
>>> SET_NO:: 2
>>> SET:: 1
>>> URI:: sip:1.1.1.1:10001 flags=AX priority=0 attrs=
>>> URI:: sip:1.1.1.1:10000 flags=AX priority=0 attrs=
>>> SET:: 2
>>> URI:: sip:1.1.1.1:10003 flags=AX priority=0 attrs=
>>> URI:: sip:1.1.1.1:10002 flags=AX priority=0 attrs=
>>>
>>> Now try set state for destination to inactive (Worked)
>>>
>>> # kamctl fifo ds_set_state i 1 sip:1.1.1.1:10001
>>> # kamctl dispatcher dump
>>> SET_NO:: 2
>>> SET:: 1
>>> URI:: sip:1.1.1.1:10001 flags=IX priority=0 attrs=
>>> URI:: sip:1.1.1.1:10000 flags=AX priority=0 attrs=
>>> SET:: 2
>>> URI:: sip:1.1.1.1:10003 flags=AX priority=0 attrs=
>>> URI:: sip:1.1.1.1:10002 flags=AX priority=0 attrs=
>>>
>>> Now try set state for destination to active (Worked)
>>>
>>> # kamctl fifo ds_set_state a 1 sip:1.1.1.1:10001
>>> # kamctl dispatcher dump
>>> SET_NO:: 2
>>> SET:: 1
>>> URI:: sip:1.1.1.1:10001 flags=AX priority=0 attrs=
>>> URI:: sip:1.1.1.1:10000 flags=AX priority=0 attrs=
>>> SET:: 2
>>> URI:: sip:1.1.1.1:10003 flags=AX priority=0 attrs=
>>> URI:: sip:1.1.1.1:10002 flags=AX priority=0 attrs=
>>>
>>> Now try set state for destination to inactive-probing (Worked)
>>>
>>> # kamctl fifo ds_set_state ip 1 sip:1.1.1.1:10001
>>> # kamctl dispatcher dump
>>> SET_NO:: 2
>>> SET:: 1
>>> URI:: sip:1.1.1.1:10001 flags=IP priority=0 attrs=
>>> URI:: sip:1.1.1.1:10000 flags=AX priority=0 attrs=
>>> SET:: 2
>>> URI:: sip:1.1.1.1:10003 flags=AX priority=0 attrs=
>>> URI:: sip:1.1.1.1:10002 flags=AX priority=0 attrs=
>>>
>>> I waited for OPTIONS/INFO message and checked state (Worked), changed
>>> state from IP -> AX
>>>
>>> # kamctl dispatcher dump
>>> SET_NO:: 2
>>> SET:: 1
>>> URI:: sip:1.1.1.1:10001 flags=AX priority=0 attrs=
>>> URI:: sip:1.1.1.1:10000 flags=AX priority=0 attrs=
>>> SET:: 2
>>> URI:: sip:1.1.1.1:10003 flags=AX priority=0 attrs=
>>> URI:: sip:1.1.1.1:10002 flags=AX priority=0 attrs=
>>>
>>> Now try set state to active-probing (Worked)
>>>
>>> # kamctl fifo ds_set_state ap 1 sip:1.1.1.1:10001
>>> # kamctl dispatcher dump
>>> SET_NO:: 2
>>> SET:: 1
>>> URI:: sip:1.1.1.1:10001 flags=AP priority=0 attrs=
>>> URI:: sip:1.1.1.1:10000 flags=AX priority=0 attrs=
>>> SET:: 2
>>> URI:: sip:1.1.1.1:10003 flags=AX priority=0 attrs=
>>> URI:: sip:1.1.1.1:10002 flags=AX priority=0 attrs=
>>>
>>> I waited for OPTIONS/INFO message and checked state (Worked), changed
>>> state from AP -> AX
>>>
>>> # kamctl dispatcher dump
>>> SET_NO:: 2
>>> SET:: 1
>>> URI:: sip:1.1.1.1:10001 flags=AX priority=0 attrs=
>>> URI:: sip:1.1.1.1:10000 flags=AX priority=0 attrs=
>>> SET:: 2
>>> URI:: sip:1.1.1.1:10003 flags=AX priority=0 attrs=
>>> URI:: sip:1.1.1.1:10002 flags=AX priority=0 attrs=
>>>
>>> Now try set state to disabled (Worked). At this point no pinging
>>> occured
>>> as destination is admin down.
>>>
>>> # kamctl fifo ds_set_state d 1 sip:1.1.1.1:10001
>>> # kamctl dispatcher dump
>>> SET_NO:: 2
>>> SET:: 1
>>> URI:: sip:1.1.1.1:10001 flags=DX priority=0 attrs=
>>> URI:: sip:1.1.1.1:10000 flags=AX priority=0 attrs=
>>> SET:: 2
>>> URI:: sip:1.1.1.1:10003 flags=AX priority=0 attrs=
>>> URI:: sip:1.1.1.1:10002 flags=AX priority=0 attrs=
>>>
>>> Now try set state to inactive-probing again, from disabled state
>>> (Worked)
>>>
>>> # kamctl fifo ds_set_state ip 1 sip:1.1.1.1:10001
>>> # kamctl dispatcher dump
>>> SET_NO:: 2
>>> SET:: 1
>>> URI:: sip:1.1.1.1:10001 flags=IP priority=0 attrs=
>>> URI:: sip:1.1.1.1:10000 flags=AX priority=0 attrs=
>>> SET:: 2
>>> URI:: sip:1.1.1.1:10003 flags=AX priority=0 attrs=
>>> URI:: sip:1.1.1.1:10002 flags=AX priority=0 attrs=
>>>
>>> I waited for OPTIONS/INFO message and checked state (Worked), changed
>>> state from IP -> AX
>>>
>>> # kamctl dispatcher dump
>>> SET_NO:: 2
>>> SET:: 1
>>> URI:: sip:1.1.1.1:10001 flags=AX priority=0 attrs=
>>> URI:: sip:1.1.1.1:10000 flags=AX priority=0 attrs=
>>> SET:: 2
>>> URI:: sip:1.1.1.1:10003 flags=AX priority=0 attrs=
>>> URI:: sip:1.1.1.1:10002 flags=AX priority=0 attrs=
>>>
>>> Thanks.
>>>
>>> On 20/10/2011 22:54, Daniel-Constantin Mierla wrote:
>>>> Hello,
>>>>
>>>> indeed the setting of active state via MI command was wrong. Should be
>>>> fixed now in master branch. Can you test it and see if works ok now (I
>>>> had no option to test it myself yet). If all goes fine with your
>>>> tests, then I will backport.
>>>>
>>>> Guidelines to install master branch can be found at:
>>>> https://www.kamailio.org/wiki/install/devel/git
>>>>
>>>> Cheers,
>>>> Daniel
>>>>
>>>>
>>>> On 10/20/11 7:52 PM, Asgaroth wrote:
>>>>> Hi All,
>>>>>
>>>>> Just been doing some testing with Kamailio 3.2 and the dispatcher
>>>>> module, and have some strange behaviour, can we confirm if it is
>>>>> expected behaviour or a bug of some sort.
>>>>>
>>>>> It appears that I can only set the dspatcher state via fifo command
>>>>> once, and not reset it again, here is an example:
>>>>>
>>>>> version: kamailio 3.2.0 (i386/linux) 94d3b8
>>>>> flags: STATS: Off, USE_IPV6, USE_TCP, USE_TLS, TLS_HOOKS,
>>>>> USE_RAW_SOCKS,
>>>>> DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC,
>>>>> DBG_QM_MALLOC, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE,
>>>>> USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES
>>>>> ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16,
>>>>> MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 4MB
>>>>> poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
>>>>> id: 94d3b8
>>>>> compiled on 12:34:43 Oct 20 2011 with gcc 4.1.2
>>>>>
>>>>>
>>>>> Here we start with the dispatcher list as it was pulled from the
>>>>> database and currently loaded in memory.
>>>>>
>>>>> # kamctl dispatcher dump
>>>>> SET_NO:: 2
>>>>> SET:: 1
>>>>> URI:: sip:1.1.1.1:10001 flags=AX priority=0 attrs=
>>>>> URI:: sip:1.1.1.1:10000 flags=AX priority=0 attrs=
>>>>> SET:: 2
>>>>> URI:: sip:1.1.1.1:10003 flags=AX priority=0 attrs=
>>>>> URI:: sip:1.1.1.1:10002 flags=AX priority=0 attrs=
>>>>>
>>>>> Now I try to set one gateway inactive:
>>>>>
>>>>> # kamctl fifo ds_set_state i 1 sip:1.1.1.1:10001
>>>>> # kamctl dispatcher dump
>>>>> SET_NO:: 2
>>>>> SET:: 1
>>>>> URI:: sip:1.1.1.1:10001 flags=IX priority=0 attrs=
>>>>> URI:: sip:1.1.1.1:10000 flags=AX priority=0 attrs=
>>>>> SET:: 2
>>>>> URI:: sip:1.1.1.1:10003 flags=AX priority=0 attrs=
>>>>> URI:: sip:1.1.1.1:10002 flags=AX priority=0 attrs=
>>>>>
>>>>> All normal so far, now if I try to set the same gateway active again,
>>>>> the state still stays inactive, i dont seem to be able to change the
>>>>> state back to active. The same thing happens if I attempt to set this
>>>>> state to disabled and then try to set it back to active
>>>>> afterwards. The
>>>>> state does not change from what the first attempt was.
>>>>>
>>>>> # kamctl fifo ds_set_state a 1 sip:1.1.1.1:10001
>>>>> # kamctl dispatcher dump
>>>>> SET_NO:: 2
>>>>> SET:: 1
>>>>> URI:: sip:1.1.1.1:10001 flags=IX priority=0 attrs=
>>>>> URI:: sip:1.1.1.1:10000 flags=AX priority=0 attrs=
>>>>> SET:: 2
>>>>> URI:: sip:1.1.1.1:10003 flags=AX priority=0 attrs=
>>>>> URI:: sip:1.1.1.1:10002 flags=AX priority=0 attrs=
>>>>>
>>>>> Now I can try to set the state to inactive probing, that works.
>>>>>
>>>>> # kamctl fifo ds_set_state ip 1 sip:1.1.1.1:10001
>>>>> # kamctl dispatcher dump
>>>>> SET_NO:: 2
>>>>> SET:: 1
>>>>> URI:: sip:1.1.1.1:10001 flags=IP priority=0 attrs=
>>>>> URI:: sip:1.1.1.1:10000 flags=AX priority=0 attrs=
>>>>> SET:: 2
>>>>> URI:: sip:1.1.1.1:10003 flags=AX priority=0 attrs=
>>>>> URI:: sip:1.1.1.1:10002 flags=AX priority=0 attrs=
>>>>>
>>>>> Now if I try to disable the same destination:
>>>>>
>>>>> # kamctl fifo ds_set_state d 1 sip:1.1.1.1:10001
>>>>> # kamctl dispatcher dump
>>>>> SET_NO:: 2
>>>>> SET:: 1
>>>>> URI:: sip:1.1.1.1:10001 flags=IP priority=0 attrs=
>>>>> URI:: sip:1.1.1.1:10000 flags=AX priority=0 attrs=
>>>>> SET:: 2
>>>>> URI:: sip:1.1.1.1:10003 flags=AX priority=0 attrs=
>>>>> URI:: sip:1.1.1.1:10002 flags=AX priority=0 attrs=
>>>>>
>>>>> State still stays in Inactive/Probing mode.
>>>>>
>>>>> Now if I try to set the same gateway to active/probing mode, I cant:
>>>>>
>>>>> # kamctl fifo ds_set_state ap 1 sip:1.1.1.1:10001
>>>>> # kamctl dispatcher dump
>>>>> SET_NO:: 2
>>>>> SET:: 1
>>>>> URI:: sip:1.1.1.1:10001 flags=IP priority=0 attrs=
>>>>> URI:: sip:1.1.1.1:10000 flags=AX priority=0 attrs=
>>>>> SET:: 2
>>>>> URI:: sip:1.1.1.1:10003 flags=AX priority=0 attrs=
>>>>> URI:: sip:1.1.1.1:10002 flags=AX priority=0 attrs=
>>>>>
>>>>> The only way for me to get the state back to active after the
>>>>> above is
>>>>> to do a reload on the dispatch table:
>>>>>
>>>>> # kamctl dispatcher reload
>>>>> # kamctl dispatcher dump
>>>>> SET_NO:: 2
>>>>> SET:: 1
>>>>> URI:: sip:1.1.1.1:10001 flags=AX priority=0 attrs=
>>>>> URI:: sip:1.1.1.1:10000 flags=AX priority=0 attrs=
>>>>> SET:: 2
>>>>> URI:: sip:1.1.1.1:10003 flags=AX priority=0 attrs=
>>>>> URI:: sip:1.1.1.1:10002 flags=AX priority=0 attrs=
>>>>>
>>>>> Is the dispatcher expected to behave as above?
>>>>>
>>>>> I'm still trying to understand the different states assuming the
>>>>> following options are set in the configuration file:
>>>>>
>>>>> modparam("dispatcher", "ds_ping_interval", 10)
>>>>> modparam("dispatcher", "ds_probing_threshhold", 1)
>>>>> modparam("dispatcher", "ds_probing_mode", 0)
>>>>>
>>>>> AX = Active/No Probing
>>>>> (*) Gateway is used in ds_select_*, ds_next_*
>>>>> (*) No OPTIONS/INFO messages are sent to destination to
>>>>> check alive
>>>>> state
>>>>>
>>>>> This appears to be the normal operating state with no issues
>>>>> detected.
>>>>>
>>>>> AP = Active/Probing
>>>>> (*) Gateway is NOT used in ds_select_*, ds_next_*
>>>>> (*) OPTIONS/INFO messages ARE sent to destination to check
>>>>> alive
>>>>> state
>>>>>
>>>>> This appears to be the state taken when a gateway times out and
>>>>> ds_mark_dst("p") is set in failure route, the gateway is "pinged"
>>>>> while
>>>>> down and set into AX mode once a response is recieved.
>>>>>
>>>>> IX = Inactive/No Probing
>>>>> (*) Gateway is NOT used in ds_select_*, ds_next_*
>>>>> (*) No OPTIONS/INFO messages sent to destination
>>>>>
>>>>> This appears to be the state taken when a gateway times out
>>>>> and if
>>>>> ds_mark_dst("i") is set in failure route, this gateway is set
>>>>> inactive
>>>>> and no "pinging" is performed.
>>>>>
>>>>> IP = Inactive/Probing
>>>>> (*) Gateway is NOT used in ds_select_*, ds_next_*
>>>>> (*) OPTIONS/INFO messages ARE sent to destination
>>>>>
>>>>> I'm not sure when this state would be reached unless it is
>>>>> set by
>>>>> fifo command. The ds_mark_dst function only allows "a","i" or "p". So
>>>>> the probing method would only become active is ds_probing_mode =
>>>>> 1. In
>>>>> my case it is 0, so I'm not sure if I would ever reach this state
>>>>> unless
>>>>> I forced it. What would happen here is the gateway came back alive
>>>>> once
>>>>> in "IP" state, would it go back to "AX" state?
>>>>>
>>>>> DX = Disabled/No Probing
>>>>> (*) Gateway is administratively set to down
>>>>>
>>>>> This is where manual intervention has occured and the
>>>>> destination
>>>>> has been set to administrativly down, for example, work being carried
>>>>> out on the destination gateway.
>>>>>
>>>>> All these states make sense, for the most part, I'm a little
>>>>> unsure of
>>>>> how IP/IX is meant to work unless you manually set it to
>>>>> "inactive" in
>>>>> your failure route.
>>>>>
>>>>> The main concern though is the the fifo commands dont appear to be
>>>>> working as expected, I can only set the state for a destination
>>>>> manually
>>>>> once, and not again after that, unless I perform a dispatcher reload.
>>>>>
>>>>> Has anyone else experienced this behaviour?
>>>>>
>>>>> Any comments/suggestions explanations are most welcome.
>>>>>
>>>>> Thanks
>>>>>
>>>>> _______________________________________________
>>>>> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing
>>>>> list
>>>>> sr-users at lists.sip-router.org
>>>>> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
>>> _______________________________________________
>>> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
>>> sr-users at lists.sip-router.org
>>> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
>>>
>>>
>>
>> _______________________________________________
>> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
>> sr-users at lists.sip-router.org
>> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.sip-router.org/pipermail/sr-users/attachments/20111024/3732b8a9/attachment-0001.htm>
More information about the sr-users
mailing list