Hi,
I analyzed and reproduced your issue and here's my conclusions:

1) in your scenario TSILO module is not involved nor needed. TSILO implements "deferred" parallel forking while, in your case, what you are trying to implement is serial forking with chained transaction suspensions. You can see in your logs that TSILO's functions are called just to store the transaction, never to append to it new branches.
2) Kamailio is not currently able to suspend a transaction which already received a final reply. As a proof of this in your logs, at row 6483, we can see that Kamailio sent out a 500 reply. In your case the fact that you are able to resume somehow the transaction is a race condition due to the fact that the REGISTER for the 2nd r-uri is coming when the transaction has still not been deleted. I could reproduce this exact scenario. I opened a PR with a fix (https://github.com/kamailio/kamailio/pull/955) which I successfully tested. Let's wait for Daniel's feedback. 
3) The problem of the new r-uri not taken in count while resuming the transaction is due to the fact that, in a failure route we are working on a copy of the original message and, the changes to it are not ported to the shm message clone that's used later for resuming the transaction. I don't have an easy solution for this, maybe Daniel can help :) Anyway I have a workaround for this: set and avp in the failure route with the new ruri you whish to use and use it in the resume route to set $ru for the message. Somethig like:

route[INVRESUME] {
    ....
    if (is_avp_set("$avp(ru)"))
               $ru = $avp(ru);
  .....
}

With the above patch on tm and this workaround I could successfully implement your scenario.

Best regards,

Federico


On Tue, Jan 24, 2017 at 10:16 AM, Tomas Zanet <tzanet@came.com> wrote:
Hi Federico, sorry you’re right about log level.
Kamailio was running with right debug level but my syslog was configured to split log messages based on level so all logs with debug went to another file called debug.
I reconfigured syslog, please see the new attached file.

I should have found the log lines where there's $ru overwriting, should be the following log lines:
2017-01-24T08:39:30+00:00 server-xip-99 /usr/local/sbin/kamailio[26021]: ERROR: <script>: resuming trasaction [58698:23306391] 00......02181 58698:23306391)sip:7ffe85fb456f629b......net
2017-01-24T08:39:30+00:00 server-xip-99 /usr/local/sbin/kamailio[26021]: ERROR: *** cfgtrace:request_route=[PUSHJOIN] c=[/usr/local/etc/kamailio/kamailio.cfg] l=1011 a=27 n=t_continue
2017-01-24T08:39:30+00:00 server-xip-99 /usr/local/sbin/kamailio[26021]: DEBUG: tm [t_lookup.c:1574]: t_lookup_ident(): DEBUG: t_lookup_ident: transaction found
2017-01-24T08:39:30+00:00 server-xip-99 /usr/local/sbin/kamailio[26021]: ERROR: *** cfgtrace:failure_route=[INVRESUME] c=[/usr/local/etc/kamailio/kamailio.cfg] l=1016 a=26 n=xlog
2017-01-24T08:39:30+00:00 server-xip-99 /usr/local/sbin/kamailio[26021]: ALERT: <script>: Invite remused for sip:00......00000@7FFE85FB456F629B........net
2017-01-24T08:39:30+00:00 server-xip-99 /usr/local/sbin/kamailio[26021]: ERROR: *** cfgtrace:failure_route=[INVRESUME] c=[/usr/local/etc/kamailio/kamailio.cfg] l=1017 a=25 n=lookup

In fact, after resuming the transaction with right 2181 account, it's called  t_continue and then t_lookup_ident funcs which says " transaction found", but maybe the transaction is not the right one...
But I don't know why that happens.

I also tried to add append_branch() function as you suggested but with no luck…

I don’t know why but when I copied and pasted the code statements, all branches disappeared…but they are in the Kamailio.cfg....
Let me retry changing email body text format.
# USER location service
route[LOCATION] {

#!ifdef WITH_SPEEDDIAL
        # search for short dialing - 2-digit extension
        if($rU=~"^[0-9][0-9]$")
                if(sd_lookup("speed_dial"))
                        route(SIPOUT);
#!endif

#!ifdef WITH_ALIASDB
        # search in DB-based aliases
        if(alias_db_lookup("dbaliases"))
                route(SIPOUT);
#!endif
#!ifdef WITH_FORK
        if($rU == "9999") {
                route(FORK);
        }
#!endif

#!ifdef WITH_PDT
        xdbg("before pdt $rU\n");
        #prefix2domain("0", "1");
        xdbg("after pdt $rU\n");
#!endif
        #Check if domains match

        if($fd != $rd) {
                xdbg("From domain $fd does not match request uri domain $rd\n");
                send_reply("488", "Not Accetable Here");
                exit;
        }

        #$rd = $td
        $avp(oexten) = $rU;
        if (!lookup("location")) {
                $var(rc) = $rc;
                t_newtran();
                switch ($var(rc)) {
                        case -1:
                        case -3:
                                xlog("L_ALERT","ALERT: User not found\n");
                                if (isflagset(FLT_ASTERISK_TRY)) {
                                        # Try calling with PUSH KIT
                                        send_reply("404", "Not Found");
                                } else {
                                        # User not found, check if is an app, then call it with pushkit
                                        if($(rU{s.substr,0,6})=="007001") {
                                                xlog("L_ALERT","ALERT: Trying calling $ru with PUSUH KIT\n");
                                                # device token check missing
                                                route(SUSPEND);
                                                route(SENDPUSH);
                                        } else {
                                                setflag(FLT_ASTERISK_TRY);
                                                route(ASTERISK);
                                        }
                                }
                                exit;
                        case -2:
                                send_reply("405", "Method Not Allowed");
                                exit;
                }
        }

        # when routing via usrloc, log the missed calls also
        if (is_method("INVITE"))
                setflag(FLT_ACCMISSED);

        route(RELAY);
        route(STORETRAN);
        exit;
}

route[STORETRAN] {
        ts_store();
        $sht(vtp=>stored::$rU) = 1;
        xlog("L_ALERT", "stored transaction [$T(id_index):$T(id_label)] $fU => $rU\n");
}

# manage failure routing cases
failure_route[MANAGE_FAILURE] {
        xlog("L_ALERT", "MANAGE_FAILURE $rm $ru\n");
        route(NATMANAGE);

        if (t_is_canceled()) {
                xlog("L_ALERT", "MANAGE_FAILURE CANCELED\n");
                exit;
        }

#!ifdef WITH_BLOCK3XX
        # block call redirect based on 3xx replies.
        if (t_check_status("3[0-9][0-9]")) {
                t_reply("404","Not found");
                exit;
        }
#!endif

       if (is_avp_set("$avp(group_members_db)")) {
                $ru ="sip:"+$avp(group_members_db)+"@"+$fd;
                #append_branch();
                #$ru = $avp(group_members);
                $avp(group_members_db) = $null;
                t_on_failure("MANAGE_FAILURE");
        xlog("L_ALERT", "MANAGE_FAILURE $rm $ru $tu $fu\n");
                route(STORETRAN);
                route(LOCATION);
                exit;
        }
}



Tomas Zanet
Software Design Department
tzanet@came.com
CAME S.p.A.


Da: sr-users [mailto:sr-users-bounces@lists.sip-router.org] Per conto di Federico Cabiddu
Inviato: martedì 24 gennaio 2017 07:57
A: Kamailio (SER) - Users Mailing List <sr-users@lists.sip-router.org>
Oggetto: Re: [SR-Users] R: serial forking calls using TSILO

Hi,
the logs are not at level 3 (there are no DEBUG messages). Could you please repeat the test?
In the meanwhile can you also try adding a branch after setting the r-uri in the failure route?

$ru ="sip:"+$avp(group_members_db)+"@"+$fd;
append_branch();

I've also noticed that, in the failure route, the statements after

if (is_avp_set("$avp(group_members_db)"))

are not enclosed in braces, is this intentional?

Regards,

Federico

On Mon, Jan 23, 2017 at 10:01 AM, Tomas Zanet <tzanet@came.com> wrote:
Hi, please find below what you requested.

@Daniel: yes, I did ts_store() after relaying for both hops, but for the second call which should call account2, $ru is overwritten to account1 after t_continue

@Federico: as you requested, here is my route[LOCATION] and please the attached Kamailio log.
# USER location service
route[LOCATION]

#!ifdef WITH_SPEEDDIAL
        # search for short dialing - 2-digit extension
        if($rU=~"^[0-9][0-9]$")
                if(sd_lookup("speed_dial"))
                        route(SIPOUT);
#!endif

#!ifdef WITH_ALIASDB
        # search in DB-based aliases
        if(alias_db_lookup("dbaliases"))
                route(SIPOUT);
#!endif

#!ifdef WITH_FORK
        if($rU == "9999")
                route(FORK);

#!endif

#!ifdef WITH_PDT
        xdbg("before pdt $rU\n");
        #prefix2domain("0", "1");
        xdbg("after pdt $rU\n");
#!endif
        #Check if domains match

        if($fd != $rd)
                xdbg("From domain $fd does not match request uri domain $rd\n");
                send_reply("488", "Not Accetable Here");
                exit;


        #$rd = $td
        $avp(oexten) = $rU;
        if (!lookup("location"))
                $var(rc) = $rc;
                t_newtran();
                switch ($var(rc))
                        case -1:
                        case -3:
                                xlog("L_ALERT","ALERT: User not found\n");
                                if (isflagset(FLT_ASTERISK_TRY))
                                        # Try calling with PUSH KIT
                                        send_reply("404", "Not Found");
                                 else
                                        # User not found, check if is an app based on uri prefix, then call it with pushkit
                                        if($(rUs.substr,0,6)=="007001")
                                                xlog("L_ALERT","ALERT: Trying calling $ru with PUSUH KIT\n");
                                                # device token check missing
                                                route(SUSPEND);
                                                route(SENDPUSH);
                                         else
                                                setflag(FLT_ASTERISK_TRY);
                                                route(ASTERISK);


      exit;
                        case -2:
                                send_reply("405", "Method Not Allowed");
                                exit;



        # when routing via usrloc, log the missed calls also
        if (is_method("INVITE"))
                setflag(FLT_ACCMISSED);

        route(RELAY);
        route(STORETRAN);
        exit;


route[STORETRAN]
        ts_store();
        $sht(vtp=>stored::$rU) = 1;
        xlog("L_ALERT", "stored transaction [$T(id_index):$T(id_label)] $fU => $rU\n");



Thanks to both,
Regards,
T.

Da: sr-users [mailto:sr-users-bounces@lists.sip-router.org] Per conto di Federico Cabiddu
Inviato: domenica 22 gennaio 2017 16:24
A: Kamailio (SER) - Users Mailing List <sr-users@lists.sip-router.org>
Oggetto: Re: [SR-Users] serial forking calls using TSILO

Hi,
could you please provide the LOCATION route and the Kamailio's log at level 3?
Thank you.

Regards,

Federico

On Fri, Jan 20, 2017 at 11:05 AM, Tomas Zanet <tzanet@came.com> wrote:
Hello,
I’m using Kamailio 4.4.4 with TSILO module in order to support Push Notifications used by our voip app on Apple ios10.
So far, everything works fine:  Kamailio can get an incoming call, suspend it, then send push notification with an external script, receive a new app registration and then call it after resuming the invite.
To do so, thanks to Mr, Cabiddu, I used all the functions described here
http://www.kamailio.org/events/2015-KamailioWorld/Day2/20-Federico.Cabiddu-Kamailio-In-A-Mobile-World.pdf


Right now, I would like to support serial forking calls using push notification because it was already supported with legacy voip.
To do so, I used failure_route function where, after getting the call destinations and setting them into an avp, I set the new SIP request uri $ru.
Here is my failure_route:
failure_route[MANAGE_FAILURE]
        route(NATMANAGE);
        if (t_is_canceled())
                exit;


#!ifdef WITH_BLOCK3XX
        # block call redirect based on 3xx replies.
        if (t_check_status("3[0-9][0-9]"))
                t_reply("404","Not found");
                exit;

#!endif

       if (is_avp_set("$avp(group_members_db)"))
                $ru ="sip:"+$avp(group_members_db)+"@"+$fd;
                $avp(group_members_db) = $null;
                route(LOCATION);
                exit;



I am troubleshooting this scenario and this is what I see:
1) Kamailio receives incoming call
2) Suspend it
3) Send Push notification to account1
4) Kamailio receives account1 registration -> INVRESUME route is now called
5) Then call account1
6) Nobody answers the call
7) failure_route[MANAGE_FAILURE] is now called -> set $ru with next sipaccount: account2
8) suspend invite
9) send push notification to account2
10) account2 sends its sip registration -> INVRESUME route is now called
11) Kamailio calls account1 instead of account2 -> this is my issue

On step 10: despite account2 is registered, I checked it using “kamctl ul show”, the second call is forwarded all'account1 and not to account2.
I also checked the TSILO logs and everything seems ok. Here is the logs:
First call:
      suspended transaction [53945:1648394094] asterisk => account1
      htable key value [53945:1648394094]
      resuming trasaction [53945:1648394094] account1 53945:1648394094)
second call:
      suspended transaction [53945:1648394094] asterisk => account2
      htable key value [53945:1648394094]
      resuming trasaction [53945:1648394094] account2 53945:1648394094)

In order to find out the issue I put some xlogs, printing $ru value: what I see is the $ru value is set correctly on failure_route but as soon as the t_continue is called, the ru overwritten back to account1

Can anyone address me to find out the solution?
Thanks in advance

T.



Tomas Zanet
Software Design Department
tzanet@came.com



_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users@lists.sip-router.org
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users



Tomas Zanet
Software Design Department
tzanet@came.com



_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users@lists.sip-router.org
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users


_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users@lists.sip-router.org
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users