Hi all,
We are using kamailio 5.1.0 as S-CSCF as follows:
We have two machines and two kamailio instances, one on each machine, both working on the same database (together they build one highly available S-CSCF).
After registering a user via the first kamailio instance, then the first instance is stopped (simulating an outage), and the HSS sends an RTR to the second instance.
However, the second instance does not find the registered user in its cache and it does not access the database --> the deregistration fails.
Configuration: kamailio-scscf/kamailio-scscf-modules.cfg:modparam("ims_usrloc_scscf", "enable_debug_file", 0) kamailio-scscf/kamailio-scscf-modules.cfg:modparam("ims_usrloc_scscf", "matching_mode", 0) kamailio-scscf/kamailio-scscf-modules.cfg:modparam("ims_usrloc_scscf", "db_url", DB_URL) kamailio-scscf/kamailio-scscf-modules.cfg:modparam("ims_usrloc_scscf", "db_mode", 2) kamailio-scscf/kamailio-scscf-modules.cfg:modparam("ims_usrloc_scscf", "timer_interval", 10) kamailio-scscf/kamailio-scscf-modules.cfg:modparam("ims_usrloc_scscf", "timer_procs", 4)
Does anyone immediately "see" a mis-configuration, or should we create an issue?
Details in the mail below.
Thanks, Christoph
From: Friedrich Peter Sent: Friday, February 16, 2018 6:38 PM To: Valentin Christoph Christoph.Valentin@kapsch.net Cc: Onic Roman Roman.Onic@kapsch.net; Habashizadeh Mahmoud Mahmoud.Habashizadeh@kapsch.net Subject: AW: Kein RTR Message nachdem SCSCF-1 outage
Hello,
following scenario at REGISTER of catt subscriber: start with empty db and empty usrloc cache
node1(.23): MariaDB [scscf]> select * from impu; Empty set (0.00 sec)
[root@vict-ims ~]# kamctl kamcmd ulscscf.showimpu sip:catt178@vict-ims.net [root@vict-ims ~]#
[root@vict-ims2 etc]# kamctl kamcmd ulscscf.showimpu sip:catt178@vict-ims.net [root@vict-ims2 etc]#
After catt REGISTER: catt_subscriber visible in DB and usrloc cache of .23 but missing in usrloc cache of .24
MariaDB [scscf]> select * from impu; +-----+---------------------------------+---------+-----------+------+------+------+------+----------------------------------------------------------------------------------------------------------------------+ | id | impu | barring | reg_state | ccf1 | ccf2 | ecf1 | ecf2 | ims_subscription_data | +-----+---------------------------------+---------+-----------+------+------+------+------+----------------------------------------------------------------------------------------------------------------------+ | 196 | sip:catt178@vict-ims.net | 0 | 1 | NULL | NULL | NULL | NULL | catt178@vict-ims.netmailto:catt178@vict-ims.net sip:catt178@vict-ims.net sip:+4350255872178@vict-ims.net tel:+4350255872178 | | 197 | sip:+4350255872178@vict-ims.net | 0 | 1 | NULL | NULL | NULL | NULL | catt178@vict-ims.netmailto:catt178@vict-ims.net sip:catt178@vict-ims.net sip:+4350255872178@vict-ims.net tel:+4350255872178 | | 198 | tel:+4350255872178 | 0 | 1 | NULL | NULL | NULL | NULL | catt178@vict-ims.netmailto:catt178@vict-ims.net sip:catt178@vict-ims.net sip:+4350255872178@vict-ims.net tel:+4350255872178 | +-----+---------------------------------+---------+-----------+------+------+------+------+----------------------------------------------------------------------------------------------------------------------+ 3 rows in set (0.00 sec)
MariaDB [scscf]>
[root@vict-ims ~]# kamctl kamcmd ulscscf.showimpu sip:catt178@vict-ims.net { impu: sip:catt178@vict-ims.net state: registered barring: 0 ccf1: ccf2: ecf1: ecf2: subscription: { impi: catt178@vict-ims.netmailto:catt178@vict-ims.net service profiles: { 1: { impus: { 1: sip:catt178@vict-ims.net 2: sip:+4350255872178@vict-ims.net 3: tel:+4350255872178 } } } } contacts: { sip:catt178@193.154.113.40:5060;transport=UDP;rinstance=317-catt178: { expires: 3542 client: n/a } } } [root@vict-ims ~]#
[root@vict-ims2 etc]# kamctl kamcmd ulscscf.showimpu sip:catt178@vict-ims.net [root@vict-ims2 etc]#
After kamctl stop + kamctl start of .24 catt_subscriber is visible in usrloc cache of (.24): preload_data loads data from db at module init
[root@vict-ims2 ~]# kamctl kamcmd ulscscf.showimpu sip:catt178@vict-ims.net { impu: sip:catt178@vict-ims.net state: registered barring: 0 ccf1: ccf2: ecf1: ecf2: subscription: { impi: catt178@vict-ims.netmailto:catt178@vict-ims.net service profiles: { 1: { impus: { 1: sip:catt178@vict-ims.net 2: sip:+4350255872178@vict-ims.net 3: tel:+4350255872178 } } } } contacts: { sip:catt178@193.154.113.40:5060;transport=UDP;rinstance=317-catt178: { expires: 2949 client: n/a } } } [root@vict-ims2 ~]#
Now deregister catt_subscriber from HSS1: catt_subscriber disappears from db and usrloc cache of (.23) but persists on (.24) as registered in usrloc cache
HSS >hss.deregister_subscriber catt178@vict-ims.netmailto:catt178@vict-ims.net 0
[root@vict-ims ~]# kamctl kamcmd ulscscf.showimpu sip:catt178@vict-ims.net [root@vict-ims ~]#
MariaDB [scscf]> select * from impu; Empty set (0.00 sec)
MariaDB [scscf]>
[root@vict-ims2 ~]# kamctl kamcmd ulscscf.showimpu sip:catt178@vict-ims.net { impu: sip:catt178@vict-ims.net state: registered barring: 0 ccf1: ccf2: ecf1: ecf2: subscription: { impi: catt178@vict-ims.netmailto:catt178@vict-ims.net service profiles: { 1: { impus: { 1: sip:catt178@vict-ims.net 2: sip:+4350255872178@vict-ims.net 3: tel:+4350255872178 } } } } contacts: { sip:catt178@193.154.113.40:5060;transport=UDP;rinstance=317-catt178: { expires: 2571 client: n/a } } } [root@vict-ims2 ~]#
Scenario of RTR from HSS after REGISTER of catt_subscriber at (.23) and outage of (.23): subscr. visible in db but not visible in usrloc cache of .24 HSS sends diameter RTR message to (.24) and deregister callback in (.24) searches in usrloc cache and does not find catt_subscriber and fails.
MariaDB [scscf]> select * from impu; +-----+---------------------------------+---------+-----------+------+------+------+------+----------------------------------------------------------------------------------------------------------------------+ | id | impu | barring | reg_state | ccf1 | ccf2 | ecf1 | ecf2 | ims_subscription_data | +-----+---------------------------------+---------+-----------+------+------+------+------+----------------------------------------------------------------------------------------------------------------------+ | 196 | sip:catt178@vict-ims.net | 0 | 1 | NULL | NULL | NULL | NULL | catt178@vict-ims.netmailto:catt178@vict-ims.net sip:catt178@vict-ims.net sip:+4350255872178@vict-ims.net tel:+4350255872178 | | 197 | sip:+4350255872178@vict-ims.net | 0 | 1 | NULL | NULL | NULL | NULL | catt178@vict-ims.netmailto:catt178@vict-ims.net sip:catt178@vict-ims.net sip:+4350255872178@vict-ims.net tel:+4350255872178 | | 198 | tel:+4350255872178 | 0 | 1 | NULL | NULL | NULL | NULL | catt178@vict-ims.netmailto:catt178@vict-ims.net sip:catt178@vict-ims.net sip:+4350255872178@vict-ims.net tel:+4350255872178 | +-----+---------------------------------+---------+-----------+------+------+------+------+----------------------------------------------------------------------------------------------------------------------+ 3 rows in set (0.00 sec)
Cfg file of (.24): kamailio-scscf/kamailio-scscf-modules.cfg:modparam("ims_usrloc_scscf", "enable_debug_file", 0) kamailio-scscf/kamailio-scscf-modules.cfg:modparam("ims_usrloc_scscf", "matching_mode", 0) kamailio-scscf/kamailio-scscf-modules.cfg:modparam("ims_usrloc_scscf", "db_url", "cluster://cluster1") kamailio-scscf/kamailio-scscf-modules.cfg:modparam("ims_usrloc_scscf", "db_url", DB_URL) kamailio-scscf/kamailio-scscf-modules.cfg:modparam("ims_usrloc_scscf", "db_mode", 2) kamailio-scscf/kamailio-scscf-modules.cfg:modparam("ims_usrloc_scscf", "timer_interval", 10) kamailio-scscf/kamailio-scscf-modules.cfg:modparam("ims_usrloc_scscf", "timer_procs", 4)
lg P
The information contained in this e-mail message is privileged and confidential and is for the exclusive use of the addressee. The person who receives this message and who is not the addressee, one of his employees or an agent entitled to hand it over to the addressee, is informed that he may not use, disclose or reproduce the contents thereof, and is kindly asked to notify the sender and delete the e-mail immediately.
Hi,
the challenge is, that Kamailio's S-CSCF-Usrloc does not support DB-Realtime like the "usual" usrloc module does. The database is always just a backup in order to survive a restart and will never be accessed, when an entry is not found in memory (PR are welcome). The Bug i see here, is that it's not clearly stated in the docs... ;-)
Thanks, Carsten
2018-02-19 13:56 GMT+01:00 Valentin Christoph Christoph.Valentin@kapsch.net:
Hi all,
We are using kamailio 5.1.0 as S-CSCF as follows:
We have two machines and two kamailio instances, one on each machine, both working on the same database (together they build one highly available S-CSCF).
After registering a user via the first kamailio instance, then the first instance is stopped (simulating an outage), and the HSS sends an RTR to the second instance.
However, the second instance does not find the registered user in its cache and it does not access the database --> the deregistration fails.
Configuration:
kamailio-scscf/kamailio-scscf-modules.cfg:modparam("ims_usrloc_scscf", "enable_debug_file", 0)
kamailio-scscf/kamailio-scscf-modules.cfg:modparam("ims_usrloc_scscf", "matching_mode", 0)
kamailio-scscf/kamailio-scscf-modules.cfg:modparam("ims_usrloc_scscf", "db_url", DB_URL)
kamailio-scscf/kamailio-scscf-modules.cfg:modparam("ims_usrloc_scscf", "db_mode", 2)
kamailio-scscf/kamailio-scscf-modules.cfg:modparam("ims_usrloc_scscf", "timer_interval", 10)
kamailio-scscf/kamailio-scscf-modules.cfg:modparam("ims_usrloc_scscf", "timer_procs", 4)
Does anyone immediately “see” a mis-configuration, or should we create an issue?
Details in the mail below.
Thanks,
Christoph
From: Friedrich Peter Sent: Friday, February 16, 2018 6:38 PM To: Valentin Christoph Christoph.Valentin@kapsch.net Cc: Onic Roman Roman.Onic@kapsch.net; Habashizadeh Mahmoud Mahmoud.Habashizadeh@kapsch.net Subject: AW: Kein RTR Message nachdem SCSCF-1 outage
Hello,
following scenario at REGISTER of catt subscriber: start with empty db and empty usrloc cache
node1(.23):
MariaDB [scscf]> select * from impu;
Empty set (0.00 sec)
[root@vict-ims ~]# kamctl kamcmd ulscscf.showimpu sip:catt178@vict-ims.net
[root@vict-ims ~]#
[root@vict-ims2 etc]# kamctl kamcmd ulscscf.showimpu sip:catt178@vict-ims.net
[root@vict-ims2 etc]#
After catt REGISTER: catt_subscriber visible in DB and usrloc cache of .23 but missing in usrloc cache of .24
MariaDB [scscf]> select * from impu;
+-----+---------------------------------+---------+-----------+------+------+------+------+----------------------------------------------------------------------------------------------------------------------+
| id | impu | barring | reg_state | ccf1 | ccf2 | ecf1 | ecf2 | ims_subscription_data |
+-----+---------------------------------+---------+-----------+------+------+------+------+----------------------------------------------------------------------------------------------------------------------+
| 196 | sip:catt178@vict-ims.net | 0 | 1 | NULL | NULL | NULL | NULL | catt178@vict-ims.net sip:catt178@vict-ims.net sip:+4350255872178@vict-ims.net tel:+4350255872178 |
| 197 | sip:+4350255872178@vict-ims.net | 0 | 1 | NULL | NULL | NULL | NULL | catt178@vict-ims.net sip:catt178@vict-ims.net sip:+4350255872178@vict-ims.net tel:+4350255872178 |
| 198 | tel:+4350255872178 | 0 | 1 | NULL | NULL | NULL | NULL | catt178@vict-ims.net sip:catt178@vict-ims.net sip:+4350255872178@vict-ims.net tel:+4350255872178 |
+-----+---------------------------------+---------+-----------+------+------+------+------+----------------------------------------------------------------------------------------------------------------------+
3 rows in set (0.00 sec)
MariaDB [scscf]>
[root@vict-ims ~]# kamctl kamcmd ulscscf.showimpu sip:catt178@vict-ims.net
{
impu: sip:catt178@vict-ims.net state: registered barring: 0 ccf1: ccf2: ecf1: ecf2: subscription: { impi: catt178@vict-ims.net service profiles: { 1: { impus: { 1: sip:catt178@vict-ims.net 2: sip:+4350255872178@vict-ims.net 3: tel:+4350255872178 } } } } contacts: {sip:catt178@193.154.113.40:5060;transport=UDP;rinstance=317-catt178: {
expires: 3542 client: n/a } }}
[root@vict-ims ~]#
[root@vict-ims2 etc]# kamctl kamcmd ulscscf.showimpu sip:catt178@vict-ims.net
[root@vict-ims2 etc]#
After kamctl stop + kamctl start of .24 catt_subscriber is visible in usrloc cache of (.24): preload_data loads data from db at module init
[root@vict-ims2 ~]# kamctl kamcmd ulscscf.showimpu sip:catt178@vict-ims.net
{
impu: sip:catt178@vict-ims.net state: registered barring: 0 ccf1: ccf2: ecf1: ecf2: subscription: { impi: catt178@vict-ims.net service profiles: { 1: { impus: { 1: sip:catt178@vict-ims.net 2: sip:+4350255872178@vict-ims.net 3: tel:+4350255872178 } } } } contacts: {sip:catt178@193.154.113.40:5060;transport=UDP;rinstance=317-catt178: {
expires: 2949 client: n/a } }}
[root@vict-ims2 ~]#
Now deregister catt_subscriber from HSS1: catt_subscriber disappears from db and usrloc cache of (.23) but persists on (.24) as registered in usrloc cache
HSS >hss.deregister_subscriber catt178@vict-ims.net 0
[root@vict-ims ~]# kamctl kamcmd ulscscf.showimpu sip:catt178@vict-ims.net
[root@vict-ims ~]#
MariaDB [scscf]> select * from impu;
Empty set (0.00 sec)
MariaDB [scscf]>
[root@vict-ims2 ~]# kamctl kamcmd ulscscf.showimpu sip:catt178@vict-ims.net
{
impu: sip:catt178@vict-ims.net state: registered barring: 0 ccf1: ccf2: ecf1: ecf2: subscription: { impi: catt178@vict-ims.net service profiles: { 1: { impus: { 1: sip:catt178@vict-ims.net 2: sip:+4350255872178@vict-ims.net 3: tel:+4350255872178 } } } } contacts: {sip:catt178@193.154.113.40:5060;transport=UDP;rinstance=317-catt178: {
expires: 2571 client: n/a } }}
[root@vict-ims2 ~]#
Scenario of RTR from HSS after REGISTER of catt_subscriber at (.23) and outage of (.23):
subscr. visible in db but not visible in usrloc cache of .24
HSS sends diameter RTR message to (.24) and deregister callback in (.24) searches in usrloc cache and does not find catt_subscriber and fails.
MariaDB [scscf]> select * from impu;
+-----+---------------------------------+---------+-----------+------+------+------+------+----------------------------------------------------------------------------------------------------------------------+
| id | impu | barring | reg_state | ccf1 | ccf2 | ecf1 | ecf2 | ims_subscription_data |
+-----+---------------------------------+---------+-----------+------+------+------+------+----------------------------------------------------------------------------------------------------------------------+
| 196 | sip:catt178@vict-ims.net | 0 | 1 | NULL | NULL | NULL | NULL | catt178@vict-ims.net sip:catt178@vict-ims.net sip:+4350255872178@vict-ims.net tel:+4350255872178 |
| 197 | sip:+4350255872178@vict-ims.net | 0 | 1 | NULL | NULL | NULL | NULL | catt178@vict-ims.net sip:catt178@vict-ims.net sip:+4350255872178@vict-ims.net tel:+4350255872178 |
| 198 | tel:+4350255872178 | 0 | 1 | NULL | NULL | NULL | NULL | catt178@vict-ims.net sip:catt178@vict-ims.net sip:+4350255872178@vict-ims.net tel:+4350255872178 |
+-----+---------------------------------+---------+-----------+------+------+------+------+----------------------------------------------------------------------------------------------------------------------+
3 rows in set (0.00 sec)
Cfg file of (.24):
kamailio-scscf/kamailio-scscf-modules.cfg:modparam("ims_usrloc_scscf", "enable_debug_file", 0)
kamailio-scscf/kamailio-scscf-modules.cfg:modparam("ims_usrloc_scscf", "matching_mode", 0)
kamailio-scscf/kamailio-scscf-modules.cfg:modparam("ims_usrloc_scscf", "db_url", "cluster://cluster1")
kamailio-scscf/kamailio-scscf-modules.cfg:modparam("ims_usrloc_scscf", "db_url", DB_URL)
kamailio-scscf/kamailio-scscf-modules.cfg:modparam("ims_usrloc_scscf", "db_mode", 2)
kamailio-scscf/kamailio-scscf-modules.cfg:modparam("ims_usrloc_scscf", "timer_interval", 10)
kamailio-scscf/kamailio-scscf-modules.cfg:modparam("ims_usrloc_scscf", "timer_procs", 4)
lg
P
The information contained in this e-mail message is privileged and confidential and is for the exclusive use of the addressee. The person who receives this message and who is not the addressee, one of his employees or an agent entitled to hand it over to the addressee, is informed that he may not use, disclose or reproduce the contents thereof, and is kindly asked to notify the sender and delete the e-mail immediately.
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Hi Carsten,
that Kamailio's S-CSCF-Usrloc does not support DB-Realtime like the "usual" usrloc module does.
Just being curious: If high availability is not supported this way by the S-CSCF-Usrloc, then what is your plan how to support high availability?
Thanks for any hints, Christoph
The information contained in this e-mail message is privileged and confidential and is for the exclusive use of the addressee. The person who receives this message and who is not the addressee, one of his employees or an agent entitled to hand it over to the addressee, is informed that he may not use, disclose or reproduce the contents thereof, and is kindly asked to notify the sender and delete the e-mail immediately.
Hi Christoph,
We are currently using a combination of the following mechanisms to provide HA (and these have been accepted by several carriers):
1) We do DNS based failover between locations according to RFC3263 (Locating SIP-Servers). E.g. if a server or datacenter goes down, the I-CSCF would re-route the traffic to the other server or datacenter 2) We run the S-CSCF as Active/Passive Node, where we restart Kamailio on the Passive node in case of single-node failure.
Nothing spectacular I guess, but it works for us.
Thanks, Carsten
2018-02-20 6:54 GMT+01:00 Valentin Christoph Christoph.Valentin@kapsch.net:
Hi Carsten,
that Kamailio's S-CSCF-Usrloc does not support DB-Realtime like the "usual" usrloc module does.
Just being curious: If high availability is not supported this way by the S-CSCF-Usrloc, then what is your plan how to support high availability?
Thanks for any hints, Christoph
The information contained in this e-mail message is privileged and confidential and is for the exclusive use of the addressee. The person who receives this message and who is not the addressee, one of his employees or an agent entitled to hand it over to the addressee, is informed that he may not use, disclose or reproduce the contents thereof, and is kindly asked to notify the sender and delete the e-mail immediately. _______________________________________________ Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Hi Carsten,
Just to be sure, let me add some questions, below:
1) We do DNS based failover between locations according to RFC3263 (Locating SIP-Servers). E.g. if a server or datacenter goes down, the I-CSCF would re-route the traffic to the other server or datacenter [CV]: Is it true, that you mean with "server" e.g. one logical S-CSCF that runs as active/passive node. That means, if one "server" goes down, the registration data and all calls are gone and the client has to register again (should happen very seldom, e.g. less than once every 5 years).
2) We run the S-CSCF as Active/Passive Node, where we restart Kamailio on the Passive node in case of single-node failure. [CV]: do you have an idea of switchover time in dependence from number of active registrations, from the instance the error occurs, until the instance the passive node has restored the data from the database (e.g. range from msec to some sec ?) - may happen several times a year
Kr, Christoph
The information contained in this e-mail message is privileged and confidential and is for the exclusive use of the addressee. The person who receives this message and who is not the addressee, one of his employees or an agent entitled to hand it over to the addressee, is informed that he may not use, disclose or reproduce the contents thereof, and is kindly asked to notify the sender and delete the e-mail immediately.
-
Carsten Bock -
Valentin Christoph