2 Nov
2017
2 Nov
'17
1:01 p.m.
### Description
We have 3 registrar servers storing location information in memory only. The registrations
are replicated between each other using DMQ/DMQ USRLOC. When one of these registrars
processes a registration it is saved and then replicated to the remaining two nodes.
When nat pinging is enabled then all three of these registrars attempt to send options
pings to the endpoints. Ideally, only the registrar that serviced the registration should
be sending the ping out.
I see that there is a core parameter to set the server_id for each server, and I also see
that nathelper has a "filter_server_id" parameter as well, however, this appears
to only work in database mode, it does not work with in memory mode. Would it be possible
to extend this to work for in-memory mode too?
### Troubleshooting
Module definitions
registrar
```
modparam("registrar", "method_filtering", 1)
modparam("registrar", "case_sensitive", 1)
modparam("registrar", "append_branches", 0)
modparam("registrar", "use_path", 1)
modparam("registrar", "path_mode", 0)
modparam("registrar", "path_use_received", 1)
modparam("registrar", "path_check_local", 1)
modparam("registrar", "max_contacts", 1)
```
usrloc
```
modparam("usrloc", "db_mode", 0)
modparam("usrloc", "use_domain", 1)
modparam("usrloc", "timer_interval", 60)
modparam("usrloc", "timer_procs", 4)
modparam("usrloc", "nat_bflag", 6)
```
nathelper
NOTE: force_socket is set to match each registrar server
```
modparam("nathelper", "natping_interval", 20)
modparam("nathelper", "natping_processes", 4)
modparam("nathelper", "ping_nated_only", 0)
modparam("nathelper", "sipping_from",
"sip:keepalive@example.com")
modparam("nathelper", "sipping_method", "OPTIONS")
modparam("nathelper", "sipping_bflag", 6)
modparam("nathelper", "force_socket",
"10.7.0.189:5060")
modparam("nathelper", "udpping_from_path", 1)
```
Kamailio listen directives:
NOTE: These are set to match the interfaces on each registrar
```
listen=udp:10.6.0.189:5060
listen=udp:10.7.0.189:5060
listen=tcp:10.6.0.189:80
```
dmq
```
modparam("dmq", "server_address", DMQ_ADDRESS)
modparam("dmq", "notification_address", DMQ_NOTIFY_ADDRESS)
modparam("dmq", "multi_notify", 1)
modparam("dmq", "num_workers", 4)
```
dmq_usrloc
```
modparam("dmq_usrloc", "enable", 1)
```
#### Reproduction
Using the settings above, when the location information is replicated to a server, the
servers that are replicated to should not send options messages for that AOR, only the
registrar that serviced the registration should send nat keepalives out.
#### Log Messages
#### SIP Traffic
Here is the traffic from the registrar that serviced the registration, this is expected
and working:
```
U 2017/11/02 07:50:54.191717 10.7.0.190:5060 -> 10.7.0.186:5062
OPTIONS sip:example_user@212.2.172.228:39808;rinstance=d74acdb581467154;transport=UDP
SIP/2.0.
Via: SIP/2.0/UDP 10.7.0.190:5060;branch=z9hG4bK5526436.
Route: <sip:10.7.0.186:5062;lr;received=sip:212.2.172.228:39808>.
From: sip:keepalive@example.com;tag=uloc-2-59fa1f9d-711-3-9968b2da-c2c36781.
To: sip:example_user@212.2.172.228:39808;rinstance=d74acdb581467154;transport=UDP.
Call-ID: d9993f61-a1fd4752-73cdc76(a)10.7.0.190.
CSeq: 1 OPTIONS.
Content-Length: 0.
.
U 2017/11/02 07:50:54.192896 194.213.29.33:5062 -> 212.2.172.228:39808
OPTIONS sip:example_user@212.2.172.228:39808;rinstance=d74acdb581467154;transport=UDP
SIP/2.0.
Max-Forwards: 10.
Record-Route:
<sip:194.213.29.33:5062;r2=on;lr;ftag=uloc-2-59fa1f9d-711-3-9968b2da-c2c36781>.
Record-Route:
<sip:10.7.0.186:5062;r2=on;lr;ftag=uloc-2-59fa1f9d-711-3-9968b2da-c2c36781>.
Via: SIP/2.0/UDP
194.213.29.33:5062;branch=z9hG4bK13ab.9ccb0733fcecc331893d95f2e09485ee.0.
Via: SIP/2.0/UDP 10.7.0.190:5060;rport=5060;branch=z9hG4bK5526436.
From: sip:keepalive@example.com;tag=uloc-2-59fa1f9d-711-3-9968b2da-c2c36781.
To: sip:example_user@212.2.172.228:39808;rinstance=d74acdb581467154;transport=UDP.
Call-ID: d9993f61-a1fd4752-73cdc76(a)10.7.0.190.
CSeq: 1 OPTIONS.
Content-Length: 0.
.
U 2017/11/02 07:50:54.248234 212.2.172.228:39808 -> 194.213.29.33:5062
SIP/2.0 200 OK.
Via: SIP/2.0/UDP
194.213.29.33:5062;branch=z9hG4bK13ab.9ccb0733fcecc331893d95f2e09485ee.0.
Via: SIP/2.0/UDP 10.7.0.190:5060;rport=5060;branch=z9hG4bK5526436.
Record-Route:
<sip:194.213.29.33:5062;r2=on;lr;ftag=uloc-2-59fa1f9d-711-3-9968b2da-c2c36781>.
Record-Route:
<sip:10.7.0.186:5062;r2=on;lr;ftag=uloc-2-59fa1f9d-711-3-9968b2da-c2c36781>.
Contact: <sip:192.168.1.64:39808>.
To:
<sip:example_user@212.2.172.228:39808;rinstance=d74acdb581467154;transport=UDP>;tag=726ffa30.
From: sip:keepalive@example.com;tag=uloc-2-59fa1f9d-711-3-9968b2da-c2c36781.
Call-ID: d9993f61-a1fd4752-73cdc76(a)10.7.0.190.
CSeq: 1 OPTIONS.
Accept: application/sdp, application/sdp.
Accept-Language: en.
Allow: INVITE, ACK, CANCEL, BYE, NOTIFY, REFER, MESSAGE, OPTIONS, INFO, SUBSCRIBE.
Supported: replaces, norefersub, extended-refer, timer, outbound, path,
X-cisco-serviceuri.
User-Agent: Z 3.15.40006 rv2.8.20.
Allow-Events: presence, kpml, talk.
Content-Length: 0.
.
U 2017/11/02 07:50:54.248922 10.7.0.186:5062 -> 10.7.0.190:5060
SIP/2.0 200 OK.
Via: SIP/2.0/UDP 10.7.0.190:5060;rport=5060;branch=z9hG4bK5526436.
Record-Route:
<sip:194.213.29.33:5062;r2=on;lr;ftag=uloc-2-59fa1f9d-711-3-9968b2da-c2c36781>.
Record-Route:
<sip:10.7.0.186:5062;r2=on;lr;ftag=uloc-2-59fa1f9d-711-3-9968b2da-c2c36781>.
Contact: <sip:192.168.1.64:39808>.
To:
<sip:example_user@212.2.172.228:39808;rinstance=d74acdb581467154;transport=UDP>;tag=726ffa30.
From: sip:keepalive@example.com;tag=uloc-2-59fa1f9d-711-3-9968b2da-c2c36781.
Call-ID: d9993f61-a1fd4752-73cdc76(a)10.7.0.190.
CSeq: 1 OPTIONS.
Accept: application/sdp, application/sdp.
Accept-Language: en.
Allow: INVITE, ACK, CANCEL, BYE, NOTIFY, REFER, MESSAGE, OPTIONS, INFO, SUBSCRIBE.
Supported: replaces, norefersub, extended-refer, timer, outbound, path,
X-cisco-serviceuri.
User-Agent: Z 3.15.40006 rv2.8.20.
Allow-Events: presence, kpml, talk.
Content-Length: 0.
.
```
Here is a ping attempt from a server replicated to (we should not be pinging from this
registrar)
```
U 2017/11/02 08:37:59.426608 10.6.0.189:5060 -> 10.7.0.186:5062
OPTIONS sip:example_user@212.2.172.228:39808;rinstance=ed8aa63e90f53e97;transport=UDP
SIP/2.0.
Via: SIP/2.0/UDP 10.6.0.189:5060;branch=z9hG4bK8416926.
Route: <sip:10.7.0.186:5062;lr;received=sip:212.2.172.228:39808>.
From: sip:keepalive@example.com;tag=uloc-2-59fa1f9d-714-17-9968b2da-13812ff4.
To: sip:example_user@212.2.172.228:39808;rinstance=ed8aa63e90f53e97;transport=UDP.
Call-ID: c0cec5f7-555ac383-68bb313(a)10.6.0.189.
CSeq: 1 OPTIONS.
Content-Length: 0.
.
U 2017/11/02 08:38:19.431937 10.6.0.189:5060 -> 10.7.0.186:5062
OPTIONS sip:example_user@212.2.172.228:39808;rinstance=ed8aa63e90f53e97;transport=UDP
SIP/2.0.
Via: SIP/2.0/UDP 10.6.0.189:5060;branch=z9hG4bK8345318.
Route: <sip:10.7.0.186:5062;lr;received=sip:212.2.172.228:39808>.
From: sip:keepalive@example.com;tag=uloc-2-59fa1f9d-714-17-9968b2da-23812ff4.
To: sip:example_user@212.2.172.228:39808;rinstance=ed8aa63e90f53e97;transport=UDP.
Call-ID: c0cec5f7-655ac383-a9bb313(a)10.6.0.189.
CSeq: 1 OPTIONS.
Content-Length: 0.
.
```
Here is a ping attempt from the last server replicated to (we should not be pinging from
this registrar)
```
U 2017/11/02 08:53:06.927374 10.6.0.191:5060 -> 10.7.0.186:5062
OPTIONS sip:example_user@212.2.172.228:39808;rinstance=ed8aa63e90f53e97;transport=UDP
SIP/2.0.
Via: SIP/2.0/UDP 10.6.0.191:5060;branch=z9hG4bK8539212.
Route: <sip:10.7.0.186:5062;lr;received=sip:212.2.172.228:39808>.
From: sip:keepalive@example.com;tag=uloc-2-59fa1f9d-714-17-9968b2da-a25dfc84.
To: sip:example_user@212.2.172.228:39808;rinstance=ed8aa63e90f53e97;transport=UDP.
Call-ID: 89dde755-495a3694-04ae655(a)10.6.0.191.
CSeq: 1 OPTIONS.
Content-Length: 0.
.
U 2017/11/02 08:53:26.931991 10.6.0.191:5060 -> 10.7.0.186:5062
OPTIONS sip:example_user@212.2.172.228:39808;rinstance=ed8aa63e90f53e97;transport=UDP
SIP/2.0.
Via: SIP/2.0/UDP 10.6.0.191:5060;branch=z9hG4bK7581592.
Route: <sip:10.7.0.186:5062;lr;received=sip:212.2.172.228:39808>.
From: sip:keepalive@example.com;tag=uloc-2-59fa1f9d-714-17-9968b2da-b25dfc84.
To: sip:example_user@212.2.172.228:39808;rinstance=ed8aa63e90f53e97;transport=UDP.
Call-ID: 89dde755-595a3694-45ae655(a)10.6.0.191.
CSeq: 1 OPTIONS.
Content-Length: 0.
.
```
### Possible Solutions
Unknown
### Additional Information
* **Kamailio Version** - output of `kamailio -v`
```
version: kamailio 5.0.4 (x86_64/linux)
flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE,
USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC,
DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER,
USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024,
BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
id: unknown
compiled on 10:57:22 Oct 26 2017 with gcc 4.8.5
```
* **Operating System**:
```
CentOS Linux release 7.4.1708 (Core)
Linux localhost 3.10.0-693.5.2.el7.x86_64 #1 SMP Fri Oct 20 20:32:50 UTC 2017 x86_64
x86_64 x86_64 GNU/Linux
```
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1299
3 Nov
3 Nov
2:31 p.m.
I haven't developed the dmq_usrloc module, so I am not sure what the developer had in
mind (or needed), but the idea of full replication is that each node can function
independent of the others, with all features on, so if the first node goes down, the
clients still get the keepalives. By replicating to two other nodes, you get indeed three
times keepalives.
Again, speaking only from concept point of view and my opinion on that.
Maybe others can comment on this from their expectation. I know @charlesrchance does a lot
of dmq, so I am mentioning him here, but others are welcome to join the discussion.
Of course, there can be an enhancement to add an option (mod param or similar) to achieve
what you want.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1299#issuecomment-341690739
3:18 p.m.
Thanks for the info, just a quick question, if the same situation was in place when
running 3 registrar's in db-only mode, then we can use the server_id and
filter_server_id so that only 1 registrar will ping the contact. At lease, that is what I
am interpreting in the module documentation. Is that correct?
I think that if you don't have the server_id and filter_server_id parameters set for
nathelper, then all 3 database-only registrars will also ping the contact? Is that
correct? Or, does nathelper/registrar us the aor "socket" parameter to decide if
it need to send the ping from the local registrar? What i mean is, if the socket is set,
and is a local socket, then send the ping, if there is not a socket set, or, if there is a
socket set and is not local, then don't send the keep-alive message.
How does nathelper currently decide if it needs to send a keep-alive message from the
local registrar?
Currently, I'm lucky that only 1 keep-alive message is actually making it through to
the endpoint, however, if kamailio used the force_socket parameter correclty then all 3
keep-alives would make it through to the endpoint, which, I'm guessing, is not desired
behaviour, or am I wrong with that assumption?
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1299#issuecomment-341700749
4:11 p.m.
I don't think this is a DMQ issue specifically, but more a question of what should
happen in *any* replicated situation (including shared DB).
Perhaps the solution is, indeed, for nathelper to compare the received socket on the
contact against the local socket(s) and either send or not send the keepalive accordingly.
I'm not sure if this is already the case since I haven't checked the code, but I
assume not given the observed behaviour.
Alternatively, given that functionality already exists in nathelper for filtering on
server_id in DB mode, we should also add the server_id to the contact in memory and
include it in the replicated copy.
The question remains, however, in the event of a node going down, which node is going to
take over the keepalives and how will it know to do so? Again, this applies to shared DB
as much as it does in-memory/DMQ.
Either way, I'm happy to add any necessary mods to dmq/dmq_usrloc modules.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1299#issuecomment-341713760
4:42 p.m.
With regards dmq/dmq_usrloc and in-memory replication, doesn't dmq know when a node is
down/removed from the bus? If it does, then maybe a background thread could fire to
(e|se)lect one of the remaining nodes to take over from the node that is down and
manipulate/override aor's of the downed host on the (ese)lected registrar. I don't
think this would work for shared-db mode though.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1299#issuecomment-341722924
7:24 p.m.
It's a nice idea, but to implement would be quite complex/involved. Firstly, there
would likely need to be the concept of a master node along with some mechanism for
voting/election. Also, there would need to be changes across all 4 modules (dmq,
dmq_usrloc, usrloc and nathelper) to make it work. On top of that, we'd need to
consider what happens when a node is restored and takes back its IP.
In any case, to select the correct node to take over keepalives, we'd need to know
where the IP had been (re)assigned. In your current setup, how do you decide which host
takes over the IP (if at all) in the event of a failure?
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1299#issuecomment-341771689
9:08 p.m.
Currently our 3 registrar servers are not configured for fail-over at all. When we have a
planned/unplanned outage we rely on the proxies dispatcher set to route the message to an
available registrar. We also shorten the registration interval to take into account that
the nat pings wont be available for those endpoints that don't re-register on an
available registrar, although, this method isn't completely foolproof.
Once the "failed" registrar is brought back into service dmq_usrloc replicates
state back to it when it comes back and we are in service again.
On the other-hand, we do have an ha setup for the proxies involving corosync/pacemaker
where there are floating addresses assigned to each kamailio instance, but for the
registrars we don't run the ha setup.
Would it be possible to have dmq_usrloc updated to reflect what is currently done in
shared-db mode? Shared-db mode currently has some flaws in the scenarios identified in
previous posts above, but, at least, for this particular issue, it should, hopefully,
resolve the current issue at hand, whereby we have one registrar servicing the nat keep
alives for a particular endpoint.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1299#issuecomment-341799026
6 Nov
6 Nov
2:05 p.m.
Sure - are you referring to server_id filtering?
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1299#issuecomment-342129493
4:17 p.m.
Hi, yes, server_id filtering is what I was referring to.
It looks like this is the current way the shared db scheme overcomes all registrars from
pinging an available aor. I cannot see, from the module docs, if there are any other ways
to overcome this issue.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1299#issuecomment-342161313
5:53 p.m.
dmq_usrloc - server_id replication: 684059ca
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1299#issuecomment-342191338
7 Nov
7 Nov
9:52 a.m.
Thanks for the addition, do I need to recompile with this patch and test it out, or, are
there other elements that need to be updated before I can test?
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1299#issuecomment-342402437
10:34 a.m.
Yes, it is ready to use - just set the core parameter server_id on each server and enable
server_id filtering in nathelper and it will work in the same way as with shared DB.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1299#issuecomment-342411278
11:20 a.m.
Thanks again, I will recompile with this patch and update this issue with testing.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1299#issuecomment-342422293
4:56 p.m.
Hi,
I'm trying to apply this patch against the 5.0.4 source release and I'm getting
the following errors:
patch -p1 < dmq_usrloc.patch
```
patching file src/modules/dmq_usrloc/usrloc_sync.c
Hunk #1 FAILED at 323.
Hunk #2 succeeded at 412 (offset 37 lines).
Hunk #3 succeeded at 434 (offset 38 lines).
Hunk #4 FAILED at 649.
Hunk #5 FAILED at 686.
Hunk #6 succeeded at 551 (offset -181 lines).
```
I downloaded the commit with ".patch" added to the end url into the above
mentioned dmq_usrloc.patch file.
I must be missing something here, I just cannot see what it is. Any thoughts?
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1299#issuecomment-342507471
7:06 p.m.
The patch is made against the current master (development) branch - there have been some
other changes to the same file since the 5.0 branch which will prevent the patch being
applied directly to the old version.
It can be backported eventually but in the meantime, are you able to test with the
master?
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1299#issuecomment-342551796
7:54 p.m.
Hi, at the moment we cannot test with the master or 5.1.x branches as they are not
"released" yet, which is a requirement here unfortunately.
I see that Daniel mentioned in an email on the mailing lists that 5.1.x should be release
soon (couple weeks time), will this patch make it into that release? If it does then I can
test against the 5.1.0 release when it is made available.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1299#issuecomment-342566824
7:55 p.m.
Actually, I had some time on the train to modify the patch for 5.0 (attached) - please
test first and I will commit later assuming no issues.
[dmq_usrloc_patch_5_0.txt](https://github.com/kamailio/kamailio/files/145108…
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1299#issuecomment-342567096
8 p.m.
Thanks Charles, I'll try to build in the morning and get back to you tomorrow with
some results.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1299#issuecomment-342568536
9 Nov
9 Nov
11:31 a.m.
I've just updated my test registrars with 5.0.4 with the above patch, I've not
enabled server_id_filetering in nathelper yet. Is there a way I can see if the server_id
is being replicated with the aor? Will I be able to see it in the usrloc database (kamctl
ul show)?
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1299#issuecomment-343097779
11:55 a.m.
Which DB mode are you using? Can you see the server_id using `kamcmd ul.dump`?
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1299#issuecomment-343104020
12:04 p.m.
I'm using memory mode only with dmq/dmq_usrloc replication. I dont see the server_id
in the AOR dump using either kamctl or kamcmd.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1299#issuecomment-343106370
12:08 p.m.
Did you clear the records after setting server_id? It seems usrloc only stores server_id
on initial insert of a contact, not on subsequent updates/refreshes.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1299#issuecomment-343107285
12:08 p.m.
No I did not, let me try that quick
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1299#issuecomment-343107462
12:24 p.m.
hmm, running this for a while longer and I still see some pings on registrars that were
replicated to, I have the nathelper server_id filtering enabled now too. So there is still
something amiss:
nathelper parameter
```
modparam("nathelper", "filter_server_id", 1)
```
Ping being sent from registrar that was replicated to:
```
U 2017/11/09 10:16:50.143477 10.6.0.189:5060 -> 10.7.0.186:5062
OPTIONS sip:example_user@78.143.152.30:59947 SIP/2.0.
Via: SIP/2.0/UDP 10.6.0.189:5060;branch=z9hG4bK1016964.
Route: <sip:10.7.0.186:5062;lr;received=sip:78.143.152.30:59947>.
From: sip:keepalive@example.com;tag=uloc-1-5a042966-4d55-1-6821c98a-e1450563.
To: sip:example_user@78.143.152.30:59947.
Call-ID: ad921322-9fd17d71-2782516(a)10.6.0.189.
CSeq: 1 OPTIONS.
Content-Length: 0.
.
```
Could this be down to the srv_id not showing up in the usrloc database?
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1299#issuecomment-343111548
1:23 p.m.
I suspect, given that the server_id is not showing in the usrloc dump, this is exactly why
the server_id filtering is not working.
We need to understand first why server_id is not present, before moving on to the
keepalives.
I have run a test locally with two registrar nodes (server_ids 1 and 2) and two
subscribers, each registering to a different node.
Output of ul.dump on one of the nodes:
```
{
Domains: {
Domain: {
Domain: location
Size: 1024
AoRs: {
Info: {
AoR: 123456
HashID: 924664970
Contacts: {
Contact: {
Address: sip:123456@10.8.0.151:5066
Expires: 33
Q: -1.000000
Call-ID: 863771482-5066-1(a)BA.I.A.BFB
CSeq: 21177
User-Agent: Grandstream GXP2160 1.0.5.33
Received: [not set]
Path: [not set]
State: CS_NEW
Flags: 0
CFlags: 0
Socket: [not set]
Methods: 7135
Ruid: uloc-1-5a007f47-206b-11
Instance: <urn:uuid:00000000-0000-1000-8000-000B825C66C2>
Reg-Id: 4
Server-Id: 1
Tcpconn-Id: -1
Keepalive: 0
Last-Keepalive: 1510225981
Last-Modified: 1510225981
}
}
}
Info: {
AoR: 123457
HashID: 924664971
Contacts: {
Contact: {
Address: sip:123457@10.8.0.135:5066
Expires: 37
Q: -1.000000
Call-ID: 1296753078-5066-1(a)BA.I.A.BDF
CSeq: 2000
User-Agent: Grandstream GXP2100 1.0.8.4
Received: [not set]
Path: [not set]
State: CS_NEW
Flags: 0
CFlags: 0
Socket: udp:10.28.0.21:5060
Methods: 7135
Ruid: uloc-2-5a007f22-17c4-1
Instance: <urn:uuid:00000000-0000-1000-8000-000B823BB51D>
Reg-Id: 4
Server-Id: 2
Tcpconn-Id: -1
Keepalive: 0
Last-Keepalive: 1510225985
Last-Modified: 1510225985
}
}
}
}
Stats: {
Records: 2
Max-Slots: 1
}
}
}
}
```
As you can see, server_id is present in both contacts and showing the correct value(s).
Can you paste the output (or a snippet) of the same command on one of your nodes?
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1299#issuecomment-343125793
1:40 p.m.
Sure, here is an example of an aor replicated to registrars_1:
```
{
"jsonrpc": "2.0",
"result": {
"AoR": "example_user(a)example.com".com",
"Contacts": [{
"Contact": {
"Address":
"sip:example_user@212.2.172.228:43356;rinstance=4dc262b9af47682f;transport=UDP",
"Expires": 98,
"Q": -1,
"Call-ID": "4HEiFls2hkky2XR6hL8Nrg..",
"CSeq": 30,
"User-Agent": "Z 3.15.40006 rv2.8.20",
"Received": "sip:212.2.172.228:43356",
"Path":
"<sip:10.7.0.186:5062;lr;received=sip:212.2.172.228:43356>",
"State": "CS_NEW",
"Flags": 2,
"CFlags": 64,
"Socket": "[not set]",
"Methods": -1,
"Ruid": "uloc-2-5a04296d-4ed0-e1",
"Instance": "[not set]",
"Reg-Id": 0,
"Last-Keepalive": 1510227057,
"Last-Modified": 1510227057
}
}]
},
"id": 20184
}
```
This aor is being serviced by registrar_2 and I can see the server_id being sent in the
kdmq message to registrar_1, server_id is clearly set to 2:
```
KDMQ sip:usrloc@10.6.0.189:5060 SIP/2.0.
Via: SIP/2.0/UDP 10.6.0.190;branch=z9hG4bK5568.b7ee8e11000000000000000000000000.0.
To: <sip:usrloc@10.6.0.189:5060>.
From: <sip:usrloc@10.6.0.190:5060>;tag=21afb82da9b0dd18e43e09ed8956ffc8-9f49.
CSeq: 10 KDMQ.
Call-ID: 1a806e9948d055d0-20402(a)10.6.0.190.
Content-Length: 514.
User-Agent: kamailio (Registrar 2).
Max-Forwards: 1.
Content-Type: application/json.
.
{"action":1,"aor":"example_user@example.com","ruid":"uloc-2-5a04296d-4ed0-e1","c":"sip:example_user@212.2.172.228:43356;rinstance=4dc262b9af47682f;transport=UDP","received":"sip:212.2.172.228:43356","path":"<sip:10.7.0.186:5062;lr;received=sip:212.2.172.228:43356>","callid":"4HEiFls2hkky2XR6hL8Nrg..","user_agent":"Z
3.15.40006
rv2.8.20","instance":"","expires":1510227474,"cseq":34,"flags":0,"cflags":64,"q":-1,"last_modified":1510227361,"methods":4294967295,"reg_id":0,"server_id":2}
```
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1299#issuecomment-343129653
1:57 p.m.
Ok - there were some attributes missing in the usrloc rpc output in 5.0 - updated in
master, here:
https://github.com/kamailio/kamailio/commit/a57465ff47d46fb5d64c692a72f4276…
I have tested and it is fine to apply this patch to 5.0 - please apply and post again the
output afterwards.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1299#issuecomment-343133562
2:27 p.m.
ok, i'll re-build with the patch and come back here to update.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1299#issuecomment-343139712
3:26 p.m.
ok, now I can see the server_id in the usrloc dump.
```
{
"Info": {
"AoR": "example_user(a)example.com".com",
"HashID": -1389656423,
"Contacts": [{
"Contact": {
"Address":
"sip:example_user@212.2.172.228:43356;rinstance=4dc262b9af47682f;transport=UDP",
"Expires": 111,
"Q": -1,
"Call-ID": "4HEiFls2hkky2XR6hL8Nrg..",
"CSeq": 86,
"User-Agent": "Z 3.15.40006 rv2.8.20",
"Received": "sip:212.2.172.228:43356",
"Path":
"<sip:10.7.0.186:5062;lr;received=sip:212.2.172.228:43356>",
"State": "CS_NEW",
"Flags": 2,
"CFlags": 64,
"Socket": "[not set]",
"Methods": -1,
"Ruid": "uloc-2-5a04296d-4ed0-e1",
"Instance": "[not set]",
"Reg-Id": 0,
"Server-Id": 2,
"Tcpconn-Id": -1,
"Keepalive": 1,
"Last-Keepalive": 1510231303,
"Last-Modified": 1510231303
}
}]
}
}
```
Let me run this for a while longer and see if I can see any options messages being sent on
the "wrong" registrar(s). After the update and restart, I currently don't
see any "incorrect" options messages being sent, which indicates, for now, that
nathelpers filtering is working as expected.
On a side note, is the aor server_id attribute exposed to $ulc pseudo variable? So if I
wanted to see which server_id an aor was loaded on I could do a reg_fetch_contacts and
check the server_id attribute, and if an aor expired, I should be able to see the
server_id in the "usrloc:contact-expired" event route when calling, for example
"aor: $ulc(exp=>aor), server_id: $ulc(exp=>server_id)"
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1299#issuecomment-343153522
3:34 p.m.
Hmm, just seen one happen now.
```
U 2017/11/09 13:25:56.263541 10.6.0.189:5060 -> 10.7.0.186:5062
OPTIONS sip:example_user@78.143.152.30:59947 SIP/2.0.
Via: SIP/2.0/UDP 10.6.0.189:5060;branch=z9hG4bK5844925.
Route: <sip:10.7.0.186:5062;lr;received=sip:78.143.152.30:59947>.
From: sip:keepalive@example.com;tag=uloc-1-5a042966-4d55-1-6821c98a-af4da324.
To: sip:example_user@78.143.152.30:59947.
Call-ID: 8ad70ab7-8e92721-ffcfa07(a)10.6.0.189.
CSeq: 1 OPTIONS.
Content-Length: 0.
.
```
What looks like happened here is that the server_id is set to 1, but the socket is not
set, so nathelper is getting the default interface from the os and sending the ping on the
wrong interface.
Here's the aor on registrar_1:
```
{
"jsonrpc": "2.0",
"result": {
"AoR": "example_user(a)example.com".com",
"Contacts": [{
"Contact": {
"Address": "sip:example_user@78.143.152.30:59947",
"Expires": 58,
"Q": -1,
"Call-ID":
"1af6284f-59888f215eb1ff0e91850080f0808080@KX-HDV430X",
"CSeq": 1288,
"User-Agent": "BF/IE/KX-HDV430X/06.001/BCC3422AAF2C",
"Received": "sip:78.143.152.30:59947",
"Path":
"<sip:10.7.0.186:5062;lr;received=sip:78.143.152.30:59947>",
"State": "CS_NEW",
"Flags": 2,
"CFlags": 64,
"Socket": "[not set]",
"Methods": 8095,
"Ruid": "uloc-1-5a042966-4d55-1",
"Instance": "[not set]",
"Reg-Id": 0,
"Server-Id": 1,
"Tcpconn-Id": -1,
"Keepalive": 1,
"Last-Keepalive": 1510233939,
"Last-Modified": 1510233939
}
}]
},
"id": 20508
}
```
This contact is serviced by registrar_2 and it looks like it has its socket set to its
local interface, but the server_id is still that of registrar_1 so registrar_1 is trying
to ping it.
here's the aor on registrar_2 (this one "saved" the location, why is
it's server_id still 1?).
```
{
"jsonrpc": "2.0",
"result": {
"AoR": "example_user(a)example.com".com",
"Contacts": [{
"Contact": {
"Address": "sip:example_user@78.143.152.30:59947",
"Expires": 68,
"Q": -1,
"Call-ID":
"1af6284f-59888f215eb1ff0e91850080f0808080@KX-HDV430X",
"CSeq": 1290,
"User-Agent": "BF/IE/KX-HDV430X/06.001/BCC3422AAF2C",
"Received": "sip:78.143.152.30:59947",
"Path":
"<sip:10.7.0.186:5062;lr;received=sip:78.143.152.30:59947>",
"State": "CS_NEW",
"Flags": 0,
"CFlags": 64,
"Socket": "udp:10.7.0.190:5060",
"Methods": 8095,
"Ruid": "uloc-1-5a042966-4d55-1",
"Instance": "[not set]",
"Reg-Id": 0,
"Server-Id": 1,
"Tcpconn-Id": -1,
"Keepalive": 1,
"Last-Keepalive": 1510234052,
"Last-Modified": 1510234052
}
}]
},
"id": 20758
}
```
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1299#issuecomment-343155712
4:33 p.m.
This is because usrloc only stores the server_id on initial insert of a contact, not on
subsequent updates/refreshes.
The following patch should fix:
[usrloc_5_0_patch.txt](https://github.com/kamailio/kamailio/files/1457907/us…
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1299#issuecomment-343171972
5:14 p.m.
Charles thanks, I'll apply the patch and come back to you.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1299#issuecomment-343185320
10 Nov
10 Nov
11:53 a.m.
I just want to confirm that those 3 patches applied to v5.0.4 release seem to have done
the trick. I don't see any rogue keep alive messages being sent now. Thank you for
looking into this issue Charles, much appreciated.
On the question of exposing the servid_id attribute to the $ulc psuedo variable, can we do
this as part of this ticket, or shall I create a new issue for that?
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1299#issuecomment-343428419
12:52 p.m.
Thanks for testing and confirming. Two out of the three patches are in the 5.0 branch now.
I expect the third to be in today or early next week.
As for the $ulc pv, I’ll take a look later today and report back here.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1299#issuecomment-343441988
3:29 p.m.
Please try with the following:
[registrar_5_0_patch.txt](https://github.com/kamailio/kamailio/files/1461467…
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1299#issuecomment-343473500
8:49 p.m.
I have the $ulc patch active now, I am testing it with the following routing block:
```
event_route[usrloc:contact-expired] {
xlog("expired contact for $ulc(exp=>aor) server_id:
$ulc(exp=>server_id)\n");
}
```
I'll leave it running for some time and report back here.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1299#issuecomment-343555187
14 Nov
14 Nov
11:21 a.m.
I can confirm that I am able to access the server_id attribute in the ulc pvar now.
Thanks for sorting that out too :)
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1299#issuecomment-344194823
12:12 p.m.
Closed #1299.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1299#event-1340519143
12:12 p.m.
No problem, thanks for confirming. Will close this now - feel free to reopen if
necessary.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1299#issuecomment-344209615
2595
days inactive
2607
days old
38 comments
3 participants
participants (3)
-
00Asgaroth00 -
Charles Chance
-
Daniel-Constantin Mierla