<p></p>
<h3 dir="auto">Description</h3>
<p dir="auto">When handling a <code>PUBLISH</code> we call <a href="https://kamailio.org/docs/modules/5.5.x/modules/presence.html#presence.f.handle_publish" rel="nofollow">handle_publish()</a> and <code>NOTIFY</code>s are sent to all of the corresponding <code>active_watchers</code> (as expected). However, when <code>NOTIFY</code>s timeout (<code>408</code>), we're seeing that the corresponding entries in the <code>active_watchers</code> table are <em>not</em> being deleted as expected. Furthermore, we've noticed that <code>NOTIFY</code>s are being sent to <code>active_watchers</code> which are expired (i.e. <code>expires < UNIX_TIMESTAMP()</code>) and when we run <code>kamcmd presence.cleanup</code>, no expired entries are removed from the <code>active_watchers</code> table.</p>
<p dir="auto">We suspect that all of these things might be related--the common theme being that records aren't deleted when expected.</p>
<h3 dir="auto">Troubleshooting</h3>
<h4 dir="auto">Reproduction</h4>
<p dir="auto">In our setup, we're using Kamailio as a "presence server" (via the <a href="https://kamailio.org/docs/modules/5.5.x/modules/presence.html" rel="nofollow">presence</a>, <a href="https://kamailio.org/docs/modules/5.5.x/modules/presence_dialoginfo.html" rel="nofollow">presence_dialoginfo</a>, and <a href="https://kamailio.org/docs/modules/5.5.x/modules/presence_xml.html" rel="nofollow">presence_xml</a> modules). We're using <a href="https://kamailio.org/docs/modules/5.5.x/modules/presence.html#presence.p.subs_db_mode" rel="nofollow"><code>subs_db_mode</code></a> <code>3</code> (DB-only scheme) and we have multiple Kamailio instances connected to a shared database (MySQL <code>8.0.27</code>).</p>
<p dir="auto">Everything seems to be working as expected. However, as we accumulated stale entries in the <code>active_watchers</code> table we're finding that we're wasting more and more time on sending <code>NOTIFY</code>s to black holes. We're generating a <em>lot</em> of traffic and waiting for the timeouts to hit is causing memory issues and backups.</p>
<p dir="auto">Here are the relevant portions of our <code>kamailio.cfg</code> file:</p>
<pre><code># ----- presence params -----
modparam("presence", "db_table_lock_type", 0) # Disable locking; MySQL has issues with this is enabled.
modparam("presence", "db_update_period", -1) # Disable synchronization.
modparam("presence", "db_url", PRESENCE_DB_URL)
modparam("presence", "expires_offset", 60) # Force the client to send an UPDATE before the old PUBLISH expires.
modparam("presence", "max_expires", 1800)
modparam("presence", "min_expires", 1700)
modparam("presence", "publ_cache", 0) # Disable the PUBLISH cache since the database is shared.
modparam("presence", "server_address", "sip:$CLUSTER_DOMAIN_NAME:5060") # This becomes the value of the Contact header.
modparam("presence", "sip_uri_match", 1) # Use case insensitive URI matching.
modparam("presence", "subs_db_mode", 3) # Database-only scheme; everything is stored in the database.
modparam("presence", "notifier_processes", 0) # Caution! Under load a race condition can cause CSeq's to be reused.
modparam("presence", "timeout_rm_subs", 1)
# ----- presence_dialoginfo params -----
modparam("presence_dialoginfo", "force_single_dialog", 1) # Maybe not all phones support multiple "dialog" elements?
modparam("presence_dialoginfo", "force_dummy_dialog", 1) # Maybe not all phones support a null body?
# ----- presence_xml params -----
modparam("presence_xml", "db_url", PRESENCE_DB_URL)
modparam("presence_xml", "force_active", 1) # Skip permission/XCAP checks.
modparam("presence_xml", "force_dummy_presence", 1) # Default to a simple "open" status when presentity info is unavailable.
# ...
route[PRESENCE] {
if (!is_method("PUBLISH|SUBSCRIBE")) {
return;
}
if (!t_newtran()) {
sl_reply_error();
exit;
}
if (is_method("PUBLISH")) {
handle_publish();
t_release();
} else if (is_method("SUBSCRIBE")) {
handle_subscribe();
t_release();
}
exit;
}
</code></pre>
<h4 dir="auto">SIP Traffic</h4>
<p dir="auto">Here's a somewhat sanitized example (the message seems OK to us; however, the <code>Subscription-State: terminated; reason=timeout</code> does make us wonder--do we as the sender <em>know</em> that the client is terminated/timed-out?):</p>
<pre><code>2022/04/05 21:09:55.209846 10.21.3.12:5060 -> 10.31.0.226:6060
NOTIFY sip:SomeUser@192.168.86.24:54639;alias=123.21.125.232~54639~1 SIP/2.0
Via: SIP/2.0/UDP presence-w.staging.internal:5060;branch=z9hG4bK43ea.648a1952000000000000000000000000.0
To: <sip:SomeOtherUser@9bfadf66-a77b-6a69-25f3-02d96d4aa946>;tag=2607596073
From: <sip:SomeUser@9bfadf66-a77b-6a69-25f3-02d96d4aa946>;tag=69309ea83adcd977af8788878e9f31b3-42e32342
CSeq: 66 NOTIFY
Call-ID: 0_2607659559@192.168.86.24
Route: <sip:10.31.0.226:6060;r2=on;lr;ftag=2607596073>, <sip:55.8.122.110;r2=on;lr;ftag=2607596073>
Content-Length: 710
Max-Forwards: 70
Event: dialog
Contact: <sip:presence-w.staging.internal:5060>
Subscription-State: terminated;reason=timeout
Content-Type: application/dialog-info+xml
<?xml version="1.0"?>
<dialog-info xmlns="urn:ietf:params:xml:ns:dialog-info" version="66" state="full" entity="sip:SomeUser@9bfadf66-a77b-6a69-25f3-02d96d4aa946">
<dialog id="0_1364146118@192.168.1.244" call-id="0_1364146118@192.168.1.244" direction="initiator">
<state>confirmed</state>
<remote>
<identity>sip:4355558565@9bfadf66-a77b-6a69-25f3-02d96d4aa945:5060</identity>
<target uri="sip:4355558565@9bfadf66-a77b-6a69-25f3-02d96d4aa946:5060"/>
</remote>
<local>
<identity>sip:SomeUser@9bfadf66-a77b-6a69-25f3-02d96d4aa946:5060</identity>
<target uri="sip:SomeUser@123.130.50.202:58872"/>
</local>
</dialog>
</dialog-info>
</code></pre>
<h3 dir="auto">Possible Solutions</h3>
<p dir="auto">We didn't see any functions in the <code>presence</code> module that we could call directly to clean things up. One thought we had was to manually run some database commands from <code>event_route[presence:notify-reply]</code> (or in a <code>reply_route</code>). We've noticed that once the problematic entries are manually removed from the database that we no longer attempt to send <code>NOTIFY</code>s to the defunct destinations.</p>
<h3 dir="auto">Additional Information</h3>
<ul dir="auto">
<li><strong>Kamailio Version</strong> - output of <code>kamailio -v</code></li>
</ul>
<pre><code>version: kamailio 5.5.4 (x86_64/linux)
flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLOCKLIST, HAVE_RESOLV_RES, TLS_PTHREAD_MUTEX_SHARED
ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
id: unknown
compiled with gcc 10.2.1
</code></pre>
<ul dir="auto">
<li><strong>Operating System</strong>:</li>
</ul>
<pre><code>$ lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description: Debian GNU/Linux 11 (bullseye)
Release: 11
Codename: bullseye
$ uname -a
Linux ip-10-21-3-12 5.10.0-13-cloud-amd64 #1 SMP Debian 5.10.106-1 (2022-03-17) x86_64 GNU/Linux
</code></pre>
<p style="font-size:small;-webkit-text-size-adjust:none;color:#666;">—<br />Reply to this email directly, <a href="https://github.com/kamailio/kamailio/issues/3074">view it on GitHub</a>, or <a href="https://github.com/notifications/unsubscribe-auth/ABO7UZJS35KYK3QPFHDYLILVDS2MTANCNFSM5SUDWWFA">unsubscribe</a>.<br />You are receiving this because you are subscribed to this thread.<img src="https://github.com/notifications/beacon/ABO7UZJ6UZLPA5RKKJDPXTDVDS2MTA5CNFSM5SUDWWFKYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4RZHOWPQ.gif" height="1" width="1" alt="" /><span style="color: transparent; font-size: 0; display: none; visibility: hidden; overflow: hidden; opacity: 0; width: 0; height: 0; max-width: 0; max-height: 0; mso-hide: all">Message ID: <span><kamailio/kamailio/issues/3074</span><span>@</span><span>github</span><span>.</span><span>com></span></span></p>
<script type="application/ld+json">[
{
"@context": "http://schema.org",
"@type": "EmailMessage",
"potentialAction": {
"@type": "ViewAction",
"target": "https://github.com/kamailio/kamailio/issues/3074",
"url": "https://github.com/kamailio/kamailio/issues/3074",
"name": "View Issue"
},
"description": "View this Issue on GitHub",
"publisher": {
"@type": "Organization",
"name": "GitHub",
"url": "https://github.com"
}
}
]</script>