[SR-Users] usrloc db_mode 2 and dropped contacts

Sat Mar 7 09:25:06 CET 2020

Well, since you have UDP endpoints, so most likely they are fixed endpoint
e.g. desk phones which almost never have network change. I know i am
assuming a lot here but this seems like a corporate network with hardware
router/firewall. The hardware routers usually have UDP NAT mapping
allocation interval of 25 seconds (e.g. CISCO ASA 55xx). It means that if
there is no UDP traffic on NATted connection for 25 seconds then router
will close the NAT mapping and SIP client will have to request for another
one when it sends next register after 60 seconds.

Also, if sip endpoint registers for 60 seconds then LCM(60, 25) = 300
seconds i.e. there will be one sip re-register (SQL UPDATE) after every 5
registers (SQL INSERT) for same endpoint due to matching_mode 0 (Contact
Only). If the SIP register interval is 120 seconds then LCM(120, 25) = 600
seconds, i.e. 1 in 12 will be SQL UPDATE. Now, since you have
timer_interval of 30 seconds [i.e. LCM(300, 30) = 300 and LCM(600, 30) =
600] so depending on sorting order of usrloc records fetched from db, the
SQL UPDATE is likely to fail (i.e. have no matching db record) and kept in
memory till next clean up.

Here are a few recommendations.

1. Enable UDP keepalives with very short intervals, e.g. 10 seconds to keep
NAT mapping at client side router open indefinitely. Set more then one
dedicated NAT ping processes if you have high number of active users
(roughly 1 process per 1K active users at peak times).

https://kamailio.org/docs/modules/4.4.x/modules/nathelper.html#nathelper.p.natping_processes

2. Enable db_check_update module option, to force insert the record in db,
this will convert orphan SQL Updates to SQL Inserts.

https://kamailio.org/docs/modules/4.4.x/modules/usrloc.html#usrloc.p.db_check_update

Or alternatively, use db_update_as_insert (somewhat more efficient),

https://kamailio.org/docs/modules/4.4.x/modules/usrloc.html#usrloc.p.db_update_as_insert

3. Enable db_clean_timer, it is kind of required option for db mode 2,
especially if sip endpoint disconnects from network and its network changes
or kamailio restarts, all so often.

https://kamailio.org/docs/modules/4.4.x/modules/usrloc.html#usrloc.p.db_timer_clean

4. Also optionally, consider using matching_mode 3 (Call-ID Only), this
solves a lot of duplication problems in SIP re-registration and SIP forking
over UDP transport.

https://kamailio.org/docs/modules/4.4.x/modules/usrloc.html#usrloc.p.matching_mode

5. Optionally, for fixed sip endpoints, it is better to set larger SIP
expiry when possible. I recommend using minimum 300 seconds and maximum 600
seconds.

Hope this help.

On Sat, Mar 7, 2020 at 7:37 AM Alex Balashov <abalashov at evaristesys.com>
wrote:

> Hi,
>
> The endpoints are just plain UDP.
>
> I don’t have an answer on #2, except to say that the majority of endpoints
> are behind NAT and therefore that their re-registration interval is quite
> low—perhaps 60-120 seconds.
>
> I don’t have a great knowledge of Kamailio modules’ startup sequence, but
> my suspicion is that a certain number of registrations might be flooding
> into SIP workers that are already available and responsive while the
> database connections are still being initialised, but before they are
> available. Or perhaps some other race condition along those general lines.
>
> That still leaves some questions surrounding the purpose and necessity of
> various settings, however.
>
> — Alex
>
> —
> Sent from mobile, with due apologies for brevity and errors.
>
> On Mar 7, 2020, at 1:28 AM, M S <shaheryarkh at gmail.com> wrote:
>
> 
> Hi,
>
> There are multiple scenarios and possibilities for this behavior, please
> check,
>
> 1. What kind of sip endpoints you have? Any TCP based transport e.g. TLS
> or WSS endpoint? Are they mobile apps (especially iOS devices) with push
> notification and "backgrounding" support?
>
> 2. How many distinct SIP re-registers (SQL Updates) you have at a time?
> Please also check if they are lower then actual endpoints.
>
> Thank you.
>
>
> On Fri, Mar 6, 2020 at 7:32 PM Alex Balashov <abalashov at evaristesys.com>
> wrote:
>
>> And another question that goes along with this:
>>
>> Do I need `db_timer_clean` to make db_mode 2 work as expected?
>>
>> A reasonable person would conclude from the documentation for
>> `timer_interval` and `db_mode 2` ...
>>
>>    "2 - Write-Back scheme. This is a combination of previous two
>>    schemes. All changes are made to memory and database synchronization
>>    is done in the timer. The timer deletes all expired contacts and
>>    flushes all modified or new contacts to database."
>>
>>    "The module uses a timer to delete expired contacts, synchronize with
>>    database and other tasks, that need to be run periodically."
>>
>> ... that deletion of expired contacts is propagated to underlying DB
>> storage (passed through the logic of whichever db_mode) in the form of
>> periodic deletions of expired contacts.
>>
>> It appears not to be enabled by default. DELETE statements appear to be
>> issued periodically from usrloc even though it's not enabled.
>>
>> So, what's that setting for? When and why do I need it?
>>
>> -- Alex
>>
>> On Fri, Mar 06, 2020 at 12:58:16PM -0500, Alex Balashov wrote:
>>
>> > Hi,
>> >
>> > On 4.4.7, since switching to db_mode 2 for usrloc with the following
>> settings:
>> >
>> >    modparam("usrloc", "db_mode", 2)
>> >    modparam("usrloc", "timer_procs", 2)
>> >    modparam("usrloc", "skip_remote_socket", 1)
>> >    modparam("usrloc", "timer_interval", 30)
>> >    modparam("usrloc", "matching_mode", 0)
>> >
>> > I seem to run into a situation where, despite starting from zero local
>> > registrations and an empty DB `location` table, out of ~1700
>> > registrations visible in the runtime usrloc table, about 60-80 never
>> > make it into the database.
>> >
>> > This is because they're being continuously UPDATE'd rather than
>> > INSERT'd, so there is some presumption that they should already be in
>> > the Postgres database. But they're not.
>> >
>> > This gap seems to very slowly grow over time.
>> >
>> > I understand that this option is intended to deal with a scenario like
>> > that:
>> >
>> >
>> https://kamailio.org/docs/modules/5.3.x/modules/usrloc.html#usrloc.p.db_check_update
>> >
>> > but the documentation doesn't say it needs to be on for db_mode 2 to
>> > work correctly, and it's off by default.
>> >
>> > I'm wondering what causes certain registrants to get into this state in
>> > the first place, where the registrar presumes that they've already been
>> > synced to the DB and only need to be update'd on renewal, where in fact
>> > they have not been synced.
>> >
>> > Causes I've ruled out:
>> >
>> > 1) The contacts being ignored due to non-local socket or other
>> > invalidation;
>> >
>> > 2) Contacts being explicitly deleted by an outside process.
>> >
>> > Any ideas appreciated!
>> >
>> > -- Alex
>> >
>> > --
>> > Alex Balashov | Principal | Evariste Systems LLC
>> >
>> > Tel: +1-706-510-6800 / +1-800-250-5920 (toll-free)
>> > Web: http://www.evaristesys.com/, http://www.csrpswitch.com/
>> >
>> > _______________________________________________
>> > Kamailio (SER) - Users Mailing List
>> > sr-users at lists.kamailio.org
>> > https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
>>
>> --
>> Alex Balashov | Principal | Evariste Systems LLC
>>
>> Tel: +1-706-510-6800 / +1-800-250-5920 (toll-free)
>> Web: http://www.evaristesys.com/, http://www.csrpswitch.com/
>>
>> _______________________________________________
>> Kamailio (SER) - Users Mailing List
>> sr-users at lists.kamailio.org
>> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
>>
> _______________________________________________
> Kamailio (SER) - Users Mailing List
> sr-users at lists.kamailio.org
> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
>
> _______________________________________________
> Kamailio (SER) - Users Mailing List
> sr-users at lists.kamailio.org
> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.kamailio.org/pipermail/sr-users/attachments/20200307/4733c831/attachment.html>