[sr-dev] [kamailio] DMQ under high load kills Kamailio (#399)

[Paul Arnold]

I came cross across and looked into this problem a while ago but never
got around to writing a patch.

On start up dmq_usrloc will request a sync of all contacts from a peer. The
peer sends each contact in a separate message. Each message registers a tm
callback which consumes shared memory. The problem is that when sending
contacts dmq_usrloc simply generates the messages as fast as possible
with no limit to the number of un-replied messages. It should maintain a
counter of sent but not yet replied messages (those with outstanding
callbacks) and limit sending to stay below some threshold.

If you allocate sufficient shared memory to fit tm callbacks for a large
number of contacts you can avoid the above problem, however you will
then likely hit a problem with exhausting private memory instead. Again
with start-up synchronisation the peer will try and send all contacts. It
currently fetches the complete list of contacts into private memory and
if you have large numbers of contacts they won't fit. Private memory
exhaustion could be avoided by using the partitioning mechanism in
usrloc to only fetch and process a portion of the contacts at a time.

The short version is both problems can be avoided by having sufficiently
large memory pools as to fit all contacts in private memory and
callbacks for all of them in shared memory. The real answer is that
dmq_usrloc needs to manage its memory consumption by processing the
registry in partitions and limiting the number of un-replied
transactions.