Re: [sr-dev] [kamailio] DMQ under high load kills Kamailio (#399)

I came cross across and looked into this problem a while ago but never got around to writing a patch. On start up dmq_usrloc will request a sync of all contacts from a peer. The peer sends each contact in a separate message. Each message registers a tm callback which consumes shared memory. The problem is that when sending contacts dmq_usrloc simply generates the messages as fast as possible with no limit to the number of un-replied messages. It should maintain a counter of sent but not yet replied messages (those with outstanding callbacks) and limit sending to stay below some threshold. If you allocate sufficient shared memory to fit tm callbacks for a large number of contacts you can avoid the above problem, however you will then likely hit a problem with exhausting private memory instead. Again with start-up synchronisation the peer will try and send all contacts. It currently fetches the complete list of contacts into private memory and if you have large numbers of contacts they won't fit. Private memory exhaustion could be avoided by using the partitioning mechanism in usrloc to only fetch and process a portion of the contacts at a time. The short version is both problems can be avoided by having sufficiently large memory pools as to fit all contacts in private memory and callbacks for all of them in shared memory. The real answer is that dmq_usrloc needs to manage its memory consumption by processing the registry in partitions and limiting the number of un-replied transactions.