[SR-Users] Memory leak in tm with push notifications

[Ivaylo Markov]

Hello,

> could you look at those transactions and see more of their details? You
> can try with rpc command:
The "delayed free" transactions are not visible using the RPC command.
You would think it is something external that keeps a reference to them,
but the leak stops if I remove the call to t_continue(), so it sounds
like something is going on inside tm or tmx.

> Among the scopes is to figure out if the related call was completed, if
> the transaction was resumed/continued...

If the INVITE times out, the transaction is freed as it should be -
which makes sense, since it does not reach the t_continue() call (see
above).

I am running Kamailio in a VM on ESXi 5.1.

I guess I will keep digging around in the source and try to trace things
with gdb, this time starting from t_continue() in tmx.

Greetings,
Ivo

On 05/29/2018 12:02 AM, Daniel-Constantin Mierla wrote:
> Hello,
>
> could you look at those transactions and see more of their details? You
> can try with rpc command:
>
>   - https://www.kamailio.org/docs/modules/stable/modules/tm.html#tm.rpc.list
>
> Or also with gdb if you are familiar with this tool.
>
> Among the scopes is to figure out if the related call was completed, if
> the transaction was resumed/continued...
>
> Is this running on a virtual machine/cloud? If yes, what kind?
>
> Cheers,
> Daniel
>
>
> On 28.05.18 17:01, Ivaylo Markov wrote:
>> Hello,
>>
>> I am trying to set up Kamailio as a push notifications proxy, closely
>> following the example in the "Kamailio in a Mobile World" presentation
>> (https://www.slideshare.net/FedericoCabiddu/kamailioinamobileworld-51617342).
>> I am running Debian 9 and Kamailio 5.1.3 from the official Debian
>> repositories.
>> I believe the main modules involved in the issue below are tm, tmx, and
>> tsilo.
>>
>> Every call passing through the proxy leads to a small memory leak in the tm
>> module - there is a large amount of "delayed free" memory cells from tm's
>> internal hash table. At some point the shared memory runs out and Kamailio
>> restarts. Using the "kamcmd corex.shm_summary" command I was able to see
>> that the top users of shared memory are "tm: h_table.c: build_cell" and
>> "core: core/sip_msg_clone.c: sip_msg_shm_clone" with the same allocation
>> count.
>>
>> I experimented with removing different parts of the configuration and
>> noticed that commenting out the "t_continue(...)" call in the "PUSHJOIN"
>> route
>> (see slide #22) prevents the leak from happening. Maybe something in that
>> function is incrementing the reference counter to the hash table cell, but
>> it is not decrementing the counter when done?
>>
>> I tried looking around the source code of the tm and tmx modules, but saw
>> nothing suspicious. I also tried using gdb with a breakpoint in
>> t_continue_helper (tm/t_suspend.c:166) hoping to see what else is accessing
>> the htable cell, but was unable to find anything of use.
>>
>> Has someone encountered anything like this? Can you provide more directions
>> on debuggin this? I can provide some bits of configuration, but an entire
>> test setup would be rather difficult, unfortunately.
>>
>> Thank you for your time,
>> Ivo
>>
>>
>> _______________________________________________
>> Kamailio (SER) - Users Mailing List
>> sr-users at lists.kamailio.org
>> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users