Hello,
could you look at those transactions and see more of their details? You can try with rpc command:
The "delayed free" transactions are not visible using the RPC command. You would think it is something external that keeps a reference to them, but the leak stops if I remove the call to t_continue(), so it sounds like something is going on inside tm or tmx.
Among the scopes is to figure out if the related call was completed, if the transaction was resumed/continued...
If the INVITE times out, the transaction is freed as it should be - which makes sense, since it does not reach the t_continue() call (see above).
I am running Kamailio in a VM on ESXi 5.1.
I guess I will keep digging around in the source and try to trace things with gdb, this time starting from t_continue() in tmx.
Greetings, Ivo
On 05/29/2018 12:02 AM, Daniel-Constantin Mierla wrote:
Hello,
could you look at those transactions and see more of their details? You can try with rpc command:
- https://www.kamailio.org/docs/modules/stable/modules/tm.html#tm.rpc.list
Or also with gdb if you are familiar with this tool.
Among the scopes is to figure out if the related call was completed, if the transaction was resumed/continued...
Is this running on a virtual machine/cloud? If yes, what kind?
Cheers, Daniel
On 28.05.18 17:01, Ivaylo Markov wrote:
Hello,
I am trying to set up Kamailio as a push notifications proxy, closely following the example in the "Kamailio in a Mobile World" presentation (https://www.slideshare.net/FedericoCabiddu/kamailioinamobileworld-51617342). I am running Debian 9 and Kamailio 5.1.3 from the official Debian repositories. I believe the main modules involved in the issue below are tm, tmx, and tsilo.
Every call passing through the proxy leads to a small memory leak in the tm module - there is a large amount of "delayed free" memory cells from tm's internal hash table. At some point the shared memory runs out and Kamailio restarts. Using the "kamcmd corex.shm_summary" command I was able to see that the top users of shared memory are "tm: h_table.c: build_cell" and "core: core/sip_msg_clone.c: sip_msg_shm_clone" with the same allocation count.
I experimented with removing different parts of the configuration and noticed that commenting out the "t_continue(...)" call in the "PUSHJOIN" route (see slide #22) prevents the leak from happening. Maybe something in that function is incrementing the reference counter to the hash table cell, but it is not decrementing the counter when done?
I tried looking around the source code of the tm and tmx modules, but saw nothing suspicious. I also tried using gdb with a breakpoint in t_continue_helper (tm/t_suspend.c:166) hoping to see what else is accessing the htable cell, but was unable to find anything of use.
Has someone encountered anything like this? Can you provide more directions on debuggin this? I can provide some bits of configuration, but an entire test setup would be rather difficult, unfortunately.
Thank you for your time, Ivo
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users