Re: [SR-Users] Memory leak in tm with push notifications

Hello, The "delayed free" transactions are not visible using the RPC command. You would think it is something external that keeps a reference to them, but the leak stops if I remove the call to t_continue(), so it sounds like something is going on inside tm or tmx. If the INVITE times out, the transaction is freed as it should be - which makes sense, since it does not reach the t_continue() call (see above). I am running Kamailio in a VM on ESXi 5.1. I guess I will keep digging around in the source and try to trace things with gdb, this time starting from t_continue() in tmx. Greetings, Ivo On 05/29/2018 12:02 AM, Daniel-Constantin Mierla wrote: > Hello, > > >   - https://www.kamailio.org/docs/modules/stable/modules/tm.html#tm.rpc.list > > Or also with gdb if you are familiar with this tool. > > > Is this running on a virtual machine/cloud? If yes, what kind? > > Cheers, > Daniel > > > On 28.05.18 17:01, Ivaylo Markov wrote: >> Hello, >> >> I am trying to set up Kamailio as a push notifications proxy, closely >> following the example in the "Kamailio in a Mobile World" presentation >> (https://www.slideshare.net/FedericoCabiddu/kamailioinamobileworld-51617342). >> I am running Debian 9 and Kamailio 5.1.3 from the official Debian >> repositories. >> I believe the main modules involved in the issue below are tm, tmx, and >> tsilo. >> >> Every call passing through the proxy leads to a small memory leak in the tm >> module - there is a large amount of "delayed free" memory cells from tm's >> internal hash table. At some point the shared memory runs out and Kamailio >> restarts. Using the "kamcmd corex.shm_summary" command I was able to see >> that the top users of shared memory are "tm: h_table.c: build_cell" and >> "core: core/sip_msg_clone.c: sip_msg_shm_clone" with the same allocation >> count. >> >> I experimented with removing different parts of the configuration and >> noticed that commenting out the "t_continue(...)" call in the "PUSHJOIN" >> route >> (see slide #22) prevents the leak from happening. Maybe something in that >> function is incrementing the reference counter to the hash table cell, but >> it is not decrementing the counter when done? >> >> I tried looking around the source code of the tm and tmx modules, but saw >> nothing suspicious. I also tried using gdb with a breakpoint in >> t_continue_helper (tm/t_suspend.c:166) hoping to see what else is accessing >> the htable cell, but was unable to find anything of use. >> >> Has someone encountered anything like this? Can you provide more directions >> on debuggin this? I can provide some bits of configuration, but an entire >> test setup would be rather difficult, unfortunately. >> >> Thank you for your time, >> Ivo >> >> >> _______________________________________________ >> Kamailio (SER) - Users Mailing List >> sr-users(a)lists.kamailio.org >> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users