Hi Kristian and Daniel. Kristian, hhanks for you feedback and patch. I'll try your patch here and will let you know the outcome soon. Thanks again guys. Cheers, -- *Nuno Miguel Reis* | *Unified Communication** Systems* M. +351 913907481 | nreis(a)wavecom.pt WAVECOM-Soluções Rádio, S.A. Cacia Park | Rua do Progresso, Lote 15 3800-639 AVEIRO | Portugal T. +351 309 700 225 | F. +351 234 919 191 *GPS <http://maps.google.com/maps/ms?msa=0&msid=202333747613191340808.0004b4b227a6144f0df88> | www.wavecom.pt <http://www.wavecom.pt/>** <http://www.wavecom.pt/>* [image: Description: Description: WavecomSignature] <http://www.wavecom.pt/pt/wavecom/premios.php> [image: Publicity] <http://www.wavecom.pt/pt/mail_eventos.php> On Tue, Jan 13, 2015 at 10:00 AM, Daniel-Constantin Mierla < miconda(a)gmail.com&gt; wrote:

Hello all. Just want to tell you that the problem remains even after the patch Daniel add to the latest 4.2. I've sent another email yesterday (partially quoted bellow) to the mailing list which was already answered by Juha Heinanen in cc regarding an odd behavior (seems odd to me) i'm seeing. I've been testing presence module for a while and I do notice that I have a small production environment with ~70 extensions and in less than 24H (~20H), the presentity table grew from 0 to about ~2000 records, pua table grew to about ~850 records and kamailio memory usage grew from 600MB to 2GB on a 4G Linux server. Yesterday Juha said and I'll quote: presence module does not do anything with calls. it handles publish and Here my publisher is Kamailio itself. Can someone elaborate a bit more on this issue and maybe we can get to bottom of it? Thanks. Cheers, -- *Nuno Miguel Reis* | *Unified Communication** Systems* M. +351 913907481 | nreis(a)wavecom.pt WAVECOM-Soluções Rádio, S.A. Cacia Park | Rua do Progresso, Lote 15 3800-639 AVEIRO | Portugal T. +351 309 700 225 | F. +351 234 919 191 *GPS <http://maps.google.com/maps/ms?msa=0&msid=202333747613191340808.0004b4b227a6144f0df88> | www.wavecom.pt <http://www.wavecom.pt/>** <http://www.wavecom.pt/>* [image: Description: Description: WavecomSignature] <http://www.wavecom.pt/pt/wavecom/premios.php> [image: Publicity] <http://www.wavecom.pt/pt/mail_eventos.php> On Thu, Jan 15, 2015 at 4:56 PM, Daniel-Constantin Mierla &lt;miconda(a)gmail.com > Hello, > > I applied the patch, with some adjustments. Now in master, to be > backported to stable branches soon. > > Cheers, > Daniel > > > On 13/01/15 20:16, Nuno Reis wrote: > > Hi Kristian and Daniel. > > Kristian, hhanks for you feedback and patch. > I'll try your patch here and will let you know the outcome soon. > Thanks again guys. > > Cheers, > > -- > > *Nuno Miguel Reis* | *Unified Communication** Systems* > M. +351 913907481 | nreis(a)wavecom.pt > WAVECOM-Soluções Rádio, S.A. > Cacia Park | Rua do Progresso, Lote 15 > 3800-639 AVEIRO | Portugal > T. +351 309 700 225 | F. +351 234 919 191 > *GPS > <http://maps.google.com/maps/ms?msa=0&msid=202333747613191340808.0004b4b227a6144f0df88> > | www.wavecom.pt <http://www.wavecom.pt/>** <http://www.wavecom.pt/>* > > [image: Description: Description: WavecomSignature] > <http://www.wavecom.pt/pt/wavecom/premios.php> > > [image: Publicity] <http://www.wavecom.pt/pt/mail_eventos.php> > > > > On Tue, Jan 13, 2015 at 10:00 AM, Daniel-Constantin Mierla < > miconda(a)gmail.com> >> Hello, >> >> thanks for the details and patch. I will try to look at later today. >> >> Cheers, >> Daniel >> >> >> On 13/01/15 08:35, Kristian F. Høgh wrote: >> >> Hi, >> >> >> >> I've been hunting a memory error in publish handling the last couple of >> days. >> >> The error is on our old but good 3.1.x presence server. >> >> Using memory debug, I located the memory leak in modules/presence/hash.c, >> function insert_phtable, line 492 (in trunk): >> >> p= (pres_entry_t*)shm_malloc(size); >> >> >> >> As far I can see there are two errors when deleting publish htable entries >> >> 1. When calling delete_phtable pres.event->type is used instead of >> pres.event->evp->type >> >> >> >> 2. When creating publish hashtable, p->publ_count is not set. (defaults >> to 0) >> >> In delete_phtable, the following code is present >> >> p->publ_count--; >> >> if(p->publ_count== 0) >> >> p->publ_count is probably decremented to -1 (unless the user have two >> active dialogs) >> >> >> >> I attach a patch, which I would carefully test in a test environment :-) >> >> >> >> Regards, >> >> Kristian Høgh >> >> Uni-tel >> >> >> >> >> >> On Monday 12 January 2015 15:39:27 Nuno Reis wrote: >> >> > Hello all. >> >> > >> >> > I'm consistently watching a memory increase in kamailio when dealing >> with >> >> > PRESENCE events, namely SIP PUBLISH events. The system eventually hangs >> >> > running out of memory. >> >> > This behavior is seen at least in kamailio 4.1 and 4.2. I'm currently >> using >> >> > the latest stable 4.2.2. >> >> > If I disable the SIP PUBLISH handling in kamailio i don't observe the >> issue >> >> > anymore but as a side effect I don't have presence (name BLFs) also. >> >> > What do you think can be the right approach here? Should I open an >> issue in >> >> > github for this? Should I run kamailio under valgrind for some time? Are >> >> > there any other possible debug hints here? >> >> > Please find attached a code snippet with the presence related parts I'm >> >> > using right now. >> >> > Looking forward to hear from you. >> >> > >> >> > Best Regards, >> >> > >> >> > -- >> >> > >> >> > *Nuno Miguel Reis* | *Unified Communication** Systems* >> >> > M. +351 913907481 <%2B351%20913907481> | nreis(a)wavecom.pt >> >> > WAVECOM-Soluções Rádio, S.A. >> >> > Cacia Park | Rua do Progresso, Lote 15 >> >> > 3800-639 AVEIRO | Portugal >> >> > T. +351 309 700 225 <%2B351%20309%20700%20225> | F. +351 234 919 191 >> >> > *GPS >> >> > >> <http://maps.google.com/maps/ms?msa=0&msid=202333747613191340808.0004b4b227a6144f0df88> >> <http://maps.google.com/maps/ms?msa=0&msid=202333747613191340808.0004b4b227a6144f0df88> >> >> > | www.wavecom.pt <http://www.wavecom.pt/> <http://www.wavecom.pt/>** >> <http://www.wavecom.pt/> <http://www.wavecom.pt/>* >> >> > >> >> > [image: Description: Description: WavecomSignature] >> >> > <http://www.wavecom.pt/pt/wavecom/premios.php> >> <http://www.wavecom.pt/pt/wavecom/premios.php> >> >> > >> >> > [image: Publicity] <http://www.wavecom.pt/pt/mail_eventos.php> >> <http://www.wavecom.pt/pt/mail_eventos.php> >> >> >> >> >> _______________________________________________ >> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing listsr-users@lists.sip-router.orghttp://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users >> >> >> -- >> Daniel-Constantin Mierlahttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda >> >> >> _______________________________________________ >> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list >> sr-users(a)lists.sip-router.org >> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users >> >> > > -- > Daniel-Constantin Mierlahttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda > >

Hi Juha and all. I understand that and that is what the RFC says. It seems pua module does that right. Although something is clearly not right in my production environment because kamailio memory consumption still grows pretty fast. Kamailio memory usage starts in ~500MB and after ~24H kamailio is using ~3GB. If I disable kamailio from listening on the localhost(127.0.0.1) where pua is generating the SIP Publishes kamailio just keeps around the ~500MB all the time. This is a small production environment with 70 extensions with Yealink phones. Any ideas on how to chase down this memory leak? Should I open a git issue for this one? -- *Nuno Miguel Reis* | *Unified Communication** Systems* M. +351 913907481 | nreis(a)wavecom.pt WAVECOM-Soluções Rádio, S.A. Cacia Park | Rua do Progresso, Lote 15 3800-639 AVEIRO | Portugal T. +351 309 700 225 | F. +351 234 919 191 *GPS <http://maps.google.com/maps/ms?msa=0&msid=202333747613191340808.0004b4b227a6144f0df88> | www.wavecom.pt <http://www.wavecom.pt/>** <http://www.wavecom.pt/>* [image: Description: Description: WavecomSignature] <http://www.wavecom.pt/pt/wavecom/premios.php> [image: Publicity] <http://www.wavecom.pt/pt/mail_eventos.php> On Wed, Jan 21, 2015 at 8:45 PM, Juha Heinanen &lt;jh(a)tutpro.com&gt; wrote:

Hi Daniel. Thanks for answering me back. I'll follow the exact procedures from here: http://www.kamailio.org/wiki/tutorials/troubleshooting/memory and will let you know about my exact finding soon. Cheers, -- *Nuno Miguel Reis* | *Unified Communication** Systems* M. +351 913907481 | nreis(a)wavecom.pt WAVECOM-Soluções Rádio, S.A. Cacia Park | Rua do Progresso, Lote 15 3800-639 AVEIRO | Portugal T. +351 309 700 225 | F. +351 234 919 191 *GPS <http://maps.google.com/maps/ms?msa=0&msid=202333747613191340808.0004b4b227a6144f0df88> | www.wavecom.pt <http://www.wavecom.pt/>** <http://www.wavecom.pt/>* [image: Description: Description: WavecomSignature] <http://www.wavecom.pt/pt/wavecom/premios.php> [image: Publicity] <http://www.wavecom.pt/pt/mail_eventos.php> On Fri, Jan 30, 2015 at 5:23 AM, Daniel-Constantin Mierla &lt;miconda(a)gmail.com > Hello, > > which memory is increasing? shared or private memory? or is system memory? > > Cheers, > Daniel > > On Fri, Jan 30, 2015 at 4:24 AM, Nuno Reis &lt;nreis(a)wavecom.pt> >> Hi Juha and all. >> >> I understand that and that is what the RFC says. It seems pua module does >> that right. Although something is clearly not right in my production >> environment because kamailio memory consumption still grows pretty fast. >> Kamailio memory usage starts in ~500MB and after ~24H kamailio is using >> ~3GB. If I disable kamailio from listening on the localhost(127.0.0.1) >> where pua is generating the SIP Publishes kamailio just keeps around the >> ~500MB all the time. >> This is a small production environment with 70 extensions with Yealink >> phones. >> Any ideas on how to chase down this memory leak? Should I open a git >> issue for this one? >> >> >> >> -- >> >> *Nuno Miguel Reis* | *Unified Communication** Systems* >> M. +351 913907481 | nreis(a)wavecom.pt >> WAVECOM-Soluções Rádio, S.A. >> Cacia Park | Rua do Progresso, Lote 15 >> 3800-639 AVEIRO | Portugal >> T. +351 309 700 225 | F. +351 234 919 191 >> *GPS >> <http://maps.google.com/maps/ms?msa=0&msid=202333747613191340808.0004b4b227a6144f0df88> >> | www.wavecom.pt <http://www.wavecom.pt/>** <http://www.wavecom.pt/>* >> >> [image: Description: Description: WavecomSignature] >> <http://www.wavecom.pt/pt/wavecom/premios.php> >> >> [image: Publicity] <http://www.wavecom.pt/pt/mail_eventos.php> >> >> >> >> On Wed, Jan 21, 2015 at 8:45 PM, Juha Heinanen &lt;jh(a)tutpro.com>> >>> Nuno Reis writes: >>> >>> > Here my publisher is Kamailio itself. Can someone elaborate a bit more >>> on >>> > this issue and maybe we can get to bottom of it? >>> >>> when your application issues initial publish request, it does so without >>> SIP-If-Match header. 200 ok from presence server then contains an etag >>> in SIP-ETag header. when your application refreshes the publish, it must >>> place this etag in SIP-If-Match header to prevent presence server from >>> creating a new publication. >>> >>> for subscribes, your application must place in re-subscribe the >>> same event header id param as the previous one had in order for the >>> presence server to know that subscribe was not a new subscription. >>> >>> -- juha >>> >> >> > > > -- > Daniel-Constantin Mierla - http://www.asipto.com > http://twitter.com/#!/miconda - http://www.linkedin.com/in/micond > <http://www.linkedin.com/in/miconda> >

Hi Daniel. Thank you for your suggestion and feedback on this. I've tried that already and here's what I've found after 15h on a running kamailio: All records in pua and presentity DB tables are gone by the end of the day, so the expire time seems to be working. I still see system memory growing and not being released again. You can find attached (tar.gz) various dumps of pkgstats and shmem usage. The number after the 'underscore' in file names corresponds to system memory allocation in kamailio at the various timestamps. The earlier timestamp corresponds to minute 1. So I started with 638632kb system mem allocation and I'm now with 1086548kb being used after 15 hours. I'll continue to investigate the issue but if you have any other suggestions on how to tackle this, I'm of course available to test those. Looking forward to hear from you. Best Regards, -- *Nuno Miguel Reis* | *Unified Communication** Systems* M. +351 913907481 | nreis(a)wavecom.pt WAVECOM-Soluções Rádio, S.A. Cacia Park | Rua do Progresso, Lote 15 3800-639 AVEIRO | Portugal T. +351 309 700 225 | F. +351 234 919 191 *GPS <http://maps.google.com/maps/ms?msa=0&msid=202333747613191340808.0004b4b227a6144f0df88> | www.wavecom.pt <http://www.wavecom.pt/>** <http://www.wavecom.pt/>* [image: Description: Description: WavecomSignature] <http://www.wavecom.pt/pt/wavecom/premios.php> [image: Publicity] <http://www.wavecom.pt/pt/mail_eventos.php> On Mon, Feb 9, 2015 at 5:05 PM, Daniel-Constantin Mierla &lt;miconda(a)gmail.com&gt; wrote:

The statistics of kamailio pkg and shm show rather steady values across the period of time. PKG is like less than 4MB for each process always and SHM stays free at 426MB. So obviously there is no leak in pkg and shm. Check the details at the next link for OS memory reports: - http://www.kamailio.org/wiki/tutorials/troubleshooting/memory#os_memory_rep… It can be unclaimed free cache what you can see as used memory. Cheers, Daniel On 11/02/15 04:28, Nuno Reis wrote: -- Daniel-Constantin Mierla http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Kamailio World Conference, May 27-29, 2015 Berlin, Germany - http://www.kamailioworld.com