[Users] memory issues

Ovidiu Sas sip.nslu at gmail.com
Fri Mar 23 16:04:27 CET 2007


Hi Daniel,

Can we backport this one to 1.2?


Regards,
Ovidiu Sas

On 3/22/07, Daniel-Constantin Mierla <daniel at voice-system.ro> wrote:
> Hello,
>
> the supposed fragmentation turned out to be a mem leak in pkg. Please
> take the latest SVN version and try again to see if you got same results.
>
> Thanks,
> Daniel
>
> On 03/19/07 18:52, Christian Schlatter wrote:
> > ...
> >>> The memory statistics indeed show a high number of memory fragments:
> >>>
> >>> before 'out of memory':
> >>>
> >>> shmem:total_size = 536870912
> >>> shmem:used_size = 59607040
> >>> shmem:real_used_size = 60106488
> >>> shmem:max_used_size = 68261536
> >>> shmem:free_size = 476764424
> >>> shmem:fragments = 9897
> >>>
> >>> after 'out of memory' (about 8000 calls per process):
> >>>
> >>> shmem:total_size = 536870912
> >>> shmem:used_size = 4171160
> >>> shmem:real_used_size = 4670744
> >>> shmem:max_used_size = 68261536
> >>> shmem:free_size = 532200168
> >>> shmem:fragments = 57902
> >>>
> >>>>
> >>>> You can try to compile openser with -DQM_JOIN_FREE (add it in DEFS
> >>>> variable of Makefile.defs) and test again. Free fragments should be
> >>>> merged and fragmentation should not occur -- processing will be
> >>>> slower. We will try for next release to provide a better solution
> >>>> for that.
> >>>
> >>> Compiling openser with -DQM_JOIN_FREE did not help. I'm not sure how
> >>> big of a problem this fragmentation issue is.
> >> What is the number of fragments with QM_JOIN_FREE after flooding?
> >
> > The numbers included above are with QM_JOIN_FREE enabled.
> >
> >>> Do you think it would make sense to restart our production openser
> >>> instances from time to time just to make sure they're not running
> >>> into this memory fragmentation limits?
> >> The issue will occur only when the call rate reaches the limits of
> >> the proxy's memory. Otherwise the chunks are reused. Transactions and
> >> avps are rounded up to be sure there will be minimized the number of
> >> different sizes for memory chunks. It wasn't reported too often,
> >> maybe that's why no big attention was paid to it. This memory system
> >> is in place since the beginning of ser. Alternative is to use sysv
> >> shared memory, but is much slower, along with libc private memory
> >> manager.
> >
> > I've done some more testing and the same out-of-memory stuff happens
> > when I run sipp with 10 calls per second only. I tested with
> > 'children=1' and I only could get through about 8200 calls (again
> > those 8000 calls / process). And this is with QM_JOIN_FREE enabled.
> >
> > Memory statistics:
> >
> > before:
> > shmem:total_size = 536870912
> > shmem:used_size = 2311976
> > shmem:real_used_size = 2335720
> > shmem:max_used_size = 2465816
> > shmem:free_size = 534535192
> > shmem:fragments = 183
> >
> > after:
> > shmem:total_size = 536870912
> > shmem:used_size = 1853472
> > shmem:real_used_size = 1877224
> > shmem:max_used_size = 2465816
> > shmem:free_size = 534993688
> > shmem:fragments = 547
> >
> > So I'm not sure if this is really a fragmentation issue. 10 cps surely
> > doesn't reach the proxy's memory.
> >
> > Thoughts?
> >
> > Christian
> >
> >
> >
> >> Cheers,
> >> Daniel
> >>
> >>>
> >>> thanks,
> >>> Christian
> >>>
> >>>>
> >>>> Cheers,
> >>>> Daniel
> >>>>
> >>>> On 03/18/07 01:21, Christian Schlatter wrote:
> >>>>> Christian Schlatter wrote:
> >>>>> ...
> >>>>>>
> >>>>>> I always had 768MB shared memory configured though, so I still
> >>>>>> can't explain the memory allocation errors I got. Some more test
> >>>>>> runs revealed that I only get these errors when using a more
> >>>>>> production oriented config that loads more modules than the one
> >>>>>> posted in my earlier email. I now try to figure out what exactly
> >>>>>> causes these memory allocation errors that happen reproducibly
> >>>>>> after about 220s at 400 cps.
> >>>>>
> >>>>> I think I found the cause for the memory allocation errors. As
> >>>>> soon as I include an AVP write operation in the routing script, I
> >>>>> get 'out of memory' messages after a certain number of calls
> >>>>> generated with sipp.
> >>>>>
> >>>>> The routing script to reproduce this behavior looks like (full
> >>>>> config available at
> >>>>> http://www.unc.edu/~cschlatt/openser/openser.cfg):
> >>>>>
> >>>>> route{
> >>>>>         $avp(s:ct) = $ct; # commenting this line solves
> >>>>>               # the memory problem
> >>>>>
> >>>>>         if (!method=="REGISTER") record_route();
> >>>>>         if (loose_route()) route(1);
> >>>>>
> >>>>>         if (uri==myself) rewritehost("xx.xx.xx.xx");
> >>>>>         route(1);
> >>>>> }
> >>>>>
> >>>>> route[1] {
> >>>>>         if (!t_relay()) sl_reply_error();
> >>>>>         exit;
> >>>>> }
> >>>>>
> >>>>> An example log file showing the 'out of memory' messages is
> >>>>> available at http://www.unc.edu/~cschlatt/openser/openser.log .
> >>>>>
> >>>>> Some observations:
> >>>>>
> >>>>> - The 'out of memory' messages always appear after about 8000 test
> >>>>> calls per worker process. One call consists of two SIP
> >>>>> transactions and six end-to-end SIP messages. An openser with 8
> >>>>> children handles about 64'000 calls, whereas 4 children only
> >>>>> handle about 32'000 calls. The sipp call rate doesn't matter, only
> >>>>> number of calls.
> >>>>>
> >>>>> - The 8000 calls per worker process are independent from the
> >>>>> amount of shared memory available. Running openser with -m 128 or
> >>>>> -m 768 does not make a difference.
> >>>>>
> >>>>> - The more AVP writes are done in the script, the less calls go
> >>>>> through. It looks like each AVP write is leaking memory (unnoticed
> >>>>> by the memory statistics).
> >>>>>
> >>>>> - The fifo memory statistics do not reflect the 'out of memory'
> >>>>> syslog messages. Even if openser does not route a single SIP
> >>>>> message because of memory issues, the statistics still show a lot
> >>>>> of 'free' memory.
> >>>>>
> >>>>>
> >>>>> All tests were done with openser SVN 1.2 branch on Ubuntu dapper
> >>>>> x86. I think the same is true for 1.1 version but I haven't tested
> >>>>> that yet.
> >>>>>
> >>>>>
> >>>>> Christian
> >>>>>
> >>>
> >>>
> >
> >
> > _______________________________________________
> > Users mailing list
> > Users at openser.org
> > http://openser.org/cgi-bin/mailman/listinfo/users
> >
>
> _______________________________________________
> Users mailing list
> Users at openser.org
> http://openser.org/cgi-bin/mailman/listinfo/users
>




More information about the Users mailing list