[Users] Re: memory issues

Daniel-Constantin Mierla daniel at voice-system.ro
Tue Mar 27 10:48:22 CEST 2007


Hello,

On 03/27/07 11:24, Papadopoulos Georgios wrote:
> Hi Daniel,
>  
> Does this problem affect 1.1.1 or only 1.2? Is there a patch to use 
> for 1.1.1?
it was a new one introduced in 1.2, not valid for 1.1.x

Cheers,
Daniel

>  
> thank you
>  
> George
>  
>  
> --------------------------------------------------------------------------------
>  
> Hello Ovidiu,
>
> On 03/23/07 17:04, Ovidiu Sas wrote:
> >/ Hi Daniel,
> />/
> />/ Can we backport this one to 1.2?
> /already done, two minutes after the commit in trunk.
>
> Cheers,
> Daniel
>
> >/
> />/
> />/ Regards,
> />/ Ovidiu Sas
> />/
> />/ On 3/22/07, Daniel-Constantin Mierla <daniel at voice-system.ro 
> <http://openser.org/cgi-bin/mailman/listinfo/users>> wrote:
> />>/ Hello,
> />>/
> />>/ the supposed fragmentation turned out to be a mem leak in pkg. Please
> />>/ take the latest SVN version and try again to see if you got same
> />>/ results.
> />>/
> />>/ Thanks,
> />>/ Daniel
> />>/
> />>/ On 03/19/07 18:52, Christian Schlatter wrote:
> />>/ > ...
> />>/ >>> The memory statistics indeed show a high number of memory 
> fragments:
> />>/ >>>
> />>/ >>> before 'out of memory':
> />>/ >>>
> />>/ >>> shmem:total_size = 536870912
> />>/ >>> shmem:used_size = 59607040
> />>/ >>> shmem:real_used_size = 60106488
> />>/ >>> shmem:max_used_size = 68261536
> />>/ >>> shmem:free_size = 476764424
> />>/ >>> shmem:fragments = 9897
> />>/ >>>
> />>/ >>> after 'out of memory' (about 8000 calls per process):
> />>/ >>>
> />>/ >>> shmem:total_size = 536870912
> />>/ >>> shmem:used_size = 4171160
> />>/ >>> shmem:real_used_size = 4670744
> />>/ >>> shmem:max_used_size = 68261536
> />>/ >>> shmem:free_size = 532200168
> />>/ >>> shmem:fragments = 57902
> />>/ >>>
> />>/ >>>>
> />>/ >>>> You can try to compile openser with -DQM_JOIN_FREE (add it 
> in DEFS
> />>/ >>>> variable of Makefile.defs) and test again. Free fragments 
> should be
> />>/ >>>> merged and fragmentation should not occur -- processing will be
> />>/ >>>> slower. We will try for next release to provide a better 
> solution
> />>/ >>>> for that.
> />>/ >>>
> />>/ >>> Compiling openser with -DQM_JOIN_FREE did not help. I'm not 
> sure how
> />>/ >>> big of a problem this fragmentation issue is.
> />>/ >> What is the number of fragments with QM_JOIN_FREE after flooding?
> />>/ >
> />>/ > The numbers included above are with QM_JOIN_FREE enabled.
> />>/ >
> />>/ >>> Do you think it would make sense to restart our production 
> openser
> />>/ >>> instances from time to time just to make sure they're not running
> />>/ >>> into this memory fragmentation limits?
> />>/ >> The issue will occur only when the call rate reaches the limits of
> />>/ >> the proxy's memory. Otherwise the chunks are reused. 
> Transactions and
> />>/ >> avps are rounded up to be sure there will be minimized the 
> number of
> />>/ >> different sizes for memory chunks. It wasn't reported too often,
> />>/ >> maybe that's why no big attention was paid to it. This memory 
> system
> />>/ >> is in place since the beginning of ser. Alternative is to use sysv
> />>/ >> shared memory, but is much slower, along with libc private memory
> />>/ >> manager.
> />>/ >
> />>/ > I've done some more testing and the same out-of-memory stuff 
> happens
> />>/ > when I run sipp with 10 calls per second only. I tested with
> />>/ > 'children=1' and I only could get through about 8200 calls (again
> />>/ > those 8000 calls / process). And this is with QM_JOIN_FREE enabled.
> />>/ >
> />>/ > Memory statistics:
> />>/ >
> />>/ > before:
> />>/ > shmem:total_size = 536870912
> />>/ > shmem:used_size = 2311976
> />>/ > shmem:real_used_size = 2335720
> />>/ > shmem:max_used_size = 2465816
> />>/ > shmem:free_size = 534535192
> />>/ > shmem:fragments = 183
> />>/ >
> />>/ > after:
> />>/ > shmem:total_size = 536870912
> />>/ > shmem:used_size = 1853472
> />>/ > shmem:real_used_size = 1877224
> />>/ > shmem:max_used_size = 2465816
> />>/ > shmem:free_size = 534993688
> />>/ > shmem:fragments = 547
> />>/ >
> />>/ > So I'm not sure if this is really a fragmentation issue. 10 cps 
> surely
> />>/ > doesn't reach the proxy's memory.
> />>/ >
> />>/ > Thoughts?
> />>/ >
> />>/ > Christian
> />>/ >
> />>/ >
> />>/ >
> />>/ >> Cheers,
> />>/ >> Daniel
> />>/ >>
> />>/ >>>
> />>/ >>> thanks,
> />>/ >>> Christian
> />>/ >>>
> />>/ >>>>
> />>/ >>>> Cheers,
> />>/ >>>> Daniel
> />>/ >>>>
> />>/ >>>> On 03/18/07 01:21, Christian Schlatter wrote:
> />>/ >>>>> Christian Schlatter wrote:
> />>/ >>>>> ...
> />>/ >>>>>>
> />>/ >>>>>> I always had 768MB shared memory configured though, so I still
> />>/ >>>>>> can't explain the memory allocation errors I got. Some 
> more test
> />>/ >>>>>> runs revealed that I only get these errors when using a more
> />>/ >>>>>> production oriented config that loads more modules than 
> the one
> />>/ >>>>>> posted in my earlier email. I now try to figure out what 
> exactly
> />>/ >>>>>> causes these memory allocation errors that happen reproducibly
> />>/ >>>>>> after about 220s at 400 cps.
> />>/ >>>>>
> />>/ >>>>> I think I found the cause for the memory allocation errors. As
> />>/ >>>>> soon as I include an AVP write operation in the routing 
> script, I
> />>/ >>>>> get 'out of memory' messages after a certain number of calls
> />>/ >>>>> generated with sipp.
> />>/ >>>>>
> />>/ >>>>> The routing script to reproduce this behavior looks like (full
> />>/ >>>>> config available at
> />>/ >>>>> http://www.unc.edu/~cschlatt/openser/openser.cfg 
> <http://www.unc.edu/%7Ecschlatt/openser/openser.cfg>):
> />>/ >>>>>
> />>/ >>>>> route{
> />>/ >>>>>         $avp(s:ct) = $ct; # commenting this line solves
> />>/ >>>>>               # the memory problem
> />>/ >>>>>
> />>/ >>>>>         if (!method=="REGISTER") record_route();
> />>/ >>>>>         if (loose_route()) route(1);
> />>/ >>>>>
> />>/ >>>>>         if (uri==myself) rewritehost("xx.xx.xx.xx");
> />>/ >>>>>         route(1);
> />>/ >>>>> }
> />>/ >>>>>
> />>/ >>>>> route[1] {
> />>/ >>>>>         if (!t_relay()) sl_reply_error();
> />>/ >>>>>         exit;
> />>/ >>>>> }
> />>/ >>>>>
> />>/ >>>>> An example log file showing the 'out of memory' messages is
> />>/ >>>>> available at 
> http://www.unc.edu/~cschlatt/openser/openser.log 
> <http://www.unc.edu/%7Ecschlatt/openser/openser.log> .
> />>/ >>>>>
> />>/ >>>>> Some observations:
> />>/ >>>>>
> />>/ >>>>> - The 'out of memory' messages always appear after about 
> 8000 test
> />>/ >>>>> calls per worker process. One call consists of two SIP
> />>/ >>>>> transactions and six end-to-end SIP messages. An openser with 8
> />>/ >>>>> children handles about 64'000 calls, whereas 4 children only
> />>/ >>>>> handle about 32'000 calls. The sipp call rate doesn't 
> matter, only
> />>/ >>>>> number of calls.
> />>/ >>>>>
> />>/ >>>>> - The 8000 calls per worker process are independent from the
> />>/ >>>>> amount of shared memory available. Running openser with -m 
> 128 or
> />>/ >>>>> -m 768 does not make a difference.
> />>/ >>>>>
> />>/ >>>>> - The more AVP writes are done in the script, the less calls go
> />>/ >>>>> through. It looks like each AVP write is leaking memory 
> (unnoticed
> />>/ >>>>> by the memory statistics).
> />>/ >>>>>
> />>/ >>>>> - The fifo memory statistics do not reflect the 'out of memory'
> />>/ >>>>> syslog messages. Even if openser does not route a single SIP
> />>/ >>>>> message because of memory issues, the statistics still show 
> a lot
> />>/ >>>>> of 'free' memory.
> />>/ >>>>>
> />>/ >>>>>
> />>/ >>>>> All tests were done with openser SVN 1.2 branch on Ubuntu 
> dapper
> />>/ >>>>> x86. I think the same is true for 1.1 version but I haven't 
> tested
> />>/ >>>>> that yet.
> />>/ >>>>>
> />>/ >>>>>
> />>/ >>>>> Christian
> />>/ >>>>>
> />>/ >>>
> />>/ >>>
> />>/ >
> />>/ >
> />>/ > _______________________________________________
> />>/ > Users mailing list
> />>/ > Users at openser.org 
> <http://openser.org/cgi-bin/mailman/listinfo/users>
> />>/ > http://openser.org/cgi-bin/mailman/listinfo/users
> />>/ >
> />>/
> />>/ _______________________________________________
> />>/ Users mailing list
> />>/ Users at openser.org 
> <http://openser.org/cgi-bin/mailman/listinfo/users>
> />>/ http://openser.org/cgi-bin/mailman/listinfo/users
> />>/
> />/
> /
>  
> Γιώργος Παπαδόπουλος
> Altec Telecoms
> R&D Department
>  
> Πάτμου 14, 151 23 Μαρούσι
> τηλ.: +30 211 687 2933
> fax.: +30 211 687 2911
> e-mail: geop at altectelecoms.gr <mailto:geop at altectelecoms.gr>
>  
>
>
>   Disclaimer
>
> The information in this e-mail and any attachments is confidential. It 
> is intended solely for the attention and use of the named 
> addressee(s). If you are not the intended recipient, or person 
> responsible for delivering this information to the intended recipient, 
> please notify the sender immediately. Unless you are the intended 
> recipient or his/her representative you are not authorized to, and 
> must not, read, copy, distribute, use or retain this message or any 
> part of it. E-mail transmission cannot be guaranteed to be secure or 
> error-free as information could be intercepted, corrupted, lost, 
> destroyed, arrive late or incomplete, or contain viruses.
>




More information about the sr-users mailing list