New subject: [Users] memory issues

23 Apr 2007


      Hello,
daily snapshots are enabled now for 1.2.x as well:
http://www.openser.org/downloads/snapshots/openser-1.2.x/
Cheers,
Daniel
On 04/23/07 17:47, Ovidiu Sas wrote:
...
Hi Tim,
Check the download page from openser website:
http://www.openser.org/mos/view/Download/:
The command that you need to run:
svn co http://openser.svn.sourceforge.net/svnroot/openser/branches/1.2 
openser
Make sure that you have svn installed.
Regards,
Ovidiu Sas
On 4/23/07, Tim Madorma tmadorma@gmail.com wrote:
...
Hi Daniel,
I have run into a leak in 1.2 and I assume it is the same one that
Ovidiu ran into. I see in your response that it was "backported to
1.2", but I'm not sure how to get the fix. When I look at the SVN
repository at:
http://www.openser.org/pub/openser/latest-1.2.x/, the date is earlier
than the date of your email exchange so I don't think the fix has been
added there. Can you please let me know how I can get it?
thanks,
Tim
On 3/23/07, Daniel-Constantin Mierla daniel@voice-system.ro wrote:
...
Hello Ovidiu,
On 03/23/07 17:04, Ovidiu Sas wrote:
...
Hi Daniel,
Can we backport this one to 1.2?
already done, two minutes after the commit in trunk.
Cheers,
Daniel
...
Regards,
Ovidiu Sas
On 3/22/07, Daniel-Constantin Mierla daniel@voice-system.ro wrote:
...
Hello,
the supposed fragmentation turned out to be a mem leak in pkg.
Please
...
...
...
take the latest SVN version and try again to see if you got same
results.
Thanks,
Daniel
On 03/19/07 18:52, Christian Schlatter wrote:
...
...
>> The memory statistics indeed show a high number of memory
fragments:
...
...
...
...
>>
>> before 'out of memory':
>>
>> shmem:total_size = 536870912
>> shmem:used_size = 59607040
>> shmem:real_used_size = 60106488
>> shmem:max_used_size = 68261536
>> shmem:free_size = 476764424
>> shmem:fragments = 9897
>>
>> after 'out of memory' (about 8000 calls per process):
>>
>> shmem:total_size = 536870912
>> shmem:used_size = 4171160
>> shmem:real_used_size = 4670744
>> shmem:max_used_size = 68261536
>> shmem:free_size = 532200168
>> shmem:fragments = 57902
>>
>>>
>>> You can try to compile openser with -DQM_JOIN_FREE (add it
in DEFS
...
...
...
...
>>> variable of Makefile.defs) and test again. Free fragments
should be
...
...
...
...
>>> merged and fragmentation should not occur -- processing
will be
...
...
...
...
>>> slower. We will try for next release to provide a better
solution
...
...
...
...
>>> for that.
>>
>> Compiling openser with -DQM_JOIN_FREE did not help. I'm not
sure how
...
...
...
...
>> big of a problem this fragmentation issue is.
> What is the number of fragments with QM_JOIN_FREE after
flooding?
...
...
...
...
The numbers included above are with QM_JOIN_FREE enabled.
>> Do you think it would make sense to restart our production
openser
...
...
...
...
>> instances from time to time just to make sure they're not
running
...
...
...
...
>> into this memory fragmentation limits?
> The issue will occur only when the call rate reaches the
limits of
...
...
...
...
> the proxy's memory. Otherwise the chunks are reused.
Transactions and
...
...
...
...
> avps are rounded up to be sure there will be minimized the
number of
...
...
...
...
> different sizes for memory chunks. It wasn't reported too often,
> maybe that's why no big attention was paid to it. This memory
system
...
...
...
...
> is in place since the beginning of ser. Alternative is to use
sysv
...
...
...
...
> shared memory, but is much slower, along with libc private
memory
...
...
...
...
> manager.
I've done some more testing and the same out-of-memory stuff
happens
...
...
...
...
when I run sipp with 10 calls per second only. I tested with
'children=1' and I only could get through about 8200 calls (again
those 8000 calls / process). And this is with QM_JOIN_FREE
enabled.
...
...
...
...
Memory statistics:
before:
shmem:total_size = 536870912
shmem:used_size = 2311976
shmem:real_used_size = 2335720
shmem:max_used_size = 2465816
shmem:free_size = 534535192
shmem:fragments = 183
after:
shmem:total_size = 536870912
shmem:used_size = 1853472
shmem:real_used_size = 1877224
shmem:max_used_size = 2465816
shmem:free_size = 534993688
shmem:fragments = 547
So I'm not sure if this is really a fragmentation issue. 10
cps surely
...
...
...
...
doesn't reach the proxy's memory.
Thoughts?
Christian
> Cheers,
> Daniel
>
>>
>> thanks,
>> Christian
>>
>>>
>>> Cheers,
>>> Daniel
>>>
>>> On 03/18/07 01:21, Christian Schlatter wrote:
>>>> Christian Schlatter wrote:
>>>> ...
>>>>>
>>>>> I always had 768MB shared memory configured though, so I
still
...
...
...
...
>>>>> can't explain the memory allocation errors I got. Some
more test
...
...
...
...
>>>>> runs revealed that I only get these errors when using a more
>>>>> production oriented config that loads more modules than
the one
...
...
...
...
>>>>> posted in my earlier email. I now try to figure out what
exactly
...
...
...
...
>>>>> causes these memory allocation errors that happen
reproducibly
...
...
...
...
>>>>> after about 220s at 400 cps.
>>>>
>>>> I think I found the cause for the memory allocation
errors. As
...
...
...
...
>>>> soon as I include an AVP write operation in the routing
script, I
...
...
...
...
>>>> get 'out of memory' messages after a certain number of calls
>>>> generated with sipp.
>>>>
>>>> The routing script to reproduce this behavior looks like
(full
...
...
...
...
>>>> config available at
>>>> http://www.unc.edu/~cschlatt/openser/openser.cfg):
>>>>
>>>> route{
>>>>         $avp(s:ct) = $ct; # commenting this line solves
>>>>               # the memory problem
>>>>
>>>>         if (!method=="REGISTER") record_route();
>>>>         if (loose_route()) route(1);
>>>>
>>>>         if (uri==myself) rewritehost("xx.xx.xx.xx");
>>>>         route(1);
>>>> }
>>>>
>>>> route[1] {
>>>>         if (!t_relay()) sl_reply_error();
>>>>         exit;
>>>> }
>>>>
>>>> An example log file showing the 'out of memory' messages is
>>>> available at
http://www.unc.edu/~cschlatt/openser/openser.log .
...
...
...
...
>>>>
>>>> Some observations:
>>>>
>>>> - The 'out of memory' messages always appear after about
8000 test
...
...
...
...
>>>> calls per worker process. One call consists of two SIP
>>>> transactions and six end-to-end SIP messages. An openser
with 8
...
...
...
...
>>>> children handles about 64'000 calls, whereas 4 children only
>>>> handle about 32'000 calls. The sipp call rate doesn't
matter, only
...
...
...
...
>>>> number of calls.
>>>>
>>>> - The 8000 calls per worker process are independent from the
>>>> amount of shared memory available. Running openser with -m
128 or
...
...
...
...
>>>> -m 768 does not make a difference.
>>>>
>>>> - The more AVP writes are done in the script, the less
calls go
...
...
...
...
>>>> through. It looks like each AVP write is leaking memory
(unnoticed
...
...
...
...
>>>> by the memory statistics).
>>>>
>>>> - The fifo memory statistics do not reflect the 'out of
memory'
...
...
...
...
>>>> syslog messages. Even if openser does not route a single SIP
>>>> message because of memory issues, the statistics still
show a lot
...
...
...
...
>>>> of 'free' memory.
>>>>
>>>>
>>>> All tests were done with openser SVN 1.2 branch on Ubuntu
dapper
...
...
...
...
>>>> x86. I think the same is true for 1.1 version but I
haven't tested
...
...
...
...
>>>> that yet.
>>>>
>>>>
>>>> Christian
>>>>
>>
>>

Users mailing list
Users@openser.org
http://openser.org/cgi-bin/mailman/listinfo/users

Users mailing list
Users@openser.org
http://openser.org/cgi-bin/mailman/listinfo/users

Users mailing list
Users@openser.org
http://openser.org/cgi-bin/mailman/listinfo/users

Re: [Users] memory issues