openser
Make sure that you have svn installed.
Regards,
Ovidiu Sas
On 4/23/07, Tim Madorma <tmadorma(a)gmail.com> wrote:
Hi Daniel,
I have run into a leak in 1.2 and I assume it is the same one that
Ovidiu ran into. I see in your response that it was "backported to
1.2", but I'm not sure how to get the fix. When I look at the SVN
repository at:
http://www.openser.org/pub/openser/latest-1.2.x/, the date is earlier
than the date of your email exchange so I don't think the fix has been
added there. Can you please let me know how I can get it?
thanks,
Tim
On 3/23/07, Daniel-Constantin Mierla <daniel(a)voice-system.ro> wrote:
Hello Ovidiu,
On 03/23/07 17:04, Ovidiu Sas wrote:
> Hi Daniel,
>
> Can we backport this one to 1.2?
already done, two minutes after the commit in trunk.
Cheers,
Daniel
>
>
> Regards,
> Ovidiu Sas
>
> On 3/22/07, Daniel-Constantin Mierla <daniel(a)voice-system.ro> wrote:
>> Hello,
>>
>> the supposed fragmentation turned out to be a mem leak in pkg.
Please
>> take the latest SVN version and try
again to see if you got same
>> results.
>>
>> Thanks,
>> Daniel
>>
>> On 03/19/07 18:52, Christian Schlatter wrote:
>> > ...
>> >>> The memory statistics indeed show a high number of memory
fragments:
>> >>>
>> >>> before 'out of memory':
>> >>>
>> >>> shmem:total_size = 536870912
>> >>> shmem:used_size = 59607040
>> >>> shmem:real_used_size = 60106488
>> >>> shmem:max_used_size = 68261536
>> >>> shmem:free_size = 476764424
>> >>> shmem:fragments = 9897
>> >>>
>> >>> after 'out of memory' (about 8000 calls per process):
>> >>>
>> >>> shmem:total_size = 536870912
>> >>> shmem:used_size = 4171160
>> >>> shmem:real_used_size = 4670744
>> >>> shmem:max_used_size = 68261536
>> >>> shmem:free_size = 532200168
>> >>> shmem:fragments = 57902
>> >>>
>> >>>>
>> >>>> You can try to compile openser with -DQM_JOIN_FREE (add it
in DEFS
>> >>>> variable of
Makefile.defs) and test again. Free fragments
should be
>> >>>> merged and
fragmentation should not occur -- processing
will be
>> >>>> slower. We will try for
next release to provide a better
solution
>> >>>> for that.
>> >>>
>> >>> Compiling openser with -DQM_JOIN_FREE did not help. I'm not
sure how
>> >>> big of a problem this
fragmentation issue is.
>> >> What is the number of fragments with QM_JOIN_FREE after
flooding?
>> >
>> > The numbers included above are with QM_JOIN_FREE enabled.
>> >
>> >>> Do you think it would make sense to restart our production
openser
>> >>> instances from time to time
just to make sure they're not
running
>> >>> into this memory
fragmentation limits?
>> >> The issue will occur only when the call rate reaches the
limits
of
>> >> the proxy's memory.
Otherwise the chunks are reused.
Transactions and
>> >> avps are rounded up to be sure
there will be minimized the
number of
>> >> different sizes for memory
chunks. It wasn't reported too often,
>> >> maybe that's why no big attention was paid to it. This memory
system
>> >> is in place since the beginning
of ser. Alternative is to use
sysv
>> >> shared memory, but is much
slower, along with libc private
memory
>> >> manager.
>> >
>> > I've done some more testing and the same out-of-memory stuff
happens
>> > when I run sipp with 10 calls per
second only. I tested with
>> > 'children=1' and I only could get through about 8200 calls (again
>> > those 8000 calls / process). And this is with QM_JOIN_FREE
enabled.
>> >
>> > Memory statistics:
>> >
>> > before:
>> > shmem:total_size = 536870912
>> > shmem:used_size = 2311976
>> > shmem:real_used_size = 2335720
>> > shmem:max_used_size = 2465816
>> > shmem:free_size = 534535192
>> > shmem:fragments = 183
>> >
>> > after:
>> > shmem:total_size = 536870912
>> > shmem:used_size = 1853472
>> > shmem:real_used_size = 1877224
>> > shmem:max_used_size = 2465816
>> > shmem:free_size = 534993688
>> > shmem:fragments = 547
>> >
>> > So I'm not sure if this is really a fragmentation issue. 10
cps
surely
>> > doesn't reach the proxy's
memory.
>> >
>> > Thoughts?
>> >
>> > Christian
>> >
>> >
>> >
>> >> Cheers,
>> >> Daniel
>> >>
>> >>>
>> >>> thanks,
>> >>> Christian
>> >>>
>> >>>>
>> >>>> Cheers,
>> >>>> Daniel
>> >>>>
>> >>>> On 03/18/07 01:21, Christian Schlatter wrote:
>> >>>>> Christian Schlatter wrote:
>> >>>>> ...
>> >>>>>>
>> >>>>>> I always had 768MB shared memory configured though, so
I
still
>> >>>>>> can't
explain the memory allocation errors I got. Some
more test
>> >>>>>> runs revealed
that I only get these errors when using a more
>> >>>>>> production oriented config that loads more modules than
the one
>> >>>>>> posted in my
earlier email. I now try to figure out what
exactly
>> >>>>>> causes these
memory allocation errors that happen
reproducibly
>> >>>>>> after about
220s at 400 cps.
>> >>>>>
>> >>>>> I think I found the cause for the memory allocation
errors. As
>> >>>>> soon as I include
an AVP write operation in the routing
script, I
>> >>>>> get 'out of
memory' messages after a certain number of calls
>> >>>>> generated with sipp.
>> >>>>>
>> >>>>> The routing script to reproduce this behavior looks like
(full
>> >>>>> config available
at
>> >>>>>
http://www.unc.edu/~cschlatt/openser/openser.cfg):
>> >>>>>
>> >>>>> route{
>> >>>>> $avp(s:ct) = $ct; # commenting this line solves
>> >>>>> # the memory problem
>> >>>>>
>> >>>>> if (!method=="REGISTER") record_route();
>> >>>>> if (loose_route()) route(1);
>> >>>>>
>> >>>>> if (uri==myself)
rewritehost("xx.xx.xx.xx");
>> >>>>> route(1);
>> >>>>> }
>> >>>>>
>> >>>>> route[1] {
>> >>>>> if (!t_relay()) sl_reply_error();
>> >>>>> exit;
>> >>>>> }
>> >>>>>
>> >>>>> An example log file showing the 'out of memory'
messages is
>> >>>>> available at
http://www.unc.edu/~cschlatt/openser/openser.log .
>> >>>>>
>> >>>>> Some observations:
>> >>>>>
>> >>>>> - The 'out of memory' messages always appear after
about
8000 test
>> >>>>> calls per worker
process. One call consists of two SIP
>> >>>>> transactions and six end-to-end SIP messages. An openser
with 8
>> >>>>> children handles
about 64'000 calls, whereas 4 children only
>> >>>>> handle about 32'000 calls. The sipp call rate
doesn't
matter, only
>> >>>>> number of calls.
>> >>>>>
>> >>>>> - The 8000 calls per worker process are independent from
the
>> >>>>> amount of shared memory available. Running openser with -m
128 or
>> >>>>> -m 768 does not
make a difference.
>> >>>>>
>> >>>>> - The more AVP writes are done in the script, the less
calls go
>> >>>>> through. It looks
like each AVP write is leaking memory
(unnoticed
>> >>>>> by the memory
statistics).
>> >>>>>
>> >>>>> - The fifo memory statistics do not reflect the 'out of
memory'
>> >>>>> syslog messages.
Even if openser does not route a single SIP
>> >>>>> message because of memory issues, the statistics still
show a lot
>> >>>>> of 'free'
memory.
>> >>>>>
>> >>>>>
>> >>>>> All tests were done with openser SVN 1.2 branch on Ubuntu
dapper
>> >>>>> x86. I think the
same is true for 1.1 version but I
haven't tested
>> >>>>> that yet.
>> >>>>>
>> >>>>>
>> >>>>> Christian
>> >>>>>
>> >>>
>> >>>
>> >
>> >
>> > _______________________________________________
>> > Users mailing list
>> > Users(a)openser.org
>> >
http://openser.org/cgi-bin/mailman/listinfo/users
>> >
>>
>> _______________________________________________
>> Users mailing list
>> Users(a)openser.org
>>
http://openser.org/cgi-bin/mailman/listinfo/users
>>
>
_______________________________________________
Users mailing list
Users(a)openser.org
http://openser.org/cgi-bin/mailman/listinfo/users