<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML><HEAD>
<META http-equiv=Content-Type content="text/html; charset=iso-8859-7">
<META content="MSHTML 6.00.2900.2180" name=GENERATOR></HEAD>
<BODY>
<DIV><FONT face=Arial><SPAN class=246441108-27032007>Hi
Daniel,</SPAN></FONT></DIV>
<DIV><FONT face=Arial><SPAN class=246441108-27032007></SPAN></FONT> </DIV>
<DIV><FONT face=Arial><SPAN class=246441108-27032007>Does this problem affect
1.1.1 or only 1.2? Is there a patch to use for
1.1.1?</SPAN></FONT></DIV>
<DIV><FONT face=Arial><SPAN class=246441108-27032007></SPAN></FONT> </DIV>
<DIV><FONT face=Arial><SPAN class=246441108-27032007>thank
you</SPAN></FONT></DIV>
<DIV><FONT face=Arial><SPAN class=246441108-27032007></SPAN></FONT> </DIV>
<DIV><FONT face=Arial><SPAN class=246441108-27032007>George</SPAN></FONT></DIV>
<DIV><FONT face=Arial></FONT> </DIV>
<DIV><FONT face=Arial size=2><FONT size=3></FONT></FONT> </DIV>
<DIV><FONT face=Arial><SPAN
class=246441108-27032007>--------------------------------------------------------------------------------</SPAN></FONT></DIV>
<DIV><FONT face=Arial><SPAN class=246441108-27032007></SPAN></FONT> </DIV>
<DIV><FONT face=Arial size=2><FONT size=3>Hello Ovidiu,<BR><BR>On 03/23/07
17:04, Ovidiu Sas wrote:<BR>></FONT><FONT size=3><I> Hi
Daniel,<BR></I>></FONT><I><BR></I><FONT size=3>></FONT><FONT size=3><I>
Can we backport this one to 1.2?<BR></I>already done, two minutes after the
commit in trunk.<BR><BR>Cheers,<BR>Daniel<BR><BR>></FONT><I><BR></I><FONT
size=3>></FONT><I><BR></I><FONT size=3>></FONT><FONT size=3><I>
Regards,<BR></I>></FONT><FONT size=3><I> Ovidiu
Sas<BR></I>></FONT><I><BR></I><FONT size=3>></FONT><I><FONT size=3> On
3/22/07, Daniel-Constantin Mierla <</FONT><A
href="http://openser.org/cgi-bin/mailman/listinfo/users"><FONT size=3>daniel at
voice-system.ro</FONT></A><FONT size=3>> wrote:<BR></FONT></I><FONT
size=3>>></FONT><FONT size=3><I>
Hello,<BR></I>>></FONT><I><BR></I><FONT size=3>>></FONT><FONT
size=3><I> the supposed fragmentation turned out to be a mem leak in pkg.
Please<BR></I>>></FONT><FONT size=3><I> take the latest SVN version and
try again to see if you got same <BR></I>>></FONT><FONT size=3><I>
results.<BR></I>>></FONT><I><BR></I><FONT size=3>>></FONT><FONT
size=3><I> Thanks,<BR></I>>></FONT><FONT size=3><I>
Daniel<BR></I>>></FONT><I><BR></I><FONT size=3>>></FONT><FONT
size=3><I> On 03/19/07 18:52, Christian Schlatter
wrote:<BR></I>>></FONT><FONT size=3><I> >
...<BR></I>>></FONT><FONT size=3><I> >>> The memory statistics
indeed show a high number of memory fragments:<BR></I>>></FONT><FONT
size=3><I> >>><BR></I>>></FONT><FONT size=3><I> >>>
before 'out of memory':<BR></I>>></FONT><FONT size=3><I>
>>><BR></I>>></FONT><FONT size=3><I> >>>
shmem:total_size = 536870912<BR></I>>></FONT><FONT size=3><I> >>>
shmem:used_size = 59607040<BR></I>>></FONT><FONT size=3><I> >>>
shmem:real_used_size = 60106488<BR></I>>></FONT><FONT size=3><I>
>>> shmem:max_used_size = 68261536<BR></I>>></FONT><FONT
size=3><I> >>> shmem:free_size = 476764424<BR></I>>></FONT><FONT
size=3><I> >>> shmem:fragments = 9897<BR></I>>></FONT><FONT
size=3><I> >>><BR></I>>></FONT><FONT size=3><I> >>>
after 'out of memory' (about 8000 calls per
process):<BR></I>>></FONT><FONT size=3><I>
>>><BR></I>>></FONT><FONT size=3><I> >>>
shmem:total_size = 536870912<BR></I>>></FONT><FONT size=3><I> >>>
shmem:used_size = 4171160<BR></I>>></FONT><FONT size=3><I> >>>
shmem:real_used_size = 4670744<BR></I>>></FONT><FONT size=3><I>
>>> shmem:max_used_size = 68261536<BR></I>>></FONT><FONT
size=3><I> >>> shmem:free_size = 532200168<BR></I>>></FONT><FONT
size=3><I> >>> shmem:fragments = 57902<BR></I>>></FONT><FONT
size=3><I> >>><BR></I>>></FONT><FONT size=3><I>
>>>><BR></I>>></FONT><FONT size=3><I> >>>> You can
try to compile openser with -DQM_JOIN_FREE (add it in
DEFS<BR></I>>></FONT><FONT size=3><I> >>>> variable of
Makefile.defs) and test again. Free fragments should
be<BR></I>>></FONT><FONT size=3><I> >>>> merged and
fragmentation should not occur -- processing will be<BR></I>>></FONT><FONT
size=3><I> >>>> slower. We will try for next release to provide a
better solution<BR></I>>></FONT><FONT size=3><I> >>>> for
that.<BR></I>>></FONT><FONT size=3><I>
>>><BR></I>>></FONT><FONT size=3><I> >>> Compiling
openser with -DQM_JOIN_FREE did not help. I'm not sure
how<BR></I>>></FONT><FONT size=3><I> >>> big of a problem this
fragmentation issue is.<BR></I>>></FONT><FONT size=3><I> >> What is
the number of fragments with QM_JOIN_FREE after
flooding?<BR></I>>></FONT><FONT size=3><I>
><BR></I>>></FONT><FONT size=3><I> > The numbers included above are
with QM_JOIN_FREE enabled.<BR></I>>></FONT><FONT size=3><I>
><BR></I>>></FONT><FONT size=3><I> >>> Do you think it would
make sense to restart our production openser<BR></I>>></FONT><FONT
size=3><I> >>> instances from time to time just to make sure they're
not running<BR></I>>></FONT><FONT size=3><I> >>> into this memory
fragmentation limits?<BR></I>>></FONT><FONT size=3><I> >> The issue
will occur only when the call rate reaches the limits
of<BR></I>>></FONT><FONT size=3><I> >> the proxy's memory. Otherwise
the chunks are reused. Transactions and<BR></I>>></FONT><FONT size=3><I>
>> avps are rounded up to be sure there will be minimized the number
of<BR></I>>></FONT><FONT size=3><I> >> different sizes for memory
chunks. It wasn't reported too often,<BR></I>>></FONT><FONT size=3><I>
>> maybe that's why no big attention was paid to it. This memory
system<BR></I>>></FONT><FONT size=3><I> >> is in place since the
beginning of ser. Alternative is to use sysv<BR></I>>></FONT><FONT
size=3><I> >> shared memory, but is much slower, along with libc private
memory<BR></I>>></FONT><FONT size=3><I> >>
manager.<BR></I>>></FONT><FONT size=3><I> ><BR></I>>></FONT><FONT
size=3><I> > I've done some more testing and the same out-of-memory stuff
happens<BR></I>>></FONT><FONT size=3><I> > when I run sipp with 10
calls per second only. I tested with<BR></I>>></FONT><FONT size=3><I> >
'children=1' and I only could get through about 8200 calls
(again<BR></I>>></FONT><FONT size=3><I> > those 8000 calls / process).
And this is with QM_JOIN_FREE enabled.<BR></I>>></FONT><FONT size=3><I>
><BR></I>>></FONT><FONT size=3><I> > Memory
statistics:<BR></I>>></FONT><FONT size=3><I>
><BR></I>>></FONT><FONT size=3><I> >
before:<BR></I>>></FONT><FONT size=3><I> > shmem:total_size =
536870912<BR></I>>></FONT><FONT size=3><I> > shmem:used_size =
2311976<BR></I>>></FONT><FONT size=3><I> > shmem:real_used_size =
2335720<BR></I>>></FONT><FONT size=3><I> > shmem:max_used_size =
2465816<BR></I>>></FONT><FONT size=3><I> > shmem:free_size =
534535192<BR></I>>></FONT><FONT size=3><I> > shmem:fragments =
183<BR></I>>></FONT><FONT size=3><I> ><BR></I>>></FONT><FONT
size=3><I> > after:<BR></I>>></FONT><FONT size=3><I> >
shmem:total_size = 536870912<BR></I>>></FONT><FONT size=3><I> >
shmem:used_size = 1853472<BR></I>>></FONT><FONT size=3><I> >
shmem:real_used_size = 1877224<BR></I>>></FONT><FONT size=3><I> >
shmem:max_used_size = 2465816<BR></I>>></FONT><FONT size=3><I> >
shmem:free_size = 534993688<BR></I>>></FONT><FONT size=3><I> >
shmem:fragments = 547<BR></I>>></FONT><FONT size=3><I>
><BR></I>>></FONT><FONT size=3><I> > So I'm not sure if this is
really a fragmentation issue. 10 cps surely<BR></I>>></FONT><FONT
size=3><I> > doesn't reach the proxy's memory.<BR></I>>></FONT><FONT
size=3><I> ><BR></I>>></FONT><FONT size=3><I> >
Thoughts?<BR></I>>></FONT><FONT size=3><I>
><BR></I>>></FONT><FONT size=3><I> >
Christian<BR></I>>></FONT><FONT size=3><I>
><BR></I>>></FONT><FONT size=3><I> ><BR></I>>></FONT><FONT
size=3><I> ><BR></I>>></FONT><FONT size=3><I> >>
Cheers,<BR></I>>></FONT><FONT size=3><I> >>
Daniel<BR></I>>></FONT><FONT size=3><I>
>><BR></I>>></FONT><FONT size=3><I>
>>><BR></I>>></FONT><FONT size=3><I> >>>
thanks,<BR></I>>></FONT><FONT size=3><I> >>>
Christian<BR></I>>></FONT><FONT size=3><I>
>>><BR></I>>></FONT><FONT size=3><I>
>>>><BR></I>>></FONT><FONT size=3><I> >>>>
Cheers,<BR></I>>></FONT><FONT size=3><I> >>>>
Daniel<BR></I>>></FONT><FONT size=3><I>
>>>><BR></I>>></FONT><FONT size=3><I> >>>> On
03/18/07 01:21, Christian Schlatter wrote:<BR></I>>></FONT><FONT
size=3><I> >>>>> Christian Schlatter
wrote:<BR></I>>></FONT><FONT size=3><I> >>>>>
...<BR></I>>></FONT><FONT size=3><I>
>>>>>><BR></I>>></FONT><FONT size=3><I>
>>>>>> I always had 768MB shared memory configured though, so
I still<BR></I>>></FONT><FONT size=3><I> >>>>>> can't
explain the memory allocation errors I got. Some more
test<BR></I>>></FONT><FONT size=3><I> >>>>>> runs
revealed that I only get these errors when using a
more<BR></I>>></FONT><FONT size=3><I> >>>>>> production
oriented config that loads more modules than the one<BR></I>>></FONT><FONT
size=3><I> >>>>>> posted in my earlier email. I now try to
figure out what exactly<BR></I>>></FONT><FONT size=3><I>
>>>>>> causes these memory allocation errors that happen
reproducibly<BR></I>>></FONT><FONT size=3><I> >>>>>>
after about 220s at 400 cps.<BR></I>>></FONT><FONT size=3><I>
>>>>><BR></I>>></FONT><FONT size=3><I> >>>>>
I think I found the cause for the memory allocation errors.
As<BR></I>>></FONT><FONT size=3><I> >>>>> soon as I include
an AVP write operation in the routing script, I<BR></I>>></FONT><FONT
size=3><I> >>>>> get 'out of memory' messages after a certain
number of calls<BR></I>>></FONT><FONT size=3><I> >>>>>
generated with sipp.<BR></I>>></FONT><FONT size=3><I>
>>>>><BR></I>>></FONT><FONT size=3><I> >>>>>
The routing script to reproduce this behavior looks like
(full<BR></I>>></FONT><FONT size=3><I> >>>>> config
available at<BR></I>>></FONT><I><FONT size=3> >>>>>
</FONT><A href="http://www.unc.edu/~cschlatt/openser/openser.cfg"><FONT
size=3>http://www.unc.edu/~cschlatt/openser/openser.cfg</FONT></A><FONT
size=3>):<BR></FONT></I><FONT size=3>>></FONT><FONT size=3><I>
>>>>><BR></I>>></FONT><FONT size=3><I> >>>>>
route{<BR></I>>></FONT><FONT size=3><I>
>>>>> $avp(s:ct)
= $ct; # commenting this line solves<BR></I>>></FONT><FONT size=3><I>
>>>>>
# the memory problem<BR></I>>></FONT><FONT size=3><I>
>>>>><BR></I>>></FONT><FONT size=3><I>
>>>>> if
(!method=="REGISTER") record_route();<BR></I>>></FONT><FONT size=3><I>
>>>>> if
(loose_route()) route(1);<BR></I>>></FONT><FONT size=3><I>
>>>>><BR></I>>></FONT><FONT size=3><I>
>>>>> if
(uri==myself) rewritehost("xx.xx.xx.xx");<BR></I>>></FONT><FONT size=3><I>
>>>>>
route(1);<BR></I>>></FONT><FONT size=3><I> >>>>>
}<BR></I>>></FONT><FONT size=3><I>
>>>>><BR></I>>></FONT><FONT size=3><I> >>>>>
route[1] {<BR></I>>></FONT><FONT size=3><I>
>>>>> if
(!t_relay()) sl_reply_error();<BR></I>>></FONT><FONT size=3><I>
>>>>>
exit;<BR></I>>></FONT><FONT size=3><I> >>>>>
}<BR></I>>></FONT><FONT size=3><I>
>>>>><BR></I>>></FONT><FONT size=3><I> >>>>>
An example log file showing the 'out of memory' messages
is<BR></I>>></FONT><I><FONT size=3> >>>>> available at
</FONT><A href="http://www.unc.edu/~cschlatt/openser/openser.log"><FONT
size=3>http://www.unc.edu/~cschlatt/openser/openser.log</FONT></A><FONT size=3>
.<BR></FONT></I><FONT size=3>>></FONT><FONT size=3><I>
>>>>><BR></I>>></FONT><FONT size=3><I> >>>>>
Some observations:<BR></I>>></FONT><FONT size=3><I>
>>>>><BR></I>>></FONT><FONT size=3><I> >>>>>
- The 'out of memory' messages always appear after about 8000
test<BR></I>>></FONT><FONT size=3><I> >>>>> calls per
worker process. One call consists of two SIP<BR></I>>></FONT><FONT
size=3><I> >>>>> transactions and six end-to-end SIP messages. An
openser with 8<BR></I>>></FONT><FONT size=3><I> >>>>>
children handles about 64'000 calls, whereas 4 children
only<BR></I>>></FONT><FONT size=3><I> >>>>> handle about
32'000 calls. The sipp call rate doesn't matter,
only<BR></I>>></FONT><FONT size=3><I> >>>>> number of
calls.<BR></I>>></FONT><FONT size=3><I>
>>>>><BR></I>>></FONT><FONT size=3><I> >>>>>
- The 8000 calls per worker process are independent from
the<BR></I>>></FONT><FONT size=3><I> >>>>> amount of shared
memory available. Running openser with -m 128 or<BR></I>>></FONT><FONT
size=3><I> >>>>> -m 768 does not make a
difference.<BR></I>>></FONT><FONT size=3><I>
>>>>><BR></I>>></FONT><FONT size=3><I> >>>>>
- The more AVP writes are done in the script, the less calls
go<BR></I>>></FONT><FONT size=3><I> >>>>> through. It looks
like each AVP write is leaking memory (unnoticed<BR></I>>></FONT><FONT
size=3><I> >>>>> by the memory
statistics).<BR></I>>></FONT><FONT size=3><I>
>>>>><BR></I>>></FONT><FONT size=3><I> >>>>>
- The fifo memory statistics do not reflect the 'out of
memory'<BR></I>>></FONT><FONT size=3><I> >>>>> syslog
messages. Even if openser does not route a single
SIP<BR></I>>></FONT><FONT size=3><I> >>>>> message because
of memory issues, the statistics still show a lot<BR></I>>></FONT><FONT
size=3><I> >>>>> of 'free' memory.<BR></I>>></FONT><FONT
size=3><I> >>>>><BR></I>>></FONT><FONT size=3><I>
>>>>><BR></I>>></FONT><FONT size=3><I> >>>>>
All tests were done with openser SVN 1.2 branch on Ubuntu
dapper<BR></I>>></FONT><FONT size=3><I> >>>>> x86. I think
the same is true for 1.1 version but I haven't
tested<BR></I>>></FONT><FONT size=3><I> >>>>> that
yet.<BR></I>>></FONT><FONT size=3><I>
>>>>><BR></I>>></FONT><FONT size=3><I>
>>>>><BR></I>>></FONT><FONT size=3><I> >>>>>
Christian<BR></I>>></FONT><FONT size=3><I>
>>>>><BR></I>>></FONT><FONT size=3><I>
>>><BR></I>>></FONT><FONT size=3><I>
>>><BR></I>>></FONT><FONT size=3><I>
><BR></I>>></FONT><FONT size=3><I> ><BR></I>>></FONT><FONT
size=3><I> >
_______________________________________________<BR></I>>></FONT><FONT
size=3><I> > Users mailing list<BR></I>>></FONT><I><FONT size=3> >
</FONT><A href="http://openser.org/cgi-bin/mailman/listinfo/users"><FONT
size=3>Users at openser.org</FONT></A><BR></I><FONT
size=3>>></FONT><I><FONT size=3> > </FONT><A
href="http://openser.org/cgi-bin/mailman/listinfo/users"><FONT
size=3>http://openser.org/cgi-bin/mailman/listinfo/users</FONT></A><BR></I><FONT
size=3>>></FONT><FONT size=3><I>
><BR></I>>></FONT><I><BR></I><FONT size=3>>></FONT><FONT
size=3><I>
_______________________________________________<BR></I>>></FONT><FONT
size=3><I> Users mailing list<BR></I>>></FONT><I><FONT size=3> </FONT><A
href="http://openser.org/cgi-bin/mailman/listinfo/users"><FONT size=3>Users at
openser.org</FONT></A><BR></I><FONT size=3>>></FONT><I><FONT size=3>
</FONT><A href="http://openser.org/cgi-bin/mailman/listinfo/users"><FONT
size=3>http://openser.org/cgi-bin/mailman/listinfo/users</FONT></A><BR></I><FONT
size=3>>></FONT><I><BR></I><FONT
size=3>></FONT><I><BR></DIV></I></FONT>
<DIV> </DIV>
<DIV align=left><FONT face=Arial size=2>Γιώργος Παπαδόπουλος</FONT></DIV>
<DIV align=left><FONT face=Arial size=2>Altec Telecoms</FONT></DIV>
<DIV align=left><FONT face=Arial size=2>R&D Department</FONT></DIV>
<DIV align=left><FONT face=Arial size=2></FONT> </DIV>
<DIV align=left><FONT face=Arial size=2>Πάτμου 14, 151 23 Μαρούσι</FONT></DIV>
<DIV align=left><FONT face=Arial size=2>τηλ.: +30 211 687 2933</FONT></DIV>
<DIV align=left><FONT face=Arial size=2>fax.: +30 211 687 2911</FONT></DIV>
<DIV align=left><FONT face=Arial size=2>e-mail: <A
href="mailto:geop@altectelecoms.gr">geop@altectelecoms.gr</A></FONT></DIV>
<DIV> </DIV></BODY><!--[object_id=#altectelecoms.gr#]--><FONT face=Tahoma size=2><FONT color=#0000ff>
<H1><FONT face="Arial Greek" color=#c0c0c0 size=2><SPAN lang=EN-GB>Disclaimer</SPAN></FONT></H1>
<P class=MsoNormal style="MARGIN: 5pt 0cm; mso-layout-grid-align: none"><SPAN lang=EN-US style="FONT-SIZE: 8pt; COLOR: #999999; FONT-FAMILY: Arial; mso-ansi-language: EN-US; mso-fareast-language: EN-US">The information in this e-mail and any attachments is confidential. It is intended solely for the attention and use of the named addressee(s). If you are not the intended recipient, or person responsible for delivering this information to the intended recipient, please notify the sender immediately. Unless you are </SPAN><SPAN lang=EN-GB style="FONT-SIZE: 8pt; COLOR: #999999; FONT-FAMILY: Arial; mso-ansi-language: EN-GB; mso-fareast-language: EN-US">the</SPAN><SPAN lang=EN-US style="FONT-SIZE: 8pt; COLOR: #999999; FONT-FAMILY: Arial; mso-ansi-language: EN-US; mso-fareast-language: EN-US"> intended recipient or his/her representative you are not authorized to, and must not, read, copy, distribute, use or retain this message or any part of it. E-mail transmission cannot be</SPAN><SPAN lang=EN-US style="FONT-SIZE: 8pt; COLOR: #999999; FONT-FAMILY: Arial; mso-ansi-language: EN-GB; mso-fareast-language: EN-US"> </SPAN><SPAN lang=EN-US style="FONT-SIZE: 8pt; COLOR: #999999; FONT-FAMILY: Arial; mso-ansi-language: EN-US; mso-fareast-language: EN-US">guaranteed to be secure or error-free as information could be</SPAN><SPAN lang=EN-US style="FONT-SIZE: 8pt; COLOR: #999999; FONT-FAMILY: Arial; mso-ansi-language: EN-GB; mso-fareast-language: EN-US"> </SPAN><SPAN lang=EN-US style="FONT-SIZE: 8pt; COLOR: #999999; FONT-FAMILY: Arial; mso-ansi-language: EN-US; mso-fareast-language: EN-US">intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses.</SPAN><SPAN lang=EN-GB style="FONT-SIZE: 8pt; COLOR: #999999; FONT-FAMILY: Arial; mso-ansi-language: EN-GB; mso-fareast-language: EN-US"><?xml:namespace prefix = o ns = "urn:schemas-microsoft-com:office:office" /><o:p></o:p></SPAN></P></FONT></FONT></HTML>