[Users] memory issues

[Christian Schlatter]

Christian Schlatter wrote:
...
> 
> I always had 768MB shared memory configured though, so I still can't 
> explain the memory allocation errors I got. Some more test runs revealed 
> that I only get these errors when using a more production oriented 
> config that loads more modules than the one posted in my earlier email. 
> I now try to figure out what exactly causes these memory allocation 
> errors that happen reproducibly after about 220s at 400 cps.

I think I found the cause for the memory allocation errors. As soon as I 
include an AVP write operation in the routing script, I get 'out of 
memory' messages after a certain number of calls generated with sipp.

The routing script to reproduce this behavior looks like (full config 
available at http://www.unc.edu/~cschlatt/openser/openser.cfg):

route{
         $avp(s:ct) = $ct; # commenting this line solves
			  # the memory problem

         if (!method=="REGISTER") record_route();
         if (loose_route()) route(1);

         if (uri==myself) rewritehost("xx.xx.xx.xx");
         route(1);
}

route[1] {
         if (!t_relay()) sl_reply_error();
         exit;
}

An example log file showing the 'out of memory' messages is available at 
http://www.unc.edu/~cschlatt/openser/openser.log .

Some observations:

- The 'out of memory' messages always appear after about 8000 test calls 
per worker process. One call consists of two SIP transactions and six 
end-to-end SIP messages. An openser with 8 children handles about 64'000 
calls, whereas 4 children only handle about 32'000 calls. The sipp call 
rate doesn't matter, only number of calls.

- The 8000 calls per worker process are independent from the amount of 
shared memory available. Running openser with -m 128 or -m 768 does not 
make a difference.

- The more AVP writes are done in the script, the less calls go through. 
It looks like each AVP write is leaking memory (unnoticed by the memory 
statistics).

- The fifo memory statistics do not reflect the 'out of memory' syslog 
messages. Even if openser does not route a single SIP message because of 
memory issues, the statistics still show a lot of 'free' memory.


All tests were done with openser SVN 1.2 branch on Ubuntu dapper x86. I 
think the same is true for 1.1 version but I haven't tested that yet.


Christian