[Serusers] Memory leak?

Andrei Pelinescu-Onciul pelinescu-onciul at fokus.fraunhofer.de
Wed May 28 13:31:45 CEST 2003


On May 27, 2003 at 13:41, Jamin W. Collins <jcollins at asgardsrealm.net> wrote:
> On Tue, May 27, 2003 at 07:34:10PM +0200, Andrei Pelinescu-Onciul wrote:
> > On May 27, 2003 at 11:16, Jamin W. Collins <jcollins at asgardsrealm.net> wrote:
> > 
> > > I'm not sure which log you are referring to.  The server itself has been
> > > rebooted by the customer to return functionality.  I found the errors
> > > after the fact.
> > 
> > Look in the log for lines conatining: "qm_status" , "heap size=", 
> > "dumping all allocked. fragments:". They should appear before ser
> > shutdown.
> 
> Nothing like that in the syslog or any other log that I can find.  There
> are errors in the logs from SER right up to just prior to the reboot:
> 
> May 27 09:41:02 hillcrest /usr/sbin/ser[8258]: process_ins_list(): Error
> while deleting from database
> May 27 09:41:02 hillcrest ntpd[269]: ntpd exiting on signal 15
> May 27 09:41:04 hillcrest kernel: Kernel logging (proc) stopped.
> May 27 09:41:04 hillcrest kernel: Kernel log daemon terminating.
> May 27 09:41:04 hillcrest exiting on signal 15
> May 27 09:41:51 hillcrest syslogd 1.4.1#10: restart.

Probably your debug level is < 4 (4 generates lots of debugging
messages).

Could you send us the whole log, (or  grep ser syslog), the cfg. file
and the packet dumps you mentioned? Use serhelp at lists.iptel.org (no limits and
not public, only ser developers on it). Maybe we can figure out where
the mem. leak is.

> 
> > > Is the local memory allocated per listening process (as reported by
> > > 'ps') or per child process (as configured in ser.cfg) or based on
> > > something else?  Just trying to get a grasp on how much the change will
> > > effect the memory usage in my environment. 
> > 
> > per listening process ( total no. of processes = child_no *
> >  no_of_listening_addresses + 3).
> 
> So, with a child_no of 4 and 1 IP listed for listening, I'm looking at 8
> listening processes or ~8 Megs of local memory allocated to SER?

That's the maximum. In practice 3 of the processes (the timer, fifo
server and the "main" one) hardly use any local memory (probably only a
few hundred bytes). There is also a very small probability that all the
others will use all their local memory, and due to OS features like
copy-on-write when forking, there will be no real 8Mb used.

You could try to start ser with maximum debuging (e.g debug=9) let it
run a few days under your expected load, then stop it /send SIGUSR1 and
look in the log for the lines:
qm_status  
heap size= ...
used= ..., used+overhead=..., free
max used (+overhead)= ...

for each process.

max used will be very interesting.

If you do this and max used is close to 1Mb for some processes, then
please send us the whole log, it would be much easier to catch the
memory leak.

BTW: is your ser compiled with DBG_QM_MALLOC?
(try ser -V, it should be unless you edited the Makefile)
> 
> > > Also how much shared memory does SER allocate by default?  Would
> > > increasing this number possibly help?
> > 
> > 32 Mb. In this case it will not help since this is not shared mem. The
> > shared mem. runs out (usually :-)) when ser is in statefull mode (tm is
> > used for forwarding) and a lot of calls are handled almost
> > simultaneously (within sip expire time limits). This also depends on the
> > CPU power (if the machine can handle only 100 cps due to cpu limits you
> > will never exceed the default 32 Mb). On a dual athlon 2000+ which can
> > handle ~4900 cps, 256 Mb are enough (ser -m 256).
> 
> The system is a 1.1 Ghz Celeron with 128 Megs and SER's it's only
> function.  So, if I'm following correctly, it might be a good idea to
> increase the share memory a bit, but the local memory would help more?

Well if ser it's the only function than increase the shared mem.. 80Mb
would be more than enough (I don't think a 1.1Ghz Celeron can handle
more than 1500-2000cps).
If this is a memory leak increasing the local memory won't help, it will
only prolong the agony :-) However it could be that in your setup 1Mb of
local memory is not enough (maybe there is a "legitimate" consumer) and
we should increase the default.


Andrei




More information about the sr-users mailing list