On May 27, 2003 at 13:41, Jamin W. Collins jcollins@asgardsrealm.net wrote:
On Tue, May 27, 2003 at 07:34:10PM +0200, Andrei Pelinescu-Onciul wrote:
On May 27, 2003 at 11:16, Jamin W. Collins jcollins@asgardsrealm.net wrote:
I'm not sure which log you are referring to. The server itself has been rebooted by the customer to return functionality. I found the errors after the fact.
Look in the log for lines conatining: "qm_status" , "heap size=", "dumping all allocked. fragments:". They should appear before ser shutdown.
Nothing like that in the syslog or any other log that I can find. There are errors in the logs from SER right up to just prior to the reboot:
May 27 09:41:02 hillcrest /usr/sbin/ser[8258]: process_ins_list(): Error while deleting from database May 27 09:41:02 hillcrest ntpd[269]: ntpd exiting on signal 15 May 27 09:41:04 hillcrest kernel: Kernel logging (proc) stopped. May 27 09:41:04 hillcrest kernel: Kernel log daemon terminating. May 27 09:41:04 hillcrest exiting on signal 15 May 27 09:41:51 hillcrest syslogd 1.4.1#10: restart.
Probably your debug level is < 4 (4 generates lots of debugging messages).
Could you send us the whole log, (or grep ser syslog), the cfg. file and the packet dumps you mentioned? Use serhelp@lists.iptel.org (no limits and not public, only ser developers on it). Maybe we can figure out where the mem. leak is.
Is the local memory allocated per listening process (as reported by 'ps') or per child process (as configured in ser.cfg) or based on something else? Just trying to get a grasp on how much the change will effect the memory usage in my environment.
per listening process ( total no. of processes = child_no * no_of_listening_addresses + 3).
So, with a child_no of 4 and 1 IP listed for listening, I'm looking at 8 listening processes or ~8 Megs of local memory allocated to SER?
That's the maximum. In practice 3 of the processes (the timer, fifo server and the "main" one) hardly use any local memory (probably only a few hundred bytes). There is also a very small probability that all the others will use all their local memory, and due to OS features like copy-on-write when forking, there will be no real 8Mb used.
You could try to start ser with maximum debuging (e.g debug=9) let it run a few days under your expected load, then stop it /send SIGUSR1 and look in the log for the lines: qm_status heap size= ... used= ..., used+overhead=..., free max used (+overhead)= ...
for each process.
max used will be very interesting.
If you do this and max used is close to 1Mb for some processes, then please send us the whole log, it would be much easier to catch the memory leak.
BTW: is your ser compiled with DBG_QM_MALLOC? (try ser -V, it should be unless you edited the Makefile)
Also how much shared memory does SER allocate by default? Would increasing this number possibly help?
32 Mb. In this case it will not help since this is not shared mem. The shared mem. runs out (usually :-)) when ser is in statefull mode (tm is used for forwarding) and a lot of calls are handled almost simultaneously (within sip expire time limits). This also depends on the CPU power (if the machine can handle only 100 cps due to cpu limits you will never exceed the default 32 Mb). On a dual athlon 2000+ which can handle ~4900 cps, 256 Mb are enough (ser -m 256).
The system is a 1.1 Ghz Celeron with 128 Megs and SER's it's only function. So, if I'm following correctly, it might be a good idea to increase the share memory a bit, but the local memory would help more?
Well if ser it's the only function than increase the shared mem.. 80Mb would be more than enough (I don't think a 1.1Ghz Celeron can handle more than 1500-2000cps). If this is a memory leak increasing the local memory won't help, it will only prolong the agony :-) However it could be that in your setup 1Mb of local memory is not enough (maybe there is a "legitimate" consumer) and we should increase the default.
Andrei