24 Jul
2013
24 Jul
'13
10:31 a.m.
Hello,
We don't do any kamctl commands at all. We do have various modules loaded,
as follows. The primary functions we use Kamailio for are phone
registrations through usrloc, and routing calls to Asterisk through logic
contained in Perl via perl_exec().
Thanks for all your advice so far!
loadmodule "tm.so"
loadmodule "tmx.so"
loadmodule "usrloc.so"
loadmodule "auth.so"
loadmodule "auth_db.so"
loadmodule "ctl.so"
loadmodule "db_mysql.so"
loadmodule "kex.so"
loadmodule "maxfwd.so"
loadmodule "mi_fifo.so"
loadmodule "mi_rpc.so"
loadmodule "nathelper.so"
loadmodule "perl.so"
loadmodule "pv.so"
loadmodule "registrar.so"
loadmodule "rr.so"
loadmodule "sanity.so"
loadmodule "siputils.so"
loadmodule "sl.so"
loadmodule "textops.so"
loadmodule "xlog.so"
On 24 July 2013 16:33, Daniel-Constantin Mierla <miconda(a)gmail.com> wrote:
Hello,
On 7/24/13 4:24 AM, David Cunningham wrote:
Hello,
Thank you very much for the email. In reply:
1. The system ran out of memory. Linux's oom-killer killed Kamailio.
then all the instructions I gave are useless, they are for debugging
kamailio's internal memory manager, which handles pkg and shm mallocs.
The chances to be from kamailio itself are very low now. Do you do lot of
mi commands (e.g., kamctl ...)? The mi api uses system malloc, but the rest
of code should use internal memory manager which does not go beyond the
limits set with -m and -M, thus not causing an OS memory exhaustion.
Can you list what modules are you loading? At some point it was a leak in
libssl, in case you use tls a lot. But could be another external library...
Cheers,
Daniel
2. You're right, DEBUG_MEMORY is a local configuration setting. If defined
it sets memdbg to -2, and memlog to -2. The debug setting is -1.
3. We'll try setting mem_summary=12, thanks.
4. We'll try setting asynchronous syslog, thanks.
5. Our configuration totals 338 lines, or approx 8.5kb. Is that a lot?
6. We'll try setting mem_join=1, thanks.
On 23 July 2013 16:53, Daniel-Constantin Mierla <miconda(a)gmail.com> wrote:
--
David Cunningham, Voisonics
http://voisonics.com/
USA: +1 213 221 1092
UK: +44 (0) 20 3298 1642
Australia: +61 (0) 2 8063 9019
Hello,
first, to clarify, is the system memory or kamailio's pkg/shm memory
running out? If the operating system runs out of memory, then should be a
leak in a library, because kamailio modules uses only from a pre-allocated
chunk, not going over it.
On 7/23/13 7:33 AM, David Cunningham wrote:
--
David Cunningham, Voisonics
http://voisonics.com/
USA: +1 213 221 1092
UK: +44 (0) 20 3298 1642
Australia: +61 (0) 2 8063 9019
--
Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda
Hello,
We're running a Kamailio 3.3.4 system, and Kamailio is slowly using more
and more memory. Over a couple of weeks it will run out of system memory.
We tried to enable memory debugging doing the following, but it resulted
in Kamailio not responding to any SIP packets. Would anyone have advice
please on how to debug the situation?
1. In Makefile.defs set MEMDBG to 1 and recompile Kamailio.
2. In kamailio.cfg add the line:
#!define DEBUG_MEMORY 1
do you set something special in config when DEBUG_MEMORY is 1? It is not
by default there, so I assume you added some rules based on this
pre-processor directive.
For memory troubleshooting, set memlog to a value lower than debug
parameter in config file and try with mem_summary=12 for a more compact
output. See more about these parameters in the wiki:
- http://www.kamailio.org/wiki/cookbooks/3.3.x/core#memlog
Run kamailio for a while in normal conditions, then restart it to get the
memory usage summaries. There should be indication if there is some leak,
by seeing memory chunks allocated many times from a function used at
runtime. You can send the memory summary for a process here, we can look at
it.
While this was running and Kamailio didn't
respond to packets, it logged
lots of lines like this:
Do you have syslog to be configured in asynchronous mode? See the notes
from:
- http://www.kamailio.org/wiki/tutorials/3.2.x/syslog
The memdbg is less than debug value, that means printing few log messages
for each memory operation. You can make memdbg higher and rely on memlog
for memory summaries, otherwise will be lot of log messages related to
memory.
Jul 22 21:32:22 hostname kamailio: : <core>
[mem/q_malloc.c:369]:
qm_malloc(0x4000e008, 128) called from <core>: cfg.lex: addstr(1438)
Jul 22 21:32:22 hostname kamailio: : <core> [mem/q_malloc.c:413]:
qm_malloc(0x4000e008, 128) returns address 0x40048918 frag. 0x40048900
(size=128) on 1 -th hit
Jul 22 21:32:22 hostname kamailio: : <core> [mem/q_malloc.c:369]:
qm_malloc(0x4000e008, 128) called from <core>: cfg.lex: addstr(1438)
Jul 22 21:32:22 hostname kamailio: : <core> [mem/q_malloc.c:413]:
qm_malloc(0x4000e008, 128) returns address 0x400489c8 frag. 0x400489b0
(size=128) on 1 -th hit
addstr() is a function used only for parsing configuration file, as long
as you can still see them, the configuration file parsing was not finish.
addstr() is not a source of leaks because it is not used at runtime.
If you have large config file, then you can get close to the limits of
the private memory, which is set to 4MB. You can increase its value using
-M parameter (e.g., start kamailio with -M 8 to set it to use 8MB of
memory).
Over the time, the private memory can get used due to fragmentation, you
can set the mem_join parameter in config file to avoid it (works when
compiled with MEMDBG=1).
To monitor usage of internal pkg memory, then you can use sercmd with
pkg.stats command:
http://kamailio.org/docs/modules/3.3.x/modules_k/kex.html#idp16972640
Shared memory stats are printed by 'kamctl fifo get_statistics shmem:'
When you see significant increase of the memory usage, then you can
restart to get the summaries.
You should run these commands after start, just to see the initial usage
of memory.
Cheers,
Daniel
--
Daniel-Constantin Mierla - http://www.asipto.com
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users(a)lists.sip-router.org
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users