Hello,
that backtrace is from shutdown, if you don't have another core file, then the initial one was overwritten. You have to enable one core per process. In most of system it is:
echo "1" > /proc/sys/kernel/core_uses_pid
You have to reproduce the issue again and get the backtrace from all core files.
Cheers, Daniel
On 12/3/13 8:42 AM, ziad habchi wrote:
Hi ,
Yes it generated a core file; the debuginfos are missing, how can I complie kamailio without stripping them?
below is the output of bt full:
#0 0x00359bc1 in __strlen_ia32 () from /lib/libc.so.6 No symbol table info available. #1 0x003244df in vfprintf () from /lib/libc.so.6 No symbol table info available. #2 0x003beea6 in __vsyslog_chk () from /lib/libc.so.6 No symbol table info available. #3 0x003bf027 in syslog () from /lib/libc.so.6 No symbol table info available. #4 0x0819f0a2 in qm_status (qm=0xf52e8000) at mem/q_malloc.c:761 f = 0xf54ccfbc i = 56 j = <value optimized out> h = <value optimized out> memlog = 1 mem_summary = <value optimized out> __FUNCTION__ = "qm_status" #5 0x081a1555 in qm_debug_frag (qm=0xf52e8000, f=<value optimized out>) at mem/q_malloc.c:160 __FUNCTION__ = "qm_debug_frag" #6 0x081a2820 in qm_free (qm=0xf52e8000, p=0xf54ccfd4, file=0xea2ac9 "kex: pkg_stats.c", func=0xea2b60 "pkg_proc_stats_destroy", line=111) at mem/q_malloc.c:462 f = 0xf54ccfbc size = <value optimized out> next = <value optimized out> prev = <value optimized out> __FUNCTION__ = "qm_free" #7 0x00ea1ab6 in pkg_proc_stats_destroy () at pkg_stats.c:111 __FUNCTION__ = "pkg_proc_stats_destroy" #8 0x00e9e157 in destroy () at kex_mod.c:170 No locals. #9 0x0813aeac in destroy_modules () at sr_module.c:790 t = 0xf73287ec foo = 0xf73287ec __FUNCTION__ = "destroy_modules" #10 0x080b4af0 in cleanup (show_status=1) at main.c:573 memlog = <value optimized out> __FUNCTION__ = "cleanup" #11 0x080b5af9 in shutdown_children (sig=<value optimized out>, show_status=1) at main.c:715 __FUNCTION__ = "shutdown_children" #12 0x080b5ff3 in handle_sigs () at main.c:745 chld = <value optimized out> chld_status = <value optimized out> memlog = <value optimized out> __FUNCTION__ = "handle_sigs" #13 0x080b7f57 in main_loop () at main.c:1767 i = <value optimized out> pid = <value optimized out> si = <value optimized out> si_desc = "udp receiver child=3 sock=193.100.200.18:5070\000\063\367\030\311\341\001\001\000\000\000\234\266 L\365жL\365\v\020\000\000l\302\061\365\270\266L\365\270q\364\377\000\000\000 \000X\243\360\b\000\000\000\000\2---Type <return> to continue, or q <return> to quit--- 03\000\000\000X\243\360\b\001", '\000' <repeats 23 times>"\270, q\364\377" nrprocs = 4 __FUNCTION__ = "main_loop" #14 0x080bb57e in main (argc=5, argv=0xfff47364) at main.c:2566 cfg_stream = 0x8ebf008 c = <value optimized out> r = 70188 tmp = 0x80512dd "__libc_start_main" tmp_len = 4657636 port = 136523065 proto = -757064 ret = <value optimized out> seed = 3203921318 rfd = 4 debug_save = 136559068 debug_flag = 0 dont_fork_cnt = 0 n_lst = 0x823b9dc p = 0x804aea8 "E\005" __FUNCTION__ = "main"
-----Original Message----- From: sr-users-bounces@lists.sip-router.org [mailto:sr-users-bounces@lists.sip-router.org] On Behalf Of Daniel-Constantin Mierla Sent: Monday, December 2, 2013 7:40 PM To: Kamailio (SER) - Users Mailing List Subject: Re: [SR-Users] Memory Leaks with Kamailio using SEAS module
Hello,
did you get a core file? If yes, send here the output for 'bt full' taken in gdb.
Cheers, Daniel
On 12/2/13 5:14 PM, zhabchi wrote:
the following is the error i got when the client connection to SEAS exited
:
tm [t_reply.c:604]: _reply_light(): ERROR: _reply_light: cannot allocate shmem buffer
i recompiled Kamailio with MEMDBG=1 and here what i got
<core> [mem/q_malloc.c:159]: qm_debug_frag(): BUG: qm_*: prev. fragm. tail overwritten(c0c0c000, abcdefed)[0xf7336d18:0xf7336d30]!
I am running Kamailio with -m 2048 paramter
zhabchi wrote
Dear Support,
I am having an "out of memory" problem while using kamailio with SEAS module.
After using Kamailio with the SEAS module for few hours with a high load , I am getting "out of memory" error that I believe is caused from a memory leak.
I would like your help interpreting this error.
The output of kamailio -v is :
version: kamailio 4.0.4 (i386/linux) cabe58
flags: STATS: Off, USE_IPV6, USE_TCP, USE_TLS, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 4MB
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
id: cabe58
compiled on 14:23:27 Dec 2 2013 with gcc 4.1.2
kamctl ps :
Process:: ID=0 PID=31191 Type=attendant
Process:: ID=1 PID=31192 Type=udp receiver child=0 sock=193.100.200.18:5070
Process:: ID=2 PID=31193 Type=udp receiver child=1 sock=193.100.200.18:5070
Process:: ID=3 PID=31194 Type=udp receiver child=2 sock=193.100.200.18:5070
Process:: ID=4 PID=31195 Type=udp receiver child=3 sock=193.100.200.18:5070
Process:: ID=5 PID=31196 Type=slow timer
Process:: ID=6 PID=31197 Type=timer
Process:: ID=7 PID=31198 Type=ctl handler
Process:: ID=8 PID=31199 Type=MI FIFO
Process:: ID=9 PID=31200 Type=SEAS
Process:: ID=10 PID=31201 Type=tcp receiver (generic) child=0
Process:: ID=11 PID=31202 Type=tcp receiver (generic) child=1
Process:: ID=12 PID=31203 Type=tcp receiver (generic) child=2
Process:: ID=13 PID=31204 Type=tcp receiver (generic) child=3
Process:: ID=14 PID=31205 Type=tcp main process
kamcmd pkg.stats:
{
entry: 0 pid: 31191 rank: 0 used: 72480 free: 4091840 real_used: 102448}
{
entry: 1 pid: 31192 rank: 1 used: 80624 free: 4083696 real_used: 110592}
{
entry: 2 pid: 31193 rank: 2 used: 80624 free: 4083696 real_used: 110592}
{
entry: 3 pid: 31194 rank: 3 used: 80624 free: 4083696 real_used: 110592}
{
entry: 4 pid: 31195 rank: 4 used: 80624 free: 4083696 real_used: 110592}
{
entry: 5 pid: 31196 rank: -1 used: 2276560 free: 4083744 real_used: 17770256}
{
entry: 6 pid: 31197 rank: -1 used: 1787296 free: 4083744 real_used: 17280992}
{
entry: 7 pid: 31198 rank: -2 used: 77760 free: 4086544 real_used: 107744}
{
entry: 8 pid: 0 rank: 0 used: 89920 free: 4074352 real_used: 119936}
{
entry: 9 pid: 0 rank: 0 used: 0 free: 0 real_used: 0}
{
entry: 10 pid: 31201 rank: 5 used: 140240 free: 4023728 real_used: 170560}
{
entry: 11 pid: 31202 rank: 6 used: 140240 free: 4023712 real_used: 170576}
{
entry: 12 pid: 31203 rank: 7 used: 140240 free: 4023856 real_used: 170432}
{
entry: 13 pid: 31204 rank: 8 used: 140240 free: 4023712 real_used: 170576}
{
entry: 14 pid: 31205 rank: -4 used: 3877344 free: 4030176 real_used: 19269056}
As you can see above , the output of the SEAS module is not showing (pid 31200)
Moreover , I can see that the real_used value in the kamcmd core.shmmem keep on increasing , and free decrease until I finally get an error "out of memory"
When my SEAS client exit the output of kamcmd core.shmmem :
{
total: 33554432 free: 16273440 used: 1787296 real_used: 17280992 max_used: 33553520 fragments: 8098}
The traffic is a simple SIP MESSAGE from a seagull simulator, with a 200OK reply from my SEAS client.
I appreciate your support,
Thank you in advance,
Ziad Habchi
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@.sip-router http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
-- View this message in context: http://sip-router.1086192.n5.nabble.com/Memory-Leaks-with-Kamailio-usi ng-SEAS-module-tp123410p123417.html Sent from the Users mailing list archive at Nabble.com.
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
-- Daniel-Constantin Mierla - http://www.asipto.com http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
No virus found in this message. Checked by AVG - www.avg.com Version: 2014.0.4259 / Virus Database: 3629/6886 - Release Date: 12/02/13