Hello,

that backtrace is from shutdown, if you don't have another core file, then the initial one was overwritten. You have to enable one core per process. In most of system it is:

echo "1" > /proc/sys/kernel/core_uses_pid
You have to reproduce the issue again and get the backtrace from all core files.

Cheers,
Daniel

On 12/3/13 8:42 AM, ziad habchi wrote:
Hi ,

Yes it generated a core file; the debuginfos are missing, how can I complie
kamailio without stripping them?

below is the output of bt full:

#0  0x00359bc1 in __strlen_ia32 () from /lib/libc.so.6
No symbol table info available.
#1  0x003244df in vfprintf () from /lib/libc.so.6
No symbol table info available.
#2  0x003beea6 in __vsyslog_chk () from /lib/libc.so.6
No symbol table info available.
#3  0x003bf027 in syslog () from /lib/libc.so.6
No symbol table info available.
#4  0x0819f0a2 in qm_status (qm=0xf52e8000) at mem/q_malloc.c:761
        f = 0xf54ccfbc
        i = 56
        j = <value optimized out>
        h = <value optimized out>
        memlog = 1
        mem_summary = <value optimized out>
        __FUNCTION__ = "qm_status"
#5  0x081a1555 in qm_debug_frag (qm=0xf52e8000, f=<value optimized out>) at
mem/q_malloc.c:160
        __FUNCTION__ = "qm_debug_frag"
#6  0x081a2820 in qm_free (qm=0xf52e8000, p=0xf54ccfd4, file=0xea2ac9 "kex:
pkg_stats.c", func=0xea2b60 "pkg_proc_stats_destroy", line=111) at
mem/q_malloc.c:462
        f = 0xf54ccfbc
        size = <value optimized out>
        next = <value optimized out>
        prev = <value optimized out>
        __FUNCTION__ = "qm_free"
#7  0x00ea1ab6 in pkg_proc_stats_destroy () at pkg_stats.c:111
        __FUNCTION__ = "pkg_proc_stats_destroy"
#8  0x00e9e157 in destroy () at kex_mod.c:170
No locals.
#9  0x0813aeac in destroy_modules () at sr_module.c:790
        t = 0xf73287ec
        foo = 0xf73287ec
        __FUNCTION__ = "destroy_modules"
#10 0x080b4af0 in cleanup (show_status=1) at main.c:573
        memlog = <value optimized out>
        __FUNCTION__ = "cleanup"
#11 0x080b5af9 in shutdown_children (sig=<value optimized out>,
show_status=1) at main.c:715
        __FUNCTION__ = "shutdown_children"
#12 0x080b5ff3 in handle_sigs () at main.c:745
        chld = <value optimized out>
        chld_status = <value optimized out>
        memlog = <value optimized out>
        __FUNCTION__ = "handle_sigs"
#13 0x080b7f57 in main_loop () at main.c:1767
        i = <value optimized out>
        pid = <value optimized out>
        si = <value optimized out>
        si_desc = "udp receiver child=3
sock=193.100.200.18:5070\000\063\367\030\311\341\001\001\000\000\000\234\266
L\365ÖL\365\v\020\000\000l\302\061\365\270\266L\365\270q\364\377\000\000\000
\000X\243\360\b\000\000\000\000\2---Type <return> to continue, or q <return>
to quit---
03\000\000\000X\243\360\b\001", '\000' <repeats 23 times>"\270, q\364\377"
        nrprocs = 4
        __FUNCTION__ = "main_loop"
#14 0x080bb57e in main (argc=5, argv=0xfff47364) at main.c:2566
        cfg_stream = 0x8ebf008
        c = <value optimized out>
        r = 70188
        tmp = 0x80512dd "__libc_start_main"
        tmp_len = 4657636
        port = 136523065
        proto = -757064
        ret = <value optimized out>
        seed = 3203921318
        rfd = 4
        debug_save = 136559068
        debug_flag = 0
        dont_fork_cnt = 0
        n_lst = 0x823b9dc
        p = 0x804aea8 "E\005"
        __FUNCTION__ = "main"

-----Original Message-----
From: sr-users-bounces@lists.sip-router.org
[mailto:sr-users-bounces@lists.sip-router.org] On Behalf Of
Daniel-Constantin Mierla
Sent: Monday, December 2, 2013 7:40 PM
To: Kamailio (SER) - Users Mailing List
Subject: Re: [SR-Users] Memory Leaks with Kamailio using SEAS module

Hello,

did you get a core file? If yes, send here the output for 'bt full' 
taken in gdb.

Cheers,
Daniel

On 12/2/13 5:14 PM, zhabchi wrote:

the following is the error i got when the client connection to SEAS exited

:

  tm [t_reply.c:604]: _reply_light(): ERROR: _reply_light: cannot 
allocate shmem buffer

i recompiled Kamailio with MEMDBG=1 and here what i got

<core> [mem/q_malloc.c:159]: qm_debug_frag(): BUG: qm_*: prev. fragm. 
tail overwritten(c0c0c000, abcdefed)[0xf7336d18:0xf7336d30]!



I am running Kamailio with -m 2048 paramter


zhabchi wrote

Dear Support,

  

I am having an "out of memory" problem while using kamailio with SEAS 
module.

After using Kamailio with the SEAS module for few hours with a high 
load , I am getting "out of memory" error that I believe is caused 
from a memory leak.

I would like your help interpreting this error.

  

The output of kamailio -v is :

  

version: kamailio 4.0.4 (i386/linux) cabe58

flags: STATS: Off, USE_IPV6, USE_TCP, USE_TLS, TLS_HOOKS, 
USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, 
SHM_MMAP, PKG_MALLOC, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, 
USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, 
HAVE_RESOLV_RES

ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, 
MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 4MB

poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.

id: cabe58

compiled on 14:23:27 Dec  2 2013 with gcc 4.1.2

  

kamctl ps :

  

Process::  ID=0 PID=31191 Type=attendant

Process::  ID=1 PID=31192 Type=udp receiver child=0
sock=193.100.200.18:5070

Process::  ID=2 PID=31193 Type=udp receiver child=1
sock=193.100.200.18:5070

Process::  ID=3 PID=31194 Type=udp receiver child=2
sock=193.100.200.18:5070

Process::  ID=4 PID=31195 Type=udp receiver child=3
sock=193.100.200.18:5070

Process::  ID=5 PID=31196 Type=slow timer

Process::  ID=6 PID=31197 Type=timer

Process::  ID=7 PID=31198 Type=ctl handler

Process::  ID=8 PID=31199 Type=MI FIFO

Process::  ID=9 PID=31200 Type=SEAS

Process::  ID=10 PID=31201 Type=tcp receiver (generic) child=0

Process::  ID=11 PID=31202 Type=tcp receiver (generic) child=1

Process::  ID=12 PID=31203 Type=tcp receiver (generic) child=2

Process::  ID=13 PID=31204 Type=tcp receiver (generic) child=3

Process::  ID=14 PID=31205 Type=tcp main process

  

  

kamcmd pkg.stats:

{

                 entry: 0

                 pid: 31191

                 rank: 0

                 used: 72480

                 free: 4091840

                 real_used: 102448

}

{

                 entry: 1

                 pid: 31192

                 rank: 1

                 used: 80624

                 free: 4083696

                 real_used: 110592

}

{

                 entry: 2

                 pid: 31193

                 rank: 2

                 used: 80624

                 free: 4083696

                 real_used: 110592

}

{

                 entry: 3

                 pid: 31194

                 rank: 3

                 used: 80624

                 free: 4083696

                 real_used: 110592

}

{

                 entry: 4

                 pid: 31195

                 rank: 4

                 used: 80624

                 free: 4083696

                 real_used: 110592

}

{

                 entry: 5

                 pid: 31196

                 rank: -1

                 used: 2276560

                 free: 4083744

                 real_used: 17770256

}

{

                 entry: 6

                 pid: 31197

                 rank: -1

                 used: 1787296

                 free: 4083744

                 real_used: 17280992

}

{

                 entry: 7

                 pid: 31198

                 rank: -2

                 used: 77760

                 free: 4086544

                 real_used: 107744

}

{

                 entry: 8

                 pid: 0

                 rank: 0

                 used: 89920

                 free: 4074352

                 real_used: 119936

}

{

                 entry: 9

                 pid: 0

                 rank: 0

                 used: 0

                 free: 0

                 real_used: 0

}

{

                 entry: 10

                 pid: 31201

                 rank: 5

                 used: 140240

                 free: 4023728

                 real_used: 170560

}

{

                 entry: 11

                 pid: 31202

                 rank: 6

                 used: 140240

                 free: 4023712

                 real_used: 170576

}

{

                 entry: 12

                 pid: 31203

                 rank: 7

                 used: 140240

                 free: 4023856

                 real_used: 170432

}

{

                 entry: 13

                 pid: 31204

                 rank: 8

                 used: 140240

                 free: 4023712

                 real_used: 170576

}

{

                 entry: 14

                 pid: 31205

                 rank: -4

                 used: 3877344

                 free: 4030176

                 real_used: 19269056

}

  

As you can see above , the output of the SEAS module is not showing 
(pid
31200)

Moreover , I can see that the real_used value in the kamcmd 
core.shmmem keep on increasing , and free decrease until I finally 
get an error "out of memory"

  

When my SEAS client exit the output of kamcmd core.shmmem :

{

                 total: 33554432

                 free: 16273440

                 used: 1787296

                 real_used: 17280992

                 max_used: 33553520

                 fragments: 8098

}

  

The traffic is a simple SIP MESSAGE from a seagull simulator, with a 
200OK reply from my SEAS client.

  

I appreciate your support,

Thank you in advance,

Ziad Habchi

  

  

  

  


_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing 
list sr-users@.sip-router 
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users




--
View this message in context: 
http://sip-router.1086192.n5.nabble.com/Memory-Leaks-with-Kamailio-usi
ng-SEAS-module-tp123410p123417.html
Sent from the Users mailing list archive at Nabble.com.

_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing 
list sr-users@lists.sip-router.org 
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users

--
Daniel-Constantin Mierla - http://www.asipto.com
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda


_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users@lists.sip-router.org
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users


-----
No virus found in this message.
Checked by AVG - www.avg.com
Version: 2014.0.4259 / Virus Database: 3629/6886 - Release Date: 12/02/13



-- 
Daniel-Constantin Mierla - http://www.asipto.com
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda