[SR-Users] Memory Leaks with Kamailio using SEAS module

Daniel-Constantin Mierla miconda at gmail.com
Tue Dec 3 09:00:08 CET 2013


Hello,

that backtrace is from shutdown, if you don't have another core file, 
then the initial one was overwritten. You have to enable one core per 
process. In most of system it is:

echo "1" > /proc/sys/kernel/core_uses_pid

You have to reproduce the issue again and get the backtrace from all 
core files.

Cheers,
Daniel

On 12/3/13 8:42 AM, ziad habchi wrote:
> Hi ,
>
> Yes it generated a core file; the debuginfos are missing, how can I complie
> kamailio without stripping them?
>
> below is the output of bt full:
>
> #0  0x00359bc1 in __strlen_ia32 () from /lib/libc.so.6
> No symbol table info available.
> #1  0x003244df in vfprintf () from /lib/libc.so.6
> No symbol table info available.
> #2  0x003beea6 in __vsyslog_chk () from /lib/libc.so.6
> No symbol table info available.
> #3  0x003bf027 in syslog () from /lib/libc.so.6
> No symbol table info available.
> #4  0x0819f0a2 in qm_status (qm=0xf52e8000) at mem/q_malloc.c:761
>          f = 0xf54ccfbc
>          i = 56
>          j = <value optimized out>
>          h = <value optimized out>
>          memlog = 1
>          mem_summary = <value optimized out>
>          __FUNCTION__ = "qm_status"
> #5  0x081a1555 in qm_debug_frag (qm=0xf52e8000, f=<value optimized out>) at
> mem/q_malloc.c:160
>          __FUNCTION__ = "qm_debug_frag"
> #6  0x081a2820 in qm_free (qm=0xf52e8000, p=0xf54ccfd4, file=0xea2ac9 "kex:
> pkg_stats.c", func=0xea2b60 "pkg_proc_stats_destroy", line=111) at
> mem/q_malloc.c:462
>          f = 0xf54ccfbc
>          size = <value optimized out>
>          next = <value optimized out>
>          prev = <value optimized out>
>          __FUNCTION__ = "qm_free"
> #7  0x00ea1ab6 in pkg_proc_stats_destroy () at pkg_stats.c:111
>          __FUNCTION__ = "pkg_proc_stats_destroy"
> #8  0x00e9e157 in destroy () at kex_mod.c:170
> No locals.
> #9  0x0813aeac in destroy_modules () at sr_module.c:790
>          t = 0xf73287ec
>          foo = 0xf73287ec
>          __FUNCTION__ = "destroy_modules"
> #10 0x080b4af0 in cleanup (show_status=1) at main.c:573
>          memlog = <value optimized out>
>          __FUNCTION__ = "cleanup"
> #11 0x080b5af9 in shutdown_children (sig=<value optimized out>,
> show_status=1) at main.c:715
>          __FUNCTION__ = "shutdown_children"
> #12 0x080b5ff3 in handle_sigs () at main.c:745
>          chld = <value optimized out>
>          chld_status = <value optimized out>
>          memlog = <value optimized out>
>          __FUNCTION__ = "handle_sigs"
> #13 0x080b7f57 in main_loop () at main.c:1767
>          i = <value optimized out>
>          pid = <value optimized out>
>          si = <value optimized out>
>          si_desc = "udp receiver child=3
> sock=193.100.200.18:5070\000\063\367\030\311\341\001\001\000\000\000\234\266
> L\365жL\365\v\020\000\000l\302\061\365\270\266L\365\270q\364\377\000\000\000
> \000X\243\360\b\000\000\000\000\2---Type <return> to continue, or q <return>
> to quit---
> 03\000\000\000X\243\360\b\001", '\000' <repeats 23 times>"\270, q\364\377"
>          nrprocs = 4
>          __FUNCTION__ = "main_loop"
> #14 0x080bb57e in main (argc=5, argv=0xfff47364) at main.c:2566
>          cfg_stream = 0x8ebf008
>          c = <value optimized out>
>          r = 70188
>          tmp = 0x80512dd "__libc_start_main"
>          tmp_len = 4657636
>          port = 136523065
>          proto = -757064
>          ret = <value optimized out>
>          seed = 3203921318
>          rfd = 4
>          debug_save = 136559068
>          debug_flag = 0
>          dont_fork_cnt = 0
>          n_lst = 0x823b9dc
>          p = 0x804aea8 "E\005"
>          __FUNCTION__ = "main"
>
> -----Original Message-----
> From: sr-users-bounces at lists.sip-router.org
> [mailto:sr-users-bounces at lists.sip-router.org] On Behalf Of
> Daniel-Constantin Mierla
> Sent: Monday, December 2, 2013 7:40 PM
> To: Kamailio (SER) - Users Mailing List
> Subject: Re: [SR-Users] Memory Leaks with Kamailio using SEAS module
>
> Hello,
>
> did you get a core file? If yes, send here the output for 'bt full'
> taken in gdb.
>
> Cheers,
> Daniel
>
> On 12/2/13 5:14 PM, zhabchi wrote:
>> the following is the error i got when the client connection to SEAS exited
> :
>>    tm [t_reply.c:604]: _reply_light(): ERROR: _reply_light: cannot
>> allocate shmem buffer
>>
>> i recompiled Kamailio with MEMDBG=1 and here what i got
>>
>> <core> [mem/q_malloc.c:159]: qm_debug_frag(): BUG: qm_*: prev. fragm.
>> tail overwritten(c0c0c000, abcdefed)[0xf7336d18:0xf7336d30]!
>>
>>
>>
>> I am running Kamailio with -m 2048 paramter
>>
>>
>> zhabchi wrote
>>> Dear Support,
>>>
>>>    
>>>
>>> I am having an "out of memory" problem while using kamailio with SEAS
>>> module.
>>>
>>> After using Kamailio with the SEAS module for few hours with a high
>>> load , I am getting "out of memory" error that I believe is caused
>>> from a memory leak.
>>>
>>> I would like your help interpreting this error.
>>>
>>>    
>>>
>>> The output of kamailio -v is :
>>>
>>>    
>>>
>>> version: kamailio 4.0.4 (i386/linux) cabe58
>>>
>>> flags: STATS: Off, USE_IPV6, USE_TCP, USE_TLS, TLS_HOOKS,
>>> USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM,
>>> SHM_MMAP, PKG_MALLOC, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT,
>>> USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST,
>>> HAVE_RESOLV_RES
>>>
>>> ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16,
>>> MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 4MB
>>>
>>> poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
>>>
>>> id: cabe58
>>>
>>> compiled on 14:23:27 Dec  2 2013 with gcc 4.1.2
>>>
>>>    
>>>
>>> kamctl ps :
>>>
>>>    
>>>
>>> Process::  ID=0 PID=31191 Type=attendant
>>>
>>> Process::  ID=1 PID=31192 Type=udp receiver child=0
>>> sock=193.100.200.18:5070
>>>
>>> Process::  ID=2 PID=31193 Type=udp receiver child=1
>>> sock=193.100.200.18:5070
>>>
>>> Process::  ID=3 PID=31194 Type=udp receiver child=2
>>> sock=193.100.200.18:5070
>>>
>>> Process::  ID=4 PID=31195 Type=udp receiver child=3
>>> sock=193.100.200.18:5070
>>>
>>> Process::  ID=5 PID=31196 Type=slow timer
>>>
>>> Process::  ID=6 PID=31197 Type=timer
>>>
>>> Process::  ID=7 PID=31198 Type=ctl handler
>>>
>>> Process::  ID=8 PID=31199 Type=MI FIFO
>>>
>>> Process::  ID=9 PID=31200 Type=SEAS
>>>
>>> Process::  ID=10 PID=31201 Type=tcp receiver (generic) child=0
>>>
>>> Process::  ID=11 PID=31202 Type=tcp receiver (generic) child=1
>>>
>>> Process::  ID=12 PID=31203 Type=tcp receiver (generic) child=2
>>>
>>> Process::  ID=13 PID=31204 Type=tcp receiver (generic) child=3
>>>
>>> Process::  ID=14 PID=31205 Type=tcp main process
>>>
>>>    
>>>
>>>    
>>>
>>> kamcmd pkg.stats:
>>>
>>> {
>>>
>>>                   entry: 0
>>>
>>>                   pid: 31191
>>>
>>>                   rank: 0
>>>
>>>                   used: 72480
>>>
>>>                   free: 4091840
>>>
>>>                   real_used: 102448
>>>
>>> }
>>>
>>> {
>>>
>>>                   entry: 1
>>>
>>>                   pid: 31192
>>>
>>>                   rank: 1
>>>
>>>                   used: 80624
>>>
>>>                   free: 4083696
>>>
>>>                   real_used: 110592
>>>
>>> }
>>>
>>> {
>>>
>>>                   entry: 2
>>>
>>>                   pid: 31193
>>>
>>>                   rank: 2
>>>
>>>                   used: 80624
>>>
>>>                   free: 4083696
>>>
>>>                   real_used: 110592
>>>
>>> }
>>>
>>> {
>>>
>>>                   entry: 3
>>>
>>>                   pid: 31194
>>>
>>>                   rank: 3
>>>
>>>                   used: 80624
>>>
>>>                   free: 4083696
>>>
>>>                   real_used: 110592
>>>
>>> }
>>>
>>> {
>>>
>>>                   entry: 4
>>>
>>>                   pid: 31195
>>>
>>>                   rank: 4
>>>
>>>                   used: 80624
>>>
>>>                   free: 4083696
>>>
>>>                   real_used: 110592
>>>
>>> }
>>>
>>> {
>>>
>>>                   entry: 5
>>>
>>>                   pid: 31196
>>>
>>>                   rank: -1
>>>
>>>                   used: 2276560
>>>
>>>                   free: 4083744
>>>
>>>                   real_used: 17770256
>>>
>>> }
>>>
>>> {
>>>
>>>                   entry: 6
>>>
>>>                   pid: 31197
>>>
>>>                   rank: -1
>>>
>>>                   used: 1787296
>>>
>>>                   free: 4083744
>>>
>>>                   real_used: 17280992
>>>
>>> }
>>>
>>> {
>>>
>>>                   entry: 7
>>>
>>>                   pid: 31198
>>>
>>>                   rank: -2
>>>
>>>                   used: 77760
>>>
>>>                   free: 4086544
>>>
>>>                   real_used: 107744
>>>
>>> }
>>>
>>> {
>>>
>>>                   entry: 8
>>>
>>>                   pid: 0
>>>
>>>                   rank: 0
>>>
>>>                   used: 89920
>>>
>>>                   free: 4074352
>>>
>>>                   real_used: 119936
>>>
>>> }
>>>
>>> {
>>>
>>>                   entry: 9
>>>
>>>                   pid: 0
>>>
>>>                   rank: 0
>>>
>>>                   used: 0
>>>
>>>                   free: 0
>>>
>>>                   real_used: 0
>>>
>>> }
>>>
>>> {
>>>
>>>                   entry: 10
>>>
>>>                   pid: 31201
>>>
>>>                   rank: 5
>>>
>>>                   used: 140240
>>>
>>>                   free: 4023728
>>>
>>>                   real_used: 170560
>>>
>>> }
>>>
>>> {
>>>
>>>                   entry: 11
>>>
>>>                   pid: 31202
>>>
>>>                   rank: 6
>>>
>>>                   used: 140240
>>>
>>>                   free: 4023712
>>>
>>>                   real_used: 170576
>>>
>>> }
>>>
>>> {
>>>
>>>                   entry: 12
>>>
>>>                   pid: 31203
>>>
>>>                   rank: 7
>>>
>>>                   used: 140240
>>>
>>>                   free: 4023856
>>>
>>>                   real_used: 170432
>>>
>>> }
>>>
>>> {
>>>
>>>                   entry: 13
>>>
>>>                   pid: 31204
>>>
>>>                   rank: 8
>>>
>>>                   used: 140240
>>>
>>>                   free: 4023712
>>>
>>>                   real_used: 170576
>>>
>>> }
>>>
>>> {
>>>
>>>                   entry: 14
>>>
>>>                   pid: 31205
>>>
>>>                   rank: -4
>>>
>>>                   used: 3877344
>>>
>>>                   free: 4030176
>>>
>>>                   real_used: 19269056
>>>
>>> }
>>>
>>>    
>>>
>>> As you can see above , the output of the SEAS module is not showing
>>> (pid
>>> 31200)
>>>
>>> Moreover , I can see that the real_used value in the kamcmd
>>> core.shmmem keep on increasing , and free decrease until I finally
>>> get an error "out of memory"
>>>
>>>    
>>>
>>> When my SEAS client exit the output of kamcmd core.shmmem :
>>>
>>> {
>>>
>>>                   total: 33554432
>>>
>>>                   free: 16273440
>>>
>>>                   used: 1787296
>>>
>>>                   real_used: 17280992
>>>
>>>                   max_used: 33553520
>>>
>>>                   fragments: 8098
>>>
>>> }
>>>
>>>    
>>>
>>> The traffic is a simple SIP MESSAGE from a seagull simulator, with a
>>> 200OK reply from my SEAS client.
>>>
>>>    
>>>
>>> I appreciate your support,
>>>
>>> Thank you in advance,
>>>
>>> Ziad Habchi
>>>
>>>    
>>>
>>>    
>>>
>>>    
>>>
>>>    
>>>
>>>
>>> _______________________________________________
>>> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing
>>> list sr-users at .sip-router
>>> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
>>
>>
>>
>> --
>> View this message in context:
>> http://sip-router.1086192.n5.nabble.com/Memory-Leaks-with-Kamailio-usi
>> ng-SEAS-module-tp123410p123417.html
>> Sent from the Users mailing list archive at Nabble.com.
>>
>> _______________________________________________
>> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing
>> list sr-users at lists.sip-router.org
>> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
> --
> Daniel-Constantin Mierla - http://www.asipto.com
> http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
>
>
> _______________________________________________
> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
> sr-users at lists.sip-router.org
> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
>
>
> -----
> No virus found in this message.
> Checked by AVG - www.avg.com
> Version: 2014.0.4259 / Virus Database: 3629/6886 - Release Date: 12/02/13
>

-- 
Daniel-Constantin Mierla - http://www.asipto.com
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.sip-router.org/pipermail/sr-users/attachments/20131203/e5369041/attachment-0001.html>


More information about the sr-users mailing list