[SR-Users] Memory leak

Daniel-Constantin Mierla miconda at gmail.com
Tue Sep 21 16:50:54 CEST 2010


  Hello,

is it compile with memory debug on? The status shows it is still fast 
malloc.

Cheers,
Daniel

On 9/21/10 4:41 PM, Santiago Soares wrote:
> Thanks for the answers.
> Here are the config file and the memory status right now.
> I didn't change the memory pool.
> I don't know how to interpret the memory log. Can you help me?
>
>
> Config file
>
> # ----------- global configuration parameters ------------------------
> # vim:bg=dark
>
> check_via=no      # (cmd. line: -v)
> dns=no          # (cmd. line: -r)
> rev_dns=no      # (cmd. line: -R)
> dns_cache_init=no
> use_dns_cache=no
> disable_tcp=yes
> server_header="Server: SIP ROUTER"
> user_agent_header="User-Agent: SIP ROUTER"
> sip_warning=0
> debug=2
> memlog=1
>
> port=5060
>
> # ------------------ module loading ----------------------------------
>
> mpath="/usr/local/lib/kamailio/modules/:/usr/local/lib/kamailio/modules_k/"
>
> loadmodule "maxfwd.so"
> loadmodule "exec.so"
> loadmodule "tm.so"
> loadmodule "sl.so"
> loadmodule "dispatcher.so"
> loadmodule "textops.so"
> loadmodule "xlog.so"
> loadmodule "mi_fifo.so"
> loadmodule "regex.so"
> loadmodule "pv.so"
>
> # ----------- setting module-specific parameters ---------------------
>
> modparam("exec", "setvars", 0)
> modparam("dispatcher", "list_file", 
> "/usr/local/etc/kamailio/servers.list")
> modparam("mi_fifo", "fifo_name", "/tmp/kamailio_fifo")
> modparam("dispatcher", "ds_ping_interval", 5)
> modparam("dispatcher", "ds_probing_mode", 1)
> modparam("dispatcher", "ds_probing_threshhold", 3)
> modparam("dispatcher", "flags", 2)
> modparam("tm", "fr_timer", 3)
>
> # ------------------ request routing logic ---------------------------
>
> route {
>
>
>         # initial sanity checks:
>         # messages with max_forwards==0
>         if (!mf_process_maxfwd_header("10")) {
>                 sl_send_reply("483","Too Many Hops");
>                 exit;
>         };
>
>         # excessively long requests
>         if (msg:len >=  2048 ) {
>                 sl_send_reply("513", "Message too big");
>                 exit;
>         };
>
>         # excessively long display name
>         if (!pcre_match("$fn","^.{0,62}$$")) {
>                 sl_send_reply("403", "Forbidden - Display Name Max 
> Size Exceeded");
>                 exit;
>         };
>
>         #Register
>         if (method=="REGISTER") {
>                 if 
> (from_uri=~".*(<?sip:[a-z][a-z0-9_.]{3,11}(\$[0-9]{1,3})?@voiprovider\.com(:[0-9]{4})?(;[^>]+)?>?) 
> ?(;tag=[a-zA-Z0-9]{4,}.*)?$" ||
>                     
> from_uri=~".*(<?sip:\$voipmonitor at voiprovider\.com(:[0-9]{4})?(;[^>]+)?>?) 
> ?(;tag=[a-zA-Z0-9]{4,}.*)?$") {
>                         route(4);
>                         exit;
>                 };
>                 sl_send_reply("403","Forbidden - Wrong domain or 
> Username format");
>                 exit;
>         };
>
>         #Deny OPTIONS
>         if (method=="OPTIONS") {
>                 sl_send_reply("404", "Not found");
>                 exit;
>         };
>
>         #Deny PING
>         if (method=="PING") {
>                 sl_send_reply("404", "Not found");
>                 exit;
>         };
>
>
>         # INVITE
>         if (method=="INVITE") {
>
>                  if 
> (from_uri=~".*(<?sip:[a-z][a-z0-9_.]{3,11}(\$[0-9]{1,3})?@voiprovider\.com(:[0-9]{4})?(;[^>]+)?>?) 
> ?(;tag=[a-zA-Z0-9]{4,}.*)?$" ||
>                     
> from_uri=~"@equip[0-9][0-9]?.voiprovider\.com($|;|:|>)" ||
>                     from_uri=~"@10.0.0.1 <http://10.0.0.1>($|;|:|>)" ||
>                     from_uri=~"@10.0.0.2 <http://10.0.0.2>($|;|:|>)" ||
>                     from_uri=~"@10.0.0.3 <http://10.0.0.3>($|;|:|>)") {
>
>                         # Skype
>                         if (to_uri=~"\#skype@") {
>                                 route(2);
>                                 exit;
>                         };
>
>                         # Skype
>                         if (to_uri=~"%23skype@") {
>                                 route(2);
>                                 exit;
>                         };
>
>                         # default
>                         route(1);
>                         exit;
>
>                 };
>
>                 sl_send_reply("403","Forbidden - Wrong domain or 
> Username format");
>                 exit;
>
>         };
>
>         # Default
>         route(1);
>         exit;
> }
> #Default (softswitches)
> route[1] {
>         ds_select_dst("1", "0");
>         forward();
>         exit;
> }
> #Skype
> route[2] {
>         ds_select_dst("2", "0");
>         forward();
>         exit;
> }
>
> #Register
> route[4] {
>         ds_select_dst("4", "0");
>         forward();
>         exit;
> }
>
>
> Memory log:
>
> Sep 21 11:31:02 siprouter /usr/local/sbin/kamailio[766]: NOTICE: 
> <core> [main.c:691]: Memory status (pkg):
> Sep 21 11:31:02 siprouter /usr/local/sbin/kamailio[766]: NOTICE: 
> fm_status: fm_status (0x88d5c0):
> Sep 21 11:31:02 siprouter /usr/local/sbin/kamailio[766]: NOTICE: 
> fm_status:  heap size= 4194304
> Sep 21 11:31:02 siprouter /usr/local/sbin/kamailio[766]: NOTICE: 
> fm_status:  used= 126576, used+overhead=169192, free=4025112
> Sep 21 11:31:02 siprouter /usr/local/sbin/kamailio[766]: NOTICE: 
> fm_status:  max used (+overhead)= 204008
> Sep 21 11:31:02 siprouter /usr/local/sbin/kamailio[766]: NOTICE: 
> fm_status: dumping free list:
> Sep 21 11:31:02 siprouter /usr/local/sbin/kamailio[766]: NOTICE: 
> fm_status: hash =   1 fragments no.:     1, unused:     
> 0                  bucket size:         8 -         8 (first         8)
> Sep 21 11:31:02 siprouter /usr/local/sbin/kamailio[766]: NOTICE: 
> fm_status: hash =   2 fragments no.:     2, unused:     
> 0                  bucket size:        16 -        16 (first        16)
> Sep 21 11:31:02 siprouter /usr/local/sbin/kamailio[766]: NOTICE: 
> fm_status: hash =   3 fragments no.:     8, unused:     
> 0                  bucket size:        24 -        24 (first        24)
> Sep 21 11:31:02 siprouter /usr/local/sbin/kamailio[766]: NOTICE: 
> fm_status: hash =   4 fragments no.:     1, unused:     
> 0                  bucket size:        32 -        32 (first        32)
> Sep 21 11:31:02 siprouter /usr/local/sbin/kamailio[766]: NOTICE: 
> fm_status: hash =   5 fragments no.:     3, unused:     
> 0                  bucket size:        40 -        40 (first        40)
> Sep 21 11:31:02 siprouter /usr/local/sbin/kamailio[766]: NOTICE: 
> fm_status: hash =  54 fragments no.:     1, unused:     
> 0                  bucket size:       432 -       432 (first       432)
> Sep 21 11:31:02 siprouter /usr/local/sbin/kamailio[766]: NOTICE: 
> fm_status: hash = 216 fragments no.:    20, unused:     
> 0                  bucket size:      1728 -      1728 (first      1728)
> Sep 21 11:31:02 siprouter /usr/local/sbin/kamailio[766]: NOTICE: 
> fm_status: hash = 2056 fragments no.:     1, unused:     
> 0                 bucket size:   2097152 -   4194304 (first   3989736)
> Sep 21 11:31:02 siprouter /usr/local/sbin/kamailio[766]: NOTICE: 
> fm_status: TOTAL:     37 free fragments = 4025112 free bytes
> Sep 21 11:31:02 siprouter /usr/local/sbin/kamailio[766]: NOTICE: 
> fm_status: -----------------------------
> Sep 21 11:31:02 siprouter /usr/local/sbin/kamailio[766]: NOTICE: 
> <core> [main.c:703]: Memory status (shm):
> Sep 21 11:31:02 siprouter /usr/local/sbin/kamailio[766]: NOTICE: 
> fm_status: fm_status (0x2b982b03b000):
> Sep 21 11:31:02 siprouter /usr/local/sbin/kamailio[766]: NOTICE: 
> fm_status:  heap size= 33554432
> Sep 21 11:31:02 siprouter /usr/local/sbin/kamailio[766]: NOTICE: 
> fm_status:  used= 3071760, used+overhead=3109608, free=30444824
> Sep 21 11:31:02 siprouter /usr/local/sbin/kamailio[766]: NOTICE: 
> fm_status:  max used (+overhead)= 3235640
> Sep 21 11:31:02 siprouter /usr/local/sbin/kamailio[766]: NOTICE: 
> fm_status: dumping free list:
> Sep 21 11:31:02 siprouter /usr/local/sbin/kamailio[766]: NOTICE: 
> fm_status: hash =   3 fragments no.:     2, unused:     
> 0                  bucket size:        24 -        24 (first        24)
> Sep 21 11:31:02 siprouter /usr/local/sbin/kamailio[766]: NOTICE: 
> fm_status: hash =   5 fragments no.:     8, unused:     
> 0                  bucket size:        40 -        40 (first        40)
> Sep 21 11:31:02 siprouter /usr/local/sbin/kamailio[766]: NOTICE: 
> fm_status: hash =   8 fragments no.:    19, unused:     
> 0                  bucket size:        64 -        64 (first        64)
> Sep 21 11:31:02 siprouter /usr/local/sbin/kamailio[766]: NOTICE: 
> fm_status: hash =  27 fragments no.:     1, unused:     
> 0                  bucket size:       216 -       216 (first       216)
> Sep 21 11:31:02 siprouter /usr/local/sbin/kamailio[766]: NOTICE: 
> fm_status: hash =  37 fragments no.:     1, unused:     
> 0                  bucket size:       296 -       296 (first       296)
> Sep 21 11:31:02 siprouter /usr/local/sbin/kamailio[766]: NOTICE: 
> fm_status: hash =  38 fragments no.:    22, unused:     
> 0                  bucket size:       304 -       304 (first       304)
> Sep 21 11:31:02 siprouter /usr/local/sbin/kamailio[766]: NOTICE: 
> fm_status: hash =  39 fragments no.:     2, unused:     
> 0                  bucket size:       312 -       312 (first       312)
> Sep 21 11:31:02 siprouter /usr/local/sbin/kamailio[766]: NOTICE: 
> fm_status: hash =  40 fragments no.:     1, unused:     
> 0                  bucket size:       320 -       320 (first       320)
> Sep 21 11:31:02 siprouter /usr/local/sbin/kamailio[766]: NOTICE: 
> fm_status: hash = 583 fragments no.:    25, unused:     
> 0                  bucket size:      4664 -      4664 (first      4664)
> Sep 21 11:31:02 siprouter /usr/local/sbin/kamailio[766]: NOTICE: 
> fm_status: hash = 2059 fragments no.:     1, unused:     
> 0                 bucket size:  16777216 -  33554432 (first  30318496)
> Sep 21 11:31:02 siprouter /usr/local/sbin/kamailio[766]: NOTICE: 
> fm_status: TOTAL:     82 free fragments = 30444824 free bytes
> Sep 21 11:31:02 siprouter /usr/local/sbin/kamailio[766]: NOTICE: 
> fm_status: -----------------------------
>
> Santiago Soares
>
>
>
> On Tue, Sep 21, 2010 at 5:29 AM, Daniel-Constantin Mierla 
> <miconda at gmail.com <mailto:miconda at gmail.com>> wrote:
>
>      For a shorter version of memory status you can set
>     memory_summary=2 or 3:
>
>     http://www.kamailio.org/dokuwiki/doku.php/core-cookbook:3.0.x#mem_summary
>
>     This will show quickly if there is a leak.
>     Cheers,
>     Daniel
>
>
>     On 9/21/10 9:57 AM, marius zbihlei wrote:
>
>         On 09/21/2010 09:18 AM, Marius Zbihlei wrote:
>
>             Hello
>
>             Can you give us more information on the modules you are
>             loading/ your specific config ?
>
>             Marius
>
>
>         Also, is it shared memory or private memory? If you have a
>         dump of the memory distribution, can you spot anything out of
>         the ordinary. As a tip look for functions in the ending part
>         of the dump that repeat for multiple times. The lower offset
>         is usually alloc'ed during module initialization so any big
>         leak will happen at higher offsets (I don't know if this is
>         the rule but this is what I observed).
>
>         Have you tried increasing the memory pool?
>
>         Cheers
>         Marius
>
>             ________________________________________
>             From: sr-users-bounces at lists.sip-router.org
>             <mailto:sr-users-bounces at lists.sip-router.org>
>             [sr-users-bounces at lists.sip-router.org
>             <mailto:sr-users-bounces at lists.sip-router.org>] On Behalf
>             Of Santiago Soares [santiagosoares at gmail.com
>             <mailto:santiagosoares at gmail.com>]
>             Sent: Monday, September 20, 2010 9:21 PM
>             To: SR-Users; users at lists.kamailio.org
>             <mailto:users at lists.kamailio.org>
>             Subject: [SR-Users] Memory leak
>
>             Hello,
>
>             We use Kamailio as a stateless load balancer.
>             It runs on a VM, with 512 MB RAM, and it seems to have
>             some kind of memory leak.
>             The kamailio processes consume all available memory, until
>             Kamailio crashes and restart.
>             This occurs about once a day.
>             Since it's stateless, and the restart is very fast, it's
>             not a big deal, but I'd like to fix it.
>             The strange thing is that I updated kamailio from 1.5.3 to
>             3.0.3 and the problem still ocurrs.
>             I followed the instructions on
>             http://www.kamailio.org/dokuwiki/doku.php/troubleshooting:memory
>             in order to determine the cause, and stopped the traffic
>             on the proxy. The allocated memory were not freed, after 1
>             hour the amount of free memory is still the same.
>             Anybody has an ideia about what's happening, or how can I
>             debug it?
>
>             Thank you!
>
>             Santiago Soares
>
>
>             _______________________________________________
>             SIP Express Router (SER) and Kamailio (OpenSER) - sr-users
>             mailing list
>             sr-users at lists.sip-router.org
>             <mailto:sr-users at lists.sip-router.org>
>             http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
>
>
>
>         _______________________________________________
>         SIP Express Router (SER) and Kamailio (OpenSER) - sr-users
>         mailing list
>         sr-users at lists.sip-router.org
>         <mailto:sr-users at lists.sip-router.org>
>         http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
>
>
>     -- 
>     Daniel-Constantin Mierla
>     http://www.asipto.com
>
>
>
>     _______________________________________________
>     SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing
>     list
>     sr-users at lists.sip-router.org <mailto:sr-users at lists.sip-router.org>
>     http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
>
>

-- 
Daniel-Constantin Mierla
http://www.asipto.com

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.sip-router.org/pipermail/sr-users/attachments/20100921/3c339c7a/attachment-0001.htm>


More information about the sr-users mailing list