[sr-dev] [SR-Users] No memory left in send_subscribe (PUA) from update_pua while building the tm dlg_t structure

Fri Dec 23 11:31:19 CET 2011

Hi Daniel,
I just follow the instruction in the link you sent
(http://www.asipto.com/pub/kamailio-devel-guide/#c04troubleshooting)
to use gdb to print PKG fragments. When I got the error in PUA:

Dec 23 11:10:53 /.../sbin/kamailio[23276]: ERROR: pua
[send_subscribe.c:158]: No memory left for size:439
Dec 23 11:10:53 /.../sbin/kamailio[23276]: ERROR: pua [pua.c:747]:
while building tm dlg_t structure
Dec 23 11:10:53 /.../sbin/kamailio[23276]: ERROR: pua [pua.c:652]:
while updating record

The I run the command
gdb /.../sbin/kamailio 23276
and write the following commands in the gdb:

set $i=0
set $a = mem_block->first_frag
while($i<10000)
  if($i>2000)
    if($a->u.is_free==0)
      p *$a
    end
  end
  set $a = ((struct qm_frag*)((char*)($a)+sizeof(struct
qm_frag)+((struct qm_frag*)$a)->size+sizeof(struct qm_frag_end)))
  set $i = $i + 1
end
...

after a while I got a lot of prints on the screen like these:

  func = 0x5d7b00 "do_parse_rr_body", line = 74, check = 4042322160}
$1348 = {size = 72, u = {nxt_free = 0x0, is_free = 0},
  file = 0x5d76c9 "<core>: parser/parse_rr.c",
  func = 0x5d7b00 "do_parse_rr_body", line = 74, check = 4042322160}
$1349 = {size = 72, u = {nxt_free = 0x0, is_free = 0},
  file = 0x5d76c9 "<core>: parser/parse_rr.c",
  func = 0x5d7b00 "do_parse_rr_body", line = 74, check = 4042322160}
$1350 = {size = 72, u = {nxt_free = 0x0, is_free = 0},
  file = 0x5d76c9 "<core>: parser/parse_rr.c",
  func = 0x5d7b00 "do_parse_rr_body", line = 74, check = 4042322160}
$1351 = {size = 104, u = {nxt_free = 0x0, is_free = 0},
  file = 0x5d76c9 "<core>: parser/parse_rr.c",
  func = 0x5d7b00 "do_parse_rr_body", line = 74, check = 4042322160}
$1352 = {size = 104, u = {nxt_free = 0x0, is_free = 0},
  file = 0x5d76c9 "<core>: parser/parse_rr.c",
  func = 0x5d7b00 "do_parse_rr_body", line = 74, check = 4042322160}
$1353 = {size = 72, u = {nxt_free = 0x0, is_free = 0},
  file = 0x5d76c9 "<core>: parser/parse_rr.c",
  func = 0x5d7b00 "do_parse_rr_body", line = 74, check = 4042322160}
$1354 = {size = 72, u = {nxt_free = 0x0, is_free = 0},
---Type <return> to continue, or q <return> to quit---

...

But I don't understand if these are normal or something goes wrong....

Can you help

Best Regards,
Laura

On Wed, Dec 21, 2011 at 12:18 PM, Daniel-Constantin Mierla
<miconda at gmail.com> wrote:
> Hello,
>
> pkg.stats was added in 3.2.0, iirc. For 3.1, you can walk the packets in
> memory with gdb -- 3.1 has memory debug on, so you don't need to recompile
> (unless you turned it off).
>
> Just attach to the pid of a sip worker  (gdb /path/to/kamailio _pid_value_)
> and run the gdb script.
>
> Cheers,
> Daniel
>
>
> On 12/21/11 11:58 AM, laura testi wrote:
>>
>> Hi Daniel,
>> I try the sercmd for pkg memory but it return 500 error:
>>
>>
>> # sercmd
>> sercmd 0.2
>> Copyright 2006 iptelorg GmbH
>> This is free software with ABSOLUTELY NO WARRANTY.
>> For details type `warranty'.
>> sercmd>  pkg.stats
>> error: 500 - command pkg.stats not found
>> sercmd>
>>
>>
>>
>> Is it available only for 3.2.x and master branch? Because we are using
>> 3.1.5. But take the PUA module from master branch for the fetch_row
>> parameter you have patched ;-)
>>
>>
>> core.shmmem is ok.
>>
>>
>>
>> Thanks a lot
>>
>> Laura
>>
>> On Wed, Dec 21, 2011 at 10:58 AM, Daniel-Constantin Mierla
>> <miconda at gmail.com>  wrote:
>>>
>>> Hello,
>>>
>>> you can see the available pkg via sercmd, sending command pkg.stats
>>> (match
>>> the entry for the pid printing the error). If there is no free memory,
>>> then
>>> might be a leak.
>>>
>>> You can attach with gdb to the pid printing these errors and walk to pkg,
>>> you see the commands for gdb at:
>>>
>>> http://www.asipto.com/pub/kamailio-devel-guide/#c04troubleshooting
>>>
>>> See if you have lot of allocated chunks from same place in source code
>>> (ignore those at the beginning, mainly related to cfg parsing) and send
>>> the
>>> details here.
>>>
>>> Cheers,
>>> Daniel
>>>
>>>
>>> On 12/21/11 10:44 AM, laura testi wrote:
>>>
>>> Hi,
>>> we are using the PUA_XMPP and PUA modules from the master branch. When
>>> the modules are started, everything are ok, the presence events from
>>> XMPP are sent to kamailio SIP servers (PUBLISH/SUBSCRIBE) and cached
>>> in the hash. But when there are several thousands records in the hash
>>> tabel, the update_pua function called in the hashT_clean gives a lot
>>> of "No memory left" error when the hashT_clean is waked up from the
>>> time:
>>>
>>> ERROR: pua [send_subscribe.c]: No memory left
>>> ERROR: pua [pua.c]: while building tm dlg_t structure
>>>
>>> The failed call is:
>>>        td = (dlg_t*)pkg_malloc(size);
>>>        if(td == NULL)
>>>        {
>>>                LM_ERR("No memory left\n");
>>>                return NULL;
>>>        }
>>>
>>> in dlg_t* pua_build_dlg_t(ua_pres_t* presentity) function in
>>> send_subscribe.c. The size is about 400 and something... It's
>>> strange.....
>>>
>>> Is it the memory leak in the PUA module?
>>>
>>>
>>>
>>> I also try to increase the pkg_memory from 4MB default to 16MB, but it
>>> doesn't help.
>>>
>>>
>>> Any Idea?
>>>
>>> Thanks in advanced
>>>
>>> Laura
>>>
>>> _______________________________________________
>>> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
>>> sr-users at lists.sip-router.org
>>> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
>>>
>>>
>>> --
>>> Daniel-Constantin Mierla -- http://www.asipto.com
>>> http://linkedin.com/in/miconda -- http://twitter.com/miconda
>>
>> _______________________________________________
>> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
>> sr-users at lists.sip-router.org
>> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
>
>
> --
> Daniel-Constantin Mierla -- http://www.asipto.com
> http://linkedin.com/in/miconda -- http://twitter.com/miconda
>