No memory left in send_subscribe (PUA) from update_pua while building the tm dlg_t structure
Hi, we are using the PUA_XMPP and PUA modules from the master branch. When the modules are started, everything are ok, the presence events from XMPP are sent to kamailio SIP servers (PUBLISH/SUBSCRIBE) and cached in the hash. But when there are several thousands records in the hash tabel, the update_pua function called in the hashT_clean gives a lot of "No memory left" error when the hashT_clean is waked up from the time:
ERROR: pua [send_subscribe.c]: No memory left ERROR: pua [pua.c]: while building tm dlg_t structure
The failed call is: td = (dlg_t*)pkg_malloc(size); if(td == NULL) { LM_ERR("No memory left\n"); return NULL; }
in dlg_t* pua_build_dlg_t(ua_pres_t* presentity) function in send_subscribe.c. The size is about 400 and something... It's strange.....
Is it the memory leak in the PUA module?
I also try to increase the pkg_memory from 4MB default to 16MB, but it doesn't help.
Any Idea?
Thanks in advanced
Laura
Hello,
you can see the available pkg via sercmd, sending command pkg.stats (match the entry for the pid printing the error). If there is no free memory, then might be a leak.
You can attach with gdb to the pid printing these errors and walk to pkg, you see the commands for gdb at:
http://www.asipto.com/pub/kamailio-devel-guide/#c04troubleshooting
See if you have lot of allocated chunks from same place in source code (ignore those at the beginning, mainly related to cfg parsing) and send the details here.
Cheers, Daniel
On 12/21/11 10:44 AM, laura testi wrote:
Hi, we are using the PUA_XMPP and PUA modules from the master branch. When the modules are started, everything are ok, the presence events from XMPP are sent to kamailio SIP servers (PUBLISH/SUBSCRIBE) and cached in the hash. But when there are several thousands records in the hash tabel, the update_pua function called in the hashT_clean gives a lot of "No memory left" error when the hashT_clean is waked up from the time:
ERROR: pua [send_subscribe.c]: No memory left ERROR: pua [pua.c]: while building tm dlg_t structure
The failed call is: td = (dlg_t*)pkg_malloc(size); if(td == NULL) { LM_ERR("No memory left\n"); return NULL; }
in dlg_t* pua_build_dlg_t(ua_pres_t* presentity) function in send_subscribe.c. The size is about 400 and something... It's strange.....
Is it the memory leak in the PUA module?
I also try to increase the pkg_memory from 4MB default to 16MB, but it doesn't help.
Any Idea?
Thanks in advanced
Laura
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
Hi Daniel, I try the sercmd for pkg memory but it return 500 error:
# sercmd sercmd 0.2 Copyright 2006 iptelorg GmbH This is free software with ABSOLUTELY NO WARRANTY. For details type `warranty'. sercmd> pkg.stats error: 500 - command pkg.stats not found sercmd>
Is it available only for 3.2.x and master branch? Because we are using 3.1.5. But take the PUA module from master branch for the fetch_row parameter you have patched ;-)
core.shmmem is ok.
Thanks a lot
Laura
On Wed, Dec 21, 2011 at 10:58 AM, Daniel-Constantin Mierla miconda@gmail.com wrote:
Hello,
you can see the available pkg via sercmd, sending command pkg.stats (match the entry for the pid printing the error). If there is no free memory, then might be a leak.
You can attach with gdb to the pid printing these errors and walk to pkg, you see the commands for gdb at:
http://www.asipto.com/pub/kamailio-devel-guide/#c04troubleshooting
See if you have lot of allocated chunks from same place in source code (ignore those at the beginning, mainly related to cfg parsing) and send the details here.
Cheers, Daniel
On 12/21/11 10:44 AM, laura testi wrote:
Hi, we are using the PUA_XMPP and PUA modules from the master branch. When the modules are started, everything are ok, the presence events from XMPP are sent to kamailio SIP servers (PUBLISH/SUBSCRIBE) and cached in the hash. But when there are several thousands records in the hash tabel, the update_pua function called in the hashT_clean gives a lot of "No memory left" error when the hashT_clean is waked up from the time:
ERROR: pua [send_subscribe.c]: No memory left ERROR: pua [pua.c]: while building tm dlg_t structure
The failed call is: td = (dlg_t*)pkg_malloc(size); if(td == NULL) { LM_ERR("No memory left\n"); return NULL; }
in dlg_t* pua_build_dlg_t(ua_pres_t* presentity) function in send_subscribe.c. The size is about 400 and something... It's strange.....
Is it the memory leak in the PUA module?
I also try to increase the pkg_memory from 4MB default to 16MB, but it doesn't help.
Any Idea?
Thanks in advanced
Laura
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
-- Daniel-Constantin Mierla -- http://www.asipto.com http://linkedin.com/in/miconda -- http://twitter.com/miconda
Hello,
pkg.stats was added in 3.2.0, iirc. For 3.1, you can walk the packets in memory with gdb -- 3.1 has memory debug on, so you don't need to recompile (unless you turned it off).
Just attach to the pid of a sip worker (gdb /path/to/kamailio _pid_value_) and run the gdb script.
Cheers, Daniel
On 12/21/11 11:58 AM, laura testi wrote:
Hi Daniel, I try the sercmd for pkg memory but it return 500 error:
# sercmd sercmd 0.2 Copyright 2006 iptelorg GmbH This is free software with ABSOLUTELY NO WARRANTY. For details type `warranty'. sercmd> pkg.stats error: 500 - command pkg.stats not found sercmd>
Is it available only for 3.2.x and master branch? Because we are using 3.1.5. But take the PUA module from master branch for the fetch_row parameter you have patched ;-)
core.shmmem is ok.
Thanks a lot
Laura
On Wed, Dec 21, 2011 at 10:58 AM, Daniel-Constantin Mierla miconda@gmail.com wrote:
Hello,
you can see the available pkg via sercmd, sending command pkg.stats (match the entry for the pid printing the error). If there is no free memory, then might be a leak.
You can attach with gdb to the pid printing these errors and walk to pkg, you see the commands for gdb at:
http://www.asipto.com/pub/kamailio-devel-guide/#c04troubleshooting
See if you have lot of allocated chunks from same place in source code (ignore those at the beginning, mainly related to cfg parsing) and send the details here.
Cheers, Daniel
On 12/21/11 10:44 AM, laura testi wrote:
Hi, we are using the PUA_XMPP and PUA modules from the master branch. When the modules are started, everything are ok, the presence events from XMPP are sent to kamailio SIP servers (PUBLISH/SUBSCRIBE) and cached in the hash. But when there are several thousands records in the hash tabel, the update_pua function called in the hashT_clean gives a lot of "No memory left" error when the hashT_clean is waked up from the time:
ERROR: pua [send_subscribe.c]: No memory left ERROR: pua [pua.c]: while building tm dlg_t structure
The failed call is: td = (dlg_t*)pkg_malloc(size); if(td == NULL) { LM_ERR("No memory left\n"); return NULL; }
in dlg_t* pua_build_dlg_t(ua_pres_t* presentity) function in send_subscribe.c. The size is about 400 and something... It's strange.....
Is it the memory leak in the PUA module?
I also try to increase the pkg_memory from 4MB default to 16MB, but it doesn't help.
Any Idea?
Thanks in advanced
Laura
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
-- Daniel-Constantin Mierla -- http://www.asipto.com http://linkedin.com/in/miconda -- http://twitter.com/miconda
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
Hi Daniel, I just follow the instruction in the link you sent (http://www.asipto.com/pub/kamailio-devel-guide/#c04troubleshooting) to use gdb to print PKG fragments. When I got the error in PUA:
Dec 23 11:10:53 /.../sbin/kamailio[23276]: ERROR: pua [send_subscribe.c:158]: No memory left for size:439 Dec 23 11:10:53 /.../sbin/kamailio[23276]: ERROR: pua [pua.c:747]: while building tm dlg_t structure Dec 23 11:10:53 /.../sbin/kamailio[23276]: ERROR: pua [pua.c:652]: while updating record
The I run the command gdb /.../sbin/kamailio 23276 and write the following commands in the gdb:
set $i=0 set $a = mem_block->first_frag while($i<10000) if($i>2000) if($a->u.is_free==0) p *$a end end set $a = ((struct qm_frag*)((char*)($a)+sizeof(struct qm_frag)+((struct qm_frag*)$a)->size+sizeof(struct qm_frag_end))) set $i = $i + 1 end ...
after a while I got a lot of prints on the screen like these:
func = 0x5d7b00 "do_parse_rr_body", line = 74, check = 4042322160} $1348 = {size = 72, u = {nxt_free = 0x0, is_free = 0}, file = 0x5d76c9 "<core>: parser/parse_rr.c", func = 0x5d7b00 "do_parse_rr_body", line = 74, check = 4042322160} $1349 = {size = 72, u = {nxt_free = 0x0, is_free = 0}, file = 0x5d76c9 "<core>: parser/parse_rr.c", func = 0x5d7b00 "do_parse_rr_body", line = 74, check = 4042322160} $1350 = {size = 72, u = {nxt_free = 0x0, is_free = 0}, file = 0x5d76c9 "<core>: parser/parse_rr.c", func = 0x5d7b00 "do_parse_rr_body", line = 74, check = 4042322160} $1351 = {size = 104, u = {nxt_free = 0x0, is_free = 0}, file = 0x5d76c9 "<core>: parser/parse_rr.c", func = 0x5d7b00 "do_parse_rr_body", line = 74, check = 4042322160} $1352 = {size = 104, u = {nxt_free = 0x0, is_free = 0}, file = 0x5d76c9 "<core>: parser/parse_rr.c", func = 0x5d7b00 "do_parse_rr_body", line = 74, check = 4042322160} $1353 = {size = 72, u = {nxt_free = 0x0, is_free = 0}, file = 0x5d76c9 "<core>: parser/parse_rr.c", func = 0x5d7b00 "do_parse_rr_body", line = 74, check = 4042322160} $1354 = {size = 72, u = {nxt_free = 0x0, is_free = 0}, ---Type <return> to continue, or q <return> to quit---
...
But I don't understand if these are normal or something goes wrong....
Can you help
Best Regards, Laura
On Wed, Dec 21, 2011 at 12:18 PM, Daniel-Constantin Mierla miconda@gmail.com wrote:
Hello,
pkg.stats was added in 3.2.0, iirc. For 3.1, you can walk the packets in memory with gdb -- 3.1 has memory debug on, so you don't need to recompile (unless you turned it off).
Just attach to the pid of a sip worker (gdb /path/to/kamailio _pid_value_) and run the gdb script.
Cheers, Daniel
On 12/21/11 11:58 AM, laura testi wrote:
Hi Daniel, I try the sercmd for pkg memory but it return 500 error:
# sercmd sercmd 0.2 Copyright 2006 iptelorg GmbH This is free software with ABSOLUTELY NO WARRANTY. For details type `warranty'. sercmd> pkg.stats error: 500 - command pkg.stats not found sercmd>
Is it available only for 3.2.x and master branch? Because we are using 3.1.5. But take the PUA module from master branch for the fetch_row parameter you have patched ;-)
core.shmmem is ok.
Thanks a lot
Laura
On Wed, Dec 21, 2011 at 10:58 AM, Daniel-Constantin Mierla miconda@gmail.com wrote:
Hello,
you can see the available pkg via sercmd, sending command pkg.stats (match the entry for the pid printing the error). If there is no free memory, then might be a leak.
You can attach with gdb to the pid printing these errors and walk to pkg, you see the commands for gdb at:
http://www.asipto.com/pub/kamailio-devel-guide/#c04troubleshooting
See if you have lot of allocated chunks from same place in source code (ignore those at the beginning, mainly related to cfg parsing) and send the details here.
Cheers, Daniel
On 12/21/11 10:44 AM, laura testi wrote:
Hi, we are using the PUA_XMPP and PUA modules from the master branch. When the modules are started, everything are ok, the presence events from XMPP are sent to kamailio SIP servers (PUBLISH/SUBSCRIBE) and cached in the hash. But when there are several thousands records in the hash tabel, the update_pua function called in the hashT_clean gives a lot of "No memory left" error when the hashT_clean is waked up from the time:
ERROR: pua [send_subscribe.c]: No memory left ERROR: pua [pua.c]: while building tm dlg_t structure
The failed call is: td = (dlg_t*)pkg_malloc(size); if(td == NULL) { LM_ERR("No memory left\n"); return NULL; }
in dlg_t* pua_build_dlg_t(ua_pres_t* presentity) function in send_subscribe.c. The size is about 400 and something... It's strange.....
Is it the memory leak in the PUA module?
I also try to increase the pkg_memory from 4MB default to 16MB, but it doesn't help.
Any Idea?
Thanks in advanced
Laura
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
-- Daniel-Constantin Mierla -- http://www.asipto.com http://linkedin.com/in/miconda -- http://twitter.com/miconda
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
-- Daniel-Constantin Mierla -- http://www.asipto.com http://linkedin.com/in/miconda -- http://twitter.com/miconda
Hello,
looks like a leak in a module that is storing record-routes and use them later, perhaps pua module, I will check it soon.
Cheers, Daniel
On 12/23/11 11:31 AM, laura testi wrote:
Hi Daniel, I just follow the instruction in the link you sent (http://www.asipto.com/pub/kamailio-devel-guide/#c04troubleshooting) to use gdb to print PKG fragments. When I got the error in PUA:
Dec 23 11:10:53 /.../sbin/kamailio[23276]: ERROR: pua [send_subscribe.c:158]: No memory left for size:439 Dec 23 11:10:53 /.../sbin/kamailio[23276]: ERROR: pua [pua.c:747]: while building tm dlg_t structure Dec 23 11:10:53 /.../sbin/kamailio[23276]: ERROR: pua [pua.c:652]: while updating record
The I run the command gdb /.../sbin/kamailio 23276 and write the following commands in the gdb:
set $i=0 set $a = mem_block->first_frag while($i<10000) if($i>2000) if($a->u.is_free==0) p *$a end end set $a = ((struct qm_frag*)((char*)($a)+sizeof(struct qm_frag)+((struct qm_frag*)$a)->size+sizeof(struct qm_frag_end))) set $i = $i + 1 end ...
after a while I got a lot of prints on the screen like these:
func = 0x5d7b00 "do_parse_rr_body", line = 74, check = 4042322160} $1348 = {size = 72, u = {nxt_free = 0x0, is_free = 0}, file = 0x5d76c9 "<core>: parser/parse_rr.c", func = 0x5d7b00 "do_parse_rr_body", line = 74, check = 4042322160} $1349 = {size = 72, u = {nxt_free = 0x0, is_free = 0}, file = 0x5d76c9 "<core>: parser/parse_rr.c", func = 0x5d7b00 "do_parse_rr_body", line = 74, check = 4042322160} $1350 = {size = 72, u = {nxt_free = 0x0, is_free = 0}, file = 0x5d76c9 "<core>: parser/parse_rr.c", func = 0x5d7b00 "do_parse_rr_body", line = 74, check = 4042322160} $1351 = {size = 104, u = {nxt_free = 0x0, is_free = 0}, file = 0x5d76c9 "<core>: parser/parse_rr.c", func = 0x5d7b00 "do_parse_rr_body", line = 74, check = 4042322160} $1352 = {size = 104, u = {nxt_free = 0x0, is_free = 0}, file = 0x5d76c9 "<core>: parser/parse_rr.c", func = 0x5d7b00 "do_parse_rr_body", line = 74, check = 4042322160} $1353 = {size = 72, u = {nxt_free = 0x0, is_free = 0}, file = 0x5d76c9 "<core>: parser/parse_rr.c", func = 0x5d7b00 "do_parse_rr_body", line = 74, check = 4042322160} $1354 = {size = 72, u = {nxt_free = 0x0, is_free = 0}, ---Type<return> to continue, or q<return> to quit---
...
But I don't understand if these are normal or something goes wrong....
Can you help
Best Regards, Laura
On Wed, Dec 21, 2011 at 12:18 PM, Daniel-Constantin Mierla miconda@gmail.com wrote:
Hello,
pkg.stats was added in 3.2.0, iirc. For 3.1, you can walk the packets in memory with gdb -- 3.1 has memory debug on, so you don't need to recompile (unless you turned it off).
Just attach to the pid of a sip worker (gdb /path/to/kamailio _pid_value_) and run the gdb script.
Cheers, Daniel
On 12/21/11 11:58 AM, laura testi wrote:
Hi Daniel, I try the sercmd for pkg memory but it return 500 error:
# sercmd sercmd 0.2 Copyright 2006 iptelorg GmbH This is free software with ABSOLUTELY NO WARRANTY. For details type `warranty'. sercmd> pkg.stats error: 500 - command pkg.stats not found sercmd>
Is it available only for 3.2.x and master branch? Because we are using 3.1.5. But take the PUA module from master branch for the fetch_row parameter you have patched ;-)
core.shmmem is ok.
Thanks a lot
Laura
On Wed, Dec 21, 2011 at 10:58 AM, Daniel-Constantin Mierla miconda@gmail.com wrote:
Hello,
you can see the available pkg via sercmd, sending command pkg.stats (match the entry for the pid printing the error). If there is no free memory, then might be a leak.
You can attach with gdb to the pid printing these errors and walk to pkg, you see the commands for gdb at:
http://www.asipto.com/pub/kamailio-devel-guide/#c04troubleshooting
See if you have lot of allocated chunks from same place in source code (ignore those at the beginning, mainly related to cfg parsing) and send the details here.
Cheers, Daniel
On 12/21/11 10:44 AM, laura testi wrote:
Hi, we are using the PUA_XMPP and PUA modules from the master branch. When the modules are started, everything are ok, the presence events from XMPP are sent to kamailio SIP servers (PUBLISH/SUBSCRIBE) and cached in the hash. But when there are several thousands records in the hash tabel, the update_pua function called in the hashT_clean gives a lot of "No memory left" error when the hashT_clean is waked up from the time:
ERROR: pua [send_subscribe.c]: No memory left ERROR: pua [pua.c]: while building tm dlg_t structure
The failed call is: td = (dlg_t*)pkg_malloc(size); if(td == NULL) { LM_ERR("No memory left\n"); return NULL; }
in dlg_t* pua_build_dlg_t(ua_pres_t* presentity) function in send_subscribe.c. The size is about 400 and something... It's strange.....
Is it the memory leak in the PUA module?
I also try to increase the pkg_memory from 4MB default to 16MB, but it doesn't help.
Any Idea?
Thanks in advanced
Laura
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
-- Daniel-Constantin Mierla -- http://www.asipto.com http://linkedin.com/in/miconda -- http://twitter.com/miconda
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
-- Daniel-Constantin Mierla -- http://www.asipto.com http://linkedin.com/in/miconda -- http://twitter.com/miconda
Hello,
can you try with this patch:
http://git.sip-router.org/cgi-bin/gitweb.cgi/sip-router/?a=commit;h=1b3cfa60...
Cheers, Daniel
On 12/23/11 12:36 PM, Daniel-Constantin Mierla wrote:
Hello,
looks like a leak in a module that is storing record-routes and use them later, perhaps pua module, I will check it soon.
Cheers, Daniel
On 12/23/11 11:31 AM, laura testi wrote:
Hi Daniel, I just follow the instruction in the link you sent (http://www.asipto.com/pub/kamailio-devel-guide/#c04troubleshooting) to use gdb to print PKG fragments. When I got the error in PUA:
Dec 23 11:10:53 /.../sbin/kamailio[23276]: ERROR: pua [send_subscribe.c:158]: No memory left for size:439 Dec 23 11:10:53 /.../sbin/kamailio[23276]: ERROR: pua [pua.c:747]: while building tm dlg_t structure Dec 23 11:10:53 /.../sbin/kamailio[23276]: ERROR: pua [pua.c:652]: while updating record
The I run the command gdb /.../sbin/kamailio 23276 and write the following commands in the gdb:
set $i=0 set $a = mem_block->first_frag while($i<10000) if($i>2000) if($a->u.is_free==0) p *$a end end set $a = ((struct qm_frag*)((char*)($a)+sizeof(struct qm_frag)+((struct qm_frag*)$a)->size+sizeof(struct qm_frag_end))) set $i = $i + 1 end ...
after a while I got a lot of prints on the screen like these:
func = 0x5d7b00 "do_parse_rr_body", line = 74, check = 4042322160} $1348 = {size = 72, u = {nxt_free = 0x0, is_free = 0}, file = 0x5d76c9 "<core>: parser/parse_rr.c", func = 0x5d7b00 "do_parse_rr_body", line = 74, check = 4042322160} $1349 = {size = 72, u = {nxt_free = 0x0, is_free = 0}, file = 0x5d76c9 "<core>: parser/parse_rr.c", func = 0x5d7b00 "do_parse_rr_body", line = 74, check = 4042322160} $1350 = {size = 72, u = {nxt_free = 0x0, is_free = 0}, file = 0x5d76c9 "<core>: parser/parse_rr.c", func = 0x5d7b00 "do_parse_rr_body", line = 74, check = 4042322160} $1351 = {size = 104, u = {nxt_free = 0x0, is_free = 0}, file = 0x5d76c9 "<core>: parser/parse_rr.c", func = 0x5d7b00 "do_parse_rr_body", line = 74, check = 4042322160} $1352 = {size = 104, u = {nxt_free = 0x0, is_free = 0}, file = 0x5d76c9 "<core>: parser/parse_rr.c", func = 0x5d7b00 "do_parse_rr_body", line = 74, check = 4042322160} $1353 = {size = 72, u = {nxt_free = 0x0, is_free = 0}, file = 0x5d76c9 "<core>: parser/parse_rr.c", func = 0x5d7b00 "do_parse_rr_body", line = 74, check = 4042322160} $1354 = {size = 72, u = {nxt_free = 0x0, is_free = 0}, ---Type<return> to continue, or q<return> to quit---
...
But I don't understand if these are normal or something goes wrong....
Can you help
Best Regards, Laura
On Wed, Dec 21, 2011 at 12:18 PM, Daniel-Constantin Mierla miconda@gmail.com wrote:
Hello,
pkg.stats was added in 3.2.0, iirc. For 3.1, you can walk the packets in memory with gdb -- 3.1 has memory debug on, so you don't need to recompile (unless you turned it off).
Just attach to the pid of a sip worker (gdb /path/to/kamailio _pid_value_) and run the gdb script.
Cheers, Daniel
On 12/21/11 11:58 AM, laura testi wrote:
Hi Daniel, I try the sercmd for pkg memory but it return 500 error:
# sercmd sercmd 0.2 Copyright 2006 iptelorg GmbH This is free software with ABSOLUTELY NO WARRANTY. For details type `warranty'. sercmd> pkg.stats error: 500 - command pkg.stats not found sercmd>
Is it available only for 3.2.x and master branch? Because we are using 3.1.5. But take the PUA module from master branch for the fetch_row parameter you have patched ;-)
core.shmmem is ok.
Thanks a lot
Laura
On Wed, Dec 21, 2011 at 10:58 AM, Daniel-Constantin Mierla miconda@gmail.com wrote:
Hello,
you can see the available pkg via sercmd, sending command pkg.stats (match the entry for the pid printing the error). If there is no free memory, then might be a leak.
You can attach with gdb to the pid printing these errors and walk to pkg, you see the commands for gdb at:
http://www.asipto.com/pub/kamailio-devel-guide/#c04troubleshooting
See if you have lot of allocated chunks from same place in source code (ignore those at the beginning, mainly related to cfg parsing) and send the details here.
Cheers, Daniel
On 12/21/11 10:44 AM, laura testi wrote:
Hi, we are using the PUA_XMPP and PUA modules from the master branch. When the modules are started, everything are ok, the presence events from XMPP are sent to kamailio SIP servers (PUBLISH/SUBSCRIBE) and cached in the hash. But when there are several thousands records in the hash tabel, the update_pua function called in the hashT_clean gives a lot of "No memory left" error when the hashT_clean is waked up from the time:
ERROR: pua [send_subscribe.c]: No memory left ERROR: pua [pua.c]: while building tm dlg_t structure
The failed call is: td = (dlg_t*)pkg_malloc(size); if(td == NULL) { LM_ERR("No memory left\n"); return NULL; }
in dlg_t* pua_build_dlg_t(ua_pres_t* presentity) function in send_subscribe.c. The size is about 400 and something... It's strange.....
Is it the memory leak in the PUA module?
I also try to increase the pkg_memory from 4MB default to 16MB, but it doesn't help.
Any Idea?
Thanks in advanced
Laura
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
-- Daniel-Constantin Mierla -- http://www.asipto.com http://linkedin.com/in/miconda -- http://twitter.com/miconda
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
-- Daniel-Constantin Mierla -- http://www.asipto.com http://linkedin.com/in/miconda -- http://twitter.com/miconda
Ok, I'll try it.
Thank you very much!
On Fri, Dec 23, 2011 at 1:20 PM, Daniel-Constantin Mierla miconda@gmail.com wrote:
Hello,
can you try with this patch:
http://git.sip-router.org/cgi-bin/gitweb.cgi/sip-router/?a=commit;h=1b3cfa60...
Cheers, Daniel
On 12/23/11 12:36 PM, Daniel-Constantin Mierla wrote:
Hello,
looks like a leak in a module that is storing record-routes and use them later, perhaps pua module, I will check it soon.
Cheers, Daniel
On 12/23/11 11:31 AM, laura testi wrote:
Hi Daniel, I just follow the instruction in the link you sent (http://www.asipto.com/pub/kamailio-devel-guide/#c04troubleshooting) to use gdb to print PKG fragments. When I got the error in PUA:
Dec 23 11:10:53 /.../sbin/kamailio[23276]: ERROR: pua [send_subscribe.c:158]: No memory left for size:439 Dec 23 11:10:53 /.../sbin/kamailio[23276]: ERROR: pua [pua.c:747]: while building tm dlg_t structure Dec 23 11:10:53 /.../sbin/kamailio[23276]: ERROR: pua [pua.c:652]: while updating record
The I run the command gdb /.../sbin/kamailio 23276 and write the following commands in the gdb:
set $i=0 set $a = mem_block->first_frag while($i<10000) if($i>2000) if($a->u.is_free==0) p *$a end end set $a = ((struct qm_frag*)((char*)($a)+sizeof(struct qm_frag)+((struct qm_frag*)$a)->size+sizeof(struct qm_frag_end))) set $i = $i + 1 end ...
after a while I got a lot of prints on the screen like these:
func = 0x5d7b00 "do_parse_rr_body", line = 74, check = 4042322160} $1348 = {size = 72, u = {nxt_free = 0x0, is_free = 0}, file = 0x5d76c9 "<core>: parser/parse_rr.c", func = 0x5d7b00 "do_parse_rr_body", line = 74, check = 4042322160} $1349 = {size = 72, u = {nxt_free = 0x0, is_free = 0}, file = 0x5d76c9 "<core>: parser/parse_rr.c", func = 0x5d7b00 "do_parse_rr_body", line = 74, check = 4042322160} $1350 = {size = 72, u = {nxt_free = 0x0, is_free = 0}, file = 0x5d76c9 "<core>: parser/parse_rr.c", func = 0x5d7b00 "do_parse_rr_body", line = 74, check = 4042322160} $1351 = {size = 104, u = {nxt_free = 0x0, is_free = 0}, file = 0x5d76c9 "<core>: parser/parse_rr.c", func = 0x5d7b00 "do_parse_rr_body", line = 74, check = 4042322160} $1352 = {size = 104, u = {nxt_free = 0x0, is_free = 0}, file = 0x5d76c9 "<core>: parser/parse_rr.c", func = 0x5d7b00 "do_parse_rr_body", line = 74, check = 4042322160} $1353 = {size = 72, u = {nxt_free = 0x0, is_free = 0}, file = 0x5d76c9 "<core>: parser/parse_rr.c", func = 0x5d7b00 "do_parse_rr_body", line = 74, check = 4042322160} $1354 = {size = 72, u = {nxt_free = 0x0, is_free = 0}, ---Type<return> to continue, or q<return> to quit---
...
But I don't understand if these are normal or something goes wrong....
Can you help
Best Regards, Laura
On Wed, Dec 21, 2011 at 12:18 PM, Daniel-Constantin Mierla miconda@gmail.com wrote:
Hello,
pkg.stats was added in 3.2.0, iirc. For 3.1, you can walk the packets in memory with gdb -- 3.1 has memory debug on, so you don't need to recompile (unless you turned it off).
Just attach to the pid of a sip worker (gdb /path/to/kamailio _pid_value_) and run the gdb script.
Cheers, Daniel
On 12/21/11 11:58 AM, laura testi wrote:
Hi Daniel, I try the sercmd for pkg memory but it return 500 error:
# sercmd sercmd 0.2 Copyright 2006 iptelorg GmbH This is free software with ABSOLUTELY NO WARRANTY. For details type `warranty'. sercmd> pkg.stats error: 500 - command pkg.stats not found sercmd>
Is it available only for 3.2.x and master branch? Because we are using 3.1.5. But take the PUA module from master branch for the fetch_row parameter you have patched ;-)
core.shmmem is ok.
Thanks a lot
Laura
On Wed, Dec 21, 2011 at 10:58 AM, Daniel-Constantin Mierla miconda@gmail.com wrote:
Hello,
you can see the available pkg via sercmd, sending command pkg.stats (match the entry for the pid printing the error). If there is no free memory, then might be a leak.
You can attach with gdb to the pid printing these errors and walk to pkg, you see the commands for gdb at:
http://www.asipto.com/pub/kamailio-devel-guide/#c04troubleshooting
See if you have lot of allocated chunks from same place in source code (ignore those at the beginning, mainly related to cfg parsing) and send the details here.
Cheers, Daniel
On 12/21/11 10:44 AM, laura testi wrote:
Hi, we are using the PUA_XMPP and PUA modules from the master branch. When the modules are started, everything are ok, the presence events from XMPP are sent to kamailio SIP servers (PUBLISH/SUBSCRIBE) and cached in the hash. But when there are several thousands records in the hash tabel, the update_pua function called in the hashT_clean gives a lot of "No memory left" error when the hashT_clean is waked up from the time:
ERROR: pua [send_subscribe.c]: No memory left ERROR: pua [pua.c]: while building tm dlg_t structure
The failed call is: td = (dlg_t*)pkg_malloc(size); if(td == NULL) { LM_ERR("No memory left\n"); return NULL; }
in dlg_t* pua_build_dlg_t(ua_pres_t* presentity) function in send_subscribe.c. The size is about 400 and something... It's strange.....
Is it the memory leak in the PUA module?
I also try to increase the pkg_memory from 4MB default to 16MB, but it doesn't help.
Any Idea?
Thanks in advanced
Laura
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
-- Daniel-Constantin Mierla -- http://www.asipto.com http://linkedin.com/in/miconda -- http://twitter.com/miconda
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
-- Daniel-Constantin Mierla -- http://www.asipto.com http://linkedin.com/in/miconda -- http://twitter.com/miconda
-- Daniel-Constantin Mierla -- http://www.asipto.com http://linkedin.com/in/miconda -- http://twitter.com/miconda
-
Daniel-Constantin Mierla -
laura testi