24 Jan
2012
24 Jan
'12
12:21 p.m.
Hi,
we are using the XMPP gateway modules:
• PUA
• PUA_XMPP
• XMPP
The behavior is ok but since the modules are started, the memory usage
is getting increased and now also the swap memory area is used.
I send to you the following information:
1. Numer of records from PUA and xmpp_presentity tables, they are quite stable:
SQL> select count(*) from pua;
COUNT(*)
----------
10218
SQL> select count(*) from xmpp_presentity;
COUNT(*)
----------
754
2. list of PKG fragments with gdb :
$1 = {size = 40, u = {nxt_free = 0x0, is_free = 0}, file =
0x2b657aa12c00 "presence: utils_func.h", func = 0x2b657aa1aed0
"uandd_to_uri", line = 66,
check = 4042322160}
$2 = {size = 32, u = {nxt_free = 0x0, is_free = 0}, file =
0x2b657aa12c00 "presence: utils_func.h", func = 0x2b657aa1aed0
"uandd_to_uri", line = 66,
check = 4042322160}
$3 = {size = 40, u = {nxt_free = 0x0, is_free = 0}, file =
0x2b657aa12c00 "presence: utils_func.h", func = 0x2b657aa1aed0
"uandd_to_uri", line = 66,
check = 4042322160}
$4 = {size = 40, u = {nxt_free = 0x0, is_free = 0}, file =
0x2b657aa12c00 "presence: utils_func.h", func = 0x2b657aa1aed0
"uandd_to_uri", line = 66,
check = 4042322160}
$5 = {size = 40, u = {nxt_free = 0x0, is_free = 0}, file =
0x2b657aa12c00 "presence: utils_func.h", func = 0x2b657aa1aed0
"uandd_to_uri", line = 66,
check = 4042322160}
$6 = {size = 104, u = {nxt_free = 0x0, is_free = 0}, file =
0x2b657aa12c00 "presence: utils_func.h", func = 0x2b657aa1aed0
"uandd_to_uri", line = 66,
check = 4042322160}
$7 = {size = 32, u = {nxt_free = 0x0, is_free = 0}, file =
0x2b657aa12c00 "presence: utils_func.h", func = 0x2b657aa1aed0
"uandd_to_uri", line = 66,
check = 4042322160}
$8 = {size = 32, u = {nxt_free = 0x0, is_free = 0}, file =
0x2b657aa12c00 "presence: utils_func.h", func = 0x2b657aa1aed0
"uandd_to_uri", line = 66,
check = 4042322160}
$9 = {size = 56, u = {nxt_free = 0x0, is_free = 0}, file =
0x2b657aa12c00 "presence: utils_func.h", func = 0x2b657aa1aed0
"uandd_to_uri", line = 66,
check = 4042322160}
$10 = {size = 96, u = {nxt_free = 0x0, is_free = 0}, file =
0x2b657aa12c00 "presence: utils_func.h", func = 0x2b657aa1aed0
"uandd_to_uri", line = 66,
check = 4042322160}
$11 = {size = 56, u = {nxt_free = 0x0, is_free = 0}, file =
0x2b657aa12c00 "presence: utils_func.h", func = 0x2b657aa1aed0
"uandd_to_uri", line = 66,
check = 4042322160}
$12 = {size = 112, u = {nxt_free = 0x0, is_free = 0}, file =
0x2b657aa12c00 "presence: utils_func.h", func = 0x2b657aa1aed0
"uandd_to_uri", line = 66,
check = 4042322160}
$13 = {size = 24, u = {nxt_free = 0x0, is_free = 0}, file =
0x2b657aa12c00 "presence: utils_func.h", func = 0x2b657aa1aed0
"uandd_to_uri", line = 66,
check = 4042322160}
$14 = {size = 24, u = {nxt_free = 0x0, is_free = 0}, file =
0x2b657aa12c00 "presence: utils_func.h", func = 0x2b657aa1aed0
"uandd_to_uri", line = 66,
check = 4042322160}
$15 = {size = 40, u = {nxt_free = 0x0, is_free = 0}, file =
0x2b657aa12c00 "presence: utils_func.h", func = 0x2b657aa1aed0
"uandd_to_uri", line = 66,
check = 4042322160}
$16 = {size = 40, u = {nxt_free = 0x0, is_free = 0}, file =
0x2b657aa12c00 "presence: utils_func.h", func = 0x2b657aa1aed0
"uandd_to_uri", line = 66,
check = 4042322160}
$17 = {size = 40, u = {nxt_free = 0x0, is_free = 0}, file =
0x2b657aa12c00 "presence: utils_func.h", func = 0x2b657aa1aed0
"uandd_to_uri", line = 66,
check = 4042322160}………….
3. Output of sercmd cfg.set_now_int core mem_dump_pkg :
……………….. 10969. N address=0xb1b320 frag=0xb1b2f0 size=104 used=1
qm_status: alloc'd from presence: utils_func.h: uandd_to_uri(66)
qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
qm_status: 10970. N address=0xb1b3e8 frag=0xb1b3b8 size=32 used=1
qm_status: alloc'd from presence: utils_func.h: uandd_to_uri(66)
qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
qm_status: 10971. N address=0xb1b468 frag=0xb1b438 size=32 used=1
qm_status: alloc'd from presence: utils_func.h: uandd_to_uri(66)
qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
qm_status: dumping free list stats :
qm_status: hash= 1. fragments no.: 4, unused: 0
bucket size: 8 - 8 (first 8)
qm_status: hash= 2. fragments no.: 8, unused: 0
bucket size: 16 - 16 (first 16)
qm_status: hash= 3. fragments no.: 6, unused: 0
bucket size: 24 - 24 (first 24)
qm_status: hash= 4. fragments no.: 1, unused: 0
bucket size: 32 - 32 (first 32)
qm_status: hash= 6. fragments no.: 3, unused: 0
bucket size: 48 - 48 (first 48)
qm_status: hash= 8. fragments no.: 8, unused: 0
bucket size: 64 - 64 (first 64)
qm_status: hash= 9. fragments no.: 6, unused: 0
bucket size: 72 - 72 (first 72)
qm_status: hash= 12. fragments no.: 5, unused: 0
bucket size: 96 - 96 (first 96)
qm_status: hash= 14. fragments no.: 6, unused: 0
bucket size: 112 - 112 (first 112)
qm_status: hash= 17. fragments no.: 32, unused: 0
bucket size: 136 - 136 (first 136)
qm_status: hash= 19. fragments no.: 1, unused: 0
bucket size: 152 - 152 (first 152)
qm_status: hash= 21. fragments no.: 1, unused: 0
bucket size: 168 - 168 (first 168)
qm_status: hash= 29. fragments no.: 1, unused: 0
bucket size: 232 - 232 (first 232)
qm_status: hash= 31. fragments no.: 2, unused: 0
bucket size: 248 - 248 (first 248)
qm_status: hash= 33. fragments no.: 2, unused: 0
bucket size: 264 - 264 (first 264)
qm_status: hash= 36. fragments no.: 1, unused: 0
bucket size: 288 - 288 (first 288)
qm_status: hash= 57. fragments no.: 1, unused: 0
bucket size: 456 - 456 (first 456)
qm_status: hash= 60. fragments no.: 1, unused: 0
bucket size: 480 - 480 (first 480)
qm_status: hash= 76. fragments no.: 709, unused: 0
bucket size: 608 - 608 (first 608)
qm_status: hash= 88. fragments no.: 1, unused: 0
bucket size: 704 - 704 (first 704)
qm_status: hash= 116. fragments no.: 1, unused: 0
bucket size: 928 - 928 (first 928)
qm_status: hash= 145. fragments no.: 1, unused: 0
bucket size: 1160 - 1160 (first 1160)
qm_status: hash= 216. fragments no.: 2, unused: 0
bucket size: 1728 - 1728 (first 1728)
qm_status: hash= 2049. fragments no.: 1, unused: 0
bucket size: 16384 - 32768 (first 22032)
qm_status: hash= 2057. fragments no.: 1, unused: 0
bucket size: 4194304 - 8388608 (first 5982824)
Is it a memory leak in one of these modules?
Thanks in advance and kind regards,
laura
25 Jan
25 Jan
7:18 p.m.
On Tuesday 24 January 2012, laura testi wrote:
we are using the XMPP gateway modules:
• PUA
• PUA_XMPP
• XMPP
The behavior is ok but since the modules are started, the memory usage
is getting increased and now also the swap memory area is used.
I send to you the following information:
Hello Laura,
so kamailio uses that much memory that your machine runs out of memory? This
sounds like a shared memory issue to me. Can you have a look to the shm_mem as
well?
1. Numer of records from PUA and xmpp_presentity
tables, they are quite
stable:
SQL> select count(*) from pua;
COUNT(*)
----------
10218
SQL> select count(*) from xmpp_presentity;
COUNT(*)
----------
754
This output is in PKG mem, but it looks indeed a bit suspicious. Have you
looked in to the respective code for this module, presence: utils_func.h and
uandd_to_uri?
2. list of PKG fragments with gdb :
$1 = {size = 40, u = {nxt_free = 0x0, is_free = 0}, file =
0x2b657aa12c00 "presence: utils_func.h", func = 0x2b657aa1aed0
"uandd_to_uri", line = 66,
check = 4042322160}
[..]
$17 = {size = 40, u = {nxt_free = 0x0, is_free = 0}, file =
0x2b657aa12c00 "presence: utils_func.h", func = 0x2b657aa1aed0
"uandd_to_uri", line = 66,
check = 4042322160}………….
3. Output of sercmd cfg.set_now_int core mem_dump_pkg :
……………….. 10969. N address=0xb1b320 frag=0xb1b2f0 size=104 used=1
qm_status: alloc'd from presence: utils_func.h: uandd_to_uri(66)
qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
qm_status: 10970. N address=0xb1b3e8 frag=0xb1b3b8 size=32 used=1
qm_status: alloc'd from presence: utils_func.h: uandd_to_uri(66)
qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
qm_status: 10971. N address=0xb1b468 frag=0xb1b438 size=32 used=1
qm_status: alloc'd from presence: utils_func.h: uandd_to_uri(66)
qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
qm_status: dumping free list stats :
qm_status: hash= 1. fragments no.: 4, unused: 0
bucket size: 8 - 8 (first 8)
qm_status: hash= 2. fragments no.: 8, unused: 0
bucket size: 16 - 16 (first 16)
qm_status: hash= 3. fragments no.: 6, unused: 0
bucket size: 24 - 24 (first 24)
qm_status: hash= 4. fragments no.: 1, unused: 0
bucket size: 32 - 32 (first 32)
qm_status: hash= 6. fragments no.: 3, unused: 0
bucket size: 48 - 48 (first 48)
qm_status: hash= 8. fragments no.: 8, unused: 0
bucket size: 64 - 64 (first 64)
qm_status: hash= 9. fragments no.: 6, unused: 0
bucket size: 72 - 72 (first 72)
qm_status: hash= 12. fragments no.: 5, unused: 0
bucket size: 96 - 96 (first 96)
qm_status: hash= 14. fragments no.: 6, unused: 0
bucket size: 112 - 112 (first 112)
qm_status: hash= 17. fragments no.: 32, unused: 0
bucket size: 136 - 136 (first 136)
qm_status: hash= 19. fragments no.: 1, unused: 0
bucket size: 152 - 152 (first 152)
qm_status: hash= 21. fragments no.: 1, unused: 0
bucket size: 168 - 168 (first 168)
qm_status: hash= 29. fragments no.: 1, unused: 0
bucket size: 232 - 232 (first 232)
qm_status: hash= 31. fragments no.: 2, unused: 0
bucket size: 248 - 248 (first 248)
qm_status: hash= 33. fragments no.: 2, unused: 0
bucket size: 264 - 264 (first 264)
qm_status: hash= 36. fragments no.: 1, unused: 0
bucket size: 288 - 288 (first 288)
qm_status: hash= 57. fragments no.: 1, unused: 0
bucket size: 456 - 456 (first 456)
qm_status: hash= 60. fragments no.: 1, unused: 0
bucket size: 480 - 480 (first 480)
qm_status: hash= 76. fragments no.: 709, unused: 0
bucket size: 608 - 608 (first 608)
qm_status: hash= 88. fragments no.: 1, unused: 0
bucket size: 704 - 704 (first 704)
qm_status: hash= 116. fragments no.: 1, unused: 0
bucket size: 928 - 928 (first 928)
qm_status: hash= 145. fragments no.: 1, unused: 0
bucket size: 1160 - 1160 (first 1160)
qm_status: hash= 216. fragments no.: 2, unused: 0
bucket size: 1728 - 1728 (first 1728)
qm_status: hash= 2049. fragments no.: 1, unused: 0
bucket size: 16384 - 32768 (first 22032)
qm_status: hash= 2057. fragments no.: 1, unused: 0
bucket size: 4194304 - 8388608 (first 5982824)
Is it a memory leak in one of these modules?
Peter found recently a lot of leaks in other presence modules (take a look to
the sr-dev list), so possible that they are more in this module.
Viele Grüße/ best regards,
Henning Westerholt
26 Jan
26 Jan
1:34 p.m.
Hi Hennings,
thanks a lot for your answer.
I'm sending you other information about our Kamailio xmpp server.
in attach the "sercmd cfg.set_now_int core mem_dump_shm" output.
And here:
kamctl fifo get_statistics | grep mem
shmem:fragments = 13107
shmem:free_size = 99724520
shmem:max_used_size = 57559216
shmem:real_used_size = 34493208
shmem:total_size = 134217728
shmem:used_size = 25651280
Thanks an kind regards,
laura
On Wed, Jan 25, 2012 at 7:18 PM, Henning Westerholt <hw(a)kamailio.org> wrote:
On Tuesday 24 January 2012, laura testi wrote:
we are using the XMPP gateway modules:
• PUA
• PUA_XMPP
• XMPP
The behavior is ok but since the modules are started, the memory usage
is getting increased and now also the swap memory area is used.
I send to you the following information:
Hello Laura,
so kamailio uses that much memory that your machine runs out of memory? This
sounds like a shared memory issue to me. Can you have a look to the shm_mem as
well?
1. Numer of records from PUA and
xmpp_presentity tables, they are quite
stable:
SQL> select count(*) from pua;
COUNT(*)
----------
10218
SQL> select count(*) from xmpp_presentity;
COUNT(*)
----------
754
This output is in PKG mem, but it looks indeed a bit suspicious. Have you
looked in to the respective code for this module, presence: utils_func.h and
uandd_to_uri?
2. list of PKG fragments with gdb :
$1 = {size = 40, u = {nxt_free = 0x0, is_free = 0}, file =
0x2b657aa12c00 "presence: utils_func.h", func = 0x2b657aa1aed0
"uandd_to_uri", line = 66,
check = 4042322160}
[..]
$17 = {size = 40, u = {nxt_free = 0x0, is_free = 0}, file =
0x2b657aa12c00 "presence: utils_func.h", func = 0x2b657aa1aed0
"uandd_to_uri", line = 66,
check = 4042322160}………….
3. Output of sercmd cfg.set_now_int core mem_dump_pkg :
……………….. 10969. N address=0xb1b320 frag=0xb1b2f0 size=104 used=1
qm_status: alloc'd from presence: utils_func.h: uandd_to_uri(66)
qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
qm_status: 10970. N address=0xb1b3e8 frag=0xb1b3b8 size=32 used=1
qm_status: alloc'd from presence: utils_func.h: uandd_to_uri(66)
qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
qm_status: 10971. N address=0xb1b468 frag=0xb1b438 size=32 used=1
qm_status: alloc'd from presence: utils_func.h: uandd_to_uri(66)
qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
qm_status: dumping free list stats :
qm_status: hash= 1. fragments no.: 4, unused: 0
bucket size: 8 - 8 (first 8)
qm_status: hash= 2. fragments no.: 8, unused: 0
bucket size: 16 - 16 (first 16)
qm_status: hash= 3. fragments no.: 6, unused: 0
bucket size: 24 - 24 (first 24)
qm_status: hash= 4. fragments no.: 1, unused: 0
bucket size: 32 - 32 (first 32)
qm_status: hash= 6. fragments no.: 3, unused: 0
bucket size: 48 - 48 (first 48)
qm_status: hash= 8. fragments no.: 8, unused: 0
bucket size: 64 - 64 (first 64)
qm_status: hash= 9. fragments no.: 6, unused: 0
bucket size: 72 - 72 (first 72)
qm_status: hash= 12. fragments no.: 5, unused: 0
bucket size: 96 - 96 (first 96)
qm_status: hash= 14. fragments no.: 6, unused: 0
bucket size: 112 - 112 (first 112)
qm_status: hash= 17. fragments no.: 32, unused: 0
bucket size: 136 - 136 (first 136)
qm_status: hash= 19. fragments no.: 1, unused: 0
bucket size: 152 - 152 (first 152)
qm_status: hash= 21. fragments no.: 1, unused: 0
bucket size: 168 - 168 (first 168)
qm_status: hash= 29. fragments no.: 1, unused: 0
bucket size: 232 - 232 (first 232)
qm_status: hash= 31. fragments no.: 2, unused: 0
bucket size: 248 - 248 (first 248)
qm_status: hash= 33. fragments no.: 2, unused: 0
bucket size: 264 - 264 (first 264)
qm_status: hash= 36. fragments no.: 1, unused: 0
bucket size: 288 - 288 (first 288)
qm_status: hash= 57. fragments no.: 1, unused: 0
bucket size: 456 - 456 (first 456)
qm_status: hash= 60. fragments no.: 1, unused: 0
bucket size: 480 - 480 (first 480)
qm_status: hash= 76. fragments no.: 709, unused: 0
bucket size: 608 - 608 (first 608)
qm_status: hash= 88. fragments no.: 1, unused: 0
bucket size: 704 - 704 (first 704)
qm_status: hash= 116. fragments no.: 1, unused: 0
bucket size: 928 - 928 (first 928)
qm_status: hash= 145. fragments no.: 1, unused: 0
bucket size: 1160 - 1160 (first 1160)
qm_status: hash= 216. fragments no.: 2, unused: 0
bucket size: 1728 - 1728 (first 1728)
qm_status: hash= 2049. fragments no.: 1, unused: 0
bucket size: 16384 - 32768 (first 22032)
qm_status: hash= 2057. fragments no.: 1, unused: 0
bucket size: 4194304 - 8388608 (first 5982824)
Is it a memory leak in one of these modules?
Peter found recently a lot of leaks in other presence modules (take a look to
the sr-dev list), so possible that they are more in this module.
Viele Grüße/ best regards,
Henning Westerholt 
1:53 p.m.
On 01/26/2012 02:34 PM, laura testi wrote:
Hi Hennings,
thanks a lot for your answer.
I'm sending you other information about our Kamailio xmpp server.
in attach the "sercmd cfg.set_now_int core mem_dump_shm" output.
And here:
kamctl fifo get_statistics | grep mem
shmem:fragments = 13107
shmem:free_size = 99724520
shmem:max_used_size = 57559216
shmem:real_used_size = 34493208
shmem:total_size = 134217728
shmem:used_size = 25651280
Thanks an kind regards,
Hello,
This seems pretty low (and normal). Probably there are some problems
with external libraries (API) using malloc and not properly freeing
memory. This is not a problem with the library itself, but more with the
way the library is used. For example the libpcre library had this sort
of leak when used in the regex modules.
Look for external libraries (by checking the dependency list) for the
custom setup you have. Then, try to match every external library
instantiation(allocation) (via library method) to a corresponding
release of resources.
Cheers
Marius
--
Zbihlei Marius
Head of
Linux Development Services Romania
1&1 Internet Development srl Tel KA: 754-9512
Str Mircea Eliade 18 Tel RO: +40-31-223-9512
Sect 1, Bucuresti mailto: marius.zbihlei(a)1and1.ro
71295, Romania
2:06 p.m.
Hello,
Some of the memory leaks we found here at Crocodile were to do with the
use of libxml2.
Various calls to libxml2 functions resulted in libxml2 malloc'ing
memory. This means you have to call the appropriate xmlFree...()
functions in the Kamailio modules to free up this memory. In some
places xmlFree() wasn't called, in other places xmlFree() was called
when a different function (for example, xmlFreeDoc()) should have been
used.
These memory leaks were not in Kamailio shared or private memory so the
statistics from Kamailio didn't show anything unusual.
As xmpp is an XML based protocol I would start by looking at where
libxml2 functions which allocate memory are being used and confirm that
the correct xmlFree...() functions are called during both normal and
abnormal cases.
Regards,
Peter
On Thu, 2012-01-26 at 14:53 +0200, Marius Zbihlei wrote:
On 01/26/2012 02:34 PM, laura testi wrote:
--
Peter Dunkley
Technical Director
Crocodile RCS Ltd
Hi Hennings,
thanks a lot for your answer.
I'm sending you other information about our Kamailio xmpp server.
in attach the "sercmd cfg.set_now_int core mem_dump_shm" output.
And here:
kamctl fifo get_statistics | grep mem
shmem:fragments = 13107
shmem:free_size = 99724520
shmem:max_used_size = 57559216
shmem:real_used_size = 34493208
shmem:total_size = 134217728
shmem:used_size = 25651280
Thanks an kind regards,
Hello,
This seems pretty low (and normal). Probably there are some problems
with external libraries (API) using malloc and not properly freeing
memory. This is not a problem with the library itself, but more with the
way the library is used. For example the libpcre library had this sort
of leak when used in the regex modules.
Look for external libraries (by checking the dependency list) for the
custom setup you have. Then, try to match every external library
instantiation(allocation) (via library method) to a corresponding
release of resources.
Cheers
Marius
2:19 p.m.
Hi,
Looking roughly in the pua_xmpp module I found some error cases where
the memory allocated by libxml2 was not freed. I have fixed them now in
master.
Not sure though if there are related to the memory leak Laura observed.
Laura, can you check if there were any error messages in the logs
printed by pua_xmpp module?
Regards,
Anca
On 01/26/2012 03:06 PM, Peter Dunkley wrote:
Hello,
Some of the memory leaks we found here at Crocodile were to do with
the use of libxml2.
Various calls to libxml2 functions resulted in libxml2 malloc'ing
memory. This means you have to call the appropriate xmlFree...()
functions in the Kamailio modules to free up this memory. In some
places xmlFree() wasn't called, in other places xmlFree() was called
when a different function (for example, xmlFreeDoc()) should have been
used.
These memory leaks were not in Kamailio shared or private memory so
the statistics from Kamailio didn't show anything unusual.
As xmpp is an XML based protocol I would start by looking at where
libxml2 functions which allocate memory are being used and confirm
that the correct xmlFree...() functions are called during both normal
and abnormal cases.
Regards,
Peter
On Thu, 2012-01-26 at 14:53 +0200, Marius Zbihlei wrote:
On 01/26/2012 02:34 PM, laura testi wrote:
--
Peter Dunkley
Technical Director
Crocodile RCS Ltd
Hi Hennings,
thanks a lot for your answer.
I'm sending you other information about our Kamailio xmpp server.
in attach the "sercmd cfg.set_now_int core mem_dump_shm" output.
And here:
kamctl fifo get_statistics | grep mem
shmem:fragments = 13107
shmem:free_size = 99724520
shmem:max_used_size = 57559216
shmem:real_used_size = 34493208
shmem:total_size = 134217728
shmem:used_size = 25651280
Thanks an kind regards,
Hello,
This seems pretty low (and normal). Probably there are some problems
with external libraries (API) using malloc and not properly freeing
memory. This is not a problem with the library itself, but more with the
way the library is used. For example the libpcre library had this sort
of leak when used in the regex modules.
Look for external libraries (by checking the dependency list) for the
custom setup you have. Then, try to match every external library
instantiation(allocation) (via library method) to a corresponding
release of resources.
Cheers
Marius
2:31 p.m.
Hi,
I also found a memory leak in pua module which is probably causing the
increase in memory that you observe.
Laura, if you could please update your code and check?
Regards,
Anca
4487
days inactive
4489
days old
6 comments
5 participants
participants (5)
-
Anca Vamanu -
Henning Westerholt -
laura testi -
Marius Zbihlei -
Peter Dunkley