2 Oct
2013
2 Oct
'13
10:39 a.m.
Hello,
on a second check, those counter chunks for counters/statistics seems to
be from startup, thus not part of a leak.
How long have you been running kamailio? Looking at the pkg stats from
previous email, the free part is still a lot, with only about 200k used.
It could mean the leak was not making any effect yet.
Cheers,
Daniel
On 10/2/13 10:19 AM, Daniel-Constantin Mierla wrote:
Hello,
the leak is due to keep defining counters or statistics. That should
happen only at startup, but if you execute from perl script some
function from a kamailio module, it may trigger this issue. Can you
share what functions from kamailio are you execute inside the perl script?
Cheers,
Daniel
On 10/2/13 9:26 AM, David Cunningham wrote:
--
Daniel-Constantin Mierla - http://www.asipto.com
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Kamailio Advanced Trainings - Berlin, Nov 25-28; Miami, Nov 18-20, 2013
- more details about Kamailio trainings at http://www.asipto.com -
Hi Daniel,
Thank you - I used the restart method and got a whole load of memory
information. 10 megabytes of it! I've compressed that and attached it
to this email.
Thanks for your help.
On 2 October 2013 17:17, Daniel-Constantin Mierla <miconda(a)gmail.com
<mailto:miconda@gmail.com>> wrote:
Hello,
ahh, right, forgot that it was disabled because it was not safe
to use at runtime for worker processes. You have to use:
kamcmd cfg.set_now_int core mem_dump_pkg <pid>
then wait for a bit so that process is receiving a SIP message.
Or, if you restart, the stats are printed in syslog for all
processes.
Another alternative is to call pkg_status() or pkg_summary() from
cfgutils module in config file, like:
if(src_ip==127.0.0.1 && is_method("OPTIONS") &&
$rU=="pkgstats") {
send_reply("200", "OK");
pkg_summary();
exit;
}
You can use sipsak to send an options request to "sip:pkgstats@..."
Cheers,
Daniel
On 10/2/13 2:30 AM, David Cunningham wrote:
--
Daniel-Constantin Mierla -http://www.asipto.com
http://twitter.com/#!/miconda -http://www.linkedin.com/in/miconda
Kamailio Advanced Trainings - Berlin, Nov 25-28; Miami, Nov 18-20, 2013
- more details about Kamailio trainings athttp://www.asipto.com - Hi Daniel,
When I send SIGUSR1 to a worker process nothing is logged. Any
ideas there? The following is the output of "sercmd pkg.stats".
{
entry: 0
pid: 14570
rank: 0
used: 191100
free: 3731504
real_used: 462800
}
{
entry: 1
pid: 14571
rank: 1
used: 202656
free: 3714236
real_used: 480068
}
{
entry: 2
pid: 14572
rank: 2
used: 202720
free: 3714220
real_used: 480084
}
{
entry: 3
pid: 14573
rank: 3
used: 202664
free: 3714324
real_used: 479980
}
{
entry: 4
pid: 14574
rank: 4
used: 202744
free: 3714100
real_used: 480204
}
{
entry: 5
pid: 14575
rank: 5
used: 201908
free: 3715176
real_used: 479128
}
{
entry: 6
pid: 14576
rank: 6
used: 202572
free: 3714368
real_used: 479936
}
{
entry: 7
pid: 14577
rank: 7
used: 202740
free: 3714488
real_used: 479816
}
{
entry: 8
pid: 14578
rank: 8
used: 199760
free: 3717084
real_used: 477220
}
{
entry: 9
pid: 14579
rank: 9
used: 199440
free: 3722828
real_used: 471476
}
{
entry: 10
pid: 14580
rank: 10
used: 199440
free: 3722828
real_used: 471476
}
{
entry: 11
pid: 14581
rank: 11
used: 199440
free: 3722828
real_used: 471476
}
{
entry: 12
pid: 14582
rank: 12
used: 199440
free: 3722828
real_used: 471476
}
{
entry: 13
pid: 14583
rank: 13
used: 199440
free: 3722828
real_used: 471476
}
{
entry: 14
pid: 14584
rank: 14
used: 199440
free: 3722828
real_used: 471476
}
{
entry: 15
pid: 14585
rank: 15
used: 199440
free: 3722828
real_used: 471476
}
{
entry: 16
pid: 14586
rank: 16
used: 199440
free: 3722828
real_used: 471476
}
{
entry: 17
pid: 14587
rank: -1
used: 199400
free: 3722868
real_used: 471436
}
{
entry: 18
pid: 14588
rank: -1
used: 2105844
free: 3722868
real_used: 2337944
}
{
entry: 19
pid: 14589
rank: -2
used: 198552
free: 3723236
real_used: 471068
}
{
entry: 20
pid: 0
rank: 0
used: 209756
free: 3712368
real_used: 481936
}
{
entry: 21
pid: 14591
rank: -4
used: 253660
free: 3668704
real_used: 525600
}
On 1 October 2013 22:45, Daniel-Constantin Mierla
<miconda(a)gmail.com <mailto:miconda@gmail.com>> wrote:
Hello,
send SIGUSR1 to one of the worker processes (e.g., UDP
listener). The main process of kamailio doesn't process SIP
messages, so it is not exposed to leaks.
You can used 'kamcmd pkg.stats" to see statistics about
usage of pkg per each process (older versions have sercmd).
'kamctl ps' gives the list of processes.
Cheers,
Daniel
On 10/1/13 2:35 PM, David Cunningham wrote:
--
Daniel-Constantin Mierla -http://www.asipto.com
http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda>
-http://www.linkedin.com/in/miconda
Kamailio Advanced Trainings - Berlin, Nov 25-28; Miami, Nov 18-20, 2013
- more details about Kamailio trainings athttp://www.asipto.com -
--
David Cunningham, Voisonics
http://voisonics.com/
USA: +1 213 221 1092
UK: +44 (0) 20 3298 1642
Australia: +61 (0) 2 8063 9019 Hi Daniel,
What I copied and pasted is all that's listed when I do a
kill -SIGUSR1 on the Kamailio master process. How do I get
fuller logs?
Thanks for your help.
On 1 October 2013 21:57, Daniel-Constantin Mierla
<miconda(a)gmail.com <mailto:miconda@gmail.com>> wrote:
Hello,
you gave only the first part of memory log output. The
chunks listed there are from startup, thus not part of
a leak, because they are not allocated again at runtime.
Can you give the full logs with messages from
qm_status? The leaks are typically listed towards the
end of those messages.
Yesterday someone else reported a potential leak when
using perl module, so it might be something else there.
I found one, which is related to an error case, but
perhaps you don't get that, otherwise you will notice
the error message.
Cheers,
Daniel
On 10/1/13 12:03 PM, David Cunningham wrote:
Hello,
Following up on this after a long break. It seems
we are still having a memory leak. We've added the
memory debugging options as per the documentation
with the following output.
I'm not sure how to read it though - can anyone
help? Thank you.
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: <core> [main.c:754]: Memory status (pkg):
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: (0x4019b008):
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: heap size= 4194304
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: used= 191100,
used+overhead=462800, free=3731504
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: max used (+overhead)= 480224
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: dumping all alloc'ed. fragments:
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: 0. N address=0x401cf414
frag=0x401cf3fc size=512 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: alloc'd from <core>:
str_hash.h: str_hash_alloc(69)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: start check=f0f0f0f0, end
check= c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: 1. N address=0x401cf644
frag=0x401cf62c size=128 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: alloc'd from <core>:
str_hash.h: str_hash_alloc(69)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: start check=f0f0f0f0, end
check= c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: 2. N address=0x401cf6f4
frag=0x401cf6dc size=256 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: alloc'd from <core>:
counters.c: cnt_hash_add(331)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: start check=f0f0f0f0, end
check= c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: 3. N address=0x401cf824
frag=0x401cf80c size=64 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: alloc'd from <core>:
counters.c: init_counters(121)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: start check=f0f0f0f0, end
check= c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: 4. N address=0x401cf894
frag=0x401cf87c size=8 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: alloc'd from <core>:
route.c: init_rlist(180)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: start check=f0f0f0f0, end
check= c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: 5. N address=0x401cf8cc
frag=0x401cf8b4 size=64 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: alloc'd from <core>:
str_hash.h: str_hash_alloc(69)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: start check=f0f0f0f0, end
check= c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: 6. N address=0x401cf93c
frag=0x401cf924 size=24 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: alloc'd from <core>:
route.c: route_add(159)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: start check=f0f0f0f0, end
check= c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: 7. N address=0x401cf984
frag=0x401cf96c size=8 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: alloc'd from <core>:
route.c: init_rlist(180)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: start check=f0f0f0f0, end
check= c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: 8. N address=0x401cf9bc
frag=0x401cf9a4 size=64 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: alloc'd from <core>:
str_hash.h: str_hash_alloc(69)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: start check=f0f0f0f0, end
check= c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: 9. N address=0x401cfa2c
frag=0x401cfa14 size=24 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: alloc'd from <core>:
route.c: route_add(159)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: start check=f0f0f0f0, end
check= c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: 10. N address=0x401cfa74
frag=0x401cfa5c size=8 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: alloc'd from <core>:
db_id.c: dupl_string(50)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: start check=f0f0f0f0, end
check= c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: 11. N address=0x401cfaac
frag=0x401cfa94 size=64 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: alloc'd from <core>:
str_hash.h: str_hash_alloc(69)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: start check=f0f0f0f0, end
check= c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: 12. N address=0x401cfb1c
frag=0x401cfb04 size=24 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: alloc'd from <core>:
route.c: route_add(159)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: start check=f0f0f0f0, end
check= c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: 13. N address=0x401cfb64
frag=0x401cfb4c size=8 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: alloc'd from <core>:
route.c: init_rlist(180)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: start check=f0f0f0f0, end
check= c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: 14. N address=0x401cfb9c
frag=0x401cfb84 size=64 used=1
--
Daniel-Constantin Mierla - http://www.asipto.com
http://twitter.com/#!/miconda
<http://twitter.com/#%21/miconda> -
http://www.linkedin.com/in/miconda
Kamailio Advanced Trainings - Berlin, Nov 25-28; Miami,
Nov 18-20, 2013
- more details about Kamailio trainings at
http://www.asipto.com -
--
David Cunningham, Voisonics
http://voisonics.com/
USA: +1 213 221 1092 <tel:%2B1%20213%20221%201092>
UK: +44 (0) 20 3298 1642
<tel:%2B44%20%280%29%2020%203298%201642>
Australia: +61 (0) 2 8063 9019
<tel:%2B61%20%280%29%202%208063%209019>
--
Daniel-Constantin Mierla -http://www.asipto.com
http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda>
-http://www.linkedin.com/in/miconda
Kamailio Advanced Trainings - Berlin, Nov 25-28; Miami, Nov 18-20, 2013
- more details about Kamailio trainings athttp://www.asipto.com -
--
David Cunningham, Voisonics
http://voisonics.com/
USA: +1 213 221 1092 <tel:%2B1%20213%20221%201092>
UK: +44 (0) 20 3298 1642 <tel:%2B44%20%280%29%2020%203298%201642>
Australia: +61 (0) 2 8063 9019
<tel:%2B61%20%280%29%202%208063%209019>