Hello,
I would say that
perl_exec() is the
one with the highest
chances to be the
reason for the leak.
Next is line would
be db_mysql module,
if liked with some
custom mysql client
library, although
even in this case
will be unlikely.
Back to perl, the
module itself does
not call any malloc,
so it might be the
embedding Perl API
that is not used
properly in the
module.
Can you use some
testbed, set
children=1 and run
kamailio under
valgrind, then do
some calls and see
if it detects the
source of the leak?
I'm not using the
perl module, I will
try to check it
whenever I get a
chance in the next
days. What version
of perl do you have
installed?
Cheers,
Daniel
On 7/24/13
10:31 AM,
David
Cunningham
wrote:
Hello,
We don't do
any kamctl
commands at
all. We do
have various
modules
loaded, as
follows. The
primary
functions we
use Kamailio
for are phone
registrations
through
usrloc, and
routing calls
to Asterisk
through logic
contained in
Perl via
perl_exec().
Thanks for all
your advice so
far!
loadmodule
"tm.so"
loadmodule
"tmx.so"
loadmodule
"usrloc.so"
loadmodule
"auth.so"
loadmodule
"auth_db.so"
loadmodule
"ctl.so"
loadmodule
"db_mysql.so"
loadmodule
"kex.so"
loadmodule
"maxfwd.so"
loadmodule
"mi_fifo.so"
loadmodule
"mi_rpc.so"
loadmodule
"nathelper.so"
loadmodule
"perl.so"
loadmodule
"pv.so"
loadmodule
"registrar.so"
loadmodule
"rr.so"
loadmodule
"sanity.so"
loadmodule
"siputils.so"
loadmodule
"sl.so"
loadmodule
"textops.so"
loadmodule
"xlog.so"
On
24 July 2013
16:33,
Daniel-Constantin
Mierla
<miconda@gmail.com>
wrote:
Hello,
On
7/24/13 4:24
AM, David
Cunningham
wrote:
Hello,
Thank
you very much
for the email.
In reply:
1. The system
ran out of
memory.
Linux's
oom-killer
killed
Kamailio.
then all the
instructions I
gave are
useless, they
are for
debugging
kamailio's
internal
memory
manager, which
handles pkg
and shm
mallocs.
The chances to
be from
kamailio
itself are
very low now.
Do you do lot
of mi commands
(e.g., kamctl
...)? The mi
api uses
system malloc,
but the rest
of code should
use internal
memory manager
which does not
go beyond the
limits set
with -m and
-M, thus not
causing an OS
memory
exhaustion.
Can you list
what modules
are you
loading? At
some point it
was a leak in
libssl, in
case you use
tls a lot. But
could be
another
external
library...
Cheers,
Daniel
2. You're
right,
DEBUG_MEMORY
is a local
configuration
setting. If
defined it
sets memdbg to
-2, and memlog
to -2. The
debug setting
is -1.
3. We'll try
setting
mem_summary=12,
thanks.
4. We'll try
setting
asynchronous
syslog,
thanks.
5. Our
configuration
totals 338
lines, or
approx 8.5kb.
Is that a lot?
6. We'll try
setting
mem_join=1,
thanks.
On
23 July 2013
16:53,
Daniel-Constantin
Mierla
<miconda@gmail.com>
wrote:
Hello,
first, to
clarify, is
the system
memory or
kamailio's
pkg/shm memory
running out?
If the
operating
system runs
out of memory,
then should be
a leak in a
library,
because
kamailio
modules uses
only from a
pre-allocated
chunk, not
going over it.
On 7/23/13
7:33 AM, David
Cunningham
wrote:
Hello,
We're running
a Kamailio
3.3.4 system,
and Kamailio
is slowly
using more and
more memory.
Over a couple
of weeks it
will run out
of system
memory.
We tried to
enable memory
debugging
doing the
following, but
it resulted in
Kamailio not
responding to
any SIP
packets. Would
anyone have
advice please
on how to
debug the
situation?
1. In
Makefile.defs
set MEMDBG to
1 and
recompile
Kamailio.
2. In
kamailio.cfg
add the line:
#!define
DEBUG_MEMORY 1
do you set
something
special in
config when
DEBUG_MEMORY
is 1? It is
not by default
there, so I
assume you
added some
rules based on
this
pre-processor
directive.
For memory
troubleshooting,
set memlog to
a value lower
than debug
parameter in
config file
and try with
mem_summary=12
for a more
compact
output. See
more about
these
parameters in
the wiki:
- http://www.kamailio.org/wiki/cookbooks/3.3.x/core#memlog
Run kamailio
for a while in
normal
conditions,
then restart
it to get the
memory usage
summaries.
There should
be indication
if there is
some leak, by
seeing memory
chunks
allocated many
times from a
function used
at runtime.
You can send
the memory
summary for a
process here,
we can look at
it.
While this was
running and
Kamailio
didn't respond
to packets, it
logged lots of
lines like
this:
Do you have
syslog to be
configured in
asynchronous
mode? See the
notes from:
- http://www.kamailio.org/wiki/tutorials/3.2.x/syslog
The memdbg is
less than
debug value,
that means
printing few
log messages
for each
memory
operation. You
can make
memdbg higher
and rely on
memlog for
memory
summaries,
otherwise will
be lot of log
messages
related to
memory.
Jul 22
21:32:22
hostname
kamailio: :
<core>
[mem/q_malloc.c:369]:
qm_malloc(0x4000e008,
128) called
from
<core>:
cfg.lex:
addstr(1438)
Jul 22
21:32:22
hostname
kamailio: :
<core>
[mem/q_malloc.c:413]:
qm_malloc(0x4000e008,
128) returns
address
0x40048918
frag.
0x40048900
(size=128) on
1 -th hit
Jul 22
21:32:22
hostname
kamailio: :
<core>
[mem/q_malloc.c:369]:
qm_malloc(0x4000e008,
128) called
from
<core>:
cfg.lex:
addstr(1438)
Jul 22
21:32:22
hostname
kamailio: :
<core>
[mem/q_malloc.c:413]:
qm_malloc(0x4000e008,
128) returns
address
0x400489c8
frag.
0x400489b0
(size=128) on
1 -th hit
addstr() is a
function used
only for
parsing
configuration
file, as long
as you can
still see
them, the
configuration
file parsing
was not
finish.
addstr() is
not a source
of leaks
because it is
not used at
runtime.
If you have
large config
file, then you
can get close
to the limits
of the private
memory, which
is set to 4MB.
You can
increase its
value using -M
parameter
(e.g., start
kamailio with
-M 8 to set it
to use 8MB of
memory).
Over the time,
the private
memory can get
used due to
fragmentation,
you can set
the mem_join
parameter in
config file to
avoid it
(works when
compiled with
MEMDBG=1).
To monitor
usage of
internal pkg
memory, then
you can use
sercmd with
pkg.stats
command:
http://kamailio.org/docs/modules/3.3.x/modules_k/kex.html#idp16972640
Shared memory
stats are
printed by
'kamctl fifo
get_statistics
shmem:'
When you see
significant
increase of
the memory
usage, then
you can
restart to get
the summaries.
You should run
these commands
after start,
just to see
the initial
usage of
memory.
Cheers,
Daniel
--
Daniel-Constantin
Mierla - http://www.asipto.com
http://twitter.com/#!/miconda
- http://www.linkedin.com/in/miconda
_______________________________________________
SIP Express
Router (SER)
and Kamailio
(OpenSER) -
sr-users
mailing list
sr-users@lists.sip-router.org
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
--
David
Cunningham,
Voisonics
http://voisonics.com/
USA: +1
213 221 1092
UK: +44
(0) 20 3298
1642
Australia: +61 (0) 2 8063 9019
--
Daniel-Constantin Mierla - http://www.asipto.com
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
--
David
Cunningham,
Voisonics
http://voisonics.com/
USA: +1
213 221 1092
UK: +44
(0) 20 3298
1642
Australia: +61 (0) 2 8063 9019
--
Daniel-Constantin Mierla - http://www.asipto.com
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda