[SR-Users] private memory leak - but where?

[Sebastian Damm]

Hi,

I'm trying to debug a Kamailio which runs out of memory after a while.
But following the hints at
https://www.kamailio.org/wiki/tutorials/troubleshooting/memory
doesn't help. I know it is the private memory that I'm missing. I
already increased the pkg size to 16M, but somehow that seems to be
not enough.

The Kamailio receives HEP messages, sends them to a lua script where
it is processed. Overall there are not more than 10 custom private
variables used.

In the log I'm seeing these messages:

Sep 19 17:04:42 salm /usr/sbin/kamailio[19134]: ERROR: app_lua
[app_lua_api.c:726]: app_lua_run_ex(): error executing: handle_packet
(err: 2)
Sep 19 17:04:42 salm /usr/sbin/kamailio[19134]: ERROR: <script>:
SCRIPT: failed to execute lua function!
Sep 19 17:04:42 salm /usr/sbin/kamailio[19134]: ERROR: <core>
[core/mem/q_malloc.c:292]: qm_find_free():
qm_find_free(0x7f9744eea010, 232); Free fragment not found!
Sep 19 17:04:42 salm /usr/sbin/kamailio[19134]: ERROR: <core>
[core/mem/q_malloc.c:425]: qm_malloc(): qm_malloc(0x7f9744eea010, 232)
called from core: core/parser/msg_parser.c: get_
hdr_field(116), module: core; Free fragment not found!
Sep 19 17:04:42 salm /usr/sbin/kamailio[19134]: ERROR: <core>
[core/parser/msg_parser.c:118]: get_hdr_field(): out of memory
Sep 19 17:04:42 salm /usr/sbin/kamailio[19134]: ERROR: <core>
[core/parser/msg_parser.c:330]: parse_headers(): bad header field
[Via: SIP/2.0/WSS o6m]
Sep 19 17:04:42 salm /usr/sbin/kamailio[19134]: ERROR: pv
[pv_core.c:647]: pv_get_callid(): cannot parse Call-Id header

kamcmd pkg.stats for this process gives me this output.

{
  entry: 61
  pid: 19134
  rank: 61
  used: 8748080
  free: 4384
  real_used: 16772832
  total_size: 16777216
  total_frags: 30
}

Now I tried to get a dump of what is in use, following the guidelines
on the wiki page mentioned above, but without success.

I can easily start and stop the traffic coming in to Kamailio (by
starting or stopping the captagent sending it there), but I can't get
any output. Here's what I did:

captagent stop
kamcmd cfg.set_now_int core memlog 1
kamcmd cfg.set_now_int core mem_dump_pkg 19134
captagent start ; sleep 1 ; captagent stop

(I started captagent so that the process actually had something to do
because that's what the wiki page says it needs for printing memory
information.)

During that second, there were quite a lot of packets running through
this process, but I did not get a memory dump. A shm_summary dump
however works as expected.

What am I missing? How do I get more information about what this
process is using its memory for?

TIA
Sebastian