[SR-Users] kamailio memory leak in private memory

[david]

Hello all

We are observing an issue in one kamailio proxy we use.
The consecuence of an unknow cause is that the private memory of some
UDP receiver processes in where we listen from a particular IP, seems to
be draining along the time while processing traffic. Quite "slowly" but
maybe aroung 200K in 3-4hours, depending on the moment of the day.

This is quite weird for us, because it seemed like it was caused by some
particular traffic profile, since we have several IP listening (8
children each) and we only see the issue in the UDP receiver processes
of some of those IPs. Besides we have moved the kamailio service
listening on the same IP to several different physical servers, and the
behaviour is the same.
We look at it ussing pkg.stats kamcmd command and for instance if we
track one of the process affected
pid: 23784
rank: 53
used: 2838616
free: 13005408
real_used: 3771808

pid: 23784
rank: 53
used: 2842584
free: 12999280
real_used: 3777936

This 2 outputs were taken with 10min diff between each other.

But we dont know where or how to look at it.
We have been these last days trying to search for patterns to allocate
some particular traffic source causing something weird, but no luck.
We have checked all the SIP INVITE messages looking for anything weird
at any field but no luck, besides we sometimes see some bad headers with
blank spaces in the uri or thing like that but those cases we have an
error and we dont think it can cause a memory leak.
Shared memory seems to be stable.

We also have checked the config we are using and we have been discarding
possible issues since we dont think the $var variables we are using
could cause that issue (but you never know....)

We tried to debug the kamailio on one affected IP compiling it with
DBG_SYS_MALLOC , and doing the dump of the pkg mem status when the
kamailio was restarted and no traffic was processed, and also when it
had traffic processed. in this last case, we removed traffic, waited for
20min and made the dump

i attach you both files, we have the debug logs, but they are very big
to be passed here, and we dont know what to look for :)

could you please give us some guidance or clue to at least start to
searching for something that can cause this issue?

thanks  alot and regards
david
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.sip-router.org/pipermail/sr-users/attachments/20170217/00e4ea75/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: mem_dump_empty.gz
Type: application/gzip
Size: 3447 bytes
Desc: not available
URL: <http://lists.sip-router.org/pipermail/sr-users/attachments/20170217/00e4ea75/attachment.gz>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: mem_dump_load.gz
Type: application/gzip
Size: 3875 bytes
Desc: not available
URL: <http://lists.sip-router.org/pipermail/sr-users/attachments/20170217/00e4ea75/attachment-0001.gz>