Hi,

I am trying to run a load test of 1000 messages/sec with 3gpp-SMS

Kamailio details

Kamailio version : 5.7.4 (downloaded from deb repo)
Ruuning under Ubuntu 18.04.6 LT
PKG Mem 32 MB
SHM Mem 1024 MB

When I run the load for around half an hour the PKG mem usgae keeps on increasing and finally after the mentioned (half an hour) time Kamailio start putting Error Logs that : could not allocate private memory from pkg pool

I referred to https://www.kamailio.org/wiki/tutorials/troubleshooting/memory, and when I use kamcmd pkg.stats I see blow out put for the udp receivers :

{
        entry: 1
        pid: 11070
        rank: 1
        used: 3652840
        free: 0
        real_used: 16777216
        total_size: 16777216
        total_frags: 0
        desc: udp receiver child=0 sock=192.168.136.151:5060
}
{
        entry: 2
        pid: 11071
        rank: 2
        used: 3652840
        free: 0
        real_used: 16777216
        total_size: 16777216
        total_frags: 0
        desc: udp receiver child=1 sock=192.168.136.151:5060
}
{
        entry: 3
        pid: 11072
        rank: 3
        used: 3652912
        free: 0
        real_used: 16777216
        total_size: 16777216
        total_frags: 0
        desc: udp receiver child=2 sock=192.168.136.151:5060
}

On further enabling memdbg for individual packets, I could see that for each message processing :
204 Malloc operations , 184 free operations .
Seems to be a memory leak condition . Am i missing something here ?

On matching the allocated and freed address, I could find below source files where address is allocated but not freed :

May  6 15:23:48 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[14646]: INFO: <core> [core/mem/f_malloc.c:450]: fm_malloc(): fm_malloc(0x7fc29f1fb010, 48) called from core: core/route.c: route_add(124)
May  6 15:23:48 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[14646]: INFO: <core> [core/mem/f_malloc.c:540]: fm_malloc(): fm_malloc(0x7fc29f1fb010, 48) returns address 0x7fc29f2a4188
May  6 15:23:48 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[14646]: INFO: <core> [core/mem/f_malloc.c:450]: fm_malloc(): fm_malloc(0x7fc29f1fb010, 48) called from core: core/route.c: route_add(124)
May  6 15:23:48 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[14646]: INFO: <core> [core/mem/f_malloc.c:540]: fm_malloc(): fm_malloc(0x7fc29f1fb010, 48) returns address 0x7fc29f2a4200
May  6 15:23:48 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[14646]: INFO: <core> [core/mem/f_malloc.c:450]: fm_malloc(): fm_malloc(0x7fc29f1fb010, 48) called from core: core/route.c: route_add(124)
May  6 15:23:48 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[14646]: INFO: <core> [core/mem/f_malloc.c:540]: fm_malloc(): fm_malloc(0x7fc29f1fb010, 48) returns address 0x7fc29f2a4278
May  6 15:23:48 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[14646]: INFO: <core> [core/mem/f_malloc.c:450]: fm_malloc(): fm_malloc(0x7fc29f1fb010, 48) called from core: core/route.c: route_add(124)
May  6 15:23:48 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[14646]: INFO: <core> [core/mem/f_malloc.c:540]: fm_malloc(): fm_malloc(0x7fc29f1fb010, 48) returns address 0x7fc29f2a42f0

May  6 15:23:48 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[14646]: INFO: <core> [core/mem/f_malloc.c:540]: fm_malloc(): fm_malloc(0x7fc29f1fb010, 64) returns address 0x7fc29f2a44f8
May  6 15:23:48 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[14646]: INFO: <core> [core/mem/f_malloc.c:450]: fm_malloc(): fm_malloc(0x7fc29f1fb010, 64) called from core: core/parser/msg_parser.c: parse_heade

May  6 15:23:48 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[14646]: INFO: <core> [core/mem/f_malloc.c:540]: fm_malloc(): fm_malloc(0x7fc29f1fb010, 64) returns address 0x7fc29f2b2df8
May  6 15:23:48 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[14646]: INFO: <core> [core/mem/f_malloc.c:450]: fm_malloc(): fm_malloc(0x7fc29f1fb010, 64) called from core: core/parser/msg_parser.c: parse_heade

Need guidance support here how to avoid this memory leak condition and how to get more details to aid further investigation.

Thank you!
Regards
Paresh Panda

—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you are subscribed to this thread.