Description

We noticed a PKG memory leak when using DMQ module's function dmq_bcast_message .

Troubleshooting

Reproduction

Issue can be reproduced with configuration below.

• 3 Kamailio nodes each one listening on dedicated port 5070 for DMQ on their private IP address.
• A dns name [KAMAILIO_DNS] resolving these 3 nodes:

# dig +short A [KAMAILIO_DNS]
10.234.97.169
10.234.97.170
10.234.92.126

DMQ module configured as below (example for node 1)

loadmodule "dmq.so"
...
modparam("dmq", "server_address", "sip:10.234.97.169:5070")
modparam("dmq", "notification_address", "sip:[KAMAILIO_DNS]:5070")
modparam("dmq", "ping_interval", 10)
modparam("dmq", "multi_notify", 1)

Each time an INVITE comes in, we use function below to broadcast custom message to all of the nodes:

...
if(is_method("INVITE")) {
    dmq_bcast_message("invitenotification", "invite received", "text/plain");
}
...

This custom message is handled as below:

request_route {
...
    if(is_method("KDMQ")) {
        if($Rp == 5070) route(DMQHANDLE);
        else drop;
    }
...
}
...

route[DMQHANDLE] {
    if($tU == "invitenotification") {
        # Doing some additional stuff here, but not the cause of the leak
        send_reply("200", "OK");
    } else {
        dmq_handle_message();
    }
}

Using sipp to send lots of messages, we can easily notice the free PKG memory decreasing over time (using kamcmd pkg.stats)

Debugging Data

Processes impacted by the leak are the ones processing SIP messages. In case of TCP, it is one of the tcp receivers.
When doing a dump of such processes with command below:

kamcmd cfg.set_now_int core mem_dump_pkg [PID]

Results are full of records like these ones:

ALERT: qm_status:           alloc'd from core: core/parser/parse_from.c: parse_from_header(63)
ALERT: qm_status:           alloc'd from core: core/parser/parse_addr_spec.c: parse_to_param(285)

We are sure that the leak is generated by this DMQ broadcast function: we isolated it and noticed no leaks when removing it from a basic script configuration.

We tried different configurations:

• disabling multi_notify => memory leak still present
• using an IP address instead of the DNS name => memory leak still present

Valgrind did not help us to find out more precisely where the issue in the code could be located.

Possible Solutions

Would a free of msg->from->parsed be missing somewhere in DMQ code?
Behavior looks very similar to #59

Additional Information

• Kamailio Version - output of kamailio -v

version: kamailio 5.3.7 (x86_64/linux) c16b3f-dirty
flags: , EXTRA_DEBUGUSE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, TIMER_DEBUG, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES

• Operating System:

Debian 9.13 under docker
Linux 4.9.0-9-amd64

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or unsubscribe.