[SR-Users] Possible memory leak in mysql driver

[Alexandru Covalschi]

Hello list,

We’re using dev version of Kamailio:
version: kamailio 5.0.0-dev4 (x86_64/linux) ff63e5
flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
id: ff63e5
compiled on 15:46:49 May 31 2016 with gcc 4.9.2

Sometimes we encounter such issue:
Nov  9 23:30:01 sbc01 /usr/local/sbin/kamailio[12276]: ERROR: <core> [db_row.c:114]: db_allocate_row(): no private memory left
Nov  9 23:30:01 sbc01 /usr/local/sbin/kamailio[12276]: ERROR: db_mysql [km_row.c:57]: db_mysql_convert_row(): could not allocate row
Nov  9 23:30:01 sbc01 /usr/local/sbin/kamailio[12276]: ERROR: db_mysql [km_res.c:188]: db_mysql_convert_rows(): error while converting row #16
Nov  9 23:30:01 sbc01 /usr/local/sbin/kamailio[12276]: ERROR: db_mysql [km_res.c:217]: db_mysql_convert_result(): error while converting rows
Nov  9 23:30:01 sbc01 /usr/local/sbin/kamailio[12276]: ERROR: db_mysql [km_dbase.c:261]: db_mysql_store_result(): error while converting result
Nov  9 23:30:01 sbc01 /usr/local/sbin/kamailio[12276]: ERROR: <core> [db_query.c:139]: db_do_query_internal(): error while storing result
Nov  9 23:30:01 sbc01 /usr/local/sbin/kamailio[12276]: ERROR: permissions [trusted.c:91]: reload_trusted_table(): failed to query database
Nov  9 23:30:01 sbc01 /usr/local/sbin/kamailio[12276]: ERROR: <core> [db_row.c:114]: db_allocate_row(): no private memory left
Nov  9 23:30:01 sbc01 /usr/local/sbin/kamailio[12276]: ERROR: db_mysql [km_row.c:57]: db_mysql_convert_row(): could not allocate row
Nov  9 23:30:01 sbc01 /usr/local/sbin/kamailio[12276]: ERROR: db_mysql [km_dbase.c:444]: db_mysql_fetch_result(): error while converting row #15
Nov  9 23:30:01 sbc01 /usr/local/sbin/kamailio[12276]: ERROR: htable [ht_db.c:234]: ht_db_load_table(): Error while fetching result
Nov  9 23:30:01 sbc01 /usr/local/sbin/kamailio[12276]: : <core> [mem/q_malloc.c:468]: qm_free(): BUG: qm_free: freeing already freed pointer (0x7f5ebda8ae18), called from db_mysql: km_dbase.c: db_mysql_free_result(305), first free db_mysql: km_dbase.c: db_mysql_free_result(305) - aborting
Nov  9 23:30:01 sbc01 /usr/local/sbin/kamailio[12281]: CRITICAL: <core> [pass_fd.c:275]: receive_fd(): EOF on 16
Nov  9 23:30:01 sbc01 /usr/local/sbin/kamailio[12268]: ALERT: <core> [main.c:739]: handle_sigs(): child process 12276 exited by a signal 6
Nov  9 23:30:01 sbc01 /usr/local/sbin/kamailio[12268]: ALERT: <core> [main.c:742]: handle_sigs(): core was not generated
Nov  9 23:30:01 sbc01 /usr/local/sbin/kamailio[12268]: INFO: <core> [main.c:754]: handle_sigs(): terminating due to SIGCHLD

The thing is we heavily use mysql module, but only to update the in-memory tables by kamcmd. Each N minutes a special script updates the trusted,address and htable executing kamcmd. Kamailio (and kamcmd as well) talks only with localhost mysql server.
What I saw when encountered that issue on a live machine is that issue happens only with one of child processes, any other are ok.
Interesting thing is that happens at the same time with machines on the same «set», I mean that issue happened simultaneously with two our test machines which actually didn’t have any load on them. 
The common thing between those machines is that they are in same subnet and local mysql databases are filled by scripts which query same external db.
I can’t confirm if there were or there weren’t any networking issues at that time with those machines, but as soon as kamcmd queries localhost that shouldn’t be the source of the issue.

So my questions are:
1. Has anyone encountered such thing?
2. Maybe the issue is already localized so it has sense to update? We actually use that on production (pls don’t throw too much rocks at me), so maintenance should be properly planned and I must be sure update won’t break anything. 
3. If update is proposed - how to do it? I mean - follow the guide https://www.kamailio.org/wiki/install/devel/git or there are some other tips? I suppose in ideal world I don’t even stop the binary, only restart after make all && make install are done, as everything is in-memory. Am I correct?
4. When can we expect stable 5.0 version? (at least tell if it’s months/years)

Thanks in advance!