[SR-Users] Possible memory leak in mysql driver
Alexandru Covalschi
568691 at gmail.com
Wed Nov 16 15:06:31 CET 2016
Thanks for response Daniel,
Can you point me to the location where the core dump should be generated? Afaik kamailio was compiled in /usr/local, what is the workdir for that setup?
I can't figure it out from systemd config file. Also, is it really necessary to run as root? What if I set enough permissions on workdir?
> 16 нояб. 2016 г., в 11:32, Daniel-Constantin Mierla <miconda at gmail.com> написал(а):
>
> Hello,
>
> the plan is to freeze development of v5.0 before the Chirstmas of the
> first week of January, then it will be a 1-1.5 months of testing,
> followed by the release. So expect like 2-3 months till the stable
> release of 5.0.
>
> As for the issue, it can be a buffer overflow somewhere, not related to
> mysql module code, but triggered by use of it.
>
> It is important to get the core file for such case, before starting
> kamailio do 'ulimit -c unlimited'. You may need to run kamailio as root
> to be able to write the core file. Enable also the option for one core
> file per process, typically is:
>
> echo "1" > /proc/sys/kernel/core_uses_pid
>
> Once you get core files, extract the output of 'bt full' with gdb and
> send it over here.
>
> Cheers,
> Daniel
>
>
> On 15/11/16 22:35, Alexandru Covalschi wrote:
>> Hello list,
>>
>> We’re using dev version of Kamailio:
>> version: kamailio 5.0.0-dev4 (x86_64/linux) ff63e5
>> flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES
>> ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB
>> poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
>> id: ff63e5
>> compiled on 15:46:49 May 31 2016 with gcc 4.9.2
>>
>> Sometimes we encounter such issue:
>> Nov 9 23:30:01 sbc01 /usr/local/sbin/kamailio[12276]: ERROR: <core> [db_row.c:114]: db_allocate_row(): no private memory left
>> Nov 9 23:30:01 sbc01 /usr/local/sbin/kamailio[12276]: ERROR: db_mysql [km_row.c:57]: db_mysql_convert_row(): could not allocate row
>> Nov 9 23:30:01 sbc01 /usr/local/sbin/kamailio[12276]: ERROR: db_mysql [km_res.c:188]: db_mysql_convert_rows(): error while converting row #16
>> Nov 9 23:30:01 sbc01 /usr/local/sbin/kamailio[12276]: ERROR: db_mysql [km_res.c:217]: db_mysql_convert_result(): error while converting rows
>> Nov 9 23:30:01 sbc01 /usr/local/sbin/kamailio[12276]: ERROR: db_mysql [km_dbase.c:261]: db_mysql_store_result(): error while converting result
>> Nov 9 23:30:01 sbc01 /usr/local/sbin/kamailio[12276]: ERROR: <core> [db_query.c:139]: db_do_query_internal(): error while storing result
>> Nov 9 23:30:01 sbc01 /usr/local/sbin/kamailio[12276]: ERROR: permissions [trusted.c:91]: reload_trusted_table(): failed to query database
>> Nov 9 23:30:01 sbc01 /usr/local/sbin/kamailio[12276]: ERROR: <core> [db_row.c:114]: db_allocate_row(): no private memory left
>> Nov 9 23:30:01 sbc01 /usr/local/sbin/kamailio[12276]: ERROR: db_mysql [km_row.c:57]: db_mysql_convert_row(): could not allocate row
>> Nov 9 23:30:01 sbc01 /usr/local/sbin/kamailio[12276]: ERROR: db_mysql [km_dbase.c:444]: db_mysql_fetch_result(): error while converting row #15
>> Nov 9 23:30:01 sbc01 /usr/local/sbin/kamailio[12276]: ERROR: htable [ht_db.c:234]: ht_db_load_table(): Error while fetching result
>> Nov 9 23:30:01 sbc01 /usr/local/sbin/kamailio[12276]: : <core> [mem/q_malloc.c:468]: qm_free(): BUG: qm_free: freeing already freed pointer (0x7f5ebda8ae18), called from db_mysql: km_dbase.c: db_mysql_free_result(305), first free db_mysql: km_dbase.c: db_mysql_free_result(305) - aborting
>> Nov 9 23:30:01 sbc01 /usr/local/sbin/kamailio[12281]: CRITICAL: <core> [pass_fd.c:275]: receive_fd(): EOF on 16
>> Nov 9 23:30:01 sbc01 /usr/local/sbin/kamailio[12268]: ALERT: <core> [main.c:739]: handle_sigs(): child process 12276 exited by a signal 6
>> Nov 9 23:30:01 sbc01 /usr/local/sbin/kamailio[12268]: ALERT: <core> [main.c:742]: handle_sigs(): core was not generated
>> Nov 9 23:30:01 sbc01 /usr/local/sbin/kamailio[12268]: INFO: <core> [main.c:754]: handle_sigs(): terminating due to SIGCHLD
>>
>> The thing is we heavily use mysql module, but only to update the in-memory tables by kamcmd. Each N minutes a special script updates the trusted,address and htable executing kamcmd. Kamailio (and kamcmd as well) talks only with localhost mysql server.
>> What I saw when encountered that issue on a live machine is that issue happens only with one of child processes, any other are ok.
>> Interesting thing is that happens at the same time with machines on the same «set», I mean that issue happened simultaneously with two our test machines which actually didn’t have any load on them.
>> The common thing between those machines is that they are in same subnet and local mysql databases are filled by scripts which query same external db.
>> I can’t confirm if there were or there weren’t any networking issues at that time with those machines, but as soon as kamcmd queries localhost that shouldn’t be the source of the issue.
>>
>> So my questions are:
>> 1. Has anyone encountered such thing?
>> 2. Maybe the issue is already localized so it has sense to update? We actually use that on production (pls don’t throw too much rocks at me), so maintenance should be properly planned and I must be sure update won’t break anything.
>> 3. If update is proposed - how to do it? I mean - follow the guide https://www.kamailio.org/wiki/install/devel/git or there are some other tips? I suppose in ideal world I don’t even stop the binary, only restart after make all && make install are done, as everything is in-memory. Am I correct?
>> 4. When can we expect stable 5.0 version? (at least tell if it’s months/years)
>>
>> Thanks in advance!
>> _______________________________________________
>> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
>> sr-users at lists.sip-router.org
>> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
>
> --
> Daniel-Constantin Mierla
> http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
> Kamailio Advanced Training, Berlin, Nov 28-30, 2016 - http://www.asipto.com
>
>
> _______________________________________________
> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
> sr-users at lists.sip-router.org
> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
More information about the sr-users
mailing list