This morning I had the unpleasant realisation that kamailio (4.0.3) crashed twice while I was running an alter statement on the acc table of a live instance.
db_mysql [km_dbase.c:122]: db_mysql_submit_query(): driver error on q uery: Too many connections <core> [db_query.c:132]: db_do_query_internal(): error while submitti ng query auth_db [authorize.c:175]: get_ha1(): failed to query database or alias_db [alookup.c:111]: alias_db_lookup(): failed to query database db_mysql [km_dbase.c:122]: db_mysql_submit_query(): driver error on query: Too many connections
just before (no core): segfault at 18 ip 00007f0d6b287a83 sp 00007fffc4467550 error 4 in usrloc.so[7f0d6b267000+27000]
So instead of just discarding the acc inserts, these pile up resulting in consuming max connections to the database and eventually segfaulting.
I know (now) that I shouldn't do this on a production server, but I'd rather have the database accounting discarded than a machine crashing (can be reconstructed from syslog).
My question: is this "normal" behavior? Can this be avoided somehow (other than ensuring 100% availability of the sql server).
Would be good to get the core and backtrace, otherwise is hard to track what went wrong. db_mysql discards the query if it is not successful after three retries. The segfault seem to refer to usrloc, not acc. The issue could be combined with libmysqlclient behaviour, iirc kamailio will try to open a new connection if old one is dead.
You could eventually switch to use insert delayed with acc module or even globally via db_mysql - there are module parameters for it.
Cheers, Daniel
On 12/4/13 3:30 PM, Daniel Tryba wrote:
This morning I had the unpleasant realisation that kamailio (4.0.3) crashed twice while I was running an alter statement on the acc table of a live instance.
db_mysql [km_dbase.c:122]: db_mysql_submit_query(): driver error on q uery: Too many connections <core> [db_query.c:132]: db_do_query_internal(): error while submitti ng query auth_db [authorize.c:175]: get_ha1(): failed to query database or alias_db [alookup.c:111]: alias_db_lookup(): failed to query database db_mysql [km_dbase.c:122]: db_mysql_submit_query(): driver error on query: Too many connections
just before (no core): segfault at 18 ip 00007f0d6b287a83 sp 00007fffc4467550 error 4 in usrloc.so[7f0d6b267000+27000]
So instead of just discarding the acc inserts, these pile up resulting in consuming max connections to the database and eventually segfaulting.
I know (now) that I shouldn't do this on a production server, but I'd rather have the database accounting discarded than a machine crashing (can be reconstructed from syslog).
My question: is this "normal" behavior? Can this be avoided somehow (other than ensuring 100% availability of the sql server).
On Wednesday 04 December 2013 21:02:20 Daniel-Constantin Mierla wrote:
Would be good to get the core and backtrace, otherwise is hard to track what went wrong. db_mysql discards the query if it is not successful after three retries. The segfault seem to refer to usrloc, not acc. The issue could be combined with libmysqlclient behaviour, iirc kamailio will try to open a new connection if old one is dead.
There are sql lookups per INVITE/REGISTER on usr_preferences and subscriber, which fail due to acc hogging all connections. I'll try to reproduce on a test server.
You could eventually switch to use insert delayed with acc module or even globally via db_mysql - there are module parameters for it.
Missed this option, thanks for pointing that out. Sounds like what is needed in these kind of scenarios.
-
Daniel Tryba -
Daniel-Constantin Mierla