On 01.03.20 21:53, Juha Heinanen wrote:
John Petrini writes:
The times we've seen this is when transactions are waiting on something so they pile up consuming shared memory. Do you have any database lookups or calls out to external services or scripts?
John,
Thanks for your reply. Yes, there are MySQL operations both during request/reply processing and also accounting to db. I would assume that any problems with those should show up in /var/log/mysql/error.log.
Failed queries should be reported to error log by kamailio as well.
On the other hand, Kamailio has some global parameters that enable logging messages for slow db queries. Maybe you can enable them to see if any query takes very long.
Long timeouts can also contribute if something stops responding because transactions are waiting for a long timeout to expire.
There is quite long "fr_inv_timer" value, that could cause problems is lots of INVITEs are ringing at the same time.
It's typically a balance between setting reasonable timeouts and allocating enough shm. In addition we implemented some watcher scripts that monitor shm and will set gflags to disable non-critical external calls beyond a certain threshold as well as send us an alert.
I also created an external script to watch shm memory usage. I would have liked K to watch itself, but could not find pseudo variables corresponding to core.shmmem entries. Do they exist?
The $stat(name) can be used for any internal statistic. I think it takes only the name of statistic, not the group, so something like $stat(free_size).
Check also statsc module, maybe it is is useful to have the evolution of some stats values in the recent past.
Cheers, Daniel