John Petrini writes:
The times we've seen this is when transactions are waiting on something so they pile up consuming shared memory. Do you have any database lookups or calls out to external services or scripts?
John,
Thanks for your reply. Yes, there are MySQL operations both during request/reply processing and also accounting to db. I would assume that any problems with those should show up in /var/log/mysql/error.log.
Long timeouts can also contribute if something stops responding because transactions are waiting for a long timeout to expire.
There is quite long "fr_inv_timer" value, that could cause problems is lots of INVITEs are ringing at the same time.
It's typically a balance between setting reasonable timeouts and allocating enough shm. In addition we implemented some watcher scripts that monitor shm and will set gflags to disable non-critical external calls beyond a certain threshold as well as send us an alert.
I also created an external script to watch shm memory usage. I would have liked K to watch itself, but could not find pseudo variables corresponding to core.shmmem entries. Do they exist?
-- Juha