[SR-Users] out of shm without any visible reason

[Daniel-Constantin Mierla]

On 01.03.20 21:53, Juha Heinanen wrote:
> John Petrini writes:
>
>> The times we've seen this is when transactions are waiting on something so
>> they pile up consuming shared memory. Do you have any database lookups or
>> calls out to external services or scripts?
> John,
>
> Thanks for your reply.  Yes, there are MySQL operations both during
> request/reply processing and also accounting to db.  I would assume that
> any problems with those should show up in /var/log/mysql/error.log.

Failed queries should be reported to error log by kamailio as well.

On the other hand, Kamailio has some global parameters that enable
logging messages for slow db queries. Maybe you can enable them to see
if any query takes very long.

>
>> Long timeouts can also contribute if something stops responding because
>> transactions are waiting for a long timeout to expire.
> There is quite long "fr_inv_timer" value, that could cause problems is
> lots of INVITEs are ringing at the same time.
>
>> It's typically a balance between setting reasonable timeouts and allocating
>> enough shm. In addition we implemented some watcher scripts that monitor
>> shm and will set gflags to disable non-critical external calls beyond a
>> certain threshold as well as send us an alert.
> I also created an external script to watch shm memory usage.  I would
> have liked K to watch itself, but could not find pseudo variables
> corresponding to core.shmmem entries.  Do they exist?

The $stat(name) can be used for any internal statistic. I think it takes
only the name of statistic, not the group, so something like
$stat(free_size).

Check also statsc module, maybe it is is useful to have the evolution of
some stats values in the recent past.

Cheers,
Daniel

-- 
Daniel-Constantin Mierla -- www.asipto.com
www.twitter.com/miconda -- www.linkedin.com/in/miconda
Kamailio Advanced Training - March 9-11, 2020, Berlin - www.asipto.com
Kamailio World Conference - April 27-29, 2020, in Berlin -- www.kamailioworld.com