[SR-Users] Best practices for troubleshooting deadlocks?

[Alex Balashov]

We just encountered another one of these famed deadlocks. Any 
suggestions for how to analyse them beyond what I've already trotted out 
here?

On 09/14/2015 05:47 PM, Alex Balashov wrote:

> Hello,
>
> Very occasionally, we encounter what appear to be deadlocks in all UDP
> receiver threads. All Kamailio processes are running, but no SIP
> messages are being processed.
>
> On one of our high-volume installation, this happens extremely
> infrequently -- maybe once every month or two. On these occasions, the
> operator restarts the proxy before we get a chance to go in and figure
> out what's going on.
>
> So, I'm trying to provide the operator with a procedure to execute prior
> to restarting the proxy on these occasions, so that we can see a
> snapshot of where the receiver threads are stuck. As far as I can tell,
> unless Kamailio itself segfaults, there's no specific PID that one can
> attach GDB to in order to get an overhead snapshot of all the child
> processes.
>
> Here's what I came up with:
>
> ---------------------------------------------
> #!/bin/bash
>
> kamcmd -s /tmp/kamailio_ctl ps > thread_log.txt
> echo >> thread_log.txt
>
> while read PID;
> do
>      gdb --pid=$PID<<EOF>>thread_log.txt
> set print elements 0
> thread apply all bt full
> generate-core-file
> detach
> EOF
> done < <(kamcmd -s /tmp/kamailio_ctl ps | grep 'udp receiver' | awk
> '{print $1}')
> ---------------------------------------------
>
> As far as I can tell, this should give me the most ample visibility into
> the state of the threads, with further core dumps to inspect if
> follow-up is needed. Hopefully this will result in some fixes back to
> the project.
>
> However, if there are any other suggestions for information to grab in
> such a scenario, I'm all ears.
>
> Thanks in advance!
>
> -- Alex
>


-- 
Alex Balashov | Principal | Evariste Systems LLC
303 Perimeter Center North, Suite 300
Atlanta, GA 30346
United States

Tel: +1-800-250-5920 (toll-free) / +1-678-954-0671 (direct)
Web: http://www.evaristesys.com/, http://www.csrpswitch.com/