<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
<title></title>
</head>
<body bgcolor="#ffffff" text="#000000">
Hello,<br>
<br>
On 3/29/10 8:53 AM, Zappasodi Daniele wrote:
<blockquote
cite="mid:14A9059A37EF9A45BC2520DE64C30F60075258@slttex002.seltatel.local"
type="cite">
<meta http-equiv="Content-Type"
content="text/html; charset=ISO-8859-1">
<title></title>
<meta content="MSHTML 6.00.6000.16981" name="GENERATOR">
<div>
<p class="MsoNormal" style="margin: 0cm 0cm 0pt;"><span style=""
lang="EN-GB">Hello,<o:p></o:p></span></p>
<p class="MsoNormal" style="margin: 0cm 0cm 0pt;"><span style=""
lang="EN-GB">I have tried a lot of times with different processes, but
backtrace shows always only this.</span></p>
</div>
</blockquote>
<br>
this is really strange. do you use mi_fifo? if yes, when you start
openser run 'openserctl ps'<br>
<br>
Spot a udp worker process, the mi fifo process, the main process and
the timer process. when the locking happens, attach to each of them and
get the backtrace. <br>
<br>
Thanks,<br>
Daniel<br>
<br>
<blockquote
cite="mid:14A9059A37EF9A45BC2520DE64C30F60075258@slttex002.seltatel.local"
type="cite">
<div>
<p class="MsoNormal" style="margin: 0cm 0cm 0pt;"><span style=""
lang="EN-GB"><o:p></o:p></span></p>
<p class="MsoNormal" style="margin: 0cm 0cm 0pt;"><span style=""
lang="EN-GB"> <o:p></o:p></span></p>
<span style="font-size: 12pt; font-family: 'Times New Roman';"
lang="EN-GB">Daniele</span></div>
<blockquote dir="ltr" style="margin-right: 0px;">
<div class="OutlookMessageHeader" dir="ltr" align="left"><font
face="Tahoma" size="2">-----Messaggio originale-----<br>
<b>Da:</b> Daniel-Constantin Mierla [<a class="moz-txt-link-freetext" href="mailto:miconda@gmail.com">mailto:miconda@gmail.com</a>]<br>
<b>Inviato:</b> venerdì 26 marzo 2010 19.40<br>
<b>A:</b> Zappasodi Daniele<br>
<b>Cc:</b> <a class="moz-txt-link-abbreviated" href="mailto:users@lists.kamailio.org">users@lists.kamailio.org</a>; sr-dev<br>
<b>Oggetto:</b> Re: R: [Kamailio-Users] Fast lock loop with arm<br>
<br>
</font></div>
Hello,<br>
<br>
On 3/26/10 4:13 PM, Zappasodi Daniele wrote:
<blockquote
cite="mid:14A9059A37EF9A45BC2520DE64C30F60075256@slttex002.seltatel.local"
type="cite">
<meta content="MSHTML 6.00.6000.16981" name="GENERATOR">
<div>
<p class="MsoNormal" style="margin: 0cm 0cm 0pt;"><span
lang="EN-GB">Hello,<o:p></o:p></span></p>
<p class="MsoNormal" style="margin: 0cm 0cm 0pt;"><span
lang="EN-GB">this is what I get with gdb:<o:p></o:p></span></p>
<p class="MsoNormal" style="margin: 0cm 0cm 0pt;"><span
lang="EN-GB"><o:p></o:p></span></p>
<p class="MsoNormal" style="margin: 0cm 0cm 0pt;"><span
lang="EN-GB">(gdb) bt full<o:p></o:p></span></p>
<p class="MsoNormal" style="margin: 0cm 0cm 0pt;"><span
lang="EN-GB">#0<span> </span>0x40137e54 in sched_yield () from
/usr/local/lib2/libc.so.6<o:p></o:p></span></p>
<p class="MsoNormal" style="margin: 0cm 0cm 0pt;"><span
lang="EN-GB">No symbol table info available.<o:p></o:p></span></p>
<p class="MsoNormal" style="margin: 0cm 0cm 0pt;"><span
lang="EN-GB"><o:p></o:p></span></p>
<p class="MsoNormal" style="margin: 0cm 0cm 0pt;"><span
lang="EN-GB">I don’t think that it can help, but I am not able to load
the symbol table for openser<span class="292020015-26032010"> on the
server</span>.</span></p>
</div>
</blockquote>
hmm, strange. Did you try with many processes? Sam result in the
backtrace?<br>
<br>
Cheers,<br>
Daniel<br>
<br>
<blockquote
cite="mid:14A9059A37EF9A45BC2520DE64C30F60075256@slttex002.seltatel.local"
type="cite">
<div>
<p class="MsoNormal" style="margin: 0cm 0cm 0pt;"><span
lang="EN-GB"><o:p></o:p></span></p>
<p class="MsoNormal" style="margin: 0cm 0cm 0pt;"><span
lang="EN-GB"><span class="292020015-26032010"></span></span><span
lang="EN-GB"></span> </p>
<p class="MsoNormal" style="margin: 0cm 0cm 0pt;"><span
lang="EN-GB"><span class="292020015-26032010"><font color="#0000ff"
face="Arial" size="2">thanks,</font></span></span></p>
<p class="MsoNormal" style="margin: 0cm 0cm 0pt;"><span
lang="EN-GB"><o:p><span class="292020015-26032010"><font
color="#0000ff" face="Arial" size="2">Daniele</font></span></o:p></span></p>
</div>
<blockquote dir="ltr" style="margin-right: 0px;">
<div class="OutlookMessageHeader" dir="ltr" align="left"><font
face="Tahoma" size="2">-----Messaggio originale-----<br>
<b>Da:</b> Daniel-Constantin Mierla [<a moz-do-not-send="true"
class="moz-txt-link-freetext" href="mailto:miconda@gmail.com">mailto:miconda@gmail.com</a>]<br>
<b>Inviato:</b> mercoledì 24 marzo 2010 16.47<br>
<b>A:</b> Zappasodi Daniele<br>
<b>Cc:</b> <a moz-do-not-send="true"
class="moz-txt-link-abbreviated" href="mailto:users@lists.kamailio.org">users@lists.kamailio.org</a>;
sr-dev<br>
<b>Oggetto:</b> Re: [Kamailio-Users] Fast lock loop with arm<br>
<br>
</font></div>
Hello,<br>
<br>
what version of kamailio do you use?<br>
<br>
Can you grab a backtrace with gdb once you get this high cpu usage?
Spot one of the processes, take the pid and do:<br>
<br>
gdb /path/to/kamailio _pid_<br>
<br>
Then:<br>
<br>
bt<br>
<br>
You should see the bactrace of executed function getting to deadloc.
Make sure you get at least one SIP worker process backtrace (do kamctl
ps to see the type of kamailio process).<br>
<br>
Cheers,<br>
Daniel<br>
<br>
On 3/24/10 4:29 PM, Zappasodi Daniele wrote:
<blockquote
cite="mid:14A9059A37EF9A45BC2520DE64C30F60075252@slttex002.seltatel.local"
type="cite">
<meta content="MS Exchange Server version 6.5.7654.12"
name="Generator">
<!-- Converted from text/rtf format -->
<p><span lang="en-gb"><font face="Times New Roman">Hi,</font></span>
<br>
<span lang="en-gb"><font face="Times New Roman">I have a big
problem with fast lock mutex on arm CPU: </font></span><br>
<span lang="en-gb"><font face="Times New Roman">sometimes I
find one or more children that go in an infinite loop, in the while
loop of get_lock function. </font></span><br>
<span lang="en-gb"><font face="Times New Roman">They remain
in Run for long time (sometimes hours) and openser keeps 100% CPU.</font></span>
</p>
<p><span lang="en-gb"><font face="Times New Roman">Now I have
changed the functions get_lock and tsl in order to obtain more info and
</font></span><br>
<span lang="en-gb"><font face="Times New Roman">I observe
that (if and) when the process finish the cycle, it have done a big
amount of cycles. </font></span></p>
<p><span lang="en-gb"><font face="Times New Roman">This the
log with my added info:</font></span> <br>
<span lang="en-gb"><font face="Times New Roman">Mar 26
20:29:08 SAM-IP openser[1645]: NOTICE:tm:_lock: ret 1 (cycle)</font></span>
<br>
<span lang="en-gb"><font face="Times New Roman">Mar 26
20:29:51 SAM-IP openser[1645]: NOTICE:tm:_lock: ret 1 (cycle)</font></span>
<br>
<span lang="en-gb"><font face="Times New Roman">Mar 26
20:29:55 SAM-IP openser[1645]: NOTICE:tm:_lock: ret 1 (cycle)</font></span>
</p>
<p><span lang="en-gb"><font face="Times New Roman">Mar 26
20:29:55 SAM-IP openser[1645]: NOTICE:tm:t_retransmit_reply: MYTM lock </font></span><br>
<span lang="en-gb"><font face="Times New Roman">[process in
loop]</font></span> <br>
<span lang="en-gb"><font face="Times New Roman">Mar 26
20:33:46 SAM-IP openser[1645]: NOTICE:tm:_lock: ret 59374917 (cycle)</font></span>
<br>
<span lang="en-gb"><font face="Times New Roman">[after 4
minutes and 59374917 cycles, this is an example with a "short" loop]</font></span>
<br>
<span lang="en-gb"><font face="Times New Roman">Mar 26
20:33:46 SAM-IP openser[1645]: NOTICE:tm:t_retransmit_reply: MYTM lock
done </font></span></p>
<p><span lang="en-gb"><font face="Times New Roman">I’m not
able to recognize the call flow that generates the loop (I know only
that it always happens when t_retransmit_reply calls LOCK_REPLIES), </font></span></p>
<p><span lang="en-gb"><font face="Times New Roman">but I
urgently need a work around to escape from the loop.</font></span> </p>
<p><span lang="en-gb"><font face="Times New Roman">This is
the get_lock function with my counter j:</font></span> </p>
<p><span lang="en-gb"><font face="Times New Roman">inline
static int get_lock(fl_lock_t* lock)</font></span> <br>
<span lang="en-gb"><font face="Times New Roman">{</font></span>
<br>
<span lang="en-gb"><font face="Times New Roman">#ifdef
ADAPTIVE_WAIT</font></span> <br>
<span lang="en-gb"> <font face="Times New Roman">int
i=ADAPTIVE_WAIT_LOOPS;</font></span> <br>
<span lang="en-gb"> <font face="Times New Roman">int
j=1; /***** my change ****/</font></span>
<br>
<span lang="en-gb"><font face="Times New Roman">#endif</font></span>
<br>
<span lang="en-gb"> </span><br>
<span lang="en-gb"> <font face="Times New Roman">while(tsl(lock)){</font></span>
<br>
<span lang="en-gb"><font face="Times New Roman">#ifdef
BUSY_WAIT</font></span> <br>
<span lang="en-gb"><font face="Times New Roman">#elif defined
ADAPTIVE_WAIT</font></span> <br>
<span lang="en-gb"> <font
face="Times New Roman">j++; /***** my
change ****/</font></span> <br>
<span lang="en-gb"> <font
face="Times New Roman">if (i>0) i--;</font></span> <br>
<span lang="en-gb"> <font
face="Times New Roman">else sched_yield();</font></span> <br>
<span lang="en-gb"><font face="Times New Roman">#else</font></span>
<br>
<span lang="en-gb"> <font
face="Times New Roman">sched_yield();</font></span> <br>
<span lang="en-gb"><font face="Times New Roman">#endif</font></span>
<br>
<span lang="en-gb"> <font face="Times New Roman">}</font></span>
<br>
<span lang="en-gb"> <font face="Times New Roman">return(j);</font></span>
<br>
<span lang="en-gb"><font face="Times New Roman">}</font></span>
</p>
<p><span lang="en-gb"><font face="Times New Roman">Can I
break the lock when my counter reaches a big value? </font></span><br>
<span lang="en-gb"><font face="Times New Roman">Should I call
the unlock function after the break?</font></span> <br>
<span lang="en-gb"><font face="Times New Roman">which value
can be considered too big?</font></span> </p>
<p><span lang="en-gb"><font face="Times New Roman">Thanks,</font></span>
<br>
<span lang="en-gb"><font face="Times New Roman">Daniele </font></span></p>
<br>
<br>
<p>**********************************************************************
The information in this message is confidential and may be legally
privileged. It is intended solely for the addressee. Access to this
message by anyone else is unauthorized. If you are not the intended
recipient, any disclosure, copying, or distribution of the message, or
any action or omission taken by you in reliance on it, is prohibited
and may be unlawful. Please immediately contact the sender if you have
received this message in error.
**********************************************************************</p>
<pre wrap=""><fieldset class="mimeAttachmentHeader"></fieldset>
_______________________________________________
Kamailio (OpenSER) - Users mailing list
<a class="moz-txt-link-abbreviated"
href="mailto:Users@lists.kamailio.org" moz-do-not-send="true">Users@lists.kamailio.org</a>
<a class="moz-txt-link-freetext"
href="http://lists.kamailio.org/cgi-bin/mailman/listinfo/users"
moz-do-not-send="true">http://lists.kamailio.org/cgi-bin/mailman/listinfo/users</a>
<a class="moz-txt-link-freetext"
href="http://lists.openser-project.org/cgi-bin/mailman/listinfo/users"
moz-do-not-send="true">http://lists.openser-project.org/cgi-bin/mailman/listinfo/users</a></pre>
</blockquote>
<br>
<div class="moz-signature">-- <br>
Daniel-Constantin Mierla * <a class="moz-txt-link-freetext"
href="http://www.asipto.com/" moz-do-not-send="true">http://www.asipto.com/</a>
</div>
</blockquote>
<p>**********************************************************************
The information in this message is confidential and may be legally
privileged. It is intended solely for the addressee. Access to this
message by anyone else is unauthorized. If you are not the intended
recipient, any disclosure, copying, or distribution of the message, or
any action or omission taken by you in reliance on it, is prohibited
and may be unlawful. Please immediately contact the sender if you have
received this message in error.
**********************************************************************</p>
</blockquote>
<br>
<div class="moz-signature">-- <br>
Daniel-Constantin Mierla * <a moz-do-not-send="true"
class="moz-txt-link-freetext" href="http://www.asipto.com/">http://www.asipto.com/</a>
</div>
</blockquote>
<p>**********************************************************************
The
information in this message is confidential and may be legally
privileged. It is intended solely for the addressee. Access to this
message
by anyone else is unauthorized. If you are not the intended recipient,
any
disclosure, copying, or distribution of the message, or any action or
omission taken by you in reliance on it, is prohibited and may be
unlawful.
Please immediately contact the sender if you have received this message
in error.
**********************************************************************</p>
<pre wrap="">
<fieldset class="mimeAttachmentHeader"></fieldset>
_______________________________________________
Kamailio (OpenSER) - Users mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Users@lists.kamailio.org">Users@lists.kamailio.org</a>
<a class="moz-txt-link-freetext" href="http://lists.kamailio.org/cgi-bin/mailman/listinfo/users">http://lists.kamailio.org/cgi-bin/mailman/listinfo/users</a>
<a class="moz-txt-link-freetext" href="http://lists.openser-project.org/cgi-bin/mailman/listinfo/users">http://lists.openser-project.org/cgi-bin/mailman/listinfo/users</a></pre>
</blockquote>
<br>
<div class="moz-signature">-- <br>
Daniel-Constantin Mierla
* <a class="moz-txt-link-freetext" href="http://www.asipto.com/">http://www.asipto.com/</a> * <a class="moz-txt-link-freetext" href="http://twitter.com/miconda">http://twitter.com/miconda</a>
* <a class="moz-txt-link-freetext" href="http://www.linkedin.com/in/danielconstantinmierla">http://www.linkedin.com/in/danielconstantinmierla</a>
</div>
</body>
</html>