[Kamailio-Users] R: R: Fast lock loop with arm
Daniel-Constantin Mierla
miconda at gmail.com
Mon Mar 29 11:00:48 CEST 2010
Hello,
On 3/29/10 8:53 AM, Zappasodi Daniele wrote:
>
> Hello,
>
> I have tried a lot of times with different processes, but backtrace
> shows always only this.
>
this is really strange. do you use mi_fifo? if yes, when you start
openser run 'openserctl ps'
Spot a udp worker process, the mi fifo process, the main process and the
timer process. when the locking happens, attach to each of them and get
the backtrace.
Thanks,
Daniel
> Daniele
>
> -----Messaggio originale-----
> *Da:* Daniel-Constantin Mierla [mailto:miconda at gmail.com]
> *Inviato:* venerdì 26 marzo 2010 19.40
> *A:* Zappasodi Daniele
> *Cc:* users at lists.kamailio.org; sr-dev
> *Oggetto:* Re: R: [Kamailio-Users] Fast lock loop with arm
>
> Hello,
>
> On 3/26/10 4:13 PM, Zappasodi Daniele wrote:
>>
>> Hello,
>>
>> this is what I get with gdb:
>>
>> (gdb) bt full
>>
>> #0 0x40137e54 in sched_yield () from /usr/local/lib2/libc.so.6
>>
>> No symbol table info available.
>>
>> I don't think that it can help, but I am not able to load the
>> symbol table for openser on the server.
>>
> hmm, strange. Did you try with many processes? Sam result in the
> backtrace?
>
> Cheers,
> Daniel
>
>> thanks,
>>
>> Daniele
>>
>> -----Messaggio originale-----
>> *Da:* Daniel-Constantin Mierla [mailto:miconda at gmail.com]
>> *Inviato:* mercoledì 24 marzo 2010 16.47
>> *A:* Zappasodi Daniele
>> *Cc:* users at lists.kamailio.org; sr-dev
>> *Oggetto:* Re: [Kamailio-Users] Fast lock loop with arm
>>
>> Hello,
>>
>> what version of kamailio do you use?
>>
>> Can you grab a backtrace with gdb once you get this high cpu
>> usage? Spot one of the processes, take the pid and do:
>>
>> gdb /path/to/kamailio _pid_
>>
>> Then:
>>
>> bt
>>
>> You should see the bactrace of executed function getting to
>> deadloc. Make sure you get at least one SIP worker process
>> backtrace (do kamctl ps to see the type of kamailio process).
>>
>> Cheers,
>> Daniel
>>
>> On 3/24/10 4:29 PM, Zappasodi Daniele wrote:
>>>
>>> Hi,
>>> I have a big problem with fast lock mutex on arm CPU:
>>> sometimes I find one or more children that go in an infinite
>>> loop, in the while loop of get_lock function.
>>> They remain in Run for long time (sometimes hours) and
>>> openser keeps 100% CPU.
>>>
>>> Now I have changed the functions get_lock and tsl in order
>>> to obtain more info and
>>> I observe that (if and) when the process finish the cycle,
>>> it have done a big amount of cycles.
>>>
>>> This the log with my added info:
>>> Mar 26 20:29:08 SAM-IP openser[1645]: NOTICE:tm:_lock: ret 1
>>> (cycle)
>>> Mar 26 20:29:51 SAM-IP openser[1645]: NOTICE:tm:_lock: ret 1
>>> (cycle)
>>> Mar 26 20:29:55 SAM-IP openser[1645]: NOTICE:tm:_lock: ret 1
>>> (cycle)
>>>
>>> Mar 26 20:29:55 SAM-IP openser[1645]:
>>> NOTICE:tm:t_retransmit_reply: MYTM lock
>>> [process in loop]
>>> Mar 26 20:33:46 SAM-IP openser[1645]: NOTICE:tm:_lock: ret
>>> 59374917 (cycle)
>>> [after 4 minutes and 59374917 cycles, this is an example
>>> with a "short" loop]
>>> Mar 26 20:33:46 SAM-IP openser[1645]:
>>> NOTICE:tm:t_retransmit_reply: MYTM lock done
>>>
>>> I'm not able to recognize the call flow that generates the
>>> loop (I know only that it always happens when
>>> t_retransmit_reply calls LOCK_REPLIES),
>>>
>>> but I urgently need a work around to escape from the loop.
>>>
>>> This is the get_lock function with my counter j:
>>>
>>> inline static int get_lock(fl_lock_t* lock)
>>> {
>>> #ifdef ADAPTIVE_WAIT
>>> int i=ADAPTIVE_WAIT_LOOPS;
>>> int j=1; /***** my
>>> change ****/
>>> #endif
>>>
>>> while(tsl(lock)){
>>> #ifdef BUSY_WAIT
>>> #elif defined ADAPTIVE_WAIT
>>> j++; /***** my change ****/
>>> if (i>0) i--;
>>> else sched_yield();
>>> #else
>>> sched_yield();
>>> #endif
>>> }
>>> return(j);
>>> }
>>>
>>> Can I break the lock when my counter reaches a big value?
>>> Should I call the unlock function after the break?
>>> which value can be considered too big?
>>>
>>> Thanks,
>>> Daniele
>>>
>>>
>>>
>>> **********************************************************************
>>> The information in this message is confidential and may be
>>> legally privileged. It is intended solely for the addressee.
>>> Access to this message by anyone else is unauthorized. If
>>> you are not the intended recipient, any disclosure, copying,
>>> or distribution of the message, or any action or omission
>>> taken by you in reliance on it, is prohibited and may be
>>> unlawful. Please immediately contact the sender if you have
>>> received this message in error.
>>> **********************************************************************
>>>
>>>
>>> _______________________________________________
>>> Kamailio (OpenSER) - Users mailing list
>>> Users at lists.kamailio.org
>>> http://lists.kamailio.org/cgi-bin/mailman/listinfo/users
>>> http://lists.openser-project.org/cgi-bin/mailman/listinfo/users
>>
>> --
>> Daniel-Constantin Mierla * http://www.asipto.com/
>>
>> **********************************************************************
>> The information in this message is confidential and may be
>> legally privileged. It is intended solely for the addressee.
>> Access to this message by anyone else is unauthorized. If you are
>> not the intended recipient, any disclosure, copying, or
>> distribution of the message, or any action or omission taken by
>> you in reliance on it, is prohibited and may be unlawful. Please
>> immediately contact the sender if you have received this message
>> in error.
>> **********************************************************************
>>
>
> --
> Daniel-Constantin Mierla * http://www.asipto.com/
>
> **********************************************************************
> The information in this message is confidential and may be legally
> privileged. It is intended solely for the addressee. Access to this
> message by anyone else is unauthorized. If you are not the intended
> recipient, any disclosure, copying, or distribution of the message, or
> any action or omission taken by you in reliance on it, is prohibited
> and may be unlawful. Please immediately contact the sender if you have
> received this message in error.
> **********************************************************************
>
>
> _______________________________________________
> Kamailio (OpenSER) - Users mailing list
> Users at lists.kamailio.org
> http://lists.kamailio.org/cgi-bin/mailman/listinfo/users
> http://lists.openser-project.org/cgi-bin/mailman/listinfo/users
--
Daniel-Constantin Mierla * http://www.asipto.com/ *
http://twitter.com/miconda *
http://www.linkedin.com/in/danielconstantinmierla
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.kamailio.org/pipermail/users/attachments/20100329/69304efe/attachment.htm>
More information about the Users
mailing list