15 Jan
2013
15 Jan
'13
11:51 a.m.
Hi,
We run Kamailio 3.2.3 (FLAVOUR=ser) on an embedded ARM platform in near-
default configuration.
In a duration test, we observe that at a certain moment Kamailio seems to
start ignoring all (re-)register messages, and eventually expires the
existing registrations.
We have not been able to reproduce the issue using debug-level logging
(-dddd). Info-level logging (-ddd) does reproduce the error, but does not
produce any error messages.
Configuration: ser-basic.cfg, with the following changes:
port=5060
alias=testnet
Command line:
ser -m 4 -f /etc/ser/ser-basic.cfg -n3 -l udp:eth0
Load:
Bursts of 8 (Re-)REGISTER messages that are repeated every 85 seconds. The
specified expiration time is 120 seconds.
Bursts of 9 (Re-)INVITE messages that are repeated every 45 seconds.
Note:
With this load it takes roughly 1 hour for the error to occur. Using a more
standard 3600 second expiration time Kamailio still stalls, it just takes
longer.
Does anyone have any idea how to tackle this issue?
Kind regards,
Michiel Veldkamp
15 Jan
15 Jan
10:04 p.m.
New subject: Kamailio stability/timing problem w.r.t. registrations?
Hello,
can you add an xlog() at the start of the main route block and log a
message for any request received? Keep the debug level low. It should
show if the sip request gets to application level, it may happen to be
dropped from the network interface or kernel.
You can put another xlog before the save() function to see if
registration requests are getting there. I am not familiar with ser
flavour of registrar and usrloc modules. You can switch to kamailio
flavour modules and see if reproduces. I haven't seen such issues with k
flavour.
Cheers,
Daniel
On 1/15/13 11:51 AM, Sotas Development wrote:
Hi,
We run Kamailio 3.2.3 (FLAVOUR=ser) on an embedded ARM platform in near-
default configuration.
In a duration test, we observe that at a certain moment Kamailio seems to
start ignoring all (re-)register messages, and eventually expires the
existing registrations.
We have not been able to reproduce the issue using debug-level logging
(-dddd). Info-level logging (-ddd) does reproduce the error, but does not
produce any error messages.
Configuration: ser-basic.cfg, with the following changes:
port=5060
alias=testnet
Command line:
ser -m 4 -f /etc/ser/ser-basic.cfg -n3 -l udp:eth0
Load:
Bursts of 8 (Re-)REGISTER messages that are repeated every 85 seconds. The
specified expiration time is 120 seconds.
Bursts of 9 (Re-)INVITE messages that are repeated every 45 seconds.
Note:
With this load it takes roughly 1 hour for the error to occur. Using a
more
standard 3600 second expiration time Kamailio still stalls, it just takes
longer.
Does anyone have any idea how to tackle this issue?
Kind regards,
Michiel Veldkamp
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users(a)lists.sip-router.org
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
--
Daniel-Constantin Mierla - http://www.asipto.com
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Kamailio World Conference, April 16-17, 2013, Berlin
- http://conference.kamailio.com -
17 Jan
17 Jan
2:41 p.m.
New subject: Kamailio stability/timing problem w.r.t. registrations?
Daniel,
can you add an xlog() at the start of the main route
block and log a
message for any
request received? [...] You can put another xlog
before the save()
function to see if
registration requests are getting there. [...] You can
switch to kamailio
flavour modules
and see if reproduces.
Thanks for the help! We switched to kamailio flavour, added some xlog
messages and managed to reproduce. See below for some logging.
To our surprise, the last message handled was an INVITE. We'll add more
logging to see whether it is handled successfully or gets stuck somewhere.
Those numbers at the start of the log message, are these child IDs? At
first they alternate, but apparently child 1 and 2 stop running early on
(each with an INVITE as last message).
Regards,
Michiel Veldkamp
0(304) WARNING: <core> [socket_info.c:1392]: WARNING: fix_hostname: could
not rev. resolve 192.168.10.1
0(304) INFO: <core> [tcp_main.c:4832]: init_tcp: using epoll_lt as the io
watch method (auto detected)
0(306) INFO: usrloc [hslot.c:53]: locks array size 512
0(306) INFO: <core> [udp_server.c:179]: INFO: udp_init: SO_RCVBUF is
initially 108544
0(306) INFO: <core> [udp_server.c:230]: INFO: udp_init: SO_RCVBUF is
finally 217088
7(315) INFO: ctl [io_listener.c:225]: io_listen_loop: using epoll_lt io
watch method (config)
1(309) ERROR: <script>: request_route start -- method=REGISTER
2(310) ERROR: <script>: request_route start -- method=REGISTER
3(311) ERROR: <script>: request_route start -- method=REGISTER
3(311) ERROR: <script>: route[REGISTRAR] start
2(310) ERROR: <script>: route[REGISTRAR] start
3(311) ERROR: <script>: route[REGISTRAR] saving location...
2(310) ERROR: <script>: route[REGISTRAR] saving location...
1(309) ERROR: <script>: route[REGISTRAR] start
1(309) ERROR: <script>: route[REGISTRAR] saving location...
1(309) ERROR: <script>: route[REGISTRAR] saving location... done
3(311) ERROR: <script>: route[REGISTRAR] saving location... done
2(310) ERROR: <script>: route[REGISTRAR] saving location... done
3(311) ERROR: <script>: request_route start -- method=REGISTER
3(311) ERROR: <script>: route[REGISTRAR] start
3(311) ERROR: <script>: route[REGISTRAR] saving location...
3(311) ERROR: <script>: route[REGISTRAR] saving location... done
2(310) ERROR: <script>: request_route start -- method=REGISTER
1(309) ERROR: <script>: request_route start -- method=REGISTER
1(309) ERROR: <script>: route[REGISTRAR] start
1(309) ERROR: <script>: route[REGISTRAR] saving location...
2(310) ERROR: <script>: route[REGISTRAR] start
2(310) ERROR: <script>: route[REGISTRAR] saving location...
1(309) ERROR: <script>: route[REGISTRAR] saving location... done
2(310) ERROR: <script>: route[REGISTRAR] saving location... done
3(311) ERROR: <script>: request_route start -- method=INVITE
3(311) ERROR: <script>: route[REGISTRAR] start
1(309) ERROR: <script>: request_route start -- method=INVITE
1(309) ERROR: <script>: route[REGISTRAR] start
2(310) ERROR: <script>: request_route start -- method=INVITE
2(310) ERROR: <script>: route[REGISTRAR] start
1(309) ERROR: <script>: request_route start -- method=INVITE
1(309) ERROR: <script>: route[REGISTRAR] start
2(310) ERROR: <script>: request_route start -- method=INVITE
2(310) ERROR: <script>: route[REGISTRAR] start
2(310) ERROR: <script>: request_route start -- method=ACK
1(309) NOTICE: acc [acc.c:275]: ACC: transaction answered:
timestamp=1350034404;method=INVITE;from_tag=966002447;to_tag=3a8473f2;call_id=
331189810(a)192.168.10.2
;code=200;reason=OK;src_user=Radio1_device;src_domain=testnet;src_ip=192.168.10.2;dst_ouser=Radio1;dst_user=3;dst_domain=192.168.10.2
3(311) NOTICE: acc [acc.c:275]: ACC: transaction answered:
timestamp=1350034404;method=INVITE;from_tag=1718943109;to_tag=22a97aa7;call_id=
956233863(a)192.168.10.2
;code=200;reason=OK;src_user=User3;src_domain=testnet;src_ip=192.168.10.2;dst_ouser=IC1;dst_user=1;dst_domain=192.168.10.2
3(311) ERROR: <script>: request_route start -- method=ACK
1(309) NOTICE: acc [acc.c:275]: ACC: transaction answered:
timestamp=1350034404;method=INVITE;from_tag=1709043140;to_tag=4aa972a8;call_id=
780056350(a)192.168.10.2
;code=200;reason=OK;src_user=User3;src_domain=testnet;src_ip=192.168.10.2;dst_ouser=IC2;dst_user=2;dst_domain=192.168.10.2
1(309) NOTICE: acc [acc.c:275]: ACC: transaction answered:
timestamp=1350034404;method=INVITE;from_tag=1506064927;to_tag=5a73a3fc;call_id=
1129604628(a)192.168.10.2
;code=200;reason=OK;src_user=User3;src_domain=testnet;src_ip=192.168.10.2;dst_ouser=emergency;dst_user=4;dst_domain=192.168.10.2
1(309) NOTICE: acc [acc.c:275]: ACC: transaction answered:
timestamp=1350034404;method=INVITE;from_tag=1200029211;to_tag=14c6d7d4;call_id=
1214006622(a)192.168.10.2
;code=200;reason=OK;src_user=User3;src_domain=testnet;src_ip=192.168.10.2;dst_ouser=Radio1;dst_user=3;dst_domain=192.168.10.2
1(309) ERROR: <script>: request_route start -- method=ACK
2(310) ERROR: <script>: request_route start -- method=ACK
3(311) ERROR: <script>: request_route start -- method=ACK
2(310) ERROR: <script>: request_route start -- method=REGISTER
2(310) ERROR: <script>: route[REGISTRAR] start
2(310) ERROR: <script>: route[REGISTRAR] saving location...
1(309) ERROR: <script>: request_route start -- method=REGISTER
1(309) ERROR: <script>: route[REGISTRAR] start
1(309) ERROR: <script>: route[REGISTRAR] saving location...
2(310) ERROR: <script>: route[REGISTRAR] saving location... done
1(309) ERROR: <script>: route[REGISTRAR] saving location... done
3(311) ERROR: <script>: request_route start -- method=INVITE
3(311) ERROR: <script>: route[REGISTRAR] start
3(311) ERROR: <script>: request_route start -- method=ACK
1(309) NOTICE: acc [acc.c:275]: ACC: transaction answered:
timestamp=1350034419;method=INVITE;from_tag=22866096;to_tag=461b3848;call_id=
119348927(a)192.168.10.1
;code=200;reason=OK;src_user=User1;src_domain=testnet;src_ip=192.168.10.1;dst_ouser=IC1;dst_user=1;dst_domain=192.168.10.2
2(310) ERROR: <script>: request_route start -- method=INVITE
2(310) ERROR: <script>: route[REGISTRAR] start
2(310) ERROR: <script>: request_route start -- method=ACK
3(311) NOTICE: acc [acc.c:275]: ACC: transaction answered:
timestamp=1350034419;method=INVITE;from_tag=1086463804;to_tag=1b28a959;call_id=
1666188208(a)192.168.10.1
;code=200;reason=OK;src_user=User1;src_domain=testnet;src_ip=192.168.10.1;dst_ouser=IC2;dst_user=2;dst_domain=192.168.10.2
1(309) ERROR: <script>: request_route start -- method=INVITE
1(309) ERROR: <script>: route[REGISTRAR] start
1(309) ERROR: <script>: request_route start -- method=ACK
2(310) NOTICE: acc [acc.c:275]: ACC: transaction answered:
timestamp=1350034419;method=INVITE;from_tag=425688445;to_tag=7c6f5fd8;call_id=
138053674(a)192.168.10.1
;code=200;reason=OK;src_user=User1;src_domain=testnet;src_ip=192.168.10.1;dst_ouser=emergency;dst_user=4;dst_domain=192.168.10.2
3(311) ERROR: <script>: request_route start -- method=INVITE
3(311) ERROR: <script>: route[REGISTRAR] start
3(311) ERROR: <script>: request_route start -- method=ACK
2(310) NOTICE: acc [acc.c:275]: ACC: transaction answered:
timestamp=1350034419;method=INVITE;from_tag=2111220874;to_tag=6a7be79d;call_id=
1381530996(a)192.168.10.1
;code=200;reason=OK;src_user=User1;src_domain=testnet;src_ip=192.168.10.1;dst_ouser=Radio1;dst_user=3;dst_domain=192.168.10.2
1(309) ERROR: <script>: request_route start -- method=INVITE
1(309) ERROR: <script>: route[REGISTRAR] start
1(309) ERROR: <script>: request_route start -- method=ACK
2(310) NOTICE: acc [acc.c:275]: ACC: transaction answered:
timestamp=1350034419;method=INVITE;from_tag=269879616;to_tag=210d7864;call_id=
907816659(a)192.168.10.1
;code=200;reason=OK;src_user=User2;src_domain=testnet;src_ip=192.168.10.1;dst_ouser=IC1;dst_user=1;dst_domain=192.168.10.2
3(311) ERROR: <script>: request_route start -- method=INVITE
3(311) ERROR: <script>: route[REGISTRAR] start
1(309) NOTICE: acc [acc.c:275]: ACC: transaction answered:
timestamp=1350034419;method=INVITE;from_tag=1891265955;to_tag=29747a68;call_id=
1205241696(a)192.168.10.1
;code=200;reason=OK;src_user=User2;src_domain=testnet;src_ip=192.168.10.1;dst_ouser=emergency;dst_user=4;dst_domain=192.168.10.2
3(311) ERROR: <script>: request_route start -- method=ACK
2(310) ERROR: <script>: request_route start -- method=INVITE
1(309) ERROR: <script>: request_route start -- method=INVITE
2(310) ERROR: <script>: request_route start -- method=ACK
2(310) ERROR: <script>: request_route start -- method=INVITE
3(311) ERROR: <script>: request_route start -- method=ACK
3(311) ERROR: <script>: request_route start -- method=INVITE
2(310) ERROR: <script>: request_route start -- method=INVITE <-- Last
message from 2(310)
3(311) ERROR: <script>: request_route start -- method=ACK
3(311) ERROR: <script>: request_route start -- method=ACK
3(311) ERROR: <script>: request_route start -- method=ACK
1(309) ERROR: <script>: request_route start -- method=INVITE <-- Last
message from 1(309)
3(311) ERROR: <script>: request_route start -- method=INVITE
3(311) ERROR: <script>: request_route start -- method=ACK
3(311) ERROR: <script>: request_route start -- method=INVITE
3(311) ERROR: <script>: request_route start -- method=ACK
3(311) ERROR: <script>: request_route start -- method=INVITE
3(311) ERROR: <script>: request_route start -- method=ACK
3(311) ERROR: <script>: request_route start -- method=INVITE
3(311) ERROR: <script>: request_route start -- method=ACK
...
3(311) ERROR: <script>: request_route start -- method=REGISTER
3(311) ERROR: <script>: route[REGISTRAR] start
3(311) ERROR: <script>: route[REGISTRAR] saving location...
3(311) ERROR: <script>: route[REGISTRAR] saving location... done
3(311) ERROR: <script>: request_route start -- method=INVITE <-- Last
message
3:20 p.m.
New subject: Kamailio stability/timing problem w.r.t. registrations?
Hello Michiel ,
You should check with top to see the status of all kamailio processes.
If you see processes that are using 100% CPU, you may have a deadlock.
Connect with gdb to the process to investigate what's going on.
Also, you are running on arm. How did you compiled kamailio: native or cross.
When you run kamailio on an embedded system, you need to check if you
have enough memory. If you start running out of memory, the OS may
start killing processes randomly (check your OS logs).
You definitely need to look at this issue from a broader prospective
as it's not a regular x86 deployment with plenty of CPU and memory.
Also tuning the config (by removing everything that you don't need
might help with memory utilization).
Regards,
Ovidiu Sas
--
VoIP Embedded, Inc.
http://www.voipembedded.com
On Thu, Jan 17, 2013 at 8:41 AM, Sotas Development <sotasdev(a)gmail.com> wrote:
Daniel,
can you add an xlog() at the start of the main
route block and log a
message for any
request received? [...] You can put another xlog before the save()
function to see if
registration requests are getting there. [...] You can switch to kamailio
flavour modules
and see if reproduces.
Thanks for the help! We switched to kamailio flavour, added some xlog
messages and managed to reproduce. See below for some logging.
To our surprise, the last message handled was an INVITE. We'll add more
logging to see whether it is handled successfully or gets stuck somewhere.
Those numbers at the start of the log message, are these child IDs? At first
they alternate, but apparently child 1 and 2 stop running early on (each
with an INVITE as last message).
Regards,
Michiel Veldkamp
0(304) WARNING: <core> [socket_info.c:1392]: WARNING: fix_hostname: could
not rev. resolve 192.168.10.1
0(304) INFO: <core> [tcp_main.c:4832]: init_tcp: using epoll_lt as the io
watch method (auto detected)
0(306) INFO: usrloc [hslot.c:53]: locks array size 512
0(306) INFO: <core> [udp_server.c:179]: INFO: udp_init: SO_RCVBUF is
initially 108544
0(306) INFO: <core> [udp_server.c:230]: INFO: udp_init: SO_RCVBUF is
finally 217088
7(315) INFO: ctl [io_listener.c:225]: io_listen_loop: using epoll_lt io
watch method (config)
1(309) ERROR: <script>: request_route start -- method=REGISTER
2(310) ERROR: <script>: request_route start -- method=REGISTER
3(311) ERROR: <script>: request_route start -- method=REGISTER
3(311) ERROR: <script>: route[REGISTRAR] start
2(310) ERROR: <script>: route[REGISTRAR] start
3(311) ERROR: <script>: route[REGISTRAR] saving location...
2(310) ERROR: <script>: route[REGISTRAR] saving location...
1(309) ERROR: <script>: route[REGISTRAR] start
1(309) ERROR: <script>: route[REGISTRAR] saving location...
1(309) ERROR: <script>: route[REGISTRAR] saving location... done
3(311) ERROR: <script>: route[REGISTRAR] saving location... done
2(310) ERROR: <script>: route[REGISTRAR] saving location... done
3(311) ERROR: <script>: request_route start -- method=REGISTER
3(311) ERROR: <script>: route[REGISTRAR] start
3(311) ERROR: <script>: route[REGISTRAR] saving location...
3(311) ERROR: <script>: route[REGISTRAR] saving location... done
2(310) ERROR: <script>: request_route start -- method=REGISTER
1(309) ERROR: <script>: request_route start -- method=REGISTER
1(309) ERROR: <script>: route[REGISTRAR] start
1(309) ERROR: <script>: route[REGISTRAR] saving location...
2(310) ERROR: <script>: route[REGISTRAR] start
2(310) ERROR: <script>: route[REGISTRAR] saving location...
1(309) ERROR: <script>: route[REGISTRAR] saving location... done
2(310) ERROR: <script>: route[REGISTRAR] saving location... done
3(311) ERROR: <script>: request_route start -- method=INVITE
3(311) ERROR: <script>: route[REGISTRAR] start
1(309) ERROR: <script>: request_route start -- method=INVITE
1(309) ERROR: <script>: route[REGISTRAR] start
2(310) ERROR: <script>: request_route start -- method=INVITE
2(310) ERROR: <script>: route[REGISTRAR] start
1(309) ERROR: <script>: request_route start -- method=INVITE
1(309) ERROR: <script>: route[REGISTRAR] start
2(310) ERROR: <script>: request_route start -- method=INVITE
2(310) ERROR: <script>: route[REGISTRAR] start
2(310) ERROR: <script>: request_route start -- method=ACK
1(309) NOTICE: acc [acc.c:275]: ACC: transaction answered:
timestamp=1350034404;method=INVITE;from_tag=966002447;to_tag=3a8473f2;call_id=331189810@192.168.10.2;code=200;reason=OK;src_user=Radio1_device;src_domain=testnet;src_ip=192.168.10.2;dst_ouser=Radio1;dst_user=3;dst_domain=192.168.10.2
3(311) NOTICE: acc [acc.c:275]: ACC: transaction answered:
timestamp=1350034404;method=INVITE;from_tag=1718943109;to_tag=22a97aa7;call_id=956233863@192.168.10.2;code=200;reason=OK;src_user=User3;src_domain=testnet;src_ip=192.168.10.2;dst_ouser=IC1;dst_user=1;dst_domain=192.168.10.2
3(311) ERROR: <script>: request_route start -- method=ACK
1(309) NOTICE: acc [acc.c:275]: ACC: transaction answered:
timestamp=1350034404;method=INVITE;from_tag=1709043140;to_tag=4aa972a8;call_id=780056350@192.168.10.2;code=200;reason=OK;src_user=User3;src_domain=testnet;src_ip=192.168.10.2;dst_ouser=IC2;dst_user=2;dst_domain=192.168.10.2
1(309) NOTICE: acc [acc.c:275]: ACC: transaction answered:
timestamp=1350034404;method=INVITE;from_tag=1506064927;to_tag=5a73a3fc;call_id=1129604628@192.168.10.2;code=200;reason=OK;src_user=User3;src_domain=testnet;src_ip=192.168.10.2;dst_ouser=emergency;dst_user=4;dst_domain=192.168.10.2
1(309) NOTICE: acc [acc.c:275]: ACC: transaction answered:
timestamp=1350034404;method=INVITE;from_tag=1200029211;to_tag=14c6d7d4;call_id=1214006622@192.168.10.2;code=200;reason=OK;src_user=User3;src_domain=testnet;src_ip=192.168.10.2;dst_ouser=Radio1;dst_user=3;dst_domain=192.168.10.2
1(309) ERROR: <script>: request_route start -- method=ACK
2(310) ERROR: <script>: request_route start -- method=ACK
3(311) ERROR: <script>: request_route start -- method=ACK
2(310) ERROR: <script>: request_route start -- method=REGISTER
2(310) ERROR: <script>: route[REGISTRAR] start
2(310) ERROR: <script>: route[REGISTRAR] saving location...
1(309) ERROR: <script>: request_route start -- method=REGISTER
1(309) ERROR: <script>: route[REGISTRAR] start
1(309) ERROR: <script>: route[REGISTRAR] saving location...
2(310) ERROR: <script>: route[REGISTRAR] saving location... done
1(309) ERROR: <script>: route[REGISTRAR] saving location... done
3(311) ERROR: <script>: request_route start -- method=INVITE
3(311) ERROR: <script>: route[REGISTRAR] start
3(311) ERROR: <script>: request_route start -- method=ACK
1(309) NOTICE: acc [acc.c:275]: ACC: transaction answered:
timestamp=1350034419;method=INVITE;from_tag=22866096;to_tag=461b3848;call_id=119348927@192.168.10.1;code=200;reason=OK;src_user=User1;src_domain=testnet;src_ip=192.168.10.1;dst_ouser=IC1;dst_user=1;dst_domain=192.168.10.2
2(310) ERROR: <script>: request_route start -- method=INVITE
2(310) ERROR: <script>: route[REGISTRAR] start
2(310) ERROR: <script>: request_route start -- method=ACK
3(311) NOTICE: acc [acc.c:275]: ACC: transaction answered:
timestamp=1350034419;method=INVITE;from_tag=1086463804;to_tag=1b28a959;call_id=1666188208@192.168.10.1;code=200;reason=OK;src_user=User1;src_domain=testnet;src_ip=192.168.10.1;dst_ouser=IC2;dst_user=2;dst_domain=192.168.10.2
1(309) ERROR: <script>: request_route start -- method=INVITE
1(309) ERROR: <script>: route[REGISTRAR] start
1(309) ERROR: <script>: request_route start -- method=ACK
2(310) NOTICE: acc [acc.c:275]: ACC: transaction answered:
timestamp=1350034419;method=INVITE;from_tag=425688445;to_tag=7c6f5fd8;call_id=138053674@192.168.10.1;code=200;reason=OK;src_user=User1;src_domain=testnet;src_ip=192.168.10.1;dst_ouser=emergency;dst_user=4;dst_domain=192.168.10.2
3(311) ERROR: <script>: request_route start -- method=INVITE
3(311) ERROR: <script>: route[REGISTRAR] start
3(311) ERROR: <script>: request_route start -- method=ACK
2(310) NOTICE: acc [acc.c:275]: ACC: transaction answered:
timestamp=1350034419;method=INVITE;from_tag=2111220874;to_tag=6a7be79d;call_id=1381530996@192.168.10.1;code=200;reason=OK;src_user=User1;src_domain=testnet;src_ip=192.168.10.1;dst_ouser=Radio1;dst_user=3;dst_domain=192.168.10.2
1(309) ERROR: <script>: request_route start -- method=INVITE
1(309) ERROR: <script>: route[REGISTRAR] start
1(309) ERROR: <script>: request_route start -- method=ACK
2(310) NOTICE: acc [acc.c:275]: ACC: transaction answered:
timestamp=1350034419;method=INVITE;from_tag=269879616;to_tag=210d7864;call_id=907816659@192.168.10.1;code=200;reason=OK;src_user=User2;src_domain=testnet;src_ip=192.168.10.1;dst_ouser=IC1;dst_user=1;dst_domain=192.168.10.2
3(311) ERROR: <script>: request_route start -- method=INVITE
3(311) ERROR: <script>: route[REGISTRAR] start
1(309) NOTICE: acc [acc.c:275]: ACC: transaction answered:
timestamp=1350034419;method=INVITE;from_tag=1891265955;to_tag=29747a68;call_id=1205241696@192.168.10.1;code=200;reason=OK;src_user=User2;src_domain=testnet;src_ip=192.168.10.1;dst_ouser=emergency;dst_user=4;dst_domain=192.168.10.2
3(311) ERROR: <script>: request_route start -- method=ACK
2(310) ERROR: <script>: request_route start -- method=INVITE
1(309) ERROR: <script>: request_route start -- method=INVITE
2(310) ERROR: <script>: request_route start -- method=ACK
2(310) ERROR: <script>: request_route start -- method=INVITE
3(311) ERROR: <script>: request_route start -- method=ACK
3(311) ERROR: <script>: request_route start -- method=INVITE
2(310) ERROR: <script>: request_route start -- method=INVITE <-- Last
message from 2(310)
3(311) ERROR: <script>: request_route start -- method=ACK
3(311) ERROR: <script>: request_route start -- method=ACK
3(311) ERROR: <script>: request_route start -- method=ACK
1(309) ERROR: <script>: request_route start -- method=INVITE <-- Last
message from 1(309)
3(311) ERROR: <script>: request_route start -- method=INVITE
3(311) ERROR: <script>: request_route start -- method=ACK
3(311) ERROR: <script>: request_route start -- method=INVITE
3(311) ERROR: <script>: request_route start -- method=ACK
3(311) ERROR: <script>: request_route start -- method=INVITE
3(311) ERROR: <script>: request_route start -- method=ACK
3(311) ERROR: <script>: request_route start -- method=INVITE
3(311) ERROR: <script>: request_route start -- method=ACK
...
3(311) ERROR: <script>: request_route start -- method=REGISTER
3(311) ERROR: <script>: route[REGISTRAR] start
3(311) ERROR: <script>: route[REGISTRAR] saving location...
3(311) ERROR: <script>: route[REGISTRAR] saving location... done
3(311) ERROR: <script>: request_route start -- method=INVITE <-- Last
message
6:43 p.m.
New subject: Kamailio stability/timing problem w.r.t. registrations?
Hi Ovidiu,
Good points. We're not running out of memory, and all processes keep
running.
Top shows 0% or 1% CPU load for the Kamailio processes.
We cross-compile with the CodeSourcery toolchain. The default build options
drag in the file atomic_unknown.h, that produces the following compile
warning:
"no native memory barrier implementations, falling back to slow lock based
workarround"
Currently we're testing a version compiled with -DNOSMP (no
atomic_unknown.h)
that will run over the weekend.
Regards,
Michiel Veldkamp
On Thu, Jan 17, 2013 at 3:20 PM, Ovidiu Sas <osas(a)voipembedded.com> wrote:
Hello Michiel ,
You should check with top to see the status of all kamailio processes.
If you see processes that are using 100% CPU, you may have a deadlock.
Connect with gdb to the process to investigate what's going on.
Also, you are running on arm. How did you compiled kamailio: native or
cross.
When you run kamailio on an embedded system, you need to check if you
have enough memory. If you start running out of memory, the OS may
start killing processes randomly (check your OS logs).
You definitely need to look at this issue from a broader prospective
as it's not a regular x86 deployment with plenty of CPU and memory.
Also tuning the config (by removing everything that you don't need
might help with memory utilization).
Regards,
Ovidiu Sas
--
VoIP Embedded, Inc.
http://www.voipembedded.com
7:01 p.m.
New subject: Kamailio stability/timing problem w.r.t. registrations?
If you are running the stable version, there's need for heavy Makefile
patching in order to properly cross compile (not to include and link
to host libs).
The trunk has everything fixed and it's cross-compiling properly for
most of the modules.
Make sure that your binaries are properly cross compiled.
Depending on your ARM CPU, atomic locks may or may not work.
I tested openser without atomic locks (using regular locks) and it worked fine.
Regards,
Ovidiu Sas
--
VoIP Embedded, Inc.
http://www.voipembedded.com
On Thu, Jan 17, 2013 at 12:43 PM, Sotas Development <sotasdev(a)gmail.com> wrote:
Hi Ovidiu,
Good points. We're not running out of memory, and all processes keep
running.
Top shows 0% or 1% CPU load for the Kamailio processes.
We cross-compile with the CodeSourcery toolchain. The default build options
drag in the file atomic_unknown.h, that produces the following compile
warning:
"no native memory barrier implementations, falling back to slow lock based
workarround"
Currently we're testing a version compiled with -DNOSMP (no
atomic_unknown.h)
that will run over the weekend.
Regards,
Michiel Veldkamp
On Thu, Jan 17, 2013 at 3:20 PM, Ovidiu Sas <osas(a)voipembedded.com> wrote:
>
> Hello Michiel ,
>
> You should check with top to see the status of all kamailio processes.
> If you see processes that are using 100% CPU, you may have a deadlock.
> Connect with gdb to the process to investigate what's going on.
>
> Also, you are running on arm. How did you compiled kamailio: native or
> cross.
> When you run kamailio on an embedded system, you need to check if you
> have enough memory. If you start running out of memory, the OS may
> start killing processes randomly (check your OS logs).
>
> You definitely need to look at this issue from a broader prospective
> as it's not a regular x86 deployment with plenty of CPU and memory.
> Also tuning the config (by removing everything that you don't need
> might help with memory utilization).
>
>
> Regards,
> Ovidiu Sas
28 Jan
28 Jan
4:08 p.m.
New subject: Kamailio stability/timing problem w.r.t. registrations?
Hi Ovidiu,
Thanks for the warning! We did not yet have much success running the
current master branch, though this may well be a resource problem on
the target platform.
For the moment, we decided to switch back to openser 1.3.5 and wait
for the official 4.0 release.
Regards,
Michiel Veldkamp
On Thu, Jan 17, 2013 at 7:01 PM, Ovidiu Sas <osas(a)voipembedded.com> wrote:
If you are running the stable version, there's
need for heavy Makefile
patching in order to properly cross compile (not to include and link
to host libs).
The trunk has everything fixed and it's cross-compiling properly for
most of the modules.
Make sure that your binaries are properly cross compiled.
Depending on your ARM CPU, atomic locks may or may not work.
I tested openser without atomic locks (using regular locks) and it worked
fine.
Regards,
Ovidiu Sas
--
VoIP Embedded, Inc.
http://www.voipembedded.com
4:44 p.m.
New subject: Kamailio stability/timing problem w.r.t. registrations?
4.0 (current trunk) is in code freeze. I would suggest to test the
trunk version (next 4.0).
Even openser 1.3 requires patches to be properly cross compiled.
Regards,
Ovidiu Sas
--
VoIP Embedded, Inc.
http://www.voipembedded.com
---------- Forwarded message ----------
From: Sotas Development <sotasdev(a)gmail.com>
Date: Mon, Jan 28, 2013 at 10:08 AM
Subject: Re: [SR-Users] Kamailio stability/timing problem w.r.t. registrations?
To: "SIP Router - Kamailio (OpenSER) and SIP Express Router (SER) -
Users Mailing List" <sr-users(a)lists.sip-router.org>
Hi Ovidiu,
Thanks for the warning! We did not yet have much success running the
current master branch, though this may well be a resource problem on
the target platform.
For the moment, we decided to switch back to openser 1.3.5 and wait
for the official 4.0 release.
Regards,
Michiel Veldkamp
On Thu, Jan 17, 2013 at 7:01 PM, Ovidiu Sas <osas(a)voipembedded.com> wrote:
If you are running the stable version, there's need for heavy Makefile
patching in order to properly cross compile (not to include and link
to host libs).
The trunk has everything fixed and it's cross-compiling properly for
most of the modules.
Make sure that your binaries are properly cross compiled.
Depending on your ARM CPU, atomic locks may or may not work.
I tested openser without atomic locks (using regular locks) and it worked fine.
Regards,
Ovidiu Sas
--
VoIP Embedded, Inc.
http://www.voipembedded.com
13 Nov
13 Nov
4:40 p.m.
New subject: Kamailio stability/timing problem w.r.t. registrations?
Hi,
Here's an update of this topic. We've tried again with the latest stable
version 4.0.4. Unfortunately the problem still exists.
In mails above it is mentioned to use kamailio without atomic locks. How do
we this (e.g. which makefile options)?
Kind regards,
Bert
(on behalf of Michiel Veldkamp)
On Mon, Jan 28, 2013 at 4:44 PM, Ovidiu Sas <osas(a)voipembedded.com> wrote:
4.0 (current trunk) is in code freeze. I would
suggest to test the
trunk version (next 4.0).
Even openser 1.3 requires patches to be properly cross compiled.
Regards,
Ovidiu Sas
--
VoIP Embedded, Inc.
http://www.voipembedded.com
---------- Forwarded message ----------
From: Sotas Development <sotasdev(a)gmail.com>
Date: Mon, Jan 28, 2013 at 10:08 AM
Subject: Re: [SR-Users] Kamailio stability/timing problem w.r.t.
registrations?
To: "SIP Router - Kamailio (OpenSER) and SIP Express Router (SER) -
Users Mailing List" <sr-users(a)lists.sip-router.org>
Hi Ovidiu,
Thanks for the warning! We did not yet have much success running the
current master branch, though this may well be a resource problem on
the target platform.
For the moment, we decided to switch back to openser 1.3.5 and wait
for the official 4.0 release.
Regards,
Michiel Veldkamp
On Thu, Jan 17, 2013 at 7:01 PM, Ovidiu Sas <osas(a)voipembedded.com> wrote:
If you are running the stable version, there's need for heavy Makefile
patching in order to properly cross compile (not to include and link
to host libs).
The trunk has everything fixed and it's cross-compiling properly for
most of the modules.
Make sure that your binaries are properly cross compiled.
Depending on your ARM CPU, atomic locks may or may not work.
I tested openser without atomic locks (using regular locks) and it
worked fine.
Regards,
Ovidiu Sas
--
VoIP Embedded, Inc.
http://www.voipembedded.com
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users(a)lists.sip-router.org
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
6:12 p.m.
New subject: Kamailio stability/timing problem w.r.t. registrations?
In a previous e-mail, you posted a warning that you had while compiling:
"no native memory barrier implementations, falling back to slow lock
based workarround"
which means that you are already running without atomic locks.
Regards,
Ovidiu Sas
On Wed, Nov 13, 2013 at 10:40 AM, Sotas Development <sotasdev(a)gmail.com> wrote:
Hi,
Here's an update of this topic. We've tried again with the latest stable
version 4.0.4. Unfortunately the problem still exists.
In mails above it is mentioned to use kamailio without atomic locks. How do
we this (e.g. which makefile options)?
Kind regards,
Bert
(on behalf of Michiel Veldkamp)
On Mon, Jan 28, 2013 at 4:44 PM, Ovidiu Sas <osas(a)voipembedded.com> wrote:
--
VoIP Embedded, Inc.
http://www.voipembedded.com
4.0 (current trunk) is in code freeze. I would suggest to test the
trunk version (next 4.0).
Even openser 1.3 requires patches to be properly cross compiled.
Regards,
Ovidiu Sas
--
VoIP Embedded, Inc.
http://www.voipembedded.com
---------- Forwarded message ----------
From: Sotas Development <sotasdev(a)gmail.com>
Date: Mon, Jan 28, 2013 at 10:08 AM
Subject: Re: [SR-Users] Kamailio stability/timing problem w.r.t.
registrations?
To: "SIP Router - Kamailio (OpenSER) and SIP Express Router (SER) -
Users Mailing List" <sr-users(a)lists.sip-router.org>
Hi Ovidiu,
Thanks for the warning! We did not yet have much success running the
current master branch, though this may well be a resource problem on
the target platform.
For the moment, we decided to switch back to openser 1.3.5 and wait
for the official 4.0 release.
Regards,
Michiel Veldkamp
On Thu, Jan 17, 2013 at 7:01 PM, Ovidiu Sas <osas(a)voipembedded.com> wrote:
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users(a)lists.sip-router.org
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
If you are running the stable version, there's need for heavy Makefile
patching in order to properly cross compile (not to include and link
to host libs).
The trunk has everything fixed and it's cross-compiling properly for
most of the modules.
Make sure that your binaries are properly cross compiled.
Depending on your ARM CPU, atomic locks may or may not work.
I tested openser without atomic locks (using regular locks) and it
worked fine.
Regards,
Ovidiu Sas
--
VoIP Embedded, Inc.
http://www.voipembedded.com
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users(a)lists.sip-router.org
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
27 Nov
27 Nov
11:16 a.m.
New subject: Kamailio stability/timing problem w.r.t. registrations?
Hi,
In the mean time we have gathered more information on this problem:
As given below, kamailio stops grabbing UDP SIP messages (SIP
registrations) after running a while on an embedded ARM and PPC platform
(which runs linux 2.6.33 kernel). Some times the hangup occures within
hours and some times after couple of days running.
NETSTAT OUTPUT:
root# netstat -pl | grep kam
udp 1047968 0 (null):sip
(null):* 8416/kamailio
raw 0 0 (null):255 (null):*
255 8416/kamailio
unix 2 [ ACC ] STREAM LISTENING 755205
8429/kamailio /tmp/kamailio_ctl
Kamailio is started with the following options = -m 4 -n 3 -f <cfg> -D
Other relevant info:
- When Kamailio hangs, I also noticed that the flag "inuse_transactions"
has always the value of '1'. Readout with "kamctl monitor".
- A simple cat to /proc/<kamailio_pid>/wchan gives us the function:
futex_wait_queue_me.
- All possible polling methods are used with -W parameter (sigio_rt, poll,
select etc) during these tests. Non of these options did solve this
problem.
I hope the additional info will clarify more. Thanks in advance.
Best regards,
Orhan Yilmaz
On Wed, Nov 13, 2013 at 6:12 PM, Ovidiu Sas <osas(a)voipembedded.com> wrote:
In a previous e-mail, you posted a warning that you
had while compiling:
"no native memory barrier implementations, falling back to slow lock
based workarround"
which means that you are already running without atomic locks.
Regards,
Ovidiu Sas
On Wed, Nov 13, 2013 at 10:40 AM, Sotas Development <sotasdev(a)gmail.com>
wrote:
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users(a)lists.sip-router.org
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
--
VoIP Embedded, Inc.
http://www.voipembedded.com
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users(a)lists.sip-router.org
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
Hi,
Here's an update of this topic. We've tried again with the latest stable
version 4.0.4. Unfortunately the problem still exists.
In mails above it is mentioned to use kamailio without atomic locks. How
do
we this (e.g. which makefile options)?
Kind regards,
Bert
(on behalf of Michiel Veldkamp)
On Mon, Jan 28, 2013 at 4:44 PM, Ovidiu Sas <osas(a)voipembedded.com>
wrote:
>
> 4.0 (current trunk) is in code freeze. I would suggest to test the
> trunk version (next 4.0).
> Even openser 1.3 requires patches to be properly cross compiled.
>
> Regards,
> Ovidiu Sas
>
> --
> VoIP Embedded, Inc.
> http://www.voipembedded.com
>
> ---------- Forwarded message ----------
> From: Sotas Development <sotasdev(a)gmail.com>
> Date: Mon, Jan 28, 2013 at 10:08 AM
> Subject: Re: [SR-Users] Kamailio stability/timing problem w.r.t.
> registrations?
> To: "SIP Router - Kamailio (OpenSER) and SIP Express Router (SER) -
> Users Mailing List" <sr-users(a)lists.sip-router.org>
>
>
> Hi Ovidiu,
>
> Thanks for the warning! We did not yet have much success running the
> current master branch, though this may well be a resource problem on
> the target platform.
>
> For the moment, we decided to switch back to openser 1.3.5 and wait
> for the official 4.0 release.
>
> Regards,
> Michiel Veldkamp
>
>
> On Thu, Jan 17, 2013 at 7:01 PM, Ovidiu Sas <osas(a)voipembedded.com>
wrote:
If you are running the stable version, there's need for heavy Makefile
patching in order to properly cross compile (not to include and link
to host libs).
The trunk has everything fixed and it's cross-compiling properly for
most of the modules.
Make sure that your binaries are properly cross compiled.
Depending on your ARM CPU, atomic locks may or may not work.
I tested openser without atomic locks (using regular locks) and it
worked fine.
Regards,
Ovidiu Sas
--
VoIP Embedded, Inc.
http://www.voipembedded.com
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users(a)lists.sip-router.org
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
4:44 p.m.
New subject: Kamailio stability/timing problem w.r.t. registrations?
Try to attach gdb to the kamailio processes and run a full backtrace.
Regards,
Ovidiu Sas
On Wed, Nov 27, 2013 at 5:16 AM, Sotas Development <sotasdev(a)gmail.com> wrote:
Hi,
In the mean time we have gathered more information on this problem:
As given below, kamailio stops grabbing UDP SIP messages (SIP registrations)
after running a while on an embedded ARM and PPC platform (which runs linux
2.6.33 kernel). Some times the hangup occures within hours and some times
after couple of days running.
NETSTAT OUTPUT:
root# netstat -pl | grep kam
udp 1047968 0 (null):sip (null):*
8416/kamailio
raw 0 0 (null):255 (null):* 255
8416/kamailio
unix 2 [ ACC ] STREAM LISTENING 755205 8429/kamailio
/tmp/kamailio_ctl
Kamailio is started with the following options = -m 4 -n 3 -f <cfg> -D
Other relevant info:
- When Kamailio hangs, I also noticed that the flag "inuse_transactions" has
always the value of '1'. Readout with "kamctl monitor".
- A simple cat to /proc/<kamailio_pid>/wchan gives us the function:
futex_wait_queue_me.
- All possible polling methods are used with -W parameter (sigio_rt, poll,
select etc) during these tests. Non of these options did solve this problem.
I hope the additional info will clarify more. Thanks in advance.
Best regards,
Orhan Yilmaz
On Wed, Nov 13, 2013 at 6:12 PM, Ovidiu Sas <osas(a)voipembedded.com> wrote:
--
VoIP Embedded, Inc.
http://www.voipembedded.com
In a previous e-mail, you posted a warning that you had while compiling:
"no native memory barrier implementations, falling back to slow lock
based workarround"
which means that you are already running without atomic locks.
Regards,
Ovidiu Sas
On Wed, Nov 13, 2013 at 10:40 AM, Sotas Development <sotasdev(a)gmail.com>
wrote:
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users(a)lists.sip-router.org
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
Hi,
Here's an update of this topic. We've tried again with the latest stable
version 4.0.4. Unfortunately the problem still exists.
In mails above it is mentioned to use kamailio without atomic locks. How
do
we this (e.g. which makefile options)?
Kind regards,
Bert
(on behalf of Michiel Veldkamp)
On Mon, Jan 28, 2013 at 4:44 PM, Ovidiu Sas <osas(a)voipembedded.com>
wrote:
--
VoIP Embedded, Inc.
http://www.voipembedded.com
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users(a)lists.sip-router.org
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
4.0 (current trunk) is in code freeze. I would suggest to test the
trunk version (next 4.0).
Even openser 1.3 requires patches to be properly cross compiled.
Regards,
Ovidiu Sas
--
VoIP Embedded, Inc.
http://www.voipembedded.com
---------- Forwarded message ----------
From: Sotas Development <sotasdev(a)gmail.com>
Date: Mon, Jan 28, 2013 at 10:08 AM
Subject: Re: [SR-Users] Kamailio stability/timing problem w.r.t.
registrations?
To: "SIP Router - Kamailio (OpenSER) and SIP Express Router (SER) -
Users Mailing List" <sr-users(a)lists.sip-router.org>
Hi Ovidiu,
Thanks for the warning! We did not yet have much success running the
current master branch, though this may well be a resource problem on
the target platform.
For the moment, we decided to switch back to openser 1.3.5 and wait
for the official 4.0 release.
Regards,
Michiel Veldkamp
On Thu, Jan 17, 2013 at 7:01 PM, Ovidiu Sas <osas(a)voipembedded.com>
wrote:
>
> If you are running the stable version, there's need for heavy
> Makefile
> patching in order to properly cross compile (not to include and link
> to host libs).
> The trunk has everything fixed and it's cross-compiling properly for
> most of the modules.
> Make sure that your binaries are properly cross compiled.
>
> Depending on your ARM CPU, atomic locks may or may not work.
> I tested openser without atomic locks (using regular locks) and it
> worked fine.
>
> Regards,
> Ovidiu Sas
>
>
> --
> VoIP Embedded, Inc.
> http://www.voipembedded.com
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users(a)lists.sip-router.org
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users(a)lists.sip-router.org
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
29 Nov
29 Nov
3:02 p.m.
New subject: Kamailio stability/timing problem w.r.t. registrations?
Attaching with gdb and getting the backtrace (as Ovidiu said), will be
very useful
The function futex_wait_queue_me() is from kernel, so it might be an
operation from the kernel about reading the udp packages. So we cannot
tell without the full trace if a kamailio function is blocking the futex
or is other standard library function.
I wanted to mention also that poll methods are for tcp only, for udp
kamailio uses recvfrom().
Can you also provide the output for 'kamailio -I' (that's upper case i)?
Cheers,
Daniel
On 11/27/13 4:44 PM, Ovidiu Sas wrote:
Try to attach gdb to the kamailio processes and run a
full backtrace.
Regards,
Ovidiu Sas
On Wed, Nov 27, 2013 at 5:16 AM, Sotas Development <sotasdev(a)gmail.com> wrote:
--
Daniel-Constantin Mierla - http://www.asipto.com
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Hi,
In the mean time we have gathered more information on this problem:
As given below, kamailio stops grabbing UDP SIP messages (SIP registrations)
after running a while on an embedded ARM and PPC platform (which runs linux
2.6.33 kernel). Some times the hangup occures within hours and some times
after couple of days running.
NETSTAT OUTPUT:
root# netstat -pl | grep kam
udp 1047968 0 (null):sip (null):*
8416/kamailio
raw 0 0 (null):255 (null):* 255
8416/kamailio
unix 2 [ ACC ] STREAM LISTENING 755205 8429/kamailio
/tmp/kamailio_ctl
Kamailio is started with the following options = -m 4 -n 3 -f <cfg> -D
Other relevant info:
- When Kamailio hangs, I also noticed that the flag "inuse_transactions" has
always the value of '1'. Readout with "kamctl monitor".
- A simple cat to /proc/<kamailio_pid>/wchan gives us the function:
futex_wait_queue_me.
- All possible polling methods are used with -W parameter (sigio_rt, poll,
select etc) during these tests. Non of these options did solve this problem.
I hope the additional info will clarify more. Thanks in advance.
Best regards,
Orhan Yilmaz
On Wed, Nov 13, 2013 at 6:12 PM, Ovidiu Sas <osas(a)voipembedded.com> wrote:
In a previous e-mail, you posted a warning that
you had while compiling:
"no native memory barrier implementations, falling back to slow lock
based workarround"
which means that you are already running without atomic locks.
Regards,
Ovidiu Sas
On Wed, Nov 13, 2013 at 10:40 AM, Sotas Development <sotasdev(a)gmail.com>
wrote:
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users(a)lists.sip-router.org
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
Hi,
Here's an update of this topic. We've tried again with the latest stable
version 4.0.4. Unfortunately the problem still exists.
In mails above it is mentioned to use kamailio without atomic locks. How
do
we this (e.g. which makefile options)?
Kind regards,
Bert
(on behalf of Michiel Veldkamp)
On Mon, Jan 28, 2013 at 4:44 PM, Ovidiu Sas <osas(a)voipembedded.com>
wrote:
> 4.0 (current trunk) is in code freeze. I would suggest to test the
> trunk version (next 4.0).
> Even openser 1.3 requires patches to be properly cross compiled.
>
> Regards,
> Ovidiu Sas
>
> --
> VoIP Embedded, Inc.
> http://www.voipembedded.com
>
> ---------- Forwarded message ----------
> From: Sotas Development <sotasdev(a)gmail.com>
> Date: Mon, Jan 28, 2013 at 10:08 AM
> Subject: Re: [SR-Users] Kamailio stability/timing problem w.r.t.
> registrations?
> To: "SIP Router - Kamailio (OpenSER) and SIP Express Router (SER) -
> Users Mailing List" <sr-users(a)lists.sip-router.org>
>
>
> Hi Ovidiu,
>
> Thanks for the warning! We did not yet have much success running the
> current master branch, though this may well be a resource problem on
> the target platform.
>
> For the moment, we decided to switch back to openser 1.3.5 and wait
> for the official 4.0 release.
>
> Regards,
> Michiel Veldkamp
>
>
> On Thu, Jan 17, 2013 at 7:01 PM, Ovidiu Sas <osas(a)voipembedded.com>
> wrote:
>> If you are running the stable version, there's need for heavy
>> Makefile
>> patching in order to properly cross compile (not to include and link
>> to host libs).
>> The trunk has everything fixed and it's cross-compiling properly for
>> most of the modules.
>> Make sure that your binaries are properly cross compiled.
>>
>> Depending on your ARM CPU, atomic locks may or may not work.
>> I tested openser without atomic locks (using regular locks) and it
>> worked fine.
>>
>> Regards,
>> Ovidiu Sas
>>
>>
>> --
>> VoIP Embedded, Inc.
>> http://www.voipembedded.com
> _______________________________________________
> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
> sr-users(a)lists.sip-router.org
> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users(a)lists.sip-router.org
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
--
VoIP Embedded, Inc.
http://www.voipembedded.com
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users(a)lists.sip-router.org
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
12 Dec
12 Dec
11:13 a.m.
New subject: Kamailio stability/timing problem w.r.t. registrations?
Hi All,
We have solved this issue now. The problem was the wrong interpretation of
the Kamailio Makefile.defs makefile;
The ARCH script variable was set to a wrong string by our higher level
makefile. Ea; for powerPC, Kamilio.defs expects 'ppc' as the value while
the higher level makefile sets the ARCH variable to 'powerpc'. So Kamailio
was compiled with wrong and missing GCC arguments. Also some C-defines were
not set correctly which resulted in linking wrong 'locking' routines, like
atomic_unknown.h. Instead atomic_ppc.h should be linked/compiled.
Thanks for your support anyway.
Best regards,
Orhan Yilmaz
On Fri, Nov 29, 2013 at 3:02 PM, Daniel-Constantin Mierla <miconda(a)gmail.com
wrote:
> Attaching with gdb and getting the backtrace (as Ovidiu said), will be
> very useful
>
> The function futex_wait_queue_me() is from kernel, so it might be an
> operation from the kernel about reading the udp packages. So we cannot tell
> without the full trace if a kamailio function is blocking the futex or is
> other standard library function.
>
> I wanted to mention also that poll methods are for tcp only, for udp
> kamailio uses recvfrom().
>
> Can you also provide the output for 'kamailio -I' (that's upper case i)?
>
> Cheers,
> Daniel
>
>
>
> On 11/27/13 4:44 PM, Ovidiu Sas wrote:
>
>> Try to attach gdb to the kamailio processes and run a full backtrace.
>>
>> Regards,
>> Ovidiu Sas
>>
>> On Wed, Nov 27, 2013 at 5:16 AM, Sotas Development <sotasdev(a)gmail.com>
> wrote:
>>
>>> Hi,
>>>
>>> In the mean time we have gathered more information on this problem:
>>>
>>> As given below, kamailio stops grabbing UDP SIP messages (SIP
>>> registrations)
>>> after running a while on an embedded ARM and PPC platform (which runs
>>> linux
>>> 2.6.33 kernel). Some times the hangup occures within hours and some times
>>> after couple of days running.
>>>
>>> NETSTAT OUTPUT:
>>> root# netstat -pl | grep kam
>>> udp 1047968 0 (null):sip (null):*
>>> 8416/kamailio
>>> raw 0 0 (null):255 (null):* 255
>>> 8416/kamailio
>>> unix 2 [ ACC ] STREAM LISTENING 755205 8429/kamailio
>>> /tmp/kamailio_ctl
>>>
>>> Kamailio is started with the following options = -m 4 -n 3 -f <cfg> -D
>>>
>>> Other relevant info:
>>> - When Kamailio hangs, I also noticed that the flag
"inuse_transactions"
>>> has
>>> always the value of '1'. Readout with "kamctl monitor".
>>> - A simple cat to /proc/<kamailio_pid>/wchan gives us the function:
>>> futex_wait_queue_me.
>>> - All possible polling methods are used with -W parameter (sigio_rt,
>>> poll,
>>> select etc) during these tests. Non of these options did solve this
>>> problem.
>>>
>>> I hope the additional info will clarify more. Thanks in advance.
>>>
>>> Best regards,
>>> Orhan Yilmaz
>>>
>>>
>>>
>>> On Wed, Nov 13, 2013 at 6:12 PM, Ovidiu Sas <osas(a)voipembedded.com>
>> wrote:
>>>
>>>> In a previous e-mail, you posted a warning that you had while compiling:
>>>> "no native memory barrier implementations, falling back to slow
lock
>>>> based workarround"
>>>> which means that you are already running without atomic locks.
>>>>
>>>> Regards,
>>>> Ovidiu Sas
>>>>
>>>> On Wed, Nov 13, 2013 at 10:40 AM, Sotas Development
<sotasdev(a)gmail.com
>>>> >
>>> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> Here's an update of this topic. We've tried again with the
latest
>>>>> stable
>>>>> version 4.0.4. Unfortunately the problem still exists.
>>>>>
>>>>> In mails above it is mentioned to use kamailio without atomic locks.
>>>>> How
>>>>> do
>>>>> we this (e.g. which makefile options)?
>>>>>
>>>>> Kind regards,
>>>>>
>>>>> Bert
>>>>> (on behalf of Michiel Veldkamp)
>>>>>
>>>>>
>>>>>
>>>>> On Mon, Jan 28, 2013 at 4:44 PM, Ovidiu Sas
<osas(a)voipembedded.com>
>>>> wrote:
>>>>>
>>>>>> 4.0 (current trunk) is in code freeze. I would suggest to test
the
>>>>>> trunk version (next 4.0).
>>>>>> Even openser 1.3 requires patches to be properly cross compiled.
>>>>>>
>>>>>> Regards,
>>>>>> Ovidiu Sas
>>>>>>
>>>>>> --
>>>>>> VoIP Embedded, Inc.
>>>>>> http://www.voipembedded.com
>>>>>>
>>>>>> ---------- Forwarded message ----------
>>>>>> From: Sotas Development <sotasdev(a)gmail.com>
>>>>>> Date: Mon, Jan 28, 2013 at 10:08 AM
>>>>>> Subject: Re: [SR-Users] Kamailio stability/timing problem w.r.t.
>>>>>> registrations?
>>>>>> To: "SIP Router - Kamailio (OpenSER) and SIP Express Router
(SER) -
>>>>>> Users Mailing List" <sr-users(a)lists.sip-router.org>
>>>>>>
>>>>>>
>>>>>> Hi Ovidiu,
>>>>>>
>>>>>> Thanks for the warning! We did not yet have much success running
the
>>>>>> current master branch, though this may well be a resource problem
on
>>>>>> the target platform.
>>>>>>
>>>>>> For the moment, we decided to switch back to openser 1.3.5 and
wait
>>>>>> for the official 4.0 release.
>>>>>>
>>>>>> Regards,
>>>>>> Michiel Veldkamp
>>>>>>
>>>>>>
>>>>>> On Thu, Jan 17, 2013 at 7:01 PM, Ovidiu Sas
<osas(a)voipembedded.com>
>>>>> wrote:
>>>>>>
>>>>>>> If you are running the stable version, there's need for
heavy
>>>>>>> Makefile
>>>>>>> patching in order to properly cross compile (not to include
and link
>>>>>>> to host libs).
>>>>>>> The trunk has everything fixed and it's cross-compiling
properly for
>>>>>>> most of the modules.
>>>>>>> Make sure that your binaries are properly cross compiled.
>>>>>>>
>>>>>>> Depending on your ARM CPU, atomic locks may or may not work.
>>>>>>> I tested openser without atomic locks (using regular locks)
and it
>>>>>>> worked fine.
>>>>>>>
>>>>>>> Regards,
>>>>>>> Ovidiu Sas
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> VoIP Embedded, Inc.
>>>>>>> http://www.voipembedded.com
>>>>>>>
>>>>>> _______________________________________________
>>>>>> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users
mailing
>>>>>> list
>>>>>> sr-users(a)lists.sip-router.org
>>>>>> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
>>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing
list
>>>>> sr-users(a)lists.sip-router.org
>>>>> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
>>>>>
>>>>>
>>>>
>>>> --
>>>> VoIP Embedded, Inc.
>>>> http://www.voipembedded.com
>>>>
>>>> _______________________________________________
>>>> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
>>>> sr-users(a)lists.sip-router.org
>>>> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
>>>>
>>>
>>>
>>> _______________________________________________
>>> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
>>> sr-users(a)lists.sip-router.org
>>> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
>>>
>>>
>>
>>
> --
> Daniel-Constantin Mierla - http://www.asipto.com
> http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
>
>
16 Dec
16 Dec
1:54 p.m.
New subject: Kamailio stability/timing problem w.r.t. registrations?
Hello,
thanks for reporting back, it's good to know that is not an issue in our
Makefile.
Cheers,
Daniel
On 12/12/13 11:13, Sotas Development wrote:
Hi All,
We have solved this issue now. The problem was the wrong
interpretation of the Kamailio Makefile.defs makefile;
The ARCH script variable was set to a wrong string by our higher level
makefile. Ea; for powerPC, Kamilio.defs expects 'ppc' as the value
while the higher level makefile sets the ARCH variable to 'powerpc'.
So Kamailio was compiled with wrong and missing GCC arguments. Also
some C-defines were not set correctly which resulted in linking wrong
'locking' routines, like atomic_unknown.h. Instead atomic_ppc.h should
be linked/compiled.
Thanks for your support anyway.
Best regards,
Orhan Yilmaz
On Fri, Nov 29, 2013 at 3:02 PM, Daniel-Constantin Mierla
<miconda(a)gmail.com <mailto:miconda@gmail.com>> wrote:
Attaching with gdb and getting the backtrace (as Ovidiu said),
will be very useful
The function futex_wait_queue_me() is from kernel, so it might be
an operation from the kernel about reading the udp packages. So we
cannot tell without the full trace if a kamailio function is
blocking the futex or is other standard library function.
I wanted to mention also that poll methods are for tcp only, for
udp kamailio uses recvfrom().
Can you also provide the output for 'kamailio -I' (that's upper
case i)?
Cheers,
Daniel
On 11/27/13 4:44 PM, Ovidiu Sas wrote:
Try to attach gdb to the kamailio processes and run a full
backtrace.
Regards,
Ovidiu Sas
On Wed, Nov 27, 2013 at 5:16 AM, Sotas Development
<sotasdev(a)gmail.com <mailto:sotasdev@gmail.com>> wrote:
Hi,
In the mean time we have gathered more information on this
problem:
As given below, kamailio stops grabbing UDP SIP messages
(SIP registrations)
after running a while on an embedded ARM and PPC platform
(which runs linux
2.6.33 kernel). Some times the hangup occures within hours
and some times
after couple of days running.
NETSTAT OUTPUT:
root# netstat -pl | grep kam
udp 1047968 0 (null):sip (null):*
8416/kamailio
raw 0 0 (null):255 (null):*
255
8416/kamailio
unix 2 [ ACC ] STREAM LISTENING 755205
8429/kamailio
/tmp/kamailio_ctl
Kamailio is started with the following options = -m 4 -n 3
-f <cfg> -D
Other relevant info:
- When Kamailio hangs, I also noticed that the flag
"inuse_transactions" has
always the value of '1'. Readout with "kamctl monitor".
- A simple cat to /proc/<kamailio_pid>/wchan gives us the
function:
futex_wait_queue_me.
- All possible polling methods are used with -W parameter
(sigio_rt, poll,
select etc) during these tests. Non of these options did
solve this problem.
I hope the additional info will clarify more. Thanks in
advance.
Best regards,
Orhan Yilmaz
On Wed, Nov 13, 2013 at 6:12 PM, Ovidiu Sas
<osas(a)voipembedded.com <mailto:osas@voipembedded.com>> wrote:
In a previous e-mail, you posted a warning that you
had while compiling:
"no native memory barrier implementations, falling
back to slow lock
based workarround"
which means that you are already running without
atomic locks.
Regards,
Ovidiu Sas
On Wed, Nov 13, 2013 at 10:40 AM, Sotas Development
<sotasdev(a)gmail.com <mailto:sotasdev@gmail.com>>
wrote:
Hi,
Here's an update of this topic. We've tried again
with the latest stable
version 4.0.4. Unfortunately the problem still exists.
In mails above it is mentioned to use kamailio
without atomic locks. How
do
we this (e.g. which makefile options)?
Kind regards,
Bert
(on behalf of Michiel Veldkamp)
On Mon, Jan 28, 2013 at 4:44 PM, Ovidiu Sas
<osas(a)voipembedded.com <mailto:osas@voipembedded.com>>
wrote:
4.0 (current trunk) is in code freeze. I
would suggest to test the
trunk version (next 4.0).
Even openser 1.3 requires patches to be
properly cross compiled.
Regards,
Ovidiu Sas
--
VoIP Embedded, Inc.
http://www.voipembedded.com
---------- Forwarded message ----------
From: Sotas Development <sotasdev(a)gmail.com
<mailto:sotasdev@gmail.com>>
Date: Mon, Jan 28, 2013 at 10:08 AM
Subject: Re: [SR-Users] Kamailio
stability/timing problem w.r.t.
registrations?
To: "SIP Router - Kamailio (OpenSER) and SIP
Express Router (SER) -
Users Mailing List"
<sr-users(a)lists.sip-router.org
<mailto:sr-users@lists.sip-router.org>>
Hi Ovidiu,
Thanks for the warning! We did not yet have
much success running the
current master branch, though this may well be
a resource problem on
the target platform.
For the moment, we decided to switch back to
openser 1.3.5 and wait
for the official 4.0 release.
Regards,
Michiel Veldkamp
On Thu, Jan 17, 2013 at 7:01 PM, Ovidiu Sas
<osas(a)voipembedded.com
<mailto:osas@voipembedded.com>>
wrote:
If you are running the stable version,
there's need for heavy
Makefile
patching in order to properly cross
compile (not to include and link
to host libs).
The trunk has everything fixed and it's
cross-compiling properly for
most of the modules.
Make sure that your binaries are properly
cross compiled.
Depending on your ARM CPU, atomic locks
may or may not work.
I tested openser without atomic locks
(using regular locks) and it
worked fine.
Regards,
Ovidiu Sas
--
VoIP Embedded, Inc.
http://www.voipembedded.com
_______________________________________________
SIP Express Router (SER) and Kamailio
(OpenSER) - sr-users mailing list
sr-users(a)lists.sip-router.org
<mailto:sr-users@lists.sip-router.org>
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) -
sr-users mailing list
sr-users(a)lists.sip-router.org
<mailto:sr-users@lists.sip-router.org>
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
--
VoIP Embedded, Inc.
http://www.voipembedded.com
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) -
sr-users mailing list
sr-users(a)lists.sip-router.org
<mailto:sr-users@lists.sip-router.org>
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users
mailing list
sr-users(a)lists.sip-router.org
<mailto:sr-users@lists.sip-router.org>
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
--
Daniel-Constantin Mierla - http://www.asipto.com
http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> -
http://www.linkedin.com/in/miconda
--
Daniel-Constantin Mierla - http://www.asipto.com
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
3785
days inactive
4120
days old
14 comments
3 participants
participants (3)
-
Daniel-Constantin Mierla -
Ovidiu Sas -
Sotas Development