15 Mar
2011
15 Mar
'11
4:24 p.m.
All,
I am a long time lurker but never posted. I am running ser 0.9.6 for many years with no
problems. Just recently none of my users can register. Well - they register once then
expire out. I noticed that several ser processes are cpu- bound consuming all
available cycles.
Once they drop register they can never reregister.
I restored the ser DB to an older one that was saved when the system was working properly.
Any ideas why the ser processes would be CPU bound?
Nothing has been changed on the ser proxy in years and the system has been super stable
for 2 years. Has anyone seen this behavior in the past?
This is from top:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
4466 root 25 0 39960 1948 1368 R 35.8 0.0 1:43.97 ser
4477 root 25 0 39960 2028 1448 R 34.5 0.1 0:21.85 ser
4471 root 25 0 39960 2040 1456 R 32.8 0.1 0:47.63 ser
4475 root 25 0 39960 1940 1360 R 32.5 0.0 2:08.90 ser
4470 root 25 0 39960 1916 1336 R 32.2 0.0 1:09.92 ser
4480 root 25 0 39960 1828 1248 R 30.5 0.0 4:45.16 ser
Any ideas would be greatly apreciated.
Michael
See what's next from IPC
www.UnigyfromIPC.com<http://www.UnigyfromIPC.com>
[cid:unigy-logo-compr64b7.png]<http://www.unigyfromipc.com/>
Join the Conversation:
LinkedIn<http://www.linkedin.com/groups?mostPopular=&gid=3808474> *
Twitter<http://twitter.com/IPC_Systems_Inc>
[cid:tree5326.png]Please consider the environment before printing this email.
________________________________
DISCLAIMER: This e-mail may contain information that is confidential, privileged or
otherwise protected from disclosure. If you are not an intended recipient of this e-mail,
do not duplicate or redistribute it by any means. Please delete it and any attachments and
notify the sender that you have received it in error. Unintended recipients are prohibited
from taking action on the basis of information in this e-mail.E-mail messages may contain
computer viruses or other defects, may not be accurately replicated on other systems, or
may be intercepted, deleted or interfered with without the knowledge of the sender or the
intended recipient. If you are not comfortable with the risks associated with e-mail
messages, you may decide not to use e-mail to communicate with IPC. IPC reserves the
right, to the extent and under circumstances permitted by applicable law, to retain,
monitor and intercept e-mail messages to and from its systems.
![cid:unigy-logo-compr64b7.png]<http://www.unigyfromipc.com/>](http://cid:unigy-logo-compr64b7.png]%3Chttp://www.unigyfromipc.com/%3E){kind=link}
15 Mar
15 Mar
5:10 p.m.
Here is some info from /var/log/messages which may help.
Mar 15 10:39:01 nasip1 /usr/local/sbin/ser[1843]: ERROR: parse_uri: uri too short:
<sip:> (4)
Mar 15 10:39:06 nasip1 /usr/local/sbin/ser[1854]: ERROR: t_newtran: new_t failed
Mar 15 10:39:11 nasip1 /usr/local/sbin/ser[1852]: WARNING: do_action:error in expression
Mar 15 10:39:16 nasip1 /usr/local/sbin/ser[1843]: ERROR: parse_sip_msg_uri: bad uri
<sip:>
Mar 15 10:39:21 nasip1 /usr/local/sbin/ser[1856]: Binding
'168061','sip:168061@71.5.65.10:49152' has expired
Mar 15 10:39:21 nasip1 /usr/local/sbin/ser[1854]: ERROR: sl_reply_error used: Regretfully,
we were not able to process the URI (479/SL)
Mar 15 10:39:21 nasip1 /usr/local/sbin/ser[1850]: ERROR: parse_uri: uri too short:
<sip:> (4)
Mar 15 10:39:26 nasip1 /usr/local/sbin/ser[1852]: ERROR: parse_uri: uri too short:
<sip:> (4)
Mar 15 10:39:31 nasip1 /usr/local/sbin/ser[1843]: WARNING: do_action:error in expression
Mar 15 10:39:37 nasip1 /usr/local/sbin/ser[1856]: Binding
'168003','sip:168003@71.5.65.10:49152' has expired
Mar 15 10:39:47 nasip1 /usr/local/sbin/ser[1850]: ERROR: parse_sip_msg_uri: bad uri
<sip:>
Mar 15 10:39:52 nasip1 /usr/local/sbin/ser[1852]: ERROR: parse_sip_msg_uri: bad uri
<sip:>
Mar 15 10:40:02 nasip1 /usr/local/sbin/ser[1856]: Binding
'168049','sip:168049@71.5.65.10:49152' has expired
Mar 15 10:40:07 nasip1 /usr/local/sbin/ser[1850]: ERROR: new_t: uri invalid
Michael
From: sr-users-bounces(a)lists.sip-router.org [mailto:sr-users-bounces@lists.sip-router.org]
On Behalf Of Sabatella, Michael
Sent: Tuesday, March 15, 2011 10:25 AM
To: sr-users(a)lists.sip-router.org
Subject: [SR-Users] problem with ser 0.9.6
All,
I am a long time lurker but never posted. I am running ser 0.9.6 for many years with no
problems. Just recently none of my users can register. Well - they register once then
expire out. I noticed that several ser processes are cpu- bound consuming all
available cycles.
Once they drop register they can never reregister.
I restored the ser DB to an older one that was saved when the system was working properly.
Any ideas why the ser processes would be CPU bound?
Nothing has been changed on the ser proxy in years and the system has been super stable
for 2 years. Has anyone seen this behavior in the past?
This is from top:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
4466 root 25 0 39960 1948 1368 R 35.8 0.0 1:43.97 ser
4477 root 25 0 39960 2028 1448 R 34.5 0.1 0:21.85 ser
4471 root 25 0 39960 2040 1456 R 32.8 0.1 0:47.63 ser
4475 root 25 0 39960 1940 1360 R 32.5 0.0 2:08.90 ser
4470 root 25 0 39960 1916 1336 R 32.2 0.0 1:09.92 ser
4480 root 25 0 39960 1828 1248 R 30.5 0.0 4:45.16 ser
Any ideas would be greatly apreciated.
Michael
See what's next from IPC
www.UnigyfromIPC.com<http://www.UnigyfromIPC.com>
[cid:image001.png@01CBE301.91783DC0]<http://www.unigyfromipc.com/>
Join the Conversation:
LinkedIn<http://www.linkedin.com/groups?mostPopular=&gid=3808474> *
Twitter<http://twitter.com/IPC_Systems_Inc>
[cid:image002.png@01CBE301.91783DC0]Please consider the environment before printing this
email.
________________________________
DISCLAIMER: This e-mail may contain information that is confidential, privileged or
otherwise protected from disclosure. If you are not an intended recipient of this e-mail,
do not duplicate or redistribute it by any means. Please delete it and any attachments and
notify the sender that you have received it in error. Unintended recipients are prohibited
from taking action on the basis of information in this e-mail.E-mail messages may contain
computer viruses or other defects, may not be accurately replicated on other systems, or
may be intercepted, deleted or interfered with without the knowledge of the sender or the
intended recipient. If you are not comfortable with the risks associated with e-mail
messages, you may decide not to use e-mail to communicate with IPC. IPC reserves the
right, to the extent and under circumstances permitted by applicable law, to retain,
monitor and intercept e-mail messages to and from its systems.
5:39 p.m.
On Tuesday 15 March 2011, Sabatella, Michael wrote:
I am a long time lurker but never posted. I am
running ser 0.9.6 for many
years with no problems. Just recently none of my users can register.
Well – they register once then expire out. I noticed that several ser
processes are cpu- bound consuming all available cycles.
Hi Michael,
i remember some similar issue from a few years ago, we also saw this every few
months on some really old openser (more or less similar to the ser version you
run). Have you tried to observe the process state e.g. with strace? If i
remember correctly, the processes then hang in sched_yield() or something like
this..
Honestly, instead of spending a lot of time in debugging this, I'd recommend
you to update to a newer version of sip-router/ kamailio, you'll get also a
lot of new features this way. :-)
Cheers,
Henning
5:51 p.m.
The problem with upgrading is we have a whole provisioning system built for ser and if
upgrading changes the databse I would have to rewrite the whole front end to integrate the
system and that could take me months..
Thanks for the response.
Michael
-----------------------------------------------------
Please consider the environment before printing this email.
-----Original Message-----
From: Henning Westerholt [mailto:henning.westerholt@1und1.de]
Sent: Tuesday, March 15, 2011 11:39 AM
To: sr-users(a)lists.sip-router.org
Cc: Sabatella, Michael
Subject: Re: [SR-Users] problem with ser 0.9.6
On Tuesday 15 March 2011, Sabatella, Michael wrote:
I am a long time lurker but never posted. I am
running ser 0.9.6 for many
years with no problems. Just recently none of my users can register.
Well – they register once then expire out. I noticed that several ser
processes are cpu- bound consuming all available cycles.
Hi Michael,
i remember some similar issue from a few years ago, we also saw this every few
months on some really old openser (more or less similar to the ser version you
run). Have you tried to observe the process state e.g. with strace? If i
remember correctly, the processes then hang in sched_yield() or something like
this..
Honestly, instead of spending a lot of time in debugging this, I'd recommend
you to update to a newer version of sip-router/ kamailio, you'll get also a
lot of new features this way. :-)
Cheers,
Henning
DISCLAIMER: This e-mail may contain information that is confidential, privileged or
otherwise protected from disclosure. If you are not an intended recipient of this e-mail,
do not duplicate or redistribute it by any means. Please delete it and any attachments and
notify the sender that you have received it in error. Unintended recipients are prohibited
from taking action on the basis of information in this e-mail.E-mail messages may contain
computer viruses or other defects, may not be accurately replicated on other systems, or
may be intercepted, deleted or interfered with without the knowledge of the sender or the
intended recipient. If you are not comfortable with the risks associated with e-mail
messages, you may decide not to use e-mail to communicate with IPC. IPC reserves the
right, to the extent and under circumstances permitted by applicable law, to retain,
monitor and intercept e-mail messages to and from its systems.
6:25 p.m.
On Tuesday 15 March 2011, Sabatella, Michael wrote:
The problem with upgrading is we have a whole
provisioning system built for
ser and if upgrading changes the databse I would have to rewrite the whole
front end to integrate the system and that could take me months..
Thanks for the response.
Michael
Hi Michael,
ok, i see. Then just have a look to the process state, maybe this gives you
more informations.
Cheers,
Henning
16 Mar
16 Mar
11:24 a.m.
On 15.03.2011 16:51, Sabatella, Michael wrote:
The problem with upgrading is we have a whole
provisioning system
built for ser and if upgrading changes the databse I would have to
rewrite the whole front end to integrate the system and that could
take me months..
It depends on which tables you are really using. E.g. subscriber, avpops
changed only minimal. If the provisioning system uses proper SQL
queries, then the additional columns should cause much problems. IIRC
only LCR was heavily changed.
regards
klaus
11:27 a.m.
Klaus Darilion writes:
It depends on which tables you are really using. E.g.
subscriber, avpops
changed only minimal. If the provisioning system uses proper SQL
queries, then the additional columns should cause much problems. IIRC
only LCR was heavily changed.
and there is a script that migrates lcr tables.
-- juha
19 Mar
19 Mar
3:37 a.m.
I will look at upgrading in the lab first. I need to stabilize this software. What I
notice is the ser process stops responding to external register requests. This is why
everyone is expiring out because the sip process decides to stop responding. The cpu is
pegged at 100% but at times it comes down to 0.
Lsof and strace do not uncover any issues.
This tool has been stable for us since I deployed it so I cant figure this one out.
Michael Sabatella
IPC Systems
Sr. Network Engineer
Office: 1-201-253-2280
Cell: 1-908-380-5321
@ IPCGlobalNetworks.com
-----------------------------------------------------
Please consider the environment before printing this email.
-----Original Message-----
From: sr-users-bounces(a)lists.sip-router.org [mailto:sr-users-bounces@lists.sip-router.org]
On Behalf Of Juha Heinanen
Sent: Wednesday, March 16, 2011 5:27 AM
To: SIP Router - Kamailio (OpenSER) and SIP Express Router (SER) - Users Mailing List
Subject: Re: [SR-Users] problem with ser 0.9.6
Klaus Darilion writes:
It depends on which tables you are really using. E.g.
subscriber, avpops
changed only minimal. If the provisioning system uses proper SQL
queries, then the additional columns should cause much problems. IIRC
only LCR was heavily changed.
and there is a script that migrates lcr tables.
-- juha
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users(a)lists.sip-router.org
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
DISCLAIMER: This e-mail may contain information that is confidential, privileged or
otherwise protected from disclosure. If you are not an intended recipient of this e-mail,
do not duplicate or redistribute it by any means. Please delete it and any attachments and
notify the sender that you have received it in error. Unintended recipients are prohibited
from taking action on the basis of information in this e-mail.E-mail messages may contain
computer viruses or other defects, may not be accurately replicated on other systems, or
may be intercepted, deleted or interfered with without the knowledge of the sender or the
intended recipient. If you are not comfortable with the risks associated with e-mail
messages, you may decide not to use e-mail to communicate with IPC. IPC reserves the
right, to the extent and under circumstances permitted by applicable law, to retain,
monitor and intercept e-mail messages to and from its systems.
5027
days inactive
5031
days old
7 comments
4 participants
participants (4)
-
Henning Westerholt -
Juha Heinanen -
Klaus Darilion -
Sabatella, Michael