All, I am a long time lurker but never posted. I am running ser 0.9.6 for many years with no problems. Just recently none of my users can register. Well - they register once then expire out. I noticed that several ser processes are cpu- bound consuming all available cycles.
Once they drop register they can never reregister.
I restored the ser DB to an older one that was saved when the system was working properly. Any ideas why the ser processes would be CPU bound?
Nothing has been changed on the ser proxy in years and the system has been super stable for 2 years. Has anyone seen this behavior in the past?
This is from top: PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 4466 root 25 0 39960 1948 1368 R 35.8 0.0 1:43.97 ser 4477 root 25 0 39960 2028 1448 R 34.5 0.1 0:21.85 ser 4471 root 25 0 39960 2040 1456 R 32.8 0.1 0:47.63 ser 4475 root 25 0 39960 1940 1360 R 32.5 0.0 2:08.90 ser 4470 root 25 0 39960 1916 1336 R 32.2 0.0 1:09.92 ser 4480 root 25 0 39960 1828 1248 R 30.5 0.0 4:45.16 ser
Any ideas would be greatly apreciated. Michael
See what's next from IPC www.UnigyfromIPC.comhttp://www.UnigyfromIPC.com [cid:unigy-logo-compr64b7.png]http://www.unigyfromipc.com/ Join the Conversation: LinkedInhttp://www.linkedin.com/groups?mostPopular=&gid=3808474 * Twitterhttp://twitter.com/IPC_Systems_Inc
[cid:tree5326.png]Please consider the environment before printing this email.
________________________________ DISCLAIMER: This e-mail may contain information that is confidential, privileged or otherwise protected from disclosure. If you are not an intended recipient of this e-mail, do not duplicate or redistribute it by any means. Please delete it and any attachments and notify the sender that you have received it in error. Unintended recipients are prohibited from taking action on the basis of information in this e-mail.E-mail messages may contain computer viruses or other defects, may not be accurately replicated on other systems, or may be intercepted, deleted or interfered with without the knowledge of the sender or the intended recipient. If you are not comfortable with the risks associated with e-mail messages, you may decide not to use e-mail to communicate with IPC. IPC reserves the right, to the extent and under circumstances permitted by applicable law, to retain, monitor and intercept e-mail messages to and from its systems.
Here is some info from /var/log/messages which may help.
Mar 15 10:39:01 nasip1 /usr/local/sbin/ser[1843]: ERROR: parse_uri: uri too short: sip: (4) Mar 15 10:39:06 nasip1 /usr/local/sbin/ser[1854]: ERROR: t_newtran: new_t failed Mar 15 10:39:11 nasip1 /usr/local/sbin/ser[1852]: WARNING: do_action:error in expression Mar 15 10:39:16 nasip1 /usr/local/sbin/ser[1843]: ERROR: parse_sip_msg_uri: bad uri sip: Mar 15 10:39:21 nasip1 /usr/local/sbin/ser[1856]: Binding '168061','sip:168061@71.5.65.10:49152' has expired Mar 15 10:39:21 nasip1 /usr/local/sbin/ser[1854]: ERROR: sl_reply_error used: Regretfully, we were not able to process the URI (479/SL) Mar 15 10:39:21 nasip1 /usr/local/sbin/ser[1850]: ERROR: parse_uri: uri too short: sip: (4) Mar 15 10:39:26 nasip1 /usr/local/sbin/ser[1852]: ERROR: parse_uri: uri too short: sip: (4) Mar 15 10:39:31 nasip1 /usr/local/sbin/ser[1843]: WARNING: do_action:error in expression Mar 15 10:39:37 nasip1 /usr/local/sbin/ser[1856]: Binding '168003','sip:168003@71.5.65.10:49152' has expired Mar 15 10:39:47 nasip1 /usr/local/sbin/ser[1850]: ERROR: parse_sip_msg_uri: bad uri sip: Mar 15 10:39:52 nasip1 /usr/local/sbin/ser[1852]: ERROR: parse_sip_msg_uri: bad uri sip: Mar 15 10:40:02 nasip1 /usr/local/sbin/ser[1856]: Binding '168049','sip:168049@71.5.65.10:49152' has expired Mar 15 10:40:07 nasip1 /usr/local/sbin/ser[1850]: ERROR: new_t: uri invalid
Michael
From: sr-users-bounces@lists.sip-router.org [mailto:sr-users-bounces@lists.sip-router.org] On Behalf Of Sabatella, Michael Sent: Tuesday, March 15, 2011 10:25 AM To: sr-users@lists.sip-router.org Subject: [SR-Users] problem with ser 0.9.6
All, I am a long time lurker but never posted. I am running ser 0.9.6 for many years with no problems. Just recently none of my users can register. Well - they register once then expire out. I noticed that several ser processes are cpu- bound consuming all available cycles.
Once they drop register they can never reregister.
I restored the ser DB to an older one that was saved when the system was working properly. Any ideas why the ser processes would be CPU bound?
Nothing has been changed on the ser proxy in years and the system has been super stable for 2 years. Has anyone seen this behavior in the past?
This is from top: PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 4466 root 25 0 39960 1948 1368 R 35.8 0.0 1:43.97 ser 4477 root 25 0 39960 2028 1448 R 34.5 0.1 0:21.85 ser 4471 root 25 0 39960 2040 1456 R 32.8 0.1 0:47.63 ser 4475 root 25 0 39960 1940 1360 R 32.5 0.0 2:08.90 ser 4470 root 25 0 39960 1916 1336 R 32.2 0.0 1:09.92 ser 4480 root 25 0 39960 1828 1248 R 30.5 0.0 4:45.16 ser
Any ideas would be greatly apreciated. Michael
See what's next from IPC www.UnigyfromIPC.comhttp://www.UnigyfromIPC.com [cid:image001.png@01CBE301.91783DC0]http://www.unigyfromipc.com/ Join the Conversation: LinkedInhttp://www.linkedin.com/groups?mostPopular=&gid=3808474 * Twitterhttp://twitter.com/IPC_Systems_Inc
[cid:image002.png@01CBE301.91783DC0]Please consider the environment before printing this email.
________________________________ DISCLAIMER: This e-mail may contain information that is confidential, privileged or otherwise protected from disclosure. If you are not an intended recipient of this e-mail, do not duplicate or redistribute it by any means. Please delete it and any attachments and notify the sender that you have received it in error. Unintended recipients are prohibited from taking action on the basis of information in this e-mail.E-mail messages may contain computer viruses or other defects, may not be accurately replicated on other systems, or may be intercepted, deleted or interfered with without the knowledge of the sender or the intended recipient. If you are not comfortable with the risks associated with e-mail messages, you may decide not to use e-mail to communicate with IPC. IPC reserves the right, to the extent and under circumstances permitted by applicable law, to retain, monitor and intercept e-mail messages to and from its systems.
On Tuesday 15 March 2011, Sabatella, Michael wrote:
I am a long time lurker but never posted. I am running ser 0.9.6 for many years with no problems. Just recently none of my users can register. Well – they register once then expire out. I noticed that several ser processes are cpu- bound consuming all available cycles.
Hi Michael,
i remember some similar issue from a few years ago, we also saw this every few months on some really old openser (more or less similar to the ser version you run). Have you tried to observe the process state e.g. with strace? If i remember correctly, the processes then hang in sched_yield() or something like this..
Honestly, instead of spending a lot of time in debugging this, I'd recommend you to update to a newer version of sip-router/ kamailio, you'll get also a lot of new features this way. :-)
Cheers,
Henning
The problem with upgrading is we have a whole provisioning system built for ser and if upgrading changes the databse I would have to rewrite the whole front end to integrate the system and that could take me months..
Thanks for the response. Michael
----------------------------------------------------- Please consider the environment before printing this email.
-----Original Message----- From: Henning Westerholt [mailto:henning.westerholt@1und1.de] Sent: Tuesday, March 15, 2011 11:39 AM To: sr-users@lists.sip-router.org Cc: Sabatella, Michael Subject: Re: [SR-Users] problem with ser 0.9.6
On Tuesday 15 March 2011, Sabatella, Michael wrote:
I am a long time lurker but never posted. I am running ser 0.9.6 for many years with no problems. Just recently none of my users can register. Well – they register once then expire out. I noticed that several ser processes are cpu- bound consuming all available cycles.
Hi Michael,
i remember some similar issue from a few years ago, we also saw this every few months on some really old openser (more or less similar to the ser version you run). Have you tried to observe the process state e.g. with strace? If i remember correctly, the processes then hang in sched_yield() or something like this..
Honestly, instead of spending a lot of time in debugging this, I'd recommend you to update to a newer version of sip-router/ kamailio, you'll get also a lot of new features this way. :-)
Cheers,
Henning
DISCLAIMER: This e-mail may contain information that is confidential, privileged or otherwise protected from disclosure. If you are not an intended recipient of this e-mail, do not duplicate or redistribute it by any means. Please delete it and any attachments and notify the sender that you have received it in error. Unintended recipients are prohibited from taking action on the basis of information in this e-mail.E-mail messages may contain computer viruses or other defects, may not be accurately replicated on other systems, or may be intercepted, deleted or interfered with without the knowledge of the sender or the intended recipient. If you are not comfortable with the risks associated with e-mail messages, you may decide not to use e-mail to communicate with IPC. IPC reserves the right, to the extent and under circumstances permitted by applicable law, to retain, monitor and intercept e-mail messages to and from its systems.
On Tuesday 15 March 2011, Sabatella, Michael wrote:
The problem with upgrading is we have a whole provisioning system built for ser and if upgrading changes the databse I would have to rewrite the whole front end to integrate the system and that could take me months..
Thanks for the response. Michael
Hi Michael,
ok, i see. Then just have a look to the process state, maybe this gives you more informations.
Cheers,
Henning
On 15.03.2011 16:51, Sabatella, Michael wrote:
The problem with upgrading is we have a whole provisioning system built for ser and if upgrading changes the databse I would have to rewrite the whole front end to integrate the system and that could take me months..
It depends on which tables you are really using. E.g. subscriber, avpops changed only minimal. If the provisioning system uses proper SQL queries, then the additional columns should cause much problems. IIRC only LCR was heavily changed.
regards klaus
Klaus Darilion writes:
It depends on which tables you are really using. E.g. subscriber, avpops changed only minimal. If the provisioning system uses proper SQL queries, then the additional columns should cause much problems. IIRC only LCR was heavily changed.
and there is a script that migrates lcr tables.
-- juha
I will look at upgrading in the lab first. I need to stabilize this software. What I notice is the ser process stops responding to external register requests. This is why everyone is expiring out because the sip process decides to stop responding. The cpu is pegged at 100% but at times it comes down to 0. Lsof and strace do not uncover any issues.
This tool has been stable for us since I deployed it so I cant figure this one out.
Michael Sabatella IPC Systems Sr. Network Engineer Office: 1-201-253-2280 Cell: 1-908-380-5321 @ IPCGlobalNetworks.com
----------------------------------------------------- Please consider the environment before printing this email.
-----Original Message----- From: sr-users-bounces@lists.sip-router.org [mailto:sr-users-bounces@lists.sip-router.org] On Behalf Of Juha Heinanen Sent: Wednesday, March 16, 2011 5:27 AM To: SIP Router - Kamailio (OpenSER) and SIP Express Router (SER) - Users Mailing List Subject: Re: [SR-Users] problem with ser 0.9.6
Klaus Darilion writes:
It depends on which tables you are really using. E.g. subscriber, avpops changed only minimal. If the provisioning system uses proper SQL queries, then the additional columns should cause much problems. IIRC only LCR was heavily changed.
and there is a script that migrates lcr tables.
-- juha
_______________________________________________ SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
DISCLAIMER: This e-mail may contain information that is confidential, privileged or otherwise protected from disclosure. If you are not an intended recipient of this e-mail, do not duplicate or redistribute it by any means. Please delete it and any attachments and notify the sender that you have received it in error. Unintended recipients are prohibited from taking action on the basis of information in this e-mail.E-mail messages may contain computer viruses or other defects, may not be accurately replicated on other systems, or may be intercepted, deleted or interfered with without the knowledge of the sender or the intended recipient. If you are not comfortable with the risks associated with e-mail messages, you may decide not to use e-mail to communicate with IPC. IPC reserves the right, to the extent and under circumstances permitted by applicable law, to retain, monitor and intercept e-mail messages to and from its systems.
-
Henning Westerholt -
Juha Heinanen -
Klaus Darilion -
Sabatella, Michael