Hello,
I can't figure out why Kamailio reports hundreds of calls with the following error:
"/usr/local/sbin/kamailio[2997]: ERROR: rtpproxy [rtpproxy.c:2727]: force_rtp_proxy(): incorrect port 0 in reply from rtp proxy"
It seems that each time this error occurs, we got a one-way audio call.
I already check and increase the limit for opening files that the process can open with ulimit -n 500000. But, anyway, if I look to /proc/$PID/fd, I never see more than 1000 files open.
I also check the socket with netstat: same; I can see the error while there's only less than 800 sockets open.
Is someone know what can be the trigger of this?
Regards,
Igor.
To complete my message, each "incorrect port 0 " is preceded with the following logs:
rtpproxy[3289]: INFO:rtpp_command_ul_handle:ebf8863557e5c8709e46ba48c1f8f780@10.5.129.117: RTP packets from caller will be resized to 20 milliseconds
rtpproxy[3289]: ERR:rtpp_command_pre_parse:GLOBAL: update/create command syntax error: invalid number of arguments (1)
I use the following version:
rtpproxy -v
Basic version: 20040107
Extension 20050322: Support for multiple RTP streams and MOH
Extension 20060704: Support for extra parameter in the V command
Extension 20071116: Support for RTP re-packetization
Extension 20071218: Support for forking (copying) RTP stream
Extension 20080403: Support for RTP statistics querying
Extension 20081102: Support for setting codecs in the update/lookup command
Extension 20081224: Support for session timeout notifications
Extension 20090810: Support for automatic bridging
Extension 20140323: Support for tracking/reporting load
Extension 20140617: Support for anchoring session connect time
Extension 20141004: Support for extendable performance counters
Extension 20150330: Support for allocating a new port ("Un"/"Ln" commands)
Extension 20150420: Support for SEQ tracking and new rtpa_ counters; Q command extended
Regards,
Igor.
De : Igor Potjevlesch [mailto:igor.potjevlesch@gmail.com] Envoyé : vendredi 20 novembre 2015 19:09 À : sr-users@lists.sip-router.org Objet : Kamailio/RTPProxy incorrect port 0
Hello,
I can't figure out why Kamailio reports hundreds of calls with the following error:
"/usr/local/sbin/kamailio[2997]: ERROR: rtpproxy [rtpproxy.c:2727]: force_rtp_proxy(): incorrect port 0 in reply from rtp proxy"
It seems that each time this error occurs, we got a one-way audio call.
I already check and increase the limit for opening files that the process can open with ulimit -n 500000. But, anyway, if I look to /proc/$PID/fd, I never see more than 1000 files open.
I also check the socket with netstat: same; I can see the error while there's only less than 800 sockets open.
Is someone know what can be the trigger of this?
Regards,
Igor.
I'm seeing the exact same message sometimes as well.
For me it seems to be tied to volume, as soon as a lot of calls are coming from an interconnect this could occur. Also, as soon as this happens both incoming and outgoing calls have audio problems.
My RTP instance isn't as much patched as your version, and I'm also not seeing the other INFO packages, just the 0 port error message.
Any insight would be appreciated. Dirk
On 20-11-15 19:25, Igor Potjevlesch wrote:
Hello,
typical reasons for this error are:
- the session is not found by rtpproxy (if it was an update command) - rtpproxy run out of file descriptors (check the limits of your system regarding opened files per process) - some internal error of rtpproxy (see if there are errors in syslog from rtpproxy, you can also run it in debug mode with -D DBUG command line parameters)
Cheers, Daniel
On 23/11/15 09:09, Dirk Teurlings - SIGNET B.V. wrote:
Hello Daniel, Dirk,
I'm also seeing that these errors seems to be related to the load. Because the errors are present during peak of traffic.
Many of the cases are not updates but new calls. Regarding to the file descriptors : I set the limit to 500000:
Max open files 500000 500000 files
And if I had a look to the opened files with, globally, "lsof | grep rtppro | wc -l", there is still less than 1000.
RTPProxy is already running with -d DBUG. I can see that there are update which fail because the log is associated to a call handled into On Reply Route.
How can be explained that the RTPProxy can't find the call? And what is the impact on the call?
Regards,
Igor.
-----Message d'origine----- De : sr-users [mailto:sr-users-bounces@lists.sip-router.org] De la part de Daniel-Constantin Mierla Envoyé : lundi 23 novembre 2015 09:33 À : Kamailio (SER) - Users Mailing List sr-users@lists.sip-router.org Objet : Re: [SR-Users] Kamailio/RTPProxy incorrect port 0
Hello,
typical reasons for this error are:
- the session is not found by rtpproxy (if it was an update command) - rtpproxy run out of file descriptors (check the limits of your system regarding opened files per process) - some internal error of rtpproxy (see if there are errors in syslog from rtpproxy, you can also run it in debug mode with -D DBUG command line parameters)
Cheers, Daniel
On 23/11/15 09:09, Dirk Teurlings - SIGNET B.V. wrote:
INFO:rtpp_command_ul_handle:ebf8863557e5c8709e46ba48c1f8f780@10.5.129.117:
-- Daniel-Constantin Mierla http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com Kamailio Advanced Training, Nov 30-Dec 2, Berlin - http://asipto.com/kat
_______________________________________________ SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
Try setting some more reasonable FD limit, i.e. say 10,000. Bumping from 1,000 to 500,000 seems somewhat excessive. You can also see if you also need to use matching -L option, there might be separate soft limit that application needs to lift by itself. In general, when session creation fails you should see any of those error messages: 10,000 is plenty, that's 2,000 sessions with 5 sockets per session. One instance is going to be CPU-limited at around 1,000 sessions with the current top CPUs and software architecture.
RTPP_ELOG(cf->glog, RTPP_LOG_ERR, "can't create %s socket", (ia->sa_family == AF_INET) ? "IPv4" : "IPv6");
RTPP_ELOG(cf->glog, RTPP_LOG_ERR, "can't bind to the %s port %d", (ia->sa_family == AF_INET) ? "IPv4" : "IPv6", port);
I suggest checking logs around that "0" reply, There should be some clue.
On Mon, Nov 23, 2015 at 5:37 AM, Igor Potjevlesch < igor.potjevlesch@gmail.com> wrote:
Hi Maxim,
There are no errors like "can't create" or "can't bind".
I don't find the -L option into the manual. I'm not sure to understand what the option is used for?
Regards,
Igor.
De : sr-users [mailto:sr-users-bounces@lists.sip-router.org] De la part de Maxim Sobolev Envoyé : mardi 24 novembre 2015 06:26 À : Kamailio (SER) - Users Mailing List sr-users@lists.sip-router.org Objet : Re: [SR-Users] Kamailio/RTPProxy incorrect port 0
Try setting some more reasonable FD limit, i.e. say 10,000. Bumping from 1,000 to 500,000 seems somewhat excessive. You can also see if you also need to use matching -L option, there might be separate soft limit that application needs to lift by itself. In general, when session creation fails you should see any of those error messages: 10,000 is plenty, that's 2,000 sessions with 5 sockets per session. One instance is going to be CPU-limited at around 1,000 sessions with the current top CPUs and software architecture.
RTPP_ELOG(cf->glog, RTPP_LOG_ERR, "can't create %s socket",
(ia->sa_family == AF_INET) ? "IPv4" : "IPv6");
RTPP_ELOG(cf->glog, RTPP_LOG_ERR, "can't bind to the %s port %d",
(ia->sa_family == AF_INET) ? "IPv4" : "IPv6", port);
I suggest checking logs around that "0" reply, There should be some clue.
On Mon, Nov 23, 2015 at 5:37 AM, Igor Potjevlesch <igor.potjevlesch@gmail.com mailto:igor.potjevlesch@gmail.com > wrote:
Hello Daniel, Dirk,
I'm also seeing that these errors seems to be related to the load. Because the errors are present during peak of traffic.
Many of the cases are not updates but new calls. Regarding to the file descriptors : I set the limit to 500000:
Max open files 500000 500000 files
And if I had a look to the opened files with, globally, "lsof | grep rtppro | wc -l", there is still less than 1000.
RTPProxy is already running with -d DBUG. I can see that there are update which fail because the log is associated to a call handled into On Reply Route.
How can be explained that the RTPProxy can't find the call? And what is the impact on the call?
Regards,
Igor.
-----Message d'origine----- De : sr-users [mailto:sr-users-bounces@lists.sip-router.org mailto:sr-users-bounces@lists.sip-router.org ] De la part de Daniel-Constantin Mierla Envoyé : lundi 23 novembre 2015 09:33 À : Kamailio (SER) - Users Mailing List <sr-users@lists.sip-router.org mailto:sr-users@lists.sip-router.org > Objet : Re: [SR-Users] Kamailio/RTPProxy incorrect port 0
Hello,
typical reasons for this error are:
- the session is not found by rtpproxy (if it was an update command) - rtpproxy run out of file descriptors (check the limits of your system regarding opened files per process) - some internal error of rtpproxy (see if there are errors in syslog from rtpproxy, you can also run it in debug mode with -D DBUG command line parameters)
Cheers, Daniel
On 23/11/15 09:09, Dirk Teurlings - SIGNET B.V. wrote:
INFO:rtpp_command_ul_handle:ebf8863557e5c8709e46ba48c1f8f780@10.5.129.117 mailto:INFO%3Artpp_command_ul_handle%3Aebf8863557e5c8709e46ba48c1f8f780@10.5.129.117 :
-- Daniel-Constantin Mierla http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com Kamailio Advanced Training, Nov 30-Dec 2, Berlin - http://asipto.com/kat
_______________________________________________ SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org mailto:sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
_______________________________________________ SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org mailto:sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
That option sets RLIMIT_NOFILE, which is the maximum number of open files for this process to the specified number, i.e. 10,000 in my example. See "man setrlimit" for details.
On Tue, Nov 24, 2015 at 1:41 AM, Igor Potjevlesch < igor.potjevlesch@gmail.com> wrote:
If your system is under heavy loads you can use multiple RTPProxy servers to balance the load between them. You can do this using rtpproxy_sock parameter or read RTPProxy's information from a database. Here I created a GIT repository which you can use to auto-deploy such a configuration:
https://github.com/ghrst/Kamailio-HA
On Tue, Nov 24, 2015 at 6:15 PM, Maxim Sobolev sobomax@sippysoft.com wrote:
Nice, thank you Gholamreza. Not sure the RTPPRoxy is loaded. I mean from a CPU point of view, the process that consumes the most is around 20% of CPU usage.
Maxim: thank you. I will edit my script to put this option. Do you know the default value? I guess that it's inherited from limits.conf or ulimit settings?
Regards,
Igor.
De : sr-users [mailto:sr-users-bounces@lists.sip-router.org] De la part de Gholamreza Sabery Envoyé : mardi 24 novembre 2015 19:22 À : Kamailio (SER) - Users Mailing List sr-users@lists.sip-router.org Objet : Re: [SR-Users] Kamailio/RTPProxy incorrect port 0
If your system is under heavy loads you can use multiple RTPProxy servers to balance the load between them. You can do this using rtpproxy_sock parameter or read RTPProxy's information from a database. Here I created a GIT repository which you can use to auto-deploy such a configuration:
https://github.com/ghrst/Kamailio-HA
On Tue, Nov 24, 2015 at 6:15 PM, Maxim Sobolev <sobomax@sippysoft.com mailto:sobomax@sippysoft.com > wrote:
That option sets RLIMIT_NOFILE, which is the maximum number of open files for this process to the specified number, i.e. 10,000 in my example. See "man setrlimit" for details.
On Tue, Nov 24, 2015 at 1:41 AM, Igor Potjevlesch <igor.potjevlesch@gmail.com mailto:igor.potjevlesch@gmail.com > wrote:
Hi Maxim,
There are no errors like "can't create" or "can't bind".
I don't find the -L option into the manual. I'm not sure to understand what the option is used for?
Regards,
Igor.
De : sr-users [mailto:sr-users-bounces@lists.sip-router.org mailto:sr-users-bounces@lists.sip-router.org ] De la part de Maxim Sobolev Envoyé : mardi 24 novembre 2015 06:26
À : Kamailio (SER) - Users Mailing List <sr-users@lists.sip-router.org mailto:sr-users@lists.sip-router.org > Objet : Re: [SR-Users] Kamailio/RTPProxy incorrect port 0
Try setting some more reasonable FD limit, i.e. say 10,000. Bumping from 1,000 to 500,000 seems somewhat excessive. You can also see if you also need to use matching -L option, there might be separate soft limit that application needs to lift by itself. In general, when session creation fails you should see any of those error messages: 10,000 is plenty, that's 2,000 sessions with 5 sockets per session. One instance is going to be CPU-limited at around 1,000 sessions with the current top CPUs and software architecture.
RTPP_ELOG(cf->glog, RTPP_LOG_ERR, "can't create %s socket",
(ia->sa_family == AF_INET) ? "IPv4" : "IPv6");
RTPP_ELOG(cf->glog, RTPP_LOG_ERR, "can't bind to the %s port %d",
(ia->sa_family == AF_INET) ? "IPv4" : "IPv6", port);
I suggest checking logs around that "0" reply, There should be some clue.
On Mon, Nov 23, 2015 at 5:37 AM, Igor Potjevlesch <igor.potjevlesch@gmail.com mailto:igor.potjevlesch@gmail.com > wrote:
Hello Daniel, Dirk,
I'm also seeing that these errors seems to be related to the load. Because the errors are present during peak of traffic.
Many of the cases are not updates but new calls. Regarding to the file descriptors : I set the limit to 500000:
Max open files 500000 500000 files
And if I had a look to the opened files with, globally, "lsof | grep rtppro | wc -l", there is still less than 1000.
RTPProxy is already running with -d DBUG. I can see that there are update which fail because the log is associated to a call handled into On Reply Route.
How can be explained that the RTPProxy can't find the call? And what is the impact on the call?
Regards,
Igor.
-----Message d'origine----- De : sr-users [mailto:sr-users-bounces@lists.sip-router.org mailto:sr-users-bounces@lists.sip-router.org ] De la part de Daniel-Constantin Mierla Envoyé : lundi 23 novembre 2015 09:33 À : Kamailio (SER) - Users Mailing List <sr-users@lists.sip-router.org mailto:sr-users@lists.sip-router.org > Objet : Re: [SR-Users] Kamailio/RTPProxy incorrect port 0
Hello,
typical reasons for this error are:
- the session is not found by rtpproxy (if it was an update command) - rtpproxy run out of file descriptors (check the limits of your system regarding opened files per process) - some internal error of rtpproxy (see if there are errors in syslog from rtpproxy, you can also run it in debug mode with -D DBUG command line parameters)
Cheers, Daniel
On 23/11/15 09:09, Dirk Teurlings - SIGNET B.V. wrote:
INFO:rtpp_command_ul_handle:ebf8863557e5c8709e46ba48c1f8f780@10.5.129.117 mailto:INFO%3Artpp_command_ul_handle%3Aebf8863557e5c8709e46ba48c1f8f780@10.5.129.117 :
-- Daniel-Constantin Mierla http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com Kamailio Advanced Training, Nov 30-Dec 2, Berlin - http://asipto.com/kat
_______________________________________________ SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org mailto:sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
_______________________________________________ SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org mailto:sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
I'm still not sure to understand the reason why the update failed and how to determine the impact on the calls concerned?
With some days of close supervision of the errors, I can see that, indeed, many of errors are about update, not new calls.
Regards,
Igor.
De : Igor Potjevlesch [mailto:igor.potjevlesch@gmail.com] Envoyé : mercredi 25 novembre 2015 12:00 À : 'Kamailio (SER) - Users Mailing List' sr-users@lists.sip-router.org Objet : RE: [SR-Users] Kamailio/RTPProxy incorrect port 0
Nice, thank you Gholamreza. Not sure the RTPPRoxy is loaded. I mean from a CPU point of view, the process that consumes the most is around 20% of CPU usage.
Maxim: thank you. I will edit my script to put this option. Do you know the default value? I guess that it's inherited from limits.conf or ulimit settings?
Regards,
Igor.
De : sr-users [ mailto:sr-users-bounces@lists.sip-router.org mailto:sr-users-bounces@lists.sip-router.org] De la part de Gholamreza Sabery Envoyé : mardi 24 novembre 2015 19:22 À : Kamailio (SER) - Users Mailing List <sr-users@lists.sip-router.org mailto:sr-users@lists.sip-router.org > Objet : Re: [SR-Users] Kamailio/RTPProxy incorrect port 0
If your system is under heavy loads you can use multiple RTPProxy servers to balance the load between them. You can do this using rtpproxy_sock parameter or read RTPProxy's information from a database. Here I created a GIT repository which you can use to auto-deploy such a configuration:
https://github.com/ghrst/Kamailio-HA
On Tue, Nov 24, 2015 at 6:15 PM, Maxim Sobolev <sobomax@sippysoft.com mailto:sobomax@sippysoft.com > wrote:
That option sets RLIMIT_NOFILE, which is the maximum number of open files for this process to the specified number, i.e. 10,000 in my example. See "man setrlimit" for details.
On Tue, Nov 24, 2015 at 1:41 AM, Igor Potjevlesch <igor.potjevlesch@gmail.com mailto:igor.potjevlesch@gmail.com > wrote:
Hi Maxim,
There are no errors like "can't create" or "can't bind".
I don't find the -L option into the manual. I'm not sure to understand what the option is used for?
Regards,
Igor.
De : sr-users [mailto:sr-users-bounces@lists.sip-router.org mailto:sr-users-bounces@lists.sip-router.org ] De la part de Maxim Sobolev Envoyé : mardi 24 novembre 2015 06:26
À : Kamailio (SER) - Users Mailing List <sr-users@lists.sip-router.org mailto:sr-users@lists.sip-router.org > Objet : Re: [SR-Users] Kamailio/RTPProxy incorrect port 0
Try setting some more reasonable FD limit, i.e. say 10,000. Bumping from 1,000 to 500,000 seems somewhat excessive. You can also see if you also need to use matching -L option, there might be separate soft limit that application needs to lift by itself. In general, when session creation fails you should see any of those error messages: 10,000 is plenty, that's 2,000 sessions with 5 sockets per session. One instance is going to be CPU-limited at around 1,000 sessions with the current top CPUs and software architecture.
RTPP_ELOG(cf->glog, RTPP_LOG_ERR, "can't create %s socket",
(ia->sa_family == AF_INET) ? "IPv4" : "IPv6");
RTPP_ELOG(cf->glog, RTPP_LOG_ERR, "can't bind to the %s port %d",
(ia->sa_family == AF_INET) ? "IPv4" : "IPv6", port);
I suggest checking logs around that "0" reply, There should be some clue.
On Mon, Nov 23, 2015 at 5:37 AM, Igor Potjevlesch <igor.potjevlesch@gmail.com mailto:igor.potjevlesch@gmail.com > wrote:
Hello Daniel, Dirk,
I'm also seeing that these errors seems to be related to the load. Because the errors are present during peak of traffic.
Many of the cases are not updates but new calls. Regarding to the file descriptors : I set the limit to 500000:
Max open files 500000 500000 files
And if I had a look to the opened files with, globally, "lsof | grep rtppro | wc -l", there is still less than 1000.
RTPProxy is already running with -d DBUG. I can see that there are update which fail because the log is associated to a call handled into On Reply Route.
How can be explained that the RTPProxy can't find the call? And what is the impact on the call?
Regards,
Igor.
-----Message d'origine----- De : sr-users [mailto:sr-users-bounces@lists.sip-router.org mailto:sr-users-bounces@lists.sip-router.org ] De la part de Daniel-Constantin Mierla Envoyé : lundi 23 novembre 2015 09:33 À : Kamailio (SER) - Users Mailing List <sr-users@lists.sip-router.org mailto:sr-users@lists.sip-router.org > Objet : Re: [SR-Users] Kamailio/RTPProxy incorrect port 0
Hello,
typical reasons for this error are:
- the session is not found by rtpproxy (if it was an update command) - rtpproxy run out of file descriptors (check the limits of your system regarding opened files per process) - some internal error of rtpproxy (see if there are errors in syslog from rtpproxy, you can also run it in debug mode with -D DBUG command line parameters)
Cheers, Daniel
On 23/11/15 09:09, Dirk Teurlings - SIGNET B.V. wrote:
INFO:rtpp_command_ul_handle:ebf8863557e5c8709e46ba48c1f8f780@10.5.129.117 mailto:INFO%3Artpp_command_ul_handle%3Aebf8863557e5c8709e46ba48c1f8f780@10.5.129.117 :
-- Daniel-Constantin Mierla http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com Kamailio Advanced Training, Nov 30-Dec 2, Berlin - http://asipto.com/kat
_______________________________________________ SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org mailto:sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
_______________________________________________ SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org mailto:sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
-
Daniel-Constantin Mierla -
Dirk Teurlings - SIGNET B.V. -
Gholamreza Sabery -
Igor Potjevlesch -
Maxim Sobolev