[Serusers] Running multiple instances of rtpproxy? How to set up in SER? (detailed answer here)
Frank Durda IV
frank.durda at hypercube-llc.com
Tue Mar 31 21:09:22 CEST 2009
A follow-up on using multiple instances of rtpproxy. I finally
got this to work, although I had to look at a lot of source
code and contradictory (and terse) bits of documentation and
finally experiment a while. So here is what I would
consider "real" documentation on how to use these functions
and features of both SER and rtpproxy in one place.
Although not a requirement to make this work, I'll mention
my hardware platform configuration is a Dell 2950 III
48V DC-powered unit with a single four-core processor. The
second CPU socket is empty. FreeBSD reports:
CPU: Intel(R) Xeon(R) CPU E5430 @ 2.66GHz (2665.33-MHz K8-class CPU)
Origin = "GenuineIntel" Id = 0x10676 Stepping = 6
Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,
MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
Features2=0xce3bd<SSE3,RSVD2,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,
PDCM,DCA,<b19>>
AMD Features=0x20100800<SYSCALL,NX,LM>
AMD Features2=0x1<LAHF>
Cores per package: 4
I run rtpproxy in the two-interface mode, which from previous
messages here I have found that I must be the only person
who does, but I do have several good reasons for doing so.
I also recently reported a coding bug in rtpproxy that at
least prevents the two-interface mode from working properly.
The fix I came up for rtpproxy is:
(diff -u this time, since apparently some OSes "patch"
program can't cope with diff -c)
--- main.c.STOCK 2008-02-20 18:51:44.000000000 +0000
+++ main.c 2009-03-25 20:31:34.000000000 +0000
@@ -930,7 +940,7 @@
* cannot be trusted and address is different from one
* that we recorded update it.
*/
- if (spa->untrusted_addr == 0 && !(spa->addr[pidx] != NULL &&
+ if (spa->untrusted_addr[pidx] == 0 && !(spa->addr[pidx] !=
NULL &&
SA_LEN(ia[0]) == SA_LEN(spa->addr[pidx]) &&
memcmp(ia[0], spa->addr[pidx], SA_LEN(ia[0])) == 0)) {
rtpp_log_write(RTPP_LOG_INFO, spa->log, "pre-filling
%s's address "
@@ -940,7 +950,7 @@
spa->addr[pidx] = ia[0];
ia[0] = NULL;
}
- if (spa->rtcp->untrusted_addr == 0 &&
!(spa->rtcp->addr[pidx] != NULL &&
+ if (spa->rtcp->untrusted_addr[pidx] == 0 &&
!(spa->rtcp->addr[pidx] != NULL &&
SA_LEN(ia[1]) == SA_LEN(spa->rtcp->addr[pidx]) &&
memcmp(ia[1], spa->rtcp->addr[pidx], SA_LEN(ia[1])) == 0)) {
if (spa->rtcp->addr[pidx] != NULL)
My setup is a bit more complicated than probably typical,
so if you don't use two interfaces, simply use the one-interface
invocation of rtpproxy ("-l oneipaddr") and build up your additional
IP addresses as needed. You will see more on this in a moment.
Each rtpproxy process needs its own IP address(es) to bind to
for RTP traffic, so you will need to alias multiple IP addresses on
the same ethernet interface or use multiple interfaces. In my case,
I put multiple IP address for each side of the rtpproxy instances
on the same physical interface using "ifconfig interface-name alias"
commands.
I chose to use multiple IP addresses on the same physical interface
because I find that on this system I can't move more than about
40Mbit/sec per interface on a single rtpproxy instance before the
processor core is at 85% utilization, so four should max-out
around 160Mbit/sec, still well below the capacity of a single
gigabit ethernet interface. Depending on your CPU speeds and maker
as well as the system bus and ethernet card type you are using,
your numbers will vary. That may mean buying plug-in cards rather
than using the interfaces that happen to be on the motherboard
(which frequently are slower, interrupt intensive and generally
inefficient compared to others you can install).
The commands to put multiple IP addresses on the same interface would
be like this on BSD and similar systems: (IP addresses are altered)
ifconfig em4 10.5.6.7 netmask 255.255.255.192
ifconfig em4 alias 10.5.6.8 netmask 255.255.255.255
ifconfig em4 alias 10.5.6.9 netmask 255.255.255.255
ifconfig em4 alias 10.5.6.10 netmask 255.255.255.255
and so on. Aliases in the same netblock as the non-aliased IP address
normally have a 255.255.255.255 netmask, at least in the BSD world.
FreeBSD lets you do this in /etc/rc.conf like this:
ifconfig_em4="inet 10.5.6.7 netmask 255.255.128.0"
ifconfig_em4_alias0="inet 10.5.6.8 netmask 255.255.255.255"
ifconfig_em4_alias1="inet 10.5.6.9 netmask 255.255.255.255"
ifconfig_em4_alias2="inet 10.5.6.10 netmask 255.255.255.255"
(If you are running Linux, the way you do aliased IP addresses is
probably different.) On FreeBSD, the result looks like:
(our inside network)
ifconfig -a
...
em4: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
options=1b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING>
inet 10.5.6.7 netmask 0xffffffc0 broadcast 10.5.6.63
inet 10.5.6.8 netmask 0xffffffff broadcast 10.5.6.8
inet 10.5.6.9 netmask 0xffffffff broadcast 10.5.6.9
inet 10.5.6.10 netmask 0xffffffff broadcast 10.5.6.10
ether 00:15:17:5b:xx:xx
media: Ethernet autoselect (1000baseTX <full-duplex>)
status: active
(public Internet side, built with similar commands)
em5: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
options=1b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING>
inet 66.1.2.3 netmask 0xffffffc0 broadcast 66.1.2.63
inet 66.1.2.4 netmask 0xffffffff broadcast 66.1.2.4
inet 66.1.2.5 netmask 0xffffffff broadcast 66.1.2.5
inet 66.1.2.6 netmask 0xffffffff broadcast 66.1.2.6
ether 00:15:17:5b:xx:xx
media: Ethernet autoselect (1000baseTX <full-duplex>)
status: active
...
Now that you have some IP addresses defined, start several instances
of rtpproxy each listening for instructions on at least a
different port than the other instances, as in:
(As found in /etc/rc.local - non BSD systems may need this elsewhere,
but this needs to be ahead of where ser is started and after the
interfaces get their IP addresses.)
/usr/local/bin/rtpproxy -F -s udp:127.0.0.1:8001 -l 66.1.2.3/10.5.6.7
/usr/local/bin/rtpproxy -F -s udp:127.0.0.1:8002 -l 66.1.2.4/10.5.6.8
/usr/local/bin/rtpproxy -F -s udp:127.0.0.1:8003 -l 66.1.2.5/10.5.6.9
/usr/local/bin/rtpproxy -F -s udp:127.0.0.1:8004 -l 66.1.2.6/10.5.6.10
CAUTION: There are probable security issues with using public IP
addresses for the rtpproxy control address, so avoid doing that.
(rtpproxy will take instructions from anybody who can get to the
port and knows how to talk to it, or might just send it junk and
crash that process. If you must use a public IP, block external
access to the port range that is used to control rtpproxy.)
Once running, reference those rtpproxy processes via a line like this
in ser.cfg:
modparam("nathelper", "rtpproxy_sock", "udp:127.0.0.1:8001=1
udp:127.0.0.1:8002=1 udp:127.0.0.1:8003=1 udp:127.0.0.1:8004=1")
That's all on one line. You will need to comment-out the existing
nathelper line that uses the AF_UNIX socket to communicate with
just the one copy of rtpproxy. If you still have the old AF_UNIX
rtpproxy running, kill it.
The "=1" part of the modparam is the weighting, and can be handy if
some of the rtpproxy processes are running on a different computer
that is faster or slower than the local one or others. If all on
the same computer, they should all be equal.
The documentation on the weighting control is unusually weak (even by
SER documentation standards), so I can't tell you exactly how the
weighting control works, what ranges of values are valid (or if
they can be floating point), or the total number of weights should
add up to a total value, or if it is simply a round-robin distribution,
where a "1" says the next call goes to this rtpproxy before advancing
to the next rtpproxy instance, while a "2" says the next two calls
go here before advancing, and so on. I'm using all "1" values for
the local system and it seems to be balancing more or less evenly,
so for other choices you will need to experiment.
Once running for a while, you should see something like this in "top"
PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND
2207 xxxxxxxx 1 8 0 4180K 1056K nanslp 1 17.5H 27.05% rtpproxy
2208 xxxxxxxx 1 8 0 4192K 1068K nanslp 2 17.5H 26.66% rtpproxy
2206 xxxxxxxx 1 102 0 4204K 1080K RUN 3 17.4H 25.29% rtpproxy
2205 xxxxxxxx 1 8 0 4228K 1104K nanslp 0 17.3H 19.92% rtpproxy
You should now also see in tcpdump/ngrep output that the SDP payloads
have different IP addresses in them for the rtpproxy that is handling
the given call. Based on limited experience so far, starting
more rtpproxy instances on a machine than there are total processors/cores
available doesn't seem to do anything bad, but I would think it would
be less efficient than having the numbers of rtpproxy equal to or
less than available CPUs.
I did not mess with any of the mechanisms SER offers that can force
a given call to a given instance. Such a thing might be useful
if you want to guarantee a higher level of stability in packet
forwarding or if some of the instances were functionally different.
In my case, all instances of rtpproxy have the same capabilities.
Good luck!
Frank Durda IV http://nemesis.lonestar.org
Copyright 2009, ask before reprinting.
More information about the sr-users
mailing list