[Serusers] SER for carrier-grade services

[Greger V. Teigre]

Thank you for sharing your views on using ser for carrier-grade services.  I 
find that open source projects often lack information/sharing on 
scalability, redundancy, monitoring and other important things in a 
carrier-grade solution.  Such information is often considered a competitive 
advantage with the result that the open source project looses a lot of the 
potential carrier-grade improvements.

    I share with you the belief that ser can be put into production for 
carrier-grade services.  My company has a focus on end-user centric service 
delivery and provide managed services, as well as turn-key solutions for 
service providers.  Our focus so far has been on home/mobile office services 
with dial-up, VPN, (two-factor) authentication etc.  Our platform has a 
powerful and flexible service authentication and authorization engine at the 
core, with RADIUS as one front-end (using LDAP backend).  In addition, we 
have a web front-end for end-user self-service and service administration, 
as well as an XML interface for CRM integration.  Finally, we have a 
Windows-based software that can automatically configure a user's PC, install 
components, keep them updated, and also be a "menu" or guide for the user in 
using the services (voicemail, forwarding etc).
    Our service providers often have multiple agents, channels, resellers. 
Each with branding requirements, as well as CRM system integration 
requirements.  So, several virtual providers can be enabled on the same 
platform. Of course, providing billing records is a must...

    We are currently adding VoIP as a service option in our platform.  I am 
sure many people on this list are working on commercial projects each with a 
slightly different angle.  :-)

    So more directly to the perspectives in your email:
- Interesting that you have selected WhiteBox as the distro. Surely, a 
RedHat subscription at $349 should not really add a lot to the costs?  We 
have looked at both CentOs, another RHEL-based distro, and WhiteBox, but 
find that the $349 is pretty cheap for the "luxury" of not having to worry 
about distros dying or introducing problems in libraries etc.
- You have opted for the development version of ser.  We have been 
(conservatively) staying with the 0.8.14.  Any particular reason for going 
for development? (Except for cool features; I think the AVP functionality is 
particularly interesting) What is the experienced stability?  Will you put 
the development version into carrier production?!
- SEMS is a bit immature and I have honestly been in doubt whether I should 
spend any time on it at this stage.  We have looked at Asterisk as an option 
(no time for religious wars even though there is some overlapping 
functionality).  In fact, we also evaluate PSTN-based voicemail solutions 
for scalability and reliability. I must admit, it's a bit backwards, but...
- I'm surprised that you have set up serweb at all.  Our experience in 
delivering end-user services is that your Achilles heel is support.  There 
is no limit to what a naive end-user can assume or believe and it will most 
definitely cause you a support call.... I look at serweb as an example 
application, and I think it would be very interesting to make a LGPL'ed 
library for ser functions that could be accessible through soap services. 
This way one can easily integrate ser functionality into the gazillion of 
CRM and customer front-end systems that you find in the telco world.
- You mention an enum directory for subscribers. I'm pretty sure we're going 
there, but there quite some issues related to service provider roaming, how 
it is charged etc.  Of course, existing telcos are keen to stop any all-IP, 
non-billable efforts, and those of us who deliver to telcos must (at least 
to a certain degree) adhere to their demands.  Non-incumbents will lead the 
way here, but I believe that we need to establish a viable business model 
for exchanging traffic.  The market (or the providers) are not ready for 
all-you-can-all-around-the-world options yet.  As long as minutes are 
charged between IP and PSTN, there will be an incentive to move as much 
traffic as possible through PSTN (also for the non-incumbents with such 
business models).

Well, talking about rambling...  I think the point is:  By showing our cards 
a bit, do we have some common interests that can benefit the ser community 
or create a new business relationship?
g-)

Java Rockx wrote:
> Greger,
>
> I believe the small change to the onfailure_route[] was the thing
> that made the difference. Andrei pointed out a small error to me
> where I was calling fix_nated_contact() for more messages than I
> should have. (THANKS ANDREI!!!)
>
> The other thing that made a **substantial** improvement was the fact
> that our PSTN gateway provider that uses Sonus equipment as added as
> an "alias" at the top of the ser.cfg. This caused ser to not include
> "Route:" headers on ACKs and this was causing the Sonus equipment to
> drop calls after two minutes.
>
> It's funny how such small changes to ser.cfg can have dramatic
> effects.
>
> I'd love to see a complete book published on ser. I'm sure, with the
> help of the maintainers and serusers participants this could become a
> reality. I mean this is the best damned SIP router I know of.
> Commercial solutions such as Sonus are not really a "solution"
> because they run as much as US$2 million per box and most small
> companies can't afford "one". And even if you could afford =>one<=
> you may not be able to afford others for scaling your VoIP platform -
> so whats the point of attempting VoIP if you can't scale?
>
> Our solution is modeled after the Google network whereby:
>
>   unlimited horizontal scalablity == unlimited cps (calls per second)
>
> And this equates to:
>
>   successful VoIP Company == retired employee
>
> If we can indeed do this for our implementation and do it on low end
> x86 or Opteron boxes (or blade servers), then the sky is the limit.
> Our engineer whom we acquired from RedHat a few months ago already
> has a full TCP/IP and UDP socket patch working so we can seperate ser
> and serweb and scale them independently of each other (over a VPN)
> since we don't need ser's existing FIFO.
>
> SEMS is a whole other beast. We are now working on this using the ivr
> module with sipums for a carrier-grade voicemail solution. Early
> indications is that we still have plenty to do in order to get
> 99.999% reliablity with sems. Our goal is to not have sems run on the
> same box as ser as we already know this is bad. Voicemail and SIP
> proxy cannot be the same box if you expect to be a big VoIP provider,
> IMHO.
>
> So our high-level VoIP platform uses the following as independent
> server farms:
>
> MySQL 4.1.7 (farm 1)
> ser-0.8.99-dev17 (farm 2)
> serweb on Apache2 farm 3)
> sems/ivr/sipums (farm 4, still a work in progress)
> Third party PSTN gateway *** see comment below
>
> All servers use Whitebox Linux Respin 1 and our network
> implementation allows us to completely administer all servers at any
> remote co-lo site --- even down to remote installation of the
> operating system. This includes make a change to the master ser.cfg
> and having it replicate to all ser proxies.
>
> Monitoring is a big part of this network architecture and we've only
> got basic monitoring right now, so this will become a focus of our
> efforts soon.
>
> We also use third parties for providing PSTN access. IMHO, PSTN
> gateways are much more trouble than their worth. We'd rather rely on
> multiple third parties for this than to try and do it ourselves. The
> cost of DSP boards is expensive, especially if you wanted to handle
> say 50,000 concurrent callers. For this very reason we use companies
> such as Level 3 and Global Crossing and all we have to do is simply
> monitor their QoS and route PSTN callers to other PSTN gateways as
> needed. And this is all at the network level, so ser is totally
> oblivious to this.
>
> Another thing I've love to see is a database whereby VoIP service
> providers can lookup 10-digit DIDs in a common database (much like a
> root DNS server) and determine if a callee is a VoIP user and route
> the SIP call directly from our SIP proxy to their SIP proxy rather
> than making a SIP->PSTN->SIP call. We have the Chairman of the IPCC
> deeply invovled in our company and he could get the ball rolling on
> this, but we don't want to get distracted right now.
>
> Anyhow, I've rambled long enough.
>
> Regards,
> Paul Hazlett
>
>
>
> We'll publish our
>
> --- "Greger V. Teigre" <greger at teigre.com> wrote:
>
>> Paul,
>> Thanks for posting your resulting config after so many hours of
>> testing! I did a diff and can see that you have done several
>> (smaller) changes. But what did the trick? (i.e. for Route problem)
>> g-)
>>
>> Java Rockx wrote:
>>> Greger,
>>>
>>> I finally got everything working without STUN and without an
>>> Outbound Proxy. I an only using rtpproxy/nathelper.
>>>
>>> I've tested these settings with the following SIP Phones/ATAs
>>>
>>> Sipura
>>> UTstarcom iAN-02EX
>>> Grandstream ATA486
>>> Grandstream BT100
>>> Cisco ATA186
>>> Cisco 7960G
>>> WorldAccxx ATA
>>> X-Ten Pro
>>> X-Ten Lite
>>>
>>> We are very happy with everything now. The only piece of the puzzle
>>> that I don't have working yet is sems/sipums for voicemail - but I'm
>>> working on that.
>>>
>>> Attached is my complete ser.cfg file that is working. Please note
>>> that I'm using ser-0.8.99-dev17 for this rather than ser-0.8.14
>>>
>>> If anyone finds problems in this ser.cfg then please let me know -
>>> but like I said before, things seem very stable right now with
>>> -dev17.
>>>
>>> Regards,
>>> Paul
>>>
>>>
>>>
>>> --- "Greger V. Teigre" <greger at teigre.com> wrote:
>>>
>>>> Good to get an authoriative answer on this.  I think I should
>>>> allocate some time to read the RFC thoroughly.
>>>> This leads back to my guess that it could be Paul's outbound proxy
>>>> setting that fixed the problem.  I would think that are some
>>>> Grandstream people this list, but anyway a bug report should be
>>>> submitted to Grandstream.  This was a hard bug to track.  Paul?
>>>> g-)
>>>>
>>>> Jan Janak wrote:
>>>>> No, ser does not change Record-Route to Route header field, user
>>>>> agents are supposed to do it. SER does only two things:
>>>>>
>>>>> 1) Adds Record-Route header fields with its own IP address (this
>>>>>   is what record_route function does)
>>>>> 2) Removes the topmost Route header field if it contains its own
>>>>>   IP address (according to the IP) and forwards the message to
>>>>>   the IP in the next Route header field if any. If there is no
>>>>>   other Route header field then the Request-URI would be used.
>>>>>
>>>>> If there are some Route header fields missing in ACK then this is
>>>>> a bug in the calling user agent, not SER.
>>>>>
>>>>>  Jan.
>>>>>
>>>>> On 18-11 21:16, Greger V. Teigre wrote:
>>>>>> I believe the changes are done in the rr module, in the loose.c
>>>>>> file. RFC3261 defines this (as mentioned by the Sonus guys).
>>>>>>    I remember vaguely reading something about equivalence between
>>>>>> defining outbund proxy on the client and a Route header, but I'm
>>>>>> way off stable ground here... However, if I remember correctly,
>>>>>> it is probably the outbound proxy and not the stun settings that
>>>>>>    does the trick. I have seen some discussions on loose routing
>>>>>> earlier this fall, maybe a search on loose routing in the
>>>>>> archives can turn up some new approaches?
>>>>>>
>>>>>> I'm afraid I don't have anything more to contribute here. From
>>>>>> all I can see, ser should change the Record-Routes to Route, but
>>>>>> doesn't, and I don't understand why.  I think we need somebody
>>>>>> with a more in-depth understanding of the ser inner workings.
>>>>>> g-)
>>>>>>
>>>>>>
>>>>>> ----- Original Message -----
>>>>>> From: "Java Rockx" <javarockx at yahoo.com>
>>>>>> To: "Greger V. Teigre" <greger at teigre.com>; "ser users"
>>>>>> <serusers at lists.iptel.org> Sent: Thursday, November 18, 2004 08:21 PM
>>>>>> Subject: Re: [Serusers] Revisted Error: force_rtp_proxy2: can't
>>>>>> extract bodyfrom the message
>>>>>>
>>>>>>
>>>>>>> Greger,
>>>>>>>
>>>>>>> Do you have any idea how SER decides to include a "Route:"
>>>>>>> versus a "Record-Route:" header? If so,
>>>>>>> which piece of code in ser would write the second ACK below?
>>>>>>>
>>>>>>> Here is a "200 OK" and two ACKs - The first ACK is good and the
>>>>>>> second ACK is bad because it
>>>>>>> should have a "Route:" header referring to the Sonus box.
>>>>>>>
>>>>>>> 100.99.99.99.99 is my SER proxy
>>>>>>> 100.10.10.10 is the public side of my firewall
>>>>>>> 216.50.50.50 is the ip of the Sonus box
>>>>>>>
>>>>>>> So the ACK from SER to Sonus is incorrect.
>>>>>>>
>>>>>>> Do you think this is worth posing to Jiri, Andrei, and company?
>>>>>>> All I know is that this ACK is bad
>>>>>>> when STUN is not used and it is good when STUN is used. I did
>>>>>>> upgrade my Grandstream, but that
>>>>>>> didn't help, and I've modified my nat_uac_test to use mode==19
>>>>>>> rather than mode==3, but still get
>>>>>>> the same results.
>>>>>>>
>>>>>>> Regards,
>>>>>>> Paul
>>>>>>>
>>>>>>> U 2004/11/18 14:13:08.419098 100.99.99.99:5060 ->
>>>>>>> 100.10.10.10:5060 SIP/2.0 200 OK.
>>>>>>> Via: SIP/2.0/UDP
>>>>>>> 192.168.0.83;rport=5060;received=100.10.10.10;branch=z9hG4bKa70081ccdd52daf0.
>>>>>>> To: <sip:14075551212 at sip.mycompany.com;user=phone>;tag=069c9797.
>>>>>>> From: "Paul (1002)"
>>>>>>> <sip:9990010001 at sip.mycompany.com;user=phone>;tag=92691bb29380c100.
>>>>>>> Call-ID: e37c04be3e50ea72 at 192.168.0.83.
>>>>>>> CSeq: 21752 INVITE.
>>>>>>> Contact: sip:4075551212 at 216.50.50.50:5060.
>>>>>>> Record-Route: <sip:216.50.50.50:5060;lr>.
>>>>>>> Record-Route: <sip:100.99.99.99;ftag=92691bb29380c100;lr=on>.
>>>>>>> Accept: multipart/mixed, application/sdp, application/isup,
>>>>>>> application/dtmf,
>>>>>>> application/dtmf-relay.
>>>>>>> Allow: OPTIONS, INVITE, CANCEL, ACK, BYE, PRACK, INFO.
>>>>>>> Supported: timer.
>>>>>>> Content-Disposition: session;handling=required.
>>>>>>> Content-Type: application/sdp.
>>>>>>> Session-Expires: 240;refresher=uas.
>>>>>>> .
>>>>>>> v=0.
>>>>>>> o=Sonus_UAC 18748 26881 IN IP4 216.229.118.76.
>>>>>>> s=SIP Media Capabilities.
>>>>>>> c=IN IP4 100.99.99.99.
>>>>>>> t=0 0.
>>>>>>> m=audio 35552 RTP/AVP 18 101.
>>>>>>> a=rtpmap:18 G729/8000.
>>>>>>> a=rtpmap:101 telephone-event/8000.
>>>>>>> a=fmtp:101 0-15.
>>>>>>> a=sendrecv.
>>>>>>> a=nortpproxy:yes.
>>>>>>>
>>>>>>> #
>>>>>>> U 2004/11/18 14:13:08.428394 100.10.10.10:5060 ->
>>>>>>> 100.99.99.99:5060 ACK sip:4075551212 at 216.50.50.50:5060 SIP/2.0.
>>>>>>> Via: SIP/2.0/UDP 192.168.0.83;branch=z9hG4bKf4bb608e498ec61d.
>>>>>>> Route: <sip:100.99.99.99;ftag=92691bb29380c100;lr=on>.
>>>>>>> Route: <sip:216.50.50.50:5060;lr>.
>>>>>>> From: "Paul (1002)"
>>>>>>> <sip:9990010001 at sip.mycompany.com;user=phone>;tag=92691bb29380c100.
>>>>>>> To: <sip:14075551212 at sip.mycompany.com;user=phone>;tag=069c9797.
>>>>>>> Contact: <sip:9990010001 at 192.168.0.83;user=phone>.
>>>>>>> Call-ID: e37c04be3e50ea72 at 192.168.0.83.
>>>>>>> CSeq: 21752 ACK.
>>>>>>> User-Agent: Grandstream BT100 1.0.5.16.
>>>>>>> Max-Forwards: 70.
>>>>>>> Allow:
>>>>>>> INVITE,ACK,CANCEL,BYE,NOTIFY,REFER,OPTIONS,INFO,SUBSCRIBE. .
>>>>>>>
>>>>>>> #
>>>>>>> U 2004/11/18 14:13:08.429879 100.99.99.99:5060 ->
>>>>>>> 216.50.50.50:5060 ACK sip:216.50.50.50:5060;lr SIP/2.0.
>>>>>>> Via: SIP/2.0/UDP
>>>>>>> 100.99.99.99;branch=z9hG4bK2b35.552edb80cbf475b9be9ae3f9db23f960.0.
>>>>>>> Via: SIP/2.0/UDP
>>>>>>> 192.168.0.83;rport=5060;received=100.10.10.10;branch=z9hG4bKf4bb608e498ec61d.
>>>>>>> From: "Paul (1002)"
>>>>>>> <sip:9990010001 at sip.mycompany.com;user=phone>;tag=92691bb29380c100.
>>>>>>> To: <sip:14075551212 at sip.mycompany.com;user=phone>;tag=069c9797.
>>>>>>> Contact: <sip:9990010001 at 100.10.10.10:5060;user=phone>.
>>>>>>> Call-ID: e37c04be3e50ea72 at 192.168.0.83.
>>>>>>> CSeq: 21752 ACK.
>>>>>>> User-Agent: Grandstream BT100 1.0.5.16.
>>>>>>> Max-Forwards: 16.
>>>>>>> Allow:
>>>>>>> INVITE,ACK,CANCEL,BYE,NOTIFY,REFER,OPTIONS,INFO,SUBSCRIBE. .
>>>>>>>
>>>>>>>
>>>>>>> --- "Greger V. Teigre" <greger at teigre.com> wrote:
>>>>>>>
>>>>>>>> A few comments/questions after having looked at your config
>>>>>>>> file: - You are using 1.0.5.11 firmware on your Grandstream.
>>>>>>>> 1.0.5.16 is the latest and a lot of things have happened since
>>>>>>>> 11. I would suggest you upgrade the firmware before you spend
>>>>>>>> more time on it. - The SIP conversation you sent earlier shows
>>>>>>>> a conversation without using
>>>>>>>> STUN.  In later messages you write about setting outbound and
>>>>>>>> then stun. Have you tried setting outbound, but not stun?  Any
>>>>>>>> difference in Route?
>>>>>>>> - From 1.0.5.7, Grandstream started to send stun-detected ip
>>>>>>>> and port when
>>>>>>>> symmetric NAT was found.  My experience is that if stun is set
>>>>>>>> on
>>
> === message truncated ===
>
>
>
>
> __________________________________
> Do you Yahoo!?
> The all-new My Yahoo! - Get yours free!
> http://my.yahoo.com