[Serusers] Via: branch tags from SER for CANCELs/BYEs frequently don't match value in the INVITE

Frank Durda IV frank.durda at hypercube-llc.com
Fri Dec 4 18:48:35 CET 2009


Hello,

You are a bit out of date on this.  The problem was specific
to CANCELs being ignored because the tag SER added to the
CANCEL did not match the tag that SER added to the INVITE
for the same call.

This flaw is in at least 2.0.0RC1  (we could not use 0.x
versions due to even more bugs), and adding syn_branch=0
merely changed the failure from CANCELs being ignored 100%
of the time by commercial switches to being ignored perhaps
90% of the time (but at least now the messages looked more
like what the RFC says they should look like).

This bug, where in many cases for calls sent from
major name commercial equipment*, the CANCELs would not
have the exact same tag as SER created for the INVITE
(which two brands of commercial switch correctly ignored),
was fixed by changing SER so that it creates its branch
tag from header lines that do not legally change between
INVITE and CANCEL.   Exactly what I stated in later posts
in this thread.  I had to come up with the fix for SER
myself.

* Strangely, calls from simple systems like Asterisk,
always produced the header lines in the CANCEL that were
identical to those in the INVITE, so the same tag was
generated and the CANCEL from such sources worked.
The fancier equipment opted to add legal information to
header lines between INVITE and CANCEL and so doomed
SER to compute a different branch tag.  SER could have
solved this by using data for the computation only up to
the ";" in those headers lines where data could be added
or not using header lines that have a risk of being
different, OR by saving a copy of the tag computed earlier.
Yeah, yeah, I know about the stateless dream.  Whatever.


My code change has been running for several million calls now
at all our sites and solved the problem.  CANCELs are now
accepted and acted-on by both Sonus and Alacatel-Lucent
switches where previously both brands of switch would
ignore most CANCEL messages passed through SER.   The calls
were coming to us via/from ACME packet, Sonus, Nortel,
Cisco and other switch brands, eg equipment who should know
what an INVITE and CANCEL should look like and would have
been accepted if SER hadn't put a mismatched tag in there.


Despite solving this issue in-house, the annoyance threshold
has been reached and we will be removing SER from the
production call path.   SER seems fine for smaller setups,
but shove 100-500Mbit/sec or more of calls through it around
the clock (times several similar sites and all running on a
number of high-high-end servers that are barely ticking over)
and strange things start happening.

Most of that incoming traffic is coming from high-dollar
SBCs and switches, built by people who probably have the
standards implemented mostly right, and with enough volume,
the "infrequent and rare" problems in SER that are known but
no one wants to talk about or fix decide to appear frequently,
and so SER becomes a pain.

No doubt, we set ourselves up for extra pain by electing to
use NAT (for additional Internet isolation and for overlapping
customer private network addressing) and rtpproxy, and worse,
the two-interface mode of rtpproxy, which we chose for
physical network isolation as well as to expand RTP bandwidth
capacity.  The two-interface mode part was extremely painful,
required many code fixes in both SER and rtpproxy to make that
usable. as well as new functions in SER, yet bugs that
were there in SER and rtpproxy all along remain.   On the good
side, SER did protect a particularly unhardened and vulnerable
commercial switch from the stupider SIP messages some of our
clients generated, garbled or forwarded "as is" to us, as well
as the garbage packets one has to expect coming from the
Internet.

To, some the concept of using SER and rtpproxy only as
cannon-fodder to take and if needed crash in response to
the junk we were sometimes sent seems like a demeaning
assignment for a piece of software that really is meant to
do so much more, but we didn't need a lot of that, and the
things we hoped we could do with SER were gradually whittled
to a few because of the huge .cfg language restriction of not
being able to pass variables or the contents of variables
to built-in functions.  (Developers, you have got to
implement a way to pass dynamic data to functions, not just
constants.)

In our case, Everything coming in from the outside had only
one place to go, a PSTN gateway switch sitting nearby.
This meant our database was empty, and if there had been a way
to disconnect mysql and eliminate two or three thousand mysql
connections on each server doing little or nothing (other than
whining that I needed to enable more connections), we would
have done so. 

All SER configuration was done via the .cfg file, of which we
had to run dozens of instances (one per SIP "trunk")
due to limitations in SER that had to be gotten-around by
brute force.  See the lack of variable passing mentioned above.
In a NAT environment where you have to substitute public and
private addresses in various headers of each message, the
lack of variable passing means you have to hard-code
constants for the old and substituted IP addresses everywhere
and had to have giant if-else trees to decide which pair to
substitute and in which direction this message was going
so as to get the substiutions right.  We finally elected to
run multiple separate copies SER on each server so as to not
disturb other customers when adding or changing another.

For almost two years SER did a reasonable job doing the task of
blocking traffic from the bad/malformed packets, and undesired
sources, even though a few got through that killed the PSTN
switch.  However, in the end, it was the good packets from
desired sources that SER then turned into bad packets before
passing them on (such as concoting different tags between
INVITE and CANCEL messages with the same Cseq) that
contributed to the forced retirement of SER.


A probably-final suggestion for SER.   Fixing/updating the
documentation for SER or its successor(s) would be an immense
help to those using or considering its use.  The web site(s)
I have tried to use are dreadful, and even a simple index that
lists functions names and a one-liner of what they are good
for IN THE CURRENT VERSION is tough to come by.   On one I
got the impression that more time was spent implementing
some cutesy HTML/javascript feature than on making the content
usable.

There is too much expectation that someone using SER will
troll all the source files, .c's and .h's looking for any
comments on how functions work or  what parameters they
accept that might have gotten left in there, or better still,
updated to match how the code actually works today.
Be sure to match up dates because you may find three
description of the same function and what parameters it takes,
and only or none of them is how it actually works today.
I realize few people like to write documentation, but it
desperately needs to be done.

Finally, mere mortals can't send e-mails to the SER developers
lists!  It appears to be a closed club with no doorman.
I gave up trying that ages ago.   That being the case, we
"users" have to rely on the developers coming out once in a while
to read what their users are saying out here on the "mortals"
serusers list, OR maybe they sould open access to some list both
parties read and write to.  Having worked on another open-source
project (FreeBSD, back when I had spare time long ago), these
things are handled quite differently there and so communications
between developers and users,  particularly advanced or
heavy-volume users, is pretty easy.

Good luck to all and thanks for the fish.


Frank Durda IV - send mail to this address and remove the "LOSE"
and adjust the month/year password accordingly:
<uhclemLOSE.dec09%nemesis.lonestar.org>    http://nemesis.lonestar.org
   "If you have a point, I suggest you sharpen it." - Frank Durda IV 2003
Copyright 2009, ask before reprinting.





More information about the sr-users mailing list