We've had similar problems and ended up getting our CDRs from our local PSTN gateways. This works until we begin peering with carriers via IP (which is soon). The gateway records are much more accurate.

One thing I've wondered about but did not have time to check is whether some of the accounting entries are generated by SER? Let me explain. Our accounting script looked for invite and bye records with matching call-id tags. In some cases a bye or an invite was missing. In other cases there were duplicates of one record or another. We enabled accounting for cancel messages and found some problems were due to hangups (invite/cancel instead of invite/bye).

When a call goes to our Asterisk voicemail server the r-uri is re-written and the message t_relay'ed. I suspect some of our mismatched records were due to these calls not being identified properly. I don't have concrete data for this point but I think either the invite to the voicemail server had a different call-id tag or the corresponding bye never made it back from Asterisk.

We also had problems with multiple message from the phones themselves. I think this is due to an older version of SIP code on the phone.

Hopefully some of these thoughts help, Steve

Jiri Kuthan wrote:

At 17:08 18/08/2006, Darren Sessions wrote:

...

Content-Type: multipart/alternative; boundary="----_=_NextPart_001_01C6C2D8.5D25DDC8" Content-class: urn:content-classes:message

All,

We’re experiencing a problem with missing call records. There is nothing in the log files that indicate a problem either with SER or our database server and likewise there is nothing in the database server’s logs to indicate a problem. System loads are within reason on both servers, network connectivity has been tested and verified, and a through hardware shakedown on both boxes has been completed without any errors.

We first noticed this problem when our underlying vendor’s bill (based on minutes of course) did not correspond with our own. Upon doing a CDR comparison (based on what was in both the acc and calls tables), we were able to verify that the calls in dispute were indeed ours (based on the sip call-id’s IP addresses).

We’re processing roughly 1.4 million calls per day on this SER server in question. Out of those calls, 508 calls (give or take a couple hundred) with a total duration of over 5,300 minutes - can not be accounted for in any of the SER database tables.

We’re running 0.9.5-pre1 with Suse Enterprise Server 9.0 (2.6.11 Kernel). Server is an HP DL380 with dual 3.2GHz processors with 6GB physical ram (only 1.4GB in use system wide); disk space is not an issue as well (only 10% in use).

We’ve scoured the mailing lists and change logs for any hint of an issue similar to ours without any results. The closest we’ve been able to come that’s anywhere near along the lines of our issue in any way that made any sense were these posts:

http://lists.iptel.org/pipermail/serdev/2003-August/000602.htmlhttp://lists.iptel.org/pipermail/serdev/2003-August/000602.html

thats usrloc specific, unrelated to accounting.

...

http://lists.iptel.org/pipermail/serusers/2005-February/015102.htmlhttp://lists.iptel.org/pipermail/serusers/2005-February/015102.html

This may be related, even though it is hard to say what the problem cause might have been.

I don't know your script and use case, but I would probably begin to suspect the script whether it correctly identifies all transactions in question to be accounted. Maybe, if you don't net do it, it may be reasonable to account all SIP transactions passing your server.

Second next thing I would do is to look at syslog. I remember that under some stressful mysql conditions, some INSERTs really got lost, which was properly logged in SER's logfiles/.

...

At this point, we’re not sure if this issue has something to do with the FIFO and SER making bulk dump uploads with call data to the database, or some other issue along those lines – or if this is a new issue all together.

That has hardly to do with FIFO.

-jiri

...

Any help would be greatly appreciated.

Thanks,

• Darren

Fifo settings:

fifo="/tmp/ser_fifo" fifo_db_url="mysql://xxx:xxx@xxx/serdb"

Here are our module specific parameters:

# -- timers -- modparam("tm", "wt_timer", 10) modparam("tm", "retr_timer1p1", 2)

# -- usrloc params -- modparam("usrloc", "db_mode", 2)

# -- database URL -- modparam("usrloc|acc|auth_db|group","db_url","mysql://xxx:xxx@xxx/serdb")

# -- auth params -- modparam("auth_db", "calculate_ha1", yes) modparam("auth_db", "password_column", "password") modparam("auth","secret","xxx")

# -- rr params -- modparam("rr", "enable_full_lr", 1)

# -- Nathelper -- modparam("registrar", "nat_flag", 6) modparam("nathelper", "natping_interval", 10) modparam("nathelper", "ping_nated_only", 1) modparam("nathelper", "rtpproxy_disable", 1)

# -- acc params -- modparam("acc", "db_flag", 3)

---

Serusers mailing list Serusers@lists.iptel.org http://lists.iptel.org/mailman/listinfo/serusers

-- Jiri Kuthan http://iptel.org/~jiri/

---

Serusers mailing list Serusers@lists.iptel.org http://lists.iptel.org/mailman/listinfo/serusers

Jiri / Everyone, here is a S. F. S. C. S. one one sanitized version of our ser.cfg file.

Any suggestions or recommendations would be appreciated.

Thank,

- Darren

# ----------- global configuration parameters ------------------------

debug=3 # debug level (cmd line: -dddddddddd) fork=yes log_stderror=no # (cmd line: -E) memlog=0

#uncomment for debug mode #debug=3 #fork=no #log_stderror=yes

#Bind IP listen=xxx.xxx.xxx.xxx

check_via=no # (cmd. line: -v) dns=yes # (cmd. line: -r) rev_dns=no # (cmd. line: -R) port=5060 children=40 mhomed=no # multi-homed?

fifo="/tmp/ser_fifo" fifo_db_url="mysql://serdba:ser2u2c@xxx.xxx.xxx.xxx/serdb"

# ------------------ module loading ----------------------------------

loadmodule "/usr/local/lib/ser/modules/mysql.so" loadmodule "/usr/local/lib/ser/modules/nathelper.so" loadmodule "/usr/local/lib/ser/modules/textops.so" loadmodule "/usr/local/lib/ser/modules/sl.so" loadmodule "/usr/local/lib/ser/modules/tm.so" loadmodule "/usr/local/lib/ser/modules/rr.so" loadmodule "/usr/local/lib/ser/modules/maxfwd.so" loadmodule "/usr/local/lib/ser/modules/usrloc.so" loadmodule "/usr/local/lib/ser/modules/registrar.so" loadmodule "/usr/local/lib/ser/modules/auth.so" loadmodule "/usr/local/lib/ser/modules/auth_db.so" loadmodule "/usr/local/lib/ser/modules/acc.so" loadmodule "/usr/local/lib/ser/modules/xlog.so" loadmodule "/usr/local/lib/ser/modules/exec.so" loadmodule "/usr/local/lib/ser/modules/group.so"

# ----------------- setting module-specific parameters ---------------

# -- timers -- modparam("tm", "wt_timer", 10) modparam("tm", "retr_timer1p1", 2)

# -- usrloc params -- modparam("usrloc", "db_mode", 2)

# -- database URL -- modparam("usrloc|acc|auth_db|group","db_url","mysql://xxx:xxx@xxx.xxx.xx x.xxx/serdb")

# -- auth params -- modparam("auth_db", "calculate_ha1", yes) modparam("auth_db", "password_column", "password") modparam("auth","secret","xxx")

# -- rr params -- modparam("rr", "enable_full_lr", 1)

# -- Nathelper -- modparam("registrar", "nat_flag", 6) modparam("nathelper", "natping_interval", 10) modparam("nathelper", "ping_nated_only", 1) modparam("nathelper", "rtpproxy_disable", 1)

# -- acc params -- modparam("acc", "db_flag", 3)

alias=proxy.host

# ------------------------- request routing logic -------------------

# main routing logic

route {

# initial sanity checks -- messages with # max_forwards==0, or excessively long requests if (!mf_process_maxfwd_header("10")) { sl_send_reply("483","Too Many Hops"); break;

}; if (msg:len > max_len) { sl_send_reply("513", "Message too big"); break; };

# Nathelper # if (nat_uac_test("19")) { if (method == "REGISTER" || !search("^Record-Route:.*xxx.xxx.xxx.xxx")) { fix_nated_contact(); # Rewrite contact with source IP of signaling force_rport(); setflag(6); }; # };

# Turn on SIP accounting setflag(3);

# we record-route all messages -- to make sure that # subsequent messages will go through our proxy; that's # particularly good if upstream and downstream entities # use different transport protocol if (!method=="REGISTER") record_route();

# loose-route processing if (loose_route()) { route(1); break; };

lookup("aliases"); if (!uri==myself) { route(1); break; };

# if the request is for other domain use UsrLoc # (in case, it does not work, use the following command # with proper names and addresses in it) if (uri==myself) { if (method=="REGISTER") { fix_nated_register();

if (src_ip!=xxx.xxx.xxx.xxx && src_ip!=xxx.xxx.xxx.xxx && src_ip!=xxx.xxx.xxx.xxx && src_ip!=xxx.xxx.xxx.xxx && src_ip!=xxx.xxx.xxx.xxx && src_ip!=xxx.xxx.xxx.xxx && src_ip!=xxx.xxx.xxx.xxx && src_ip!=xxx.xxx.xxx.xxx && src_ip!=xxx.xxx.xxx.xxx) { if (!www_authorize("proxy.host", "subscriber")) { www_challenge("proxy.host", "0"); break; }; };

save("location_novm"); save("location"); break; };

# If not in our list of allowed hosts - authenticate the call originator # If this is not one of our customers we don't want them making calls if (src_ip!=xxx.xxx.xxx.xxx && src_ip!=xxx.xxx.xxx.xxx && src_ip!=xxx.xxx.xxx.xxx && src_ip!=xxx.xxx.xxx.xxx && src_ip!=xxx.xxx.xxx.xxx && src_ip!=xxx.xxx.xxx.xxx && src_ip!=xxx.xxx.xxx.xxx && src_ip!=xxx.xxx.xxx.xxx && src_ip!=xxx.xxx.xxx.xxx && src_ip!=xxx.xxx.xxx.xxx) { if (!(proxy_authorize("proxy.host","subscriber"))) { proxy_challenge("proxy.host", "1"); break; } }

## Save the Remote-Party-ID for anonymous calls exec_msg("/usr/local/sbin/save_rpid");

### Process URI routing

#strip enhanced mobile prefix (only used for cdr processing) if (uri =~ "^sip:333#") { strip(4); };

if (uri =~ "^sip:666#") { rewriteuri("sip:666@xxx.xxx.xxx.xxx"); if (!t_relay_to_udp("xxx.xxx.xxx.xxx","5060")) { sl_reply_error(); }; break; };

if ((uri =~ "^sip:555#") && (src_ip != xxx.xxx.xxx.xxx)) { rewritehostport("xxx.xxx.xxx.xxx"); if (!t_relay_to_udp("xxx.xxx.xxx.xxx","5060")) { sl_reply_error(); }; break; };

if ((uri =~ "^sip:999#") && (src_ip != xxx.xxx.xxx.xxx)) { rewritehostport("xxx.xxx.xxx.xxx"); if (!t_relay_to_udp("xxx.xxx.xxx.xxx","5060")) { sl_reply_error(); }; break; };

if ((uri =~ "^sip:888#") || (uri =~ "^sip:444#")) { rewritehostport("xxx.xxx.xxx.xxx"); if (!t_relay_to_udp("xxx.xxx.xxx.xxx","5060")) { sl_reply_error(); }; break; };

if ((uri =~ "^sip:33#") || (uri =~ "^sip:33%23")) { if (uri =~ "^sip:33#") { strip(3); } if (uri =~ "^sip:33%23") { strip(5); } rewritehostport("xxx.xxx.xxx.xxx"); if (!t_relay_to_udp("xxx.xxx.xxx.xxx","5060")) { sl_reply_error(); }; break; };

if ((uri =~ "^sip:*[0-9][0-9]") || (uri =~ "^sip:511")) { rewritehostport("xxx.xxx.xxx.xxx"); if (!t_relay_to_udp("xxx.xxx.xxx.xxx","5060")) { sl_reply_error(); }; break; }; if ((uri =~ "^sip:+17005551111") || (uri =~ "^sip:17005551111")) { rewriteuri("sip:17005551111@xxx.xxx.xxx.xxx:5060"); if (!t_relay_to_udp("xxx.xxx.xxx.xxx","5060")) { sl_reply_error(); }; break; }; if ((uri =~ "^sip:+17005551111") || (uri =~ "^sip:17005551111")) { rewriteuri("sip:17005552222@xxx.xxx.xxx.xxx:5060"); if (!t_relay_to_udp("xxx.xxx.xxx.xxx","5060")) { sl_reply_error(); }; break; };

if ((uri =~ "^sip:411") || (uri =~ "^sip:+411") || (uri =~ "^sip:.*5551212")) { if (uri =~ "^sip:411") { prefix("+"); } rewritehostport("xxx.xxx.xxx.xxx:5060"); if (!t_relay_to_tcp("xxx.xxx.xxx.xxx","5060")) { sl_reply_error(); }; break; };

if ((uri =~ "^sip:911") || (uri =~ "^sip:+911")) { rewriteuri("sip:+911@xxx.xxx.xxx.xxx:5060"); if (!t_relay_to_udp("xxx.xxx.xxx.xxx","5060")) { sl_reply_error(); }; break; };

if (src_ip==xxx.xxx.xxx.xxx || src_ip==xxx.xxx.xxx.xxx || src_ip==xxx.xxx.xxx.xxx || src_ip==xxx.xxx.xxx.xxx || src_ip==xxx.xxx.xxx.xxx) { if (!lookup("location_novm")) { route(2); # Nothing found - Route Off-Net } else { route(1); # Direct match - Route On-Net } } else { # Not originating from a vm box # Lookup location table for uri if (!lookup("location")) { route(2); # Nothing found - Route Off-Net } else { route(1); # Direct match - Route On-Net }; };

}; }

# # No direct route via location or location_novm (Off-Net Route) # route[2] { ## Send 404 Not Found if call originated Off-Net if ((search("^(Via|v):.*xxx.xxx.xxx.xxx")) || (search("^(Via|v):.*xxx.xxx.xxx.xxx")) || (search("^(Via|v):.*xxx.xxx.xxx.xxx")) || (search("^(From|f):.*<sip:+1700.*@")) || (uri =~ "^sip:+1700.*@")) { sl_send_reply("404", "Not Found"); break; };

# Handle Toll-Free Calls if ((uri =~ "^sip:1800") || (uri =~ "^sip:+1800") || (uri =~ "^sip:1888") || (uri =~ "^sip:+1888") || (uri =~ "^sip:1877") || (uri =~ "^sip:+1877") || (uri =~ "^sip:1866") || (uri =~ "^sip:+1866")) { if (uri=~"^sip:1") { prefix("+"); } rewritehostport("xxx.xxx.xxx.xxx:5060"); if (!t_relay_to_tcp("xxx.xxx.xxx.xxx","5060")) { sl_reply_error(); }; break; };

## Handle International Traffic if ((uri=~"^sip:011") || (uri=~"^sip:+011")) {

# Kill calls to 011+1... (invalid dialing) if ((uri=~"^sip:0111") || (uri=~"^sip:+0111")) { sl_send_reply("406", "Not Acceptable"); break; }

## Send ALL Intl Traffic if (uri=~"^sip:+011") { strip(1); } strip(3); prefix("+"); rewritehostport("xxx.xxx.xxx.xxx:5060"); if (!t_relay_to_tcp("xxx.xxx.xxx.xxx","5060")) { sl_reply_error(); };

## Handle 1+ traffic } else {

if ((is_user_in("credentials","local")) || (src_ip==xxx.xxx.xxx.xxx) || (src_ip==xxx.xxx.xxx.xxx)) {

rewritehostport("xxx.xxx.xxx.xxx:5060"); if (!t_relay_to_tcp("xxx.xxx.xxx.xxx","5060")) { sl_reply_error(); };

#Send ALL other Domestic calls to xyz } else {

# 1+ Intl Traffic if ((uri=~"^sip:+1242") || (uri=~"^sip:+1246") || (uri=~"^sip:+1264") || (uri=~"^sip:+1268") || (uri=~"^sip:+1284") || (uri=~"^sip:+1340") || (uri=~"^sip:+1345") || (uri=~"^sip:+1441") || (uri=~"^sip:+1473") || (uri=~"^sip:+1649") || (uri=~"^sip:+1664") || (uri=~"^sip:+1670") || (uri=~"^sip:+1671") || (uri=~"^sip:+1758") || (uri=~"^sip:+1767") || (uri=~"^sip:+1784") || (uri=~"^sip:+1787") || (uri=~"^sip:+1939") || (uri=~"^sip:+1809") || (uri=~"^sip:+1868") || (uri=~"^sip:+1869") || (uri=~"^sip:+1876") || (uri=~"^sip:+1684")) {

rewritehostport("xxx.xxx.xxx.xxx:5060"); if (!t_relay_to_udp("xxx.xxx.xxx.xxx","5060")) { sl_reply_error(); }; break; };

# Canada Traffic if ((uri=~"^sip:+1204") || (uri=~"^sip:+1250") || (uri=~"^sip:+1289") || (uri=~"^sip:+1306") || (uri=~"^sip:+1402") || (uri=~"^sip:+1403") || (uri=~"^sip:+1416") || (uri=~"^sip:+1418") || (uri=~"^sip:+1450") || (uri=~"^sip:+1506") || (uri=~"^sip:+1514") || (uri=~"^sip:+1519") || (uri=~"^sip:+1604") || (uri=~"^sip:+1605") || (uri=~"^sip:+1613") || (uri=~"^sip:+1647") || (uri=~"^sip:+1705") || (uri=~"^sip:+1708") || (uri=~"^sip:+1709") || (uri=~"^sip:+1778") || (uri=~"^sip:+1780") || (uri=~"^sip:+1807") || (uri=~"^sip:+1819") || (uri=~"^sip:+1867") || (uri=~"^sip:+1902") || (uri=~"^sip:+1905") || (uri=~"^sip:+1906")) {

Since rewritehostport("xxx.xxx.xxx.xxx:5060"); if (!t_relay_to_udp("xxx.xxx.xxx.xxx","5060")) { sl_reply_error(); }; break; };

## Force Tech Support Test Line 2 if (search("^(f|From):.*<sip:+17005553333@")) { rewritehostport("xxx.xxx.xxx.xxx:5060"); if (!t_relay_to_tcp("xxx.xxx.xxx.xxx","5060")) { sl_reply_error(); }; break; };

# Route all Domestic Continental US Calls rewritehostport("xxx.xxx.xxx.xxx:5060"); if (!t_relay_to_udp("xxx.xxx.xxx.xxx","5060")) { sl_reply_error(); }; break;

}; }; }

# # Direct route found via location or location_novm (On-Net Route) # route[1] { # !! Nathelper if (uri=~"[@:](192.168.|10.|172.(1[6-9]|2[0-9]|3[0-1]).)" && !search("^Route:")) { sl_send_reply("479", "We don't forward to private IP addresses"); break; };

# if client or server know to be behind a NAT, enable relay if (isflagset(6)) { # NAT processing of replies; apply to all transactions (for a example, # re-INVITEs from public to private UA are hard to identify as # NATed at the moment of request processing); look at replies t_on_reply("1"); append_hf("P-Behind-NAT: Yes\r\n"); };

setflag(3);

# send it out now; use stateful forwarding as it works reliably # even for UDP2TCP if (!t_relay()) { sl_reply_error(); }; }

onreply_route[1] { if (src_ip != xxx.xxx.xxx.xxx) { if (status =~ "(183)|(180)|2[0-9][0-9]") { fix_nated_contact(); }; if (nat_uac_test("1")) { fix_nated_contact(); }; break; }; }in

-----Original Message----- From: Jiri Kuthan [mailto:jiri@iptel.org] Sent: Saturday, August 19, 2006 7:03 PM To: Darren Sessions; serusers@lists.iptel.org Cc: George Law; James Bailey Subject: Re: [Serusers] SER / Missing call records

At 17:08 18/08/2006, Darren Sessions wrote:

Content-Type: multipart/alternative; boundary="----_=_NextPart_001_01C6C2D8.5D25DDC8" Content-class: urn:content-classes:message

All,

We're experiencing a problem with missing call records. There is

nothing in the log files that indicate a problem either with SER or our database server and likewise there is nothing in the database server's logs to indicate a problem. System loads are within reason on both servers, network connectivity has been tested and verified, and a through hardware shakedown on both boxes has been completed without any errors.

We first noticed this problem when our underlying vendor's bill (based

on minutes of course) did not correspond with our own. Upon doing a CDR comparison (based on what was in both the acc and calls tables), we were able to verify that the calls in dispute were indeed ours (based on the sip call-id's IP addresses).

We're processing roughly 1.4 million calls per day on this SER server

in question. Out of those calls, 508 calls (give or take a couple hundred) with a total duration of over 5,300 minutes - can not be accounted for in any of the SER database tables.

We're running 0.9.5-pre1 with Suse Enterprise Server 9.0 (2.6.11

Kernel). Server is an HP DL380 with dual 3.2GHz processors with 6GB physical ram (only 1.4GB in use system wide); disk space is not an issue as well (only 10% in use).

We've scoured the mailing lists and change logs for any hint of an

issue similar to ours without any results. The closest we've been able to come that's anywhere near along the lines of our issue in any way that made any sense were these posts:

http://lists.iptel.org/pipermail/serdev/2003-August/000602.htmlhttp:/

/lists.iptel.org/pipermail/serdev/2003-August/000602.html

thats usrloc specific, unrelated to accounting.

http://lists.iptel.org/pipermail/serusers/2005-February/015102.htmlht

tp://lists.iptel.org/pipermail/serusers/2005-February/015102.html

This may be related, even though it is hard to say what the problem cause might have been.

I don't know your script and use case, but I would probably begin to suspect the script whether it correctly identifies all transactions in question to be accounted. Maybe, if you don't net do it, it may be reasonable to account all SIP transactions passing your server.

Second next thing I would do is to look at syslog. I remember that under some stressful mysql conditions, some INSERTs really got lost, which was properly logged in SER's logfiles/.

At this point, we're not sure if this issue has something to do with

the FIFO and SER making bulk dump uploads with call data to the database, or some other issue along those lines - or if this is a new issue all together.

That has hardly to do with FIFO.

-jiri