<div dir="ltr">Hello,<br>we have openser 1.3.3 running in production (current rev.: 4943).<br>For 3 times in 50 days we had to restart openser to correct pkg memory problem.<br>After some time logging messages like this:<br>
/openser.log:Aug 19 10:39:18 ipx022 /usr/local/sbin/openser[16991]: ERROR:core:new_credentials: no pkg memory left,<br>openser will eventually run out of pkg memory and refuse all subsequent requests.<br><br>We are trying to recreate this in our lab so that we can follow memory troubleshooting instructions at <a href="http://kamailio.net/dokuwiki/doku.php/troubleshooting:memory" target="_blank">http://kamailio.net/dokuwiki/doku.php/troubleshooting:memory</a>, but so far we were unable to do it even when generating millions of calls and registration transactions (we are using SIPp to generate normal call flows and even abnormal call flows detected when reading openser.log, like 'invalid cseq for aor', malformed SIP messages etc).<br>
And this is much more than in our production environment, with just 600 subscribers and about 2000 calls a day.<br><br>The frequency the problem happens is increasing with the number of
subscribers, so we are performing periodic restart of openser (actually,
what we do is to switch over to the standby server). We already recompiled openser
with pkg memory pool size set to 4MB so that this will not have to be
done frequently.<br><br>Since we cannot recreate this in our lab, we suspect there is a situation happening in production that might not be having been properly handled by openser.cfg. So my question is: would it be possible to an overlooked detail in openser.cfg to cause pkg memory problem? <br>
<br><br>In case someone could take a look at it, here's our cfg file:<br><br>####### Global Parameters #########<br><br>debug=0<br>log_stderror=no<br>log_facility=LOG_LOCAL0<br><br>fork=yes<br>children=4<br><br>/* uncomment the following lines to enable debugging */<br>
#debug=6<br>#fork=no<br>#log_stderror=yes<br><br>/* uncomment the next line to disable TCP (default on) */<br>disable_tcp=yes<br><br>/* uncomment the next line to enable the auto temporary blacklisting of <br> not available destinations (default disabled) */<br>
#disable_dns_blacklist=no<br><br>/* uncomment the next line to enable IPv6 lookup after IPv4 dns <br> lookup failures (default disabled) */<br>#dns_try_ipv6=yes<br><br>/* uncomment the next line to disable the auto discovery of local aliases<br>
based on revers DNS on IPs (default on) */<br>#auto_aliases=no<br><br>/* uncomment the following lines to enable TLS support (default off) */<br>#disable_tls = no<br>#listen = tls:your_IP:5061<br>#tls_verify_server = 1<br>
#tls_verify_client = 1<br>#tls_require_client_certificate = 0<br>#tls_method = TLSv1<br>#tls_certificate = "/usr/local/etc/openser/tls/user/user-cert.pem"<br>#tls_private_key = "/usr/local/etc/openser/tls/user/user-privkey.pem"<br>
#tls_ca_list = "/usr/local/etc/openser/tls/user/user-calist.pem"<br><br><br>port=5060<br><br>/* uncomment and configure the following line if you want openser to <br> bind on a specific interface/port/proto (default bind on all available) */<br>
#listen=udp:<a href="http://202.173.5.181:5060" target="_blank">202.173.5.181:5060</a><br><br><br>####### Modules Section ########<br><br>#set module path<br>mpath="/usr/local/lib/openser/modules/"<br><br>/* uncomment next line for MySQL DB support */<br>
loadmodule "mysql.so"<br>loadmodule "sl.so"<br>loadmodule "tm.so"<br>loadmodule "rr.so"<br>loadmodule "maxfwd.so"<br>loadmodule "usrloc.so"<br>loadmodule "registrar.so"<br>
loadmodule "textops.so"<br>loadmodule "mi_fifo.so"<br>loadmodule "uri_db.so"<br>loadmodule "uri.so"<br>loadmodule "xlog.so"<br>loadmodule "acc.so"<br>loadmodule "carrierroute.so"<br>
loadmodule "nathelper.so"<br>loadmodule "dialog.so"<br>loadmodule "snmpstats.so"<br>/* uncomment next lines for MySQL based authentication support <br> NOTE: a DB (like mysql) module must be also loaded */<br>
loadmodule "auth.so"<br>loadmodule "auth_db.so"<br>loadmodule "lcr.so"<br>/* uncomment next line for aliases support<br> NOTE: a DB (like mysql) module must be also loaded */<br>loadmodule "alias_db.so"<br>
/* uncomment next line for multi-domain support<br> NOTE: a DB (like mysql) module must be also loaded<br> NOTE: be sure and enable multi-domain support in all used modules<br> (see "multi-module params" section ) */<br>
loadmodule "domain.so"<br>/* uncomment the next two lines for presence server support<br> NOTE: a DB (like mysql) module must be also loaded */<br>#loadmodule "presence.so"<br>#loadmodule "presence_xml.so"<br>
loadmodule "uac.so"<br>loadmodule "avpops.so"<br># ----------------- setting module-specific parameters ---------------<br><br><br># ----- mi_fifo params -----<br>modparam("mi_fifo", "fifo_name", "/tmp/openser_fifo")<br>
<br><br># ----- rr params -----<br># add value to ;lr param to cope with most of the UAs<br>modparam("rr", "enable_full_lr", 1)<br># do not append from tag to the RR (no need for this script)<br>modparam("rr", "append_fromtag", 1)<br>
<br><br># ----- rr params -----<br>modparam("registrar", "method_filtering", 1)<br>/* uncomment the next line to disable parallel forking via location */<br># modparam("registrar", "append_branches", 0)<br>
/* uncomment the next line not to allow more than 10 contacts per AOR */<br>modparam("registrar", "max_contacts", 10)<br>modparam("registrar", "min_expires", 30)<br>modparam("registrar", "max_expires", 40)<br>
modparam("registrar", "default_expires", 35)<br># ----- uri_db params -----<br>/* by default we disable the DB support in the module as we do not need it<br> in this configuration */<br>modparam("uri_db", "use_uri_table", 0)<br>
modparam("uri_db", "db_url", "mysql://openser:openserrw@localhost/openser")<br>modparam("uri_db", "use_domain", 1)<br><br># ----- acc params -----<br>/* what sepcial events should be accounted ? */<br>
modparam("acc", "early_media", 1)<br>modparam("acc", "report_ack", 1)<br>modparam("acc", "report_cancels", 1)<br>/* by default ww do not adjust the direct of the sequential requests.<br>
if you enable this parameter, be sure the enable "append_fromtag"<br> in "rr" module */<br>modparam("acc", "detect_direction", 0)<br>/* account triggers (flags) */<br>modparam("acc", "failed_transaction_flag", 3)<br>
modparam("acc", "log_flag", 1)<br>modparam("acc", "log_missed_flag", 2)<br>/* uncomment the following lines to enable DB accounting also */<br>modparam("acc", "db_flag", 1)<br>
modparam("acc", "db_missed_flag", 2)<br><br><br># ----- usrloc params -----<br>#modparam("usrloc", "db_mode", 0)<br>/* uncomment the following lines if you want to enable DB persistency<br>
for location entries */<br>modparam("usrloc", "db_mode", 2)<br>modparam("usrloc", "db_url", "mysql://openser:openserrw@localhost/openser")<br>modparam("usrloc", "use_domain", 1)<br>
<br># ----- auth_db params -----<br>/* uncomment the following lines if you want to enable the DB based<br> authentication */<br>modparam("auth_db", "calculate_ha1", yes)<br>modparam("auth_db", "password_column", "password")<br>
modparam("auth_db", "db_url", "mysql://openser:openserrw@localhost/openser")<br>modparam("auth_db", "load_credentials", "$avp(s:rpid)=rpid;$avp(s:blocked)=subscriber_status")<br>
<br><br># ----- alias_db params -----<br>/* uncomment the following lines if you want to enable the DB based<br> aliases */<br>modparam("alias_db", "db_url",<br> "mysql://openser:openserrw@localhost/openser")<br>
modparam("alias_db", "use_domain", 0)<br><br># ----- domain params -----<br>/* uncomment the following lines to enable multi-domain detection<br> support */<br>modparam("domain", "db_url", "mysql://openser:openserrw@localhost/openser")<br>
modparam("domain", "db_mode", 1) # Use caching<br><br><br># ----- multi-module params -----<br>/* uncomment the following line if you want to enable multi-domain support<br> in the modules (dafault off) */<br>
#modparam("alias_db|auth_db|usrloc|uri_db", "use_domain", 1)<br><br><br># ----- presence params -----<br>/* uncomment the following lines if you want to enable presence */<br>#modparam("presence|presence_xml", "db_url",<br>
# "mysql://openser:openserrw@localhost/openser")<br>#modparam("presence_xml", "force_active", 1)<br>#modparam("presence", "server_address", "sip:<a href="http://192.168.1.2:5060" target="_blank">192.168.1.2:5060</a>")<br>
<br># ----- carrieroute params -----<br>modparam("carrierroute", "db_url", "mysql://openser:openserrw@localhost/openser")<br>modparam("carrierroute", "config_source", "db")<br>
modparam("carrierroute", "use_domain", 1)<br># ----- NatHelper -----<br>#para versao a partir da versao 1.2 eh necessario esse paramtro para nao dar erro qdo usa a funcao "fix_nated_register();"<br>
modparam("nathelper|registrar", "received_avp", "$avp(i:42)")<br>modparam("nathelper", "rtpproxy_sock", "udp:<a href="http://127.0.0.1:22222" target="_blank">127.0.0.1:22222</a>")<br>
<br><br># ----- LCR -----<br>modparam("lcr", "db_url", "mysql://openser:openserrw@localhost/openser")<br>modparam("lcr|tm", "fr_inv_timer_avp", "$avp(i:704)")<br>
modparam("lcr", "gw_uri_avp", "$avp(i:709)")<br>modparam("^auth$|lcr", "rpid_avp", "$avp(i:302)")<br>modparam("lcr", "contact_avp", "$avp(i:711)")<br>
modparam("lcr", "ruri_user_avp", "$avp(i:500)")<br>modparam("lcr", "dm_flag", 25)<br><br># ----- Dialog ----<br>modparam("dialog", "dlg_flag", 4)<br><br>
# ----- SnmpStat -----<br>modparam("snmpstats", "sipEntityType", "registrarServer")<br>modparam("snmpstats", "sipEntityType", "proxyServer")<br>modparam("snmpstats", "MsgQueueMinorThreshold", 2000)<br>
modparam("snmpstats", "MsgQueueMajorThreshold", 5000)<br>modparam("snmpstats", "dlg_minor_threshold", 500)<br>modparam("snmpstats", "dlg_major_threshold", 750)<br>
modparam("snmpstats", "snmpgetPath","/usr/bin/")<br>modparam("snmpstats", "snmpCommunity","public")<br><br><br><br>####### Routing Logic ########<br><br><br># main request routing logic<br>
<br>route{<br><br> if (!mf_process_maxfwd_header("10")) {<br> sl_send_reply("483","Too Many Hops");<br> exit;<br> }<br><br> ##nat<br> route(2);<br><br> if (has_totag()) {<br>
# sequential request withing a dialog should<br> # take the path determined by record-routing<br> if (loose_route()) {<br> if (is_method("BYE")) {<br> setflag(1); # do accouting ...<br>
setflag(3); # ... even if the transaction fails<br> }<br> route(1);<br> } else {<br> /* uncomment the following lines if you want to enable presence */<br> ##if (is_method("SUBSCRIBE") && $rd == "your.server.ip.address") {<br>
## # in-dialog subscribe requests<br> ## route(2);<br> ## exit;<br> ##}<br> if ( is_method("ACK") ) {<br> if ( t_check_trans() ) {<br>
# non loose-route, but stateful ACK; must be an ACK after a 487 or e.g. 404 from upstream server<br> #t_relay();<br> #exit;<br> route(1);<br> } else {<br>
# ACK without matching transaction ... ignore and discard.\n");<br> exit;<br> }<br> }<br> sl_send_reply("404","Not here");<br>
}<br> exit;<br> }<br><br> #initial requests<br> setflag(4); #for dialog statistics<br><br><br> # CANCEL processing<br> if (is_method("CANCEL"))<br> {<br> if (t_check_trans()) route(1);<br>
# t_relay();<br> # exit;<br> }<br><br> #t_check_trans();<br><br> if (is_method("PUBLISH|SUBSCRIBE|REFER|OPTIONS|MESSAGE"))<br> {<br> sl_send_reply("405", "Method not allowed");<br>
exit;<br> }<br><br> # authenticate if from local subscriber (uncomment to enable auth)<br> if (!(method=="REGISTER") && (!from_gw()))<br> {<br> if (!proxy_authorize("", "subscriber")) {<br>
proxy_challenge("", "0");<br> exit;<br> }<br> if (!check_from()) {<br> sl_send_reply("403","Forbidden auth ID");<br> exit;<br>
}else if (avp_check("$avp(s:blocked)", "eq/0")) {<br> sl_send_reply("603","Subscriber disabled");<br> exit;<br> }else if (avp_check("$avp(s:blocked)", "eq/1")) {<br>
sl_send_reply("603","Subscriber with outgoing blocked");<br> exit; <br> }<br><br> consume_credentials();<br> # caller authenticated<br>
}<br><br> # record routing<br> if (!is_method("REGISTER|MESSAGE"))<br> record_route();<br><br> # account only INVITEs<br> if (is_method("INVITE")) {<br> setflag(1); # do accouting<br>
}<br><br> if (is_method("REGISTER"))<br> {<br> # authenticate the REGISTER requests (uncomment to enable auth)<br> if (!proxy_authorize("", "subscriber"))<br> {<br>
proxy_challenge("", "0");<br> exit;<br> }<br> <br> if (!check_to()) <br> {<br> sl_send_reply("403","Forbidden auth ID");<br>
exit;<br> }else if (avp_check("$avp(s:blocked)", "eq/0")) {<br> sl_send_reply("403","Subscriber disabled");<br> exit;<br>
} <br><br> if (!save("location"))<br> sl_reply_error();<br><br> exit;<br> }<br> <br> if ($rU==NULL) {<br> # request with no Username in RURI<br> sl_send_reply("484","Address Incomplete");<br>
exit;<br> }<br><br> # apply DB based aliases (uncomment to enable)<br> ##alias_db_lookup("dbaliases");<br> #if the call came from a known gateway it is not authenticated and we cannot use the function check_from()<br>
if (from_gw()) {<br> route(4);<br> }else if (!check_from()) {#if the check_from() returns false the call is not from a subscriber<br> route(4); <br> } else {#it is a subscriber, route using flip domain<br>
xlog("L_INFO", "routing using carrierroute $rm to $ru\n");<br> if (!cr_user_rewrite_uri("$fu", "flip"))<br> {<br> t_newtran();<br> t_reply("404", "No Route");<br>
exit;<br> }<br> #replaces from by it's default DID<br> uac_replace_from("sip:$avp(s:rpid)@$fd");<br> }<br><br> # when routing via usrloc, log the missed calls also<br>
setflag(2);<br><br> route(1);<br>}<br><br>route[1] {<br> xlog("L_INFO", "ROUTE_1 $rm to $ru\n");<br> if (subst_uri('/(sip:.*);nat=yes/\1/')) <br> {<br> setflag(6);<br>
};<br><br> if (isflagset(5)||isflagset(6)) {<br> route(3);<br> }<br> <br> if (!t_relay()) {<br> sl_reply_error();<br> };<br> exit;<br>}<br><br>
route[2] {<br> xlog("L_INFO", "ROUTE_2 $rm to $ru\n");<br> if (method=="REGISTER") {<br> fix_nated_register();<br> } else if (!from_gw()){<br> fix_nated_contact();<br>
};<br> setflag(5);<br>}<br>route[3] {<br> xlog("L_INFO", "ROUTE_3 $rm to $ru\n");<br> if (is_method("BYE|CANCEL")) {<br> unforce_rtp_proxy();<br> } else if (is_method("INVITE")) {<br>
xlog("L_INFO", "FORCE RTP w/ parameter.\n");<br> force_rtp_proxy("r");<br> t_on_failure("1");<br> };<br> if (isflagset(5))<br>
search_append('Contact:.*sip:[^>[:cntrl:]]*', ';nat=yes');<br> t_on_reply("1");<br>}<br><br>route[4] {<br><br> xlog("L_INFO", "uri does exist $rm to $ru \n");<br>
if (alias_db_lookup("dbaliases")){<br><br> if (!lookup("location")) {<br> switch ($retcode) {<br> case -1:<br> t_newtran();<br>
t_reply("404", "Subscriber not online");<br> exit;<br> case -2:<br> sl_send_reply("405", "Method Not Allowed");<br>
exit;<br> }<br> }<br> }else{#check if did is blocked<br> $rU = "(BLK)" + $rU;<br> if (alias_db_lookup("dbaliases")){<br>
sl_send_reply("403", "DID blocked");<br> exit;<br><br> }else{# if it is not a valid DID nor a blocked DID tries to route it using peering domain<br> if (!cr_rewrite_uri("peering", "call_id"))<br>
{<br> t_newtran();<br> t_reply("404", "Peering Not Found");<br> exit;<br> }<br>
<br> }<br> }<br>}<br><br>failure_route[1] {<br> xlog("L_INFO", "FAILURE $rm to $ru\n");<br> if (isflagset(6)||isflagset(5)) {<br> unforce_rtp_proxy();<br> }<br>
}<br><br>onreply_route[1] {<br> xlog("L_INFO", "ONREPLY_1 - Status $rs from $si $rm .\n");<br> if (is_method("INVITE")) {<br> if ((isflagset(5)||isflagset(6)) && status=~"(183)|(2[0-9][0-9])") {<br>
force_rtp_proxy();<br> }<br> search_append('Contact:.*sip:[^>[:cntrl:]]*', ';nat=yes');<br><br> if (!from_gw()){ #if (isflagset(6)) {<br>
xlog("L_INFO", "ONREPLY_1 - ! from gw.\n");<br> fix_nated_contact();<br> }<br> exit;<br> }<br>}<br><br>Regards,<br>takeshi<br><br><br></div>