[Users] tls. soket queues not flushed

Wed Oct 4 16:28:57 CEST 2006

Hello,

I'm sorry for a big lag. I test system yesterday on other linux kernel (2.6) and obtain the same results. In this case i suppose, 
that this problem don't bases on system socket implementation. I tested openser on other numbers of tcp workers and obtain next results:
20 children: 2 sockets don't flushed (all had data on receive and send queues), cpu usage is above 48% on each of 2 openser processes, 
one of them invokes poll function in tls_server.c, other trying to write into tls socket.
1 child: 1 socket don't flushed (receive queue not empty, send queue empty), cpu usage is 90% on one process.
100 children: no problems, all work correctly
2 children: no problems.

>Hello,
>
>On 09/29/06 16:06, serega wrote:
>> Hi Klaus and all other who read this.
>>
>> It's not a problem of tcp socket - yesterday i rewrite gateway config for fowarding directed to extgateway requests to gateway itself.
>> In tcp this problems not occured, but after i switched on tls, i see overfilled queues again. In that case i haven't extgateway, communications with 
>> remote sockets and etc. Did anybody test openser whith big payload? I think this problem occurs because in my case i have big traffic and several process that simultaniously use tls. Did anybody have the same problem?
>>   
>
>If you increase the number of the TLS/TCP workers do you get same 
>results? In case of TLS, there is more CPU used because of 
>encryption/decryption. How is the CPU usage in teh system while you are 
>testing?
>
>As Klaus said, the management of socket queue is done by OS.
>
>Cheers,
>Daniel
>
>>   
>>> Hi Serega!
>>>
>>> I'm not sure - just a guess. openser uses "worker threads". This worker
>>> threads get a SIP message from the listener thread, process them, and
>>> then they send them.
>>>
>>> If for some reason the sending fails - e.g. the gateway sends via
>>> TCP/TLS to extgateway and extgateway's receiving queue is full. Then
>>> gateway's sending buffer gets full and the threads wait until they can
>>> send. Thus, as the threads are all busy, also the receiving queue of
>>> gateway gets full. I guess after some time the threads give up sending
>>> and will read again.
>>>
>>> The question is: why does the sending buffer gets full? Where is gateway
>>> sending the messages to?
>>>
>>> The sending queue must not get flushed. The queue is in the OS, not in 
>>> openser. As TCP guarantees that there is no loss/reordering the TCP 
>>> stack must not flush the queue.
>>>
>>> regards
>>> klaus
>>>
>>> serega wrote:
>>>     
>>>> Hi all.
>>>>
>>>> I have next problem while using openser with tls. System consists of
>>>> sip statefull server (in next time, simply, server) and stateless sip
>>>> gateway (gateway). Server used for connect to jabber server. Server
>>>> contains rewrote jabber module logic and also use presence module.
>>>> Gateway used to connect to other sip gateway (extgateway) via tls
>>>> protocol. Gateway use 1.1 openser version and doesn’t contain changed
>>>> code. When server together with gateway restarts and server in its
>>>> database contains above 260 subscriptions (in watcherinfo table) i
>>>> have error. Socket used by gateway to connect with extgateway
>>>> contains in receive and sent queues a lot of data (above 50kByte on
>>>> each sides). This data never flushed out. This happens because after
>>>> restart sip server through gateway send notifications to extgateway.
>>>>  When I attached using gdb to process that send data, I saw that it
>>>> was in infinity loop because tls library returns SSL_ERROR_WANT_WRITE
>>>> and I think it’s correct because we have overfilled send queue. In
>>>> this case I don’t interest why receive queue not empty (I think it
>>>> happens because sending process have got block on socket). But I
>>>> don’t understand why sent queue not flushed. I test this behavior
>>>> using tcp – all was correct. Socket not closed by other side because
>>>> this status can be unchanged above one day. Anybody can help me? Why
>>>> this happens?
>>>>
>>>> I use openssl-0.9.8c and redhat os (Linux xdevel1 2.4.21-4.ELsmp #1
>>>> SMP Fri Oct 3 17:52:56 EDT 2003 i686 i686 i386 GNU/Linux). Computer
>>>> have too Pentium 4 processors).
>>>>
>>>>
>>>> Gateway.cfg (sip.qa.hbex.com – gateway address):
>>>>
>>>> # # $Id: router_qa.cfg,v 1.1 2006/08/02 18:14:34 ilya Exp $ # #
>>>> simple quick-start config script #
>>>>
>>>> # ----------- global configuration parameters
>>>> ------------------------
>>>>
>>>> debug=9            # debug level (cmd line: -dddddddddd) 
>>>> log_facility=LOG_LOCAL0 fork=yes log_stderror=yes    # (cmd line: -E)
>>>>
>>>>
>>>> check_via=no	# (cmd. line: -v) dns=no          # (cmd. line: -r) 
>>>> rev_dns=no      # (cmd. line: -R) children=20 
>>>> fifo="/tmp/openser_fifo" server_signature=no
>>>>
>>>> fifo_db_url="mysql://root:@localhost/openser"
>>>>
>>>> listen = udp:sip.qa.hbex.com:5060 #incoming sip server address listen
>>>> = tcp:sip.qa.hbex.com:5060
>>>>
>>>> #tls address disable_tls = 0 listen = tls:sip.qa.hbex.com:5061 listen
>>>> = udp:sip.qa.hbex.com:5061
>>>>
>>>> tls_certificate = "/ home/inop/ex.com-cert.pem" tls_private_key =
>>>> "/home/inop/ex.com-privkey.pem" tls_ca_list =
>>>> "/home/inop/ex.com-calist.pem" tls_require_client_certificate=0 
>>>> tls_verify_client=0 tls_verify_server=0
>>>>
>>>> # ------------------ module loading
>>>> ---------------------------------- loadmodule
>>>> "/home/interop/openser/lib/openser/modules/rr.so" loadmodule
>>>> "/home/interop/openser/lib/openser/modules/xlog.so" loadmodule
>>>> "/home/interop/openser/lib/openser/modules/textops.so" loadmodule
>>>> "/home/interop/openser/lib/openser/modules/maxfwd.so" loadmodule
>>>> "/home/interop/openser/lib/openser/modules/sl.so" loadmodule
>>>> "/home/interop/openser/lib/openser/modules/mysql.so" loadmodule
>>>> "/home/interop/openser/lib/openser/modules/tm_unchanged.so" 
>>>> loadmodule "/home/interop/openser/lib/openser/modules/usrloc.so" 
>>>> loadmodule "/home/interop/openser/lib/openser/modules/registrar.so"
>>>>
>>>> # ----------------- setting module-specific parameters
>>>> --------------- modparam("registrar", "default_expires", 120) 
>>>> modparam("registrar", "use_domain", 1)
>>>>
>>>> modparam("usrloc", "use_domain", 1) modparam("usrloc", "db_mode",
>>>> 0) # Uncomment this if you want to use SQL database # for persistent
>>>> storage and comment the previous line #modparam("usrloc", "db_mode",
>>>> 2)
>>>>
>>>> # add value to ;lr param to make some broken UAs happy modparam("rr",
>>>> "enable_full_lr", 1) modparam("maxfwd", "max_limit", 10) #
>>>> -------------------------  request routing logic ------------------- 
>>>> # main routing logic route{ if (!mf_process_maxfwd_header("10")) { 
>>>> sl_send_reply("483","To Many Hops"); drop(); }; if (dst_port==5061) {
>>>>  if (search("^To:.*@.*cmnicator.com") || search("^To:.*@.*cihb.com"))
>>>> { # rewrite destination and forward to jabber (sip server) route(1); 
>>>> return; }; sl_reply_error(); return; } if (method=="REGISTER") { 
>>>> xlog("XXX: saving location msg=$mb\n"); if(!save_noreply("location"))
>>>> { log("XXX: Error saving location!\n"); sl_reply_error(); } 
>>>> sl_send_reply("200","OK"); return; } if
>>>> (search("^To:.*@.*cmnicator.com") || search("^To:.*@.*cihb.com")) { 
>>>> #forward to self route(3); return; } ##forward to ext gateway 
>>>> route(2); }
>>>>
>>>> route[1]{ if (method!="MESSAGE" && method!="SUBSCRIBE" &&
>>>> method!="NOTIFY") { log("XXX: Request not forwarded to sip
>>>> server!\n"); sl_send_reply("202","Accepted"); return; } 
>>>> t_on_reply("1"); record_route();
>>>>
>>>> if(!lookup("location")) { if (method!="SUBSCRIBE") { log("XXX: only
>>>> subscribes are processed for user that are not registered\n"); 
>>>> sl_reply_error(); return; } 
>>>> subst_uri('/(.*)@(.*)/\1#\2*jabber.qa.im.hb.com at jabber01.qa.im.hb.com/ig');
>>>>  if(!t_relay()) { log("XXX: error forwarding jabber01...\n"); 
>>>> sl_reply_error(); return; } else { sl_send_reply("200","OK"); return;
>>>>  } } else { log("XXX: forwarding to the address of record...\n"); 
>>>> if(!t_relay()) { log("XXX: error forwarding to address of record
>>>> \n"); sl_reply_error(); return; } else { sl_send_reply("200", "OK"); 
>>>> return; } }
>>>>
>>>> }
>>>>
>>>> # forwarding to external gateway route[2]{ log("XXX: rewriting
>>>> headers\n"); 
>>>> subst('/^(From:[^@#]*)#([^@]*)\*jabber.qa.im.hb.com at jabber...qa.im.hb.com(.*)/\1@\2\3/ig');
>>>>  
>>>> subst('/^(Contact:[^@#]*)#([^@]*)\*jabber.qa.im.hb.com at jabber...qa.im.hb.com(.*)/\1@\2\3/ig');
>>>>
>>>>
>>>> if(!forward("tls:43.123.141.166:3000")) { log("XXX: Error forwarding
>>>> to external gateway!\n"); sl_reply_error(); } else { 
>>>> sl_send_reply("200", "Accepted"); }; }
>>>>
>>>> # forwarding to SELF route[3]{ log("XXX: rewriting headers\n"); 
>>>> subst('/^(From:[^@#]*)#([^@]*)\*jabber.qa.im.hb.com at jabber...qa.im.hb.com(.*)/\1@\2\3/ig');
>>>>  
>>>> subst('/^(Contact:[^@#]*)#([^@]*)\*jabber.qa.im.hb.com at jabber...qa.im.hb.com(.*)/\1@\2\3/ig');
>>>>
>>>>
>>>> log("XXX: about to forward to self\n"); 
>>>> if(!forward("tcp:sip.qa.hbex.com:5061")) { log("XXX: Error forwarding
>>>> to self!\n"); sl_reply_error(); } else { sl_send_reply("200",
>>>> "Accepted"); }; }
>>>>
>>>> # this is executed for replies onreply_route[1]{ 
>>>> subst('/^(To:[^@#]*)#([^@]*)\*jabber.qa.im.hb.com at jabber...qa.im.hb.com(.*)/\1@\2\3/ig');
>>>>  
>>>> subst('/^(Contact:[^@#]*)#([^@]*)\*jabber.qa.im.hb.com at jabber...qa.im.hb.com(.*)/\1@\2\3/ig');
>>>>  route(2); }
>>>>
>>>>
>>>>
>>>> _______________________________________________ Users mailing list 
>>>> Users at openser.org http://openser.org/cgi-bin/mailman/listinfo/users
>>>>       
>>>     
>>
>>
>> _______________________________________________
>> Users mailing list
>> Users at openser.org
>> http://openser.org/cgi-bin/mailman/listinfo/users
>>
>>   
>