[Users] tls. soket queues not flushed

Klaus Darilion klaus.mailinglists at pernau.at
Thu Oct 5 10:28:58 CEST 2006 

Hi!

Do the queues get flushed after some time (some minutes, hours)?

How do you generate the TCP traffic? I once used sipp and had similar 
problems, but suspected sipp to be the bad guy.

If a sending queue does not get flushed, this means the remote peer of 
this TCP connection has set the TCP windows size to zero (meaning that 
the receiving buffer on the remote peer is full). can you check the 
receiving buffer at the remote peer?

regards
klaus



serega wrote:
> Hello,
> 
> I'm sorry for a big lag. I test system yesterday on other linux
> kernel (2.6) and obtain the same results. In this case i suppose, 
> that this problem don't bases on system socket implementation. I
> tested openser on other numbers of tcp workers and obtain next
> results: 20 children: 2 sockets don't flushed (all had data on
> receive and send queues), cpu usage is above 48% on each of 2 openser
> processes, one of them invokes poll function in tls_server.c, other
> trying to write into tls socket. 1 child: 1 socket don't flushed
> (receive queue not empty, send queue empty), cpu usage is 90% on one
> process. 100 children: no problems, all work correctly 2 children: no
> problems.
> 
> 
>> Hello,
>> 
>> On 09/29/06 16:06, serega wrote:
>>> Hi Klaus and all other who read this.
>>> 
>>> It's not a problem of tcp socket - yesterday i rewrite gateway
>>> config for fowarding directed to extgateway requests to gateway
>>> itself. In tcp this problems not occured, but after i switched on
>>> tls, i see overfilled queues again. In that case i haven't
>>> extgateway, communications with remote sockets and etc. Did
>>> anybody test openser whith big payload? I think this problem
>>> occurs because in my case i have big traffic and several process
>>> that simultaniously use tls. Did anybody have the same problem?
>>> 
>> If you increase the number of the TLS/TCP workers do you get same 
>> results? In case of TLS, there is more CPU used because of 
>> encryption/decryption. How is the CPU usage in teh system while you
>> are testing?
>> 
>> As Klaus said, the management of socket queue is done by OS.
>> 
>> Cheers, Daniel
>> 
>>> 
>>>> Hi Serega!
>>>> 
>>>> I'm not sure - just a guess. openser uses "worker threads".
>>>> This worker threads get a SIP message from the listener thread,
>>>> process them, and then they send them.
>>>> 
>>>> If for some reason the sending fails - e.g. the gateway sends
>>>> via TCP/TLS to extgateway and extgateway's receiving queue is
>>>> full. Then gateway's sending buffer gets full and the threads
>>>> wait until they can send. Thus, as the threads are all busy,
>>>> also the receiving queue of gateway gets full. I guess after
>>>> some time the threads give up sending and will read again.
>>>> 
>>>> The question is: why does the sending buffer gets full? Where
>>>> is gateway sending the messages to?
>>>> 
>>>> The sending queue must not get flushed. The queue is in the OS,
>>>> not in openser. As TCP guarantees that there is no
>>>> loss/reordering the TCP stack must not flush the queue.
>>>> 
>>>> regards klaus
>>>> 
>>>> serega wrote:
>>>> 
>>>>> Hi all.
>>>>> 
>>>>> I have next problem while using openser with tls. System
>>>>> consists of sip statefull server (in next time, simply,
>>>>> server) and stateless sip gateway (gateway). Server used for
>>>>> connect to jabber server. Server contains rewrote jabber
>>>>> module logic and also use presence module. Gateway used to
>>>>> connect to other sip gateway (extgateway) via tls protocol.
>>>>> Gateway use 1.1 openser version and doesn’t contain changed 
>>>>> code. When server together with gateway restarts and server
>>>>> in its database contains above 260 subscriptions (in
>>>>> watcherinfo table) i have error. Socket used by gateway to
>>>>> connect with extgateway contains in receive and sent queues a
>>>>> lot of data (above 50kByte on each sides). This data never
>>>>> flushed out. This happens because after restart sip server
>>>>> through gateway send notifications to extgateway. When I
>>>>> attached using gdb to process that send data, I saw that it 
>>>>> was in infinity loop because tls library returns
>>>>> SSL_ERROR_WANT_WRITE and I think it’s correct because we have
>>>>> overfilled send queue. In this case I don’t interest why
>>>>> receive queue not empty (I think it happens because sending
>>>>> process have got block on socket). But I don’t understand why
>>>>> sent queue not flushed. I test this behavior using tcp – all
>>>>> was correct. Socket not closed by other side because this
>>>>> status can be unchanged above one day. Anybody can help me?
>>>>> Why this happens?
>>>>> 
>>>>> I use openssl-0.9.8c and redhat os (Linux xdevel1
>>>>> 2.4.21-4.ELsmp #1 SMP Fri Oct 3 17:52:56 EDT 2003 i686 i686
>>>>> i386 GNU/Linux). Computer have too Pentium 4 processors).
>>>>> 
>>>>> 
>>>>> Gateway.cfg (sip.qa.hbex.com – gateway address):
>>>>> 
>>>>> # # $Id: router_qa.cfg,v 1.1 2006/08/02 18:14:34 ilya Exp $ #
>>>>> # simple quick-start config script #
>>>>> 
>>>>> # ----------- global configuration parameters 
>>>>> ------------------------
>>>>> 
>>>>> debug=9            # debug level (cmd line: -dddddddddd) 
>>>>> log_facility=LOG_LOCAL0 fork=yes log_stderror=yes    # (cmd
>>>>> line: -E)
>>>>> 
>>>>> 
>>>>> check_via=no	# (cmd. line: -v) dns=no          # (cmd. line:
>>>>> -r) rev_dns=no      # (cmd. line: -R) children=20 
>>>>> fifo="/tmp/openser_fifo" server_signature=no
>>>>> 
>>>>> fifo_db_url="mysql://root:@localhost/openser"
>>>>> 
>>>>> listen = udp:sip.qa.hbex.com:5060 #incoming sip server
>>>>> address listen = tcp:sip.qa.hbex.com:5060
>>>>> 
>>>>> #tls address disable_tls = 0 listen =
>>>>> tls:sip.qa.hbex.com:5061 listen = udp:sip.qa.hbex.com:5061
>>>>> 
>>>>> tls_certificate = "/ home/inop/ex.com-cert.pem"
>>>>> tls_private_key = "/home/inop/ex.com-privkey.pem" tls_ca_list
>>>>> = "/home/inop/ex.com-calist.pem"
>>>>> tls_require_client_certificate=0 tls_verify_client=0
>>>>> tls_verify_server=0
>>>>> 
>>>>> # ------------------ module loading 
>>>>> ---------------------------------- loadmodule 
>>>>> "/home/interop/openser/lib/openser/modules/rr.so" loadmodule 
>>>>> "/home/interop/openser/lib/openser/modules/xlog.so"
>>>>> loadmodule 
>>>>> "/home/interop/openser/lib/openser/modules/textops.so"
>>>>> loadmodule 
>>>>> "/home/interop/openser/lib/openser/modules/maxfwd.so"
>>>>> loadmodule "/home/interop/openser/lib/openser/modules/sl.so"
>>>>> loadmodule 
>>>>> "/home/interop/openser/lib/openser/modules/mysql.so"
>>>>> loadmodule 
>>>>> "/home/interop/openser/lib/openser/modules/tm_unchanged.so" 
>>>>> loadmodule
>>>>> "/home/interop/openser/lib/openser/modules/usrloc.so" 
>>>>> loadmodule
>>>>> "/home/interop/openser/lib/openser/modules/registrar.so"
>>>>> 
>>>>> # ----------------- setting module-specific parameters 
>>>>> --------------- modparam("registrar", "default_expires", 120)
>>>>>  modparam("registrar", "use_domain", 1)
>>>>> 
>>>>> modparam("usrloc", "use_domain", 1) modparam("usrloc",
>>>>> "db_mode", 0) # Uncomment this if you want to use SQL
>>>>> database # for persistent storage and comment the previous
>>>>> line #modparam("usrloc", "db_mode", 2)
>>>>> 
>>>>> # add value to ;lr param to make some broken UAs happy
>>>>> modparam("rr", "enable_full_lr", 1) modparam("maxfwd",
>>>>> "max_limit", 10) # -------------------------  request routing
>>>>> logic ------------------- # main routing logic route{ if
>>>>> (!mf_process_maxfwd_header("10")) { sl_send_reply("483","To
>>>>> Many Hops"); drop(); }; if (dst_port==5061) { if
>>>>> (search("^To:.*@.*cmnicator.com") ||
>>>>> search("^To:.*@.*cihb.com")) { # rewrite destination and
>>>>> forward to jabber (sip server) route(1); return; };
>>>>> sl_reply_error(); return; } if (method=="REGISTER") { 
>>>>> xlog("XXX: saving location msg=$mb\n");
>>>>> if(!save_noreply("location")) { log("XXX: Error saving
>>>>> location!\n"); sl_reply_error(); } sl_send_reply("200","OK");
>>>>> return; } if (search("^To:.*@.*cmnicator.com") ||
>>>>> search("^To:.*@.*cihb.com")) { #forward to self route(3);
>>>>> return; } ##forward to ext gateway route(2); }
>>>>> 
>>>>> route[1]{ if (method!="MESSAGE" && method!="SUBSCRIBE" && 
>>>>> method!="NOTIFY") { log("XXX: Request not forwarded to sip 
>>>>> server!\n"); sl_send_reply("202","Accepted"); return; } 
>>>>> t_on_reply("1"); record_route();
>>>>> 
>>>>> if(!lookup("location")) { if (method!="SUBSCRIBE") {
>>>>> log("XXX: only subscribes are processed for user that are not
>>>>> registered\n"); sl_reply_error(); return; } 
>>>>> subst_uri('/(.*)@(.*)/\1#\2*jabber.qa.im.hb.com at jabber01.qa.im.hb.com/ig');
>>>>>  if(!t_relay()) { log("XXX: error forwarding jabber01...\n");
>>>>>  sl_reply_error(); return; } else {
>>>>> sl_send_reply("200","OK"); return; } } else { log("XXX:
>>>>> forwarding to the address of record...\n"); if(!t_relay()) {
>>>>> log("XXX: error forwarding to address of record \n");
>>>>> sl_reply_error(); return; } else { sl_send_reply("200",
>>>>> "OK"); return; } }
>>>>> 
>>>>> }
>>>>> 
>>>>> # forwarding to external gateway route[2]{ log("XXX:
>>>>> rewriting headers\n"); 
>>>>> subst('/^(From:[^@#]*)#([^@]*)\*jabber.qa.im.hb.com at jabber...qa.im.hb.com(.*)/\1@\2\3/ig');
>>>>> 
>>>>> 
>>>>> subst('/^(Contact:[^@#]*)#([^@]*)\*jabber.qa.im.hb.com at jabber...qa.im.hb.com(.*)/\1@\2\3/ig');
>>>>> 
>>>>> 
>>>>> 
>>>>> if(!forward("tls:43.123.141.166:3000")) { log("XXX: Error
>>>>> forwarding to external gateway!\n"); sl_reply_error(); } else
>>>>> { sl_send_reply("200", "Accepted"); }; }
>>>>> 
>>>>> # forwarding to SELF route[3]{ log("XXX: rewriting
>>>>> headers\n"); 
>>>>> subst('/^(From:[^@#]*)#([^@]*)\*jabber.qa.im.hb.com at jabber...qa.im.hb.com(.*)/\1@\2\3/ig');
>>>>> 
>>>>> 
>>>>> subst('/^(Contact:[^@#]*)#([^@]*)\*jabber.qa.im.hb.com at jabber...qa.im.hb.com(.*)/\1@\2\3/ig');
>>>>> 
>>>>> 
>>>>> 
>>>>> log("XXX: about to forward to self\n"); 
>>>>> if(!forward("tcp:sip.qa.hbex.com:5061")) { log("XXX: Error
>>>>> forwarding to self!\n"); sl_reply_error(); } else {
>>>>> sl_send_reply("200", "Accepted"); }; }
>>>>> 
>>>>> # this is executed for replies onreply_route[1]{ 
>>>>> subst('/^(To:[^@#]*)#([^@]*)\*jabber.qa.im.hb.com at jabber...qa.im.hb.com(.*)/\1@\2\3/ig');
>>>>> 
>>>>> 
>>>>> subst('/^(Contact:[^@#]*)#([^@]*)\*jabber.qa.im.hb.com at jabber...qa.im.hb.com(.*)/\1@\2\3/ig');
>>>>>  route(2); }
>>>>> 
>>>>> 
>>>>> 
>>>>> _______________________________________________ Users mailing
>>>>> list Users at openser.org
>>>>> http://openser.org/cgi-bin/mailman/listinfo/users
>>>>> 
>>>> 
>>> 
>>> _______________________________________________ Users mailing
>>> list Users at openser.org 
>>> http://openser.org/cgi-bin/mailman/listinfo/users
>>> 
>>> 
> 
> 
> _______________________________________________ Users mailing list 
> Users at openser.org http://openser.org/cgi-bin/mailman/listinfo/users

More information about the sr-users mailing list