[sr-dev] Problem with TCP and EPOLL

[Paul Pankhurst]

Thanks for that  information Andrei - it has opened up a new path of 
investigation and we think we understand what is going wrong now.
I will post an update when I get a bit further.

Paul

-----Original Message----- 
From: Andrei Pelinescu-Onciul
Sent: Thursday, February 16, 2012 1:53 PM
To: Daniel-Constantin Mierla
Cc: Development mailing list of the sip-router project ; Paul Pankhurst
Subject: Re: [sr-dev] Problem with TCP and EPOLL

On Feb 15, 2012 at 12:05, Daniel-Constantin Mierla <miconda at gmail.com> 
wrote:
> Hello,
>
> I am cc-ing Andrei, since he authored that part, maybe he is
> available these days and can give a quick answer regarding the
> issue.
>
> Cheers,
> Daniel
>
> On 2/14/12 6:06 PM, Paul Pankhurst wrote:
> >Sorry this was originally posted incorrectly, so I'm reposting....
> >I have been having problems with TCP under load.  What I have been
> >seeing is
> >TCP buffers failing to be serviced and, when wr_timeout exceeds the
> >configured value for tcp_send_timeout, kamailio kills the connection.
> >Increasing tcp_send_timeout doesn't help, even setting this to a
> >big value
> >(such as 45 seconds) just delays the disconnection.
> >
> >Putting some tracing into the code shows that wbufq_add() is repeatedly
> >called, but wbufq_run() is called for that connection far less
> >than I would
> >expect.  wbufq_run() is frequently called for other connections.
> >It looks
> >like wbufq_run() doesn't get called when lots of wbufq_add()s are
> >happening
> >for a connection?  wbufq_run() only appears to be called for a connection
> >after some time has passed from the last wbufq_add().

It's called when the kernel says it can write again on the respective
socket.
It might be that your consumer cannot read fast enough and so the
buffers fill on ser/kamailio side.

> >
> >The connection in question is a local loopback between the RLS and
> >Presence
> >modules (both running in the same Kamailio instance).  However, it
> >may just
> >be a coincidence that this is the affected connection as it is
> >also the one
> >with the most traffic.

You might do something much more resource intensive on the receive side
and it might not be able to keep up with the traffic (one connection is
handled by one process, so if that process is too slow for some reason
it might not read fast enough => on the transmit side the send buffers
will fill-up).

> >
> >My suspicion is that the bug is in the io_wait_loop_epoll() routine.

You could try changing the poll method and see if that makes any
difference, e.g.:
tcp_poll_method = sigio_rt in the .cfg file.
The default is epoll-lt, so try "epoll-et", "sigio_rt2 and maybe "poll"
(slow for lots of connections).


Andrei

> >
> >Can anybody with experience of this part of the code help?
> >
> >Paul Pankhurst
> >Engineering Director
> >Crocodile RCS Ltd
> >
> >
> >_______________________________________________
> >sr-dev mailing list
> >sr-dev at lists.sip-router.org
> >http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-dev
>
> -- 
> Daniel-Constantin Mierla -- http://www.asipto.com
> http://linkedin.com/in/miconda -- http://twitter.com/miconda
>