<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3c.org/TR/1999/REC-html401-19991224/loose.dtd">
<HTML><HEAD>
<META http-equiv=Content-Type content="text/html; charset=iso-8859-1">
<META content=1 name=qrichtext>
<STYLE type=text/css>p, li { white-space: pre-wrap; }</STYLE>
<META content="MSHTML 6.00.2900.3527" name=GENERATOR></HEAD>
<BODY
style="FONT-WEIGHT: 400; FONT-SIZE: 9pt; FONT-STYLE: normal; FONT-FAMILY: 'Lucida Console'">
<DIV dir=ltr align=left><SPAN class=378143009-16072009><FONT face=Arial
size=2>Hi everybody,</FONT></SPAN></DIV>
<DIV dir=ltr align=left><SPAN class=378143009-16072009><FONT face=Arial
size=2></FONT></SPAN> </DIV>
<DIV><SPAN class=378143009-16072009><FONT face=Arial>regarding our TCP/TLS
stability problems we have no decided to make test with kamailio
1.5.1</FONT></SPAN></DIV>
<DIV><SPAN class=378143009-16072009><FONT face=Arial>Nevertheless it would be
interesting if there is a chance to get rid of this
problems.</FONT></SPAN></DIV>
<DIV><SPAN class=378143009-16072009><FONT face=Arial></FONT></SPAN> </DIV>
<DIV><SPAN class=378143009-16072009><FONT face=Arial>Is anybody using
TLS?</FONT></SPAN></DIV>
<DIV><SPAN class=378143009-16072009><FONT face=Arial></FONT></SPAN> </DIV>
<DIV><SPAN class=378143009-16072009><FONT face=Arial>Used modules: SNMP,
mySQL</FONT></SPAN></DIV>
<DIV><SPAN class=378143009-16072009><FONT face=Arial></FONT></SPAN> </DIV>
<DIV><SPAN class=378143009-16072009><FONT face=Arial>Summary of
problems</FONT></SPAN></DIV>
<DIV><SPAN class=378143009-16072009><FONT face=Arial>Errors may be related to
the following log file entries</FONT></SPAN></DIV>
<DIV><SPAN class=378143009-16072009><SPAN lang=DE>
<P>un 17 09:01:27 si-…. /usr/local/sbin/openser[13921]: WARNING:core:send2child:
no free tcp receiver, connection passed to the leastbusy one (6) </P>
<P>Jun 17 08:54:52 si-…. /usr/local/sbin/openser[13921]: ERROR:core:tcpconn_new:
shared memory allocation failure </P>
<P>Jun 17 08:54:52 si-… /usr/local/sbin/openser[13921]:
ERROR:core:handle_new_connect: tcpconn_new failed, closing socket </P>
<P>And a few of these also (7613 times):</P>
<P>Jun 17 08:57:24 si-… /usr/local/sbin/openser[13880]: ERROR:core:tls_accept:
some error in SSL: </P>
<P>Jun 17 08:57:24 si-… /usr/local/sbin/openser[13880]:
ERROR:core:tls_print_errstack: error:1409C041:SSL
routines:SSL3_SETUP_BUFFERS:malloc failure</P></SPAN></SPAN></DIV>
<DIV><SPAN class=378143009-16072009><FONT face=Arial><STRONG>shared memory
consumption</STRONG></FONT></SPAN></DIV>
<DIV><SPAN class=378143009-16072009><FONT face=Arial>shared memory is
continously increasing (set to 1024)</FONT></SPAN></DIV>
<DIV><SPAN class=378143009-16072009><FONT face=Arial>PKG_MEM is 1
MB</FONT></SPAN></DIV>
<DIV><SPAN class=378143009-16072009><FONT face=Arial></FONT></SPAN> </DIV>
<DIV><SPAN class=378143009-16072009><FONT face=Arial><STRONG>high CPU load for
some openser processes</STRONG></FONT></SPAN></DIV>
<DIV><SPAN class=378143009-16072009><FONT face=Arial>normally after some days we
get a high CPU load (50-90%) for a small number of the openser
processes</FONT></SPAN></DIV>
<DIV><SPAN class=378143009-16072009><FONT face=Arial>It looks like an endless
loop and requires restart of openser</FONT></SPAN></DIV>
<DIV><SPAN class=378143009-16072009><FONT face=Arial>There may be an endless
loop in</FONT></SPAN></DIV>
<DIV><SPAN class=378143009-16072009>
<P><FONT size=2>Pass_fd.c</FONT></P>
<P><FONT size=2>again:</FONT></P>
<P><FONT size=2>ret=sendmsg(unix_socket, &msg, 0);</FONT></P>
<P><FONT size=2>if (ret<0){</FONT></P>
<P><FONT size=2>if (errno==EINTR) goto again;</FONT></P>
<P><FONT size=2>LM_CRIT("sendmsg failed on %d: %s\n", unix_socket,
strerror(errno));</FONT></P>
<P><FONT size=2>}</FONT></P>
<P><SPAN class=378143009-16072009><FONT face=Arial size=2>any comments on
that?</FONT></SPAN></P></SPAN></DIV><!-- Converted from text/rtf format -->
<P><SPAN lang=de><FONT face=Arial size=2>Mit besten Grüßen | Best
regards</FONT><FONT face="Times New Roman"> </FONT></SPAN><BR><SPAN
lang=de><B><FONT face=Arial size=2>Albert Munder</FONT></B><FONT
face="Times New Roman"> </FONT></SPAN><BR><SPAN lang=de><FONT face=Arial
size=2>Robert Bosch GmbH</FONT><FONT face="Times New Roman"><BR></FONT><FONT
face=Arial size=2>IT Systems Engineering (CI/ISE)</FONT><FONT
face="Times New Roman"><BR></FONT><FONT face=Arial size=2>Postfach 30 02
20</FONT><FONT face="Times New Roman"><BR></FONT><FONT face=Arial size=2>70442
Stuttgart</FONT><BR><FONT face=Arial size=2>GERMANY</FONT><FONT
face="Times New Roman"><BR></FONT><FONT face=Arial
size=2>www.bosch.com</FONT><FONT face="Times New Roman"> </FONT></SPAN><BR><SPAN
lang=de><FONT face=Arial size=2>Tel. +49 711 811-40562</FONT><FONT
face="Times New Roman"><BR></FONT><FONT face=Arial size=2>Fax +49 711
811-5113333</FONT><FONT face="Times New Roman"><BR><U></U></FONT><U><FONT
face=Arial color=#0000ff size=2>Albert.Munder@de.bosch.com</FONT></U><FONT
face="Times New Roman"> </FONT></SPAN><BR><SPAN lang=de><FONT face=Arial
size=2>Robert Bosch GmbH, Sitz: Stuttgart, Registergericht: Amtsgericht
Stuttgart HRB 14000</FONT><FONT face="Times New Roman"><BR></FONT><FONT
face=Arial size=2>Aufsichtsratsvorsitzender: Hermann Scholl; Geschäftsführung:
Franz Fehrenbach, Siegfried Dais;</FONT><FONT
face="Times New Roman"><BR></FONT><FONT face=Arial size=2>Bernd Bohr, Wolfgang
Chur, Rudolf Colm, Gerhard Kümmel, Wolfgang Malchow, Peter Marks;</FONT><FONT
face="Times New Roman"><BR></FONT><FONT face=Arial size=2>Volkmar Denner, Peter
Tyroller.</FONT><FONT face="Times New Roman"></FONT> </SPAN></P>
<DIV> </DIV><BR>
<DIV class=OutlookMessageHeader lang=de dir=ltr align=left>
<HR tabIndex=-1>
<FONT face=Tahoma size=2><B>Von:</B> Henning Westerholt
[mailto:henning.westerholt@1und1.de] <BR><B>Gesendet:</B> Dienstag, 30. Juni
2009 17:25<BR><B>An:</B> users@lists.kamailio.org<BR><B>Cc:</B> Munder Albert
(CI/ISE)<BR><B>Betreff:</B> Re: [Kamailio-Users] OpenSER stability problems in
pilot project<BR></FONT><BR></DIV>
<DIV></DIV>On Dienstag, 30. Juni 2009, Munder Albert (CI/ISE) wrote:<BR>>
[..]<BR>> We are running OpenSER in a pilot project and<BR>> unfortunately
have some stability problems. <BR>
<P
style="MARGIN: 0px; TEXT-INDENT: 0px; qt-paragraph-type: empty; qt-block-indent: 0; qt-user-state: 0"><BR></P>Hallo
Albert,<BR>
<P
style="MARGIN: 0px; TEXT-INDENT: 0px; qt-paragraph-type: empty; qt-block-indent: 0; qt-user-state: 0"><BR></P>>
* Appr. 5000 subscriber accounts<BR>> * Appr. 1200 simultaneously registered
users<BR>> * Signalling encrypted with TLS<BR>> * Media data encrypted
with SRTP<BR>> * Clients: softphones and hardphones<BR>> * Re-registration
time for clients: 3600 sec<BR>
<P
style="MARGIN: 0px; TEXT-INDENT: 0px; qt-paragraph-type: empty; qt-block-indent: 0; qt-user-state: 0"><BR></P>I've
not that much experience with TCP, but don't think that this numbers should be a
problem in a setup like this.<BR>
<P
style="MARGIN: 0px; TEXT-INDENT: 0px; qt-paragraph-type: empty; qt-block-indent: 0; qt-user-state: 0"><BR></P>>
OpenSER configuration<BR>> · Works as stateful SIP Proxy<BR>> 1 mySQL
database<BR>> 2 Version 1.3.4.-TLS<BR>> 3 Tcp_children: 100 --> is it
recommended to increase this number?<BR>
<P
style="MARGIN: 0px; TEXT-INDENT: 0px; qt-paragraph-type: empty; qt-block-indent: 0; qt-user-state: 0"><BR></P>This
are quite a lot of children, but ok.<BR>
<P
style="MARGIN: 0px; TEXT-INDENT: 0px; qt-paragraph-type: empty; qt-block-indent: 0; qt-user-state: 0"><BR></P>>
4 Udp_children: 20<BR>> 5 Tcp_connection_timeout: 3600<BR>> 6 Shared
memory:<BR>> · -m 512 when error occurred<BR>> 1 Now set to 1024<BR>
<P
style="MARGIN: 0px; TEXT-INDENT: 0px; qt-paragraph-type: empty; qt-block-indent: 0; qt-user-state: 0"><BR></P>How
much PKG_MEM do you use? The default value?<BR>
<P
style="MARGIN: 0px; TEXT-INDENT: 0px; qt-paragraph-type: empty; qt-block-indent: 0; qt-user-state: 0"><BR></P>>
Problems<BR>> * Shared memory consumption<BR>> Shared memory usage is
permanently increasing (about 50 MB per day)<BR>> Application already crashed
twice<BR>
<P
style="MARGIN: 0px; TEXT-INDENT: 0px; qt-paragraph-type: empty; qt-block-indent: 0; qt-user-state: 0"><BR></P>This
could be a memory leak, what modules do you use? And do you use any proprietary
modules? You could use the memory debugging to further investigate this:
http://www.kamailio.org/dokuwiki/doku.php/troubleshooting:memory<BR>
<P
style="MARGIN: 0px; TEXT-INDENT: 0px; qt-paragraph-type: empty; qt-block-indent: 0; qt-user-state: 0"><BR></P>>
First messages were, these, repeated thousands of times (5915 times):<BR>>
Jun 17 08:54:52 si-.... /usr/local/sbin/openser[13921]:<BR>>
ERROR:core:tcpconn_new: shared memory allocation failure Jun 17 08:54:52<BR>>
si-... /usr/local/sbin/openser[13921]: ERROR:core:handle_new_connect:<BR>>
tcpconn_new failed, closing socket And a few of these also (7613 times):<BR>>
Jun 17 08:57:24 si-... /usr/local/sbin/openser[13880]:<BR>>
ERROR:core:tls_accept: some error in SSL: Jun 17 08:57:24 si-...<BR>>
/usr/local/sbin/openser[13880]: ERROR:core:tls_print_errstack:<BR>>
error:1409C041:SSL routines:SSL3_SETUP_BUFFERS:malloc failure<BR>
<P
style="MARGIN: 0px; TEXT-INDENT: 0px; qt-paragraph-type: empty; qt-block-indent: 0; qt-user-state: 0"><BR></P>This
are caused from insufficient memory conditions. I can't comment on the TCP and
TLS errors. But before really starting to investigate this problem, would it be
possible for you to use a more recent version, e.g. kamailio 1.5.1 for
testing?<BR>
<P
style="MARGIN: 0px; TEXT-INDENT: 0px; qt-paragraph-type: empty; qt-block-indent: 0; qt-user-state: 0"><BR></P>>
* TCP errors, lost SIP messages<BR>><BR>> Examples from error
messages:<BR>> 14.100 times in log file from 17.06.09<BR>> Jun 17 04:03:15
si-... /usr/local/sbin/openser[13863]:<BR>> ERROR:core:tcp_blocking_connect:
poll error: flags 18 Jun 17 04:03:15<BR>> si-...
/usr/local/sbin/openser[13863]: ERROR:core:tcp_blocking_connect:<BR>> failed
to retrieve SO_ERROR (111) Connection refused Jun 17 04:03:15 si-...<BR>>
/usr/local/sbin/openser[13863]: ERROR:core:tcpconn_connect:<BR>>
tcp_blocking_connect failed Jun 17 04:03:15 si-...<BR>>
/usr/local/sbin/openser[13863]: ERROR:core:tcp_send: connect failed Jun
17<BR>> 04:03:15 si-.. /usr/local/sbin/openser[13863]: ERROR:tm:msg_send:
tcp_send<BR>> failed Jun 17 04:03:15 si-...
/usr/local/sbin/openser[13863]:<BR>> ERROR:tm:t_forward_nonack: sending
request failed<BR>><BR>> Appears at least 20 000 times; and in the day of
the last shared memory<BR>> errors, it was 225.794 times in the log file
(note that the number in<BR>> parenthesis is usually 1 or 2, but on that day
it has reached 6): Jun 17<BR>> 09:01:27 si-....
/usr/local/sbin/openser[13921]: WARNING:core:send2child:<BR>> no free tcp
receiver, connection passed to the leastbusy one (6) Jun 17<BR>> 09:01:27
si-... /usr/local/sbin/openser[13921]: WARNING:core:send2child: no<BR>> free
tcp receiver, connection passed to the leastbusy one (5)<BR>><BR>> *
Certificate validation problems<BR>> TCP traffic is currently significantly
increased by some ( appr. 70)<BR>> clients which failed to validate the TLS
certificate. Registration is<BR>> repeated every 5 sec.<BR>><BR>> Circa
30 thousand per day (on that day, it was 37.162 times in log)<BR>> Jun 17
04:03:10 si-024lc008 /usr/local/sbin/openser[13801]:<BR>>
ERROR:core:tls_accept: some error in SSL: Jun 17 04:03:10 si-024lc008<BR>>
/usr/local/sbin/openser[13801]: ERROR:core:tls_print_errstack:<BR>>
error:14094418:SSL routines:SSL3_READ_BYTES:tlsv1 alert unknown ca<BR>
<P
style="MARGIN: 0px; TEXT-INDENT: 0px; qt-paragraph-type: empty; qt-block-indent: 0; qt-user-state: 0"><BR></P>Best
regards,<BR>
<P
style="MARGIN: 0px; TEXT-INDENT: 0px; qt-paragraph-type: empty; qt-block-indent: 0; qt-user-state: 0"><BR></P>Henning
<P></P></BODY></HTML>