[SR-Users] TLS not enough memory issue with git master
Anthony Messina
amessina at messinet.com
Wed Nov 18 02:16:49 CET 2015
Sorry for the delay, I just got home from my $PAYINGJOB. And thanks a lot for
helping figure this out.
I build Kamailio RPMs from the latest release tarball, with the changes
between the release and git master applied via patch, but here is the version
output:
# kamailio -v
version: kamailio 4.4.0-dev6 (x86_64/linux) e275bc
flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS,
DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC,
Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-
ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST,
HAVE_RESOLV_RES
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16,
MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
id: e275bc
compiled on 22:23:43 Nov 13 2015 with gcc 5.1.1
I am using tcpops functions as follows, though again, these were in use for
quite some time before the issue appeared:
To update the tcp connection lifetime after successful auth:
# User authenticated - update tcp_connection_lifetime
if(proto!=UDP)
tcp_set_connection_lifetime("3605");
and in the reply route:
onreply_route[MANAGE_REPLY] {
xdbg("incoming reply\n");
if(status=~"[12][0-9][0-9]")
route(NATMANAGE);
#!ifdef WITH_TCPOPS
if(proto!=UDP && status=="200") {
if(is_method("INVITE")) {
# enable on callee's connection
tcp_keepalive_enable("60", "5", "5");
# enable on caller's connection
if($avp(caller_conid)!=$null)
tcp_keepalive_enable("$avp(caller_conid)",
"60", "5", "2");
}
if(is_method("BYE")) {
tcp_keepalive_disable();
tcp_keepalive_disable("$avp(bye_conid)");
}
}
On Tuesday, November 17, 2015 09:39:28 PM Daniel-Constantin Mierla wrote:
> Looking at the logs of last commits, I couldn't spot the change that
> would add the leak.
>
> What is the exact version you are running (kamailio -v)?
>
> Are you using any of the functions exported by tcpops?
>
> Cheers,
> Daniel
>
> On 17/11/15 15:24, Anthony Messina wrote:
> > I wish that were the case...
> >
> > # kamcmd core.tcp_info
> > {
> >
> > readers: 2
> > max_connections: 2048
> > max_tls_connections: 2048
> > opened_connections: 0
> > opened_tls_connections: 0
> > write_queued_bytes: 0
> >
> > }
> >
> > # kamcmd tls.info
> > {
> >
> > max_connections: 2048
> > opened_connections: 0
> > clear_text_write_queued_bytes: 0
> >
> > }
> >
> > On Tuesday, November 17, 2015 03:08:59 PM Daniel-Constantin Mierla wrote:
> >> Looks like a lot of connections being open, can you get the output for:
> >>
> >> kamcmd core.tcp_info
> >>
> >> kamcmd tls.info
> >>
> >> Cheers,
> >> Daniel
> >>
> >> On 17/11/15 14:59, Anthony Messina wrote:
> >>> Attached. -A
> >>>
> >>> On Tuesday, November 17, 2015 02:50:21 PM Daniel-Constantin Mierla
wrote:
> >>>> Can you run the following commands:
> >>>>
> >>>> kamcmd cfg.set_now_int core memlog 1
> >>>> kamcmd corex.shm_summary
> >>>>
> >>>> Then grab the log messages from syslog related to shared memory summary
> >>>> and send them over here.
> >>>>
> >>>> Cheers,
> >>>> Daniel
> >>>>
> >>>> On 17/11/15 14:31, Anthony Messina wrote:
> >>>>> After I reported last night, I restarted Kamailio and even though the
> >>>>> 5
> >>>>> UACs did nothing but ensure they had a registration overnight, this
> >>>>> morning the issue has recurred. The following is the output you
> >>>>> requested. Not sure how the memory is being used up by Kamailio.
> >>>>>
> >>>>> # kamctl stats shmem
> >>>>> shmem:fragments = 181
> >>>>> shmem:free_size = 8922584
> >>>>> shmem:max_used_size = 58243792
> >>>>> shmem:real_used_size = 58186280
> >>>>> shmem:total_size = 67108864
> >>>>> shmem:used_size = 54346088
> >>>>>
> >>>>> On Tuesday, November 17, 2015 09:03:24 AM Daniel-Constantin Mierla
> >
> > wrote:
> >>>>>> As you are using the master branch (development), do you run latest
> >>>>>> version?
> >>>>>>
> >>>>>> Can you look at available shared memory?
> >>>>>>
> >>>>>> kamctl stats shmem
> >>>>>>
> >>>>>> Check it over time and see if the free memory is decreasing.
> >>>>>>
> >>>>>> Cheers,
> >>>>>> Daniel
> >>>>>>
> >>>>>> On 17/11/15 00:44, Anthony Messina wrote:
> >>>>>>> I have noticed the following issue which began with builds somewhere
> >>>>>>> between git master commits bff0a08 and 6173ef7. I did not see this
> >>>>>>> issue
> >>>>>>> with my previous builds and haven't been able to pin down the
> >>>>>>> problem,
> >>>>>>> which is why I haven't formally filed a bug.
> >>>>>>>
> >>>>>>> Any help or guidance is appreciated, because this has crippled my
> >>>>>>> use
> >>>>>>> of
> >>>>>>> Kamailio. Only a restart enables it to work again until the issue
> >>>>>>> recurs.
> >>>>>>>
> >>>>>>> ERROR: tls [tls_server.c:189]: tls_complete_init(): tls: ssl bug
> >>>>>>> #1491
> >>>>>>> workaround: not enough memory for safe operation: 8870536
> >>>>>>> ERROR: <core> [tcp_read.c:1303]: tcp_read_req(): ERROR:
> >>>>>>> tcp_read_req:
> >>>>>>> error
> >>>>>>> reading
> >>>>>>>
> >>>>>>> I currently build against and run openssl-1.0.1k-12.fc22.x86_64.
> >>>>>>>
> >>>>>>> I have a very small operation and the only change on the operational
> >>>>>>> side
> >>>>>>> is that all 5 of my mobile UACs (yes, that's all) have switched from
> >>>>>>> CSipSimple/Android to Zoiper/Android, which doesn't yet have support
> >>>>>>> for
> >>>>>>> client-side certificates so verify_certificate and
> >>>>>>> require_certificate
> >>>>>>> are
> >>>>>>> off for both the server and client config.
> >>>>>>>
> >>>>>>> The server is started with:
> >>>>>>> /usr/sbin/kamailio -P /run/kamailio/kamailio.pid -m 64 -M 8
> >>>>>>>
> >>>>>>> I have tried modifying the shared mem to 128 but the issue still
> >>>>>>> occurs.
> >>>>>>>
> >>>>>>> Even right now, I am seeing the error when only one UAC has
> >>>>>>> established
> >>>>>>> a
> >>>>>>> TLS connection:
> >>>>>>>
> >>>>>>> # kamcmd tls.list
> >>>>>>> {
> >>>>>>>
> >>>>>>> id: 572
> >>>>>>> timeout: 3475
> >>>>>>> src_ip: 10.77.79.156
> >>>>>>> src_port: 58688
> >>>>>>> dst_ip: 10.77.79.3
> >>>>>>> dst_port: 5061
> >>>>>>> cipher: ECDHE-RSA-RC4-SHA SSLv3 Kx=ECDH Au=RSA
> >>>>>>> Enc=RC4(128)
> >>>>>>> Mac=SHA1
> >>>>>>> ct_wq_size: 0
> >>>>>>> enc_rd_buf: 0
> >>>>>>> flags: 2
> >>>>>>> state: established
> >>>>>>>
> >>>>>>> }
> >>>>>>>
> >>>>>>> # kamailio.cfg
> >>>>>>> enable_tls=yes
> >>>>>>> loadmodule "tls.so"
> >>>>>>> modparam("tls", "connection_timeout", 60)
> >>>>>>> #modparam("tls", "tls_log", 1)
> >>>>>>> #modparam("tls", "tls_debug", 1)
> >>>>>>> #modparam("tls", "low_mem_threshold1", -1)
> >>>>>>> #modparam("tls", "low_mem_threshold2", 0)
> >>>>>>> modparam("tls", "session_cache", 1)
> >>>>>>>
> >>>>>>> # tls.cfg
> >>>>>>> [server:default]
> >>>>>>> method = TLSv1+
> >>>>>>> verify_certificate = no
> >>>>>>> require_certificate = no
> >>>>>>> private_key = /etc/kamailio/example.org.key.pem
> >>>>>>> certificate = /etc/kamailio/example.org.crt.pem
> >>>>>>> server_name = example.org
> >>>>>>> cipher_list =
> >>>>>>> ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-
> >>>>>>> AES128-GCM-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-RC4-SHA:ECDHE-RS
> >>>>>>> A-
> >>>>>>> AE
> >>>>>>> S
> >>>>>>> 256-
> >>>>>>> SHA:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-SHA256:DHE-RSA-AES128-G
> >>>>>>> CM
> >>>>>>> -
> >>>>>>> SHA256:DHE-RSA-AES128-SHA256:DHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA:R
> >>>>>>> C4
> >>>>>>> -
> >>>>>>> SHA:AES256-GCM-SHA384:AES256-SHA256:CAMELLIA256-SHA:ECDHE-RSA-AES128
> >>>>>>> -
> >>>>>>> SHA:AES128-GCM-SHA256:AES128-SHA256:AES128-SHA:CAMELLIA128-
> >>>>>>> SHA:!aNULL:!eNULL:!EXPORT:!DES:!3DES:!MD5:!PSK
> >>>>>>>
> >>>>>>> [client:default]
> >>>>>>> method = TLSv1+
> >>>>>>> verify_certificate = no
> >>>>>>> require_certificate = no
> >>>>>>> private_key = /etc/kamailio/example.org.key.pem
> >>>>>>> certificate = /etc/kamailio/example.org.crt.pem
> >>>>>>> cipher_list =
> >>>>>>> ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-
> >>>>>>> AES128-GCM-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-RC4-SHA:ECDHE-RS
> >>>>>>> A-
> >>>>>>> AE
> >>>>>>> S
> >>>>>>> 256-
> >>>>>>> SHA:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-SHA256:DHE-RSA-AES128-G
> >>>>>>> CM
> >>>>>>> -
> >>>>>>> SHA256:DHE-RSA-AES128-SHA256:DHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA:R
> >>>>>>> C4
> >>>>>>> -
> >>>>>>> SHA:AES256-GCM-SHA384:AES256-SHA256:CAMELLIA256-SHA:ECDHE-RSA-AES128
> >>>>>>> -
> >>>>>>> SHA:AES128-GCM-SHA256:AES128-SHA256:AES128-SHA:CAMELLIA128-
> >>>>>>> SHA:!aNULL:!eNULL:!EXPORT:!DES:!3DES:!MD5:!PSK
> >>>>>>>
> >>>>>>>
> >>>>>>> Thanks. -Anthony
--
Anthony - https://messinet.com/ - https://messinet.com/~amessina/gallery
8F89 5E72 8DF0 BCF0 10BE 9967 92DC 35DC B001 4A4E
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 181 bytes
Desc: This is a digitally signed message part.
URL: <http://lists.sip-router.org/pipermail/sr-users/attachments/20151117/f635e797/attachment.sig>
More information about the sr-users
mailing list