Sorry for the delay, I just got home from my $PAYINGJOB. And thanks a lot for helping figure this out.
I build Kamailio RPMs from the latest release tarball, with the changes between the release and git master applied via patch, but here is the version output:
# kamailio -v version: kamailio 4.4.0-dev6 (x86_64/linux) e275bc flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK- ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: e275bc compiled on 22:23:43 Nov 13 2015 with gcc 5.1.1
I am using tcpops functions as follows, though again, these were in use for quite some time before the issue appeared:
To update the tcp connection lifetime after successful auth: # User authenticated - update tcp_connection_lifetime if(proto!=UDP) tcp_set_connection_lifetime("3605");
and in the reply route:
onreply_route[MANAGE_REPLY] { xdbg("incoming reply\n");
if(status=~"[12][0-9][0-9]") route(NATMANAGE);
#!ifdef WITH_TCPOPS if(proto!=UDP && status=="200") { if(is_method("INVITE")) { # enable on callee's connection tcp_keepalive_enable("60", "5", "5"); # enable on caller's connection if($avp(caller_conid)!=$null) tcp_keepalive_enable("$avp(caller_conid)", "60", "5", "2"); } if(is_method("BYE")) { tcp_keepalive_disable(); tcp_keepalive_disable("$avp(bye_conid)"); } }
On Tuesday, November 17, 2015 09:39:28 PM Daniel-Constantin Mierla wrote:
Looking at the logs of last commits, I couldn't spot the change that would add the leak.
What is the exact version you are running (kamailio -v)?
Are you using any of the functions exported by tcpops?
Cheers, Daniel
On 17/11/15 15:24, Anthony Messina wrote:
I wish that were the case...
# kamcmd core.tcp_info {
readers: 2 max_connections: 2048 max_tls_connections: 2048 opened_connections: 0 opened_tls_connections: 0 write_queued_bytes: 0}
# kamcmd tls.info {
max_connections: 2048 opened_connections: 0 clear_text_write_queued_bytes: 0}
On Tuesday, November 17, 2015 03:08:59 PM Daniel-Constantin Mierla wrote:
Looks like a lot of connections being open, can you get the output for:
kamcmd core.tcp_info
kamcmd tls.info
Cheers, Daniel
On 17/11/15 14:59, Anthony Messina wrote:
Attached. -A
On Tuesday, November 17, 2015 02:50:21 PM Daniel-Constantin Mierla
wrote:
Can you run the following commands:
kamcmd cfg.set_now_int core memlog 1 kamcmd corex.shm_summary
Then grab the log messages from syslog related to shared memory summary and send them over here.
Cheers, Daniel
On 17/11/15 14:31, Anthony Messina wrote:
After I reported last night, I restarted Kamailio and even though the 5 UACs did nothing but ensure they had a registration overnight, this morning the issue has recurred. The following is the output you requested. Not sure how the memory is being used up by Kamailio.
# kamctl stats shmem shmem:fragments = 181 shmem:free_size = 8922584 shmem:max_used_size = 58243792 shmem:real_used_size = 58186280 shmem:total_size = 67108864 shmem:used_size = 54346088
On Tuesday, November 17, 2015 09:03:24 AM Daniel-Constantin Mierla
wrote:
> As you are using the master branch (development), do you run latest > version? > > Can you look at available shared memory? > > kamctl stats shmem > > Check it over time and see if the free memory is decreasing. > > Cheers, > Daniel > > On 17/11/15 00:44, Anthony Messina wrote: >> I have noticed the following issue which began with builds somewhere >> between git master commits bff0a08 and 6173ef7. I did not see this >> issue >> with my previous builds and haven't been able to pin down the >> problem, >> which is why I haven't formally filed a bug. >> >> Any help or guidance is appreciated, because this has crippled my >> use >> of >> Kamailio. Only a restart enables it to work again until the issue >> recurs. >> >> ERROR: tls [tls_server.c:189]: tls_complete_init(): tls: ssl bug >> #1491 >> workaround: not enough memory for safe operation: 8870536 >> ERROR: <core> [tcp_read.c:1303]: tcp_read_req(): ERROR: >> tcp_read_req: >> error >> reading >> >> I currently build against and run openssl-1.0.1k-12.fc22.x86_64. >> >> I have a very small operation and the only change on the operational >> side >> is that all 5 of my mobile UACs (yes, that's all) have switched from >> CSipSimple/Android to Zoiper/Android, which doesn't yet have support >> for >> client-side certificates so verify_certificate and >> require_certificate >> are >> off for both the server and client config. >> >> The server is started with: >> /usr/sbin/kamailio -P /run/kamailio/kamailio.pid -m 64 -M 8 >> >> I have tried modifying the shared mem to 128 but the issue still >> occurs. >> >> Even right now, I am seeing the error when only one UAC has >> established >> a >> TLS connection: >> >> # kamcmd tls.list >> { >> >> id: 572 >> timeout: 3475 >> src_ip: 10.77.79.156 >> src_port: 58688 >> dst_ip: 10.77.79.3 >> dst_port: 5061 >> cipher: ECDHE-RSA-RC4-SHA SSLv3 Kx=ECDH Au=RSA >> Enc=RC4(128) >> Mac=SHA1 >> ct_wq_size: 0 >> enc_rd_buf: 0 >> flags: 2 >> state: established >> >> } >> >> # kamailio.cfg >> enable_tls=yes >> loadmodule "tls.so" >> modparam("tls", "connection_timeout", 60) >> #modparam("tls", "tls_log", 1) >> #modparam("tls", "tls_debug", 1) >> #modparam("tls", "low_mem_threshold1", -1) >> #modparam("tls", "low_mem_threshold2", 0) >> modparam("tls", "session_cache", 1) >> >> # tls.cfg >> [server:default] >> method = TLSv1+ >> verify_certificate = no >> require_certificate = no >> private_key = /etc/kamailio/example.org.key.pem >> certificate = /etc/kamailio/example.org.crt.pem >> server_name = example.org >> cipher_list = >> ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-RSA- >> AES128-GCM-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-RC4-SHA:ECDHE-RS >> A- >> AE >> S >> 256- >> SHA:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-SHA256:DHE-RSA-AES128-G >> CM >> - >> SHA256:DHE-RSA-AES128-SHA256:DHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA:R >> C4 >> - >> SHA:AES256-GCM-SHA384:AES256-SHA256:CAMELLIA256-SHA:ECDHE-RSA-AES128 >> - >> SHA:AES128-GCM-SHA256:AES128-SHA256:AES128-SHA:CAMELLIA128- >> SHA:!aNULL:!eNULL:!EXPORT:!DES:!3DES:!MD5:!PSK >> >> [client:default] >> method = TLSv1+ >> verify_certificate = no >> require_certificate = no >> private_key = /etc/kamailio/example.org.key.pem >> certificate = /etc/kamailio/example.org.crt.pem >> cipher_list = >> ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-RSA- >> AES128-GCM-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-RC4-SHA:ECDHE-RS >> A- >> AE >> S >> 256- >> SHA:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-SHA256:DHE-RSA-AES128-G >> CM >> - >> SHA256:DHE-RSA-AES128-SHA256:DHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA:R >> C4 >> - >> SHA:AES256-GCM-SHA384:AES256-SHA256:CAMELLIA256-SHA:ECDHE-RSA-AES128 >> - >> SHA:AES128-GCM-SHA256:AES128-SHA256:AES128-SHA:CAMELLIA128- >> SHA:!aNULL:!eNULL:!EXPORT:!DES:!3DES:!MD5:!PSK >> >> >> Thanks. -Anthony