As you are using the master branch (development), do you run latest version? Can you look at available shared memory? kamctl stats shmem Check it over time and see if the free memory is decreasing. Cheers, Daniel On 17/11/15 00:44, Anthony Messina wrote: > I have noticed the following issue which began with builds somewhere > between git master commits bff0a08 and 6173ef7. I did not see this issue > with my previous builds and haven't been able to pin down the problem, > which is why I haven't formally filed a bug. > > Any help or guidance is appreciated, because this has crippled my use of > Kamailio. Only a restart enables it to work again until the issue recurs. > > ERROR: tls [tls_server.c:189]: tls_complete_init(): tls: ssl bug #1491 > workaround: not enough memory for safe operation: 8870536 > ERROR: <core> [tcp_read.c:1303]: tcp_read_req(): ERROR: tcp_read_req: > error > reading > > I currently build against and run openssl-1.0.1k-12.fc22.x86_64. > > I have a very small operation and the only change on the operational side > is that all 5 of my mobile UACs (yes, that's all) have switched from > CSipSimple/Android to Zoiper/Android, which doesn't yet have support for > client-side certificates so verify_certificate and require_certificate are > off for both the server and client config. > > The server is started with: > /usr/sbin/kamailio -P /run/kamailio/kamailio.pid -m 64 -M 8 > > I have tried modifying the shared mem to 128 but the issue still occurs. > > Even right now, I am seeing the error when only one UAC has established a > TLS connection: > > # kamcmd tls.list > { > > id: 572 > timeout: 3475 > src_ip: 10.77.79.156 > src_port: 58688 > dst_ip: 10.77.79.3 > dst_port: 5061 > cipher: ECDHE-RSA-RC4-SHA SSLv3 Kx=ECDH Au=RSA Enc=RC4(128) > Mac=SHA1 > ct_wq_size: 0 > enc_rd_buf: 0 > flags: 2 > state: established > > } > > # kamailio.cfg > enable_tls=yes > loadmodule "tls.so" > modparam("tls", "connection_timeout", 60) > #modparam("tls", "tls_log", 1) > #modparam("tls", "tls_debug", 1) > #modparam("tls", "low_mem_threshold1", -1) > #modparam("tls", "low_mem_threshold2", 0) > modparam("tls", "session_cache", 1) > > # tls.cfg > [server:default] > method = TLSv1+ > verify_certificate = no > require_certificate = no > private_key = /etc/kamailio/example.org.key.pem > certificate = /etc/kamailio/example.org.crt.pem > server_name = example.org > cipher_list = > ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-RSA- > AES128-GCM-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-RC4-SHA:ECDHE-RSA-AES > 256- > SHA:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-SHA256:DHE-RSA-AES128-GCM- > SHA256:DHE-RSA-AES128-SHA256:DHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA:RC4- > SHA:AES256-GCM-SHA384:AES256-SHA256:CAMELLIA256-SHA:ECDHE-RSA-AES128- > SHA:AES128-GCM-SHA256:AES128-SHA256:AES128-SHA:CAMELLIA128- > SHA:!aNULL:!eNULL:!EXPORT:!DES:!3DES:!MD5:!PSK > > [client:default] > method = TLSv1+ > verify_certificate = no > require_certificate = no > private_key = /etc/kamailio/example.org.key.pem > certificate = /etc/kamailio/example.org.crt.pem > cipher_list = > ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-RSA- > AES128-GCM-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-RC4-SHA:ECDHE-RSA-AES > 256- > SHA:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-SHA256:DHE-RSA-AES128-GCM- > SHA256:DHE-RSA-AES128-SHA256:DHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA:RC4- > SHA:AES256-GCM-SHA384:AES256-SHA256:CAMELLIA256-SHA:ECDHE-RSA-AES128- > SHA:AES128-GCM-SHA256:AES128-SHA256:AES128-SHA:CAMELLIA128- > SHA:!aNULL:!eNULL:!EXPORT:!DES:!3DES:!MD5:!PSK > > > Thanks. -Anthony

Can you run the following commands: kamcmd cfg.set_now_int core memlog 1 kamcmd corex.shm_summary Then grab the log messages from syslog related to shared memory summary and send them over here. Cheers, Daniel On 17/11/15 14:31, Anthony Messina wrote: > After I reported last night, I restarted Kamailio and even though the 5 > UACs did nothing but ensure they had a registration overnight, this > morning the issue has recurred. The following is the output you > requested. Not sure how the memory is being used up by Kamailio. > > # kamctl stats shmem > shmem:fragments = 181 > shmem:free_size = 8922584 > shmem:max_used_size = 58243792 > shmem:real_used_size = 58186280 > shmem:total_size = 67108864 > shmem:used_size = 54346088 > > On Tuesday, November 17, 2015 09:03:24 AM Daniel-Constantin Mierla wrote: >> As you are using the master branch (development), do you run latest >> version? >> >> Can you look at available shared memory? >> >> kamctl stats shmem >> >> Check it over time and see if the free memory is decreasing. >> >> Cheers, >> Daniel >> >> On 17/11/15 00:44, Anthony Messina wrote: >>> I have noticed the following issue which began with builds somewhere >>> between git master commits bff0a08 and 6173ef7. I did not see this issue >>> with my previous builds and haven't been able to pin down the problem, >>> which is why I haven't formally filed a bug. >>> >>> Any help or guidance is appreciated, because this has crippled my use of >>> Kamailio. Only a restart enables it to work again until the issue >>> recurs. >>> >>> ERROR: tls [tls_server.c:189]: tls_complete_init(): tls: ssl bug #1491 >>> workaround: not enough memory for safe operation: 8870536 >>> ERROR: <core> [tcp_read.c:1303]: tcp_read_req(): ERROR: tcp_read_req: >>> error >>> reading >>> >>> I currently build against and run openssl-1.0.1k-12.fc22.x86_64. >>> >>> I have a very small operation and the only change on the operational >>> side >>> is that all 5 of my mobile UACs (yes, that's all) have switched from >>> CSipSimple/Android to Zoiper/Android, which doesn't yet have support for >>> client-side certificates so verify_certificate and require_certificate >>> are >>> off for both the server and client config. >>> >>> The server is started with: >>> /usr/sbin/kamailio -P /run/kamailio/kamailio.pid -m 64 -M 8 >>> >>> I have tried modifying the shared mem to 128 but the issue still occurs. >>> >>> Even right now, I am seeing the error when only one UAC has established >>> a >>> TLS connection: >>> >>> # kamcmd tls.list >>> { >>> >>> id: 572 >>> timeout: 3475 >>> src_ip: 10.77.79.156 >>> src_port: 58688 >>> dst_ip: 10.77.79.3 >>> dst_port: 5061 >>> cipher: ECDHE-RSA-RC4-SHA SSLv3 Kx=ECDH Au=RSA Enc=RC4(128) >>> Mac=SHA1 >>> ct_wq_size: 0 >>> enc_rd_buf: 0 >>> flags: 2 >>> state: established >>> >>> } >>> >>> # kamailio.cfg >>> enable_tls=yes >>> loadmodule "tls.so" >>> modparam("tls", "connection_timeout", 60) >>> #modparam("tls", "tls_log", 1) >>> #modparam("tls", "tls_debug", 1) >>> #modparam("tls", "low_mem_threshold1", -1) >>> #modparam("tls", "low_mem_threshold2", 0) >>> modparam("tls", "session_cache", 1) >>> >>> # tls.cfg >>> [server:default] >>> method = TLSv1+ >>> verify_certificate = no >>> require_certificate = no >>> private_key = /etc/kamailio/example.org.key.pem >>> certificate = /etc/kamailio/example.org.crt.pem >>> server_name = example.org >>> cipher_list = >>> ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-RSA- >>> AES128-GCM-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-RC4-SHA:ECDHE-RSA-AE >>> S >>> 256- >>> SHA:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-SHA256:DHE-RSA-AES128-GCM- >>> SHA256:DHE-RSA-AES128-SHA256:DHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA:RC4- >>> SHA:AES256-GCM-SHA384:AES256-SHA256:CAMELLIA256-SHA:ECDHE-RSA-AES128- >>> SHA:AES128-GCM-SHA256:AES128-SHA256:AES128-SHA:CAMELLIA128- >>> SHA:!aNULL:!eNULL:!EXPORT:!DES:!3DES:!MD5:!PSK >>> >>> [client:default] >>> method = TLSv1+ >>> verify_certificate = no >>> require_certificate = no >>> private_key = /etc/kamailio/example.org.key.pem >>> certificate = /etc/kamailio/example.org.crt.pem >>> cipher_list = >>> ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-RSA- >>> AES128-GCM-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-RC4-SHA:ECDHE-RSA-AE >>> S >>> 256- >>> SHA:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-SHA256:DHE-RSA-AES128-GCM- >>> SHA256:DHE-RSA-AES128-SHA256:DHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA:RC4- >>> SHA:AES256-GCM-SHA384:AES256-SHA256:CAMELLIA256-SHA:ECDHE-RSA-AES128- >>> SHA:AES128-GCM-SHA256:AES128-SHA256:AES128-SHA:CAMELLIA128- >>> SHA:!aNULL:!eNULL:!EXPORT:!DES:!3DES:!MD5:!PSK >>> >>> >>> Thanks. -Anthony

Looks like a lot of connections being open, can you get the output for: kamcmd core.tcp_info kamcmd tls.info Cheers, Daniel On 17/11/15 14:59, Anthony Messina wrote: > Attached. -A > > On Tuesday, November 17, 2015 02:50:21 PM Daniel-Constantin Mierla wrote: >> Can you run the following commands: >> >> kamcmd cfg.set_now_int core memlog 1 >> kamcmd corex.shm_summary >> >> Then grab the log messages from syslog related to shared memory summary >> and send them over here. >> >> Cheers, >> Daniel >> >> On 17/11/15 14:31, Anthony Messina wrote: >>> After I reported last night, I restarted Kamailio and even though the 5 >>> UACs did nothing but ensure they had a registration overnight, this >>> morning the issue has recurred. The following is the output you >>> requested. Not sure how the memory is being used up by Kamailio. >>> >>> # kamctl stats shmem >>> shmem:fragments = 181 >>> shmem:free_size = 8922584 >>> shmem:max_used_size = 58243792 >>> shmem:real_used_size = 58186280 >>> shmem:total_size = 67108864 >>> shmem:used_size = 54346088 >>> >>> On Tuesday, November 17, 2015 09:03:24 AM Daniel-Constantin Mierla

>>>> As you are using the master branch (development), do you run latest >>>> version? >>>> >>>> Can you look at available shared memory? >>>> >>>> kamctl stats shmem >>>> >>>> Check it over time and see if the free memory is decreasing. >>>> >>>> Cheers, >>>> Daniel >>>> >>>> On 17/11/15 00:44, Anthony Messina wrote: >>>>> I have noticed the following issue which began with builds somewhere >>>>> between git master commits bff0a08 and 6173ef7. I did not see this >>>>> issue >>>>> with my previous builds and haven't been able to pin down the problem, >>>>> which is why I haven't formally filed a bug. >>>>> >>>>> Any help or guidance is appreciated, because this has crippled my use >>>>> of >>>>> Kamailio. Only a restart enables it to work again until the issue >>>>> recurs. >>>>> >>>>> ERROR: tls [tls_server.c:189]: tls_complete_init(): tls: ssl bug #1491 >>>>> workaround: not enough memory for safe operation: 8870536 >>>>> ERROR: <core> [tcp_read.c:1303]: tcp_read_req(): ERROR: tcp_read_req: >>>>> error >>>>> reading >>>>> >>>>> I currently build against and run openssl-1.0.1k-12.fc22.x86_64. >>>>> >>>>> I have a very small operation and the only change on the operational >>>>> side >>>>> is that all 5 of my mobile UACs (yes, that's all) have switched from >>>>> CSipSimple/Android to Zoiper/Android, which doesn't yet have support >>>>> for >>>>> client-side certificates so verify_certificate and require_certificate >>>>> are >>>>> off for both the server and client config. >>>>> >>>>> The server is started with: >>>>> /usr/sbin/kamailio -P /run/kamailio/kamailio.pid -m 64 -M 8 >>>>> >>>>> I have tried modifying the shared mem to 128 but the issue still >>>>> occurs. >>>>> >>>>> Even right now, I am seeing the error when only one UAC has >>>>> established >>>>> a >>>>> TLS connection: >>>>> >>>>> # kamcmd tls.list >>>>> { >>>>> >>>>> id: 572 >>>>> timeout: 3475 >>>>> src_ip: 10.77.79.156 >>>>> src_port: 58688 >>>>> dst_ip: 10.77.79.3 >>>>> dst_port: 5061 >>>>> cipher: ECDHE-RSA-RC4-SHA SSLv3 Kx=ECDH Au=RSA Enc=RC4(128) >>>>> Mac=SHA1 >>>>> ct_wq_size: 0 >>>>> enc_rd_buf: 0 >>>>> flags: 2 >>>>> state: established >>>>> >>>>> } >>>>> >>>>> # kamailio.cfg >>>>> enable_tls=yes >>>>> loadmodule "tls.so" >>>>> modparam("tls", "connection_timeout", 60) >>>>> #modparam("tls", "tls_log", 1) >>>>> #modparam("tls", "tls_debug", 1) >>>>> #modparam("tls", "low_mem_threshold1", -1) >>>>> #modparam("tls", "low_mem_threshold2", 0) >>>>> modparam("tls", "session_cache", 1) >>>>> >>>>> # tls.cfg >>>>> [server:default] >>>>> method = TLSv1+ >>>>> verify_certificate = no >>>>> require_certificate = no >>>>> private_key = /etc/kamailio/example.org.key.pem >>>>> certificate = /etc/kamailio/example.org.crt.pem >>>>> server_name = example.org >>>>> cipher_list = >>>>> ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-RSA- >>>>> AES128-GCM-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-RC4-SHA:ECDHE-RSA- >>>>> AE >>>>> S >>>>> 256- >>>>> SHA:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-SHA256:DHE-RSA-AES128-GCM >>>>> - >>>>> SHA256:DHE-RSA-AES128-SHA256:DHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA:RC4 >>>>> - >>>>> SHA:AES256-GCM-SHA384:AES256-SHA256:CAMELLIA256-SHA:ECDHE-RSA-AES128- >>>>> SHA:AES128-GCM-SHA256:AES128-SHA256:AES128-SHA:CAMELLIA128- >>>>> SHA:!aNULL:!eNULL:!EXPORT:!DES:!3DES:!MD5:!PSK >>>>> >>>>> [client:default] >>>>> method = TLSv1+ >>>>> verify_certificate = no >>>>> require_certificate = no >>>>> private_key = /etc/kamailio/example.org.key.pem >>>>> certificate = /etc/kamailio/example.org.crt.pem >>>>> cipher_list = >>>>> ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-RSA- >>>>> AES128-GCM-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-RC4-SHA:ECDHE-RSA- >>>>> AE >>>>> S >>>>> 256- >>>>> SHA:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-SHA256:DHE-RSA-AES128-GCM >>>>> - >>>>> SHA256:DHE-RSA-AES128-SHA256:DHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA:RC4 >>>>> - >>>>> SHA:AES256-GCM-SHA384:AES256-SHA256:CAMELLIA256-SHA:ECDHE-RSA-AES128- >>>>> SHA:AES128-GCM-SHA256:AES128-SHA256:AES128-SHA:CAMELLIA128- >>>>> SHA:!aNULL:!eNULL:!EXPORT:!DES:!3DES:!MD5:!PSK >>>>> >>>>> >>>>> Thanks. -Anthony

Looking at the logs of last commits, I couldn't spot the change that would add the leak. What is the exact version you are running (kamailio -v)? Are you using any of the functions exported by tcpops? Cheers, Daniel On 17/11/15 15:24, Anthony Messina wrote: > I wish that were the case... > > # kamcmd core.tcp_info > { > > readers: 2 > max_connections: 2048 > max_tls_connections: 2048 > opened_connections: 0 > opened_tls_connections: 0 > write_queued_bytes: 0 > > } > > # kamcmd tls.info > { > > max_connections: 2048 > opened_connections: 0 > clear_text_write_queued_bytes: 0 > > } > > On Tuesday, November 17, 2015 03:08:59 PM Daniel-Constantin Mierla wrote: >> Looks like a lot of connections being open, can you get the output for: >> >> kamcmd core.tcp_info >> >> kamcmd tls.info >> >> Cheers, >> Daniel >> >> On 17/11/15 14:59, Anthony Messina wrote: >>> Attached. -A >>> >>> On Tuesday, November 17, 2015 02:50:21 PM Daniel-Constantin Mierla

>>>> Can you run the following commands: >>>> >>>> kamcmd cfg.set_now_int core memlog 1 >>>> kamcmd corex.shm_summary >>>> >>>> Then grab the log messages from syslog related to shared memory summary >>>> and send them over here. >>>> >>>> Cheers, >>>> Daniel >>>> >>>> On 17/11/15 14:31, Anthony Messina wrote: >>>>> After I reported last night, I restarted Kamailio and even though the >>>>> 5 >>>>> UACs did nothing but ensure they had a registration overnight, this >>>>> morning the issue has recurred. The following is the output you >>>>> requested. Not sure how the memory is being used up by Kamailio. >>>>> >>>>> # kamctl stats shmem >>>>> shmem:fragments = 181 >>>>> shmem:free_size = 8922584 >>>>> shmem:max_used_size = 58243792 >>>>> shmem:real_used_size = 58186280 >>>>> shmem:total_size = 67108864 >>>>> shmem:used_size = 54346088 >>>>> >>>>> On Tuesday, November 17, 2015 09:03:24 AM Daniel-Constantin Mierla > > wrote: >>>>>> As you are using the master branch (development), do you run latest >>>>>> version? >>>>>> >>>>>> Can you look at available shared memory? >>>>>> >>>>>> kamctl stats shmem >>>>>> >>>>>> Check it over time and see if the free memory is decreasing. >>>>>> >>>>>> Cheers, >>>>>> Daniel >>>>>> >>>>>> On 17/11/15 00:44, Anthony Messina wrote: >>>>>>> I have noticed the following issue which began with builds somewhere >>>>>>> between git master commits bff0a08 and 6173ef7. I did not see this >>>>>>> issue >>>>>>> with my previous builds and haven't been able to pin down the >>>>>>> problem, >>>>>>> which is why I haven't formally filed a bug. >>>>>>> >>>>>>> Any help or guidance is appreciated, because this has crippled my >>>>>>> use >>>>>>> of >>>>>>> Kamailio. Only a restart enables it to work again until the issue >>>>>>> recurs. >>>>>>> >>>>>>> ERROR: tls [tls_server.c:189]: tls_complete_init(): tls: ssl bug >>>>>>> #1491 >>>>>>> workaround: not enough memory for safe operation: 8870536 >>>>>>> ERROR: <core> [tcp_read.c:1303]: tcp_read_req(): ERROR: >>>>>>> tcp_read_req: >>>>>>> error >>>>>>> reading >>>>>>> >>>>>>> I currently build against and run openssl-1.0.1k-12.fc22.x86_64. >>>>>>> >>>>>>> I have a very small operation and the only change on the operational >>>>>>> side >>>>>>> is that all 5 of my mobile UACs (yes, that's all) have switched from >>>>>>> CSipSimple/Android to Zoiper/Android, which doesn't yet have support >>>>>>> for >>>>>>> client-side certificates so verify_certificate and >>>>>>> require_certificate >>>>>>> are >>>>>>> off for both the server and client config. >>>>>>> >>>>>>> The server is started with: >>>>>>> /usr/sbin/kamailio -P /run/kamailio/kamailio.pid -m 64 -M 8 >>>>>>> >>>>>>> I have tried modifying the shared mem to 128 but the issue still >>>>>>> occurs. >>>>>>> >>>>>>> Even right now, I am seeing the error when only one UAC has >>>>>>> established >>>>>>> a >>>>>>> TLS connection: >>>>>>> >>>>>>> # kamcmd tls.list >>>>>>> { >>>>>>> >>>>>>> id: 572 >>>>>>> timeout: 3475 >>>>>>> src_ip: 10.77.79.156 >>>>>>> src_port: 58688 >>>>>>> dst_ip: 10.77.79.3 >>>>>>> dst_port: 5061 >>>>>>> cipher: ECDHE-RSA-RC4-SHA SSLv3 Kx=ECDH Au=RSA >>>>>>> Enc=RC4(128) >>>>>>> Mac=SHA1 >>>>>>> ct_wq_size: 0 >>>>>>> enc_rd_buf: 0 >>>>>>> flags: 2 >>>>>>> state: established >>>>>>> >>>>>>> } >>>>>>> >>>>>>> # kamailio.cfg >>>>>>> enable_tls=yes >>>>>>> loadmodule "tls.so" >>>>>>> modparam("tls", "connection_timeout", 60) >>>>>>> #modparam("tls", "tls_log", 1) >>>>>>> #modparam("tls", "tls_debug", 1) >>>>>>> #modparam("tls", "low_mem_threshold1", -1) >>>>>>> #modparam("tls", "low_mem_threshold2", 0) >>>>>>> modparam("tls", "session_cache", 1) >>>>>>> >>>>>>> # tls.cfg >>>>>>> [server:default] >>>>>>> method = TLSv1+ >>>>>>> verify_certificate = no >>>>>>> require_certificate = no >>>>>>> private_key = /etc/kamailio/example.org.key.pem >>>>>>> certificate = /etc/kamailio/example.org.crt.pem >>>>>>> server_name = example.org >>>>>>> cipher_list = >>>>>>> ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-RSA- >>>>>>> AES128-GCM-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-RC4-SHA:ECDHE-RS >>>>>>> A- >>>>>>> AE >>>>>>> S >>>>>>> 256- >>>>>>> SHA:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-SHA256:DHE-RSA-AES128-G >>>>>>> CM >>>>>>> - >>>>>>> SHA256:DHE-RSA-AES128-SHA256:DHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA:R >>>>>>> C4 >>>>>>> - >>>>>>> SHA:AES256-GCM-SHA384:AES256-SHA256:CAMELLIA256-SHA:ECDHE-RSA-AES128 >>>>>>> - >>>>>>> SHA:AES128-GCM-SHA256:AES128-SHA256:AES128-SHA:CAMELLIA128- >>>>>>> SHA:!aNULL:!eNULL:!EXPORT:!DES:!3DES:!MD5:!PSK >>>>>>> >>>>>>> [client:default] >>>>>>> method = TLSv1+ >>>>>>> verify_certificate = no >>>>>>> require_certificate = no >>>>>>> private_key = /etc/kamailio/example.org.key.pem >>>>>>> certificate = /etc/kamailio/example.org.crt.pem >>>>>>> cipher_list = >>>>>>> ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-RSA- >>>>>>> AES128-GCM-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-RC4-SHA:ECDHE-RS >>>>>>> A- >>>>>>> AE >>>>>>> S >>>>>>> 256- >>>>>>> SHA:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-SHA256:DHE-RSA-AES128-G >>>>>>> CM >>>>>>> - >>>>>>> SHA256:DHE-RSA-AES128-SHA256:DHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA:R >>>>>>> C4 >>>>>>> - >>>>>>> SHA:AES256-GCM-SHA384:AES256-SHA256:CAMELLIA256-SHA:ECDHE-RSA-AES128 >>>>>>> - >>>>>>> SHA:AES128-GCM-SHA256:AES128-SHA256:AES128-SHA:CAMELLIA128- >>>>>>> SHA:!aNULL:!eNULL:!EXPORT:!DES:!3DES:!MD5:!PSK >>>>>>> >>>>>>> >>>>>>> Thanks. -Anthony

It is not clear how what sources you are using, what does it mean 'latest release tarball' -- version 4.3.3? Then did you take all the patches from master since version 4.3.0? I tested master with 20000 registrations over TLS sent by sipp, but I couldn't spot any leak there. Can you test with bare master branch? Cheers, Daniel On 18/11/15 02:16, Anthony Messina wrote: > Sorry for the delay, I just got home from my $PAYINGJOB. And thanks a lot > for helping figure this out. > > I build Kamailio RPMs from the latest release tarball, with the changes > between the release and git master applied via patch, but here is the > version output: > > # kamailio -v > version: kamailio 4.4.0-dev6 (x86_64/linux) e275bc > flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, > DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, > Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK- > ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, > USE_DST_BLACKLIST, HAVE_RESOLV_RES > ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, > MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB > poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. > id: e275bc > compiled on 22:23:43 Nov 13 2015 with gcc 5.1.1 > > I am using tcpops functions as follows, though again, these were in use > for > quite some time before the issue appeared: > > To update the tcp connection lifetime after successful auth: > # User authenticated - update tcp_connection_lifetime > if(proto!=UDP) > > tcp_set_connection_lifetime("3605"); > > and in the reply route: > > onreply_route[MANAGE_REPLY] { > > xdbg("incoming reply\n"); > > if(status=~"[12][0-9][0-9]") > > route(NATMANAGE); > > #!ifdef WITH_TCPOPS > > if(proto!=UDP && status=="200") { > > if(is_method("INVITE")) { > > # enable on callee's connection > tcp_keepalive_enable("60", "5", "5"); > # enable on caller's connection > if($avp(caller_conid)!=$null) > > tcp_keepalive_enable("$avp(caller_conid)", > > "60", "5", "2"); > > } > if(is_method("BYE")) { > > tcp_keepalive_disable(); > tcp_keepalive_disable("$avp(bye_conid)"); > > } > > }

I did tow things on 2015-11-19 which seem to have (at least temporarily) resolved this issue: 1. Upgraded to git master@b056aed 2. Commented out #modparam("usrloc", "close_expired_tcp", 1) based on http://lists.sip-router.org/pipermail/sr-users/2015-November/090733.html

I have re-enabled the close_expired_tcp modparam and will report back when I have results. Thanks Camille. -A -- Anthony - https://messinet.com/ On November 23, 2015 3:46:55 AM CST, Camille Oudot &lt;camille.oudot(a)orange.com&gt; wrote: > Le Sun, 22 Nov 2015 15:22:06 -0600, > Anthony Messina &lt;amessina(a)messinet.com&gt; a écrit : > >> I did tow things on 2015-11-19 which seem to have (at least >> temporarily) resolved this issue: >> >> 1. Upgraded to git master@b056aed >> >> 2. Commented out #modparam("usrloc", "close_expired_tcp", 1) based on > >> > http://lists.sip-router.org/pipermail/sr-users/2015-November/090733.html > > Hi Anthony, > > does this memory issue occur again if you re-enable close_expired_tcp > on this latest version you recompiled? This would help isolating the > issue. > Thanks

When the close_expired_tcp modparam was disabled, Kamailio never displayed this warning and continued to process all TLS connections from 2015-11-19 through 2015-11-23, when I re-enabled the close_expired_tcp modparam for this test.