I have noticed the following issue which began with builds somewhere between git master commits bff0a08 and 6173ef7. I did not see this issue with my previous builds and haven't been able to pin down the problem, which is why I haven't formally filed a bug.
Any help or guidance is appreciated, because this has crippled my use of Kamailio. Only a restart enables it to work again until the issue recurs.
ERROR: tls [tls_server.c:189]: tls_complete_init(): tls: ssl bug #1491 workaround: not enough memory for safe operation: 8870536 ERROR: <core> [tcp_read.c:1303]: tcp_read_req(): ERROR: tcp_read_req: error reading
I currently build against and run openssl-1.0.1k-12.fc22.x86_64.
I have a very small operation and the only change on the operational side is that all 5 of my mobile UACs (yes, that's all) have switched from CSipSimple/Android to Zoiper/Android, which doesn't yet have support for client-side certificates so verify_certificate and require_certificate are off for both the server and client config.
The server is started with: /usr/sbin/kamailio -P /run/kamailio/kamailio.pid -m 64 -M 8
I have tried modifying the shared mem to 128 but the issue still occurs.
Even right now, I am seeing the error when only one UAC has established a TLS connection:
# kamcmd tls.list { id: 572 timeout: 3475 src_ip: 10.77.79.156 src_port: 58688 dst_ip: 10.77.79.3 dst_port: 5061 cipher: ECDHE-RSA-RC4-SHA SSLv3 Kx=ECDH Au=RSA Enc=RC4(128) Mac=SHA1 ct_wq_size: 0 enc_rd_buf: 0 flags: 2 state: established }
# kamailio.cfg enable_tls=yes loadmodule "tls.so" modparam("tls", "connection_timeout", 60) #modparam("tls", "tls_log", 1) #modparam("tls", "tls_debug", 1) #modparam("tls", "low_mem_threshold1", -1) #modparam("tls", "low_mem_threshold2", 0) modparam("tls", "session_cache", 1)
# tls.cfg [server:default] method = TLSv1+ verify_certificate = no require_certificate = no private_key = /etc/kamailio/example.org.key.pem certificate = /etc/kamailio/example.org.crt.pem server_name = example.org cipher_list = ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-RSA- AES128-GCM-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-RC4-SHA:ECDHE-RSA-AES256- SHA:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-SHA256:DHE-RSA-AES128-GCM- SHA256:DHE-RSA-AES128-SHA256:DHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA:RC4- SHA:AES256-GCM-SHA384:AES256-SHA256:CAMELLIA256-SHA:ECDHE-RSA-AES128- SHA:AES128-GCM-SHA256:AES128-SHA256:AES128-SHA:CAMELLIA128- SHA:!aNULL:!eNULL:!EXPORT:!DES:!3DES:!MD5:!PSK
[client:default] method = TLSv1+ verify_certificate = no require_certificate = no private_key = /etc/kamailio/example.org.key.pem certificate = /etc/kamailio/example.org.crt.pem cipher_list = ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-RSA- AES128-GCM-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-RC4-SHA:ECDHE-RSA-AES256- SHA:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-SHA256:DHE-RSA-AES128-GCM- SHA256:DHE-RSA-AES128-SHA256:DHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA:RC4- SHA:AES256-GCM-SHA384:AES256-SHA256:CAMELLIA256-SHA:ECDHE-RSA-AES128- SHA:AES128-GCM-SHA256:AES128-SHA256:AES128-SHA:CAMELLIA128- SHA:!aNULL:!eNULL:!EXPORT:!DES:!3DES:!MD5:!PSK
Thanks. -Anthony
As you are using the master branch (development), do you run latest version?
Can you look at available shared memory?
kamctl stats shmem
Check it over time and see if the free memory is decreasing.
Cheers, Daniel
On 17/11/15 00:44, Anthony Messina wrote:
I have noticed the following issue which began with builds somewhere between git master commits bff0a08 and 6173ef7. I did not see this issue with my previous builds and haven't been able to pin down the problem, which is why I haven't formally filed a bug.
Any help or guidance is appreciated, because this has crippled my use of Kamailio. Only a restart enables it to work again until the issue recurs.
ERROR: tls [tls_server.c:189]: tls_complete_init(): tls: ssl bug #1491 workaround: not enough memory for safe operation: 8870536 ERROR: <core> [tcp_read.c:1303]: tcp_read_req(): ERROR: tcp_read_req: error reading
I currently build against and run openssl-1.0.1k-12.fc22.x86_64.
I have a very small operation and the only change on the operational side is that all 5 of my mobile UACs (yes, that's all) have switched from CSipSimple/Android to Zoiper/Android, which doesn't yet have support for client-side certificates so verify_certificate and require_certificate are off for both the server and client config.
The server is started with: /usr/sbin/kamailio -P /run/kamailio/kamailio.pid -m 64 -M 8
I have tried modifying the shared mem to 128 but the issue still occurs.
Even right now, I am seeing the error when only one UAC has established a TLS connection:
# kamcmd tls.list { id: 572 timeout: 3475 src_ip: 10.77.79.156 src_port: 58688 dst_ip: 10.77.79.3 dst_port: 5061 cipher: ECDHE-RSA-RC4-SHA SSLv3 Kx=ECDH Au=RSA Enc=RC4(128) Mac=SHA1 ct_wq_size: 0 enc_rd_buf: 0 flags: 2 state: established }
# kamailio.cfg enable_tls=yes loadmodule "tls.so" modparam("tls", "connection_timeout", 60) #modparam("tls", "tls_log", 1) #modparam("tls", "tls_debug", 1) #modparam("tls", "low_mem_threshold1", -1) #modparam("tls", "low_mem_threshold2", 0) modparam("tls", "session_cache", 1)
# tls.cfg [server:default] method = TLSv1+ verify_certificate = no require_certificate = no private_key = /etc/kamailio/example.org.key.pem certificate = /etc/kamailio/example.org.crt.pem server_name = example.org cipher_list = ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-RSA- AES128-GCM-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-RC4-SHA:ECDHE-RSA-AES256- SHA:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-SHA256:DHE-RSA-AES128-GCM- SHA256:DHE-RSA-AES128-SHA256:DHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA:RC4- SHA:AES256-GCM-SHA384:AES256-SHA256:CAMELLIA256-SHA:ECDHE-RSA-AES128- SHA:AES128-GCM-SHA256:AES128-SHA256:AES128-SHA:CAMELLIA128- SHA:!aNULL:!eNULL:!EXPORT:!DES:!3DES:!MD5:!PSK
[client:default] method = TLSv1+ verify_certificate = no require_certificate = no private_key = /etc/kamailio/example.org.key.pem certificate = /etc/kamailio/example.org.crt.pem cipher_list = ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-RSA- AES128-GCM-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-RC4-SHA:ECDHE-RSA-AES256- SHA:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-SHA256:DHE-RSA-AES128-GCM- SHA256:DHE-RSA-AES128-SHA256:DHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA:RC4- SHA:AES256-GCM-SHA384:AES256-SHA256:CAMELLIA256-SHA:ECDHE-RSA-AES128- SHA:AES128-GCM-SHA256:AES128-SHA256:AES128-SHA:CAMELLIA128- SHA:!aNULL:!eNULL:!EXPORT:!DES:!3DES:!MD5:!PSK
Thanks. -Anthony
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
After I reported last night, I restarted Kamailio and even though the 5 UACs did nothing but ensure they had a registration overnight, this morning the issue has recurred. The following is the output you requested. Not sure how the memory is being used up by Kamailio.
# kamctl stats shmem shmem:fragments = 181 shmem:free_size = 8922584 shmem:max_used_size = 58243792 shmem:real_used_size = 58186280 shmem:total_size = 67108864 shmem:used_size = 54346088
On Tuesday, November 17, 2015 09:03:24 AM Daniel-Constantin Mierla wrote:
As you are using the master branch (development), do you run latest version?
Can you look at available shared memory?
kamctl stats shmem
Check it over time and see if the free memory is decreasing.
Cheers, Daniel
On 17/11/15 00:44, Anthony Messina wrote:
I have noticed the following issue which began with builds somewhere between git master commits bff0a08 and 6173ef7. I did not see this issue with my previous builds and haven't been able to pin down the problem, which is why I haven't formally filed a bug.
Any help or guidance is appreciated, because this has crippled my use of Kamailio. Only a restart enables it to work again until the issue recurs.
ERROR: tls [tls_server.c:189]: tls_complete_init(): tls: ssl bug #1491 workaround: not enough memory for safe operation: 8870536 ERROR: <core> [tcp_read.c:1303]: tcp_read_req(): ERROR: tcp_read_req: error reading
I currently build against and run openssl-1.0.1k-12.fc22.x86_64.
I have a very small operation and the only change on the operational side is that all 5 of my mobile UACs (yes, that's all) have switched from CSipSimple/Android to Zoiper/Android, which doesn't yet have support for client-side certificates so verify_certificate and require_certificate are off for both the server and client config.
The server is started with: /usr/sbin/kamailio -P /run/kamailio/kamailio.pid -m 64 -M 8
I have tried modifying the shared mem to 128 but the issue still occurs.
Even right now, I am seeing the error when only one UAC has established a TLS connection:
# kamcmd tls.list {
id: 572 timeout: 3475 src_ip: 10.77.79.156 src_port: 58688 dst_ip: 10.77.79.3 dst_port: 5061 cipher: ECDHE-RSA-RC4-SHA SSLv3 Kx=ECDH Au=RSA Enc=RC4(128) Mac=SHA1 ct_wq_size: 0 enc_rd_buf: 0 flags: 2 state: established}
# kamailio.cfg enable_tls=yes loadmodule "tls.so" modparam("tls", "connection_timeout", 60) #modparam("tls", "tls_log", 1) #modparam("tls", "tls_debug", 1) #modparam("tls", "low_mem_threshold1", -1) #modparam("tls", "low_mem_threshold2", 0) modparam("tls", "session_cache", 1)
# tls.cfg [server:default] method = TLSv1+ verify_certificate = no require_certificate = no private_key = /etc/kamailio/example.org.key.pem certificate = /etc/kamailio/example.org.crt.pem server_name = example.org cipher_list = ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-RSA- AES128-GCM-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-RC4-SHA:ECDHE-RSA-AES 256- SHA:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-SHA256:DHE-RSA-AES128-GCM- SHA256:DHE-RSA-AES128-SHA256:DHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA:RC4- SHA:AES256-GCM-SHA384:AES256-SHA256:CAMELLIA256-SHA:ECDHE-RSA-AES128- SHA:AES128-GCM-SHA256:AES128-SHA256:AES128-SHA:CAMELLIA128- SHA:!aNULL:!eNULL:!EXPORT:!DES:!3DES:!MD5:!PSK
[client:default] method = TLSv1+ verify_certificate = no require_certificate = no private_key = /etc/kamailio/example.org.key.pem certificate = /etc/kamailio/example.org.crt.pem cipher_list = ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-RSA- AES128-GCM-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-RC4-SHA:ECDHE-RSA-AES 256- SHA:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-SHA256:DHE-RSA-AES128-GCM- SHA256:DHE-RSA-AES128-SHA256:DHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA:RC4- SHA:AES256-GCM-SHA384:AES256-SHA256:CAMELLIA256-SHA:ECDHE-RSA-AES128- SHA:AES128-GCM-SHA256:AES128-SHA256:AES128-SHA:CAMELLIA128- SHA:!aNULL:!eNULL:!EXPORT:!DES:!3DES:!MD5:!PSK
Thanks. -Anthony
Can you run the following commands:
kamcmd cfg.set_now_int core memlog 1 kamcmd corex.shm_summary
Then grab the log messages from syslog related to shared memory summary and send them over here.
Cheers, Daniel
On 17/11/15 14:31, Anthony Messina wrote:
After I reported last night, I restarted Kamailio and even though the 5 UACs did nothing but ensure they had a registration overnight, this morning the issue has recurred. The following is the output you requested. Not sure how the memory is being used up by Kamailio.
# kamctl stats shmem shmem:fragments = 181 shmem:free_size = 8922584 shmem:max_used_size = 58243792 shmem:real_used_size = 58186280 shmem:total_size = 67108864 shmem:used_size = 54346088
On Tuesday, November 17, 2015 09:03:24 AM Daniel-Constantin Mierla wrote:
As you are using the master branch (development), do you run latest version?
Can you look at available shared memory?
kamctl stats shmem
Check it over time and see if the free memory is decreasing.
Cheers, Daniel
On 17/11/15 00:44, Anthony Messina wrote:
I have noticed the following issue which began with builds somewhere between git master commits bff0a08 and 6173ef7. I did not see this issue with my previous builds and haven't been able to pin down the problem, which is why I haven't formally filed a bug.
Any help or guidance is appreciated, because this has crippled my use of Kamailio. Only a restart enables it to work again until the issue recurs.
ERROR: tls [tls_server.c:189]: tls_complete_init(): tls: ssl bug #1491 workaround: not enough memory for safe operation: 8870536 ERROR: <core> [tcp_read.c:1303]: tcp_read_req(): ERROR: tcp_read_req: error reading
I currently build against and run openssl-1.0.1k-12.fc22.x86_64.
I have a very small operation and the only change on the operational side is that all 5 of my mobile UACs (yes, that's all) have switched from CSipSimple/Android to Zoiper/Android, which doesn't yet have support for client-side certificates so verify_certificate and require_certificate are off for both the server and client config.
The server is started with: /usr/sbin/kamailio -P /run/kamailio/kamailio.pid -m 64 -M 8
I have tried modifying the shared mem to 128 but the issue still occurs.
Even right now, I am seeing the error when only one UAC has established a TLS connection:
# kamcmd tls.list {
id: 572 timeout: 3475 src_ip: 10.77.79.156 src_port: 58688 dst_ip: 10.77.79.3 dst_port: 5061 cipher: ECDHE-RSA-RC4-SHA SSLv3 Kx=ECDH Au=RSA Enc=RC4(128) Mac=SHA1 ct_wq_size: 0 enc_rd_buf: 0 flags: 2 state: established}
# kamailio.cfg enable_tls=yes loadmodule "tls.so" modparam("tls", "connection_timeout", 60) #modparam("tls", "tls_log", 1) #modparam("tls", "tls_debug", 1) #modparam("tls", "low_mem_threshold1", -1) #modparam("tls", "low_mem_threshold2", 0) modparam("tls", "session_cache", 1)
# tls.cfg [server:default] method = TLSv1+ verify_certificate = no require_certificate = no private_key = /etc/kamailio/example.org.key.pem certificate = /etc/kamailio/example.org.crt.pem server_name = example.org cipher_list = ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-RSA- AES128-GCM-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-RC4-SHA:ECDHE-RSA-AES 256- SHA:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-SHA256:DHE-RSA-AES128-GCM- SHA256:DHE-RSA-AES128-SHA256:DHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA:RC4- SHA:AES256-GCM-SHA384:AES256-SHA256:CAMELLIA256-SHA:ECDHE-RSA-AES128- SHA:AES128-GCM-SHA256:AES128-SHA256:AES128-SHA:CAMELLIA128- SHA:!aNULL:!eNULL:!EXPORT:!DES:!3DES:!MD5:!PSK
[client:default] method = TLSv1+ verify_certificate = no require_certificate = no private_key = /etc/kamailio/example.org.key.pem certificate = /etc/kamailio/example.org.crt.pem cipher_list = ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-RSA- AES128-GCM-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-RC4-SHA:ECDHE-RSA-AES 256- SHA:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-SHA256:DHE-RSA-AES128-GCM- SHA256:DHE-RSA-AES128-SHA256:DHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA:RC4- SHA:AES256-GCM-SHA384:AES256-SHA256:CAMELLIA256-SHA:ECDHE-RSA-AES128- SHA:AES128-GCM-SHA256:AES128-SHA256:AES128-SHA:CAMELLIA128- SHA:!aNULL:!eNULL:!EXPORT:!DES:!3DES:!MD5:!PSK
Thanks. -Anthony
Attached. -A
On Tuesday, November 17, 2015 02:50:21 PM Daniel-Constantin Mierla wrote:
Can you run the following commands:
kamcmd cfg.set_now_int core memlog 1 kamcmd corex.shm_summary
Then grab the log messages from syslog related to shared memory summary and send them over here.
Cheers, Daniel
On 17/11/15 14:31, Anthony Messina wrote:
After I reported last night, I restarted Kamailio and even though the 5 UACs did nothing but ensure they had a registration overnight, this morning the issue has recurred. The following is the output you requested. Not sure how the memory is being used up by Kamailio.
# kamctl stats shmem shmem:fragments = 181 shmem:free_size = 8922584 shmem:max_used_size = 58243792 shmem:real_used_size = 58186280 shmem:total_size = 67108864 shmem:used_size = 54346088
On Tuesday, November 17, 2015 09:03:24 AM Daniel-Constantin Mierla wrote:
As you are using the master branch (development), do you run latest version?
Can you look at available shared memory?
kamctl stats shmem
Check it over time and see if the free memory is decreasing.
Cheers, Daniel
On 17/11/15 00:44, Anthony Messina wrote:
I have noticed the following issue which began with builds somewhere between git master commits bff0a08 and 6173ef7. I did not see this issue with my previous builds and haven't been able to pin down the problem, which is why I haven't formally filed a bug.
Any help or guidance is appreciated, because this has crippled my use of Kamailio. Only a restart enables it to work again until the issue recurs.
ERROR: tls [tls_server.c:189]: tls_complete_init(): tls: ssl bug #1491 workaround: not enough memory for safe operation: 8870536 ERROR: <core> [tcp_read.c:1303]: tcp_read_req(): ERROR: tcp_read_req: error reading
I currently build against and run openssl-1.0.1k-12.fc22.x86_64.
I have a very small operation and the only change on the operational side is that all 5 of my mobile UACs (yes, that's all) have switched from CSipSimple/Android to Zoiper/Android, which doesn't yet have support for client-side certificates so verify_certificate and require_certificate are off for both the server and client config.
The server is started with: /usr/sbin/kamailio -P /run/kamailio/kamailio.pid -m 64 -M 8
I have tried modifying the shared mem to 128 but the issue still occurs.
Even right now, I am seeing the error when only one UAC has established a TLS connection:
# kamcmd tls.list {
id: 572 timeout: 3475 src_ip: 10.77.79.156 src_port: 58688 dst_ip: 10.77.79.3 dst_port: 5061 cipher: ECDHE-RSA-RC4-SHA SSLv3 Kx=ECDH Au=RSA Enc=RC4(128) Mac=SHA1 ct_wq_size: 0 enc_rd_buf: 0 flags: 2 state: established}
# kamailio.cfg enable_tls=yes loadmodule "tls.so" modparam("tls", "connection_timeout", 60) #modparam("tls", "tls_log", 1) #modparam("tls", "tls_debug", 1) #modparam("tls", "low_mem_threshold1", -1) #modparam("tls", "low_mem_threshold2", 0) modparam("tls", "session_cache", 1)
# tls.cfg [server:default] method = TLSv1+ verify_certificate = no require_certificate = no private_key = /etc/kamailio/example.org.key.pem certificate = /etc/kamailio/example.org.crt.pem server_name = example.org cipher_list = ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-RSA- AES128-GCM-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-RC4-SHA:ECDHE-RSA-AE S 256- SHA:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-SHA256:DHE-RSA-AES128-GCM- SHA256:DHE-RSA-AES128-SHA256:DHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA:RC4- SHA:AES256-GCM-SHA384:AES256-SHA256:CAMELLIA256-SHA:ECDHE-RSA-AES128- SHA:AES128-GCM-SHA256:AES128-SHA256:AES128-SHA:CAMELLIA128- SHA:!aNULL:!eNULL:!EXPORT:!DES:!3DES:!MD5:!PSK
[client:default] method = TLSv1+ verify_certificate = no require_certificate = no private_key = /etc/kamailio/example.org.key.pem certificate = /etc/kamailio/example.org.crt.pem cipher_list = ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-RSA- AES128-GCM-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-RC4-SHA:ECDHE-RSA-AE S 256- SHA:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-SHA256:DHE-RSA-AES128-GCM- SHA256:DHE-RSA-AES128-SHA256:DHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA:RC4- SHA:AES256-GCM-SHA384:AES256-SHA256:CAMELLIA256-SHA:ECDHE-RSA-AES128- SHA:AES128-GCM-SHA256:AES128-SHA256:AES128-SHA:CAMELLIA128- SHA:!aNULL:!eNULL:!EXPORT:!DES:!3DES:!MD5:!PSK
Thanks. -Anthony
Looks like a lot of connections being open, can you get the output for:
kamcmd core.tcp_info
kamcmd tls.info
Cheers, Daniel
On 17/11/15 14:59, Anthony Messina wrote:
Attached. -A
On Tuesday, November 17, 2015 02:50:21 PM Daniel-Constantin Mierla wrote:
Can you run the following commands:
kamcmd cfg.set_now_int core memlog 1 kamcmd corex.shm_summary
Then grab the log messages from syslog related to shared memory summary and send them over here.
Cheers, Daniel
On 17/11/15 14:31, Anthony Messina wrote:
After I reported last night, I restarted Kamailio and even though the 5 UACs did nothing but ensure they had a registration overnight, this morning the issue has recurred. The following is the output you requested. Not sure how the memory is being used up by Kamailio.
# kamctl stats shmem shmem:fragments = 181 shmem:free_size = 8922584 shmem:max_used_size = 58243792 shmem:real_used_size = 58186280 shmem:total_size = 67108864 shmem:used_size = 54346088
On Tuesday, November 17, 2015 09:03:24 AM Daniel-Constantin Mierla wrote:
As you are using the master branch (development), do you run latest version?
Can you look at available shared memory?
kamctl stats shmem
Check it over time and see if the free memory is decreasing.
Cheers, Daniel
On 17/11/15 00:44, Anthony Messina wrote:
I have noticed the following issue which began with builds somewhere between git master commits bff0a08 and 6173ef7. I did not see this issue with my previous builds and haven't been able to pin down the problem, which is why I haven't formally filed a bug.
Any help or guidance is appreciated, because this has crippled my use of Kamailio. Only a restart enables it to work again until the issue recurs.
ERROR: tls [tls_server.c:189]: tls_complete_init(): tls: ssl bug #1491 workaround: not enough memory for safe operation: 8870536 ERROR: <core> [tcp_read.c:1303]: tcp_read_req(): ERROR: tcp_read_req: error reading
I currently build against and run openssl-1.0.1k-12.fc22.x86_64.
I have a very small operation and the only change on the operational side is that all 5 of my mobile UACs (yes, that's all) have switched from CSipSimple/Android to Zoiper/Android, which doesn't yet have support for client-side certificates so verify_certificate and require_certificate are off for both the server and client config.
The server is started with: /usr/sbin/kamailio -P /run/kamailio/kamailio.pid -m 64 -M 8
I have tried modifying the shared mem to 128 but the issue still occurs.
Even right now, I am seeing the error when only one UAC has established a TLS connection:
# kamcmd tls.list {
id: 572 timeout: 3475 src_ip: 10.77.79.156 src_port: 58688 dst_ip: 10.77.79.3 dst_port: 5061 cipher: ECDHE-RSA-RC4-SHA SSLv3 Kx=ECDH Au=RSA Enc=RC4(128) Mac=SHA1 ct_wq_size: 0 enc_rd_buf: 0 flags: 2 state: established}
# kamailio.cfg enable_tls=yes loadmodule "tls.so" modparam("tls", "connection_timeout", 60) #modparam("tls", "tls_log", 1) #modparam("tls", "tls_debug", 1) #modparam("tls", "low_mem_threshold1", -1) #modparam("tls", "low_mem_threshold2", 0) modparam("tls", "session_cache", 1)
# tls.cfg [server:default] method = TLSv1+ verify_certificate = no require_certificate = no private_key = /etc/kamailio/example.org.key.pem certificate = /etc/kamailio/example.org.crt.pem server_name = example.org cipher_list = ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-RSA- AES128-GCM-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-RC4-SHA:ECDHE-RSA-AE S 256- SHA:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-SHA256:DHE-RSA-AES128-GCM- SHA256:DHE-RSA-AES128-SHA256:DHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA:RC4- SHA:AES256-GCM-SHA384:AES256-SHA256:CAMELLIA256-SHA:ECDHE-RSA-AES128- SHA:AES128-GCM-SHA256:AES128-SHA256:AES128-SHA:CAMELLIA128- SHA:!aNULL:!eNULL:!EXPORT:!DES:!3DES:!MD5:!PSK
[client:default] method = TLSv1+ verify_certificate = no require_certificate = no private_key = /etc/kamailio/example.org.key.pem certificate = /etc/kamailio/example.org.crt.pem cipher_list = ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-RSA- AES128-GCM-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-RC4-SHA:ECDHE-RSA-AE S 256- SHA:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-SHA256:DHE-RSA-AES128-GCM- SHA256:DHE-RSA-AES128-SHA256:DHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA:RC4- SHA:AES256-GCM-SHA384:AES256-SHA256:CAMELLIA256-SHA:ECDHE-RSA-AES128- SHA:AES128-GCM-SHA256:AES128-SHA256:AES128-SHA:CAMELLIA128- SHA:!aNULL:!eNULL:!EXPORT:!DES:!3DES:!MD5:!PSK
Thanks. -Anthony
I wish that were the case...
# kamcmd core.tcp_info { readers: 2 max_connections: 2048 max_tls_connections: 2048 opened_connections: 0 opened_tls_connections: 0 write_queued_bytes: 0 }
# kamcmd tls.info { max_connections: 2048 opened_connections: 0 clear_text_write_queued_bytes: 0 }
On Tuesday, November 17, 2015 03:08:59 PM Daniel-Constantin Mierla wrote:
Looks like a lot of connections being open, can you get the output for:
kamcmd core.tcp_info
kamcmd tls.info
Cheers, Daniel
On 17/11/15 14:59, Anthony Messina wrote:
Attached. -A
On Tuesday, November 17, 2015 02:50:21 PM Daniel-Constantin Mierla wrote:
Can you run the following commands:
kamcmd cfg.set_now_int core memlog 1 kamcmd corex.shm_summary
Then grab the log messages from syslog related to shared memory summary and send them over here.
Cheers, Daniel
On 17/11/15 14:31, Anthony Messina wrote:
After I reported last night, I restarted Kamailio and even though the 5 UACs did nothing but ensure they had a registration overnight, this morning the issue has recurred. The following is the output you requested. Not sure how the memory is being used up by Kamailio.
# kamctl stats shmem shmem:fragments = 181 shmem:free_size = 8922584 shmem:max_used_size = 58243792 shmem:real_used_size = 58186280 shmem:total_size = 67108864 shmem:used_size = 54346088
On Tuesday, November 17, 2015 09:03:24 AM Daniel-Constantin Mierla
wrote:
As you are using the master branch (development), do you run latest version?
Can you look at available shared memory?
kamctl stats shmem
Check it over time and see if the free memory is decreasing.
Cheers, Daniel
On 17/11/15 00:44, Anthony Messina wrote:
I have noticed the following issue which began with builds somewhere between git master commits bff0a08 and 6173ef7. I did not see this issue with my previous builds and haven't been able to pin down the problem, which is why I haven't formally filed a bug.
Any help or guidance is appreciated, because this has crippled my use of Kamailio. Only a restart enables it to work again until the issue recurs.
ERROR: tls [tls_server.c:189]: tls_complete_init(): tls: ssl bug #1491 workaround: not enough memory for safe operation: 8870536 ERROR: <core> [tcp_read.c:1303]: tcp_read_req(): ERROR: tcp_read_req: error reading
I currently build against and run openssl-1.0.1k-12.fc22.x86_64.
I have a very small operation and the only change on the operational side is that all 5 of my mobile UACs (yes, that's all) have switched from CSipSimple/Android to Zoiper/Android, which doesn't yet have support for client-side certificates so verify_certificate and require_certificate are off for both the server and client config.
The server is started with: /usr/sbin/kamailio -P /run/kamailio/kamailio.pid -m 64 -M 8
I have tried modifying the shared mem to 128 but the issue still occurs.
Even right now, I am seeing the error when only one UAC has established a TLS connection:
# kamcmd tls.list {
id: 572 timeout: 3475 src_ip: 10.77.79.156 src_port: 58688 dst_ip: 10.77.79.3 dst_port: 5061 cipher: ECDHE-RSA-RC4-SHA SSLv3 Kx=ECDH Au=RSA Enc=RC4(128) Mac=SHA1 ct_wq_size: 0 enc_rd_buf: 0 flags: 2 state: established}
# kamailio.cfg enable_tls=yes loadmodule "tls.so" modparam("tls", "connection_timeout", 60) #modparam("tls", "tls_log", 1) #modparam("tls", "tls_debug", 1) #modparam("tls", "low_mem_threshold1", -1) #modparam("tls", "low_mem_threshold2", 0) modparam("tls", "session_cache", 1)
# tls.cfg [server:default] method = TLSv1+ verify_certificate = no require_certificate = no private_key = /etc/kamailio/example.org.key.pem certificate = /etc/kamailio/example.org.crt.pem server_name = example.org cipher_list = ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-RSA- AES128-GCM-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-RC4-SHA:ECDHE-RSA- AE S 256- SHA:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-SHA256:DHE-RSA-AES128-GCM
SHA256:DHE-RSA-AES128-SHA256:DHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA:RC4
SHA:AES256-GCM-SHA384:AES256-SHA256:CAMELLIA256-SHA:ECDHE-RSA-AES128- SHA:AES128-GCM-SHA256:AES128-SHA256:AES128-SHA:CAMELLIA128- SHA:!aNULL:!eNULL:!EXPORT:!DES:!3DES:!MD5:!PSK
[client:default] method = TLSv1+ verify_certificate = no require_certificate = no private_key = /etc/kamailio/example.org.key.pem certificate = /etc/kamailio/example.org.crt.pem cipher_list = ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-RSA- AES128-GCM-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-RC4-SHA:ECDHE-RSA- AE S 256- SHA:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-SHA256:DHE-RSA-AES128-GCM
SHA256:DHE-RSA-AES128-SHA256:DHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA:RC4
SHA:AES256-GCM-SHA384:AES256-SHA256:CAMELLIA256-SHA:ECDHE-RSA-AES128- SHA:AES128-GCM-SHA256:AES128-SHA256:AES128-SHA:CAMELLIA128- SHA:!aNULL:!eNULL:!EXPORT:!DES:!3DES:!MD5:!PSK
Thanks. -Anthony
Looking at the logs of last commits, I couldn't spot the change that would add the leak.
What is the exact version you are running (kamailio -v)?
Are you using any of the functions exported by tcpops?
Cheers, Daniel
On 17/11/15 15:24, Anthony Messina wrote:
I wish that were the case...
# kamcmd core.tcp_info { readers: 2 max_connections: 2048 max_tls_connections: 2048 opened_connections: 0 opened_tls_connections: 0 write_queued_bytes: 0 }
# kamcmd tls.info { max_connections: 2048 opened_connections: 0 clear_text_write_queued_bytes: 0 }
On Tuesday, November 17, 2015 03:08:59 PM Daniel-Constantin Mierla wrote:
Looks like a lot of connections being open, can you get the output for:
kamcmd core.tcp_info
kamcmd tls.info
Cheers, Daniel
On 17/11/15 14:59, Anthony Messina wrote:
Attached. -A
On Tuesday, November 17, 2015 02:50:21 PM Daniel-Constantin Mierla wrote:
Can you run the following commands:
kamcmd cfg.set_now_int core memlog 1 kamcmd corex.shm_summary
Then grab the log messages from syslog related to shared memory summary and send them over here.
Cheers, Daniel
On 17/11/15 14:31, Anthony Messina wrote:
After I reported last night, I restarted Kamailio and even though the 5 UACs did nothing but ensure they had a registration overnight, this morning the issue has recurred. The following is the output you requested. Not sure how the memory is being used up by Kamailio.
# kamctl stats shmem shmem:fragments = 181 shmem:free_size = 8922584 shmem:max_used_size = 58243792 shmem:real_used_size = 58186280 shmem:total_size = 67108864 shmem:used_size = 54346088
On Tuesday, November 17, 2015 09:03:24 AM Daniel-Constantin Mierla
wrote:
As you are using the master branch (development), do you run latest version?
Can you look at available shared memory?
kamctl stats shmem
Check it over time and see if the free memory is decreasing.
Cheers, Daniel
On 17/11/15 00:44, Anthony Messina wrote: > I have noticed the following issue which began with builds somewhere > between git master commits bff0a08 and 6173ef7. I did not see this > issue > with my previous builds and haven't been able to pin down the problem, > which is why I haven't formally filed a bug. > > Any help or guidance is appreciated, because this has crippled my use > of > Kamailio. Only a restart enables it to work again until the issue > recurs. > > ERROR: tls [tls_server.c:189]: tls_complete_init(): tls: ssl bug #1491 > workaround: not enough memory for safe operation: 8870536 > ERROR: <core> [tcp_read.c:1303]: tcp_read_req(): ERROR: tcp_read_req: > error > reading > > I currently build against and run openssl-1.0.1k-12.fc22.x86_64. > > I have a very small operation and the only change on the operational > side > is that all 5 of my mobile UACs (yes, that's all) have switched from > CSipSimple/Android to Zoiper/Android, which doesn't yet have support > for > client-side certificates so verify_certificate and require_certificate > are > off for both the server and client config. > > The server is started with: > /usr/sbin/kamailio -P /run/kamailio/kamailio.pid -m 64 -M 8 > > I have tried modifying the shared mem to 128 but the issue still > occurs. > > Even right now, I am seeing the error when only one UAC has > established > a > TLS connection: > > # kamcmd tls.list > { > > id: 572 > timeout: 3475 > src_ip: 10.77.79.156 > src_port: 58688 > dst_ip: 10.77.79.3 > dst_port: 5061 > cipher: ECDHE-RSA-RC4-SHA SSLv3 Kx=ECDH Au=RSA Enc=RC4(128) > Mac=SHA1 > ct_wq_size: 0 > enc_rd_buf: 0 > flags: 2 > state: established > > } > > # kamailio.cfg > enable_tls=yes > loadmodule "tls.so" > modparam("tls", "connection_timeout", 60) > #modparam("tls", "tls_log", 1) > #modparam("tls", "tls_debug", 1) > #modparam("tls", "low_mem_threshold1", -1) > #modparam("tls", "low_mem_threshold2", 0) > modparam("tls", "session_cache", 1) > > # tls.cfg > [server:default] > method = TLSv1+ > verify_certificate = no > require_certificate = no > private_key = /etc/kamailio/example.org.key.pem > certificate = /etc/kamailio/example.org.crt.pem > server_name = example.org > cipher_list = > ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-RSA- > AES128-GCM-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-RC4-SHA:ECDHE-RSA- > AE > S > 256- > SHA:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-SHA256:DHE-RSA-AES128-GCM > - > SHA256:DHE-RSA-AES128-SHA256:DHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA:RC4 > - > SHA:AES256-GCM-SHA384:AES256-SHA256:CAMELLIA256-SHA:ECDHE-RSA-AES128- > SHA:AES128-GCM-SHA256:AES128-SHA256:AES128-SHA:CAMELLIA128- > SHA:!aNULL:!eNULL:!EXPORT:!DES:!3DES:!MD5:!PSK > > [client:default] > method = TLSv1+ > verify_certificate = no > require_certificate = no > private_key = /etc/kamailio/example.org.key.pem > certificate = /etc/kamailio/example.org.crt.pem > cipher_list = > ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-RSA- > AES128-GCM-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-RC4-SHA:ECDHE-RSA- > AE > S > 256- > SHA:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-SHA256:DHE-RSA-AES128-GCM > - > SHA256:DHE-RSA-AES128-SHA256:DHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA:RC4 > - > SHA:AES256-GCM-SHA384:AES256-SHA256:CAMELLIA256-SHA:ECDHE-RSA-AES128- > SHA:AES128-GCM-SHA256:AES128-SHA256:AES128-SHA:CAMELLIA128- > SHA:!aNULL:!eNULL:!EXPORT:!DES:!3DES:!MD5:!PSK > > > Thanks. -Anthony
Sorry for the delay, I just got home from my $PAYINGJOB. And thanks a lot for helping figure this out.
I build Kamailio RPMs from the latest release tarball, with the changes between the release and git master applied via patch, but here is the version output:
# kamailio -v version: kamailio 4.4.0-dev6 (x86_64/linux) e275bc flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK- ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: e275bc compiled on 22:23:43 Nov 13 2015 with gcc 5.1.1
I am using tcpops functions as follows, though again, these were in use for quite some time before the issue appeared:
To update the tcp connection lifetime after successful auth: # User authenticated - update tcp_connection_lifetime if(proto!=UDP) tcp_set_connection_lifetime("3605");
and in the reply route:
onreply_route[MANAGE_REPLY] { xdbg("incoming reply\n");
if(status=~"[12][0-9][0-9]") route(NATMANAGE);
#!ifdef WITH_TCPOPS if(proto!=UDP && status=="200") { if(is_method("INVITE")) { # enable on callee's connection tcp_keepalive_enable("60", "5", "5"); # enable on caller's connection if($avp(caller_conid)!=$null) tcp_keepalive_enable("$avp(caller_conid)", "60", "5", "2"); } if(is_method("BYE")) { tcp_keepalive_disable(); tcp_keepalive_disable("$avp(bye_conid)"); } }
On Tuesday, November 17, 2015 09:39:28 PM Daniel-Constantin Mierla wrote:
Looking at the logs of last commits, I couldn't spot the change that would add the leak.
What is the exact version you are running (kamailio -v)?
Are you using any of the functions exported by tcpops?
Cheers, Daniel
On 17/11/15 15:24, Anthony Messina wrote:
I wish that were the case...
# kamcmd core.tcp_info {
readers: 2 max_connections: 2048 max_tls_connections: 2048 opened_connections: 0 opened_tls_connections: 0 write_queued_bytes: 0}
# kamcmd tls.info {
max_connections: 2048 opened_connections: 0 clear_text_write_queued_bytes: 0}
On Tuesday, November 17, 2015 03:08:59 PM Daniel-Constantin Mierla wrote:
Looks like a lot of connections being open, can you get the output for:
kamcmd core.tcp_info
kamcmd tls.info
Cheers, Daniel
On 17/11/15 14:59, Anthony Messina wrote:
Attached. -A
On Tuesday, November 17, 2015 02:50:21 PM Daniel-Constantin Mierla
wrote:
Can you run the following commands:
kamcmd cfg.set_now_int core memlog 1 kamcmd corex.shm_summary
Then grab the log messages from syslog related to shared memory summary and send them over here.
Cheers, Daniel
On 17/11/15 14:31, Anthony Messina wrote:
After I reported last night, I restarted Kamailio and even though the 5 UACs did nothing but ensure they had a registration overnight, this morning the issue has recurred. The following is the output you requested. Not sure how the memory is being used up by Kamailio.
# kamctl stats shmem shmem:fragments = 181 shmem:free_size = 8922584 shmem:max_used_size = 58243792 shmem:real_used_size = 58186280 shmem:total_size = 67108864 shmem:used_size = 54346088
On Tuesday, November 17, 2015 09:03:24 AM Daniel-Constantin Mierla
wrote:
> As you are using the master branch (development), do you run latest > version? > > Can you look at available shared memory? > > kamctl stats shmem > > Check it over time and see if the free memory is decreasing. > > Cheers, > Daniel > > On 17/11/15 00:44, Anthony Messina wrote: >> I have noticed the following issue which began with builds somewhere >> between git master commits bff0a08 and 6173ef7. I did not see this >> issue >> with my previous builds and haven't been able to pin down the >> problem, >> which is why I haven't formally filed a bug. >> >> Any help or guidance is appreciated, because this has crippled my >> use >> of >> Kamailio. Only a restart enables it to work again until the issue >> recurs. >> >> ERROR: tls [tls_server.c:189]: tls_complete_init(): tls: ssl bug >> #1491 >> workaround: not enough memory for safe operation: 8870536 >> ERROR: <core> [tcp_read.c:1303]: tcp_read_req(): ERROR: >> tcp_read_req: >> error >> reading >> >> I currently build against and run openssl-1.0.1k-12.fc22.x86_64. >> >> I have a very small operation and the only change on the operational >> side >> is that all 5 of my mobile UACs (yes, that's all) have switched from >> CSipSimple/Android to Zoiper/Android, which doesn't yet have support >> for >> client-side certificates so verify_certificate and >> require_certificate >> are >> off for both the server and client config. >> >> The server is started with: >> /usr/sbin/kamailio -P /run/kamailio/kamailio.pid -m 64 -M 8 >> >> I have tried modifying the shared mem to 128 but the issue still >> occurs. >> >> Even right now, I am seeing the error when only one UAC has >> established >> a >> TLS connection: >> >> # kamcmd tls.list >> { >> >> id: 572 >> timeout: 3475 >> src_ip: 10.77.79.156 >> src_port: 58688 >> dst_ip: 10.77.79.3 >> dst_port: 5061 >> cipher: ECDHE-RSA-RC4-SHA SSLv3 Kx=ECDH Au=RSA >> Enc=RC4(128) >> Mac=SHA1 >> ct_wq_size: 0 >> enc_rd_buf: 0 >> flags: 2 >> state: established >> >> } >> >> # kamailio.cfg >> enable_tls=yes >> loadmodule "tls.so" >> modparam("tls", "connection_timeout", 60) >> #modparam("tls", "tls_log", 1) >> #modparam("tls", "tls_debug", 1) >> #modparam("tls", "low_mem_threshold1", -1) >> #modparam("tls", "low_mem_threshold2", 0) >> modparam("tls", "session_cache", 1) >> >> # tls.cfg >> [server:default] >> method = TLSv1+ >> verify_certificate = no >> require_certificate = no >> private_key = /etc/kamailio/example.org.key.pem >> certificate = /etc/kamailio/example.org.crt.pem >> server_name = example.org >> cipher_list = >> ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-RSA- >> AES128-GCM-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-RC4-SHA:ECDHE-RS >> A- >> AE >> S >> 256- >> SHA:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-SHA256:DHE-RSA-AES128-G >> CM >> - >> SHA256:DHE-RSA-AES128-SHA256:DHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA:R >> C4 >> - >> SHA:AES256-GCM-SHA384:AES256-SHA256:CAMELLIA256-SHA:ECDHE-RSA-AES128 >> - >> SHA:AES128-GCM-SHA256:AES128-SHA256:AES128-SHA:CAMELLIA128- >> SHA:!aNULL:!eNULL:!EXPORT:!DES:!3DES:!MD5:!PSK >> >> [client:default] >> method = TLSv1+ >> verify_certificate = no >> require_certificate = no >> private_key = /etc/kamailio/example.org.key.pem >> certificate = /etc/kamailio/example.org.crt.pem >> cipher_list = >> ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-RSA- >> AES128-GCM-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-RC4-SHA:ECDHE-RS >> A- >> AE >> S >> 256- >> SHA:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-SHA256:DHE-RSA-AES128-G >> CM >> - >> SHA256:DHE-RSA-AES128-SHA256:DHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA:R >> C4 >> - >> SHA:AES256-GCM-SHA384:AES256-SHA256:CAMELLIA256-SHA:ECDHE-RSA-AES128 >> - >> SHA:AES128-GCM-SHA256:AES128-SHA256:AES128-SHA:CAMELLIA128- >> SHA:!aNULL:!eNULL:!EXPORT:!DES:!3DES:!MD5:!PSK >> >> >> Thanks. -Anthony
It is not clear how what sources you are using, what does it mean 'latest release tarball' -- version 4.3.3? Then did you take all the patches from master since version 4.3.0?
I tested master with 20000 registrations over TLS sent by sipp, but I couldn't spot any leak there.
Can you test with bare master branch?
Cheers, Daniel
On 18/11/15 02:16, Anthony Messina wrote:
Sorry for the delay, I just got home from my $PAYINGJOB. And thanks a lot for helping figure this out.
I build Kamailio RPMs from the latest release tarball, with the changes between the release and git master applied via patch, but here is the version output:
# kamailio -v version: kamailio 4.4.0-dev6 (x86_64/linux) e275bc flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK- ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: e275bc compiled on 22:23:43 Nov 13 2015 with gcc 5.1.1
I am using tcpops functions as follows, though again, these were in use for quite some time before the issue appeared:
To update the tcp connection lifetime after successful auth: # User authenticated - update tcp_connection_lifetime if(proto!=UDP) tcp_set_connection_lifetime("3605");
and in the reply route:
onreply_route[MANAGE_REPLY] { xdbg("incoming reply\n");
if(status=~"[12][0-9][0-9]") route(NATMANAGE);#!ifdef WITH_TCPOPS if(proto!=UDP && status=="200") { if(is_method("INVITE")) { # enable on callee's connection tcp_keepalive_enable("60", "5", "5"); # enable on caller's connection if($avp(caller_conid)!=$null) tcp_keepalive_enable("$avp(caller_conid)", "60", "5", "2"); } if(is_method("BYE")) { tcp_keepalive_disable(); tcp_keepalive_disable("$avp(bye_conid)"); } }
I was just letting you know how I build it, but yes, I will test with just the bare master branch this weekend. After a restart, this issue takes a few hours to happen, making it difficult to reproduce in testing. -A
On Wednesday, November 18, 2015 10:30:16 AM Daniel-Constantin Mierla wrote:
It is not clear how what sources you are using, what does it mean 'latest release tarball' -- version 4.3.3? Then did you take all the patches from master since version 4.3.0?
I tested master with 20000 registrations over TLS sent by sipp, but I couldn't spot any leak there.
Can you test with bare master branch?
Cheers, Daniel
On 18/11/15 02:16, Anthony Messina wrote:
Sorry for the delay, I just got home from my $PAYINGJOB. And thanks a lot for helping figure this out.
I build Kamailio RPMs from the latest release tarball, with the changes between the release and git master applied via patch, but here is the version output:
# kamailio -v version: kamailio 4.4.0-dev6 (x86_64/linux) e275bc flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK- ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: e275bc compiled on 22:23:43 Nov 13 2015 with gcc 5.1.1
I am using tcpops functions as follows, though again, these were in use for quite some time before the issue appeared:
To update the tcp connection lifetime after successful auth: # User authenticated - update tcp_connection_lifetime if(proto!=UDP)
tcp_set_connection_lifetime("3605");and in the reply route:
onreply_route[MANAGE_REPLY] {
xdbg("incoming reply\n"); if(status=~"[12][0-9][0-9]") route(NATMANAGE);#!ifdef WITH_TCPOPS
if(proto!=UDP && status=="200") { if(is_method("INVITE")) { # enable on callee's connection tcp_keepalive_enable("60", "5", "5"); # enable on caller's connection if($avp(caller_conid)!=$null) tcp_keepalive_enable("$avp(caller_conid)","60", "5", "2");
} if(is_method("BYE")) { tcp_keepalive_disable(); tcp_keepalive_disable("$avp(bye_conid)"); } }
I did tow things on 2015-11-19 which seem to have (at least temporarily) resolved this issue:
1. Upgraded to git master@b056aed
2. Commented out #modparam("usrloc", "close_expired_tcp", 1) based on http://lists.sip-router.org/pipermail/sr-users/2015-November/090733.html
-A
On Wednesday, November 18, 2015 06:26:46 AM Anthony Messina wrote:
I was just letting you know how I build it, but yes, I will test with just the bare master branch this weekend. After a restart, this issue takes a few hours to happen, making it difficult to reproduce in testing. -A
On Wednesday, November 18, 2015 10:30:16 AM Daniel-Constantin Mierla wrote:
It is not clear how what sources you are using, what does it mean 'latest release tarball' -- version 4.3.3? Then did you take all the patches from master since version 4.3.0?
I tested master with 20000 registrations over TLS sent by sipp, but I couldn't spot any leak there.
Can you test with bare master branch?
Cheers, Daniel
On 18/11/15 02:16, Anthony Messina wrote:
Sorry for the delay, I just got home from my $PAYINGJOB. And thanks a lot for helping figure this out.
I build Kamailio RPMs from the latest release tarball, with the changes between the release and git master applied via patch, but here is the version output:
# kamailio -v version: kamailio 4.4.0-dev6 (x86_64/linux) e275bc flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK- ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: e275bc compiled on 22:23:43 Nov 13 2015 with gcc 5.1.1
I am using tcpops functions as follows, though again, these were in use for quite some time before the issue appeared:
To update the tcp connection lifetime after successful auth: # User authenticated - update tcp_connection_lifetime if(proto!=UDP)
tcp_set_connection_lifetime("3605");and in the reply route:
onreply_route[MANAGE_REPLY] {
xdbg("incoming reply\n"); if(status=~"[12][0-9][0-9]") route(NATMANAGE);#!ifdef WITH_TCPOPS
if(proto!=UDP && status=="200") { if(is_method("INVITE")) { # enable on callee's connection tcp_keepalive_enable("60", "5", "5"); # enable on caller's connection if($avp(caller_conid)!=$null) tcp_keepalive_enable("$avp(caller_conid) ","60", "5", "2");
} if(is_method("BYE")) { tcp_keepalive_disable(); tcp_keepalive_disable("$avp(bye_conid)"); } }
Le Sun, 22 Nov 2015 15:22:06 -0600, Anthony Messina amessina@messinet.com a écrit :
I did tow things on 2015-11-19 which seem to have (at least temporarily) resolved this issue:
Upgraded to git master@b056aed
Commented out #modparam("usrloc", "close_expired_tcp", 1) based on
http://lists.sip-router.org/pipermail/sr-users/2015-November/090733.html
Hi Anthony,
does this memory issue occur again if you re-enable close_expired_tcp on this latest version you recompiled? This would help isolating the issue. Thanks
I have re-enabled the close_expired_tcp modparam and will report back when I have results. Thanks Camille. -A
After having re-enabled the close_expired_tcp modparam, Kamailio made it about 12hours before giving the following warning and blocking new TLS connections again:
ERROR: tls [tls_server.c:189]: tls_complete_init(): tls: ssl bug #1491 workaround: not enough memory for safe operation: 8870536 ERROR: <core> [tcp_read.c:1303]: tcp_read_req(): ERROR: tcp_read_req: error reading
When the close_expired_tcp modparam was disabled, Kamailio never displayed this warning and continued to process all TLS connections from 2015-11-19 through 2015-11-23, when I re-enabled the close_expired_tcp modparam for this test.
Thanks. -A
Quoting Anthony Messina amessina@messinet.com:
I have re-enabled the close_expired_tcp modparam and will report back when I have results. Thanks Camille. -A -- Anthony - https://messinet.com/
On November 23, 2015 3:46:55 AM CST, Camille Oudot camille.oudot@orange.com wrote:
Le Sun, 22 Nov 2015 15:22:06 -0600, Anthony Messina amessina@messinet.com a écrit :
I did tow things on 2015-11-19 which seem to have (at least temporarily) resolved this issue:
Upgraded to git master@b056aed
Commented out #modparam("usrloc", "close_expired_tcp", 1) based on
http://lists.sip-router.org/pipermail/sr-users/2015-November/090733.html
Hi Anthony,
does this memory issue occur again if you re-enable close_expired_tcp on this latest version you recompiled? This would help isolating the issue. Thanks
Le Tue, 24 Nov 2015 09:49:36 -0600, Anthony Messina amessina@messinet.com a écrit :
After having re-enabled the close_expired_tcp modparam, Kamailio made it about 12hours before giving the following warning and blocking new TLS connections again:
ERROR: tls [tls_server.c:189]: tls_complete_init(): tls: ssl bug #1491 workaround: not enough memory for safe operation: 8870536 ERROR: <core> [tcp_read.c:1303]: tcp_read_req(): ERROR: tcp_read_req: error reading
When the close_expired_tcp modparam was disabled, Kamailio never displayed this warning and continued to process all TLS connections from 2015-11-19 through 2015-11-23, when I re-enabled the close_expired_tcp modparam for this test.
Thanks for this feedback, I'll be working on it in the next days.
Le Tue, 24 Nov 2015 09:49:36 -0600, Anthony Messina amessina@messinet.com a écrit :
When the close_expired_tcp modparam was disabled, Kamailio never displayed this warning and continued to process all TLS connections from 2015-11-19 through 2015-11-23, when I re-enabled the close_expired_tcp modparam for this test.
The latest commit in master and in the 4.3 branch should fix it. Thanks for reporting the issue.
-
Anthony Messina -
Camille Oudot -
Daniel-Constantin Mierla