In the current Kamailio TLS module document, there is a statement about tls.reload being unsafe. But the only way to periodically update CRL without restarting Kamailio is to use tls.reload. In our test with tls.reload for CRL, it seems Kamailio would crash after about 100 times of tls.reload in 5/6 hours. The core dump indicates memory access violation, signal 11. We compiled Kamailio with openssl 1.0.0-fips. Would appreciate some insights on tls.reload and ideas to fix the crash issue. Thanks,
I remember that long time ago there was an email discussing the problem in details. MAybe it was on one of the old mailing lists (ser, openser). IIRC the feature and the detailed discussion way by Jan Janak. Maybe this helps you to refine your Google search.
regards Klaus
On 19.10.2013 21:33, Ding Ma wrote:
In the current Kamailio TLS module document, there is a statement about tls.reload being unsafe. But the only way to periodically update CRL without restarting Kamailio is to use tls.reload. In our test with tls.reload for CRL, it seems Kamailio would crash after about 100 times of tls.reload in 5/6 hours. The core dump indicates memory access violation, signal 11. We compiled Kamailio with openssl 1.0.0-fips. Would appreciate some insights on tls.reload and ideas to fix the crash issue. Thanks,
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
Klaus,
With the information you provided, I did find the emails initiated by Jan Janak on this topic. Thanks. Guess our test with large RSA keys hits one of the race conditions when reloading TLS config, which results in kamailio crash. One thing I'm not quite clear is whether this is an openssl issue or kamailio TLS module issue. And where to start if we'd like to help fix this.
Appreciate your help,
Ding
On 10/21/2013 3:14 AM, Klaus Darilion wrote:
I remember that long time ago there was an email discussing the problem in details. MAybe it was on one of the old mailing lists (ser, openser). IIRC the feature and the detailed discussion way by Jan Janak. Maybe this helps you to refine your Google search.
regards Klaus
On 19.10.2013 21:33, Ding Ma wrote:
In the current Kamailio TLS module document, there is a statement about tls.reload being unsafe. But the only way to periodically update CRL without restarting Kamailio is to use tls.reload. In our test with tls.reload for CRL, it seems Kamailio would crash after about 100 times of tls.reload in 5/6 hours. The core dump indicates memory access violation, signal 11. We compiled Kamailio with openssl 1.0.0-fips. Would appreciate some insights on tls.reload and ideas to fix the crash issue. Thanks,
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
22 okt 2013 kl. 05:20 skrev Ding Ma mading087@gmail.com:
Klaus,
With the information you provided, I did find the emails initiated by Jan Janak on this topic. Thanks.
Can you please provide a URL so that the rest of us can update ourselves?
Guess our test with large RSA keys hits one of the race conditions when reloading TLS config, which results in kamailio crash. One thing I'm not quite clear is whether this is an openssl issue or kamailio TLS module issue. And where to start if we'd like to help fix this.
A problem I have seen and that was fixed in Asterisk was that the application initialized OpenSSL from many different modules instead of from the core application. For instance, using LDAP with TLS will initialize OpenSSL in OpenLDAP libraries that is used by Kamailio - the same for Postgres and possibly other modules. Maybe reloading does something weird here.
Note that this is just a wild guess, but an area where I think Kamailio may need some improvements.
Kevin Fleming developed a wrapper library to solve this issue in Asterisk, maybe that's something we need to look into.
/O
Appreciate your help,
Ding
On 10/21/2013 3:14 AM, Klaus Darilion wrote:
I remember that long time ago there was an email discussing the problem in details. MAybe it was on one of the old mailing lists (ser, openser). IIRC the feature and the detailed discussion way by Jan Janak. Maybe this helps you to refine your Google search.
regards Klaus
On 19.10.2013 21:33, Ding Ma wrote:
In the current Kamailio TLS module document, there is a statement about tls.reload being unsafe. But the only way to periodically update CRL without restarting Kamailio is to use tls.reload. In our test with tls.reload for CRL, it seems Kamailio would crash after about 100 times of tls.reload in 5/6 hours. The core dump indicates memory access violation, signal 11. We compiled Kamailio with openssl 1.0.0-fips. Would appreciate some insights on tls.reload and ideas to fix the crash issue. Thanks,
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
Here is the URL for the tls.reload discussion http://lists.iptel.org/pipermail/serdev/2006-March/007051.html This email chain has quite detailed information about tls module implementation. Multiple modules initializing openssl separately doesn't sound like a potential cause for crash. We'll look into this. Thanks,
Here is the gdb output for the core dump. Hope this can provide some clues. Seems like a double free to one memory block.
Currently logging to "gdb.txt". Future logs will be written to /home/admin/firstcoredump. Logs will be appended to the log file. Output will be logged and displayed. #0 0x00007f62e55448a5 in raise () from /lib64/libc.so.6 No symbol table info available. #1 0x00007f62e5546085 in abort () from /lib64/libc.so.6 No symbol table info available. #2 0x00000000004645d1 in ?? () No symbol table info available. #3 <signal handler called> No symbol table info available. #4 0x0000000000540194 in qm_free () No symbol table info available. #5 0x00007f62d987fe38 in ?? () from /usr/lib64/kamailio/modules/tls.so No symbol table info available. #6 0x00007f62e430ca8d in CRYPTO_free () from /usr/lib64/libcrypto.so.10 No symbol table info available. #7 0x00007f62e467ef40 in ?? () from /usr/lib64/libssl.so.10 No symbol table info available. #8 0x00007f62e467fef5 in SSL_CTX_free () from /usr/lib64/libssl.so.10 No symbol table info available. #9 0x00007f62d9876e31 in tls_free_domain () from /usr/lib64/kamailio/modules/tls.so No symbol table info available. #10 0x00007f62d9877645 in tls_free_cfg () from /usr/lib64/kamailio/modules/tls.so No symbol table info available. #11 0x00007f62d98776dc in tls_destroy_cfg () from /usr/lib64/kamailio/modules/tls.so No symbol table info available. #12 0x00007f62d987dcdf in destroy_tls_h () from /usr/lib64/kamailio/modules/tls.so No symbol table info available. #13 0x000000000046583e in cleanup () No symbol table info available. #14 0x00000000004664df in ?? () No symbol table info available. #15 0x00000000004676c5 in handle_sigs () No symbol table info available. #16 0x0000000000468cfc in main_loop () No symbol table info available. #17 0x000000000046a912 in main () No symbol table info available.
On 10/21/2013 10:30 PM, Olle E. Johansson wrote:
22 okt 2013 kl. 05:20 skrev Ding Ma mading087@gmail.com:
Klaus,
With the information you provided, I did find the emails initiated by Jan Janak on this topic. Thanks.
Can you please provide a URL so that the rest of us can update ourselves?
Guess our test with large RSA keys hits one of the race conditions when reloading TLS config, which results in kamailio crash. One thing I'm not quite clear is whether this is an openssl issue or kamailio TLS module issue. And where to start if we'd like to help fix this.
A problem I have seen and that was fixed in Asterisk was that the application initialized OpenSSL from many different modules instead of from the core application. For instance, using LDAP with TLS will initialize OpenSSL in OpenLDAP libraries that is used by Kamailio - the same for Postgres and possibly other modules. Maybe reloading does something weird here.
Note that this is just a wild guess, but an area where I think Kamailio may need some improvements.
Kevin Fleming developed a wrapper library to solve this issue in Asterisk, maybe that's something we need to look into.
/O
Appreciate your help,
Ding
On 10/21/2013 3:14 AM, Klaus Darilion wrote:
I remember that long time ago there was an email discussing the problem in details. MAybe it was on one of the old mailing lists (ser, openser). IIRC the feature and the detailed discussion way by Jan Janak. Maybe this helps you to refine your Google search.
regards Klaus
On 19.10.2013 21:33, Ding Ma wrote:
In the current Kamailio TLS module document, there is a statement about tls.reload being unsafe. But the only way to periodically update CRL without restarting Kamailio is to use tls.reload. In our test with tls.reload for CRL, it seems Kamailio would crash after about 100 times of tls.reload in 5/6 hours. The core dump indicates memory access violation, signal 11. We compiled Kamailio with openssl 1.0.0-fips. Would appreciate some insights on tls.reload and ideas to fix the crash issue. Thanks,
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
-
Ding Ma -
Klaus Darilion -
Olle E. Johansson