[sr-dev] kamailio 3.1.0 crash on ssl-dos attack

Daniel-Constantin Mierla miconda at gmail.com
Thu Dec 15 11:38:10 CET 2011


Hello,

I know tls_max_connection is not the solution, but in the context of 
this discussion resulted that would be good to have such parameter, so I 
added it -- it was faster that setting a testbed to work on the ssl-dos 
attack as I was traveling. So I thought you can test it a bit as well, 
since you have such config in place, to be sure it works.

Thanks,
Daniel


On 12/14/11 11:21 PM, Jijo wrote:
> Hi,
> As i mentioned in my previous mail that i tested limiting the TLS 
> connection but didn't help. The problem was high 
> frequent renegotiation on existing TLS connections. This was causing 
> the kamailio to restart.
> So i was thinking to disable the renegoitation until OPENSSL comeup 
> with a solution to this issue.
> Thanks
> Jijo
>
> On Wed, Dec 14, 2011 at 5:45 AM, Daniel-Constantin Mierla 
> <miconda at gmail.com <mailto:miconda at gmail.com>> wrote:
>
>     Hello,
>
>     I committed yesterday in master branch the code that adds a new
>     global parameter tls_max_connections which sets a limit on how
>     many active tls connections are on sip server. Since tls
>     connections are over tcp, practically tls_max_connections is not
>     effective if greater than tcp_max_connections, since the last will
>     be reached first.
>
>     Can you give it a try and see if works for you.
>
>     Cheers,
>     Daniel
>
>
>     On 12/10/11 12:53 AM, Jijo wrote:
>>     Hi
>>     Since TLS run on TCP, the max TLS connections accepted is based
>>     on tcp_max_connections.  I thought its better to limit the max
>>     tls connections, since tls require more memory for each tls
>>     connection.
>>     I did look futher into the issue and the kamailio crashes only
>>     when the tool(_thc-ssl-dos)_ is ran with periodic renegotiation.
>>     Kamailo doesn't crash if the renegotiation is disabled in the tool.
>>     The htppd has fixed this issue by providing a flag to disable TLS
>>     renegotation in the server from any client. Please find the httpd
>>     patch at
>>     http://mail-archives.apache.org/mod_mbox/httpd-dev/200911.mbox/raw/%3c20091106030922.GA3764@redhat.com%3e/
>>     So I was thinking to apply the same patch in kamailio also. We
>>     could have a new flag in tls_domain structure to enable or
>>     disable renegotation.
>>     Please have a look and let me know your findings.
>>     Thanks
>>     Jijo
>>     On Fri, Dec 9, 2011 at 1:53 AM, Daniel-Constantin Mierla
>>     <miconda at gmail.com <mailto:miconda at gmail.com>> wrote:
>>
>>         Hello,
>>
>>         my plan is to look at it as I get back to office (being few
>>         days off).
>>
>>         But, isn't tcp_max_connections applied to tls connections? If
>>         not, adding such limit will be good anyhow.
>>
>>         Cheers,
>>         Daniel
>>
>>
>>
>>         On 12/7/11 8:57 PM, Jijo wrote:
>>>         Hi All..
>>>         Any comments/suggestions on this issue?
>>>
>>>         Thanks
>>>         Jijo
>>>
>>>         On Mon, Dec 5, 2011 at 6:56 PM, Jijo <realjijo at gmail.com
>>>         <mailto:realjijo at gmail.com>> wrote:
>>>
>>>             Hello
>>>
>>>             I tested using 3.2 and i got the same error. I couldn't
>>>             get the memlog, when i enable , the system doesn't come up.
>>>
>>>             The latest update from http://www.thc.org/thc-ssl-dos/  
>>>             says that
>>>
>>>             "2011-OCT-24 UPDATE:
>>>             SSL-DOS released. Some organizations already found out
>>>             about this release a while ago and mistakenly identified
>>>             it as an SSL-RENEGOTIATION BUG. This is not true. The
>>>             tool can be modified to work without SSL-RENEGOTIATION
>>>             by just establishing a new TCP connection for every new
>>>             handshake. "
>>>
>>>             So this issue could happen if we do TCP connection for
>>>             every new TLS connection with renegotiation. I believe
>>>             this could be fixed if we could block based on max
>>>             active TLS connections, probably a similar flag for max
>>>             TLS connections like "tcp_max_connections"
>>>
>>>
>>>             Core was generated by `/usr/sbin/kamailio -u swrun -g sw
>>>             -m 160 -f /etc/kamailio/kamailio.cfg'.
>>>
>>>             Program terminated with signal 11, Segmentation fault.
>>>             #0  0xb61cc2e3 in ?? () from
>>>             /usr/lib/kamailio/modules/tls.so
>>>             (gdb) bt
>>>             #0  0xb61cc2e3 in ?? () from
>>>             /usr/lib/kamailio/modules/tls.so
>>>             #1  0xb65199fd in CRYPTO_lock () from
>>>             /lib/libcrypto.so.1.0.0
>>>             #2  0xb658c133 in ?? () from /lib/libcrypto.so.1.0.0
>>>             #3  0xb658d240 in RAND_add () from /lib/libcrypto.so.1.0.0
>>>             #4  0xb6690d50 in ssl3_accept () from /lib/libssl.so.1.0.0
>>>             #5  0xb669bd2f in ssl3_read_bytes () from
>>>             /lib/libssl.so.1.0.0
>>>             #6  0xb66989b9 in ?? () from /lib/libssl.so.1.0.0
>>>             #7  0xb66ae499 in SSL_read () from /lib/libssl.so.1.0.0
>>>             #8  0xb61e1e33 in tls_read_f () from
>>>             /usr/lib/kamailio/modules/tls.so
>>>             #9  0x08177036 in tcp_read_headers ()
>>>             #10 0x081771f6 in tcp_read_req ()
>>>             #11 0x08178480 in ?? ()
>>>             #12 0x0817af77 in tcp_receive_loop ()
>>>             #13 0x08173597 in tcp_init_children ()
>>>             #14 0x080b20bf in main_loop ()
>>>             #15 0x080b38c3 in main ()
>>>             (gdb)
>>>
>>>
>>>
>>>             On Wed, Nov 23, 2011 at 10:35 AM, Daniel-Constantin
>>>             Mierla <miconda at gmail.com <mailto:miconda at gmail.com>> wrote:
>>>
>>>                 Hello,
>>>
>>>                 3.1.0 is not the right choice in 3.1 series, there
>>>                 were many fixes that were added to that release
>>>                 series branch. The latest stable version there is
>>>                 3.1.5. Try with it (even better, try with latest
>>>                 version from git branch 3.1 -- see
>>>                 http://www.kamailio.org/dokuwiki/doku.php/install:kamailio-3.1.x-from-git)
>>>
>>>                 In this way we are sure it is not a bug that was
>>>                 fixed after the time of releasing 3.1.0 -- config
>>>                 file and database is the same as for 3.1.0.
>>>
>>>                 Cheers,
>>>                 Daniel
>>>
>>>
>>>                 On 11/23/11 4:01 PM, Jijo wrote:
>>>>                 Thanks I will attach the logs soon..meanwhile here
>>>>                 is the kamailio and openssl version
>>>>
>>>>                 OB151:~ # /usr/sbin/kamailio -V
>>>>                 version: kamailio 3.1.0 (i386/linux) 21a375
>>>>                 flags: STATS: Off, USE_IPV6, USE_TCP, USE_TLS,
>>>>                 TLS_HOOKS, USE_RAW_SOCKS, USE_MCAST, DNS_IP_HACK,
>>>>                 SHM_MEM, SHM_MMAP, PKG_MALLOC, DBG_QM_MALLOC,
>>>>                 USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE,
>>>>                 USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST,
>>>>                 HAVE_RESOLV_RES
>>>>                 ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE
>>>>                 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE
>>>>                 65535, PKG_SIZE 15MB
>>>>                 poll method support: poll, epoll_lt, epoll_et,
>>>>                 sigio_rt, select.
>>>>                 id: 21a375
>>>>                 compiled on 09:22:51 Nov  4 2011 with gcc 4.5.0
>>>>
>>>>
>>>>                 OB151:~ # openssl version -a
>>>>                 OpenSSL 1.0.0 29 Mar 2010
>>>>                 built on: 2011-05-31 07:52:17.000000000 +0000
>>>>                 platform: linux-elf
>>>>                 options:  bn(64,32) rc4(4x,int)
>>>>                 des(ptr,risc1,16,long) blowfish(idx)
>>>>                 compiler: gcc -fPIC -DOPENSSL_PIC -DZLIB
>>>>                 -DOPENSSL_THREADS -D_REENTRANT -DDSO_DLFCN
>>>>                 -DHAVE_DLFCN_H
>>>>                 -DL_ENDIAN -DTERMIO -O3 -fomit-frame-pointer -Wall
>>>>                 -fomit-frame-pointer -fmessage-length=0 -O2 -Wall
>>>>                 -D_FORTIFY_SOURCE=2 -fstack-protector
>>>>                 -funwind-tables -fasynchronous-unwind-tables -g
>>>>                 -Wa,--noexecstack -fomit-frame-pointer
>>>>                 -fno-strict-aliasing
>>>>                 -DTERMIO -Wall -fstack-protector 
>>>>                 -DOPENSSL_BN_ASM_PART_WORDS -DOPENSSL_IA32_SSE2
>>>>                 -DOPENSSL_BN_ASM_MONT -DSHA1_ASM -DSHA256_ASM
>>>>                 -DSHA512_ASM -DMD5_ASM -DRMD160_ASM -DAES_ASM
>>>>                 -DWHIRLPOOL_ASM
>>>>                 OPENSSLDIR: "/etc/ssl"
>>>>
>>>>
>>>>                 On Wed, Nov 23, 2011 at 4:44 AM, Daniel-Constantin
>>>>                 Mierla <miconda at gmail.com
>>>>                 <mailto:miconda at gmail.com>> wrote:
>>>>
>>>>                     Hello,
>>>>
>>>>                     (discussion kept only on sr-dev as it is very
>>>>                     likely going to require mostly devel interaction).
>>>>
>>>>                     What is the version of kamailio (-V command
>>>>                     line option). Also, send the verision of
>>>>                     openssl library -- there were many bugs in
>>>>                     various lib versions that had to be
>>>>                     workarounded in the module, maybe this is a new
>>>>                     one that has to be fixed.
>>>>
>>>>                     Do you get any error message in the syslog at
>>>>                     the moment of the crash?
>>>>
>>>>                     What would be useful is to get the memory
>>>>                     operations log, you can get it by setting:
>>>>
>>>>                     memdbg=1
>>>>                     memlog=1
>>>>
>>>>                     in config file.
>>>>
>>>>                     Then repeat the test and make the log available
>>>>                     for download somehow (if it is too big), from
>>>>                     start to the moment of the crash.
>>>>
>>>>                     Cheers,
>>>>                     Daniel
>>>>
>>>>
>>>>                     On 11/22/11 11:30 PM, Jijo wrote:
>>>>>                     Hi All,
>>>>>
>>>>>                     Kamailio is resetting when we do TLS
>>>>>                     renegotiation dos attack using the tool
>>>>>                     available at http://www.thc.org/thc-ssl-dos/.
>>>>>
>>>>>                     Anybody looked at this issue? How we could
>>>>>                     resolve it. Any idea?
>>>>>
>>>>>                     The core generated for 3 pid's as below
>>>>>
>>>>>                     Pid 1:
>>>>>
>>>>>                     Core was generated by `/usr/sbin/kamailio -u
>>>>>                     swrun -g sw -m 120 -f /etc/kamailio/kamailio.cfg'.
>>>>>                     Program terminated with signal 11,
>>>>>                     Segmentation fault.
>>>>>                     #0  atomic_inc_int () at atomic/atomic_x86.h:225
>>>>>                     (gdb) bt
>>>>>                     #0  atomic_inc_int () at atomic/atomic_x86.h:225
>>>>>                     #1  cfg_update_local () at cfg/cfg_struct.h:228
>>>>>                     #2  timer_main () at timer.c:994
>>>>>                     #3  0x080b0579 in main_loop () at main.c:1632
>>>>>                     #4  0x080b1be4 in main (argc=9,
>>>>>                     argv=0xbfd61e54) at main.c:2446
>>>>>
>>>>>
>>>>>                     Pid 2:
>>>>>
>>>>>                     Core was generated by `/usr/sbin/kamailio -u
>>>>>                     swrun -g sw -m 120 -f /etc/kamailio/kamailio.cfg'.
>>>>>                     Program terminated with signal 11,
>>>>>                     Segmentation fault.
>>>>>                     #0  0x0819bfe8 in qm_insert_free
>>>>>                     (qm=0xaf6c5000, p=0xb05eec30, file=0xb6fb4140
>>>>>                     "tls: tls_init.c", func=0xb6fb4ce0 "ser_free",
>>>>>                     line=296)
>>>>>                         at mem/q_malloc.c:184
>>>>>                     184                     if (frag->size <=
>>>>>                     f->size) break;
>>>>>                     (gdb) bt
>>>>>                     #0  0x0819bfe8 in qm_insert_free
>>>>>                     (qm=0xaf6c5000, p=0xb05eec30, file=0xb6fb4140
>>>>>                     "tls: tls_init.c", func=0xb6fb4ce0 "ser_free",
>>>>>                     line=296)
>>>>>                         at mem/q_malloc.c:184
>>>>>                     #1  qm_free (qm=0xaf6c5000, p=0xb05eec30,
>>>>>                     file=0xb6fb4140 "tls: tls_init.c",
>>>>>                     func=0xb6fb4ce0 "ser_free", line=296) at
>>>>>                     mem/q_malloc.c:518
>>>>>                     #2  0xb6f95404 in ser_free (ptr=0xb05eec30) at
>>>>>                     tls_init.c:296
>>>>>                     #3  0xb732e9ba in CRYPTO_free (str=0xb05eec30)
>>>>>                     at mem.c:391
>>>>>                     #4  0xb7330bee in int_new_ex_data
>>>>>                     (class_index=5, obj=0xbfd414f4, ad=0xbfd41574)
>>>>>                     at ex_data.c:440
>>>>>                     #5  0xb7330443 in CRYPTO_new_ex_data
>>>>>                     (class_index=5, obj=0xbfd414f4, ad=0xbfd41574)
>>>>>                     at ex_data.c:575
>>>>>                     #6  0xb73dfde3 in X509_STORE_CTX_init
>>>>>                     (ctx=0xbfd414f4, store=0xafd8b3d0,
>>>>>                     x509=0xafe08ff0, chain=0x0) at x509_vfy.c:2114
>>>>>                     #7  0xb74b0f31 in ssl3_output_cert_chain
>>>>>                     (s=0xb0553a10, x=0xafe08ff0) at s3_both.c:349
>>>>>                     #8  0xb74a4728 in ssl3_send_server_certificate
>>>>>                     (s=0xb0553a10) at s3_srvr.c:3034
>>>>>                     #9  0xb74a5879 in ssl3_accept (s=0xb0553a10)
>>>>>                     at s3_srvr.c:353
>>>>>                     #10 0xb74afa8f in ssl3_read_bytes
>>>>>                     (s=0xb0553a10, type=23, buf=0xb0ad44ec "",
>>>>>                     len=4095, peek=0) at s3_pkt.c:1266
>>>>>                     #11 0xb74ac9c9 in ssl3_read_internal
>>>>>                     (s=0xb0553a10, buf=0xb0ad44ec, len=4095,
>>>>>                     peek=0) at s3_lib.c:3265
>>>>>                     #12 0xb74c24a9 in SSL_read (s=0xb0553a10,
>>>>>                     buf=0xb0ad44ec, num=4095) at ssl_lib.c:954
>>>>>                     #13 0xb6fad1c3 in tls_read_f (c=0xb0ad431c,
>>>>>                     flags=0xbfd619c4) at tls_server.c:1058
>>>>>                     #14 0x08171c0e in tcp_read_headers
>>>>>                     (c=0xb0ad431c, read_flags=0xbfd619c4) at
>>>>>                     tcp_read.c:406
>>>>>                     #15 0x08171db8 in tcp_read_req
>>>>>                     (con=0xb0ad431c, bytes_read=0xbfd619cc,
>>>>>                     read_flags=0xbfd619c4) at tcp_read.c:885
>>>>>                     #16 0x08172f67 in handle_io (fm=<value
>>>>>                     optimized out>, events=1, idx=<value optimized
>>>>>                     out>) at tcp_read.c:1234
>>>>>                     #17 0x0817583b in io_wait_loop_epoll
>>>>>                     (unix_sock=89) at io_wait.h:1092
>>>>>                     #18 tcp_receive_loop (unix_sock=89) at
>>>>>                     tcp_read.c:1345
>>>>>                     #19 0x0816e2e9 in tcp_init_children () at
>>>>>                     tcp_main.c:4867
>>>>>                     #20 0x080affb1 in main_loop () at main.c:1646
>>>>>                     #21 0x080b1be4 in main (argc=9,
>>>>>                     argv=0xbfd61e54) at main.c:2446
>>>>>
>>>>>                     Pid 3:
>>>>>
>>>>>                     Core was generated by `/usr/sbin/kamailio -u
>>>>>                     swrun -g sw -m 120 -f /etc/kamailio/kamailio.cfg'.
>>>>>                     Program terminated with signal 11,
>>>>>                     Segmentation fault.
>>>>>                     #0  0xb76c9e7c in memmove () from /lib/libc.so.6
>>>>>                     (gdb) bt
>>>>>                     #0  0xb76c9e7c in memmove () from /lib/libc.so.6
>>>>>                     #1  0x081724e7 in tcp_read_req
>>>>>                     (con=0xb022c8f0, bytes_read=0xbfd619cc,
>>>>>                     read_flags=0xbfd619c4) at tcp_read.c:1026
>>>>>                     #2  0x08172f67 in handle_io (fm=<value
>>>>>                     optimized out>, events=1, idx=<value optimized
>>>>>                     out>) at tcp_read.c:1234
>>>>>                     #3  0x0817583b in io_wait_loop_epoll
>>>>>                     (unix_sock=93) at io_wait.h:1092
>>>>>                     #4  tcp_receive_loop (unix_sock=93) at
>>>>>                     tcp_read.c:1345
>>>>>                     #5  0x0816e2e9 in tcp_init_children () at
>>>>>                     tcp_main.c:4867
>>>>>                     #6  0x080affb1 in main_loop () at main.c:1646
>>>>>                     #7  0x080b1be4 in main (argc=9,
>>>>>                     argv=0xbfd61e54) at main.c:2446
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>                     _______________________________________________
>>>>>                     sr-dev mailing list
>>>>>                     sr-dev at lists.sip-router.org  <mailto:sr-dev at lists.sip-router.org>
>>>>>                     http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-dev
>>>>
>>>>                     -- 
>>>>                     Daniel-Constantin Mierla --http://www.asipto.com  <http://www.asipto.com/>
>>>>                     Kamailio Advanced Training, Dec 5-8, Berlin:http://asipto.com/u/kat
>>>>                     http://linkedin.com/in/miconda  -- http://twitter.com/miconda
>>>>
>>>>
>>>
>>>                 -- 
>>>                 Daniel-Constantin Mierla --http://www.asipto.com  <http://www.asipto.com/>
>>>                 Kamailio Advanced Training, Dec 5-8, Berlin:http://asipto.com/u/kat
>>>                 http://linkedin.com/in/miconda  -- http://twitter.com/miconda
>>>
>>>
>>>
>>
>>         -- 
>>         Daniel-Constantin Mierla --http://www.asipto.com  <http://www.asipto.com/>
>>         http://linkedin.com/in/miconda  -- http://twitter.com/miconda
>>
>>
>>
>>
>>     _______________________________________________
>>     sr-dev mailing list
>>     sr-dev at lists.sip-router.org  <mailto:sr-dev at lists.sip-router.org>
>>     http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-dev
>
>     -- 
>     Daniel-Constantin Mierla --http://www.asipto.com  <http://www.asipto.com/>
>     http://linkedin.com/in/miconda  -- http://twitter.com/miconda
>
>

-- 
Daniel-Constantin Mierla -- http://www.asipto.com
http://linkedin.com/in/miconda -- http://twitter.com/miconda

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.sip-router.org/pipermail/sr-dev/attachments/20111215/24c67fb7/attachment-0001.htm>


More information about the sr-dev mailing list