[sr-dev] kamailio 3.1.0 crash on ssl-dos attack
Jijo
realjijo at gmail.com
Mon Dec 19 23:04:56 CET 2011
HI,
I merged the changes to kamailio 3.1.0 and tested. I set
tls_max_connections to 6, but i don't see the connections closed after 6.
kamailio still accepting the connection.
sercmd> core.tcp_info
{
readers: 20
max_connections: 2048
max_tls_connections: 6
opened_connections: 357
opened_tls_connections: 353
write_queued_bytes: 0
}
Thanks for implementing tls renegotiation, i tested that too, which is
working fine.
Thanks
JIjo
On Thu, Dec 15, 2011 at 5:38 AM, Daniel-Constantin Mierla <miconda at gmail.com
> wrote:
> Hello,
>
> I know tls_max_connection is not the solution, but in the context of this
> discussion resulted that would be good to have such parameter, so I added
> it -- it was faster that setting a testbed to work on the ssl-dos attack as
> I was traveling. So I thought you can test it a bit as well, since you have
> such config in place, to be sure it works.
>
> Thanks,
> Daniel
>
>
>
> On 12/14/11 11:21 PM, Jijo wrote:
>
> Hi,
>
> As i mentioned in my previous mail that i tested limiting the TLS
> connection but didn't help. The problem was high frequent renegotiation on
> existing TLS connections. This was causing the kamailio to restart.
>
> So i was thinking to disable the renegoitation until OPENSSL comeup with a
> solution to this issue.
>
> Thanks
> Jijo
>
> On Wed, Dec 14, 2011 at 5:45 AM, Daniel-Constantin Mierla <
> miconda at gmail.com> wrote:
>
>> Hello,
>>
>> I committed yesterday in master branch the code that adds a new global
>> parameter tls_max_connections which sets a limit on how many active tls
>> connections are on sip server. Since tls connections are over tcp,
>> practically tls_max_connections is not effective if greater than
>> tcp_max_connections, since the last will be reached first.
>>
>> Can you give it a try and see if works for you.
>>
>> Cheers,
>> Daniel
>>
>>
>> On 12/10/11 12:53 AM, Jijo wrote:
>>
>> Hi
>>
>> Since TLS run on TCP, the max TLS connections accepted is based on
>> tcp_max_connections. I thought its better to limit the max tls
>> connections, since tls require more memory for each tls connection.
>>
>> I did look futher into the issue and the kamailio crashes only when the
>> tool(*thc-ssl-dos)* is ran with periodic renegotiation. Kamailo doesn't
>> crash if the renegotiation is disabled in the tool.
>>
>> The htppd has fixed this issue by providing a flag to disable TLS
>> renegotation in the server from any client. Please find the httpd patch at
>>
>> http://mail-archives.apache.org/mod_mbox/httpd-dev/200911.mbox/raw/%3c20091106030922.GA3764@redhat.com%3e/
>>
>> So I was thinking to apply the same patch in kamailio also. We could have
>> a new flag in tls_domain structure to enable or disable renegotation.
>>
>> Please have a look and let me know your findings.
>>
>> Thanks
>> Jijo
>>
>>
>> On Fri, Dec 9, 2011 at 1:53 AM, Daniel-Constantin Mierla <
>> miconda at gmail.com> wrote:
>>
>>> Hello,
>>>
>>> my plan is to look at it as I get back to office (being few days off).
>>>
>>> But, isn't tcp_max_connections applied to tls connections? If not,
>>> adding such limit will be good anyhow.
>>>
>>> Cheers,
>>> Daniel
>>>
>>>
>>>
>>> On 12/7/11 8:57 PM, Jijo wrote:
>>>
>>> Hi All..
>>>
>>> Any comments/suggestions on this issue?
>>>
>>> Thanks
>>> Jijo
>>>
>>> On Mon, Dec 5, 2011 at 6:56 PM, Jijo <realjijo at gmail.com> wrote:
>>>
>>>> Hello
>>>>
>>>> I tested using 3.2 and i got the same error. I couldn't get the memlog,
>>>> when i enable , the system doesn't come up.
>>>>
>>>> The latest update from http://www.thc.org/thc-ssl-dos/ says that
>>>>
>>>> "2011-OCT-24 UPDATE:
>>>> SSL-DOS released. Some organizations already found out about this
>>>> release a while ago and mistakenly identified it as an SSL-RENEGOTIATION
>>>> BUG. This is not true. The tool can be modified to work without
>>>> SSL-RENEGOTIATION by just establishing a new TCP connection for every new
>>>> handshake. "
>>>>
>>>> So this issue could happen if we do TCP connection for every new TLS
>>>> connection with renegotiation. I believe this could be fixed if we could
>>>> block based on max active TLS connections, probably a similar flag for max
>>>> TLS connections like "tcp_max_connections"
>>>>
>>>>
>>>> Core was generated by `/usr/sbin/kamailio -u swrun -g sw -m 160 -f
>>>> /etc/kamailio/kamailio.cfg'.
>>>>
>>>> Program terminated with signal 11, Segmentation fault.
>>>> #0 0xb61cc2e3 in ?? () from /usr/lib/kamailio/modules/tls.so
>>>> (gdb) bt
>>>> #0 0xb61cc2e3 in ?? () from /usr/lib/kamailio/modules/tls.so
>>>> #1 0xb65199fd in CRYPTO_lock () from /lib/libcrypto.so.1.0.0
>>>> #2 0xb658c133 in ?? () from /lib/libcrypto.so.1.0.0
>>>> #3 0xb658d240 in RAND_add () from /lib/libcrypto.so.1.0.0
>>>> #4 0xb6690d50 in ssl3_accept () from /lib/libssl.so.1.0.0
>>>> #5 0xb669bd2f in ssl3_read_bytes () from /lib/libssl.so.1.0.0
>>>> #6 0xb66989b9 in ?? () from /lib/libssl.so.1.0.0
>>>> #7 0xb66ae499 in SSL_read () from /lib/libssl.so.1.0.0
>>>> #8 0xb61e1e33 in tls_read_f () from /usr/lib/kamailio/modules/tls.so
>>>> #9 0x08177036 in tcp_read_headers ()
>>>> #10 0x081771f6 in tcp_read_req ()
>>>> #11 0x08178480 in ?? ()
>>>> #12 0x0817af77 in tcp_receive_loop ()
>>>> #13 0x08173597 in tcp_init_children ()
>>>> #14 0x080b20bf in main_loop ()
>>>> #15 0x080b38c3 in main ()
>>>> (gdb)
>>>>
>>>>
>>>>
>>>> On Wed, Nov 23, 2011 at 10:35 AM, Daniel-Constantin Mierla <
>>>> miconda at gmail.com> wrote:
>>>>
>>>>> Hello,
>>>>>
>>>>> 3.1.0 is not the right choice in 3.1 series, there were many fixes
>>>>> that were added to that release series branch. The latest stable version
>>>>> there is 3.1.5. Try with it (even better, try with latest version from git
>>>>> branch 3.1 -- see
>>>>> http://www.kamailio.org/dokuwiki/doku.php/install:kamailio-3.1.x-from-git
>>>>> )
>>>>>
>>>>> In this way we are sure it is not a bug that was fixed after the time
>>>>> of releasing 3.1.0 -- config file and database is the same as for 3.1.0.
>>>>>
>>>>> Cheers,
>>>>> Daniel
>>>>>
>>>>>
>>>>> On 11/23/11 4:01 PM, Jijo wrote:
>>>>>
>>>>> Thanks I will attach the logs soon..meanwhile here is the kamailio and
>>>>> openssl version
>>>>>
>>>>> OB151:~ # /usr/sbin/kamailio -V
>>>>> version: kamailio 3.1.0 (i386/linux) 21a375
>>>>> flags: STATS: Off, USE_IPV6, USE_TCP, USE_TLS, TLS_HOOKS,
>>>>> USE_RAW_SOCKS, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC,
>>>>> DBG_QM_MALLOC, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE,
>>>>> USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES
>>>>> ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16,
>>>>> MAX_URI_SIZE 1024, BUF_SIZE 65535, PKG_SIZE 15MB
>>>>> poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
>>>>> id: 21a375
>>>>> compiled on 09:22:51 Nov 4 2011 with gcc 4.5.0
>>>>>
>>>>>
>>>>> OB151:~ # openssl version -a
>>>>> OpenSSL 1.0.0 29 Mar 2010
>>>>> built on: 2011-05-31 07:52:17.000000000 +0000
>>>>> platform: linux-elf
>>>>> options: bn(64,32) rc4(4x,int) des(ptr,risc1,16,long) blowfish(idx)
>>>>> compiler: gcc -fPIC -DOPENSSL_PIC -DZLIB -DOPENSSL_THREADS
>>>>> -D_REENTRANT -DDSO_DLFCN -DHAVE_DLFCN_H
>>>>> -DL_ENDIAN -DTERMIO -O3 -fomit-frame-pointer -Wall
>>>>> -fomit-frame-pointer -fmessage-length=0 -O2 -Wall
>>>>> -D_FORTIFY_SOURCE=2 -fstack-protector -funwind-tables
>>>>> -fasynchronous-unwind-tables -g -Wa,--noexecstack -fomit-frame-pointer
>>>>> -fno-strict-aliasing
>>>>> -DTERMIO -Wall -fstack-protector -DOPENSSL_BN_ASM_PART_WORDS
>>>>> -DOPENSSL_IA32_SSE2
>>>>> -DOPENSSL_BN_ASM_MONT -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DMD5_ASM
>>>>> -DRMD160_ASM -DAES_ASM -DWHIRLPOOL_ASM
>>>>> OPENSSLDIR: "/etc/ssl"
>>>>>
>>>>>
>>>>> On Wed, Nov 23, 2011 at 4:44 AM, Daniel-Constantin Mierla <
>>>>> miconda at gmail.com> wrote:
>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> (discussion kept only on sr-dev as it is very likely going to require
>>>>>> mostly devel interaction).
>>>>>>
>>>>>> What is the version of kamailio (-V command line option). Also, send
>>>>>> the verision of openssl library -- there were many bugs in various lib
>>>>>> versions that had to be workarounded in the module, maybe this is a new one
>>>>>> that has to be fixed.
>>>>>>
>>>>>> Do you get any error message in the syslog at the moment of the crash?
>>>>>>
>>>>>> What would be useful is to get the memory operations log, you can get
>>>>>> it by setting:
>>>>>>
>>>>>> memdbg=1
>>>>>> memlog=1
>>>>>>
>>>>>> in config file.
>>>>>>
>>>>>> Then repeat the test and make the log available for download somehow
>>>>>> (if it is too big), from start to the moment of the crash.
>>>>>>
>>>>>> Cheers,
>>>>>> Daniel
>>>>>>
>>>>>>
>>>>>> On 11/22/11 11:30 PM, Jijo wrote:
>>>>>>
>>>>>> Hi All,
>>>>>>
>>>>>> Kamailio is resetting when we do TLS renegotiation dos attack using
>>>>>> the tool available at http://www.thc.org/thc-ssl-dos/.
>>>>>>
>>>>>> Anybody looked at this issue? How we could resolve it. Any idea?
>>>>>>
>>>>>> The core generated for 3 pid's as below
>>>>>>
>>>>>> Pid 1:
>>>>>>
>>>>>> Core was generated by `/usr/sbin/kamailio -u swrun -g sw -m 120 -f
>>>>>> /etc/kamailio/kamailio.cfg'.
>>>>>> Program terminated with signal 11, Segmentation fault.
>>>>>> #0 atomic_inc_int () at atomic/atomic_x86.h:225
>>>>>> (gdb) bt
>>>>>> #0 atomic_inc_int () at atomic/atomic_x86.h:225
>>>>>> #1 cfg_update_local () at cfg/cfg_struct.h:228
>>>>>> #2 timer_main () at timer.c:994
>>>>>> #3 0x080b0579 in main_loop () at main.c:1632
>>>>>> #4 0x080b1be4 in main (argc=9, argv=0xbfd61e54) at main.c:2446
>>>>>>
>>>>>>
>>>>>> Pid 2:
>>>>>>
>>>>>> Core was generated by `/usr/sbin/kamailio -u swrun -g sw -m 120 -f
>>>>>> /etc/kamailio/kamailio.cfg'.
>>>>>> Program terminated with signal 11, Segmentation fault.
>>>>>> #0 0x0819bfe8 in qm_insert_free (qm=0xaf6c5000, p=0xb05eec30,
>>>>>> file=0xb6fb4140 "tls: tls_init.c", func=0xb6fb4ce0 "ser_free", line=296)
>>>>>> at mem/q_malloc.c:184
>>>>>> 184 if (frag->size <= f->size) break;
>>>>>> (gdb) bt
>>>>>> #0 0x0819bfe8 in qm_insert_free (qm=0xaf6c5000, p=0xb05eec30,
>>>>>> file=0xb6fb4140 "tls: tls_init.c", func=0xb6fb4ce0 "ser_free", line=296)
>>>>>> at mem/q_malloc.c:184
>>>>>> #1 qm_free (qm=0xaf6c5000, p=0xb05eec30, file=0xb6fb4140 "tls:
>>>>>> tls_init.c", func=0xb6fb4ce0 "ser_free", line=296) at mem/q_malloc.c:518
>>>>>> #2 0xb6f95404 in ser_free (ptr=0xb05eec30) at tls_init.c:296
>>>>>> #3 0xb732e9ba in CRYPTO_free (str=0xb05eec30) at mem.c:391
>>>>>> #4 0xb7330bee in int_new_ex_data (class_index=5, obj=0xbfd414f4,
>>>>>> ad=0xbfd41574) at ex_data.c:440
>>>>>> #5 0xb7330443 in CRYPTO_new_ex_data (class_index=5, obj=0xbfd414f4,
>>>>>> ad=0xbfd41574) at ex_data.c:575
>>>>>> #6 0xb73dfde3 in X509_STORE_CTX_init (ctx=0xbfd414f4,
>>>>>> store=0xafd8b3d0, x509=0xafe08ff0, chain=0x0) at x509_vfy.c:2114
>>>>>> #7 0xb74b0f31 in ssl3_output_cert_chain (s=0xb0553a10, x=0xafe08ff0)
>>>>>> at s3_both.c:349
>>>>>> #8 0xb74a4728 in ssl3_send_server_certificate (s=0xb0553a10) at
>>>>>> s3_srvr.c:3034
>>>>>> #9 0xb74a5879 in ssl3_accept (s=0xb0553a10) at s3_srvr.c:353
>>>>>> #10 0xb74afa8f in ssl3_read_bytes (s=0xb0553a10, type=23,
>>>>>> buf=0xb0ad44ec "", len=4095, peek=0) at s3_pkt.c:1266
>>>>>> #11 0xb74ac9c9 in ssl3_read_internal (s=0xb0553a10, buf=0xb0ad44ec,
>>>>>> len=4095, peek=0) at s3_lib.c:3265
>>>>>> #12 0xb74c24a9 in SSL_read (s=0xb0553a10, buf=0xb0ad44ec, num=4095)
>>>>>> at ssl_lib.c:954
>>>>>> #13 0xb6fad1c3 in tls_read_f (c=0xb0ad431c, flags=0xbfd619c4) at
>>>>>> tls_server.c:1058
>>>>>> #14 0x08171c0e in tcp_read_headers (c=0xb0ad431c,
>>>>>> read_flags=0xbfd619c4) at tcp_read.c:406
>>>>>> #15 0x08171db8 in tcp_read_req (con=0xb0ad431c,
>>>>>> bytes_read=0xbfd619cc, read_flags=0xbfd619c4) at tcp_read.c:885
>>>>>> #16 0x08172f67 in handle_io (fm=<value optimized out>, events=1,
>>>>>> idx=<value optimized out>) at tcp_read.c:1234
>>>>>> #17 0x0817583b in io_wait_loop_epoll (unix_sock=89) at io_wait.h:1092
>>>>>> #18 tcp_receive_loop (unix_sock=89) at tcp_read.c:1345
>>>>>> #19 0x0816e2e9 in tcp_init_children () at tcp_main.c:4867
>>>>>> #20 0x080affb1 in main_loop () at main.c:1646
>>>>>> #21 0x080b1be4 in main (argc=9, argv=0xbfd61e54) at main.c:2446
>>>>>>
>>>>>> Pid 3:
>>>>>>
>>>>>> Core was generated by `/usr/sbin/kamailio -u swrun -g sw -m 120 -f
>>>>>> /etc/kamailio/kamailio.cfg'.
>>>>>> Program terminated with signal 11, Segmentation fault.
>>>>>> #0 0xb76c9e7c in memmove () from /lib/libc.so.6
>>>>>> (gdb) bt
>>>>>> #0 0xb76c9e7c in memmove () from /lib/libc.so.6
>>>>>> #1 0x081724e7 in tcp_read_req (con=0xb022c8f0,
>>>>>> bytes_read=0xbfd619cc, read_flags=0xbfd619c4) at tcp_read.c:1026
>>>>>> #2 0x08172f67 in handle_io (fm=<value optimized out>, events=1,
>>>>>> idx=<value optimized out>) at tcp_read.c:1234
>>>>>> #3 0x0817583b in io_wait_loop_epoll (unix_sock=93) at io_wait.h:1092
>>>>>> #4 tcp_receive_loop (unix_sock=93) at tcp_read.c:1345
>>>>>> #5 0x0816e2e9 in tcp_init_children () at tcp_main.c:4867
>>>>>> #6 0x080affb1 in main_loop () at main.c:1646
>>>>>> #7 0x080b1be4 in main (argc=9, argv=0xbfd61e54) at main.c:2446
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> sr-dev mailing listsr-dev at lists.sip-router.orghttp://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-dev
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Daniel-Constantin Mierla -- http://www.asipto.com
>>>>>> Kamailio Advanced Training, Dec 5-8, Berlin: http://asipto.com/u/kathttp://linkedin.com/in/miconda -- http://twitter.com/miconda
>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> Daniel-Constantin Mierla -- http://www.asipto.com
>>>>> Kamailio Advanced Training, Dec 5-8, Berlin: http://asipto.com/u/kathttp://linkedin.com/in/miconda -- http://twitter.com/miconda
>>>>>
>>>>>
>>>>
>>>
>>> --
>>> Daniel-Constantin Mierla -- http://www.asipto.comhttp://linkedin.com/in/miconda -- http://twitter.com/miconda
>>>
>>>
>>
>>
>> _______________________________________________
>> sr-dev mailing listsr-dev at lists.sip-router.orghttp://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-dev
>>
>>
>> --
>> Daniel-Constantin Mierla -- http://www.asipto.comhttp://linkedin.com/in/miconda -- http://twitter.com/miconda
>>
>>
>
> --
> Daniel-Constantin Mierla -- http://www.asipto.comhttp://linkedin.com/in/miconda -- http://twitter.com/miconda
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.sip-router.org/pipermail/sr-dev/attachments/20111219/377e77db/attachment-0001.htm>
More information about the sr-dev
mailing list