[sr-dev] massive TLS crashes

[Andrei Pelinescu-Onciul]

On Feb 22, 2010 at 19:16, Klaus Darilion <klaus.mailinglists at pernau.at> wrote:
> Hi!
> 
> With kamailio 3.0 I encounter lots of crashes. I am using SNOM 320 and 
> eyebeam clients. I think crashes are mostly correlated with SNOM 320.
> 
> Server is CentOS5.4. OpenSSL is openssl-0.9.8e-12.el5_4.1.
> 
> First crashed Kamailio always with the same backtrace:
> 
> Program terminated with signal 11, Segmentation fault.
> [New process 1580]
> #0  0x0017451d in kssl_keytab_is_available () from /lib/libssl.so.6
> (gdb) bt
> #0  0x0017451d in kssl_keytab_is_available () from /lib/libssl.so.6
> 

This one looks like some openssl bug. What's strange is that it's in
some kerberos code, which is supposed to be disabled.
Could you try with another openssl version, or even better compile
openssl with debug info and link with it?


What exactly did you need to do for reproducing it with a snom (any
settings you think there might be relevant both on the phone and in the
tls handling in the script)? 

[...]

> 
> Then I configured RSA ciphers (as suggested by others) and now I get 
> different backtraces. Here is an example where Kamailio wrote 2 core files:
> 
> 1. Program terminated with signal 11, Segmentation fault.
> [New process 1735]
> #0  0x004d01d3 in free_hash_table () at h_table.c:423

This crashed on shutdown for some reason (corrupted list?).

[...]

> 
> 2. Program terminated with signal 11, Segmentation fault.
> [New process 1759]
> #0  0x004f24ce in t_reply_matching (p_msg=0x8298cf0, 
> p_branch=0xbfc70424) at t_lookup.c:983
> 983                             if (p_cell->label != entry_label)

Hmm, this looks like a corrupted transaction list. It's hard to say if
it's caused by tls or is something unrelated.
Could you print p_cell and p_cell->label in gdb?

> (gdb) bt
> #0  0x004f24ce in t_reply_matching (p_msg=0x8298cf0, 
> p_branch=0xbfc70424) at t_lookup.c:983
> #1  0x004f5559 in t_check_msg (p_msg=0x8298cf0, param_branch=0xbfc70424) 
> at t_lookup.c:1138
> #2  0x004f5e94 in t_check (p_msg=0x8298cf0, param_branch=0xbfc70424) at 
> t_lookup.c:1180
> #3  0x005140d9 in reply_received (p_msg=0x8298cf0) at t_reply.c:1897
> #4  0x0808c764 in forward_reply (msg=0x8298cf0) at forward.c:689
> #5  0x080c401e in receive_msg (
>     buf=0xb60fe088 "SIP/2.0 200 Ok\r\nVia: SIP/2.0/TLS 
> 83.136.32.167:5061;branch=z9hG4bKcc38.0a7d6be7.0;i=2\r\nVia: SIP/2.0/TLS 
> 10.10.0.51:40487;received=83.136.33.3;branch=z9hG4bK-d8754z-7b0f1727475bda32-1---d8754z-;rport=2"..., 
> len=1150, rcv_info=0xb60fded4) at receive.c:257
> #6  0x0813cf41 in tcp_read_req (con=0xb60fdec0, bytes_read=0xbfc70768, 
> read_flags=0xbfc70760) at tcp_read.c:761
> #7  0x0813da0b in handle_io (fm=0x82aa708, events=1, idx=-1) at 
> tcp_read.c:980
> #8  0x08141513 in tcp_receive_loop (unix_sock=25) at io_wait.h:1057
> #9  0x0810fc1b in tcp_init_children () at tcp_main.c:4253
> #10 0x0809ae69 in main_loop () at main.c:1525
> #11 0x0809bc02 in main (argc=1, argv=0xbfc70b74) at main.c:2251
> 
> 


Andrei