17 May
2019
17 May
'19
5:03 p.m.
Hi!
I haven't used the workaround yet: I'm focusing on trying to make sure I
have the same issue
or trying to figure out how to force it to happen.
I have started to check again the server today and I started by this
command:
$> sudo kamcmd tls.list
In my previous description, the above was a dead lock. Today, It finally
completed, but
after 5 minutes. (I suspect 5 minutes is abnormal)
During the long running command:
-> UDP was working
-> TCP was not:
-> The TCP connection is being ESTABLISHED, but the SIP message was not
replied.
(this was the behavior I had before)
At the same time, I took a trap "sudo kamctl trap". (during the dead lock)
-> one thread is on "tls_list" (tls_rpc.c:154)
-> one thread is on tcpconn_get (core/tcp_main.c:1449) called from tcp_send
(core/tcp_main.c:1716)
and seems to be sending a 484 Address Incomplete on a TLS connection
-> 2 threads are on CRYPTO_THREAD_write_lock on a backtrace showing
"SSL_do_handshake/tls_accept"
Suddenly, "sudo kamcmd tls.list" completed, and then, my TCP Agent received
4 answers from kamailio for the last 4 REGISTER sent.
I have a network capture for my TCP agent.
I have a trap showing 2 thread waiting on "CRYPTO_THREAD_write_lock"
Conclusion:
The use-case showed that the lock was VERY long.
The use-case showed that the lock was TEMPORARY...
Side-note: From my understanding of the multi-fork/openssl issue, I would
expect
to see dead lock happening very fast after a kamailio restart?
Do you expect the preload workaround to work in such behavior?
Or do you consider that my issue is different?
Because there is no "real" dead-lock, I don't understand why "my"
issue
would be related to libssl1.1...
My gdb trap, network capture are available in private exchange if you need!
(please ask me by direct email)
Tks
Aymeric
Le lun. 13 mai 2019 à 12:48, Daniel-Constantin Mierla <miconda(a)gmail.com> a
écrit :
Hello,
thanks for the feedback! It is good to know that it works well so far for
you. I don't see any reason not to make the library to preload as part of
the next release.
Just to let everyone know, for now, the built packages are pinned to link
against libssl 1.0.x.
Soon, I will approach the openssl project in order to find a proper
solution for long term.
Cheers,
Daniel
On 13.05.19 10:48, Floimair Florian wrote:
Hi all!
We have used the work-around with the pre-loaded library and so far this
seems to have fixed our problem (that my colleague Kristijan Vrban
reported).
At least we did not have a single failure within the last week, whereas
before the issue happened about once every 2 days.
Would be nice if this would be part of the next Kamailio version.
With best regards
*Florian Floimair *Innovation - Software-Development
*COMMEND INTERNATIONAL GMBH *A-5020 Salzburg, Saalachstraße 51
http://www.commend.com
*Security and Communication by Commend *FN 178618z | LG Salzburg
*Von: *sr-users <sr-users-bounces(a)lists.kamailio.org>
<sr-users-bounces(a)lists.kamailio.org> im Auftrag von Daniel-Constantin
Mierla <miconda(a)gmail.com> <miconda(a)gmail.com>
*Antworten an: *"miconda(a)gmail.com" <miconda(a)gmail.com>
<miconda(a)gmail.com> <miconda(a)gmail.com>om>, "Kamailio (SER) - Users
Mailing
List" <sr-users(a)lists.kamailio.org> <sr-users(a)lists.kamailio.org>
*Datum: *Montag, 15. April 2019 um 09:07
*An: *Aymeric Moizard <amoizard(a)gmail.com> <amoizard(a)gmail.com>om>,
"Kamailio (SER) - Users Mailing List" <sr-users(a)lists.kamailio.org>
<sr-users(a)lists.kamailio.org>
*Betreff: *Re: [SR-Users] Kamailio stop to process incoming SIP traffic
via TCP.
Hello Aymeric,
would you be able to test with tls module compiled against libssl 1.1 and
using the pre-loaded shared object workaround?
*
https://github.com/kamailio/kamailio/tree/master/src/modules/tls/utils/open…
<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fkamailio%2Fkamailio%2Ftree%2Fmaster%2Fsrc%2Fmodules%2Ftls%2Futils%2Fopenssl_mutex_shared&data=02%7C01%7Cf.floimair%40commend.com%7C4008d49af1b347abe20308d6c1710532%7C13b1ddb756454e7fbe663171548559da%7C0%7C0%7C636909088526480174&sdata=d9E%2Fy4cvdLkGCPUexoCJ7tws3QL4rFqz5ebnMGnsESQ%3D&reserved=0>
You should be able to use it with any version, no need to test with
kamailio master branch.
Just clone the master branch, then:
cd src/modules/tls/utils/openssl_mutex_shared
make
Either from there or copy openssl_mutex_shared.so to a location you want,
then pre-load it before starting your version of Kamailio.
The README.md in the folder has some more details.
I would like to have some validation that it works fine before approaching
this topic with libssl project to allow to init the locks with shared
process option.
Thanks,
Daniel
On 26.03.19 16:18, Daniel-Constantin Mierla wrote:
Hello,
yep, locking there is expected, as listing the tls connections wait for no
other processes to change the content of internal tls connection
structures. So it is a side effect of libssl/libcrypto getting stuck and
the other processing waiting for it to move one. I have the Kamailio
training in USA these days, so the trip and schedule of the day didn't
allow me to look more at the libsll/libcrypto code in order to find a
solution here. It is a high priority in my list, as I get time during the
next days.
Cheers,
Daniel
On 26.03.19 15:55, Aymeric Moizard wrote:
Hi All,
I was debugging a TCP issue (most probably, I may start a thread for this
question).
I was trying to get some info for TCP and TLS.
I typed:
$> sudo kamctl rpc tls.list
And waited for a while.... until... I realized that my User-Agent,
connected with TCP was not able to register any more. I think the rpc
command has introduced something wrong.
The device can successfully "connect", send the REGISTER over the
established TCP connection. The REGISTER do not appear in the logs any
more, I don't see any traffic for TCP any more. So the behavior is the same
as I had before: TCP and TLS are both not working and UDP is still working
fine.
kamctl do not work any more... so kamctl trap do not work...
I have been able to type.. manually... for (all?) kamailio threads:
gdb /usr/sbin/kamailio 16500 -batch --eval-command="bt full" >>
kamailio-trap-tcp-down.txt
I'm temporarly puting the backtrace I have here:
https://sip.antisip.com/kamailio-trap-tcp-down.txt
<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fsip.antisip.com%2Fkamailio-trap-tcp-down.txt&data=02%7C01%7Cf.floimair%40commend.com%7C4008d49af1b347abe20308d6c1710532%7C13b1ddb756454e7fbe663171548559da%7C0%7C0%7C636909088526490178&sdata=1lfFxvR0m4PVcfnYsrrIO%2FM2nbGK6zfpl2C01O2c7M0%3D&reserved=0>
You can see a thread stuck on the json command line: "tls_list"
And many other waiting on CRYPTO_THREAD_write_lock
? might be related to: https://github.com/openssl/openssl/issues/5376
<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fopenssl%2Fopenssl%2Fissues%2F5376&data=02%7C01%7Cf.floimair%40commend.com%7C4008d49af1b347abe20308d6c1710532%7C13b1ddb756454e7fbe663171548559da%7C0%7C0%7C636909088526490178&sdata=D5Fb4U3trdbRUY7ifMLSc5KE4mAxjK2%2BzOy8nSD1Rks%3D&reserved=0>
SIDE NOTE:
Right before I was typing the last gdb command for the last thread,
kamailio
has crashed: This was around 5 minutes after the dead lock started.
Mar 26 14:47:11 sip kamailio[16493]: ERROR: <core> [core/tcp_main.c:2561]:
tcpconn_do_send(): failed to send on 0x7ff8dfc2fdc8 (91.121.30.149:5061->
62.210.97.21:49351
<https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2F62.210.97.21%3A49351&data=02%7C01%7Cf.floimair%40commend.com%7C4008d49af1b347abe20308d6c1710532%7C13b1ddb756454e7fbe663171548559da%7C0%7C0%7C636909088526500195&sdata=9XqEUKoMwNEvCPFtKfvB0c43yk1GcSzYOiPdY9Pj1uo%3D&reserved=0>):
Broken pipe (32)
Mar 26 14:47:11 sip kamailio[16493]: ERROR: <core> [core/tcp_read.c:1505]:
tcp_read_req(): ERROR: tcp_read_req: error reading - c: 0x7ff8dfc2fdc8 r:
0x7ff8dfc2fe48 (-1)
Mar 26 14:47:11 sip kamailio[16493]: WARNING: <core>
[core/tcp_read.c:1848]: handle_io(): F_TCPCONN connection marked as bad:
0x7ff8dfa6a408 id 846 refcnt 3
Mar 26 14:47:11 sip kamailio[16371]: ALERT: <core> [main.c:755]:
handle_sigs(): child process 16374 exited by a signal 11
Mar 26 14:47:11 sip kamailio[16371]: ALERT: <core> [main.c:758]:
handle_sigs(): core was not generated
Mar 26 14:47:11 sip kamailio[16371]: INFO: <core> [main.c:781]:
handle_sigs(): terminating due to SIGCHLD
Mar 26 14:47:11 sip kamailio[16493]: INFO: <core> [main.c:836]: sig_usr():
signal 15 received
Mar 26 14:47:11 sip kamailio[16500]: INFO: <core> [main.c:836]: sig_usr():
signal 15 received
Mar 26 14:47:11 sip kamailio[16479]: INFO: <core> [main.c:836]: sig_usr():
signal 15 received
Unfortunalty, even if I did my best to setup my service to generate a core
on crash, I still have "core was not generated".... (debian stretch)
Tks for reading!
Regards
Aymeric
--
Antisip - http://www.antisip.com