[kamailio/kamailio] tls_wolfssl: assertion (Issue #4391)
Den4t created an issue (kamailio/kamailio#4391) Hi ! Kamailio v.5.8.6, used as webrtc registrar/GW, wolfssl v.5.7.4, high call load, OS: Ubuntu 22 Periodically kamailio dumps core with assertion: ``` Program terminated with signal SIGABRT, Aborted. #0 __pthread_kill_implementation (no_tid=0, signo=6, threadid=140585212159808) at ./nptl/pthread_kill.c:44 44 ./nptl/pthread_kill.c: No such file or directory. (gdb) backtrace #0 __pthread_kill_implementation (no_tid=0, signo=6, threadid=140585212159808) at ./nptl/pthread_kill.c:44 #1 __pthread_kill_internal (signo=6, threadid=140585212159808) at ./nptl/pthread_kill.c:78 #2 __GI___pthread_kill (threadid=140585212159808, signo=signo@entry=6) at ./nptl/pthread_kill.c:89 #3 0x00007fdc8ba5f476 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26 #4 0x00007fdc8ba457f3 in __GI_abort () at ./stdlib/abort.c:79 #5 0x00007fdc8ba4571b in __assert_fail_base (fmt=0x7fdc8bbfa130 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=0x7fdc7a6b299f "nw == bytes_read", file=0x7fdc7a6b089a "tls_server.c", line=1056, function=<optimized out>) at ./assert/assert.c:94 #6 0x00007fdc8ba56e96 in __GI___assert_fail (assertion=0x7fdc7a6b299f "nw == bytes_read", file=0x7fdc7a6b089a "tls_server.c", line=1056, function=0x7fdc7a6b5068 "tls_h_read_f") at ./assert/assert.c:103 #7 0x00007fdc7a48cd89 in tls_h_read_f () from /usr/lib64/kamailio/modules/tls_wolfssl.so #8 0x00005622baa30e8e in ?? () #9 0x00005622baa3273b in tcp_read_req () #10 0x00005622baa3c7c2 in ?? () #11 0x00005622babf61d6 in ?? () #12 0x00005622baa42b87 in tcp_receive_loop () #13 0x00005622baa24188 in tcp_init_children () #14 0x00005622ba7b1b33 in main_loop () #15 0x00005622ba7a5888 in main () ``` I couldn't reproduce this issue in a lab, but I think that possibly the issue depends on TLS traffic burst behavior. We have some buggy webrtc clients, which floods using MESSAGE requests with high CPS periodically. I assume that might be this traffic somehow fills the BIO buffers in wolfssl. As result wolfSSL_BIO_write function returns with error. I have no time to debug deeply, so I made a quick fix here: https://github.com/Den4t/kamailio/commit/20579f02354356dedb5f5ac1f9bff653143... I see following logs after the fix has been applied: ``` 2025-09-02T09:24:04.393019+03:00 iptel /usr/sbin/kamailio[892554]: BUG: tls_wolfssl [tls_server.c:1058]: tls_h_read_f(): tls_h_read_f assertion check nw == bytes_read (nw=-1, bytes_read=7115, npos=-1) 2025-09-02T10:44:21.556481+03:00 iptel /usr/sbin/kamailio[892559]: BUG: tls_wolfssl [tls_server.c:1058]: tls_h_read_f(): tls_h_read_f assertion check nw == bytes_read (nw=-1, bytes_read=6201, npos=-1) 2025-09-02T10:58:31.563456+03:00 iptel /usr/sbin/kamailio[892558]: BUG: tls_wolfssl [tls_server.c:1058]: tls_h_read_f(): tls_h_read_f assertion check nw == bytes_read (nw=-1, bytes_read=7587, npos=-1) ``` But service is alive. It is better to lose one session than the whole service. In any case this is just a crutch for quick problem fix, so it would be nice if tls_wolfssl module's author took a deeper look at the problem. P.S. I assume that module's logic should check BIO flags after the call to wolfSSL_BIO_write, f.e. this function can return error with bio flag WOLFSSL_BIO_FLAG_RETRY set. -- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/4391 You are receiving this because you are subscribed to this thread. Message ID: <kamailio/kamailio/issues/4391@github.com>
github-actions[bot] left a comment (kamailio/kamailio#4391) This issue is stale because it has been open 6 weeks with no activity. Remove stale label or comment or this will be closed in 2 weeks. -- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/4391#issuecomment-3434928569 You are receiving this because you are subscribed to this thread. Message ID: <kamailio/kamailio/issues/4391/3434928569@github.com>
henningw left a comment (kamailio/kamailio#4391) Thanks for the report, not much activity so far. Maybe @space88man has seen this before. -- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/4391#issuecomment-3450439118 You are receiving this because you are subscribed to this thread. Message ID: <kamailio/kamailio/issues/4391/3450439118@github.com>
github-actions[bot] left a comment (kamailio/kamailio#4391) This issue is stale because it has been open 6 weeks with no activity. Remove stale label or comment or this will be closed in 2 weeks. -- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/4391#issuecomment-3630096766 You are receiving this because you are subscribed to this thread. Message ID: <kamailio/kamailio/issues/4391/3630096766@github.com>
Den4t left a comment (kamailio/kamailio#4391) @space88man - ping ! -- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/4391#issuecomment-3631439286 You are receiving this because you are subscribed to this thread. Message ID: <kamailio/kamailio/issues/4391/3631439286@github.com>
github-actions[bot] left a comment (kamailio/kamailio#4391) This issue is stale because it has been open 6 weeks with no activity. Remove stale label or comment or this will be closed in 2 weeks. -- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/4391#issuecomment-3776010767 You are receiving this because you are subscribed to this thread. Message ID: <kamailio/kamailio/issues/4391/3776010767@github.com>
space88man left a comment (kamailio/kamailio#4391)
[@space88man](https://github.com/space88man) - ping !
Keepalive - ping ! -- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/4391#issuecomment-3781892188 You are receiving this because you are subscribed to this thread. Message ID: <kamailio/kamailio/issues/4391/3781892188@github.com>
Den4t left a comment (kamailio/kamailio#4391) Hi, @space88man ! About this issue, are this possible to replace assertion with error processing as in: https://github.com/Den4t/kamailio/commit/20579f02354356dedb5f5ac1f9bff653143... it is not good to loose whole service when only one session fails. This modification work in our highly loaded installations with no problems 5 months. -- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/4391#issuecomment-3790025958 You are receiving this because you are subscribed to this thread. Message ID: <kamailio/kamailio/issues/4391/3790025958@github.com>
miconda left a comment (kamailio/kamailio#4391) @Den4t: I pushed a commit to replace all assert() uses, it is not common practice in Kamailio to stop/crash on purpose on library/memory/etc... errors, but try to go on, by reporting error. Can you test and report if any obvious side effects? If all ok, it will be backported. -- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/4391#issuecomment-3895444912 You are receiving this because you are subscribed to this thread. Message ID: <kamailio/kamailio/issues/4391/3895444912@github.com>
Den4t left a comment (kamailio/kamailio#4391)
Can you test and report if any obvious side effects? If all ok, it will be backported.
Hi, Daniel ! Thanks, I'll give it a try this week, and report back here. -- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/4391#issuecomment-3906705004 You are receiving this because you are subscribed to this thread. Message ID: <kamailio/kamailio/issues/4391/3906705004@github.com>
Den4t left a comment (kamailio/kamailio#4391) I applied this patch to one of the nodes under high pressure, need to wait a while, this problem does not occur immediately. -- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/4391#issuecomment-3951573772 You are receiving this because you are subscribed to this thread. Message ID: <kamailio/kamailio/issues/4391/3951573772@github.com>
Den4t left a comment (kamailio/kamailio#4391) Hi ! No asserstion was found during 2 weeks, but not found specific logs also :( -- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/4391#issuecomment-4030907448 You are receiving this because you are subscribed to this thread. Message ID: <kamailio/kamailio/issues/4391/4030907448@github.com>
space88man left a comment (kamailio/kamailio#4391) Closing for now - if it reproduces open a new ticket. There has been recent work on master to bring the module up to date with wolfSSL(v5.8.4-stable) and MT-mode. -- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/4391#issuecomment-4081236784 You are receiving this because you are subscribed to this thread. Message ID: <kamailio/kamailio/issues/4391/4081236784@github.com>
Closed #4391 as completed. -- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/4391#event-23675758120 You are receiving this because you are subscribed to this thread. Message ID: <kamailio/kamailio/issue/4391/issue_event/23675758120@github.com>
participants (5)
-
Daniel-Constantin Mierla -
Dennis -
github-actions[bot] -
Henning Westerholt -
space88man