[sr-dev] [kamailio/kamailio] Database TLS connection through cnf file: TLS error followed by memory corruption messages (Issue #3115)
liviu-rolo
notifications at github.com
Wed May 18 11:51:28 CEST 2022
<!--
Kamailio Project uses GitHub Issues only for bugs in the code or feature requests. Please use this template only for bug reports.
If you have questions about using Kamailio or related to its configuration file, ask on sr-users mailing list:
* http://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
If you have questions about developing extensions to Kamailio or its existing C code, ask on sr-dev mailing list:
* http://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev
Please try to fill this template as much as possible for any issue. It helps the developers to troubleshoot the issue.
If there is no content to be filled in a section, the entire section can be removed.
You can delete the comments from the template sections when filling.
You can delete next line and everything above before submitting (it is a comment).
-->
### Description
Using kamailio 5.5.4 from the official Docker image (created like `create_dockerfile.sh bullseye 5.5.4`, but with no ENTRYPOINT).
Doing TLS listening like:
```
socket_workers=2
tls_max_connections=20000
listen=tls:192.168.0.11:5061
enable_tls=yes
dns_tls_pref=50
tls_port_no=5061
loadmodule "tls.so"
```
It is the second loaded module after tcpops.
Have to connect to the database through cnf files for the SSL support:
`modparam("sqlops","sqlcon","euler=>mysql://[euler]/euclid")`
Relevant section in the cnf file:
```
[euler]
port = 3306
host = 127.0.0.1
protocol = tcp
user = gauss
password = fermat
default-character-set = utf8
ssl-ca = /SSL/riemann/ca-cert.pem
ssl-cert = /SSL/riemann/client-cert.pem
ssl-key = /SSL/riemann/client-key.pem
#ssl-verify-server-cert
```
When starting kamailio:
```
8(18) ERROR: db_mysql [km_my_con.c:163]: db_mysql_new_connection(): driver error: SSL connection error: PEM lib
8(18) ERROR: <core> [db.c:319]: db_do_init2(): could not add connection to the pool
8(18) ERROR: sqlops [sql_api.c:165]: sql_connect(): failed to connect to the database [euclid]
8(18) ERROR: <core> [core/sr_module.c:874]: init_mod_child(): error while initializing module sqlops (/home/euler/lib64/euler/modules/sqlops.so) (idx: 8 rank: -2 desc: [Async Task Worker - default])
8(18) ERROR: <core> [core/pt.c:338]: fork_process(): init_child failed for process 8, pid 18, "Async Task Worker - default"
8(18) ERROR: <core> [main.c:1850]: main_loop(): error in init_child
12(22) CRITICAL: <core> [core/mem/q_malloc.c:501]: qm_free(): BUG: bad pointer 0x7fafa23e6270 (out of memory block!) called from tls: tls_init.c: ser_free(323) - ignoring
```
Then kamailio just hangs on in an incomplete state:
```
/ # ps auxww --forest
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 29 0.6 0.0 4092 3244 pts/1 Ss 08:59 0:00 bash
root 36 0.0 0.0 6692 2964 pts/1 R+ 08:59 0:00 \_ ps auxww --forest
root 1 0.0 0.0 4092 3252 pts/0 Ss 08:58 0:00 bash
root 8 0.4 0.0 110868 23764 pts/0 S+ 08:58 0:00 kamailio -A NO_JSON_LOG -f /home/euler/etc/euler.cfg
root 9 0.0 0.0 0 0 ? Zs 08:58 0:00 \_ [kamailio] <defunct>
root 10 0.5 0.0 111080 31148 ? S 08:58 0:00 kamailio -A NO_JSON_LOG -f /home/euler/etc/euler.cfg
root 11 0.0 0.0 111080 13412 ? S 08:58 0:00 \_ kamailio -A NO_JSON_LOG -f /home/euler/etc/euler.cfg
root 12 99.9 0.0 111080 13240 ? R 08:58 0:43 \_ kamailio -A NO_JSON_LOG -f /home/euler/etc/euler.cfg
root 13 0.0 0.0 111080 14252 ? S 08:58 0:00 \_ kamailio -A NO_JSON_LOG -f /home/euler/etc/euler.cfg
root 14 0.0 0.0 111080 13432 ? S 08:58 0:00 \_ kamailio -A NO_JSON_LOG -f /home/euler/etc/euler.cfg
root 15 0.0 0.0 111080 13296 ? S 08:58 0:00 \_ kamailio -A NO_JSON_LOG -f /home/euler/etc/euler.cfg
root 16 0.0 0.0 111080 13348 ? S 08:58 0:00 \_ kamailio -A NO_JSON_LOG -f /home/euler/etc/euler.cfg
root 17 0.0 0.0 111080 13220 ? S 08:58 0:00 \_ kamailio -A NO_JSON_LOG -f /home/euler/etc/euler.cfg
root 18 0.0 0.0 111072 14196 ? S 08:58 0:00 \_ kamailio -A NO_JSON_LOG -f /home/euler/etc/euler.cfg
root 19 99.7 0.0 111080 10748 ? R 08:58 0:43 \_ kamailio -A NO_JSON_LOG -f /home/euler/etc/euler.cfg
root 20 0.2 0.0 0 0 ? Z 08:58 0:00 \_ [kamailio] <defunct>
root 21 0.0 0.0 111080 13068 ? S 08:58 0:00 \_ kamailio -A NO_JSON_LOG -f /home/euler/etc/euler.cfg
root 22 0.0 0.0 111080 13156 ? S 08:58 0:00 \_ kamailio -A NO_JSON_LOG -f /home/euler/etc/euler.cfg
root 23 0.0 0.0 111080 10684 ? S 08:58 0:00 \_ kamailio -A NO_JSON_LOG -f /home/euler/etc/euler.cfg
root 24 0.0 0.0 111080 10684 ? S 08:58 0:00 \_ kamailio -A NO_JSON_LOG -f /home/euler/etc/euler.cfg
root 25 0.0 0.0 111080 10748 ? S 08:58 0:00 \_ kamailio -A NO_JSON_LOG -f /home/euler/etc/euler.cfg
root 26 0.0 0.0 111080 10748 ? S 08:58 0:00 \_ kamailio -A NO_JSON_LOG -f /home/euler/etc/euler.cfg
root 27 0.0 0.0 111080 13344 ? S 08:58 0:00 \_ kamailio -A NO_JSON_LOG -f /home/euler/etc/euler.cfg
root 28 0.0 0.0 111080 10748 ? S 08:58 0:00 \_ kamailio -A NO_JSON_LOG -f /home/euler/etc/euler.cfg
```
<!--
Explain what you did, what you expected to happen, and what actually happened.
-->
### Troubleshooting
#### Reproduction
<!--
If the issue can be reproduced, describe how it can be done.
-->
It is always happening, but not when either `listen=tls` or the SSL options in the cnf file are dropped.
#### Debugging Data
<!--
If you got a core dump, use gdb to extract troubleshooting data - full backtrace,
local variables and the list of the code at the issue location.
gdb /path/to/kamailio /path/to/corefile
bt full
info locals
list
If you are familiar with gdb, feel free to attach more of what you consider to
be relevant.
-->
```
/ # ps auxww --forest
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 29 0.6 0.0 4092 3244 pts/1 Ss 08:59 0:00 bash
root 36 0.0 0.0 6692 2964 pts/1 R+ 08:59 0:00 \_ ps auxww --forest
root 1 0.0 0.0 4092 3252 pts/0 Ss 08:58 0:00 bash
root 8 0.4 0.0 110868 23764 pts/0 S+ 08:58 0:00 kamailio -A NO_JSON_LOG -f /home/euler/etc/euler.cfg
root 9 0.0 0.0 0 0 ? Zs 08:58 0:00 \_ [kamailio] <defunct>
root 10 0.5 0.0 111080 31148 ? S 08:58 0:00 kamailio -A NO_JSON_LOG -f /home/euler/etc/euler.cfg
root 11 0.0 0.0 111080 13412 ? S 08:58 0:00 \_ kamailio -A NO_JSON_LOG -f /home/euler/etc/euler.cfg
root 12 99.9 0.0 111080 13240 ? R 08:58 0:43 \_ kamailio -A NO_JSON_LOG -f /home/euler/etc/euler.cfg
root 13 0.0 0.0 111080 14252 ? S 08:58 0:00 \_ kamailio -A NO_JSON_LOG -f /home/euler/etc/euler.cfg
root 14 0.0 0.0 111080 13432 ? S 08:58 0:00 \_ kamailio -A NO_JSON_LOG -f /home/euler/etc/euler.cfg
root 15 0.0 0.0 111080 13296 ? S 08:58 0:00 \_ kamailio -A NO_JSON_LOG -f /home/euler/etc/euler.cfg
root 16 0.0 0.0 111080 13348 ? S 08:58 0:00 \_ kamailio -A NO_JSON_LOG -f /home/euler/etc/euler.cfg
root 17 0.0 0.0 111080 13220 ? S 08:58 0:00 \_ kamailio -A NO_JSON_LOG -f /home/euler/etc/euler.cfg
root 18 0.0 0.0 111072 14196 ? S 08:58 0:00 \_ kamailio -A NO_JSON_LOG -f /home/euler/etc/euler.cfg
root 19 99.7 0.0 111080 10748 ? R 08:58 0:43 \_ kamailio -A NO_JSON_LOG -f /home/euler/etc/euler.cfg
root 20 0.2 0.0 0 0 ? Z 08:58 0:00 \_ [kamailio] <defunct>
root 21 0.0 0.0 111080 13068 ? S 08:58 0:00 \_ kamailio -A NO_JSON_LOG -f /home/euler/etc/euler.cfg
root 22 0.0 0.0 111080 13156 ? S 08:58 0:00 \_ kamailio -A NO_JSON_LOG -f /home/euler/etc/euler.cfg
root 23 0.0 0.0 111080 10684 ? S 08:58 0:00 \_ kamailio -A NO_JSON_LOG -f /home/euler/etc/euler.cfg
root 24 0.0 0.0 111080 10684 ? S 08:58 0:00 \_ kamailio -A NO_JSON_LOG -f /home/euler/etc/euler.cfg
root 25 0.0 0.0 111080 10748 ? S 08:58 0:00 \_ kamailio -A NO_JSON_LOG -f /home/euler/etc/euler.cfg
root 26 0.0 0.0 111080 10748 ? S 08:58 0:00 \_ kamailio -A NO_JSON_LOG -f /home/euler/etc/euler.cfg
root 27 0.0 0.0 111080 13344 ? S 08:58 0:00 \_ kamailio -A NO_JSON_LOG -f /home/euler/etc/euler.cfg
root 28 0.0 0.0 111080 10748 ? S 08:58 0:00 \_ kamailio -A NO_JSON_LOG -f /home/euler/etc/euler.cfg
```
#### Log Messages
<!--
Check the syslog file and if there are relevant log messages printed by Kamailio, add them next, or attach to issue, or provide a link to download them (e.g., to a pastebin site).
-->
```
8(18) ERROR: db_mysql [km_my_con.c:163]: db_mysql_new_connection(): driver error: SSL connection error: PEM lib
8(18) ERROR: <core> [db.c:319]: db_do_init2(): could not add connection to the pool
8(18) ERROR: sqlops [sql_api.c:165]: sql_connect(): failed to connect to the database [euclid]
8(18) ERROR: <core> [core/sr_module.c:874]: init_mod_child(): error while initializing module sqlops (/home/euler/lib64/euler/modules/sqlops.so) (idx: 8 rank: -2 desc: [Async Task Worker - default])
8(18) ERROR: <core> [core/pt.c:338]: fork_process(): init_child failed for process 8, pid 18, "Async Task Worker - default"
8(18) ERROR: <core> [main.c:1850]: main_loop(): error in init_child
12(22) CRITICAL: <core> [core/mem/q_malloc.c:501]: qm_free(): BUG: bad pointer 0x7fafa23e6270 (out of memory block!) called from tls: tls_init.c: ser_free(323) - ignoring
```
#### SIP Traffic
<!--
If the issue is exposed by processing specific SIP messages, grab them with ngrep or save in a pcap file, then add them next, or attach to issue, or provide a link to download them (e.g., to a pastebin site).
-->
```
(paste your sip traffic here)
```
### Possible Solutions
<!--
If you found a solution or workaround for the issue, describe it. Ideally, provide a pull request with a fix.
-->
### Additional Information
* **Kamailio Version** - output of `kamailio -v`
```
version: kamailio 5.5.4 (x86_64/linux)
flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLOCKLIST, HAVE_RESOLV_RES, TLS_PTHREAD_MUTEX_SHARED
ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
id: unknown
compiled with gcc 10.2.1
```
* **Operating System**:
<!--
Details about the operating system, the type: Linux (e.g.,: Debian 8.4, Ubuntu 16.04, CentOS 7.1, ...), MacOS, xBSD, Solaris, ...;
Kernel details (output of `lsb_release -a` and `uname -a`)
-->
```
Linux euler01 5.4.44-2-pve #1 SMP PVE 5.4.44-2 (Wed, 01 Jul 2020 16:37:57 +0200) x86_64 GNU/Linux
```
--
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/3115
You are receiving this because you are subscribed to this thread.
Message ID: <kamailio/kamailio/issues/3115 at github.com>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.kamailio.org/pipermail/sr-dev/attachments/20220518/c71fd71b/attachment-0001.htm>
More information about the sr-dev
mailing list