Description

Kamailio frequently terminates tls connections to sip clients, if db_mongodb database driver is used and one mongodb cluster member becomes unavailable.

If one mongodb secondary is shutdown (or the primary and a promotion takes place) kamailio will terminate the tls connection to a sip-client (sends a FIN/ACK) after a short period of time (differs 20s to 180s). If a sip client reconnects, registration will succeed (so mongodb access is fine) but kamailio will terminate the tls connection shortly after.
This situation happens as well if only the connection between kamailio and a secondary is blocked (the cluster members can still communicate with each other).

I assume that the mongo-c's server discovery might be the culprit. Mongoc iteratively tries to connect to all servers in the mongoc-uri to update their status (mongodb://mongodb-cluster2:27017,mongodb-cluster1:27017,mongodb-cluster0:27017/kamailio?authMechanism=MONGODB-X509&replicaSet=rs1&ssl=true&sslcertificateauthorityfile=mongo.ca&sslclientcertificatekeyfile=mongo.certkey).
Nonethless, mongoc's server discovery should be transparent to the kamailio client as long as a primary exists (and this is the case). No reason to terminate tls client connections.

Troubleshooting

Reproduction

  1. Setup a mongodb cluster with 3 members.
  2. Setup kamailio with tls using db-mongodb via tls.
    Note: The mongodb-uri must contain all three members, e.g. mongodb://mongodb-cluster2:27017,mongodb-cluster1:27017,mongodb-cluster0:27017/kamailio?ssl=true)
  3. Connect/register a sip client via tls.
  4. Stop one member.
  5. Clients will receive a ready error on its tls connection.

Log Messages

The db_mongodb kamailio logs don't reveal any errors. Operations are normal. The only thing which is shown in the logs is TLS read:error. As seen below.

Jul 23 14:22:09 bda3e8fce481 /usr/local/sbin/kamailio[318]: ERROR: tls [tls_util.h:42]: tls_err_ret(): TLS read:error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
Jul 23 14:22:09 bda3e8fce481 /usr/local/sbin/kamailio[318]: ERROR: <core> [core/tcp_read.c:1485]: tcp_read_req(): ERROR: tcp_read_req: error reading - c: 0x7fa9b6343210 r: 0x7fa9b6343290

A wireshark trace reveals that kamailio closes the tcp connection (FIN/ACK). The client receives a "stream truncated".

Additional Information

version: kamailio 5.1.4 (x86_64/linux) 
flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
id: unknown 
compiled on 13:39:15 Jul 23 2018 with gcc 6.3.0


mongoc-1.11.0 (built with cmake ../ -DENABLE_AUTOMATIC_INIT_AND_CLEANUP=OFF)

Debian 9.5 docker image running on CentOS 7 (kernel 3.10.0-514.6.1.el7.x86_64)

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.