It is the shared memory that runs out of space. As I can see it, you provided the stats for pkg. You have to take the ones for shm.

What are the values you give for `-m` and `-M` command line parameters for kamailio?

Another occurence:

``` 11(8342) ERROR: usrloc [ucontact.c:100]: new_ucontact(): no more shm memory 11(8342) ERROR: usrloc [urecord.c:149]: mem_insert_ucontact(): failed to create new contact 11(8342) ERROR: usrloc [urecord.c:573]: insert_ucontact(): failed to insert contact 11(8342) ERROR: registrar [save.c:522]: insert_contacts(): failed to insert contact 11(8342) ERROR: sl [sl_funcs.c:362]: sl_reply_error(): stateless error reply used: I'm terribly sorry, server error occurred (1/SL) 31(8362) ERROR: <core> [core/tcp_main.c:973]: tcpconn_new(): mem. allocation failure 31(8362) ERROR: <core> [core/tcp_main.c:4042]: handle_new_connect(): tcpconn_new failed, closing socket 31(8362) ERROR: <core> [core/tcp_main.c:973]: tcpconn_new(): mem. allocation failure 31(8362) ERROR: <core> [core/tcp_main.c:4042]: handle_new_connect(): tcpconn_new failed, closing socket 31(8362) ERROR: <core> [core/tcp_main.c:973]: tcpconn_new(): mem. allocation failure ```

``` kamcmd mod.stats all shm Module: core { dns_cache_mk_bad_entry(881): 7864 dns_cache_mk_rd_entry(1230): 13800 msg_lump_cloner(986): 501728 sip_msg_shm_clone(496): 2862448 tcpconn_new(971): 36272 build_req_buf_from_sip_req(2182): 814696 counters_prefork_init(207): 39168 create_avp(175): 57912 cfg_clone_str(130): 256 cfg_shmize(217): 1088 init_pt(113): 8 init_pt(108): 8 init_pt(107): 4760 cfg_parse_str(906): 192 register_timer(1012): 560 cfg_register_ctx(47): 96 init_tcp(4700): 8192 init_tcp(4694): 32768 init_tcp(4686): 8 init_tcp(4679): 8 init_tcp(4672): 8 init_tcp(4666): 8 init_tcp(4654): 8 init_avps(90): 8 init_avps(89): 8 init_dst_blacklist(437): 16384 init_dst_blacklist(430): 8 timer_alloc(515): 96 init_dns_cache(369): 8 init_dns_cache(361): 16384 init_dns_cache(354): 16 init_dns_cache(345): 8 init_timer(284): 8 init_timer(283): 16384 init_timer(282): 8 init_timer(281): 8 init_timer(270): 8 init_timer(238): 8 init_timer(221): 278544 init_timer(220): 8 init_timer(207): 8 cfg_child_cb_new(828): 64 sr_cfg_init(360): 8 sr_cfg_init(353): 8 sr_cfg_init(346): 8 sr_cfg_init(334): 8 sr_cfg_init(322): 8 qm_shm_lock_init(1195): 8 Total: 4709872 }

Module: ndb_redis { Total: 0 }

Module: pv { Total: 0 }

Module: sqlops { Total: 0 }

Module: textops { Total: 0 }

Module: xhttp_rpc { Total: 0 }

Module: db_mysql { Total: 0 }

Module: auth { Total: 0 }

Module: htable { ht_init_tables(386): 24576 ht_add_table(281): 936 Total: 25512 }

Module: xlog { Total: 0 }

Module: cnxcc { __alloc_new_credit_data(1217): 1080 __alloc_new_credit_data(1207): 480 __get_or_create_credit_data_entry(1177): 288 __add_call_by_cid(1534): 4464 __alloc_new_call_by_money(1278): 16512 __alloc_new_credit_data(1233): 48 shm_str_dup(723): 10728 __shm_str_hash_alloc(1090): 21984 __mod_init(286): 16 __mod_init(276): 16 __mod_init(275): 16 __mod_init(273): 16 __mod_init(272): 16 __mod_init(270): 16 __mod_init(269): 16 Total: 55696 }

Module: rtpengine { shm_str_dup(723): 3336 rtpp_function_call(2281): 3592 rtpengine_hash_table_init(85): 1024 rtpengine_hash_table_init(76): 2048 rtpengine_hash_table_init(67): 2048 get_rtpp_set(746): 200 add_rtpengine_socks(841): 88 get_rtpp_set(755): 8 rtpengine_hash_table_init(58): 72 add_rtpengine_socks(860): 24 mod_init(1448): 72 mod_init(1441): 24 mod_init(1428): 88 mod_init(1421): 8 rtpengine_hash_table_init(111): 14400 rtpengine_hash_table_init(96): 2048 Total: 29080 }

Module: mangler { Total: 0 }

Module: acc { Total: 0 }

Module: rr { Total: 0 }

Module: dialog { build_new_dlg(456): 151088 dlg_set_leg_info(541): 24712 dlg_get_iuid_shm_clone(287): 8328 dlg_set_toroute(1157): 8128 dlg_set_leg_info(532): 8136 dlg_set_leg_info(524): 36240 init_dlg_callback(55): 16 register_dlgcb(124): 40 init_dlg_table(295): 131088 init_dlg_table(287): 8 init_dlg_table(280): 8 init_dlg_table(275): 8 init_dlg_timer(57): 8 init_dlg_timer(48): 32 new_dlg_profile(188): 312 Total: 368152 }

Module: uac { reg_ht_add(530): 41784 reg_ht_add_byuser(479): 1760 reg_ht_add_byuuid(448): 2016 uac_reg_update(1093): 11472688 uac_reg_init_ht(237): 512 uac_reg_init_ht(223): 24 uac_reg_init_ht(211): 512 uac_reg_init_ht(199): 24 uac_reg_init_ht(187): 8 Total: 11519328 }

Module: tm { str_duplicate(196): 312 new_dlg_uac(324): 248 _reply_light(490): 72496 cancel_reason_pack(1201): 5416 update_totag_set(311): 4904 update_totag_set(310): 4136 build_local_reparse(345): 253264 relay_reply(1920): 323544 build_cell(339): 1800657088 build_uac_req(1592): 190580248 init_tm_stats_child(58): 8704 insert_tmcb(137): 15281328 init_tmcb_lists(74): 16 init_tmcb_lists(72): 16 init_hash_table(481): 2097152 Total: 2009288872 }

Module: tls { fix_domain(987): 816 tls_new_domain(157): 576 tls_new_cfg(1323): 48 tls_ct_wq_init(49): 8 mod_init(322): 8 mod_init(306): 8 ser_realloc(299): 194536 ser_malloc(293): 1404184 lock_set_alloc(70): 24 Total: 1600208 }

Module: sl { init_sl_stats_child(125): 6528 sl_startup(83): 8 init_sl_stats(110): 8 Total: 6544 }

Module: tmx { tmx_init_pretran_table(90): 512 Total: 512 }

Module: kex { pkg_proc_stats_init(79): 1632 Total: 1632 }

Module: usrloc { build_stat_name(52): 64 new_udomain(94): 49152 new_udomain(87): 48 new_dlist(563): 16 new_dlist(555): 32 init_ulcb_list(45): 16 Total: 49328 }

Module: pike { pike_init(125): 16 lock_set_alloc(70): 1040 init_ip_tree(115): 4112 pike_init(107): 8 Total: 5176 }

Module: websocket { wsconn_init(109): 16 wsconn_init(100): 8192 wsconn_init(89): 8 wsconn_init(79): 8 Total: 8224 }

Module: cfgutils { mod_init(863): 8 mod_init(857): 8 mod_init(849): 8 Total: 24 }

Module: nathelper { mod_init(483): 8 Total: 8 }

Module: dispatcher { reindex_dests(629): 1600 ds_avl_insert(3064): 2568 ds_set_attrs(257): 48 pack_dest(355): 216 init_data(226): 16 init_data(218): 16 ds_parse_reply_codes(987): 816 mod_init(308): 8 mod_init(306): 8 ds_ping_active_init(118): 8 Total: 5304 } ```

We also have a test system with the issue reproduced if a closer look is needed @miconda

With busy traveling during March, this issue got out of my sight. It looks like a problem with TM UAC, the allocated size there is big. Could be related to uac_reg (uac remote registrations), I will dig into it a bit.

@miconda the number of records in uacreg is significantly less than that, that makes me wonder if the hash table cleanup is to blame

``` Apr 16 00:05:37 ip-10-0-0-0 kamailio[12057]: ERROR: uac [uac_reg.c:913]: uac_reg_tm_callback(): authentication failed for <user1234> Apr 16 00:05:37 ip-10-0-0-0 kamailio[12040]: ERROR: uac [uac_reg.c:1032]: uac_reg_tm_callback(): got sip response 500 while registering [test] Apr 16 01:39:03 ip-10-0-0-0 kamailio[12054]: ERROR: tls [tls_server.c:192]: tls_complete_init(): tls: ssl bug #1491 workaround: not enough memory for safe operation: shm=17097536 threshold1=17301504 Apr 16 01:39:03 ip-10-0-0-0 kamailio[12054]: ERROR: <core> [core/tcp_read.c:1485]: tcp_read_req(): ERROR: tcp_read_req: error reading - c: 0x7f7ace43eb68 r: 0x7f7ace43ebe8 ```

In regards to the logs, that's something that we use for tests in the lab

@miconda what I meant is the non-expiring registration records stored in memory

As to the error messages, those are about it, everything else is repetitive and the remaining logs consist of registrations, test calls, etc.

Perhaps there's a good way to export all shm data related to uac?

@Jaykah - if they repeat, is important to see how many times -- can you count how many error messages that contain the next texts are from start to out of memory logs?

``` authentication failed for got sip response 500 while registering ```

Something like:

``` grep "authentication failed for" /path/to/logfile | wc -l ``` Just be sure that the logfile has only the logs from startup to out of memory, not older logs.

@miconda We got rid of the trunks/accounts causing 500 errors and auth failure, yet are still seeing the leak in the kamailio process.

Any other way to troubleshoot this?

@Jaykah - bug bounties are not really common in the Kamailio project so far. Commercial support options can be found on e.g. in the directory on kamailio.org. I will write you also an email with some more information.

@miconda what would be the best way to dump the kamailio uac memory segments to see those records and compare with actual registrations?

Is there a way to get access to the system exposing the issue and do some troubleshooting there (it may require sudo privileges)? The leak is identified by the number of allocated memory chunks by uac, with side effects to transactions, but the reason for it is yet unknown. With Kamailio World 2019 being finished, I expect to have more time for troubleshooting here.

You can write to miconda [at] gmail.com

Adding also that the mysql timeout might not work always, as presented in manual:

* https://dev.mysql.com/doc/refman/5.5/en/mysql-options.html

Citing:

``` MYSQL_OPT_READ_TIMEOUT (argument type: unsigned int *)

The timeout in seconds for each attempt to read from the server. There are retries if necessary, so the total effective timeout value is three times the option value. You can set the value so that a lost connection can be detected earlier than the TCP/IP Close_Wait_Timeout value of 10 minutes.

Implementation of this timeout uses mechanisms that may not be available on all platforms. On such a platform, a client that issues a read call might under certain circumstances wait without timing out. For example, a client might not time out if the server is not responding because it is waiting for a “disk full” condition to clear. ``` Therefore you should also check if there are problems with mysql server.

strings /usr/lib/x86_64-linux-gnu/libcrypto.so | grep "^OpenSSL" OpenSSLDie OpenSSL_add_all_ciphers OpenSSL_add_all_digests OpenSSL 1.0.2n 7 Dec 2017 OpenSSL HMAC method OpenSSL EC algorithm OpenSSL RSA method OpenSSL DSA method OpenSSL ECDSA method OpenSSL DH Method OpenSSL X9.42 DH method OpenSSL PKCS#3 DH method OpenSSL ECDH method OpenSSL 'dlfcn' shared library method OpenSSL default OpenSSL default user interface OpenSSL CMAC method

Yet the issue is still reproducible

@Jaykah - do you still see this error, maybe after updating to a new kamailio version?

As I looked again over the issue, I see in the backtrace (I attached) that the mysql client library is using yassl not libssl (which I expected).

@Jaykah - do you know if the library is installed from the default ubuntu repository or is something customized for RDS? Also, for you backtrace, install debugging symbols to see where the code get stuck in libmysqlclient library. There is another report on sr-dev with a blocking happening in libmysqlclient, maybe the backtrace with debug symbols will show the same like there.

@henningw we have stuck to db_mode 0, so the issue no longer resurfaces. It is still present, however, with db_mode 1.

Closed #1895.

On Debian stable I'm periodically getting following error: ``` /usr/sbin/kamailio[13996]: ERROR: tls [tls_server.c:192]: tls_complete_init(): tls: ssl bug #1491 workaround: not enough memory for safe operation: shm=7494944 threshold1=7864320 ```