[SR-Users] TLS not enough memory issue with git master
Anthony Messina
amessina at messinet.com
Tue Nov 17 14:59:50 CET 2015
Attached. -A
On Tuesday, November 17, 2015 02:50:21 PM Daniel-Constantin Mierla wrote:
> Can you run the following commands:
>
> kamcmd cfg.set_now_int core memlog 1
> kamcmd corex.shm_summary
>
> Then grab the log messages from syslog related to shared memory summary
> and send them over here.
>
> Cheers,
> Daniel
>
> On 17/11/15 14:31, Anthony Messina wrote:
> > After I reported last night, I restarted Kamailio and even though the 5
> > UACs did nothing but ensure they had a registration overnight, this
> > morning the issue has recurred. The following is the output you
> > requested. Not sure how the memory is being used up by Kamailio.
> >
> > # kamctl stats shmem
> > shmem:fragments = 181
> > shmem:free_size = 8922584
> > shmem:max_used_size = 58243792
> > shmem:real_used_size = 58186280
> > shmem:total_size = 67108864
> > shmem:used_size = 54346088
> >
> > On Tuesday, November 17, 2015 09:03:24 AM Daniel-Constantin Mierla wrote:
> >> As you are using the master branch (development), do you run latest
> >> version?
> >>
> >> Can you look at available shared memory?
> >>
> >> kamctl stats shmem
> >>
> >> Check it over time and see if the free memory is decreasing.
> >>
> >> Cheers,
> >> Daniel
> >>
> >> On 17/11/15 00:44, Anthony Messina wrote:
> >>> I have noticed the following issue which began with builds somewhere
> >>> between git master commits bff0a08 and 6173ef7. I did not see this issue
> >>> with my previous builds and haven't been able to pin down the problem,
> >>> which is why I haven't formally filed a bug.
> >>>
> >>> Any help or guidance is appreciated, because this has crippled my use of
> >>> Kamailio. Only a restart enables it to work again until the issue
> >>> recurs.
> >>>
> >>> ERROR: tls [tls_server.c:189]: tls_complete_init(): tls: ssl bug #1491
> >>> workaround: not enough memory for safe operation: 8870536
> >>> ERROR: <core> [tcp_read.c:1303]: tcp_read_req(): ERROR: tcp_read_req:
> >>> error
> >>> reading
> >>>
> >>> I currently build against and run openssl-1.0.1k-12.fc22.x86_64.
> >>>
> >>> I have a very small operation and the only change on the operational
> >>> side
> >>> is that all 5 of my mobile UACs (yes, that's all) have switched from
> >>> CSipSimple/Android to Zoiper/Android, which doesn't yet have support for
> >>> client-side certificates so verify_certificate and require_certificate
> >>> are
> >>> off for both the server and client config.
> >>>
> >>> The server is started with:
> >>> /usr/sbin/kamailio -P /run/kamailio/kamailio.pid -m 64 -M 8
> >>>
> >>> I have tried modifying the shared mem to 128 but the issue still occurs.
> >>>
> >>> Even right now, I am seeing the error when only one UAC has established
> >>> a
> >>> TLS connection:
> >>>
> >>> # kamcmd tls.list
> >>> {
> >>>
> >>> id: 572
> >>> timeout: 3475
> >>> src_ip: 10.77.79.156
> >>> src_port: 58688
> >>> dst_ip: 10.77.79.3
> >>> dst_port: 5061
> >>> cipher: ECDHE-RSA-RC4-SHA SSLv3 Kx=ECDH Au=RSA Enc=RC4(128)
> >>> Mac=SHA1
> >>> ct_wq_size: 0
> >>> enc_rd_buf: 0
> >>> flags: 2
> >>> state: established
> >>>
> >>> }
> >>>
> >>> # kamailio.cfg
> >>> enable_tls=yes
> >>> loadmodule "tls.so"
> >>> modparam("tls", "connection_timeout", 60)
> >>> #modparam("tls", "tls_log", 1)
> >>> #modparam("tls", "tls_debug", 1)
> >>> #modparam("tls", "low_mem_threshold1", -1)
> >>> #modparam("tls", "low_mem_threshold2", 0)
> >>> modparam("tls", "session_cache", 1)
> >>>
> >>> # tls.cfg
> >>> [server:default]
> >>> method = TLSv1+
> >>> verify_certificate = no
> >>> require_certificate = no
> >>> private_key = /etc/kamailio/example.org.key.pem
> >>> certificate = /etc/kamailio/example.org.crt.pem
> >>> server_name = example.org
> >>> cipher_list =
> >>> ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-
> >>> AES128-GCM-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-RC4-SHA:ECDHE-RSA-AE
> >>> S
> >>> 256-
> >>> SHA:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-SHA256:DHE-RSA-AES128-GCM-
> >>> SHA256:DHE-RSA-AES128-SHA256:DHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA:RC4-
> >>> SHA:AES256-GCM-SHA384:AES256-SHA256:CAMELLIA256-SHA:ECDHE-RSA-AES128-
> >>> SHA:AES128-GCM-SHA256:AES128-SHA256:AES128-SHA:CAMELLIA128-
> >>> SHA:!aNULL:!eNULL:!EXPORT:!DES:!3DES:!MD5:!PSK
> >>>
> >>> [client:default]
> >>> method = TLSv1+
> >>> verify_certificate = no
> >>> require_certificate = no
> >>> private_key = /etc/kamailio/example.org.key.pem
> >>> certificate = /etc/kamailio/example.org.crt.pem
> >>> cipher_list =
> >>> ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-
> >>> AES128-GCM-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-RC4-SHA:ECDHE-RSA-AE
> >>> S
> >>> 256-
> >>> SHA:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-SHA256:DHE-RSA-AES128-GCM-
> >>> SHA256:DHE-RSA-AES128-SHA256:DHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA:RC4-
> >>> SHA:AES256-GCM-SHA384:AES256-SHA256:CAMELLIA256-SHA:ECDHE-RSA-AES128-
> >>> SHA:AES128-GCM-SHA256:AES128-SHA256:AES128-SHA:CAMELLIA128-
> >>> SHA:!aNULL:!eNULL:!EXPORT:!DES:!3DES:!MD5:!PSK
> >>>
> >>>
> >>> Thanks. -Anthony
--
Anthony - https://messinet.com/ - https://messinet.com/~amessina/gallery
8F89 5E72 8DF0 BCF0 10BE 9967 92DC 35DC B001 4A4E
-------------- next part --------------
ERROR: tls [tls_server.c:189]: tls_complete_init(): tls: ssl bug #1491 workaround: not enough memory for safe operation: 8905448
ERROR: <core> [tcp_read.c:1303]: tcp_read_req(): ERROR: tcp_read_req: error reading
NOTICE: fm_status: summarizing all alloc'ed. fragments:
NOTICE: fm_status: count= 4 size= 1472 bytes from tmx: tmx_pretran.c: tmx_check_pretran(271)
NOTICE: fm_status: count= 1 size= 144 bytes from core: dns_cache.c: dns_cache_mk_rd_entry(1216)
NOTICE: fm_status: count= 1 size= 32 bytes from usrloc: urecord.c: new_urecord(65)
NOTICE: fm_status: count= 2 size= 200 bytes from core: xavp.c: xavp_new_value(94)
NOTICE: fm_status: count= 4 size= 808 bytes from tmx: tmx_pretran.c: tmx_check_pretran(250)
NOTICE: fm_status: count= 1 size= 256 bytes from usrloc: ucontact.c: new_ucontact(98)
NOTICE: fm_status: count= 1 size= 64 bytes from usrloc: urecord.c: new_urecord(58)
NOTICE: fm_status: count= 5 size= 216 bytes from usrloc: ../../ut.h: shm_str_dup(691)
NOTICE: fm_status: count= 572 size= 11733488 bytes from core: tcp_main.c: tcpconn_new(957)
NOTICE: fm_status: count= 572 size= 33456 bytes from tls: tls_server.c: tls_complete_init(227)
NOTICE: fm_status: count= 1 size= 17408 bytes from core: counters.c: counters_prefork_init(207)
NOTICE: fm_status: count= 1 size= 256 bytes from tmx: tmx_pretran.c: tmx_init_pretran_table(90)
NOTICE: fm_status: count= 1 size= 4096 bytes from sl: sl_stats.c: init_sl_stats_child(125)
NOTICE: fm_status: count= 1 size= 4352 bytes from tm: t_stats.c: init_tm_stats_child(60)
NOTICE: fm_status: count= 1 size= 40960 bytes from usrloc: udomain.c: new_udomain(93)
NOTICE: fm_status: count= 1 size= 8192 bytes from websocket: ws_conn.c: wsconn_init(103)
NOTICE: fm_status: count= 1 size= 4112 bytes from pike: ip_tree.c: init_ip_tree(115)
NOTICE: fm_status: count= 1 size= 1680 bytes from core: cfg/cfg_struct.c: cfg_shmize(217)
NOTICE: fm_status: count= 1 size= 4096 bytes from core: pt.c: init_pt(104)
NOTICE: fm_status: count= 1 size= 48 bytes from usrloc: udomain.c: new_udomain(86)
NOTICE: fm_status: count= 1 size= 8 bytes from websocket: ws_conn.c: wsconn_init(89)
NOTICE: fm_status: count= 1 size= 4096 bytes from imc: imc_mng.c: imc_htable_init(58)
NOTICE: fm_status: count= 4 size= 38160 bytes from htable: ht_api.c: ht_init_tables(336)
NOTICE: fm_status: count= 3 size= 568 bytes from imc: imc_mng.c: imc_add_room(131)
NOTICE: fm_status: count= 1 size= 8 bytes from tls: tls_mod.c: mod_init(335)
NOTICE: fm_status: count= 1 size= 8 bytes from tls: tls_ct_wrq.c: tls_ct_wq_init(49)
NOTICE: fm_status: count= 512 size= 132144 bytes from pua: hash.c: new_htable(93)
NOTICE: fm_status: count= 1 size= 8192 bytes from pua: hash.c: new_htable(79)
NOTICE: fm_status: count= 1 size= 8192 bytes from presence: hash.c: new_phtable(410)
NOTICE: fm_status: count= 1 size= 131088 bytes from dialog: dlg_hash.c: init_dlg_table(277)
NOTICE: fm_status: count= 1 size= 8 bytes from pua: hash.c: new_htable(64)
NOTICE: fm_status: count= 1 size= 24 bytes from rtpengine: rtpengine.c: add_rtpengine_socks(672)
NOTICE: fm_status: count= 10 size= 2080 bytes from imc: imc_mng.c: imc_add_member(306)
NOTICE: fm_status: count= 1 size= 56 bytes from pua: event_list.c: init_pua_evlist(36)
NOTICE: fm_status: count= 1 size= 1200 bytes from pike: ../../lock_alloc.h: lock_set_alloc(70)
NOTICE: fm_status: count= 1 size= 1032 bytes from domain: domain_mod.c: mod_init(207)
NOTICE: fm_status: count= 512 size= 168712 bytes from presence: hash.c: new_shtable(66)
NOTICE: fm_status: count= 1 size= 8192 bytes from presence: hash.c: new_shtable(53)
NOTICE: fm_status: count= 1 size= 592 bytes from lcr: lcr_mod.c: prefix_len_insert(953)
NOTICE: fm_status: count= 4 size= 355008 bytes from lcr: lcr_mod.c: mod_init(699)
NOTICE: fm_status: count= 2 size= 1184 bytes from lcr: lcr_mod.c: prefix_len_insert(968)
NOTICE: fm_status: count= 4 size= 71024 bytes from lcr: lcr_mod.c: mod_init(672)
NOTICE: fm_status: count= 2 size= 2048 bytes from permissions: hash.c: new_domain_name_table(928)
NOTICE: fm_status: count= 1 size= 32 bytes from usrloc: dlist.c: new_dlist(565)
NOTICE: fm_status: count= 3 size= 424 bytes from tls: tls_domain.c: fix_domain(962)
NOTICE: fm_status: count= 1 size= 48 bytes from tls: tls_domain.c: tls_new_cfg(1274)
NOTICE: fm_status: count= 1 size= 32 bytes from msilo: ms_msg_list.c: msg_list_init(87)
NOTICE: fm_status: count= 1 size= 8 bytes from core: pt.c: init_pt(105)
NOTICE: fm_status: count= 3 size= 672 bytes from tls: tls_domain.c: tls_new_domain(145)
NOTICE: fm_status: count= 4 size= 352 bytes from permissions: hash.c: addr_hash_table_insert(481)
NOTICE: fm_status: count= 1 size= 48 bytes from rtpengine: rtpengine.c: get_rtpp_set(587)
NOTICE: fm_status: count= 1 size= 16 bytes from pike: pike.c: pike_init(136)
NOTICE: fm_status: count= 1 size= 8 bytes from permissions: address.c: init_addresses(325)
NOTICE: fm_status: count= 1 size= 8 bytes from permissions: trusted.c: init_trusted(223)
NOTICE: fm_status: count= 1 size= 8 bytes from dialog: dlg_hash.c: init_dlg_table(262)
NOTICE: fm_status: count= 1 size= 8 bytes from core: pt.c: init_pt(110)
NOTICE: fm_status: count= 1 size= 8 bytes from permissions: address.c: init_addresses(339)
NOTICE: fm_status: count= 1 size= 16 bytes from websocket: ws_conn.c: wsconn_init(113)
NOTICE: fm_status: count= 1 size= 8 bytes from permissions: address.c: init_addresses(353)
NOTICE: fm_status: count= 1 size= 8 bytes from exec: kill.c: initialize_kill(151)
NOTICE: fm_status: count= 1 size= 928 bytes from kex: pkg_stats.c: pkg_proc_stats_init(79)
NOTICE: fm_status: count= 1 size= 8 bytes from dialog: dlg_hash.c: init_dlg_table(269)
NOTICE: fm_status: count= 1 size= 8 bytes from dialog: dlg_hash.c: init_dlg_table(257)
NOTICE: fm_status: count= 7 size= 704 bytes from pua: event_list.c: add_pua_event(72)
NOTICE: fm_status: count= 1 size= 8 bytes from tls: tls_mod.c: mod_init(319)
NOTICE: fm_status: count= 1 size= 8 bytes from nathelper: nathelper.c: mod_init(663)
NOTICE: fm_status: count= 3 size= 64 bytes from usrloc: udomain.c: build_stat_name(51)
NOTICE: fm_status: count= 1 size= 8 bytes from lcr: lcr_mod.c: mod_init(709)
NOTICE: fm_status: count= 1 size= 32 bytes from lcr: lcr_mod.c: mod_init(685)
NOTICE: fm_status: count= 4 size= 48 bytes from presence: event_list.c: add_event(156)
NOTICE: fm_status: count= 1 size= 8 bytes from pike: pike.c: pike_init(118)
NOTICE: fm_status: count= 1 size= 80 bytes from rtpengine: rtpengine.c: add_rtpengine_socks(661)
NOTICE: fm_status: count= 1 size= 8 bytes from domain: domain_mod.c: mod_init(203)
NOTICE: fm_status: count= 1 size= 16 bytes from usrloc: dlist.c: new_dlist(573)
NOTICE: fm_status: count= 1 size= 8 bytes from dialog: dlg_timer.c: init_dlg_timer(57)
NOTICE: fm_status: count= 1 size= 32 bytes from lcr: lcr_mod.c: mod_init(660)
NOTICE: fm_status: count= 1 size= 16 bytes from presence: event_list.c: init_evlist(289)
NOTICE: fm_status: count= 1 size= 8 bytes from domain: domain_mod.c: mod_init(219)
NOTICE: fm_status: count= 1 size= 16 bytes from rtpengine: rtpengine.c: get_rtpp_set(604)
NOTICE: fm_status: count= 4 size= 48 bytes from presence: event_list.c: shm_copy_event(57)
NOTICE: fm_status: count= 19 size= 344 bytes from core: cfg/cfg_struct.c: cfg_clone_str(130)
NOTICE: fm_status: count= 4 size= 744 bytes from presence: event_list.c: add_event(149)
NOTICE: fm_status: count= 1 size= 32 bytes from dialog: dlg_timer.c: init_dlg_timer(48)
NOTICE: fm_status: count= 1 size= 8 bytes from websocket: ws_conn.c: wsconn_init(77)
NOTICE: fm_status: count= 9 size= 1840 bytes from core: cfg_parser.c: cfg_parse_str(906)
NOTICE: fm_status: count= 2 size= 3584 bytes from permissions: hash.c: new_addr_hash_table(442)
NOTICE: fm_status: count= 2 size= 2048 bytes from permissions: hash.c: new_hash_table(95)
NOTICE: fm_status: count= 4 size= 424 bytes from presence: event_list.c: shm_copy_event(50)
NOTICE: fm_status: count= 2204 size= 1305168 bytes from lcr: hash.c: rule_hash_table_insert(50)
NOTICE: fm_status: count= 2 size= 14448 bytes from permissions: hash.c: new_subnet_table(678)
NOTICE: fm_status: count= 1 size= 24 bytes from pua: pua_callback.c: init_puacb_list(35)
NOTICE: fm_status: count= 512 size= 23024 bytes from presence: hash.c: new_phtable(424)
NOTICE: fm_status: count= 4 size= 120 bytes from presence: event_list.c: add_event(181)
NOTICE: fm_status: count= 2204 size= 38872 bytes from lcr: hash.c: rule_hash_table_insert_target(138)
NOTICE: fm_status: count= 1 size= 1200 bytes from domain: domain_mod.c: mod_init(205)
NOTICE: fm_status: count= 1 size= 1048576 bytes from auth: nc.c: init_nonce_count(128)
NOTICE: fm_status: count= 1 size= 256 bytes from auth: nid.c: init_nonce_id(74)
NOTICE: fm_status: count= 2 size= 32 bytes from dialog: dlg_cb.c: init_dlg_callback(55)
NOTICE: fm_status: count= 3 size= 120 bytes from dialog: dlg_cb.c: register_dlgcb(124)
NOTICE: fm_status: count= 2 size= 80 bytes from tm: t_hooks.c: insert_tmcb(137)
NOTICE: fm_status: count= 1 size= 16 bytes from usrloc: ul_callback.c: init_ulcb_list(45)
NOTICE: fm_status: count= 1 size= 4112 bytes from usrloc: ../../lock_alloc.h: lock_set_alloc(70)
NOTICE: fm_status: count= 12 size= 976 bytes from core: timer.c: register_timer(1011)
NOTICE: fm_status: count= 1 size= 24 bytes from outbound: ob_mod.c: mod_init(104)
NOTICE: fm_status: count= 1 size= 8 bytes from sl: sl_funcs.c: sl_startup(83)
NOTICE: fm_status: count= 1 size= 8 bytes from sl: sl_stats.c: init_sl_stats(110)
NOTICE: fm_status: count= 1 size= 16 bytes from tm: t_hooks.c: init_tmcb_lists(74)
NOTICE: fm_status: count= 1 size= 16 bytes from tm: t_hooks.c: init_tmcb_lists(72)
NOTICE: fm_status: count= 1 size= 2097152 bytes from tm: h_table.c: init_hash_table(458)
NOTICE: fm_status: count= 4 size= 128 bytes from core: cfg/cfg_ctx.c: cfg_register_ctx(47)
NOTICE: fm_status: count= 1 size= 80 bytes from db_postgres: ../../lock_alloc.h: lock_set_alloc(70)
NOTICE: fm_status: count= 669 size= 135352 bytes from tls: tls_init.c: ser_realloc(280)
NOTICE: fm_status: count= 1 size= 8192 bytes from core: tcp_main.c: init_tcp(4635)
NOTICE: fm_status: count= 1 size= 32768 bytes from core: tcp_main.c: init_tcp(4629)
NOTICE: fm_status: count= 1 size= 8 bytes from core: tcp_main.c: init_tcp(4621)
NOTICE: fm_status: count= 1 size= 8 bytes from core: tcp_main.c: init_tcp(4614)
NOTICE: fm_status: count= 1 size= 8 bytes from core: tcp_main.c: init_tcp(4607)
NOTICE: fm_status: count= 1 size= 8 bytes from core: tcp_main.c: init_tcp(4601)
NOTICE: fm_status: count= 1 size= 8 bytes from core: tcp_main.c: init_tcp(4589)
NOTICE: fm_status: count= 1 size= 8 bytes from core: usr_avp.c: init_avps(90)
NOTICE: fm_status: count= 1 size= 8 bytes from core: usr_avp.c: init_avps(89)
NOTICE: fm_status: count= 1 size= 16384 bytes from core: dst_blacklist.c: init_dst_blacklist(437)
NOTICE: fm_status: count= 1 size= 8 bytes from core: dst_blacklist.c: init_dst_blacklist(430)
NOTICE: fm_status: count= 2 size= 96 bytes from core: timer.c: timer_alloc(514)
NOTICE: fm_status: count= 1 size= 8 bytes from core: dns_cache.c: init_dns_cache(366)
NOTICE: fm_status: count= 1 size= 16384 bytes from core: dns_cache.c: init_dns_cache(358)
NOTICE: fm_status: count= 1 size= 16 bytes from core: dns_cache.c: init_dns_cache(351)
NOTICE: fm_status: count= 1 size= 8 bytes from core: dns_cache.c: init_dns_cache(345)
NOTICE: fm_status: count= 1 size= 8 bytes from core: timer.c: init_timer(283)
NOTICE: fm_status: count= 1 size= 16384 bytes from core: timer.c: init_timer(282)
NOTICE: fm_status: count= 1 size= 8 bytes from core: timer.c: init_timer(281)
NOTICE: fm_status: count= 1 size= 8 bytes from core: timer.c: init_timer(280)
NOTICE: fm_status: count= 1 size= 8 bytes from core: timer.c: init_timer(269)
NOTICE: fm_status: count= 1 size= 8 bytes from core: timer.c: init_timer(237)
NOTICE: fm_status: count= 1 size= 278544 bytes from core: timer.c: init_timer(220)
NOTICE: fm_status: count= 1 size= 8 bytes from core: timer.c: init_timer(219)
NOTICE: fm_status: count= 1 size= 8 bytes from core: timer.c: init_timer(206)
NOTICE: fm_status: count= 1 size= 64 bytes from core: cfg/cfg_struct.c: cfg_child_cb_new(830)
NOTICE: fm_status: count= 1 size= 8 bytes from core: cfg/cfg_struct.c: sr_cfg_init(361)
NOTICE: fm_status: count= 1 size= 8 bytes from core: cfg/cfg_struct.c: sr_cfg_init(354)
NOTICE: fm_status: count= 1 size= 8 bytes from core: cfg/cfg_struct.c: sr_cfg_init(347)
NOTICE: fm_status: count= 1 size= 8 bytes from core: cfg/cfg_struct.c: sr_cfg_init(335)
NOTICE: fm_status: count= 1 size= 8 bytes from core: cfg/cfg_struct.c: sr_cfg_init(323)
NOTICE: fm_status: count= 1 size= 32 bytes from auth_ephemeral: autheph_mod.c: add_secret(226)
NOTICE: fm_status: count= 1 size= 8 bytes from auth_ephemeral: autheph_mod.c: add_secret(213)
NOTICE: fm_status: count= 1 size= 64 bytes from auth_ephemeral: autheph_mod.c: secret_param(305)
NOTICE: fm_status: count= 4 size= 384 bytes from htable: ht_api.c: ht_add_table(275)
NOTICE: fm_status: count= 44671 size= 36504848 bytes from tls: tls_init.c: ser_malloc(274)
NOTICE: fm_status: count= 1 size= 184 bytes from tls: ../../lock_alloc.h: lock_set_alloc(70)
NOTICE: fm_status: count= 1 size= 8 bytes from core: mem/shm.c: shm_core_lock_init(153)
NOTICE: fm_status: -----------------------------
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 181 bytes
Desc: This is a digitally signed message part.
URL: <http://lists.sip-router.org/pipermail/sr-users/attachments/20151117/5cf315f1/attachment.sig>
More information about the sr-users
mailing list