<div dir="ltr">Hi Daniel,<div><br></div><div>Thanks for your reply. </div><div><br></div><div>Yes, It is a similar issue as you shared. I have installed debug symbols for libmysqlclient library, let's see what more info it can give.</div><div><br></div><div>TLS is not enabled, yet. Just to mention, MySQL instance is on RDS, however, I have not seen such behavior before in other deployments.</div><div><br></div><div>location, dialog, presence is being used with MySQL.</div><div><br></div><div>@David, thanks for trying to help. Turning DB mode to in memory is not going to work in my case scenarios. Thanks<br></div><div><br></div><div>Regards, </div><div>Salman</div><div>VoIP Professional</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Aug 5, 2019 at 7:58 PM Daniel-Constantin Mierla <<a href="mailto:miconda@gmail.com" target="_blank">miconda@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>Hello,</p>
<p>this seems similar to the issue reported in the tracker:</p>
<p> - <a href="https://github.com/kamailio/kamailio/issues/1895#issuecomment-500895303" target="_blank">https://github.com/kamailio/kamailio/issues/1895#issuecomment-500895303</a></p>
<p>I was troubleshooting on that system in the past, but recently I
couldn't get more there. Not sure why the mysql query is not
timing out, on the net there were mentions that is cannot be
ensured, it's more like a best try.</p>
<p>You should install debug symbols for libmysqlclient library and
see if you get when in the libmysqlclient code is getting stuck.</p>
<p>Do you have tls enabled in kamailio?</p>
<p>Cheers,<br>
Daniel<br>
</p>
<div class="gmail-m_2769825626893760435gmail-m_4388046060681410490moz-cite-prefix">On 05.08.19 17:03, David Villasmil
wrote:<br>
</div>
<blockquote type="cite">
<div>
<div dir="auto">At least try setting db_mode for the dialog
module to 0 (zero)</div>
</div>
<div><br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Mon, 5 Aug 2019 at 16:02,
David Villasmil <<a href="mailto:david.villasmil.work@gmail.com" target="_blank">david.villasmil.work@gmail.com</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div dir="auto">Have you tried taking out the dB support
and see if it keeps happening? The by talks a lot about
MySQL... no bT expert here...</div>
</div>
<div><br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Mon, 5 Aug 2019 at
14:07, M. Salman <<a href="mailto:msalman212@gmail.com" target="_blank">msalman212@gmail.com</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div>Hi Team,</div>
<div><br>
</div>
<div>Facing 100% Kamilio CPU issue, here are the
details, any pointers/hints would help. Thanks</div>
<div><br>
</div>
<div><b>Scenario:</b></div>
<div>Random Kamailio CPU shoot, restart fixes it.
Happens once or twice in a day.</div>
<div>It's a pre-production server with around 30
extensions, not much load. For now, it has 2 CPUs
and 4 G memory.</div>
<div><br>
</div>
<div><b>[Info]</b></div>
<div><br>
</div>
<div>
<p class="gmail-m_2769825626893760435gmail-m_4388046060681410490m_1403799627641090686m_6369401794401130355gmail-p1"><span class="gmail-m_2769825626893760435gmail-m_4388046060681410490m_1403799627641090686m_6369401794401130355gmail-s1" style="font-variant-ligatures:no-common-ligatures">uname
-a</span></p>
<p class="gmail-m_2769825626893760435gmail-m_4388046060681410490m_1403799627641090686m_6369401794401130355gmail-p1"><span class="gmail-m_2769825626893760435gmail-m_4388046060681410490m_1403799627641090686m_6369401794401130355gmail-s1" style="font-variant-ligatures:no-common-ligatures">Linux
ip-X-aws #46-Ubuntu SMP Thu Jul 4 13:38:28 UTC
2019 x86_64 x86_64 x86_64 GNU/Linux</span></p>
</div>
<div><br>
</div>
<div>
<p class="gmail-m_2769825626893760435gmail-m_4388046060681410490m_1403799627641090686m_6369401794401130355gmail-p1"><span class="gmail-m_2769825626893760435gmail-m_4388046060681410490m_1403799627641090686m_6369401794401130355gmail-s1" style="font-variant-ligatures:no-common-ligatures">kamailio<span class="gmail-m_2769825626893760435gmail-m_4388046060681410490m_1403799627641090686m_6369401794401130355gmail-Apple-converted-space">
</span>-V</span></p>
<p class="gmail-m_2769825626893760435gmail-m_4388046060681410490m_1403799627641090686m_6369401794401130355gmail-p1"><span class="gmail-m_2769825626893760435gmail-m_4388046060681410490m_1403799627641090686m_6369401794401130355gmail-s1" style="font-variant-ligatures:no-common-ligatures">version:
kamailio 5.2.3 (x86_64/linux) c36229</span></p>
<p class="gmail-m_2769825626893760435gmail-m_4388046060681410490m_1403799627641090686m_6369401794401130355gmail-p1"><span class="gmail-m_2769825626893760435gmail-m_4388046060681410490m_1403799627641090686m_6369401794401130355gmail-s1" style="font-variant-ligatures:no-common-ligatures">flags:
STATS: Off, USE_TCP, USE_TLS, USE_SCTP,
TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE,
USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP,
PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC,
DBG_SR_MEMORY, USE_FUTEX,
FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE,
USE_DNS_FAILOVER, USE_NAPTR,
USE_DST_BLACKLIST, HAVE_RESOLV_RES</span></p>
<p class="gmail-m_2769825626893760435gmail-m_4388046060681410490m_1403799627641090686m_6369401794401130355gmail-p1"><span class="gmail-m_2769825626893760435gmail-m_4388046060681410490m_1403799627641090686m_6369401794401130355gmail-s1" style="font-variant-ligatures:no-common-ligatures">ADAPTIVE_WAIT_LOOPS=1024,
MAX_RECV_BUFFER_SIZE 262144 MAX_URI_SIZE 1024,
BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB</span></p>
<p class="gmail-m_2769825626893760435gmail-m_4388046060681410490m_1403799627641090686m_6369401794401130355gmail-p1"><span class="gmail-m_2769825626893760435gmail-m_4388046060681410490m_1403799627641090686m_6369401794401130355gmail-s1" style="font-variant-ligatures:no-common-ligatures">poll
method support: poll, epoll_lt, epoll_et,
sigio_rt, select.</span></p>
<p class="gmail-m_2769825626893760435gmail-m_4388046060681410490m_1403799627641090686m_6369401794401130355gmail-p1"><span class="gmail-m_2769825626893760435gmail-m_4388046060681410490m_1403799627641090686m_6369401794401130355gmail-s1" style="font-variant-ligatures:no-common-ligatures">id:
c36229<span class="gmail-m_2769825626893760435gmail-m_4388046060681410490m_1403799627641090686m_6369401794401130355gmail-Apple-converted-space"> </span></span></p>
</div>
<div><br>
</div>
<div><b>(gdb) bt full</b><br>
#0 0x00007f0069d97aab in ?? () from
/usr/lib/x86_64-linux-gnu/libmysqlclient.so.20<br>
No symbol table info available.<br>
#1 0x00007f0069d7d6bd in ?? () from
/usr/lib/x86_64-linux-gnu/libmysqlclient.so.20<br>
No symbol table info available.<br>
#2 0x00007f0069d657c7 in ?? () from
/usr/lib/x86_64-linux-gnu/libmysqlclient.so.20<br>
No symbol table info available.<br>
#3 0x00007f0069d3ebce in ?? () from
/usr/lib/x86_64-linux-gnu/libmysqlclient.so.20<br>
No symbol table info available.<br>
#4 0x00007f0069d3fc4c in ?? () from
/usr/lib/x86_64-linux-gnu/libmysqlclient.so.20<br>
No symbol table info available.<br>
#5 0x00007f0069d33edc in ?? () from
/usr/lib/x86_64-linux-gnu/libmysqlclient.so.20<br>
No symbol table info available.<br>
#6 0x00007f0069d34fa8 in ?? () from
/usr/lib/x86_64-linux-gnu/libmysqlclient.so.20<br>
No symbol table info available.<br>
#7 0x00007f0069d36d71 in mysql_real_query () from
/usr/lib/x86_64-linux-gnu/libmysqlclient.so.20<br>
No symbol table info available.<br>
#8 0x00007f006a31e4b4 in db_mysql_submit_query
(h=0x7f006a6cdb20, s=0x7f0069af92d0
<sql_str>) at km_dbase.c:116<br>
t = 1564528172<br>
i = 0<br>
code = 0<br>
_func_ = "db_mysql_submit_query"<br>
#9 0x00007f00698e8fb3 in db_do_submit_query
(h=0x7f006a6cdb20, query=0x7f0069af92d0
<sql_str>,<br>
submit_query=0x7f006a31dd3c
<db_mysql_submit_query>) at db_query.c:58<br>
ret = 10<br>
tvb = {tv_sec = 0, tv_usec = 0}<br>
tve = {tv_sec = 0, tv_usec =
2077076196242373120}<br>
tz = {tz_minuteswest = 0, tz_dsttime = 0}<br>
tdiff = 0<br>
_func_ = "db_do_submit_query"<br>
#10 0x00007f00698ec1dc in db_do_delete
(h=0x7f006a6cdb20, k=0x7ffcdb440fa0, o=0x0,
v=0x7ffcdb440fb0, _n=2,<br>
val2str=0x7f006a340429
<db_mysql_val2str>,
submit_query=0x7f006a31dd3c
<db_mysql_submit_query>) at db_query.c:299<br>
off = 68<br>
ret = 36<br>
---Type <return> to continue, or q
<return> to quit---<br>
_func_ = "db_do_delete"<br>
#11 0x00007f006a322f6a in db_mysql_delete
(h=0x7f006a6cdb20, k=0x7ffcdb440fa0, o=0x0,
v=0x7ffcdb440fb0, _n=2) at km_dbase.c:515<br>
No locals.<br>
#12 0x00007f0065b6da70 in remove_dialog_from_db
(cell=0x7f005c5ed3d8) at dlg_db_handler.c:722<br>
values = {{type = DB1_INT, nul = 0, free =
477963473, val = {int_val = 1460, ll_val =
140720308487604,<br>
double_val =
6.9525070095907518e-310, time_val =
140720308487604,<br>
string_val = 0x7ffc000005b4
<error: Cannot access memory at address
0x7ffc000005b4>, str_val = {<br>
s = 0x7ffc000005b4 <error:
Cannot access memory at address
0x7ffc000005b4>, len = 0}, blob_val = {<br>
s = 0x7ffc000005b4 <error:
Cannot access memory at address
0x7ffc000005b4>, len = 0}, bitmap_val = 1460,
uint_val = 1460,<br>
ull_val = 140720308487604}}, {type =
DB1_INT, nul = 0, free = 0, val = {int_val = 8020,
ll_val = 139637976735572,<br>
double_val = 6.899032715982534e-310,
time_val = 139637976735572,<br>
string_val = 0x7f0000001f54
<error: Cannot access memory at address
0x7f0000001f54>, str_val = {<br>
s = 0x7f0000001f54 <error:
Cannot access memory at address
0x7f0000001f54>, len = 0}, blob_val = {<br>
s = 0x7f0000001f54 <error:
Cannot access memory at address
0x7f0000001f54>, len = 0}, bitmap_val = 8020,
uint_val = 8020,<br>
ull_val = 139637976735572}}}<br>
match_keys = {0x7f0065dfc6c0
<h_entry_column>, 0x7f0065dfc6b0
<h_id_column>}<br>
vars_match_keys = {0x7f0065dfc7f0
<vars_h_entry_column>, 0x7f0065dfc7e0
<vars_h_id_column>}<br>
_func_ = "remove_dialog_from_db"<br>
#13 0x00007f0065b49375 in destroy_dlg
(dlg=0x7f005c5ed3d8) at dlg_hash.c:371<br>
ret = 1<br>
var = 0x7f0065b45442 <futex_get+40><br>
_func_ = "destroy_dlg"<br>
#14 0x00007f0065b50ca0 in dlg_unref_helper
(dlg=0x7f005c5ed3d8, cnt=2, fname=0x7f0065be1eff
"dlg_handlers.c", fline=1182) at dlg_hash.c:961<br>
d_entry = 0x7f005c45b4c0<br>
_func_ = "dlg_unref_helper"<br>
#15 0x00007f0065b7f708 in unref_dlg_from_cb
(t=0x7f005c5b5dc0, type=131072,
param=0x7ffcdb4412a0) at dlg_handlers.c:1182<br>
dlg = 0x7f005c5ed3d8<br>
iuid = 0x7f005c568c38<br>
#16 0x00007f0068295b05 in
run_trans_callbacks_internal
(cb_lst=0x7f005c5b5e38, type=131072,
trans=0x7f005c5b5dc0, params=0x7ffcdb4412a0)<br>
at t_hooks.c:260<br>
cbp = 0x7f005c606118<br>
backup_from = 0x5569cc3da8b0
<def_list+16><br>
backup_to = 0x5569cc3da8b8
<def_list+24><br>
---Type <return> to continue, or q
<return> to quit---<br>
backup_dom_from = 0x5569cc3da8c0
<def_list+32><br>
backup_dom_to = 0x5569cc3da8c8
<def_list+40><br>
backup_uri_from = 0x5569cc3da8a0
<def_list><br>
backup_uri_to = 0x5569cc3da8a8
<def_list+8><br>
backup_xavps = 0x5569cc3da2c0
<_xavp_list_head><br>
_func_ = "run_trans_callbacks_internal"<br>
#17 0x00007f0068295c2c in run_trans_callbacks
(type=131072, trans=0x7f005c5b5dc0, req=0x0,
rpl=0x0, code=0) at t_hooks.c:287<br>
params = {req = 0x0, rpl = 0x0, param =
0x7f005c606128, code = 0, flags = 0, branch = 0,
t_rbuf = 0x0, dst = 0x0, send_buf = {<br>
s = 0x0, len = 0}}<br>
#18 0x00007f00681cbf45 in free_cell_helper
(dead_cell=0x7f005c5b5dc0, silent=0,
fname=0x7f00682de882 "timer.c", fline=689) at
h_table.c:166<br>
b = 0x7f005c5b5dc0 ""<br>
i = 1<br>
rpl = 0x7ffcdb4413a0<br>
tt = 0x0<br>
foo = 0x7ffcdb441390<br>
cbs = 0x7f005c378700<br>
cbs_tmp = 0x1681ca80f<br>
_func_ = "free_cell_helper"<br>
#19 0x00007f0068283bf6 in wait_handler
(ti=855479619, wait_tl=0x7f005c5b5e48,
data=0x7f005c5b5dc0) at timer.c:689<br>
p_cell = 0x7f005c5b5dc0<br>
ret = 1<br>
unlinked = 0<br>
rcount = 1<br>
_func_ = "wait_handler"<br>
#20 0x00005569cbdb0661 in timer_list_expire
(t=855479619, h=0x7f005c238af0,
slow_l=0x7f005c23ae08, slow_mark=5646) at
core/timer.c:874<br>
tl = 0x7f005c5b5e48<br>
ret = 0<br>
#21 0x00005569cbdb0b09 in timer_handler () at
core/timer.c:939<br>
saved_ticks = 855479619<br>
run_slow_timer = 0<br>
i = 526<br>
_func_ = "timer_handler"<br>
#22 0x00005569cbdb0fd2 in timer_main () at
core/timer.c:978<br>
---Type <return> to continue, or q
<return> to quit---<br>
No locals.<br>
#23 0x00005569cbcf1382 in main_loop () at
main.c:1727<br>
i = 8<br>
pid = 0<br>
si = 0x0<br>
si_desc = "udp receiver child=7 sock=<a href="http://172.31.2.33:6000" target="_blank">172.31.2.33:6000</a> (<a href="http://13.54.252.29:6000" target="_blank">13.54.252.29:6000</a>)\000\034\004\000\000\000\000\000\000\000\321$}\034\000\000\000\000P0\316\313iU\000\000
\031D\333\374\177", '\000' <repeats 18
times>,
"`\026D\333\374\177\000\000\360S\361\313iU\000"<br>
nrprocs = 8<br>
woneinit = 1<br>
_func_ = "main_loop"<br>
#24 0x00005569cbcf8a7b in main (argc=13,
argv=0x7ffcdb441928) at main.c:2696<br>
cfg_stream = 0x5569cd7c7260<br>
c = -1<br>
r = 0<br>
tmp = 0x7ffcdb441f39 ""<br>
tmp_len = -616294480<br>
port = 32764<br>
proto = 3<br>
options = 0x5569cc074190
":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:x:X:Y:"<br>
ret = -1<br>
seed = 3114863458<br>
rfd = 4<br>
debug_save = 0<br>
debug_flag = 0<br>
dont_fork_cnt = 0<br>
n_lst = 0x7f006af01787<br>
p = 0x9c0 <error: Cannot access memory
at address 0x9c0><br>
st = {st_dev = 23, st_ino = 456, st_nlink
= 2, st_mode = 16877, st_uid = 0, st_gid = 0,
__pad0 = 0, st_rdev = 0, st_size = 60,<br>
st_blksize = 4096, st_blocks = 0,
st_atim = {tv_sec = 1564229247, tv_nsec =
809557407}, st_mtim = {tv_sec = 1564229229,<br>
tv_nsec = 217250917}, st_ctim =
{tv_sec = 1564229279, tv_nsec = 666090748},
__glibc_reserved = {0, 0, 0}}<br>
_func_ = "main"<br>
(gdb)</div>
</div>
<div dir="ltr">
<div><br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr" class="gmail-m_2769825626893760435gmail-m_4388046060681410490m_1403799627641090686m_6369401794401130355gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr"><font face="'times new
roman', serif">Regards</font>
<div><font face="'times new roman', serif"><br>
</font>
<div>
<pre cols="72"><font face="'times new roman', serif">M. Salman
</font>
</pre>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
_______________________________________________<br>
Kamailio (SER) - Users Mailing List<br>
<a href="mailto:sr-users@lists.kamailio.org" target="_blank">sr-users@lists.kamailio.org</a><br>
<a href="https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users" rel="noreferrer" target="_blank">https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users</a><br>
</blockquote>
</div>
</div>
-- <br>
<div dir="ltr" class="gmail-m_2769825626893760435gmail-m_4388046060681410490m_1403799627641090686gmail_signature">
<div dir="ltr">
<div>Regards,</div>
<div><br>
</div>
David Villasmil
<div>email: <a href="mailto:david.villasmil.work@gmail.com" target="_blank">david.villasmil.work@gmail.com</a></div>
<div>phone: +34669448337</div>
</div>
</div>
</blockquote>
</div>
</div>
-- <br>
<div dir="ltr" class="gmail-m_2769825626893760435gmail-m_4388046060681410490gmail_signature">
<div dir="ltr">
<div>Regards,</div>
<div><br>
</div>
David Villasmil
<div>email: <a href="mailto:david.villasmil.work@gmail.com" target="_blank">david.villasmil.work@gmail.com</a></div>
<div>phone: +34669448337</div>
</div>
</div>
<br>
<fieldset class="gmail-m_2769825626893760435gmail-m_4388046060681410490mimeAttachmentHeader"></fieldset>
<pre class="gmail-m_2769825626893760435gmail-m_4388046060681410490moz-quote-pre">_______________________________________________
Kamailio (SER) - Users Mailing List
<a class="gmail-m_2769825626893760435gmail-m_4388046060681410490moz-txt-link-abbreviated" href="mailto:sr-users@lists.kamailio.org" target="_blank">sr-users@lists.kamailio.org</a>
<a class="gmail-m_2769825626893760435gmail-m_4388046060681410490moz-txt-link-freetext" href="https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users" target="_blank">https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users</a>
</pre>
</blockquote>
<pre class="gmail-m_2769825626893760435gmail-m_4388046060681410490moz-signature" cols="72">--
Daniel-Constantin Mierla -- <a class="gmail-m_2769825626893760435gmail-m_4388046060681410490moz-txt-link-abbreviated" href="http://www.asipto.com" target="_blank">www.asipto.com</a>
<a class="gmail-m_2769825626893760435gmail-m_4388046060681410490moz-txt-link-abbreviated" href="http://www.twitter.com/miconda" target="_blank">www.twitter.com/miconda</a> -- <a class="gmail-m_2769825626893760435gmail-m_4388046060681410490moz-txt-link-abbreviated" href="http://www.linkedin.com/in/miconda" target="_blank">www.linkedin.com/in/miconda</a></pre>
</div>
_______________________________________________<br>
Kamailio (SER) - Users Mailing List<br>
<a href="mailto:sr-users@lists.kamailio.org" target="_blank">sr-users@lists.kamailio.org</a><br>
<a href="https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users" rel="noreferrer" target="_blank">https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users</a><br>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail-m_2769825626893760435gmail_signature"><div dir="ltr"><div><div dir="ltr"><font face="'times new roman', serif">Regards</font><div><font face="'times new roman', serif"><br></font><div><pre cols="72"><font face="'times new roman', serif">M. Salman<br>VoIP Professional<br></font><br></pre></div></div></div></div></div></div>