[SR-Users] Kamailio Process CPU 100%

M. Salman msalman212 at gmail.com
Mon Aug 5 19:29:26 CEST 2019


Hi Daniel,

Thanks for your reply.

Yes, It is a similar issue as you shared. I have installed debug symbols
for libmysqlclient library, let's see what more info it can give.

TLS is not enabled, yet. Just to mention, MySQL instance is on RDS,
however, I have not seen such behavior before in other deployments.

location, dialog, presence is being used with MySQL.

@David, thanks for trying to help. Turning DB mode to in memory is not
going to work in my case scenarios. Thanks

Regards,
Salman
VoIP Professional

On Mon, Aug 5, 2019 at 7:58 PM Daniel-Constantin Mierla <miconda at gmail.com>
wrote:

> Hello,
>
> this seems similar to the issue reported in the tracker:
>
>   -
> https://github.com/kamailio/kamailio/issues/1895#issuecomment-500895303
>
> I was troubleshooting on that system in the past, but recently I couldn't
> get more there. Not sure why the mysql query is not timing out, on the net
> there were mentions that is cannot be ensured, it's more like a best try.
>
> You should install debug symbols for libmysqlclient library and see if you
> get when in the libmysqlclient code is getting stuck.
>
> Do you have tls enabled in kamailio?
>
> Cheers,
> Daniel
> On 05.08.19 17:03, David Villasmil wrote:
>
> At least try setting db_mode for the dialog module to 0 (zero)
>
> On Mon, 5 Aug 2019 at 16:02, David Villasmil <
> david.villasmil.work at gmail.com> wrote:
>
>> Have you tried taking out the dB support and see if it keeps happening?
>> The by talks a lot about MySQL... no bT expert here...
>>
>> On Mon, 5 Aug 2019 at 14:07, M. Salman <msalman212 at gmail.com> wrote:
>>
>>> Hi Team,
>>>
>>> Facing 100% Kamilio CPU issue, here are the details, any pointers/hints
>>> would help. Thanks
>>>
>>> *Scenario:*
>>> Random Kamailio CPU shoot, restart fixes it. Happens once or twice in a
>>> day.
>>> It's a pre-production server with around 30 extensions, not much load.
>>> For now, it has 2 CPUs and 4 G memory.
>>>
>>> *[Info]*
>>>
>>> uname -a
>>>
>>> Linux ip-X-aws #46-Ubuntu SMP Thu Jul 4 13:38:28 UTC 2019 x86_64 x86_64
>>> x86_64 GNU/Linux
>>>
>>> kamailio  -V
>>>
>>> version: kamailio 5.2.3 (x86_64/linux) c36229
>>>
>>> flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS,
>>> DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC,
>>> Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX,
>>> FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR,
>>> USE_DST_BLACKLIST, HAVE_RESOLV_RES
>>>
>>> ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144 MAX_URI_SIZE 1024,
>>> BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB
>>>
>>> poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
>>>
>>> id: c36229
>>>
>>> *(gdb) bt full*
>>> #0  0x00007f0069d97aab in ?? () from
>>> /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20
>>> No symbol table info available.
>>> #1  0x00007f0069d7d6bd in ?? () from
>>> /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20
>>> No symbol table info available.
>>> #2  0x00007f0069d657c7 in ?? () from
>>> /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20
>>> No symbol table info available.
>>> #3  0x00007f0069d3ebce in ?? () from
>>> /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20
>>> No symbol table info available.
>>> #4  0x00007f0069d3fc4c in ?? () from
>>> /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20
>>> No symbol table info available.
>>> #5  0x00007f0069d33edc in ?? () from
>>> /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20
>>> No symbol table info available.
>>> #6  0x00007f0069d34fa8 in ?? () from
>>> /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20
>>> No symbol table info available.
>>> #7  0x00007f0069d36d71 in mysql_real_query () from
>>> /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20
>>> No symbol table info available.
>>> #8  0x00007f006a31e4b4 in db_mysql_submit_query (h=0x7f006a6cdb20,
>>> s=0x7f0069af92d0 <sql_str>) at km_dbase.c:116
>>>         t = 1564528172
>>>         i = 0
>>>         code = 0
>>>         _func_ = "db_mysql_submit_query"
>>> #9  0x00007f00698e8fb3 in db_do_submit_query (h=0x7f006a6cdb20,
>>> query=0x7f0069af92d0 <sql_str>,
>>>     submit_query=0x7f006a31dd3c <db_mysql_submit_query>) at db_query.c:58
>>>         ret = 10
>>>         tvb = {tv_sec = 0, tv_usec = 0}
>>>         tve = {tv_sec = 0, tv_usec = 2077076196242373120}
>>>         tz = {tz_minuteswest = 0, tz_dsttime = 0}
>>>         tdiff = 0
>>>         _func_ = "db_do_submit_query"
>>> #10 0x00007f00698ec1dc in db_do_delete (h=0x7f006a6cdb20,
>>> k=0x7ffcdb440fa0, o=0x0, v=0x7ffcdb440fb0, _n=2,
>>>     val2str=0x7f006a340429 <db_mysql_val2str>,
>>> submit_query=0x7f006a31dd3c <db_mysql_submit_query>) at db_query.c:299
>>>         off = 68
>>>         ret = 36
>>> ---Type <return> to continue, or q <return> to quit---
>>>         _func_ = "db_do_delete"
>>> #11 0x00007f006a322f6a in db_mysql_delete (h=0x7f006a6cdb20,
>>> k=0x7ffcdb440fa0, o=0x0, v=0x7ffcdb440fb0, _n=2) at km_dbase.c:515
>>> No locals.
>>> #12 0x00007f0065b6da70 in remove_dialog_from_db (cell=0x7f005c5ed3d8) at
>>> dlg_db_handler.c:722
>>>         values = {{type = DB1_INT, nul = 0, free = 477963473, val =
>>> {int_val = 1460, ll_val = 140720308487604,
>>>               double_val = 6.9525070095907518e-310, time_val =
>>> 140720308487604,
>>>               string_val = 0x7ffc000005b4 <error: Cannot access memory
>>> at address 0x7ffc000005b4>, str_val = {
>>>                 s = 0x7ffc000005b4 <error: Cannot access memory at
>>> address 0x7ffc000005b4>, len = 0}, blob_val = {
>>>                 s = 0x7ffc000005b4 <error: Cannot access memory at
>>> address 0x7ffc000005b4>, len = 0}, bitmap_val = 1460, uint_val = 1460,
>>>               ull_val = 140720308487604}}, {type = DB1_INT, nul = 0,
>>> free = 0, val = {int_val = 8020, ll_val = 139637976735572,
>>>               double_val = 6.899032715982534e-310, time_val =
>>> 139637976735572,
>>>               string_val = 0x7f0000001f54 <error: Cannot access memory
>>> at address 0x7f0000001f54>, str_val = {
>>>                 s = 0x7f0000001f54 <error: Cannot access memory at
>>> address 0x7f0000001f54>, len = 0}, blob_val = {
>>>                 s = 0x7f0000001f54 <error: Cannot access memory at
>>> address 0x7f0000001f54>, len = 0}, bitmap_val = 8020, uint_val = 8020,
>>>               ull_val = 139637976735572}}}
>>>         match_keys = {0x7f0065dfc6c0 <h_entry_column>, 0x7f0065dfc6b0
>>> <h_id_column>}
>>>         vars_match_keys = {0x7f0065dfc7f0 <vars_h_entry_column>,
>>> 0x7f0065dfc7e0 <vars_h_id_column>}
>>>         _func_ = "remove_dialog_from_db"
>>> #13 0x00007f0065b49375 in destroy_dlg (dlg=0x7f005c5ed3d8) at
>>> dlg_hash.c:371
>>>         ret = 1
>>>         var = 0x7f0065b45442 <futex_get+40>
>>>         _func_ = "destroy_dlg"
>>> #14 0x00007f0065b50ca0 in dlg_unref_helper (dlg=0x7f005c5ed3d8, cnt=2,
>>> fname=0x7f0065be1eff "dlg_handlers.c", fline=1182) at dlg_hash.c:961
>>>         d_entry = 0x7f005c45b4c0
>>>         _func_ = "dlg_unref_helper"
>>> #15 0x00007f0065b7f708 in unref_dlg_from_cb (t=0x7f005c5b5dc0,
>>> type=131072, param=0x7ffcdb4412a0) at dlg_handlers.c:1182
>>>         dlg = 0x7f005c5ed3d8
>>>         iuid = 0x7f005c568c38
>>> #16 0x00007f0068295b05 in run_trans_callbacks_internal
>>> (cb_lst=0x7f005c5b5e38, type=131072, trans=0x7f005c5b5dc0,
>>> params=0x7ffcdb4412a0)
>>>     at t_hooks.c:260
>>>         cbp = 0x7f005c606118
>>>         backup_from = 0x5569cc3da8b0 <def_list+16>
>>>         backup_to = 0x5569cc3da8b8 <def_list+24>
>>> ---Type <return> to continue, or q <return> to quit---
>>>         backup_dom_from = 0x5569cc3da8c0 <def_list+32>
>>>         backup_dom_to = 0x5569cc3da8c8 <def_list+40>
>>>         backup_uri_from = 0x5569cc3da8a0 <def_list>
>>>         backup_uri_to = 0x5569cc3da8a8 <def_list+8>
>>>         backup_xavps = 0x5569cc3da2c0 <_xavp_list_head>
>>>         _func_ = "run_trans_callbacks_internal"
>>> #17 0x00007f0068295c2c in run_trans_callbacks (type=131072,
>>> trans=0x7f005c5b5dc0, req=0x0, rpl=0x0, code=0) at t_hooks.c:287
>>>         params = {req = 0x0, rpl = 0x0, param = 0x7f005c606128, code =
>>> 0, flags = 0, branch = 0, t_rbuf = 0x0, dst = 0x0, send_buf = {
>>>             s = 0x0, len = 0}}
>>> #18 0x00007f00681cbf45 in free_cell_helper (dead_cell=0x7f005c5b5dc0,
>>> silent=0, fname=0x7f00682de882 "timer.c", fline=689) at h_table.c:166
>>>         b = 0x7f005c5b5dc0 ""
>>>         i = 1
>>>         rpl = 0x7ffcdb4413a0
>>>         tt = 0x0
>>>         foo = 0x7ffcdb441390
>>>         cbs = 0x7f005c378700
>>>         cbs_tmp = 0x1681ca80f
>>>         _func_ = "free_cell_helper"
>>> #19 0x00007f0068283bf6 in wait_handler (ti=855479619,
>>> wait_tl=0x7f005c5b5e48, data=0x7f005c5b5dc0) at timer.c:689
>>>         p_cell = 0x7f005c5b5dc0
>>>         ret = 1
>>>         unlinked = 0
>>>         rcount = 1
>>>         _func_ = "wait_handler"
>>> #20 0x00005569cbdb0661 in timer_list_expire (t=855479619,
>>> h=0x7f005c238af0, slow_l=0x7f005c23ae08, slow_mark=5646) at core/timer.c:874
>>>         tl = 0x7f005c5b5e48
>>>         ret = 0
>>> #21 0x00005569cbdb0b09 in timer_handler () at core/timer.c:939
>>>         saved_ticks = 855479619
>>>         run_slow_timer = 0
>>>         i = 526
>>>         _func_ = "timer_handler"
>>> #22 0x00005569cbdb0fd2 in timer_main () at core/timer.c:978
>>> ---Type <return> to continue, or q <return> to quit---
>>> No locals.
>>> #23 0x00005569cbcf1382 in main_loop () at main.c:1727
>>>         i = 8
>>>         pid = 0
>>>         si = 0x0
>>>         si_desc = "udp receiver child=7 sock=172.31.2.33:6000 (
>>> 13.54.252.29:6000)\000\034\004\000\000\000\000\000\000\000\321$}\034\000\000\000\000P0\316\313iU\000\000
>>> \031D\333\374\177", '\000' <repeats 18 times>,
>>> "`\026D\333\374\177\000\000\360S\361\313iU\000"
>>>         nrprocs = 8
>>>         woneinit = 1
>>>         _func_ = "main_loop"
>>> #24 0x00005569cbcf8a7b in main (argc=13, argv=0x7ffcdb441928) at
>>> main.c:2696
>>>         cfg_stream = 0x5569cd7c7260
>>>         c = -1
>>>         r = 0
>>>         tmp = 0x7ffcdb441f39 ""
>>>         tmp_len = -616294480
>>>         port = 32764
>>>         proto = 3
>>>         options = 0x5569cc074190
>>> ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:x:X:Y:"
>>>         ret = -1
>>>         seed = 3114863458
>>>         rfd = 4
>>>         debug_save = 0
>>>         debug_flag = 0
>>>         dont_fork_cnt = 0
>>>         n_lst = 0x7f006af01787
>>>         p = 0x9c0 <error: Cannot access memory at address 0x9c0>
>>>         st = {st_dev = 23, st_ino = 456, st_nlink = 2, st_mode = 16877,
>>> st_uid = 0, st_gid = 0, __pad0 = 0, st_rdev = 0, st_size = 60,
>>>           st_blksize = 4096, st_blocks = 0, st_atim = {tv_sec =
>>> 1564229247, tv_nsec = 809557407}, st_mtim = {tv_sec = 1564229229,
>>>             tv_nsec = 217250917}, st_ctim = {tv_sec = 1564229279,
>>> tv_nsec = 666090748}, __glibc_reserved = {0, 0, 0}}
>>>         _func_ = "main"
>>> (gdb)
>>>
>>>
>>> --
>>> Regards
>>>
>>> M. Salman
>>>
>>> _______________________________________________
>>> Kamailio (SER) - Users Mailing List
>>> sr-users at lists.kamailio.org
>>> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
>>>
>> --
>> Regards,
>>
>> David Villasmil
>> email: david.villasmil.work at gmail.com
>> phone: +34669448337
>>
> --
> Regards,
>
> David Villasmil
> email: david.villasmil.work at gmail.com
> phone: +34669448337
>
> _______________________________________________
> Kamailio (SER) - Users Mailing Listsr-users at lists.kamailio.orghttps://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
>
> --
> Daniel-Constantin Mierla -- www.asipto.comwww.twitter.com/miconda -- www.linkedin.com/in/miconda
>
> _______________________________________________
> Kamailio (SER) - Users Mailing List
> sr-users at lists.kamailio.org
> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
>


-- 
Regards

M. Salman
VoIP Professional
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.kamailio.org/pipermail/sr-users/attachments/20190805/ad964acb/attachment.html>


More information about the sr-users mailing list