<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
span.E-MailFormatvorlage18
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:70.85pt 70.85pt 2.0cm 70.85pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="DE" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Hello,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="mso-fareast-language:EN-US">you need to somehow get more information from your kamailio that is possible to help you.
</span><span lang="EN-GB" style="font-family:"Segoe UI Emoji",sans-serif;mso-fareast-language:EN-US">😉</span><span lang="EN-GB" style="mso-fareast-language:EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="mso-fareast-language:EN-US">If the server is stuck the next time, I would suggest to use e.g. “strace” or “kamcmd trap” to get more information what the processes are currently doing and where they hang.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="mso-fareast-language:EN-US">Enable more verbose logging might also help.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="mso-fareast-language:EN-US">Cheers,<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="mso-fareast-language:EN-US">Henning<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="mso-fareast-language:EN-US">-- <o:p>
</o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="mso-fareast-language:EN-US">Henning Westerholt –
</span><span style="mso-fareast-language:EN-US"><a href="https://skalatan.de/blog/"><span lang="EN-GB" style="color:#0563C1">https://skalatan.de/blog/</span></a></span><span lang="EN-GB" style="mso-fareast-language:EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="mso-fareast-language:EN-US">Kamailio services –
</span><span style="mso-fareast-language:EN-US"><a href="https://gilawa.com/"><span lang="EN-GB" style="color:#0563C1">https://gilawa.com</span></a></span><span style="mso-fareast-language:EN-US">
<span lang="EN-GB"><o:p></o:p></span></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal" style="margin-left:35.4pt"><b>From:</b> M Arqum CH <marqumch@gmail.com>
<br>
<b>Sent:</b> Friday, May 21, 2021 11:16 AM<br>
<b>To:</b> Henning Westerholt <hw@skalatan.de><br>
<b>Cc:</b> Kamailio (SER) - Users Mailing List <sr-users@lists.kamailio.org><br>
<b>Subject:</b> Re: [SR-Users] Kamailio Stop Processing Call<o:p></o:p></p>
</div>
<p class="MsoNormal" style="margin-left:35.4pt"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">Hi Henning,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">Thanks for reply and information regarding memory. <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">No error log in log file regarding anything.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">Prometheus scrapper just shows as shared unable to connect with kamailio socket.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">Thanks<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:0cm;margin-right:0cm;margin-bottom:12.0pt;margin-left:35.4pt">
Arqum<o:p></o:p></p>
<div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">On Fri, May 21, 2021, 12:42 PM Henning Westerholt <<a href="mailto:hw@skalatan.de">hw@skalatan.de</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:35.4pt">
Hello,<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:35.4pt">
<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:35.4pt">
<span lang="EN-GB">just a quick note – 264 MB of private memory is a lot, as it is per process. Usually, you do not need that much.</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:35.4pt">
<span lang="EN-GB"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:35.4pt">
<span lang="EN-GB">Regarding the error/hang of Kamailio, do you see any errors in the log file, e.g., an out of memory error or similar?</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:35.4pt">
<span lang="EN-GB"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:35.4pt">
<span lang="EN-GB">Cheers,</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:35.4pt">
<span lang="EN-GB"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:35.4pt">
<span lang="EN-GB">Henning</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:35.4pt">
<span lang="EN-GB"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:35.4pt">
<span lang="EN-GB">-- </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:35.4pt">
<span lang="EN-GB">Henning Westerholt – </span><a href="https://skalatan.de/blog/" target="_blank"><span lang="EN-GB" style="color:#0563C1">https://skalatan.de/blog/</span></a><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:35.4pt">
<span lang="EN-GB">Kamailio services – </span><a href="https://gilawa.com/" target="_blank"><span lang="EN-GB" style="color:#0563C1">https://gilawa.com</span></a>
<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:35.4pt">
<span lang="EN-GB"> </span><o:p></o:p></p>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:70.8pt">
<b>From:</b> sr-users <<a href="mailto:sr-users-bounces@lists.kamailio.org" target="_blank">sr-users-bounces@lists.kamailio.org</a>>
<b>On Behalf Of </b>M Arqum CH<br>
<b>Sent:</b> Thursday, May 20, 2021 4:38 PM<br>
<b>To:</b> Kamailio (SER) - Users Mailing List <<a href="mailto:sr-users@lists.kamailio.org" target="_blank">sr-users@lists.kamailio.org</a>><br>
<b>Subject:</b> [SR-Users] Kamailio Stop Processing Call<o:p></o:p></p>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:70.8pt">
<o:p></o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:70.8pt">
Hello Members, <o:p></o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:70.8pt">
I have kamailio setup backed with mysql db, using db_cluster module for spliting read and write..<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:70.8pt">
Lisiting on three different virtual ips are bind with same/single udp port<b>1</b> for processing calls.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:70.8pt">
One static IP is bind with udp port<b>2</b> to for processing/response to option messages send from keepalived external script after every few seconds and for DMQ messages from other kamailio node. <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:70.8pt">
<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:70.8pt">
General configuation details:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:70.8pt">
children=16<br>
fork=yes<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:70.8pt">
listen=udp:VIP:SIP_PORT1<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:70.8pt">
listen=udp:VIP2:SIP_PORT1<br>
listen=udp:VIP3:SIP_PORT1<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:70.8pt">
listen=udp:StaticIP3:SIP_PORT2 <o:p></o:p></p>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:70.8pt">
listen=tcp:SIP:RPC_PORT <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:70.8pt">
<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:70.8pt">
SH_MEM=5012<br>
PRIV_MEM=264<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:70.8pt">
<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:70.8pt">
System Specification : 24 GB RAM, 12 CPUs<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:70.8pt">
<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:70.8pt">
Sometimes kamailio radomly stop processing the requests or give no response on new requests recevied on virtual ips and as well it did't show any error in syslog. <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:70.8pt">
<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:70.8pt">
To monitor the kamailio memory I used promethues (<a href="https://github.com/florentchauveau/kamailio_exporter" target="_blank">https://github.com/florentchauveau/kamailio_exporter</a>)<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:70.8pt">
<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:70.8pt">
The moment kamailio stop processing the requests it also stop responding to kamailio exporter and scrap request get timeout. i.e error log on console <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:70.8pt">
<span style="color:black;background:yellow">debian prometheus-kamailio-exporter[83440]: 2021/05/20 12:15:27 [error] cannot read header: read unix @->/tmp/kamailio__ctl: i/o timeout</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:70.8pt">
<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:70.8pt">
<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:70.8pt">
After getting first timeout occurrence of kamailio-exporter scrap request, kamailio processed sip messages for almost 30 seconds then completly stop replying to any message. But it keep replying to option message arrive on different port from keepalived script. <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:70.8pt">
<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:70.8pt">
Stats :<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:70.8pt">
Active dialog: 75 <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:70.8pt">
Shared memory used: 150 MB<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:70.8pt">
<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:70.8pt">
Kamailio Installed version :<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:70.8pt">
kamailio -v<br>
version: kamailio 5.4.1 (x86_64/linux) 2ca223<br>
flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST,
HAVE_RESOLV_RES, TLS_PTHREAD_MUTEX_SHARED<br>
ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB<br>
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.<br>
id: 2ca223<br>
compiled on 12:51:24 Sep 21 2020 with gcc 6.3.0<o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:70.8pt">
<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:70.8pt">
<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:70.8pt">
Is anyone else face similar issue or idea why kamailio is showing such behaviour.?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:70.8pt">
<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:70.8pt">
Please guide me on what I should do to find and fix the issue. <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:70.8pt">
<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:70.8pt">
Thanks <o:p></o:p></p>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:70.8pt">
-- <o:p></o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:70.8pt">
Regards<o:p></o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:70.8pt">
M Arqum<o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</body>
</html>