[SR-Users] Kamailio Stop Processing Call

[Henning Westerholt]

Hello,

Ok – in this case probably “strace” or “gdb” should still work.

You can also consider to increase debug level, to see if you get more information as well.

Cheers,

Henning
--
Henning Westerholt – https://skalatan.de/blog/
Kamailio services – https://gilawa.com<https://gilawa.com/>



From: M Arqum CH <marqumch at gmail.com>
Sent: Sunday, May 23, 2021 10:23 AM
To: Henning Westerholt <hw at skalatan.de>
Subject: Re: [SR-Users] Kamailio Stop Processing Call

Hi Henning,

Unfortunately when process hang, it did't respond on kamcmd socket, it gives timeout too, so i have no clue what was happening at that moment.

Thanks

Arqum

On Fri, May 21, 2021 at 7:56 PM Henning Westerholt <hw at skalatan.de<mailto:hw at skalatan.de>> wrote:
Hello,

you need to somehow get more information from your kamailio that is possible to help you. 😉

If the server is stuck the next time, I would suggest to use e.g. “strace” or “kamcmd trap” to get more information what the processes are currently doing and where they hang.

Enable more verbose logging might also help.

Cheers,

Henning

--
Henning Westerholt – https://skalatan.de/blog/
Kamailio services – https://gilawa.com<https://gilawa.com/>

From: M Arqum CH <marqumch at gmail.com<mailto:marqumch at gmail.com>>
Sent: Friday, May 21, 2021 11:16 AM
To: Henning Westerholt <hw at skalatan.de<mailto:hw at skalatan.de>>
Cc: Kamailio (SER) - Users Mailing List <sr-users at lists.kamailio.org<mailto:sr-users at lists.kamailio.org>>
Subject: Re: [SR-Users] Kamailio Stop Processing Call

Hi Henning,

Thanks for reply and information regarding memory.

No error log in log file  regarding anything.

Prometheus scrapper just shows as shared unable to connect with kamailio socket.

Thanks

Arqum
On Fri, May 21, 2021, 12:42 PM Henning Westerholt <hw at skalatan.de<mailto:hw at skalatan.de>> wrote:
Hello,

just a quick note – 264 MB of private memory is a lot, as it is per process. Usually, you do not need that much.

Regarding the error/hang of Kamailio, do you see any errors in the log file, e.g., an out of memory error or similar?

Cheers,

Henning

--
Henning Westerholt – https://skalatan.de/blog/
Kamailio services – https://gilawa.com<https://gilawa.com/>

From: sr-users <sr-users-bounces at lists.kamailio.org<mailto:sr-users-bounces at lists.kamailio.org>> On Behalf Of M Arqum CH
Sent: Thursday, May 20, 2021 4:38 PM
To: Kamailio (SER) - Users Mailing List <sr-users at lists.kamailio.org<mailto:sr-users at lists.kamailio.org>>
Subject: [SR-Users] Kamailio Stop Processing Call

Hello Members,
I have kamailio setup backed with mysql db, using db_cluster module for spliting read and write..
Lisiting on three  different virtual ips are bind with same/single  udp port1 for processing  calls.
 One  static IP is bind with  udp port2  to for processing/response  to option messages send from  keepalived external script  after every few seconds and for DMQ messages from other kamailio node.

General configuation details:
children=16
fork=yes
listen=udp:VIP:SIP_PORT1
listen=udp:VIP2:SIP_PORT1
listen=udp:VIP3:SIP_PORT1
listen=udp:StaticIP3:SIP_PORT2
listen=tcp:SIP:RPC_PORT

SH_MEM=5012
PRIV_MEM=264

System Specification : 24 GB RAM, 12 CPUs

Sometimes kamailio  radomly stop processing the requests or give no response on  new requests recevied on virtual ips and as well it did't show any error in syslog.

To monitor the kamailio memory I used promethues (https://github.com/florentchauveau/kamailio_exporter)

The moment kamailio stop processing the requests it also stop responding to kamailio exporter and scrap request get timeout.  i.e error log on console
debian prometheus-kamailio-exporter[83440]: 2021/05/20 12:15:27 [error] cannot read header: read unix @->/tmp/kamailio__ctl: i/o timeout


After getting first timeout occurrence of  kamailio-exporter scrap request, kamailio processed sip messages for almost 30 seconds then completly stop replying to any message.  But it keep replying to option message arrive on different port from keepalived script.

Stats :
Active dialog: 75
Shared memory used: 150 MB

Kamailio Installed version :
kamailio -v
version: kamailio 5.4.1 (x86_64/linux) 2ca223
flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES, TLS_PTHREAD_MUTEX_SHARED
ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
id: 2ca223
compiled on 12:51:24 Sep 21 2020 with gcc 6.3.0


Is anyone else face similar issue  or idea why kamailio is showing such behaviour.?

Please guide me on what I should do to find and fix the issue.

Thanks
--
Regards
M Arqum


--
Regards
M Arqum
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.kamailio.org/pipermail/sr-users/attachments/20210524/2cce248a/attachment.htm>