Hello,

thanks for the report. Could you maybe point out where the increase happens in the shared memory statistics, e.g. by doing some more calls and executing the stats command again?

If you don't get a reply in time on the open-source support list, you can also open an issue on our github tracker, to have it documented.

Regarding the mentioned production system issues, if you need defined response times on problems, maybe you might be interested in looking into the companies providing support services in our directory: https://www.kamailio.org/w/business-directory/

Cheers,

Henning

-- Henning Westerholt - https://skalatan.de/blog/ Kamailio services - https://gilawa.comhttps://gilawa.com/

From: Vanderlei Torres Batistela via sr-users sr-users@lists.kamailio.org Sent: Dienstag, 9. Juli 2024 19:43 To: sr-users@lists.kamailio.org Cc: Vanderlei Torres Batistela v.batistela@hotmail.com Subject: [SR-Users] Re: Possible Kamailio memmory leak - version 5.8.2

Hello,

Could someone help me please?

I have a production system stopped and waiting to be fixed.

My production script is more complex, but this issue also occurs with simple scripts like the one described previously.

Help on this is much appreciated

Regards,

Vanderlei ________________________________ De: Vanderlei Torres Batistela via sr-users <sr-users@lists.kamailio.orgmailto:sr-users@lists.kamailio.org> Enviado: terça-feira, 2 de julho de 2024 09:24 Para: sr-users@lists.kamailio.orgmailto:sr-users@lists.kamailio.org <sr-users@lists.kamailio.orgmailto:sr-users@lists.kamailio.org> Cc: Vanderlei Torres Batistela <v.batistela@hotmail.commailto:v.batistela@hotmail.com> Assunto: [SR-Users] Possible Kamailio memory leak - version 5.8.2

Hello all,

I am using a very simple scritp with corex module enabled . Normal calls works fine, but a scenario where INVITE does not receive response and the call is released with SIP 408 (request timeout) leads to memory leak. I verified it making a lot of calls with this scenario and printing the shared memory (using kamcmd core.shmmem) . Its ocurred with modparam("corex", "nio_intercept", 1). If I modify to modparam("corex", "nio_intercept", 0) the memory leak does not ocurr anymore. The problem also occurs in another scenario where the SIP Cancel message is not responded to.

My script has this event route

# event route event_route[network:msg] {

xlog("L_INFO", "Dentro do Event_Route \n"); if (is_incoming()) { xlog("L_INFO", "Received message '$mb' \n"); $avp(msg) = $mb; xinfo("avp(msg) = $avp(msg)"); } else { xlog("L_INFO", "Sending message '$mb' \n"); $avp(msg) = $mb; }; }

Test done wtih 4000 calls:

[root@labcom070 kamailio]# kamctl stats shmem { "jsonrpc": "2.0", "result": [ "shmem:fragments = 900", "shmem:free_size = 6180368", "shmem:max_used_size = 66980272", "shmem:real_used_size = 60928496", "shmem:total_size = 67108864", "shmem:used_size = 54884944" ], "id": 393115 }

[root@labcom070 kamailio]# kamcmd mod.stats all shm Module: core { create_avp(178): 52257504 counters_prefork_init(211): 53760 cfg_clone_str(132): 112 cfg_shmize(221): 832 main_loop(1381): 16 init_pt(104): 16 init_pt(103): 16 init_pt(102): 6224 cfg_register_ctx(47): 96 init_tcp(5197): 8192 init_tcp(5191): 32768 init_tcp(5182): 16 init_tcp(5175): 16 init_tcp(5167): 16 init_tcp(5161): 16 init_tcp(5149): 16 init_avps(92): 16 init_avps(91): 16 init_dst_blocklist(435): 16384 init_dst_blocklist(427): 16 timer_alloc(494): 96 init_dns_cache(368): 16 init_dns_cache(359): 16384 init_dns_cache(351): 16 init_dns_cache(343): 16 init_timer(264): 16 init_timer(263): 16384 init_timer(262): 16 init_timer(261): 16 init_timer(250): 16 init_timer(218): 16 init_timer(207): 278544 init_timer(206): 16 init_timer(194): 16 cfg_child_cb_new(832): 64 sr_cfg_init(371): 16 sr_cfg_init(364): 16 sr_cfg_init(356): 16 sr_cfg_init(344): 16 sr_cfg_init(332): 16 ksr_shutdown_phase_init(123): 16 rpc_hash_add(118): 16 qm_shm_lock_init(1463): 16 Total: 52687808 }

I attached my kamailio script used and a log with one call using the command "kamcmd corex.shm_summary" with modparam("corex", "nio_intercept", 1) and modparam("corex", "nio_intercept", 0)

Regards,

Vanderlei

Hello Daniel,

I tried with git master branch and it works only for a simple event-route script where avp(msg) is not modified as below:

# event route event_route[network:msg] {

xlog("L_INFO", " Event_Route \n"); if (is_incoming()) { xlog("L_INFO", "Received message '$mb' \n"); $avp(msg) = $mb; xinfo("avp(msg) = $avp(msg)"); } else { xlog("L_INFO", "Sending message '$mb' \n"); $avp(msg) = $mb; xinfo("avp(msg) = $avp(msg)"); } }

Then I tried it using this script as an example to set the message to send out:

# event route event_route[network:msg] {

xlog("L_INFO", " Event_Route \n"); if (is_incoming()) { xlog("L_INFO", "Received message '$mb' \n"); $avp(msg) = $mb; xinfo("avp(msg) = $avp(msg)"); } else { xlog("L_INFO", "Sending message '$mb' \n"); $avp(msg) = $mb; xinfo("var(x)= $var(x)"); $var(x)="a=3gOoBTC" + "\r" + "\n"; $avp(msg) = $(avp(msg){s.replace,$var(x),}); avp_print(); } }

Examining the attached log I saw that for each INVITE retransmission (handled by another processor-pid) the avp (msg) is not freed and is accumulated. So I modified the script to delete the avp(msg) before using it and that fixed the issue.

2) Delete avp(msg) before using it

# event route event_route[network:msg] {

xlog("L_INFO", " Event_Route \n"); if (is_incoming()) { xlog("L_INFO", "Received message '$mb' \n"); $avp(msg) = $mb; xinfo("avp(msg) = $avp(msg)"); } else { xlog("L_INFO", "Sending message '$mb' \n"); $(avp(msg)[*]) = $null; <---------------------------------------------- $avp(msg) = $mb; xinfo("var(x)= $var(x)"); $var(x)="a=3gOoBTC" + "\r" + "\n"; $avp(msg) = $(avp(msg){s.replace,$var(x),}); avp_print(); } }

This workaround also fixed the issue in the original 5.8.2 version.

Regards,

Vanderlei

________________________________ De: Daniel-Constantin Mierla miconda@gmail.com Enviado: quarta-feira, 10 de julho de 2024 07:05 Para: Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org Cc: Vanderlei Torres Batistela v.batistela@hotmail.com Assunto: Re: [SR-Users] Possible Kamailio memmory leak - version 5.8.2

Hello,

I couldn't spot any error message in the log file that you attached, have you taken it when you encountered the problem?

Anyhow, I am not familiar with the implementation of the event_route[network:msg], but the shm info indicates the leak to be related to avp, so I pushed a patch to destroy the nio_msg_avp avp after event route execution. Can you try with git master branch and see if works ok?

Cheers, Daniel

On 02.07.24 16:24, Vanderlei Torres Batistela via sr-users wrote: Hello all,

I am using a very simple scritp with corex module enabled . Normal calls works fine, but a scenario where INVITE does not receive response and the call is released with SIP 408 (request timeout) leads to memmory leak. I verified it making a lot of calls with this scenario and printing the shared memory (using kamcmd core.shmmem) . Its ocurred with modparam("corex", "nio_intercept", 1). If I modify to modparam("corex", "nio_intercept", 0) the memmory leak does not ocurr anymore. The problem also occurs in another scenario where the SIP Cancel message is not responded to.

My script has this event route

# event route event_route[network:msg] {

xlog("L_INFO", "Dentro do Event_Route \n"); if (is_incoming()) { xlog("L_INFO", "Received message '$mb' \n"); $avp(msg) = $mb; xinfo("avp(msg) = $avp(msg)"); } else { xlog("L_INFO", "Sending message '$mb' \n"); $avp(msg) = $mb; }; }

Test done wtih 4000 calls:

[root@labcom070 kamailio]# kamctl stats shmem { "jsonrpc": "2.0", "result": [ "shmem:fragments = 900", "shmem:free_size = 6180368", "shmem:max_used_size = 66980272", "shmem:real_used_size = 60928496", "shmem:total_size = 67108864", "shmem:used_size = 54884944" ], "id": 393115 }

[root@labcom070 kamailio]# kamcmd mod.stats all shm Module: core { create_avp(178): 52257504 counters_prefork_init(211): 53760 cfg_clone_str(132): 112 cfg_shmize(221): 832 main_loop(1381): 16 init_pt(104): 16 init_pt(103): 16 init_pt(102): 6224 cfg_register_ctx(47): 96 init_tcp(5197): 8192 init_tcp(5191): 32768 init_tcp(5182): 16 init_tcp(5175): 16 init_tcp(5167): 16 init_tcp(5161): 16 init_tcp(5149): 16 init_avps(92): 16 init_avps(91): 16 init_dst_blocklist(435): 16384 init_dst_blocklist(427): 16 timer_alloc(494): 96 init_dns_cache(368): 16 init_dns_cache(359): 16384 init_dns_cache(351): 16 init_dns_cache(343): 16 init_timer(264): 16 init_timer(263): 16384 init_timer(262): 16 init_timer(261): 16 init_timer(250): 16 init_timer(218): 16 init_timer(207): 278544 init_timer(206): 16 init_timer(194): 16 cfg_child_cb_new(832): 64 sr_cfg_init(371): 16 sr_cfg_init(364): 16 sr_cfg_init(356): 16 sr_cfg_init(344): 16 sr_cfg_init(332): 16 ksr_shutdown_phase_init(123): 16 rpc_hash_add(118): 16 qm_shm_lock_init(1463): 16 Total: 52687808 }

I attached my kamailio script used and a log with one call using the command "kamcmd corex.shm_summary" with modparam("corex", "nio_intercept", 1) and modparam("corex", "nio_intercept", 0)

Regards,

Vanderlei

__________________________________________________________ Kamailio - Users Mailing List - Non Commercial Discussions To unsubscribe send an email to sr-users-leave@lists.kamailio.orgmailto:sr-users-leave@lists.kamailio.org Important: keep the mailing list in the recipients, do not reply only to the sender! Edit mailing list options or unsubscribe:

-- Daniel-Constantin Mierla (@ asipto.com) twitter.com/miconda -- linkedin.com/in/miconda Kamailio Consultancy, Training and Development Services -- asipto.com

Hello Daniel,

Regarding your question, I think only the avp(msg) should be destroyed as other avp's can be used elsewhere in the script.

I tried the lastest git master and when restarting kamailio I got a segmentation fault:

Jul 18 21:17:16 labcom070 systemd[1]: Started Kamailio - the Open Source SIP Server. Jul 18 21:17:17 labcom070 kernel: kamailio[2285]: segfault at 0 ip 00000000007eb66a sp 00007ffd7f80cc00 error 6 in kamailio[400000+533000] Jul 18 21:17:17 labcom070 kernel: Code: 41 56 41 55 41 54 53 48 81 ec c8 00 00 00 48 89 bd 48 ff ff ff 48 89 b5 40 ff ff ff 48 89 95 38 ff ff ff 48 8b 85 48 ff ff ff <48> c7 00 00 00 00 00 48 8b 85 48 ff ff ff c7 40 08 00 00 00 00 48 Jul 18 21:17:17 labcom 070 systemd[1]: Started Process Core Dump (PID 2286/UID 0). Jul 18 21:17:18 labcom070 systemd-coredump[2287]: Resource limits disable core dumping for process 2285 (kamailio). Jul 18 21:17:18 labcom070 systemd-coredump[2287]: Process 2285 (kamailio) of user 993 dumped core. Jul 18 21:17:18 labcom070 systemd[1]: kamailio.service: Main process exited, code=dumped, status=11/SEGV Jul 18 21:17:18 labcom070 systemd[1]: kamailio.service: Failed with result 'core-dump'. Jul 18 21:17:18 labcom070 systemd[1]: systemd-coredump@6-2286-0.service: Succeeded. Jul 18 21:17:18 labcom070 systemd[1]: kamailio.service: Service RestartSec=100ms expired, scheduling restart. Jul 18 21:17:18 labcom070 systemd[1]: kamailio.service: Scheduled restart job, restart counter is at 1. Jul 18 21:17:18 labcom070 systemd[1]: Stopped Kamailio - the Open Source SIP Server. Jul 18 21:17:18 labcom070 kernel: kamailio[2298]: segfault at 0 ip 00000000007eb66a sp 00007ffcd101c240 error 6 in kamailio[400000+533000] Jul 18 21:17:18 labcom070 kernel: Code: 41 56 41 55 41 54 53 48 81 ec c8 00 00 00 48 89 bd 48 ff ff ff 48 89 b5 40 ff ff ff 48 89 95 38 ff ff ff 48 8b 85 48 ff ff ff <48> c7 00 00 00 00 00 48 8b 85 48 ff ff ff c7 40 08 00 00 00 00 48

Regards,

Vanderlei ________________________________ De: Daniel-Constantin Mierla miconda@gmail.com Enviado: quinta-feira, 18 de julho de 2024 02:51 Para: Vanderlei Torres Batistela v.batistela@hotmail.com; Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org Assunto: Re: [SR-Users] Possible Kamailio memmory leak - version 5.8.2

Hello,

indeed the previous commit was destroying a single avp, I pushed a new one to destroy all of them with that name. I am wondering if all avps, no matter the name should be destroyed.

Anyhow, could you fetch the latest git master version and try your scenario again?

Cheers, Daniel

On 16.07.24 16:11, Vanderlei Torres Batistela wrote: Hello Daniel,

I tried with git master branch and it works only for a simple event-route script where avp(msg) is not modified as below:

# event route event_route[network:msg] {

xlog("L_INFO", " Event_Route \n"); if (is_incoming()) { xlog("L_INFO", "Received message '$mb' \n"); $avp(msg) = $mb; xinfo("avp(msg) = $avp(msg)"); } else { xlog("L_INFO", "Sending message '$mb' \n"); $avp(msg) = $mb; xinfo("avp(msg) = $avp(msg)"); } }

Then I tried it using this script as an example to set the message to send out:

# event route event_route[network:msg] {

xlog("L_INFO", " Event_Route \n"); if (is_incoming()) { xlog("L_INFO", "Received message '$mb' \n"); $avp(msg) = $mb; xinfo("avp(msg) = $avp(msg)"); } else { xlog("L_INFO", "Sending message '$mb' \n"); $avp(msg) = $mb; xinfo("var(x)= $var(x)"); $var(x)="a=3gOoBTC" + "\r" + "\n"; $avp(msg) = $(avp(msg){s.replace,$var(x),}); avp_print(); } }

Examining the attached log I saw that for each INVITE retransmission (handled by another processor-pid) the avp (msg) is not freed and is accumulated. So I modified the script to delete the avp(msg) before using it and that fixed the issue.

2) Delete avp(msg) before using it

# event route event_route[network:msg] {

xlog("L_INFO", " Event_Route \n"); if (is_incoming()) { xlog("L_INFO", "Received message '$mb' \n"); $avp(msg) = $mb; xinfo("avp(msg) = $avp(msg)"); } else { xlog("L_INFO", "Sending message '$mb' \n"); $(avp(msg)[*]) = $null; <---------------------------------------------- $avp(msg) = $mb; xinfo("var(x)= $var(x)"); $var(x)="a=3gOoBTC" + "\r" + "\n"; $avp(msg) = $(avp(msg){s.replace,$var(x),}); avp_print(); } }

This workaround also fixed the issue in the original 5.8.2 version.

Regards,

Vanderlei

________________________________ De: Daniel-Constantin Mierla miconda@gmail.commailto:miconda@gmail.com Enviado: quarta-feira, 10 de julho de 2024 07:05 Para: Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.orgmailto:sr-users@lists.kamailio.org Cc: Vanderlei Torres Batistela v.batistela@hotmail.commailto:v.batistela@hotmail.com Assunto: Re: [SR-Users] Possible Kamailio memmory leak - version 5.8.2

Hello,

I couldn't spot any error message in the log file that you attached, have you taken it when you encountered the problem?

Anyhow, I am not familiar with the implementation of the event_route[network:msg], but the shm info indicates the leak to be related to avp, so I pushed a patch to destroy the nio_msg_avp avp after event route execution. Can you try with git master branch and see if works ok?

Cheers, Daniel

On 02.07.24 16:24, Vanderlei Torres Batistela via sr-users wrote: Hello all,

I am using a very simple scritp with corex module enabled . Normal calls works fine, but a scenario where INVITE does not receive response and the call is released with SIP 408 (request timeout) leads to memmory leak. I verified it making a lot of calls with this scenario and printing the shared memory (using kamcmd core.shmmem) . Its ocurred with modparam("corex", "nio_intercept", 1). If I modify to modparam("corex", "nio_intercept", 0) the memmory leak does not ocurr anymore. The problem also occurs in another scenario where the SIP Cancel message is not responded to.

My script has this event route

# event route event_route[network:msg] {

xlog("L_INFO", "Dentro do Event_Route \n"); if (is_incoming()) { xlog("L_INFO", "Received message '$mb' \n"); $avp(msg) = $mb; xinfo("avp(msg) = $avp(msg)"); } else { xlog("L_INFO", "Sending message '$mb' \n"); $avp(msg) = $mb; }; }

Test done wtih 4000 calls:

[root@labcom070 kamailio]# kamctl stats shmem { "jsonrpc": "2.0", "result": [ "shmem:fragments = 900", "shmem:free_size = 6180368", "shmem:max_used_size = 66980272", "shmem:real_used_size = 60928496", "shmem:total_size = 67108864", "shmem:used_size = 54884944" ], "id": 393115 }

[root@labcom070 kamailio]# kamcmd mod.stats all shm Module: core { create_avp(178): 52257504 counters_prefork_init(211): 53760 cfg_clone_str(132): 112 cfg_shmize(221): 832 main_loop(1381): 16 init_pt(104): 16 init_pt(103): 16 init_pt(102): 6224 cfg_register_ctx(47): 96 init_tcp(5197): 8192 init_tcp(5191): 32768 init_tcp(5182): 16 init_tcp(5175): 16 init_tcp(5167): 16 init_tcp(5161): 16 init_tcp(5149): 16 init_avps(92): 16 init_avps(91): 16 init_dst_blocklist(435): 16384 init_dst_blocklist(427): 16 timer_alloc(494): 96 init_dns_cache(368): 16 init_dns_cache(359): 16384 init_dns_cache(351): 16 init_dns_cache(343): 16 init_timer(264): 16 init_timer(263): 16384 init_timer(262): 16 init_timer(261): 16 init_timer(250): 16 init_timer(218): 16 init_timer(207): 278544 init_timer(206): 16 init_timer(194): 16 cfg_child_cb_new(832): 64 sr_cfg_init(371): 16 sr_cfg_init(364): 16 sr_cfg_init(356): 16 sr_cfg_init(344): 16 sr_cfg_init(332): 16 ksr_shutdown_phase_init(123): 16 rpc_hash_add(118): 16 qm_shm_lock_init(1463): 16 Total: 52687808 }

I attached my kamailio script used and a log with one call using the command "kamcmd corex.shm_summary" with modparam("corex", "nio_intercept", 1) and modparam("corex", "nio_intercept", 0)

Regards,

Vanderlei

__________________________________________________________ Kamailio - Users Mailing List - Non Commercial Discussions To unsubscribe send an email to sr-users-leave@lists.kamailio.orgmailto:sr-users-leave@lists.kamailio.org Important: keep the mailing list in the recipients, do not reply only to the sender! Edit mailing list options or unsubscribe:

-- Daniel-Constantin Mierla (@ asipto.com) twitter.com/miconda -- linkedin.com/in/miconda Kamailio Consultancy, Training and Development Services -- asipto.com

-- Daniel-Constantin Mierla (@ asipto.com) twitter.com/miconda -- linkedin.com/in/miconda Kamailio Consultancy, Training and Development Services -- asipto.com