Hi,
I have run in to an issue when sending diameter requests using the ims_diameter_server module. Shared memory keeps increasing and are not being freed until restart of kamailio. This leads to all shared memory is being used and no more are able to be allocated.
This only happens when the request is sent synchronously. When sent asynchronously shared memory is being freed as far as I can see.
have anyone ran into this issue and found a remedy or can give some pointers how to proceed?
Following functions have been used for sending requests.
diameter_request([peer], appid, commandcode, message) diameter_request_async([peer], appid, commandcode, message)
route[CHECK_LOCATION] { diameter_request("hss.ims.test.local", "16777217", "306", "[ " + "{ "avpCode":260, "vendorId":0, "Flags":64, "list":[ { "avpCode":266, "vendorId":0, "Flags":64, "int32":10415 }, { "avpCode":258, "vendorId":0, "Flags":64, "int32":16777217 } ] }, " + "{ "avpCode":277, "vendorId":0, "Flags":64, "int32":1 }, " + "{ "avpCode":283, "vendorId":0, "Flags":64, "string":"ims.test.local"}, " + "{ "avpCode":706, "vendorId":10415, "Flags":192, "int32":0 }, " + "{ "avpCode":700, "vendorId":10415, "Flags":192, "list": [ { "avpCode":601, "vendorId":10415, "Flags":192, "string":"tel:" + $fU + "" }]}, " + "{ "avpCode":707, "vendorId":10415, "Flags":192, "int32":0 }, " + "{ "avpCode":703, "vendorId":10415, "Flags":192, "int32":14 } ]"); xlog("L_INFO", "Sent Diameter request"); }
Shared memory after restart
kamcmd mod.stats cdp shm Module: cdp { save_peer_applications(779): 144 make_send_pipe(126): 320 dp_add_pid(92): 216 AAAAddRequestHandler(69): 40 cb_add(183): 8 cb_add(177): 32 cdp_sessions_init(178): 8 cdp_sessions_init(173): 8 cdp_sessions_init(165): 1024 cdp_sessions_init(157): 3072 cdp_sessions_init(149): 8 cdp_trans_init(67): 8 cdp_trans_init(60): 24 add_timer(150): 24 add_timer(145): 144 new_peer(77): 16 new_peer(75): 16 new_peer(72): 16 new_peer(69): 80 new_peer(66): 96 new_peer(60): 384 peer_manager_init(84): 8 peer_manager_init(83): 8 peer_manager_init(82): 8 peer_manager_init(79): 8 peer_manager_init(76): 16 worker_init(101): 16 worker_init(94): 128 worker_init(89): 32 worker_init(87): 32 worker_init(84): 8 worker_init(82): 48 timer_cdp_init(182): 8 timer_cdp_init(179): 16 diameter_peer_init_real(137): 8 diameter_peer_init_real(131): 16 diameter_peer_init_real(122): 16 diameter_peer_init_real(115): 8 diameter_peer_init_real(108): 8 diameter_peer_init_real(101): 8 diameter_peer_init_real(94): 8 new_routing_entry(88): 64 parse_dp_config(447): 16 parse_dp_config(348): 8 parse_dp_config(340): 16 parse_dp_config(333): 40 parse_dp_config(326): 144 quote_trim_dup(96): 456 new_dp_config(52): 184 Total: 7024 }
Shared memory after 10 diameter requests
Module: cdp { save_peer_applications(779): 144 AAACreateAVP(137): 5072 AAATranslateMessage(500): 1320 do_receive(600): 7392 make_send_pipe(126): 320 dp_add_pid(92): 216 AAAAddRequestHandler(69): 40 cb_add(183): 8 cb_add(177): 32 cdp_sessions_init(178): 8 cdp_sessions_init(173): 8 cdp_sessions_init(165): 1024 cdp_sessions_init(157): 3072 cdp_sessions_init(149): 8 cdp_trans_init(67): 8 cdp_trans_init(60): 24 add_timer(150): 24 add_timer(145): 144 new_peer(77): 16 new_peer(75): 16 new_peer(72): 16 new_peer(69): 80 new_peer(66): 96 new_peer(60): 384 peer_manager_init(84): 8 peer_manager_init(83): 8 peer_manager_init(82): 8 peer_manager_init(79): 8 peer_manager_init(76): 16 worker_init(101): 16 worker_init(94): 128 worker_init(89): 32 worker_init(87): 32 worker_init(84): 8 worker_init(82): 48 timer_cdp_init(182): 8 timer_cdp_init(179): 16 diameter_peer_init_real(137): 8 diameter_peer_init_real(131): 16 diameter_peer_init_real(122): 16 diameter_peer_init_real(115): 8 diameter_peer_init_real(108): 8 diameter_peer_init_real(101): 8 diameter_peer_init_real(94): 8 new_routing_entry(88): 64 parse_dp_config(447): 16 parse_dp_config(348): 8 parse_dp_config(340): 16 parse_dp_config(333): 40 parse_dp_config(326): 144 quote_trim_dup(96): 456 new_dp_config(52): 184 Total: 20808 }
Shared memory after 170k requests
do_receive(600): 131363752 AAATranslateMessage(500): 32326656 AAACreateAVP(137): 110067152 Total: 273764840
Br Martin
Hello,
thanks for the report, please indicate the version of Kamailio you are using. If its an older version (e.g. not one of the 5.7.x or 5.8.x branches), I would recommend doing an update as well.
Cheers,
Henning
-----Original Message----- From: marpett365--- via sr-users sr-users@lists.kamailio.org Sent: Donnerstag, 2. Januar 2025 11:31 To: sr-users@lists.kamailio.org Cc: marpett365@gmail.com Subject: [SR-Users] ims_diameter_server not freeing shared memory
Hi,
I have run in to an issue when sending diameter requests using the ims_diameter_server module. Shared memory keeps increasing and are not being freed until restart of kamailio. This leads to all shared memory is being used and no more are able to be allocated.
This only happens when the request is sent synchronously. When sent asynchronously shared memory is being freed as far as I can see.
have anyone ran into this issue and found a remedy or can give some pointers how to proceed?
Following functions have been used for sending requests.
diameter_request([peer], appid, commandcode, message) diameter_request_async([peer], appid, commandcode, message)
route[CHECK_LOCATION] { diameter_request("hss.ims.test.local", "16777217", "306", "[ " + "{ "avpCode":260, "vendorId":0, "Flags":64, "list":[ { "avpCode":266, "vendorId":0, "Flags":64, "int32":10415 }, { "avpCode":258, "vendorId":0, "Flags":64, "int32":16777217 } ] }, " + "{ "avpCode":277, "vendorId":0, "Flags":64, "int32":1 }, " + "{ "avpCode":283, "vendorId":0, "Flags":64, "string":"ims.test.local"}, " + "{ "avpCode":706, "vendorId":10415, "Flags":192, "int32":0 }, " + "{ "avpCode":700, "vendorId":10415, "Flags":192, "list": [ { "avpCode":601, "vendorId":10415, "Flags":192, "string":"tel:" + $fU
"" }]}, " + "{ "avpCode":707, "vendorId":10415, "Flags":192, "int32":0 }, " + "{ "avpCode":703, "vendorId":10415, "Flags":192, "int32":14 } ]");
xlog("L_INFO", "Sent Diameter request"); }
Shared memory after restart
kamcmd mod.stats cdp shm Module: cdp { save_peer_applications(779): 144 make_send_pipe(126): 320 dp_add_pid(92): 216 AAAAddRequestHandler(69): 40 cb_add(183): 8 cb_add(177): 32 cdp_sessions_init(178): 8 cdp_sessions_init(173): 8 cdp_sessions_init(165): 1024 cdp_sessions_init(157): 3072 cdp_sessions_init(149): 8 cdp_trans_init(67): 8 cdp_trans_init(60): 24 add_timer(150): 24 add_timer(145): 144 new_peer(77): 16 new_peer(75): 16 new_peer(72): 16 new_peer(69): 80 new_peer(66): 96 new_peer(60): 384 peer_manager_init(84): 8 peer_manager_init(83): 8 peer_manager_init(82): 8 peer_manager_init(79): 8 peer_manager_init(76): 16 worker_init(101): 16 worker_init(94): 128 worker_init(89): 32 worker_init(87): 32 worker_init(84): 8 worker_init(82): 48 timer_cdp_init(182): 8 timer_cdp_init(179): 16 diameter_peer_init_real(137): 8 diameter_peer_init_real(131): 16 diameter_peer_init_real(122): 16 diameter_peer_init_real(115): 8 diameter_peer_init_real(108): 8 diameter_peer_init_real(101): 8 diameter_peer_init_real(94): 8 new_routing_entry(88): 64 parse_dp_config(447): 16 parse_dp_config(348): 8 parse_dp_config(340): 16 parse_dp_config(333): 40 parse_dp_config(326): 144 quote_trim_dup(96): 456 new_dp_config(52): 184 Total: 7024 }
Shared memory after 10 diameter requests
Module: cdp { save_peer_applications(779): 144 AAACreateAVP(137): 5072 AAATranslateMessage(500): 1320 do_receive(600): 7392 make_send_pipe(126): 320 dp_add_pid(92): 216 AAAAddRequestHandler(69): 40 cb_add(183): 8 cb_add(177): 32 cdp_sessions_init(178): 8 cdp_sessions_init(173): 8 cdp_sessions_init(165): 1024 cdp_sessions_init(157): 3072 cdp_sessions_init(149): 8 cdp_trans_init(67): 8 cdp_trans_init(60): 24 add_timer(150): 24 add_timer(145): 144 new_peer(77): 16 new_peer(75): 16 new_peer(72): 16 new_peer(69): 80 new_peer(66): 96 new_peer(60): 384 peer_manager_init(84): 8 peer_manager_init(83): 8 peer_manager_init(82): 8 peer_manager_init(79): 8 peer_manager_init(76): 16 worker_init(101): 16 worker_init(94): 128 worker_init(89): 32 worker_init(87): 32 worker_init(84): 8 worker_init(82): 48 timer_cdp_init(182): 8 timer_cdp_init(179): 16 diameter_peer_init_real(137): 8 diameter_peer_init_real(131): 16 diameter_peer_init_real(122): 16 diameter_peer_init_real(115): 8 diameter_peer_init_real(108): 8 diameter_peer_init_real(101): 8 diameter_peer_init_real(94): 8 new_routing_entry(88): 64 parse_dp_config(447): 16 parse_dp_config(348): 8 parse_dp_config(340): 16 parse_dp_config(333): 40 parse_dp_config(326): 144 quote_trim_dup(96): 456 new_dp_config(52): 184 Total: 20808 }
Shared memory after 170k requests
do_receive(600): 131363752 AAATranslateMessage(500): 32326656 AAACreateAVP(137): 110067152 Total: 273764840
Br Martin __________________________________________________________ Kamailio - Users Mailing List - Non Commercial Discussions -- sr- users@lists.kamailio.org To unsubscribe send an email to sr-users- leave@lists.kamailio.org Important: keep the mailing list in the recipients, do not reply only to the sender!
Hello,
Ok, this is a fairly recent version. It seems that the module indeed does not free the used memory after sending the request.
Cheers,
Henning
-----Original Message----- From: marpett365--- via sr-users sr-users@lists.kamailio.org Sent: Donnerstag, 2. Januar 2025 12:27 To: sr-users@lists.kamailio.org Cc: marpett365@gmail.com Subject: [SR-Users] Re: ims_diameter_server not freeing shared memory
Hi,
Kamailio runs version 5.7.4
Shared memory after 170k requests
do_receive(600): 131363752 AAATranslateMessage(500): 32326656 AAACreateAVP(137): 110067152 Total: 273764840
/Martin __________________________________________________________ Kamailio - Users Mailing List - Non Commercial Discussions -- sr- users@lists.kamailio.org To unsubscribe send an email to sr-users- leave@lists.kamailio.org Important: keep the mailing list in the recipients, do not reply only to the sender!
Hi,
Any pointers in how to proceed?
I tried to follow the memory troubleshooting guide and dumped qm_sum/qm_stats after sending 10 diameter requests synchronously.
Top functions with most allocated shared memory. That keeps increasing after every request sent.
kamcmd mod.stats cdp shm Module: cdp { AAACreateAVP(137): 4608 AAATranslateMessage(500): 1200 do_receive(600): 5280 }
kamailio /usr/local/sbin/kamailio[4086588]: NOTICE: qm_sums: qm_sums(): count= 10 size= 1200 bytes from cdp: diameter_msg.c: AAATranslateMessage(500) kamailio /usr/local/sbin/kamailio[4086588]: NOTICE: qm_sums: qm_sums(): count= 10 size= 5280 bytes from cdp: receiver.c: do_receive(600) kamailio /usr/local/sbin/kamailio[4086588]: NOTICE: qm_sums: qm_sums(): count= 11 size= 456 bytes from cdp: configparser.c: quote_trim_dup(96) kamailio /usr/local/sbin/kamailio[4086588]: NOTICE: qm_sums: qm_sums(): count= 80 size= 4608 bytes from cdp: diameter_avp.c: AAACreateAVP(137) kamailio /usr/local/sbin/kamailio[4086588]: NOTICE: qm_sums: qm_sums(): count= 128 size= 1024 bytes from cdp: session.c: cdp_sessions_init(165)
1 kamailio /usr/local/sbin/kamailio[4086588]: ALERT: qm_status: qm_status(): freed from cdp: diameter_avp.c: AAAFreeAVP(382) 1 kamailio /usr/local/sbin/kamailio[4086588]: ALERT: qm_status: qm_status(): freed from cdp: diameter_msg.c: AAAFreeMessage(441) 10 kamailio /usr/local/sbin/kamailio[4086588]: ALERT: qm_status: qm_status(): alloc'd from cdp: diameter_msg.c: AAATranslateMessage(500) 10 kamailio /usr/local/sbin/kamailio[4086588]: ALERT: qm_status: qm_status(): alloc'd from cdp: receiver.c: do_receive(600) 11 kamailio /usr/local/sbin/kamailio[4086588]: ALERT: qm_status: qm_status(): alloc'd from cdp: configparser.c: quote_trim_dup(96) 80 kamailio /usr/local/sbin/kamailio[4086588]: ALERT: qm_status: qm_status(): alloc'd from cdp: diameter_avp.c: AAACreateAVP(137) 128 kamailio /usr/local/sbin/kamailio[4086588]: ALERT: qm_status: qm_status(): alloc'd from cdp: session.c: cdp_sessions_init(165)
Br Martin
Hello Martin,
if you want to fix it by yourself, have a look to the module code, starting with the cdp module probably. If you found some issues, create a pull-request for the bug fix, to integrate it in the code base.
Otherwise, you can open an issue on our github issue tracker for it.
Cheers,
Henning
-----Original Message----- From: marpett365--- via sr-users sr-users@lists.kamailio.org Sent: Donnerstag, 9. Januar 2025 15:07 To: sr-users@lists.kamailio.org Cc: marpett365@gmail.com Subject: [SR-Users] Re: ims_diameter_server not freeing shared memory
Hi,
Any pointers in how to proceed?
I tried to follow the memory troubleshooting guide and dumped qm_sum/qm_stats after sending 10 diameter requests synchronously.
Top functions with most allocated shared memory. That keeps increasing after every request sent.
kamcmd mod.stats cdp shm Module: cdp { AAACreateAVP(137): 4608 AAATranslateMessage(500): 1200 do_receive(600): 5280 }
kamailio /usr/local/sbin/kamailio[4086588]: NOTICE: qm_sums: qm_sums(): count= 10 size= 1200 bytes from cdp: diameter_msg.c: AAATranslateMessage(500) kamailio /usr/local/sbin/kamailio[4086588]: NOTICE: qm_sums: qm_sums(): count= 10 size= 5280 bytes from cdp: receiver.c: do_receive(600) kamailio /usr/local/sbin/kamailio[4086588]: NOTICE: qm_sums: qm_sums(): count= 11 size= 456 bytes from cdp: configparser.c: quote_trim_dup(96) kamailio /usr/local/sbin/kamailio[4086588]: NOTICE: qm_sums: qm_sums(): count= 80 size= 4608 bytes from cdp: diameter_avp.c: AAACreateAVP(137) kamailio /usr/local/sbin/kamailio[4086588]: NOTICE: qm_sums: qm_sums(): count= 128 size= 1024 bytes from cdp: session.c: cdp_sessions_init(165)
1 kamailio /usr/local/sbin/kamailio[4086588]: ALERT: qm_status:
qm_status(): freed from cdp: diameter_avp.c: AAAFreeAVP(382) 1 kamailio /usr/local/sbin/kamailio[4086588]: ALERT: qm_status: qm_status(): freed from cdp: diameter_msg.c: AAAFreeMessage(441)
10 kamailio /usr/local/sbin/kamailio[4086588]: ALERT: qm_status:
qm_status(): alloc'd from cdp: diameter_msg.c: AAATranslateMessage(500) 10 kamailio /usr/local/sbin/kamailio[4086588]: ALERT: qm_status: qm_status(): alloc'd from cdp: receiver.c: do_receive(600) 11 kamailio /usr/local/sbin/kamailio[4086588]: ALERT: qm_status: qm_status(): alloc'd from cdp: configparser.c: quote_trim_dup(96) 80 kamailio /usr/local/sbin/kamailio[4086588]: ALERT: qm_status: qm_status(): alloc'd from cdp: diameter_avp.c: AAACreateAVP(137) 128 kamailio /usr/local/sbin/kamailio[4086588]: ALERT: qm_status: qm_status(): alloc'd from cdp: session.c: cdp_sessions_init(165)
Br Martin __________________________________________________________ Kamailio - Users Mailing List - Non Commercial Discussions -- sr- users@lists.kamailio.org To unsubscribe send an email to sr-users- leave@lists.kamailio.org Important: keep the mailing list in the recipients, do not reply only to the sender!