Hello all,
I'm testing out release 5.6.2 on one production kamailio instance, and today our monitoring system triggered an alarm for lack of disk space on /. Indeed, kamailio.log started growing at an alarming rate at some point.
At about the same time disk space started being depleted, the following line was frequently logged in the logs (no instances of it before that time):
qm_malloc(): qm_malloc(0x7f5ade9c0010, 536) called from rtpengine: bencode.c: __bencode_piece_new(79), module: rtpengine; Free fragment not found!
They don't only concern rtpengine, core has produced similar "free fragment not found" errors as well. There's additional multiple "had header field", "bad header" and "invalid header name" errors appearing after that too, but I'm guessing that's only a symptom of the qm_malloc issue?
kamailio is started with -m 2048 -M 16 and there's another instance with almost identical traffic and configuration, but running on 5.6.1, which has never exhibited such failures in several months uptime. So I'm wondering, is there anything in 5.6.2 that could explain this? Thanks!
Best regards, George
Hello,
difficult to say, it could be also some random memory error or virtual/physical hardware issue. Are you seeing more error related to that, maybe it’s a memory leak? Make probably sense to observe it further. If it happens again, a diff between the rtpengine in 5.6.1 and 5.6.2 might get more insight. Regarding a memory leak, you can observe the memory pool by kamcmd or different tools by looking at statistics or memory status.
Cheers,
Henning
-- Henning Westerholt – https://skalatan.de/blog/ Kamailio services – https://gilawa.comhttps://gilawa.com/
From: sr-users sr-users-bounces@lists.kamailio.org On Behalf Of George Diamantopoulos Sent: Tuesday, December 13, 2022 12:14 PM To: Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org Subject: [SR-Users] Logs suddenly inundated with qm_malloc errors
Hello all,
I'm testing out release 5.6.2 on one production kamailio instance, and today our monitoring system triggered an alarm for lack of disk space on /. Indeed, kamailio.log started growing at an alarming rate at some point.
At about the same time disk space started being depleted, the following line was frequently logged in the logs (no instances of it before that time):
qm_malloc(): qm_malloc(0x7f5ade9c0010, 536) called from rtpengine: bencode.c: __bencode_piece_new(79), module: rtpengine; Free fragment not found!
They don't only concern rtpengine, core has produced similar "free fragment not found" errors as well. There's additional multiple "had header field", "bad header" and "invalid header name" errors appearing after that too, but I'm guessing that's only a symptom of the qm_malloc issue?
kamailio is started with -m 2048 -M 16 and there's another instance with almost identical traffic and configuration, but running on 5.6.1, which has never exhibited such failures in several months uptime. So I'm wondering, is there anything in 5.6.2 that could explain this? Thanks!
Best regards, George
Maybe there is a memory leak or maybe the system is not provisioned with enough memory to cope with the traffic. Increase the amount of allocated memory and monitor it, see how it evolves over time.
Regards, Ovidiu Sas
On Tue, Dec 13, 2022 at 6:14 AM George Diamantopoulos georgediam@gmail.com wrote:
Hello all,
I'm testing out release 5.6.2 on one production kamailio instance, and today our monitoring system triggered an alarm for lack of disk space on /. Indeed, kamailio.log started growing at an alarming rate at some point.
At about the same time disk space started being depleted, the following line was frequently logged in the logs (no instances of it before that time):
qm_malloc(): qm_malloc(0x7f5ade9c0010, 536) called from rtpengine: bencode.c: __bencode_piece_new(79), module: rtpengine; Free fragment not found!
They don't only concern rtpengine, core has produced similar "free fragment not found" errors as well. There's additional multiple "had header field", "bad header" and "invalid header name" errors appearing after that too, but I'm guessing that's only a symptom of the qm_malloc issue?
kamailio is started with -m 2048 -M 16 and there's another instance with almost identical traffic and configuration, but running on 5.6.1, which has never exhibited such failures in several months uptime. So I'm wondering, is there anything in 5.6.2 that could explain this? Thanks!
Best regards, George __________________________________________________________ Kamailio - Users Mailing List - Non Commercial Discussions sr-users@lists.kamailio.org Important: keep the mailing list in the recipients, do not reply only to the sender! Edit mailing list options or unsubscribe: https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Hello Henning, Ovidiu,
Thank you for your feedback.
I downgraded to 5.6.1 and the issue hasn't reappeared since then (three days ago). Note that it did manifest twice in a day when running on 5.6.2 so it seems possible this is a regression.
I'm not sure if the culprit lies in the rtpengine module. There were similar messages for other components as well later on in the logs (e.g. core).
Can you think of any changes between 5.6.1 and 5.6.2 that could be potential triggers? Should I open an issue over at github? Thanks!
Best regards, George
On Tue, 13 Dec 2022 at 20:53, Ovidiu Sas osas@voipembedded.com wrote:
Maybe there is a memory leak or maybe the system is not provisioned with enough memory to cope with the traffic. Increase the amount of allocated memory and monitor it, see how it evolves over time.
Regards, Ovidiu Sas
On Tue, Dec 13, 2022 at 6:14 AM George Diamantopoulos georgediam@gmail.com wrote:
Hello all,
I'm testing out release 5.6.2 on one production kamailio instance, and
today our monitoring system triggered an alarm for lack of disk space on /. Indeed, kamailio.log started growing at an alarming rate at some point.
At about the same time disk space started being depleted, the following
line was frequently logged in the logs (no instances of it before that time):
qm_malloc(): qm_malloc(0x7f5ade9c0010, 536) called from rtpengine:
bencode.c: __bencode_piece_new(79), module: rtpengine; Free fragment not found!
They don't only concern rtpengine, core has produced similar "free
fragment not found" errors as well. There's additional multiple "had header field", "bad header" and "invalid header name" errors appearing after that too, but I'm guessing that's only a symptom of the qm_malloc issue?
kamailio is started with -m 2048 -M 16 and there's another instance with
almost identical traffic and configuration, but running on 5.6.1, which has never exhibited such failures in several months uptime. So I'm wondering, is there anything in 5.6.2 that could explain this? Thanks!
Best regards, George __________________________________________________________ Kamailio - Users Mailing List - Non Commercial Discussions sr-users@lists.kamailio.org Important: keep the mailing list in the recipients, do not reply only to
the sender!
Edit mailing list options or unsubscribe: https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
-- VoIP Embedded, Inc. http://www.voipembedded.com
Kamailio - Users Mailing List - Non Commercial Discussions sr-users@lists.kamailio.org Important: keep the mailing list in the recipients, do not reply only to the sender! Edit mailing list options or unsubscribe: https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Hi George,
thanks for the feedback. Maybe some insight can be got from logs related to the error, as mentioned, maybe there are more error messages related to memory allocation problems in the log?
Other idea is to run it again with the “bad” version, and after some time run the memory debug to get the allocated memory details. Further details are documented here: http://www.kamailio.org/wiki/tutorials/troubleshooting/memory If you get more feedback, it would be surely good to open an issue to fix the regression, if its one.
Cheers,
Henning
-- Henning Westerholt – https://skalatan.de/blog/ Kamailio services – https://gilawa.comhttps://gilawa.com/
From: George Diamantopoulos georgediam@gmail.com Sent: Friday, December 16, 2022 11:58 AM To: Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org Subject: [SR-Users] Re: Logs suddenly inundated with qm_malloc errors
Hello Henning, Ovidiu,
Thank you for your feedback.
I downgraded to 5.6.1 and the issue hasn't reappeared since then (three days ago). Note that it did manifest twice in a day when running on 5.6.2 so it seems possible this is a regression.
I'm not sure if the culprit lies in the rtpengine module. There were similar messages for other components as well later on in the logs (e.g. core).
Can you think of any changes between 5.6.1 and 5.6.2 that could be potential triggers? Should I open an issue over at github? Thanks!
Best regards, George
On Tue, 13 Dec 2022 at 20:53, Ovidiu Sas <osas@voipembedded.commailto:osas@voipembedded.com> wrote: Maybe there is a memory leak or maybe the system is not provisioned with enough memory to cope with the traffic. Increase the amount of allocated memory and monitor it, see how it evolves over time.
Regards, Ovidiu Sas
On Tue, Dec 13, 2022 at 6:14 AM George Diamantopoulos <georgediam@gmail.commailto:georgediam@gmail.com> wrote:
Hello all,
I'm testing out release 5.6.2 on one production kamailio instance, and today our monitoring system triggered an alarm for lack of disk space on /. Indeed, kamailio.log started growing at an alarming rate at some point.
At about the same time disk space started being depleted, the following line was frequently logged in the logs (no instances of it before that time):
qm_malloc(): qm_malloc(0x7f5ade9c0010, 536) called from rtpengine: bencode.c: __bencode_piece_new(79), module: rtpengine; Free fragment not found!
They don't only concern rtpengine, core has produced similar "free fragment not found" errors as well. There's additional multiple "had header field", "bad header" and "invalid header name" errors appearing after that too, but I'm guessing that's only a symptom of the qm_malloc issue?
kamailio is started with -m 2048 -M 16 and there's another instance with almost identical traffic and configuration, but running on 5.6.1, which has never exhibited such failures in several months uptime. So I'm wondering, is there anything in 5.6.2 that could explain this? Thanks!
Best regards, George __________________________________________________________ Kamailio - Users Mailing List - Non Commercial Discussions sr-users@lists.kamailio.orgmailto:sr-users@lists.kamailio.org Important: keep the mailing list in the recipients, do not reply only to the sender! Edit mailing list options or unsubscribe: https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
-- VoIP Embedded, Inc. http://www.voipembedded.com
__________________________________________________________ Kamailio - Users Mailing List - Non Commercial Discussions sr-users@lists.kamailio.orgmailto:sr-users@lists.kamailio.org Important: keep the mailing list in the recipients, do not reply only to the sender! Edit mailing list options or unsubscribe: https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
-
George Diamantopoulos -
Henning Westerholt -
Ovidiu Sas