[SR-Users] Fwd: Possible memory leak on 5.5.x (new)?

Wed Jan 19 12:09:40 CET 2022

Hello again,

Indeed it must have been the version mismatch. I have upgraded all
instances to 5.5.3 and shm now reports reasonable numbers. Thanks!

BR,
George

On Fri, 7 Jan 2022 at 14:10, Daniel-Constantin Mierla <miconda at gmail.com>
wrote:

> Hello,
> On 07.01.22 13:04, George Diamantopoulos wrote:
>
> Hello Daniel,
>
> I see, thanks for the response. I guess I'll try upgrading all instances
> to 5.5.x for now, and hopefully that will fix it. If not, I'll revert to
> 5.4.x and post here again. If the issue does manifest with all instances on
> 5.5.x however, what information should I collect to investigate this
> further? I'm guessing shm stats is a start, but is it enough?
>
> it is hard to say, starting with the stats and shm summary is good!
>
>
> Lastly, is sip_msg_shm_clone growing consistent with DMQ incompatibility
> between 5.5.x <-> 5.4.x, or should such memory leaks manifest in some other
> way in shm stats?
>
> If the leak manifests only when dmq is used, then troubleshooting has to
> be done with the same Kamailio version on all nodes, otherwise can be waste
> of time. The cloning function is used for other features as well, not only
> for dmq, could be other reasons to leak from it.
>
> Cheers,
> Daniel
>
>
>
> Cheers,
> George
>
> On Fri, 7 Jan 2022 at 13:20, Daniel-Constantin Mierla <miconda at gmail.com>
> wrote:
>
>> Hello,
>>
>> if you do dmq replication between kamailio systems running different
>> major versions, then it is likely to get memory leaks due to replication of
>> data and most probably cannot be fixed. This is because internal structures
>> of modules (also dmq commands) can change, practically what an instance
>> does is not ensured to happen on the other instance. Just for example, from
>> my mind, htable got some changes during past releases, dmq also has
>> significant enhancements by getting support for more transport protocols.
>>
>> If you get memory leaks when you run same Kamailio major version on all
>> Kamailio nodes, then that can be troubleshoot and fixed.
>>
>> Happy new year,
>> Daniel
>> On 07.01.22 11:33, George Diamantopoulos wrote:
>>
>> Hello all and happy new year,
>>
>> I have some new information to share regarding this issue. I believe the
>> previous metrics I sent to the list might not be indicative of the way the
>> problem manifests. Here's what I believe so far:
>>   - Issue is exacerbated (or manifests) during moderate-to-high cps, or
>> grows linearly with total traffic processed since last restart
>>   - shm stats show a lot of memory consumed by sip_msg_shm_clone
>>   - also reproduced this time on bullseye with kamailio 5.5.3
>>
>> Here's some more meaningful stats taken at more appropriate times (i.e.
>> after more traffic has been processed) than the previous ones. These two
>> kamailio instances have identical configuration and traffic patterns:
>> - https://pastebin.com/gHa803kB for kamailio 5.5.3 showing high
>> sip_msg_shm_clone on debian bullseye
>> - https://pastebin.com/JbcZbbSQ for kamailio 5.4.6 on debian buster
>>
>> There is still DMQ use for these instances despite the version mismatch.
>> Unfortunately I can't migrate all DMQ nodes to 5.5.x at this time, not
>> unless I can have assurances that it is DMQ that causes this issue with shm
>> memory exhaustion...
>>
>> After shmem was exhausted on 5.5.3, it stopped processing traffic. I
>> issued a kamctl trap at that time but I'm assuming the backtrace won't show
>> much except for the inability to allocate shm? If you think the backtrace
>> at that point would be useful in any way, let me know and I'll try to share
>> it privately. In case it isn't useful, what other debugging information can
>> be gathered to dissect this issue? Thanks!
>>
>> BR,
>> George
>>
>> On Wed, 30 Jun 2021 at 19:20, Daniel-Constantin Mierla <miconda at gmail.com>
>> wrote:
>>
>>> Hello,
>>>
>>> for the sake of completion: the autoexpire should clean the items if
>>> they are not used during the expiration interval. If you want to get them
>>> deleted after first expiration interval always, see the updateexpire
>>> attribute for htable modparam.
>>>
>>> Also, note that replication should be done only between Kamailio
>>> instances with same major version, because there can be internal
>>> differences between major versions that can lead to unexpected behaviour.
>>> In other words, if you replicate, doing between two kamailio with version
>>> 5.5.x or between two kamailio with version 5.4.x, but not between a
>>> kamailkio 5.5.x and a kamailio 5.4.x.
>>>
>>> The total amount of used memory in the stats file for 5.5 does not seem
>>> to be high as a rough estimation. The highest by module is in htable, but
>>> it is around 20MB. Maybe you took the stats too early, quickly after a
>>> restart?
>>> Cheers,
>>> Daniel
>>>
>>> On 30.06.21 17:20, George Diamantopoulos wrote:
>>>
>>> Hello Daniel,
>>>
>>> Thanks for the feedback. I think I might have been too quick to blame
>>> htable for this behaviour. In fact, version 5.4 seems to consume more
>>> memory than 5.5 (175129776 bytes vs 20581096), which makes sense since it
>>> has been running for longer (I missed the extra digit previously).
>>>
>>> So I'm not sure htable is to blame. On the other hand, I don't see any
>>> other modules using up too much of shmem either, so maybe memory stats
>>> can't provide the answer here?
>>>
>>> To answer your question, though, I do use DMQ and both tables that use
>>> it have autoexpire set to the same value on both 5.4 and 5.5:
>>>
>>> /etc/kamailio# grep dmq kamailio-module-params.cfg
>>> modparam("dmq", "server_address", "sip:172.30.43.1:5090")
>>> modparam("dmq", "notification_address", "sip:
>>> dmq.services.mydomain.com:5090")
>>> modparam("dmq", "multi_notify", 1)
>>> modparam("htable", "enable_dmq", 1)
>>> modparam("htable", "htable",
>>> 'cid2hi=>size=8;autoexpire=600;dmqreplicate=1')
>>> modparam("htable", "htable",
>>> 'xcid2count=>size=8;autoexpire=600;dmqreplicate=1')
>>>
>>> On Wed, 30 Jun 2021 at 17:43, Daniel-Constantin Mierla <
>>> miconda at gmail.com> wrote:
>>>
>>>> Hello,
>>>>
>>>> do you replicate items in the htable via dmq? Does the htable have
>>>> autoexpire value set?
>>>>
>>>> Cheers,
>>>> Daniel
>>>> On 30.06.21 13:54, George Diamantopoulos wrote:
>>>>
>>>> Forwarding my reply to the list, using gmail's reply button set Henning
>>>> as the sole recipient :-\
>>>>
>>>> ---------- Forwarded message ---------
>>>> From: George Diamantopoulos <georgediam at gmail.com>
>>>> Date: Sat, 26 Jun 2021 at 02:25
>>>> Subject: Re: [SR-Users] Possible memory leak on 5.5.x (new)?
>>>> To: Henning Westerholt <hw at skalatan.de>
>>>>
>>>>
>>>> Hello Henning,
>>>>
>>>> Thanks for your reply. Here's what has come up after a few hours:
>>>>
>>>> shm55: https://pastebin.com/h9JCePmc
>>>> shm54: https://pastebin.com/Nx5xEEnA
>>>>
>>>> It seems to me htable is the culprit? Are you seeing anything
>>>> different? 54 has been running for 77020 seconds, 55 for 28521
>>>> (significantly less).
>>>>
>>>> I'm going to turn it off until we figure something out...
>>>>
>>>> BR,
>>>> George
>>>>
>>>> On Fri, 25 Jun 2021 at 18:17, Henning Westerholt <hw at skalatan.de>
>>>> wrote:
>>>>
>>>>> Hello,
>>>>>
>>>>>
>>>>>
>>>>> Good observation. Please run the memory statistics CLI commands to get
>>>>> more hints about the module that might cause it (as per below link). Then
>>>>> please report more details. If you can point to a particular module, you
>>>>> can also open an issue on our tracker.
>>>>>
>>>>>
>>>>>
>>>>> https://www.kamailio.org/wiki/tutorials/troubleshooting/memory
>>>>>
>>>>>
>>>>>
>>>>> Cheers,
>>>>>
>>>>>
>>>>>
>>>>> Henning
>>>>>
>>>>>
>>>>>
>>>>> *From:* sr-users <sr-users-bounces at lists.kamailio.org> *On Behalf Of *George
>>>>> Diamantopoulos
>>>>> *Sent:* Friday, June 25, 2021 4:53 PM
>>>>> *To:* Kamailio (SER) - Users Mailing List <sr-users at lists.kamailio.org
>>>>> >
>>>>> *Subject:* [SR-Users] Possible memory leak on 5.5.x (new)?
>>>>>
>>>>>
>>>>>
>>>>> Hello all,
>>>>>
>>>>>
>>>>>
>>>>> I'm still investigating the (most likely non-kamailio-related) memory
>>>>> leak of my previous message to the list, there have been no developments so
>>>>> far. I'll update if anything changes.
>>>>>
>>>>>
>>>>>
>>>>> This concerns a new finding which seems to affect kamailio 5.5.x. I
>>>>> have two kamailio instances receiving the same traffic via round-robin. I
>>>>> upgraded only one of them to 5.5.1 and left the other to 5.4.6 as I feared
>>>>> of any issues arising. I was lucky to do so, because with identical
>>>>> configuration, 5.5.x seems to run out of SHM very quickly. Here are links
>>>>> to graphs produced by our monitoring system:
>>>>>
>>>>>
>>>>>
>>>>> Old kamailio (no memory leak): https://pasteboard.co/K8fVBiD.png
>>>>>
>>>>> New kamailio (possible leak): https://pasteboard.co/K8fVS9N.png
>>>>>
>>>>>
>>>>>
>>>>> The configuration uses mtree, htable, vars and vns extensively. Has
>>>>> anyone come across anything similar? Let me know if I can provide any
>>>>> further information to help disect this. Thanks!
>>>>>
>>>>>
>>>>>
>>>>> BR,
>>>>>
>>>>> George
>>>>>
>>>>
>>>> __________________________________________________________
>>>> Kamailio - Users Mailing List - Non Commercial Discussions
>>>>   * sr-users at lists.kamailio.org
>>>> Important: keep the mailing list in the recipients, do not reply only to the sender!
>>>> Edit mailing list options or unsubscribe:
>>>>   * https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
>>>>
>>>> --
>>>> Daniel-Constantin Mierla -- www.asipto.comwww.twitter.com/miconda -- www.linkedin.com/in/miconda
>>>>
>>>> --
>>> Daniel-Constantin Mierla -- www.asipto.comwww.twitter.com/miconda -- www.linkedin.com/in/miconda
>>>
>>> --
>> Daniel-Constantin Mierla -- www.asipto.comwww.twitter.com/miconda -- www.linkedin.com/in/miconda
>> Kamailio Advanced Training - Online
>>   Feb 21-24, 2022 (America Timezone)
>>   * https://www.asipto.com/sw/kamailio-advanced-training-online/
>>
>> --
> Daniel-Constantin Mierla -- www.asipto.comwww.twitter.com/miconda -- www.linkedin.com/in/miconda
> Kamailio Advanced Training - Online
>   Feb 21-24, 2022 (America Timezone)
>   * https://www.asipto.com/sw/kamailio-advanced-training-online/
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.kamailio.org/pipermail/sr-users/attachments/20220119/6c5c45a3/attachment.htm>