<p>More info:</p>
<p>On another cluster, same setup..., during this troubleshooting I disabled DMQ and enabled MySQL for dialog replication, I also left one node outside of rotation to see replication behavior.</p>
<p>Well, with 0 traffic I can see this:</p>
<pre><code>root@sbc01:~# kamctl rpc dlg.stats_active
{
"jsonrpc": "2.0",
"result": {
"starting": 0,
"connecting": 0,
"answering": 0,
"ongoing": 0,
"all": 0
},
"id": 2634
}
root@sbc01:~#
</code></pre>
<p>Which is correct, but...:</p>
<pre><code>root@sbc01:~# kamctl rpc stats.fetch all | grep dialog
"dialog.active_dialogs": "38",
"dialog.early_dialogs": "0",
"dialog.expired_dialogs": "38",
"dialog.failed_dialogs": "0",
"dialog.processed_dialogs": "1",
root@sbc01:~#
</code></pre>
<p>I wonder if it's correct that those 38 <code>expired</code> dialogs still count towards the <code>active</code> counter? Could that have something to do? Maybe those non-existent-expired-dialogs because they are somewhere in the "active" counter, they get replicated only when DMQ is enabled?</p>
<p>So far I have clear that:</p>
<p>1- For sure there is a scenario where counters go below 0.<br>
2- It only happens (so far) when DMQ is enabled<br>
3- In my initial look, I found some logs about dialogs created on node1 were timed-out on node2 (see previous posts)<br>
4- Now I see a discrepancy on active vs expired dialogs on a node with 0 active dialogs.</p>
<p>I feel we are slowly narrowing down what the possible problem can be, but I still don't have a clear picture what causes what.</p>
<p style="font-size:small;-webkit-text-size-adjust:none;color:#666;">—<br />You are receiving this because you are subscribed to this thread.<br />Reply to this email directly, <a href="https://github.com/kamailio/kamailio/issues/1591#issuecomment-408690103">view it on GitHub</a>, or <a href="https://github.com/notifications/unsubscribe-auth/AF36ZYNrEXR3DFJTNOQhPo0p48njie83ks5uLebwgaJpZM4VLmd9">mute the thread</a>.<img src="https://github.com/notifications/beacon/AF36Zeiah9lanZqfTvweQPB1a3n1UPXzks5uLebwgaJpZM4VLmd9.gif" height="1" width="1" alt="" /></p>
<script type="application/json" data-scope="inboxmarkup">{"api_version":"1.0","publisher":{"api_key":"05dde50f1d1a384dd78767c55493e4bb","name":"GitHub"},"entity":{"external_key":"github/kamailio/kamailio","title":"kamailio/kamailio","subtitle":"GitHub repository","main_image_url":"https://assets-cdn.github.com/images/email/message_cards/header.png","avatar_image_url":"https://assets-cdn.github.com/images/email/message_cards/avatar.png","action":{"name":"Open in GitHub","url":"https://github.com/kamailio/kamailio"}},"updates":{"snippets":[{"icon":"PERSON","message":"@joelsdc in #1591: More info:\r\n\r\nOn another cluster, same setup..., during this troubleshooting I disabled DMQ and enabled MySQL for dialog replication, I also left one node outside of rotation to see replication behavior.\r\n\r\nWell, with 0 traffic I can see this:\r\n\r\n```\r\nroot@sbc01:~# kamctl rpc dlg.stats_active\r\n{\r\n \"jsonrpc\": \"2.0\",\r\n \"result\": {\r\n \"starting\": 0,\r\n \"connecting\": 0,\r\n \"answering\": 0,\r\n \"ongoing\": 0,\r\n \"all\": 0\r\n },\r\n \"id\": 2634\r\n}\r\nroot@sbc01:~#\r\n```\r\n\r\nWhich is correct, but...:\r\n\r\n```\r\nroot@sbc01:~# kamctl rpc stats.fetch all | grep dialog\r\n \"dialog.active_dialogs\": \"38\",\r\n \"dialog.early_dialogs\": \"0\",\r\n \"dialog.expired_dialogs\": \"38\",\r\n \"dialog.failed_dialogs\": \"0\",\r\n \"dialog.processed_dialogs\": \"1\",\r\nroot@sbc01:~#\r\n```\r\n\r\nI wonder if it's correct that those 38 `expired` dialogs still count towards the `active` counter? Could that have something to do? Maybe those non-existent-expired-dialogs because they are somewhere in the \"active\" counter, they get replicated only when DMQ is enabled?\r\n\r\nSo far I have clear that:\r\n\r\n1- For sure there is a scenario where counters go below 0.\r\n2- It only happens (so far) when DMQ is enabled\r\n3- In my initial look, I found some logs about dialogs created on node1 were timed-out on node2 (see previous posts)\r\n4- Now I see a discrepancy on active vs expired dialogs on a node with 0 active dialogs.\r\n\r\nI feel we are slowly narrowing down what the possible problem can be, but I still don't have a clear picture what causes what.\r\n"}],"action":{"name":"View Issue","url":"https://github.com/kamailio/kamailio/issues/1591#issuecomment-408690103"}}}</script>
<script type="application/ld+json">[
{
"@context": "http://schema.org",
"@type": "EmailMessage",
"potentialAction": {
"@type": "ViewAction",
"target": "https://github.com/kamailio/kamailio/issues/1591#issuecomment-408690103",
"url": "https://github.com/kamailio/kamailio/issues/1591#issuecomment-408690103",
"name": "View Issue"
},
"description": "View this Issue on GitHub",
"publisher": {
"@type": "Organization",
"name": "GitHub",
"url": "https://github.com"
}
},
{
"@type": "MessageCard",
"@context": "http://schema.org/extensions",
"hideOriginalBody": "false",
"originator": "AF6C5A86-E920-430C-9C59-A73278B5EFEB",
"title": "Re: [kamailio/kamailio] RPC command stats.get_statistics randomly reporting 9223372036854776000 for current active/early dialogs (#1591)",
"sections": [
{
"text": "",
"activityTitle": "**Joel Serrano**",
"activityImage": "https://assets-cdn.github.com/images/email/message_cards/avatar.png",
"activitySubtitle": "@joelsdc",
"facts": [
]
}
],
"potentialAction": [
{
"name": "Add a comment",
"@type": "ActionCard",
"inputs": [
{
"isMultiLine": true,
"@type": "TextInput",
"id": "IssueComment",
"isRequired": false
}
],
"actions": [
{
"name": "Comment",
"@type": "HttpPOST",
"target": "https://api.github.com",
"body": "{\n\"commandName\": \"IssueComment\",\n\"repositoryFullName\": \"kamailio/kamailio\",\n\"issueId\": 1591,\n\"IssueComment\": \"{{IssueComment.value}}\"\n}"
}
]
},
{
"name": "Close issue",
"@type": "HttpPOST",
"target": "https://api.github.com",
"body": "{\n\"commandName\": \"IssueClose\",\n\"repositoryFullName\": \"kamailio/kamailio\",\n\"issueId\": 1591\n}"
},
{
"targets": [
{
"os": "default",
"uri": "https://github.com/kamailio/kamailio/issues/1591#issuecomment-408690103"
}
],
"@type": "OpenUri",
"name": "View on GitHub"
},
{
"name": "Unsubscribe",
"@type": "HttpPOST",
"target": "https://api.github.com",
"body": "{\n\"commandName\": \"MuteNotification\",\n\"threadId\": 355362685\n}"
}
],
"themeColor": "26292E"
}
]</script>