Hi there,
Every now and then, the value of active dialogs metric spikes to an erroneous huge number, messing up my otherwise beautiful grafana gauges :(
Example: kamailio_dialog_active_dialogs 18446744073709551614 1602812239115 kamailio_dialog_early_dialogs 0 1602812239115 kamailio_dialog_expired_dialogs 2 1602812239115 kamailio_dialog_failed_dialogs 7361 1602812239115
Anybody faced the same symptoms, any clues what might be causing this?
Thanks, --Sergiu
Hello,
is this happening when using dmq for dialog with multiple sip servers? Or is a single instance sip server?
Cheers, Daniel
On 16.10.20 04:12, Sergiu Pojoga wrote:
Hi there,
Every now and then, the value of active dialogs metric spikes to an erroneous huge number, messing up my otherwise beautiful grafana gauges :(
Example: kamailio_dialog_active_dialogs 18446744073709551614 1602812239115 kamailio_dialog_early_dialogs 0 1602812239115 kamailio_dialog_expired_dialogs 2 1602812239115 kamailio_dialog_failed_dialogs 7361 1602812239115
Anybody faced the same symptoms, any clues what might be causing this?
Thanks, --Sergiu
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Hi Daniel,
You guessed right, two dmq nodes with dialog replication.
Thanks,
On Fri., Oct. 16, 2020, 2:15 a.m. Daniel-Constantin Mierla, < miconda@gmail.com> wrote:
Hello,
is this happening when using dmq for dialog with multiple sip servers? Or is a single instance sip server?
Cheers, Daniel On 16.10.20 04:12, Sergiu Pojoga wrote:
Hi there,
Every now and then, the value of active dialogs metric spikes to an erroneous huge number, messing up my otherwise beautiful grafana gauges :(
Example: kamailio_dialog_active_dialogs 18446744073709551614 1602812239115 kamailio_dialog_early_dialogs 0 1602812239115 kamailio_dialog_expired_dialogs 2 1602812239115 kamailio_dialog_failed_dialogs 7361 1602812239115
Anybody faced the same symptoms, any clues what might be causing this?
Thanks, --Sergiu
Kamailio (SER) - Users Mailing Listsr-users@lists.kamailio.orghttps://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
-- Daniel-Constantin Mierla -- www.asipto.comwww.twitter.com/miconda -- www.linkedin.com/in/miconda Funding: https://www.paypal.me/dcmierla
Hey Sergiu,
I think I've been through a similar case... have a look at: https://github.com/kamailio/kamailio/issues/1591 in case you find it helpful.
You might have to rely on the different metric that Daniel added.
Joel.
On Fri, Oct 16, 2020 at 3:35 AM Sergiu Pojoga pojogas@gmail.com wrote:
Hi Daniel,
You guessed right, two dmq nodes with dialog replication.
Thanks,
On Fri., Oct. 16, 2020, 2:15 a.m. Daniel-Constantin Mierla, < miconda@gmail.com> wrote:
Hello,
is this happening when using dmq for dialog with multiple sip servers? Or is a single instance sip server?
Cheers, Daniel On 16.10.20 04:12, Sergiu Pojoga wrote:
Hi there,
Every now and then, the value of active dialogs metric spikes to an erroneous huge number, messing up my otherwise beautiful grafana gauges :(
Example: kamailio_dialog_active_dialogs 18446744073709551614 1602812239115 kamailio_dialog_early_dialogs 0 1602812239115 kamailio_dialog_expired_dialogs 2 1602812239115 kamailio_dialog_failed_dialogs 7361 1602812239115
Anybody faced the same symptoms, any clues what might be causing this?
Thanks, --Sergiu
Kamailio (SER) - Users Mailing Listsr-users@lists.kamailio.orghttps://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
-- Daniel-Constantin Mierla -- www.asipto.comwww.twitter.com/miconda -- www.linkedin.com/in/miconda Funding: https://www.paypal.me/dcmierla
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Hi Joel,
Thanks, that's precisely it!
Indeed, `dlg.stats_active` works correctly and what's nice about it is that it shows stats summary from all DMQ nodes combined. However, I use xHTTP_PROM module to export my stats, which seems to rely on the 'problematic' counters. Sad... Let's hope this gets resolved one day.
Regards, --Sergiu
On Fri, Oct 16, 2020 at 4:17 PM Joel Serrano joel@textplus.com wrote:
Hey Sergiu,
I think I've been through a similar case... have a look at: https://github.com/kamailio/kamailio/issues/1591 in case you find it helpful.
You might have to rely on the different metric that Daniel added.
Joel.
On Fri, Oct 16, 2020 at 3:35 AM Sergiu Pojoga pojogas@gmail.com wrote:
Hi Daniel,
You guessed right, two dmq nodes with dialog replication.
Thanks,
On Fri., Oct. 16, 2020, 2:15 a.m. Daniel-Constantin Mierla, < miconda@gmail.com> wrote:
Hello,
is this happening when using dmq for dialog with multiple sip servers? Or is a single instance sip server?
Cheers, Daniel On 16.10.20 04:12, Sergiu Pojoga wrote:
Hi there,
Every now and then, the value of active dialogs metric spikes to an erroneous huge number, messing up my otherwise beautiful grafana gauges :(
Example: kamailio_dialog_active_dialogs 18446744073709551614 1602812239115 kamailio_dialog_early_dialogs 0 1602812239115 kamailio_dialog_expired_dialogs 2 1602812239115 kamailio_dialog_failed_dialogs 7361 1602812239115
Anybody faced the same symptoms, any clues what might be causing this?
Thanks, --Sergiu
Kamailio (SER) - Users Mailing Listsr-users@lists.kamailio.orghttps://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
-- Daniel-Constantin Mierla -- www.asipto.comwww.twitter.com/miconda -- www.linkedin.com/in/miconda Funding: https://www.paypal.me/dcmierla
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Sorry to insist but... looks like it's not just the active_dialogs stats that's messed up. Early_dialogs are also often erroneous and the dialog counters don't match against different RPCs.
What would the devs like me to do with this: drop it, open another GitHub issue?
# kamcmd dlg.stats_active && kamcmd stats.get_statistics all | grep dialog { starting: 0 connecting: 0 answering: 0 ongoing: 2 all: 2 } dialog:active_dialogs = 20 dialog:early_dialogs = 18446744073709551614 dialog:expired_dialogs = 0 dialog:failed_dialogs = 6978 dialog:processed_dialogs = 11248
# kamcmd dmq.list_nodes { host: 65.xx.xx.11 port: 5062 resolved_ip: 65.xx.xx.11 status: active last_notification: 0 local: 0 } { host: 65.xx.xx.22 port: 5062 resolved_ip: 65.xx.xx.22 status: active last_notification: 0 local: 1 }
# kamailio -v version: kamailio 5.3.4 (x86_64/linux) fc28a1
Thanks. --Sergiu
On Fri, Oct 16, 2020 at 5:10 PM Sergiu Pojoga pojogas@gmail.com wrote:
Hi Joel,
Thanks, that's precisely it!
Indeed, `dlg.stats_active` works correctly and what's nice about it is that it shows stats summary from all DMQ nodes combined. However, I use xHTTP_PROM module to export my stats, which seems to rely on the 'problematic' counters. Sad... Let's hope this gets resolved one day.
Regards, --Sergiu
On Fri, Oct 16, 2020 at 4:17 PM Joel Serrano joel@textplus.com wrote:
Hey Sergiu,
I think I've been through a similar case... have a look at: https://github.com/kamailio/kamailio/issues/1591 in case you find it helpful.
You might have to rely on the different metric that Daniel added.
Joel.
On Fri, Oct 16, 2020 at 3:35 AM Sergiu Pojoga pojogas@gmail.com wrote:
Hi Daniel,
You guessed right, two dmq nodes with dialog replication.
Thanks,
On Fri., Oct. 16, 2020, 2:15 a.m. Daniel-Constantin Mierla, < miconda@gmail.com> wrote:
Hello,
is this happening when using dmq for dialog with multiple sip servers? Or is a single instance sip server?
Cheers, Daniel On 16.10.20 04:12, Sergiu Pojoga wrote:
Hi there,
Every now and then, the value of active dialogs metric spikes to an erroneous huge number, messing up my otherwise beautiful grafana gauges :(
Example: kamailio_dialog_active_dialogs 18446744073709551614 1602812239115 kamailio_dialog_early_dialogs 0 1602812239115 kamailio_dialog_expired_dialogs 2 1602812239115 kamailio_dialog_failed_dialogs 7361 1602812239115
Anybody faced the same symptoms, any clues what might be causing this?
Thanks, --Sergiu
Kamailio (SER) - Users Mailing Listsr-users@lists.kamailio.orghttps://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
-- Daniel-Constantin Mierla -- www.asipto.comwww.twitter.com/miconda -- www.linkedin.com/in/miconda Funding: https://www.paypal.me/dcmierla
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Unfortunately the design of the dialog statistics, which are basically just counters, do not allow dealing easy with retransmissions or duplicate operations that can happen with dmq. That was the reason to add the rpc command to scan what the dialog module has internally. Probably the dialog statistics should be diabled when dmq is enabled or switched to use function callbacks behind and to the scanning mechanism as well, but some of them won't be possible with callbacks without refactoring.
Cheers, Daniel
On 20.10.20 02:28, Sergiu Pojoga wrote:
Sorry to insist but... looks like it's not just the active_dialogs stats that's messed up. Early_dialogs are also often erroneous and the dialog counters don't match against different RPCs.
What would the devs like me to do with this: drop it, open another GitHub issue? # kamcmd dlg.stats_active && kamcmd stats.get_statistics all | grep dialog { starting: 0 connecting: 0 answering: 0 ongoing: 2 all: 2 } dialog:active_dialogs = 20 dialog:early_dialogs = 18446744073709551614 dialog:expired_dialogs = 0 dialog:failed_dialogs = 6978 dialog:processed_dialogs = 11248
# kamcmd dmq.list_nodes { host: 65.xx.xx.11 port: 5062 resolved_ip: 65.xx.xx.11 status: active last_notification: 0 local: 0 } { host: 65.xx.xx.22 port: 5062 resolved_ip: 65.xx.xx.22 status: active last_notification: 0 local: 1 }
# kamailio -v version: kamailio 5.3.4 (x86_64/linux) fc28a1
Thanks. --Sergiu
On Fri, Oct 16, 2020 at 5:10 PM Sergiu Pojoga <pojogas@gmail.com mailto:pojogas@gmail.com> wrote:
Hi Joel, Thanks, that's precisely it! Indeed, `dlg.stats_active` works correctly and what's nice about it is that it shows stats summary from all DMQ nodes combined. However, I use xHTTP_PROM module to export my stats, which seems to rely on the 'problematic' counters. Sad... Let's hope this gets resolved one day. Regards, --Sergiu On Fri, Oct 16, 2020 at 4:17 PM Joel Serrano <joel@textplus.com <mailto:joel@textplus.com>> wrote: Hey Sergiu, I think I've been through a similar case... have a look at: https://github.com/kamailio/kamailio/issues/1591 <https://github.com/kamailio/kamailio/issues/1591> in case you find it helpful. You might have to rely on the different metric that Daniel added. Joel. On Fri, Oct 16, 2020 at 3:35 AM Sergiu Pojoga <pojogas@gmail.com <mailto:pojogas@gmail.com>> wrote: Hi Daniel, You guessed right, two dmq nodes with dialog replication. Thanks, On Fri., Oct. 16, 2020, 2:15 a.m. Daniel-Constantin Mierla, <miconda@gmail.com <mailto:miconda@gmail.com>> wrote: Hello, is this happening when using dmq for dialog with multiple sip servers? Or is a single instance sip server? Cheers, Daniel On 16.10.20 04:12, Sergiu Pojoga wrote:Hi there, Every now and then, the value of active dialogs metric spikes to an erroneous huge number, messing up my otherwise beautiful grafana gauges :( Example: kamailio_dialog_active_dialogs 18446744073709551614 1602812239115 kamailio_dialog_early_dialogs 0 1602812239115 kamailio_dialog_expired_dialogs 2 1602812239115 kamailio_dialog_failed_dialogs 7361 1602812239115 Anybody faced the same symptoms, any clues what might be causing this? Thanks, --Sergiu _______________________________________________ Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org <mailto:sr-users@lists.kamailio.org> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users <https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users>-- Daniel-Constantin Mierla -- www.asipto.com <http://www.asipto.com> www.twitter.com/miconda <http://www.twitter.com/miconda> -- www.linkedin.com/in/miconda <http://www.linkedin.com/in/miconda> Funding: https://www.paypal.me/dcmierla <https://www.paypal.me/dcmierla> _______________________________________________ Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org <mailto:sr-users@lists.kamailio.org> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users <https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users> _______________________________________________ Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org <mailto:sr-users@lists.kamailio.org> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users <https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users>
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
-
Daniel-Constantin Mierla -
Joel Serrano -
Sergiu Pojoga