### Description We have a couple of Kamailio containers running on AWS ECS Fargate. Using a Cloudwatch Agent sidecar container I should be able to collect metrics from the Kamailio containers and push them to Cloudwatch for monitoring.
``` xHTTP_PROM exposes the metrics like this: # Kamailio whole internal statistics kamailio_app_python3_traced_replies 0 1640265350616 kamailio_app_python3_traced_requests 0 1640265350616 kamailio_core_bad_URIs_rcvd 0 1640265350616 kamailio_core_bad_msg_hdr 0 1640265350616 ```
While other prometheus metrics I have seen are throwing metrics like this: ``` # HELP kamailio_core_shmmem_free Free shared memory. # TYPE kamailio_core_shmmem_free gauge ``` ### Troubleshooting
#### Reproduction
<!-- If the issue can be reproduced, describe how it can be done. -->
#### Debugging Data
<!-- If you got a core dump, use gdb to extract troubleshooting data - full backtrace, local variables and the list of the code at the issue location.
gdb /path/to/kamailio /path/to/corefile bt full info locals list
If you are familiar with gdb, feel free to attach more of what you consider to be relevant. -->
``` (paste your debugging data here) ```
#### Log Messages
<!-- Check the syslog file and if there are relevant log messages printed by Kamailio, add them next, or attach to issue, or provide a link to download them (e.g., to a pastebin site). -->
``` (paste your log messages here) ```
#### SIP Traffic
<!-- If the issue is exposed by processing specific SIP messages, grab them with ngrep or save in a pcap file, then add them next, or attach to issue, or provide a link to download them (e.g., to a pastebin site). -->
``` (paste your sip traffic here) ```
### Possible Solutions
<!-- If you found a solution or workaround for the issue, describe it. Ideally, provide a pull request with a fix. -->
### Additional Information
* **Kamailio Version** - output of `kamailio -v`
``` (paste your output here) ```
* **Operating System**:
<!-- Details about the operating system, the type: Linux (e.g.,: Debian 8.4, Ubuntu 16.04, CentOS 7.1, ...), MacOS, xBSD, Solaris, ...; Kernel details (output of `lsb_release -a` and `uname -a`) -->
``` (paste your output here) ```
Hello mneirynck,
in Prometheus metrics when there is no # TYPE metadata it defaults to UNKNOWN metric type. This is suitable for xHTTP_PROM because it blindly takes Kamailio registered counters and export them. We do not know if they are Prometheus counters or if they are Prometheus gauges.
Currently OpenMetrics recommend not to use UNKNOWN metric types https://github.com/OpenObservability/OpenMetrics/blob/main/specification/Ope... and Cloudwatch do not support unknown Prometheus metrics https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/ContainerInsi...
On the other hand kamailio_exporter exports a set of metrics and defines clearly if they are counters, gauges, etc. So at this moment I recommend you to use kamailio_exporter to interface with Cloudwatch https://github.com/florentchauveau/kamailio_exporter/blob/master/collector.g...
Hi, Sad we have to resort to a third party for this. Just tested the exporter and it's also failing because it starts instantly without waiting for the Kamailio container and fails because Kamailio doesn't respond in time.
So we either write our own exporter, or use a script which we run on xHTTP requests to /metrics.
We'll figure it out, but this would have been a very useful feature for us. I'll hear it if the feature ever gets added to xHTTP_PROM!
Thanks for your descriptive feedback.
Hello mneirynck,
I am thinking on a workaround. Does asigning gauge type to Kamailio's metrics even if they are actually counters would suit you?
Not sure if that would work for Cloudwatch, since gauges can go up and down while counters can't. No idea how they check this, if they check it at all.
In the meantime we've implemented our own script that exposes metrics to Cloudwatch directly, without prometheus. So if nobody else needs this feature it can go on hold.
@vhernando: do you still want this open to work on something related to it, or can it be closed based on @mneirynck comment?
Hello Daniel, I think we can close the issue.
Closed #3001.