Prometheus 2.9 leaves time series issues behind, improves service discovery


The team behind monitoring and alerting toolkit Prometheus have improved cooperation with Kubernetes and OpenStack, amongst other gems with their latest release.

Prometheus 2.9 can now differentiate between empty and non-existing labels in Kubernetes, since service discovery now has additional metadata for those cases available. Thanks to a change in the service discovery for Consul, developers making use of HashiCorp’s SD tool in combination with Prometheus can specify multiple tags in a request.

Version 2.9 sees the addition of ProjectID and UserID meta labels to the OpenStack SD configuration and support for POST at the series endpoint with PUT been added as a supported method for the Lifecycle and Admin APIs – something that can be helpful for reloading.

Other enhancements include the automatic loading of cert files from the disk, using a fixed length millisecond timestamp format for logs, as well as performance improvements for posting.

Issues introduced in 2.8, such as the “unknown series references” warning at startup and the missing time series when reading remotely from the WAL, have remediated with 2.9

Other fixes include the ability for the promtool to resolve relative paths in alert test files, and something that makes the system more resilient to machine crashes. Other than that, scrapes are now staggered across Prometheus servers and not just within a single one, and a honor_timestamps option can be used to handle targets producing problematic timestamps.

The complete list of changes can be found in the release notes of the project. Prometheus contributor Brian Brazil used the notes to make users aware that version 2.9 uses Go 1.12, which can mean that the memory occupied by the process is reported higher than usual. “However this is harmless and the memory is available to the kernel when it needs it,” Brazil noted.

Prometheus was initially developed at SoundCloud and is protected under the Apache License 2.0, with source code available via GitHub. The tool is one of the six projects that were able to graduate the Cloud Native Computing Foundation since the organisation’s inception in 2015. Prometheus users include Docker, DigitalOcean, and Ericsson.