Promethean effort: Version 2.25 of monitoring system intros feature flags


Prometheus 2.25 is now available for download, bringing a number of small enhancements and a commitment to feature toggles in the hopes of making more room for experimentation.

With many orgs depending on the monitoring and alerting toolkit, adding preliminary features comes with its own set of challenges – especially if you promise API stability as Prometheus does. To prevent that promise from hindering experimentation amongst the development team, it finally decided to “use feature flags more” and introduced an –enable-feature flag. The flag hides any new additions considered breaking or experimental and can be used to turn them on for testing purposes.

As long as a feature is among the by-default disabled capabilities, users must expect its API specs and behaviour to change, and can’t trust its stability. Some of the features may also break assumptions about the way Prometheus works, for example the @ modifier, which is new in v2.25. Once enabled, it lets developers “specify the evaluation time for instant vector selectors, range vector selectors, and subqueries”, breaking the rule that queries do not look for samples ahead of evaluation time.

Another experimental feature that can be found behind the new flag is a remote-write receiver, allowing Prometheus to receive remote-write requests for use cases such as push-based federation in low connectivity environments. To take it for a spin, it has to be activated via –enable-feature=remote-write-receiver. 

Since the last release, Prometheus’ time series database has seen additional buckets for the compaction duration metric, and learned to reload blocks more frequently to detect new ones and enforce retention. Meanwhile the project’s UI now exposes the number of label pairs in the head block, which was requested for more analytical purposes, and teams can make use of a new optional name property for test groups when researching which groups of a unit test failed.

Other than that, endpoint labels metadata has been added to the Kubernetes service discovery capability. The Prometheus team also fixed some bugs in the UI and the remote feature. Details for this can be found in the release notes