The Prometheus team has released v2.4 of its open-source systems monitoring and alerting toolkit, including a major update to the write ahead log (WAL) implementation of the storage layer TSDB which breaks backward compatibility with earlier versions.

Once updated, evaluation errors will be shown in the UI and users have a new set of commands for debugging and querying available in promtool. There are also limits to the samples returned by remote read endpoints and data read through remote read now. Other additions throttle the resend of alerts to one minute by default and let users send an EndsAt value for alerts to the Alertmanager. The new version also comes with a persistent alert ‘for’ state, and APIs that provide target metric metadata as well as recording and alerting rules respectively.

Prometheus 2.4 is the first release as a graduated project of the Cloud Native Computing Foundation. Maintainer Richard Hartman announced the new status at PromCom in August 2018. The newly introduced backwards incompatibility might therefore seem curious to some – after all, the graduation was supposed to signal stability, which usually means breaking changes in major releases only. Well – we’ll see if that was just a one-off.

Security check first – Prometheus installation second

Lately the project also received a CNCF sponsored security audit by Berlin-based Cure53, which uncovered five security-relevant issues – three of which were classified as actual vulnerabilities of high or medium severity. To get rid of those, stricter CORS settings, CSRF protection, and HTTP security headers would be needed.

The final report (PDF) however states, that most of the findings were dismissed as false alarms by the Prometheus team, reason being that they expect the software to run in secured environments only, shifting responsibility for security to other layers. The auditors therefore find it “extremely difficult – if not impossible -” to guarantee the safety of Prometheus’ approach. Something you might want to keep in mind when integrating Prometheus into your system.

Other than that the auditors attest the Prometheus project as “secure and clean”, even “praiseworthy” code, but advise the maintainers to adjust documentation to at least state the limitation of the system, so that it’s clear to developers and admins that security is their responsibility. They’d also like to see the team supplying guidelines on how to go about solving the “technical dilemmas” uncovered, and maybe even follow some more of the technically-demanding suggestions they came up with. On Twitter, the Prometheus project ensured incorporation of Cure 53’s suggestions into future work, without specifying next steps.