Elasticsearch has updated its software stack to v6.6, improving index management, machine learning capabilities, and distributed tracing amongst other things.

The latest version of the search and analytics engine Elasticsearch itself offers frozen indices to those that are okay with a higher search latency if this means they have a higher ratio of disk storage to heap available to them. Frozen indices take up no heap so that a single node can manage more indices with very low overhead. Something that might be interesting to those in need of ways for long time online archiving. Meanwhile Elasticsearch SQL is still not completely done but it now offers support for date histograms using SQL syntax.

Once updated, Elasticsearch’s data visualisation tool Kibana gives users the option to define multiple Elasticsearch nodes, which allows request to be distributed over a cluster. The management user interface now includes a UI to add and delete remote clusters as well as check their connectivity.

There’s also an auto-follow pattern to automatically discover remote clusters that match a certain pattern and replicate them as well as an experimental way of creating and managing index lifecycle policies to automate actions such as relocating shards for time series indices. The Machine Learning UI lets users directly annotate their findings, which might help the system learn from older issues.

Anyone who ever wanted to group or copy and paste elements in Kibana’s Canvas is in luck, since a beta of those functions is included as well. Since the release of the next major version of Elasticsearch might need some preparation, Kibana 6.6 comes with an upgrade assistant to identify issues that might need special attention.

Data processing pipeline Logstash has been fitted an Index Lifecycle Management feature to control how indices are managed as they age and grow and sports the completed version of its Java execution engine, which will become the default once v7.0 has been rolled out. Additional filters include one to invoke HTTP calls with parameters from an event to external APIs and have the response inserted into the event and one to enable key/value sets and get operations at the same time.

The Beats data shipping agents also get a version bump and some breaking changes along with that. One of them is due to an alteration of syntax when working with field references (* has to be used instead of ? now, which isn’t exactly backwards compatible).

The most notable change however has to be the introduction of the System Module to Auditbeat. Though still marked as experimental, it can already be used to record system activities relating to hosts, processes, sockets, and users. Use cases for the new module include security measures, since this can for example help to find out which process opened certain connections or if a password has been modified recently.

With the release of version 6.6 of the Elastic Stack, Elastic Application Performance Monitoring on Elasticsearch Service reaches general availability status. It can be added to Elasticsearch Service deployments for things like automated anomaly detection, and helps to keep an eye on distributed workloads.

Elasticsearch Service subscribers that upgrade their deployment to version 6.6 can try a 512MB instance of the offering for free. Those already using APM regularly might find it interesting that v6.6 comes with distributed tracing capabilities, and that its agents will now automatically pick up memory metrics, system CPU utilisation, and similar basic metrics.