With the Elastic 7.x series concluded in early February, the team behind Elasticsearch, Kibana, and Co has entered the 8.x era of its portfolio, highlighting the importance of security and machine learning for the company along the way.

Security obviously has been a topic for Elastic for quite a while now, as investments in Elastic SIEM and various acquisitions (think CMD and build.security for instance) show. The just released version 8.0 of the Elastic stack is trying to build on that by looking into ways of making protection mechanisms more accessible. 

It approaches the issue largely by relieving teams of their setup work, enabling and configuring security features such as network encryption, user authentication and authorization automatically when self-managed versions of Elasticsearch and Kibana are started for the first time. Other than that the Elastic team took the opportunity to add some changes to protect system indices from direct access by hiding them behind a allow_restricted_indices permission set and stripping superusers from their previously standard indices write access.

Machine learning, or natural language processing to be precise, is another major theme of the release and is supposed to enhance the search experience in a couple of ways. Amongst other things, Elasticsearch 8.0 includes a technical preview for a k-nearest neighbour (kNN) search API, which is often used for things like ranking search results by relevance. Though a variant of such a search was integrated before, the new API promises faster approximate kNN searches when working with larger datasets.

Vector-based searches are also hoped to get a push through the introduction of ways to import NLP models for things like text classification and embedding directly into the search engine. The interesting part here is, that Elasticsearch decided to go with support for PyTorch models instead of popular TensorFlow — a decision that might be down to PyTorch’s interoperability investment and the fact that practitioners tend to find PyTorch a bit easier to integrate into complex systems. 

The new feature comes at a price however, as Elasticsearch requires a Platinum or Enterprise licence to upload NLP models and use the Inference processor. Other Elasticsearch improvements include updates to inverted indices and multi-dimensional point indexing, which are hoped to reduce index sizes and speed up the indexing process respectively.

Apart from that, Elastic used the 8.0 release to expand its AWS integrations by offering a new AWS Lambda application to let users ingest logs from Amazon S3 into their Elastic Cloud deployments, and an Amazon S3 Storage Lens integration to get Storage Lens metrics into the Elastic Cloud. Details are available via the Elastic documentation.