A good three months after its last feature release, the team behind data collector Fluentd has finished and released v1.4 of the project.

Fluentd is a tool that uses (mostly) JSON to unify the processing of log data across multiple sources. It runs on servers, where it collects, parses, transforms, analyses and stores data (apparently 13,000 events per second and core can be processed) and is therefore meant for building logging layers.

It supports robust failover and can be set up for high availability. Memory and file based buffering should prevent data loss between nodes. Fluentd requires 30-40MB of memory and comparatively little system resources – there is however a more lightweight version available which needs ~450KB.

The project was first developed in 2011 and is currently being incubated in the Cloud Native Computing Foundation. Companies using the Apache-2-licensed software include Atlassian, AWS, Microsoft, and Nintendo.

The new version offers a way of grouping workers and supports placeholders in symlink_path parameters. It also contains a fix that lets users enable retry_forever and <secondary> at the same time. This lets the system retry forever in case errors are recoverable, and moves the unrecoverable ones to secondary output.

Speaking of which, MessagePack unpacker errors can now be found on the list of unrecoverable errors. Other than that Ruby code can now be embedded into section arguments in config and the handling of encoding parameters should work better.