Distributed stream processing framework Apache Samza has hit version 1.5 with a simplified job submission workflow to cover more security requirements, and a way to move containers without having to restart an application.

The latter is realised by a new Container Placements API and was motivated by the problem that moving containers between hosts to improve performance always called for a restart of the affected job, while also influencing other jobs on the hot host. The APIs are meant to mitigate those issues and offer ways to restart containers with the same host preferences or selectively spin up standby for one or a subset of containers.

From an implementational point of view, the Container Placement system is largely divided into a handling and a service unit. While the handler fetches a container placement control action from the control plane, queues it up and dispatches requests on containers of a job with policy, the placement service reacts to those actions and works with the cluster manager to execute them.

The second big change of the release, the reworked job runner, had become necessary because the old implementation had made the debugging of pipelines tricky. This was mainly due to the fact that failures could be caused in multiple places that all had to be checked if something went wrong. The job runner also required some level of isolation to keep the system safe, because planning invoked user code which could cause trouble if malicious.

In its new form, the job runner doesn’t split responsibilities with the application master anymore, but simply submits a Samza job to the Samza Yarn component. Configurations necessary to submit jobs, fetch their configs, or override others have to be supplied via –config now, since the runner won’t read those from local files anymore. 

Though the new behaviour is said to be consistent with other tools in the space, such as Apache Flink and Apache Spark, it’s something Samza users will have to get used to first. And for those looking to profit from other improvements there’s no way around it really, since the change is backward incompatible, and ConfigRunner has been deprecated with the release. Tips on how to make the switch can be found in the release announcement.

Other than that, Samza 1.5 comes with support for duplicate timer registration, and new metrics for tracking the size of a container thread pool or the key and value of records written to a RocksDB store. A number of bug fixes is supposed to add thread safety for Kafka consumers, and iron out some kinks linked to container shutdowns.