The Google Cloud team just celebrated the beta launch of its AI Platform Pipelines feature with a couple of additions and improvements to the machine learning workflow execution environment.
The product was started to provide those at the beginning of their machine learning journey with a way to “deploy robust, repeatable machine learning pipelines along with monitoring, auditing, version tracking, and reproducibility” in an “easy to install, secure” environment. It is therefore mainly made up of the infrastructural component needed to run the workflows, as well as tools for creating and sharing pipelines.
Since the service is part of Google Cloud, it can be quickly installed via the company’s cloud console, which also takes care of access management. Options for the building pipelines part boil down to the Kubeflow Pipelines SDK, which isn’t surprising given that the AI Platform Pipelines run on a GKE cluster, and the development kit for TensorFlow Extended (TFX), TF’s end-to-end machine learning platform.
Offering support for both SDKs makes sure that the offering isn’t confined to TensorFlow users only and customers have more flexibility when looking to modify pipelines.
In the now available version, besides providing direct access to TFX templates through the AI Platform Pipelines’ Getting Started page, the service also comes with pipeline versioning opportunities. Multiple versions of a pipeline can be grouped and managed together through the UI if needed as well.
Observability meanwhile is realised through TFX’s ML Metadata library, which is supposed to automatically log artifacts, such as models or data statistics, used in each step, along with pipeline parameters, linkage across artifacts and the associated pipeline steps.
In the near future, the Google Cloud team promises to look into multi-user isolation among other things. This is meant to help control who can do what with which resources better. Other items on the todo list include support for workload identities, ways to easily set up off-cluster storage for backend data (job histories and such), and more workflow templates.