Google has announced general availability of Vertex Pipelines, the pipeline orchestration piece of its Vertex AI machine learning platform, previously described by Google as the backbone of the Vertex AI MLOps story.
Vertex Pipelines is designed to help Vertex AI users to scale machine learning workflows by running them as a pipeline, where each step in the pipeline is a distinct piece of the ML process. It is also key to implementing MLOps, as it enables developers and data scientists to build systems that can automatically retrain and deploy models.
Announcing its availability on the Google Cloud Blog, the cloud giant explained that in Vertex Pipelines, each machine learning pipeline step is a container, and the output of each step is typically an input to the next step.
Vertex Pipelines supports two popular open source libraries — Kubeflow Pipelines (KFP) and TensorFlow Extended (TFX) — that handle the process of converting pipeline steps to containers and managing the flow of input and output artifacts between containers.
Vertex Pipelines is also serverless and has seamless integration with other Vertex AI tools and services and Google Cloud Platform, which means that users do not have to worry about provisioning or managing the underlying infrastructure and can instead focus on building and running their pipelines. It also means that customers only pay for the resources used while their pipelines are running.
To help with building pipelines, Google offers a library of pre-built components for Vertex Pipelines, which simplify the process of using Vertex AI tools in the pipeline steps, such as creating a dataset or training an AutoML model. To use these, developers just need to import the pre-built component library and use components from it directly in their pipeline definition, Google says.
Vertex Pipelines integrates directly with the Vertex ML Metadata service to capture the output generated from each step of a pipeline execution, automatically tracking the artifacts and metrics created across pipeline runs. Pipeline metadata can be inspected using the Vertex AI console or through the Vertex AI SDK.
For further details, including instructions on building a sample pipeline, see the Google Cloud Blog.