The newly released Kubernetes 1.31 has completed the removal of in-tree cloud provider integrations in what team members described as the “largest migration in Kubernetes history,” though the upgrade can break existing scripts since, for example, the only valid cloud-provider argument for kubelet is now “external.”

In the past Kubernetes has included support in its core code (“in-tree”) for five cloud providers: Google Cloud, Microsoft Azure, Amazon Web Services (AWS), OpenStack, and VMware vSphere.  While this was convenient, it undermined the notion of Kubernetes as a vendor-neutral platform. The inclusion of these providers also bloated the code, as well as making it harder to update the provider code since it was baked in, and increasing the possibility of security issues.

In late 2018 an enhancement proposal, KEP-2395 was put forward to remove these built-in cloud providers. The proposal observes though that “Kubernetes users will need to add CCM (Cloud Controller Manager) deployments to their clusters. Previously, users were able to enable the cloud controller loops of the kubernetes-controller-manager through command line flags.”

The cloud providers now have documentation to support users in deploying their CCMs, such as this for AWS and this for Azure.

The complexity of the migration was because of “the numerous impacted components and the critical code paths that relied on the built-in integrations,” the Cloud Provider SIG (Special Interest Group) explained earlier this year, including having to build “four new subsystems from the ground up,” covering the CCM, API Server network proxy, kubelet credential provider, and storage migration.

The kubelet is an agent that runs on each VM (virtual machine), or node, that is part of a Kubernetes cluster.

The migration effort, according to the team, achieved dramatic results, “removing roughly 1.5 million lines of code and reducing the binary sizes of core components by approximately 40 percent.”

The cloud provider SIG was formed for the purpose of this migration and has worked on it for several years, leaving it now determining what it should do next. Suggestions include smarter hybrid deployments – where nodes can run on both private and public cloud – and “better tools and frameworks” for those developing cloud provider code.

The change should not in theory cause problems for DevOps teams since it has been well flagged. Kubernetes 1.29, first released in December 2023, aborts by default if used with a legacy built-in cloud provider, though this can be overridden. In addition, the built-in provider for OpenStack was removed in1.26, and for AWS in Kubernetes 1.27, so organizations deploying on those platforms and versions have already made the necessary changes.

The rollout of new versions of Kubernetes is a gradual process, though, and in many cases changes will be necessary. Information on how to migrate is in this official post.