Fly.io, a platform for running containers on virtual machines, has introduced Fly Kubernetes, which lets developers use Kubernetes tools while still deploying to Fly.io, which does not itself run on Kubernetes.
Fly runs Docker containers without Docker, stating that “Docker’s isolation isn’t strong enough” for high-density multi-tenant servers. Instead, the company deploys micro virtual machines (VMs) using the AWS-sponsored Firecracker project, developed by AWS for its Lambda and Fargate serverless platforms. Docker containers submitted by developers are converted to run on VMs hosted by Firecracker.
The Fly team deliberately chose not to use Kubernetes for its platform, but has customers that do want to use it. Its answer was to build an implementation of Kubernetes on top. The two key components are K3s, a lightweight Kubernetes designed for IoT and Edge computing, and Virtual Kubelet, which supports Kubernetes “nodes” that actually run pods (the unit in which containers run) on other providers, including AWS Fargate, Azure Container Instances, or in the Fly case, Fly machines.
“When you create a cluster, we run K3s and the Virtual Kubelet on a single Fly machine,” the team said. “Your cluster scales out directly using our platform, but with K8s tooling.”
Despite being resistant to Kubernetes for running its platform, the Fly team confesses that “Fly Kubernetes has some advantages over plain flyctl and fly.toml,” where flyctl is the CLI tool for Fly.io, and fly.toml the configuration file for an application. The Fly Kubernetes approach means that the deployment is more declarative, and “when you deploy with Fly Kubernetes, Kubernetes will automatically make your definitions match the state of the world. Machines go down? Kubernetes will whack them back online.”
Fly Kubernetes is currently in early access preview, with use by request.
It sounds good, but developers are thirsty for more information.
“What exactly does the combination of k3s and virtual-kubelet provide compared to standard k8s? Does it provide secret and confmap storage and namespaces and all those expected things? Can we run things like the Kubernetes dashboard? cert-manager? nginx-ingress?,” asked a developer. While not answering the complete question, Thomas H. Ptacek, a developer at Fly, said that “this is absolutely not going to let you do everything you can possibly do with K8s! Maybe we’ll beef it up over time.”
An attraction of Fly Kubernetes is that it may solve a common Kubernetes problem, which is low Node utilization, or put another way, unused compute resources that still have to be paid for.
Fly emphasises that it has no intention of migrating its underlying platform to Kubernetes. “We’re not replacing Fly.io and the Fly Machines API and the Fly Launch stuff in `flyctl` with FKS. FKS is just there for people who want a K8s interface,” said Ptacek.