Reliable Cross-Cloud Kubernetes on Spot Instances with Architect

What if you could just pick up a Kubernetes cluster, and move it?

At KubeCon North America 2024 we demo'd just that: live-migrating a Kubernetes cluster (running a stateful Valkey database) from AWS to GCP to Azure in seconds, with zero downtime, and with a 90% reduction in compute costs.

Felicitas Pojtinger, who's the Head of R&D at Loophole Labs (and our resident KVM whisperer), gave a talk that explored how to build Kubernetes clusters that leverage spot instances and then how to move that cluster in its entirety across cloud providers and even to on-premise infrastructure.

A link to a video of the talk is below and we’ve also posted Felicitas’ full notes from the talk to our github repo.

Architect from Loophole Labs

The solution employs a combination of open-source projects that, when used together, allow not for live VM migrations, but also maintain data integrity and network connectivity in the process. Felicitas explains how each project works and how they come together to deliver Architect, the underlying project used in the demo. The projects uses:

• PVM - Pagetable-based Virtual Machine (repo): This kernel patch delivers page table virtualization and enables running VMs within VMs without hardware acceleration.
• Firecracker: A hypervisor providing consistent VM behavior across different cloud platforms via CPU templates. Its snapshot and restore features are crucial for portability.
• Silo: An open-source storage engine optimized for tracking changes in VM snapshots, facilitating rapid migration between hosts.
• Drafter: A framework that integrates PVM and Firecracker, offering tools for VM management, networking, health checks, snapshotting, and packaging.
• Conduit: Enables live migration of network connections alongside VM state, ensuring zero downtime during migrations.

Building Reliable Cross-Cloud Kubernetes Clusters on Spot Instances

The talk is available on the CNCF youtube channel:

If you want to know more please join our Discord or follow us on LinkedIn, Twitter, and GitHub.