Rethinking Spot Instances: Solving Preemption

TL;DR:

We've made spot instances practical for all Kubernetes workloads. Architect migrates pods to new nodes before spot instances are reclaimed - with zero downtime, zero client interruptions, and zero code changes. Get 75%+ cost savings on your entire Kubernetes infrastructure while maintaining production reliability.

Join our waitlist to run your production workloads on spot instances without any compromises.

The Spot Instance Problem

Spot instances are 75%+ cheaper than on-demand instances, but they can be reclaimed with just 30 seconds to 2 minutes notice. This makes them unusable for production Kubernetes workloads.

When a spot instance is reclaimed, every pod on that node experiences:

Forced termination
Rescheduling to a new node (if capacity exists)
Full application restart and initialization
Load balancer re-registration
Client request failures during the transition

Even "stateless" workloads aren't immune. Your users experience errors, timeouts, and degraded performance every time a node is preempted. The unpredictability makes it worse - preemption rates can swing from 3% to over 60% based on time, region, and instance type.

This forces a stark technical trade-off: accept service interruptions and angry users, or pay 3-4x more for on-demand instances. Most production workloads stay on on-demand infrastructure, leaving significant cost savings on the table.

Why Traditional Approaches Aren't Enough

Kubernetes already provides some mechanisms to handle node failures - pod disruption budgets, graceful termination, and cluster autoscaling. But these are fundamentally reactive systems designed for failure recovery, not proactive migration.

Each approach hits the same technical barrier:

Extra replicas: Maintaining 2-3x capacity eliminates the 70% cost savings
Fast autoscaling: New pods still require full initialization (pulling images, application startup, loading models, etc.)
Graceful shutdown handlers: Unable to prevent the pod from being terminated in the first place
Mixed node pools: Reduces the frequency of preemptions, but every preemption still causes downtime

The core problem is that all these solutions accept that a pod must die and be recreated, and that comes with inherent downtime and service disruptions.

Enter Architect: Seamless Pod Migrations

We took a different approach. Instead of managing the chaos of preemption, we eliminate it entirely.

Architect continuously snapshots your running pods without interrupting them. This results in Architect always having the complete state of your application on-hand before anything happens. When a spot instance preemption notice arrives, we migrate your pods to new nodes before the original instance terminates. Your applications keep running, connections stay alive, and clients never notice.

This isn't a restart or a reschedule - it's true live migration for any Kubernetes workload.

How It Works

Under the hood Architect is a relatively simple state machine who's only job is to keep your application running under any circumstances. Architect fundamentally moves through four possible states:

Normal Operation: In this state Architect is continuously capturing pod state and syncing it across nodes. This is inherently why our migrations are instantaneous - we're amortizing the migrations ahead of time. During this time we're also carefully watching the resource utilization across the cluster to make sure we have enough resources to handle any sudden migrations that come up.
Preemption: When a preemption notice from the cloud provider arrives (between 30 seconds and 2 minutes depending on the provider) Architect goes to work live migrating the workloads away from the preempted nodes.
Migrations: This is where Architect earns its keep. Once we have the resources scheduled (ie. the pods themselves) have been rescheduled and all ancillary resources have been recreated (file descriptors, PVC attachments, GPU resources, etc.) it's time to move the state. This is relatively instantaneous thanks to all the work we did ahead of time.
Rerouting: Migrations aren't just supposed to be seamless for the application - they should be seamless for clients as well. To achieve this Architect gets to work updating our XDP-based routing layer to transparently reroute connections. This is done in a way so that no packets are dropped and the client can't even tell that the workload has migrated.

The magic happens in our continuous state capture mechanism. We've extended CRIU (Checkpoint/Restore In Userspace) to continuously serialize running processes to passive storage (PVCs, S3, etc.) without interrupting them or impacting performance. Think of it as streaming your application's entire memory state constantly to disk in the background.

For GPU workloads, we go even further: we serialize the complete CUDA context and GPU memory state. This means your model weights, optimizer states, gradient buffers - everything stays exactly where it was. A training run that's 47 epochs deep doesn't restart from scratch; it continues from epoch 47 on the new node. We can even migrate GPU state independent from CPU state, enabling fascinating optimization strategies.

When a preemption notice arrives (30 seconds to 2 minutes warning), we:

Provision the target node using standard Kubernetes scheduling
Stream the latest checkpoint to the new node from the passive storage layer as well as peer to peer from the previous node
Restore the process with all its state intact
Update our network layer to transparently redirect existing connections

The end result is a completely seamless migration experience - neither your clients nor your application can tell that it's now running on a completely new node. The best part is that this entire process requires no code changes, just helm install and your infrastructure is spot-ready.

See It In Action: Live Demo at KubeCon NA 2024

This isn't theoretical. At KubeCon North America 2024, we demonstrated live pod migration across multiple cloud providers (AWS, GCP, and Azure), all while maintaining active client connections:

Real Cost Impact

The math for spot instances is actually quite straightforward: spot instances consistently cost 70% less than on-demand across AWS, GCP, and Azure. For a typical production cluster running 50 m5.xlarge instances:

On-Demand: ~$6,240/month
With Architect on Spot Instances: ~$1,872/month

A monthly cost savings of $4,368 (~70% reduction).

Plus, Architect's overhead is minimal - the continuous checkpointing uses existing cluster storage, and migrations are primarily peer-to-peer with S3 acceleration only when needed. The system pays for itself if you're running more than a handful of instances.

Inherently Compatible

Architect is inherently compatible with any type of workload and fixes how your infrastructure handles spot preemptions and pod migrations:

Web Services & APIs: No 502 errors, no request timeouts, no user impact during preemptions
Databases & Caches: Maintain connections, preserve buffers, no rebalancing overhead
ML/AI Workloads: Models stay loaded, GPU state is preserved, training continues uninterrupted
Streaming Systems: Kafka consumers keep their positions, WebSocket connections survive
Batch Jobs: Complete without restart, no lost progress, no wasted compute

With Architect, every pod in your cluster can now safely run on spot instances.

The Power of Workload Mobility

The same technology that enables spot instance migration also powers:

Scale-to-Zero Without Cold Starts: Hibernate idle pods, wake them in <50ms
Node Maintenance Without Downtime: Evacuate nodes anytime without service impact
Dynamic Cost Optimization: Move workloads to the cheapest compute continuously and automatically

Only when your pods can move without interruption does your infrastructure become truly elastic.

Getting Started

Architect is currently in early access. We're working with select teams to dramatically reduce their Kubernetes costs without compromising reliability.

Join our waitlist to be among the first to deploy Architect in your cluster. We'll help you identify which workloads can benefit most and guide you through maximizing your spot instance savings.

Spot instances have always been a compromise - great prices but unreliable for production. With Architect, that compromise finally disappears.