# Loophole Labs > Loophole Labs builds systems infrastructure. Two products: Architect, for running Kubernetes workloads on spot with live migration and instant hibernation, and AGX, an agent sandbox platform. Loophole Labs is a systems infrastructure company. It builds two products: - Architect (https://architect.io): A Kubernetes availability platform. Live-migrates running workloads across nodes, zones, and clouds before spot instances disappear, with zero downtime and no code changes. Idle pods hibernate (scale-to-zero without deletion) and wake in under 50ms with no cold start, using checkpoint/restore at the containerd shim layer and eBPF packet buffering. Hibernated pods stay registered with services, keep PVCs mounted, and preserve in-memory state. - AGX (https://agx.so): An agent sandbox platform. Runs untrusted agent code safely, integrates with third-party systems, and forks a running machine mid-task in milliseconds, built on a high-performance in-house hypervisor. This site (loopholelabs.io) hosts the company pages and the engineering blog. This file is the complete blog — every published post, concatenated. Architect's documentation and changelog live on https://architect.io: see https://architect.io/llms-full.txt for the complete product docs in one file, or https://architect.io/llms.txt for the curated index. --- # AI Took My Coding. What's Left For Me? > Source: https://loopholelabs.io/blog/ai-took-my-coding-whats-left-for-me By Jimmy Moore, published 2026-07-03. When the thing you got good at stops being scarce, the first reaction is to assume you're out of a job. Here's why AI codegen took the typing and left the part that was always the real work: choosing the target and knowing when you've hit it. #### TL;DR: *AI codegen does the typing. Knowing what to aim at, and whether we got there, was always the actual skill. It was never about the coding.* ## ZX81 + QY10 I grew up in a strange, lucky window. The home computer revolution and the synthesizer revolution were happening at the same time, and for the first time kids my age could afford both: a computer of their own, and a keyboard of their own. Either would have been a generation-defining toy on its own. I didn't pick. My computer was a Sinclair ZX81, built around a Z80, with 1k of RAM and a membrane keyboard that fought you for every keystroke. My synth was a Yamaha QY10, a palm-sized hardware sequencer with just enough buttons to be a complete instrument. I'd already had the odd go on my brother's computer, and I'd started learning piano, so neither was entirely foreign. But both were mine, both were new, and I embraced them with equal devotion. When I learned to program, programming was a craft. You went to the library and took out a book on the instruction set. You bought magazines with programs printed in them and typed them in line by line, hoping the computer wouldn't crash before you'd saved to tape. The code was the whole thing: finding the right combination of instructions to make the machine do what you wanted. I don't think I'd heard the phrase "automated testing" until I'd been programming for ten years. Pair programming and code review were completely unheard of. The job was the code. You sat alone with the machine, made an assumption, wrote it down, measured what came out, and adjusted. Assumption, hypothesis, measurement, adjustment. A tight, often unforgiving loop, repeated until the thing worked, or until you were so tired of rebooting after a crash that you called it a night. There was nothing else to it, because nothing else had been invented yet. ## Speeding Up Creation Every creative act starts the same way, with an idea. Then we make something, hold it up against what we had in mind, adjust, and try again. Sometimes the idea shifts as we go. Sometimes the original idea turns out to have flaws we couldn't see from the outside, and we never quite get there. The QY10 broke that loop for me in the best way. Pretty quickly I realized I could ask it to play things I hadn't learned to play yet. I could write music that I, personally, could never perform. My creation loop went from "have an idea, learn to play it, record it" to "have an idea, get the sequencer to play it, record it." The bottleneck moved off my hands and onto the idea itself. ```text ┌────────────────┐ ┌────────────►│ Create │ │ └───────┬────────┘ │ │ │ ▼ │ ┌─────────────────────────────────┐ │ │ Check what we created. │ │ │ Are we widely regarded as a │ │ │ creative genius yet? │ │ └───┬─────────────────────────┬───┘ │ │ NO YES │ │ ▼ ▼ ┌─────────┴────────┴───────────┐ ┌───────┴───────┐ │ Assess how far we missed by, │ │ Have a rest │ │ adjust our aim │ └───────────────┘ └──────────────────────────────┘ ``` ## AI Coding Something very similar has happened in software development. We no longer have to type all the code we want to write. We can get AI to write it for us. When I first saw AI codegen, my first thoughts weren't positive. I felt slightly threatened. I also told myself the output would always be awful code I could never trust, which was a tidy way of not engaging with the threat in the first place. I have to admit I'd thought the same thing about compilers. I started in BASIC, wanted things to go faster, and wrote assembly. Then I started debugging programs that other people had compiled, and what I saw horrified me. The generated assembly was wasteful, inefficient, sometimes downright stupid compared to what I'd write by hand. How could I ever use a compiler if that was the kind of code it produced? Of course, I was missing the point. I was optimizing for one thing, which wasn't always the most important thing to optimize for. Iteration speed and the ability to read and debug what I'd written six months later mattered more, by miles. And the world kept moving. Hand-rolled assembly made sense on a ZX81 with 1k of RAM and a handful of cycles to play with. Once computers had thousands of times more of both, the gap between compiler output and hand-written assembly mattered less and less. ## Steering The Arrow If the aim is creating things rather than performing them, AI codegen is exactly the kind of accelerator the sequencer was for me. The bottleneck moves off the keyboard and onto the idea, and the interesting part of the work (having something to say, sketching it fast, listening for whether it works) gets faster. Think of software development, or any creative endeavor, as hitting a moving target with an arrow. Before AI, you had to do three things at once: propel the arrow, keep your eye on the target as it moves, and steer your aim toward it. AI codegen takes propulsion off your hands. What's left is the part that was always the actual skill: the aiming. Two things follow. Because AI is doing the propelling, we hit targets faster. And because we can put our full attention on steering, we're more likely to hit the bullseye when we get there. And we still pick the targets. When code is cheap, there is a strong pull to build everything that could be built. Aim at one target and you might hit the bullseye. Aim at a hundred and you will be lucky to hit the board. Resisting that pull (choosing which target is worth aiming at) matters more when shooting is cheap, not less. ![Aiming as the targets slide past](/blog/posts/ai-took-my-coding-whats-left-for-me/aiming.svg) ## How We're Using AI Codegen You might have heard we're rewriting CRIU (the Linux checkpoint/restore tool) to serve our own needs, in the language of our choice, using AI. If you haven't, [the previous post](/blog/rewriting-oss-in-the-ai-era.md) covers the why. There's a popular fantasy floating around about how this kind of work goes. You give the AI a vague target, walk away, come back, and find the arrow in the bullseye. That can work, if the target is large. "Create me a website for my company with a contact form" often does give a good first result: many different kinds of website would satisfy the user. But we're working on low-level code, picking up customer workloads, moving them, and restoring them. If we get something wrong, it doesn't just make a page look a bit jarring; it means the workload crashes. Even with the most painstakingly detailed spec, you don't hit a small target on the first shot. Assumptions shift. Sometimes the target moves once you've started hitting near it and realized you wanted something slightly different. Creative endeavor is a loop. AI doesn't change that. And if the loop stays, we still need to measure how near we got to the target, and course-correct on the next pass. ## The Ceremony Was The Work I first ran into these ideas in the early 2000s, at a bigco that was pushing them hard. I resisted all three. They sounded like ceremony designed to keep me away from the actual work, which was writing code. The dev loop I described earlier (assumption, hypothesis, measure, adjust) had been just fine without any of this, thanks. I wasn't alone. Plenty of us pushed back, and over the years the noise around them softened. Look at what I'm doing today. I'm pair programming with Claude, in a tight back-and-forth loop. I'm reviewing code constantly (Claude makes mistakes!), and most of my time goes to reading and judging the output. And before we touch the implementation, Claude and I usually build the tests together first: the acceptance criteria, written down in code. That's how we know when we've hit a bullseye and not just something that looks like one from a distance. These practices were built for exactly this situation. They came out of a generation of trying to reduce bugs, catch mistakes early, and prove that what we built matched what we set out to build. That work felt like overhead when one person sat alone with the machine, holding the whole thing in their head. With AI doing the typing, it isn't overhead anymore. AI is fast, confident, and sometimes wrong. Catching that is the work. Every practice I dismissed in 2003 turns out to be the nuts and bolts of how I get any real work done today. ```text ┌────────────────────────────────────────────────────────┐ │ │ │ func Restore(ctx context.Context) (*Proc, error) { │ │ proc, err := loadCheckpoint(ctx) │ │ if err != nil { │ │ return nil, err │ │ } │ │ return proc, proc.Resume() │ │ } │ │ │ └────────────────────────────────────────────────────────┘ ╭───────╮ ╭───────╮ │ WDYT? │ │ LGTM! │ ╰───┬───╯ ╰───┬───╯ ▼ ▼ .-----. .-----. ( o o ) ( ^ ^ ) \ - / \ \_/ / '---' '---' /| |\ /| |\ | | | | ``` ## The Sequencer Question To a musician, the advent of mechanical music and sequencers must have been perplexing. Why learn to play the piano anymore, if a machine can play it better than I can? Programmers facing AI codegen for the first time could be forgiven for thinking the same thing. When the thing you got good at suddenly stops being scarce, the natural first reaction is to assume you're out of a job. Once I had the QY10 I had far more fun telling the sequencer which notes to play (I told it to play a lot of notes in quick succession), than I would have had trying to learn to play it on a piano myself. And once I gave compilers a chance, I had far more fun telling them what I wanted in a higher-level language than I would have had writing all that assembly out myself, and I got far more done that way too. Same with AI Codegen. I'm having much more fun telling AI what to aim at next, than I would have writing it all out myself, and I'm able to iterate much faster. That's where the real skill is: everything that isn't writing the code. AI can have the typing. The aiming is mine. --- # The Stateful Spot Instance > Source: https://loopholelabs.io/blog/stateful-spot By Gerhard Lazu, published 2026-06-16. We put Valkey on an EKS spot instance without persistence. It got reclaimed. Architect moved the running container to another host with all its in-memory data intact. > We ran Valkey on an EKS spot instance without persistence. The instance got > reclaimed. The container moved to another host with all its in-memory data > intact. [Try it on a pre-configured Kubernetes cluster in the iximiuz > Labs playground](https://labs.iximiuz.com/tutorials/architect-valkey). ## The spot instance tradeoff Spot instances are up to 90% cheaper than on-demand. We use them for web servers, CI runners, batch jobs, and anything that can restart from scratch. Valkey is not one of those things. Most Valkey deployments run without persistence because speed is the whole point, so teams design around the loss. But Valkey grew beyond caching a long time ago. Teams use it for sessions, job queues, rate limiting, pub/sub, and increasingly as the [memory layer for AI agents](https://venturebeat.com/data/context-architecture-is-replacing-rag-as-agentic-ai-pushes-enterprise-retrieval-to-its-limits/) where it holds semantic facts, conversation context, and [vector embeddings](https://developers.llamaindex.ai/python/framework-api-reference/storage/vector_store/redis/). None of that can disappear on a spot reclamation, so teams keep paying full price. Stateful means on-demand. There is no technical reason a stateful workload cannot run on spot. The problem is that when the instance disappears, everything on it disappears too. So we fixed it. ## What you lose When the cloud provider reclaims your spot instance, Valkey loses everything: ``` +------------------------------------------------------------+ | SPOT INSTANCE RECLAIMED | | | | +-----------+ | | | Valkey | --- container killed --- X | | | (memory) | | | +-----------+ | | | | | +-- All in-memory data ------------ LOST | | +-- Client connections ------------ DROPPED | | +-- Pub/sub subscriptions --------- GONE | | +-- Blocked clients (BLPOP) ------- GONE | | +-- In-flight transactions -------- GONE | | | | What happens next: | | | | +-- New Valkey starts ------------- EMPTY | | +-- Every client reconnects ------- THUNDERING HERD | | +-- Cache is cold ----------------- MISSES | | +-- Upstream database ------------- SLAMMED | | | +------------------------------------------------------------+ ``` A cache warms up again in a few minutes, but a session store logs out every user, a job queue loses tasks mid-flight, and every pub/sub subscriber disconnects and misses messages until it reconnects. Then there are the connections: they all drop. Every client has to reconnect, re-authenticate, re-subscribe. Most teams just avoid spot entirely and run Valkey on on-demand. Safe, but 2-3x the cost. Reserved instances claw most of that back, betting you'll still want that same instance one to three years out. The rest design around the loss: cache invalidation strategies, read-through patterns, warm-up scripts that pre-populate data after a restart. That works for simple caches, but breaks when Valkey holds sessions, job queues, or agent memory that cannot be reconstructed. Attaching a persistent volume looks like the fix, but EBS volumes are locked to one availability zone and slow to re-attach to a new node. Durable storage surrenders the cross-zone diversity that makes spot cheap, and still costs you minutes of downtime. Either way, a spot reclamation hurts. Spot reclamations are not sudden crashes though. The cloud provider warns you first: 2 minutes on AWS, 30 seconds on GCP and Azure. That does not sound like much, but [Architect](/architect) needs about 10 seconds to migrate most workloads (it's usually down to network throughput and amount of data that needs to be transferred). The source node needs to be alive for those seconds. A kernel panic or a sudden power loss takes the process with it, and no migration can help. But sudden death is the minority: spot warns you, drains have grace periods, autoscalers cordon before scaling down. Mature infrastructure usually says goodbye before it goes. ## 30 seconds is plenty When a Kubernetes node gets a spot reclamation notice, Architect moves the container to another spot node with all its in-memory data. The container pauses on one host and picks up on another. Yes, a checkpoint travels between the nodes, but it carries the entire running process, not a data dump that a fresh Valkey has to load. ``` +------------------------------------------------------------+ | CONTAINER MIGRATION | | | | Spot Instance (Host A) Spot Instance (Host B) | | | | +--------+ migrate container +-------------+ | | | Valkey | =========================> | SAME Valkey | | | +--------+ memory + data | container | | | +-------------+ | | | | Everything in memory preserved. Zero data loss. | | | | Clients reconnect to a warm store, not an empty one. | | | +------------------------------------------------------------+ ``` Clients reconnect, but to a store that already has everything. No cold cache, no thundering herd against an empty Valkey. Compare that to a traditional recovery. We will even be generous and assume RDB persistence was on, so only the writes since the last snapshot are lost: ``` TRADITIONAL RECOVERY (RDB SNAPSHOT) Detect Re-attach Load RDB failure EBS (PVC) snapshot ~5s ~30s ~5s |----------|------------------------------------------------------|----------| 0s ~40s ARCHITECT MIGRATION Freeze Transfer state Resume <1s ~3-8s <1s |---------|---------------|--| 0s ~10s ``` ## Try it yourself > Do not take our word for it. Run it on your own EKS cluster with [the quick > start](/docs/quick-start.md), or [try it on a pre-configured cluster on the > iximiuz Labs playground](https://labs.iximiuz.com/tutorials/architect-valkey). > Write your own data, trigger a migration, and see what survives. Try to break > it. If you manage, we are most curious to [read all about > it](mailto:contact@loopholelabs.io?subject=I%20broke%20Architect). A fresh EKS cluster takes 20+ minutes and costs money. The iximiuz Labs playground is free for up to 1 hour, ready in 3-5 minutes, and just requires GitHub authentication. One caveat: it does not run real spot instances. You drain the node yourself, the same eviction a spot interruption triggers. The migration is real, only the eviction notice is simulated. If you want to try this on your Kubernetes cluster, Architect works best on EKS with AL2023 nodes. On GKE, use the Ubuntu node image. Other Kubernetes distributions may work, but I cannot promise it: the installer integrates tightly with containerd, and distributions that relocate it, like k3s, need manual surgery first. You also need Kubernetes 1.33+ and at least 2 nodes, or there is nowhere to migrate to. Add your cluster in [the Console](https://console-v2.architect.io/) and it hands you a pre-filled `helm install`. The manifest below gives you migration with the data intact, plus hibernate and wake. Start a single Valkey instance with no persistence and no replicas. The annotations are the entire integration: Architect manages the container, hibernates it when idle, and network monitoring wakes it on traffic: ```bash title="Deploy Valkey, no persistence, no replicas" kubectl apply -f - < Source: https://loopholelabs.io/blog/scale-to-zero By Paul Lundin, published 2026-06-05. Eliminating cold starts while scaling Kubernetes pods to zero has long been incredibly complex, Architect makes it easy. ## The Cold Start Problem Imagine Kubernetes pods being turned on and off as easily as flipping the light switch when you enter or leave a room. No scheduling latency or start up delay, just instantly on like the lights. Many discussions make it appear as if scaling pods to zero is a solved problem and operates as simply as the light switch. This is misleading at best. Sure Kubernetes has had the primitives and vendors have made promises for years now. You set up an autoscaler, traffic drops, pods disappear, your bill shrinks. Simple, right? Except it is not simple, because scale to zero has a dirty secret in many cases: the moment traffic returns, you are starting an application from scratch. Your pods have been deleted. Kubernetes has to reschedule them onto a node, pull the container image, run init containers, wait for the application runtime to initialize or database cache to be warmed up, and serve a health check before a single request gets through. For a Go service, that might be a few seconds. For a Database or JVM-based application like Spring Boot or Kafka, you are looking at closer to a minute or more. The experience of flipping a light switch and waiting for 30+ seconds for the lights to come on? That is the cold-start of a Kubernetes pod, and it drives all sorts of architectural and efficiency anti-patterns. That gap between zero and ready is the cold start, and it is the real cost of conventional scale-to-zero. For plenty of workloads the cost and complexity are high enough that teams just leave pods running. They end up paying for idle capacity around the clock to avoid the user-facing latency that comes with every wake cycle. Architect was built around a different premise: pods should go to zero, but they should never cold start. ## What Actually Causes Cold Starts When most scale-to-zero tools bring a pod back, they recreate it from nothing. The scheduler finds a node, the runtime pulls the image, the application boots, and only then does traffic flow. This is not a configuration problem. It is a consequence of the underlying model: the pod was deleted, and something new has to be created in its place. The sequence looks like this: ```mermaid sequenceDiagram participant U as Incoming Traffic participant S as Kubernetes Scheduler participant N as Node Runtime participant A as Application U->>S: Request arrives (no pods running) S->>N: Schedule pod onto node N->>N: Pull container image N->>A: Run init containers A->>A: Application init A->>S: Health check passes S->>U: Ready to serve ``` Some tooling tries to paper over this with techniques like pre-warming pools or keeping a "shadow" replica running. Those approaches trade cost for latency, and they still require you to predict demand ahead of time and provision accordingly. You are back to over-provisioning, just with extra steps. The fundamental issue is that rescheduling is relatively expensive, and any approach that relies on it will have a cold start floor that application teams have to architect around (*ba-dum-tss*). ## Hibernation, Not Deletion Architect does not delete pods. It hibernates them. When a pod has been idle (for a configurable duration), Architect checkpoints the running container and preserves its execution state to local node storage, and reduces the pod's resource requests to zero. The pod remains registered with the Kubernetes API, stays attached to its Services, and keeps its PVCs mounted. This subtle difference between a hibernated pod and a deleted pod is actually quite important because from Kubernetes' perspective the pod still exists but from the node's perspective it is consuming no CPU or memory. If this sounds like much better bin packing, that's because it is. When traffic arrives, Architect restores the container from the checkpoint in under 50ms. The application picks up exactly where it left off. No image pull, no init sequence, no JVM warmup. ```mermaid sequenceDiagram participant U as Incoming Traffic participant A as Architect participant P as Your Pod Note over P: Idle threshold reached A->>P: Checkpoint container state A->>A: Set resource requests to zero Note over P: Status: SCALED_DOWN U->>A: Packet arrives A->>A: Buffer incoming packets A->>P: Restore from checkpoint (~50ms) A->>A: Release buffered packets A->>U: Ready — zero packets dropped ``` Packets that arrive during the wake cycle are buffered and delivered once the container is running. No requests are dropped. No timeout errors surface to the client. ## How Architect Fits Into Your Cluster Architect runs as a DaemonSet so the Architect agents run on each labeled node and handles the checkpoint, hibernate, and wake operations locally. A cluster-level control plane coordinates checkpoint locking during pod migrations, while the checkpoint data itself transfers daemon-to-daemon. You can learn more about the Architect components in the [How It Works](/docs/how-it-works.md) section of our docs. ```mermaid graph TB subgraph "Each Node" Pod[Your Pods] D[Architect Daemon] end subgraph "Cluster" API[Kubernetes API] AC[Admission Controller] CP[Control Plane] end Pod <--> D D --> API D --> CP API --> AC ``` Because there are no external analytics platforms and no separate optimization layer making decisions about your cluster, running Architect is simple and well integrated into native Kubernetes operational patterns and tooling. Hibernation and wake are handled by `architectd` on the node where the pod lives. The decision logic is: if the pod has been idle for `N` seconds, hibernate it; if traffic arrives, wake it. You set the threshold. Architect executes it. ```mermaid sequenceDiagram participant Pod as Your Pod participant A as Architect Note over Pod: Idle threshold reached A->>Pod: Checkpoint and stop container A->>A: Buffer incoming packets A->>A: Reduce pod resources to zero Note over Pod: Status: SCALED_DOWN ``` For a detailed view of configuration options take a look at the [configuration](/docs/configuration.md) or [examples](/docs/examples.md) section of our documentation. ## No Code Changes Required Enabling Architect on a workload is a two-line annotation change to your pod spec. There is no SDK to import, no sidecar to configure, no application-level checkpoint logic to write. ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: my-service spec: template: metadata: annotations: architect.loopholelabs.io/scale-down-timeout: "60s" spec: runtimeClassName: runc-architect containers: - name: my-service image: my-service:latest ``` Two things to note here. `runtimeClassName: runc-architect` tells the node to run this container through Architect's shim, which is what enables checkpoint and restore. The `scale-down-timeout` annotation sets how long Architect waits after the last observed activity before hibernating. Both are pod-spec concerns. Your application code is untouched. If you are running StatefulSets, the same annotations apply. PVCs stay mounted through the hibernation cycle, so your stateful workloads wake with their storage intact. For workloads where you want explicit control over checkpoint lifecycle, Architect exposes a `PersistentCheckpoint` CRD. This lets you name and manage checkpoints directly, which is useful for workflows where you want a known-good restore point rather than the most recent automatic checkpoint. ## Use Case: Developer Environments Most developer environments are a textbook idle resource problem. Your staging clusters, feature branch environments, and internal tooling are heavily used during the business day. Then the team goes home, and those pods run until morning or over the weekend, burning compute for nothing. The conventional answer is to write a cron job or similar that scales deployments to zero at night and back up in the morning. This works, but the first engineer who opens a staging tab at 08:00 before the scale-up job runs gets a 60-second wait while Spring Boot remembers who it is. If that engineer is in a different timezone, or they work a non-standard schedule, the friction compounds quickly. With Architect, the environments hibernate automatically when idle. When the first requests hit in the morning the pod wakes in under 50ms. The environment is ready before the browser tab finishes rendering. The experience is similar to that light switch I mentioned earlier than it is to other scale to zero solutions which rely on cold start of pods. This works across the full shift spectrum. A team distributed across UTC-8 to UTC+5 can have a single staging environment that is effectively always available and never burning idle compute. Engineers in later timezones do not pay a cold start tax because the environment went quiet while the earlier timezone slept. The setup is the same two-line annotation. For a cluster of developer environments, you might set a tighter idle threshold: ```yaml annotations: architect.loopholelabs.io/scale-down-timeout: "300s" # 5 minutes idle ``` ## Use Case: Spiky / Periodic Production Traffic Retail holidays, ticket sales, and any platform with highly fluctuating sale patterns share a common characteristic: demand is predictable in aggregate but unpredictable in timing. You know a sale is coming. You do not know exactly when the load will spike, or how steep the initial ramp will be. The conventional response is to pre-provision capacity before the event. You scale up replicas early, run them through the quiet period, and scale back down when you think the spike passes. This is safe (if you know the demand) and expensive. An alternative that does not work well is relying on reactive autoscaling to provision new pods during the spike. By the time the autoscaler detects the load increase, schedules new pods, and waits for them to pass health checks, you have already dropped requests or served degraded latency to the customers driving the most revenue in your quarter. Architect gives you a third option. You can maintain a pool of hibernated replicas that are ready to wake on the first packet. The replicas exist in the cluster and are registered with your Services. Their resource cost at rest is zero. When the spike hits, they wake on demand without a scheduling or initialization delay. ```mermaid sequenceDiagram participant HPA participant RS as ReplicaSet participant Pod as Hibernated Replica participant T as Traffic HPA->>RS: Scale up to N replicas RS->>Pod: Create / maintain replica Note over Pod: Hibernated by Architect T->>Pod: First packet of spike Note over Pod: Wake from checkpoint Pod-->>T: Response ``` Architect works alongside HPA without conflict. HPA manages how many replicas exist. Architect manages the resource cost of idle replicas. They operate at different layers and do not interfere with each other. For a seasonal retail workload, a reasonable pattern is to scale replica count up before an anticipated event using HPA or manual scaling, let Architect hibernate the extra replicas, and then let the live traffic wake them on demand as the spike materializes. You get burst capacity without the cold start risk and without paying for active compute on standby pods. ## TL; DR: Other Patterns Do Not Work at Scale It is worth being direct about why some common approaches to scale-to-zero hit a ceiling. **Deletion-based scale-to-zero** treats pods as disposable. Every wake cycle is a full pod creation: schedule, pull, init, boot. The cold start latency is baked into the architecture. For stateless micro-services with fast boot times this is sometimes acceptable. For anything stateful, JVM-based, or with meaningful initialization logic, it is not. **Analytics-driven optimization platforms** require you to instrument your cluster, pipe usage metrics into an external system, wait for the system to build a model of your workload patterns, and then act on recommendations the platform surfaces. The operational overhead is real and annoying with these platforms. Perhaps importantly it is also often the case that the decision logic lives outside your cluster. When something goes wrong at 2AM, you are debugging across two systems. These platforms can be useful for cost visibility and maybe even rightsizing, but they are not solving the cold start problem. **Pre-warming and shadow replicas** keep a warm instance running to absorb the first request while additional capacity scales up behind it. This eliminates cold starts at the cost of eliminating scale-to-zero. You are always paying for at least one active replica. **Architect's model is more elegant**: the pod never leaves, it just stops consuming resources. No rescheduling, no external system, no permanently warm replica. The checkpoint is the mechanism. The hibernated pod is the warm standby that costs nothing. *** ## Ready for an Architect POC? The full [Quick Start](https://loopholelabs.io/docs/quick-start) walks through installation, node labeling, and deploying your own workloads with Architect annotations. If you are evaluating Architect for a specific workload or want to talk through whether it fits your architecture, [schedule a demo](https://cal.com/team/loopholelabs/architect-intro-demo) with the team. --- # The Math Changed: Rewriting Stale Open Source In The AI Era > Source: https://loopholelabs.io/blog/rewriting-oss-in-the-ai-era By Jimmy Moore, published 2026-05-22. Rewriting an established open source project used to take years. With AI, that's no longer true. Here's why we're rewriting CRIU in Zig for our own use, what makes infrastructure projects go stale in the first place, and why the rebuild-vs-contribute math might be worth a second look. #### TL;DR: *Until recently, rewriting an established open source project could take years. With LLMs, that's changed. We're rewriting CRIU in Zig, and expect it to be complete in months, not years.* ## Reaching For The Obvious Tool Anyone who's done home renovations knows there's a tool for every job. Over the years you end up with a shed full of specialized things that each do one job well. Tools you might only need a handful of times, but when the moment comes, nothing else will do. ```text My house Tools to fix up my house ════════ ════════════════════════ ┌───────────────────────────┐ │ hammer saw drill │ │ chisel awl pliers │ │ level square axe │ ╱╲ │ jigsaw sander oil │ ╱ ╲ │ crowbar plane vise │ ┌────┐ │ tape wrench ratchet │ │ ▢▢ │ │ spanner nailer awl │ └────┘ └───────────────────────────┘ ``` Software is the same story. As we build, we reach for the specialized tools that let us move faster and ship better. And both kinds of tools need maintenance. A rusty saw is more pain than it's worth when you actually need to cut wood, and a stale software tool can cost you more time than it ever saved. The catch with software is that you usually aren't the one keeping it sharp. Someone else is, and you're trusting that they still are. The standard advice, when you notice your tool getting dull, is to be a good open-source citizen: file the bug, write the patch, send the pull request. Most of us have done it, and the reciprocity is what's kept the ecosystem running. But that model is starting to show strain. Maintainers are quietly drowning in AI-generated patches and bug reports that look right at a glance and [burn](https://daniel.haxx.se/blog/2025/07/14/death-by-a-thousand-slops/) [hours](https://www.phoronix.com/news/Torvalds-Linux-Kernel-AI-Slop) under [review](https://www.qemu.org/docs/master/devel/code-provenance.html). The contribute-upstream economics worked when humans wrote thoughtful patches for humans to review, at roughly comparable speed. When one side of that equation runs at machine speed and the other doesn't, the whole loop breaks down. It's worth asking whether the model still works at all. Checkpoint/restore is what Architect does for a living. We take our customers' running Linux processes - their memory, their threads, their open files and sockets - freeze them, pick them up, and bring them back to life on a different machine. There is exactly one tool on Linux that does this: [CRIU](https://criu.org/Main_Page). It's been around for more than a decade, it's wired into runc, podman, and the wider Kubernetes ecosystem, and it's the obvious thing every team reaches for first. So we reached for it. ## How Open Source Projects Go Stale The issues we kept running into - and the workarounds we kept piling on top of each other - weren't really CRIU problems. They were structural ones, the kind every long-lived open source project eventually runs into, and no amount of patching around the edges was going to fix them. People have passions - almost obsessions. We've all had things in our lives we were utterly obsessed with for a while, and then just grew away from. Maintaining that intensity over years, let alone decades, is genuinely hard. At the start of a project, everything is new. Every commit moves the needle, the mental model fits in your head, and progress is fast and fun. As the project matures, the work shifts. More of your time goes to maintenance, regression hunts, and keeping the existing code from rotting under you. Back to the renovation analogy: we've all got the project on the shelf that died at 90%, because the last 10% was the unfun part. When code is written by hand, people only have finite energy to push things forward. The original driving forces eventually burn out and move on, and the project keeps running on whoever's left. ### The World Doesn't Stop A long-lived project also exists in a world that keeps changing around it. When CRIU started in 2012, Go had just hit 1.0. Rust was three years from its first stable release. Zig wouldn't exist for another four. The Linux kernel didn't have io\_uring. eBPF as we know it didn't exist. Containers were a niche curiosity rather than the substrate of half the world's compute. Build systems, CI, package management, testing conventions, documentation tooling - all of it has moved, often more than once. When a project doesn't keep refactoring toward its environment, the world around it slowly drifts out of fit. The tells are subtle. CRIU's homepage is still a MediaWiki install in 2026. It's the kind of detail that doesn't matter on its own, but reads like wallpaper from the wrong decade. It quietly marks when the project stopped renovating. ```text ┌──────────────────────────────────────────────────────────────────────────┐ │ File Edit View Help │ ├──────────────────────────────────────────────────────────────────────────┤ │ ◄ ► ⟳ ⌂ http://geocities.com/~user/website-i-have-not-updated-lately │ ├──────────────────────────────────────────────────────────────────────────┤ │ │ │ ★ WELCOME TO MY HOMEPAGE ★ visitors: [ 0 0 0 0 4 2 7 ] │ │ ⚠ UNDER CONSTRUCTION ⚠ [] │ │ │ │ webring: << prev | random | next >> last updated: April 1998 │ │ sign my guestbook! best viewed in Netscape │ │ │ └──────────────────────────────────────────────────────────────────────────┘ ``` Some of that drift can be patched over. CRIU did get bolted into runc, podman, and Kubernetes after they emerged. If the existing core exposes a reasonable interface, new orchestration layers can sit on top of it without surgery. Other kinds of drift can't be patched at all. You don't migrate an established C codebase to Rust or Zig with a few pull requests. The implementation language, the memory model, the on-disk formats, and the core architecture all freeze the moment they're chosen. By the time the world has moved on, the cost of moving with it approaches the cost of starting over. ### Big Things Don't Innovate There's a third force at work, and it's structural. Big companies don't innovate. Not because the people inside them lack ideas, but because the company has been optimized, often deliberately, to reduce risk and protect existing revenue. Innovation is by definition risky and disruptive, and a system tuned for stability rejects it the way an immune system rejects a foreign cell. Mature projects work the same way. Once there are real users, a downstream ecosystem, and a core group that's been around a long time, the dominant force becomes "don't break anything." Reviewers get cautious. Scope tightens. Ambitious patches die in discussion. The project quietly converges on a position: keep it alive, keep it stable, don't touch the bottom layer. Big innovation stops being something anyone can push through, regardless of whether they wanted to. In the corporate world there's at least an escape valve. Big companies let startups take on the risk, wait until the idea proves out, and acquire the result. The innovation happens outside, and the incumbent buys it once it's a known quantity. Open source has no equivalent. Large mature projects rarely absorb small innovative ones. A successful alternative implementation has to stand on its own; it doesn't get merged into the thing it was built to replace. Meaningful innovation in mature OSS isn't an internal process. It's an external one. The pattern is everywhere once you look for it. Cassandra didn't get built inside the MySQL codebase. CockroachDB didn't get built inside Postgres, even though it speaks the Postgres wire protocol, uses Postgres-compatible SQL, and targets users who outgrew single-node Postgres. It's the thing Postgres would have become if Postgres could have evolved in that direction internally, and instead it had to be built outside. The modern database landscape (Cassandra, Redis, ClickHouse, DuckDB, Tigerbeetle, and dozens of others) is what happened when people wanted something different and the incumbents weren't going to absorb it. None of it got folded back into MySQL or Postgres. The incumbents keep running on whoever's left. ## Just Write Your Own Software development has always had a tension between reaching for an existing tool and rolling your own version. Both extremes have their famous failure modes. On one end you pull in hundreds of dependencies for trivial things, and find out six months later that one of them has been hijacked or unpublished. On the other you refuse to use ready-made solutions for anything, and spend months rewriting parsers and serialization libraries that the world has had for a decade. What's kept the second extreme in check, at least until recently, is speed. Rolling your own version of an established tool is expensive. The problem space is large, the edge cases are hostile, the long tail of platform quirks goes on forever. Even if you knew everything you needed to know to rebuild a tool, the months or years of typing got in the way. Most of the time it was cheaper to live with the tool you had than to build the one you wanted. Building from scratch was reserved for the desperate or the well-funded. ## How AI Changes The Math ```text ✻ Welcome to Claude Code cwd: ~/linux ╭─────────────────────────────────────────────────────────────╮ │ > Rewrite the linux kernel in rust on pain of death. │ │ I'll be back in 10 mins and expect it to boot. Only use │ │ unsafe if you really have to, or if you're just feeling │ │ adventurous. │ ╰─────────────────────────────────────────────────────────────╯ [10 mins later...] ╭─────────────────────────────────────────────────────────────╮ │ > Sorry I forgot about rust compile time. I'll be back in │ │ 2 days. │ ╰─────────────────────────────────────────────────────────────╯ ``` The barriers to rewriting an established project (in a different language, with different priorities, for a more specific purpose) are essentially gone. The most interesting thing isn't that AI can write code; it's that the existing project becomes a usable reference. Decades of accumulated decisions, edge cases, and platform quirks used to be the wall blocking any rewrite. Now they're a specification you can read against your own version. The incumbent stops being a prison and starts being documentation. What you build instead can be optimized for your particular usage rather than every possible usage. Off-the-shelf tools solve a wide range of problems fairly well. A purpose-built tool can solve your problem really well, free of the cruft that accumulates over years of being everything to everyone. Maybe you want it in a different language. Maybe you want to integrate it with the rest of your stack in ways the original architecture forbids. Maybe you want a different memory model, on-disk format, or concurrency story. All of these are now within reach. What AI doesn't do is the architecture or the judgment. The hard parts of building infrastructure still come from humans: what to expose, what to hide, where the abstractions break, which tradeoffs are real and which are folklore. But the months of typing, the long tail of platform quirks, the exhaustive test matrices, the boilerplate around the actual ideas: that's what's gotten cheap. Removing it doesn't make rewriting trivial. It makes rewriting finite. ## Rewriting CRIU In Zig Live migration is core to what we build. Architect's whole value proposition (moving running workloads across clouds, across nodes, across hardware) depends on checkpoint/restore being fast, predictable, and ours to fix when it breaks. We need to optimize every step of the process, identify and eliminate bottlenecks (almost always disk or network at our scale), and ship a fix the same day we find the bug. CRIU gets us most of the way there. To be clear: it's a serious piece of software, with more than a decade of careful work, deep kernel integration, and a real engineering effort we have a lot of respect for. But "most of the way" isn't enough when checkpoint/restore is the foundation of your product. And working downstream of a large project with hundreds of open issues unrelated to your needs (issues you can't ignore, because they touch the same code paths your product depends on) is a tax that doesn't go away. So we're rewriting CRIU in Zig - not as a fork, not as an upstream contribution, but as a different tool aimed squarely at our customers' problems. Why Zig (and not Rust, or C, or Go) is a post of its own; stay tuned. ## How The Rewrite Works ```text STEP 1 Build a robust test harness against the target project, covering ONLY the behavior we want to replicate. │ ▼ STEP 2 Start the rewrite with the language and tooling that makes the most sense today, using the target project as the specification. │ ▼ STEP 3 Iterate until every test passes. ``` ## How's It Going? It's early days, but the initial signs are good. We can ensure complete correctness against the slice of CRIU we care about, ignore everything we don't, and move fast. We already have a development spike that checkpoints and restores things CRIU can't currently handle (including processes using io\_uring), and we expect to drop in our replacement by summer. What we'll have is our own easy-to-maintain checkpoint/restore codebase, in Zig. Explicit allocators give us predictable memory behavior, comptime replaces brittle macros, and zero-cost C interop lets us reach into existing systems libraries directly. We'll be able to iterate fast on issues, and reach areas of the problem space that CRIU hasn't gotten to through its slower issues process. And we'll be able to integrate tightly into our specific use case, optimizing the things our customers actually care about most. An LLM rewrite of open source tailored to your use case isn't going to be the right answer for everyone. But the math has changed. For the first time, the cost of building exactly what you need might be lower than the cost of adapting to what already exists. --- # Ollama Doesn't Know Its GPU Is on Another Machine > Source: https://loopholelabs.io/blog/ollama-remote-gpu By Niklas Eiling, published 2026-05-11. No GPU. No CUDA drivers. No code changes. We ran large language models inside a container with zero GPU access by intercepting CUDA calls at the loader level and forwarding them to a remote DGX Spark. Ollama never knew. We started an Ollama container on a MacBook. There's no NVIDIA GPU, no CUDA toolkit, and macOS doesn't even have CUDA drivers. Ollama found an NVIDIA GPU anyway: A 128 GB Blackwell GPU on a DGX Spark across the network. GTAP enables this by intercepting CUDA calls and forwarding them to a remote server. It takes one command, requires no code changes, and the application has no idea. Here's what that looks like: ## Seeing it in action * [Running Ollama on a MacBook with no GPU via GTAP](/blog/posts/ollama-remote-gpu/demo-client.mp4) * [GTAP server TUI showing GPU utilization during remote Ollama inference](/blog/posts/ollama-remote-gpu/demo-server.mp4) The setup is two machines: a **[DGX Spark](https://www.nvidia.com/en-us/products/workstations/dgx-spark/)** (NVIDIA's Grace Blackwell workstation, 128 GB of unified GPU memory) running the GTAP server, and a **MacBook** running Docker Desktop with no NVIDIA hardware and no CUDA installation. With GTAP installed on the MacBook, starting Ollama is one command: ```bash $ gtap docker run \ --name ollama-demo --rm \ ollama/ollama ``` GTAP acquires a lease on the remote GPU, starts the container, and injects an interceptor. Ollama initializes, discovers "its" GPU (actually the DGX Spark's GB10), and begins serving. The first command starts Ollama's server inside the container. To load a model and interact with it, you open a second terminal and run `ollama run`: ![Running llama3.1:8b via GTAP. The model pulls and generates a response, all computed on the remote GPU.](/blog/posts/ollama-remote-gpu/llama3-terminal.png) Every token is computed on the DGX Spark's GPU. Only the generated text crosses the network back to the MacBook. Token generation is fast and interactive. Want to go bigger? The DGX Spark has 128 GB of memory. This is enough for [`mistral-large`](https://ollama.com/library/mistral-large): ![Running Mistral Large at 123 billion parameters. 73 GB of VRAM, streamed from the remote DGX Spark.](/blog/posts/ollama-remote-gpu/mistral-large-terminal.png) Mistral Large at 123 billion parameters requires roughly 73 GB of VRAM. The response streams back to the MacBook in real time, token by token. On the server side, GTAP's terminal UI shows live GPU utilization, RPC call rate, network bandwidth, and scrollable logs. During inference, you can watch GPU utilization spike every time a response is generated, confirming that the compute is happening remotely. GTAP doesn't know or care what model Ollama is running. It just forwards CUDA calls. We've verified 48 models across 15 families, from SmolLM2 at 135M parameters to Qwen3.5 at 122B. All of them work without changes. You may have noticed that the video uses a [custom container image](/blog/posts/ollama-remote-gpu/Dockerfile.txt) instead of the stock `ollama/ollama`. Since GTAP intercepts all CUDA libraries at the loader level, the container doesn't need a CUDA distribution at all. We strip CUDA from the image entirely, bringing it from 8.7 GB (`ollama/ollama:0.17.7`) down to 1.2 GB. The difference is just the toolkit you no longer have to ship. Removing the NVIDIA Container Toolkit from the host also eliminates a [recurring](https://nvidia.custhelp.com/app/answers/detail/a_id/5582) [source](https://nvidia.custhelp.com/app/answers/detail/a_id/5659) of container escape vulnerabilities. ## Why this matters GPUs are expensive and hard to get. Most development machines don't have one, and cloud GPU instances bundle hardware you don't need with long-term commitments you can't avoid. GTAP turns a GPU into a network resource. Nothing about your application changes. The GPU just shows up wherever you need it: a fleet of laptops sharing a single GPU server, Kubernetes pods on nodes without NVIDIA drivers, or a CI runner that never had a GPU installed. None of this requires code changes, special drivers, or a CUDA installation on the client. ## How this works GTAP decouples the GPU from the machine that runs the application. The approach is called *API remoting*: GTAP intercepts CUDA API calls at the loader level and forwards them over the network to a server that has the actual GPU. The application has no idea this is happening. ![CUDA calls are intercepted locally and forwarded to the remote GPU over the network.](/blog/posts/ollama-remote-gpu/gtap-flow.svg) This is not virtualization in the hypervisor sense, and it isn't the same as GPU sharing approaches like [MIG](https://docs.nvidia.com/datacenter/tesla/mig-user-guide/), [vGPU](https://docs.nvidia.com/vgpu/latest/grid-licensing-user-guide/index.html), or PCIe passthrough, which all partition or assign a GPU to VMs on the *same physical machine*. There's no virtual GPU device, no special driver, no modified application. GTAP operates below the application, at the boundary between the CUDA shared libraries and the process that calls them. When an application calls `cudaMalloc`, GTAP locally serializes the call and sends it over the network to the server. The server executes the real `cudaMalloc` on the physical GPU and returns the result. The application receives a valid device pointer and continues as normal. Getting that interception to work cleanly turns out to be the tricky part. The common way to intercept library calls on Linux is `LD_PRELOAD`, which loads a wrapper library before the real one. The problem is that `LD_PRELOAD` hides symbols but doesn't replace the underlying library. CUDA still has to be installed locally, and the real CUDA libraries still initialize, which can cause conflicts when there is no actual GPU. GTAP uses a different mechanism: Linux's `ld.so` audit interface (`LD_AUDIT`). When the loader loads a CUDA library, GTAP's audit module redirects it to its own implementation and replaces CUDA functions with its own stubs. The original CUDA libraries never load locally at all. GTAP currently supports the CUDA Driver API, Runtime API, cuBLAS, cuSPARSE, cuDNN, cuFFT, nvJPEG, and NVML, covering the vast majority of the CUDA ecosystem. ## Overhead The obvious concern with API remoting is latency. Every CUDA call crosses the network, so how does this not hurt performance? The key is that most CUDA calls are asynchronous. Kernel launches, `cudaMemcpyAsync`, and other stream operations return immediately to the application. GTAP sends them to the server asynchronously, maintaining execution order without blocking the application. The application keeps submitting work while previous calls are still in flight on the GPU. Network latency only becomes visible on *synchronization points*: calls where the application explicitly waits for the GPU to finish. The main ones are `cudaStreamSynchronize`, `cudaDeviceSynchronize`, and the synchronous variants of `cudaMemcpy`. These block until all in-flight work on the relevant stream completes, and that round trip is where you pay the network cost. Well-written CUDA applications minimize synchronization. They submit large batches of work, overlap compute with data transfers using multiple streams, and only synchronize when they need results. [Research on CUDA API remoting](https://doi.org/10.1002/cpe.6474) has shown that these workloads see negligible overhead on a low-latency network. For Ollama, each generated token requires at least one synchronization to read back the result, so each token pays at least one network round trip. On a low-latency network this is still very usable, but token rate is sensitive to network latency. Fewer synchronization points mean less overhead. vLLM, for instance, achieves higher token rates over GTAP for exactly this reason. Beyond latency, there's also bandwidth to consider. Loading a model means transferring its weights to GPU memory, and a naive implementation would send them over the network every time. For Llama 3.1 8B, that's roughly 4.7 GB. For Mistral Large, it's around 73 GB. Over a gigabit Ethernet link, that's 38 seconds and nearly 10 minutes respectively. GTAP avoids this entirely. The GTAP server caches model weights on local disk. When Ollama loads a model that's already cached, GTAP reads it directly from the server's disk instead of transferring it over the network. The weights never cross the wire. We benchmarked DeepSeek-R1 32B generating 512 tokens on a DGX Spark, comparing native execution with GTAP over a local network: ![DeepSeek-R1 32B total runtime, native vs GTAP over LAN.](/blog/posts/ollama-remote-gpu/benchmark-light.svg) The total overhead splits roughly evenly between model loading (9s) and token generation (12s). Both are areas we're actively working on improving. Once every CUDA call flows through a single point, there's more you can do with it. GPU sharing, workload migration, and more. That's for another post. ## Try it Beyond Ollama, we've tested GTAP with vLLM, ComfyUI, Stable Diffusion, and more. If you're interested in trying it with your own workloads, [get in touch](https://gtap.ai). And yes, it runs DOOM. ![DOOM running on a MacBook via GTAP. The GPU is on a DGX Spark across the network.](/blog/posts/ollama-remote-gpu/doom.png) --- # An eBPF Loophole: Using XDP for Egress Traffic > Source: https://loopholelabs.io/blog/xdp-for-egress-traffic By Alex Sørlie, published 2025-11-04. We found a loophole that lets XDP work for egress traffic - something that was considered impossible until now. By exploiting how Linux handles virtual network interfaces, we achieved line-rate packet processing for outbound traffic with zero kernel modifications. #### TL;DR: *XDP (eXpress Data Path) is the fastest packet processing framework in linux - but it only works for incoming (ingress) traffic. We discovered how to use it for outgoing (egress) traffic by exploiting a loophole in how the linux kernel determines packet direction. Our technique delivers 10x better performance than current solutions, works with existing Docker/Kubernetes containers, and requires zero kernel modifications.* *This post not only expands on the overall implementation but also outlines how existing container and VM workloads can immediately take advantage with minimal effort and zero infrastructure changes.* ## The Line-Rate Problem At Loophole Labs, we live migrate everything - [containers](https://loophole.sh/rejekts2024), [VMs](https://loophole.sh/kc2024), and even [network connections](https://youtu.be/HrtX0JrjekE?si=h1xDagSnR4vaOB2z\&t=1683). During a migration every single packet for a workload needs to be intercepted, modified, encapsulated, encrypted, and rerouted to its new destination - all without the application noticing. Our scale requires us to be able to move workloads across clouds at *hundreds of gigabits per second* - and with that sort of performance requirement, every single CPU cycle matters. All of this is to say, we need to be able to process packets at **line-rate** (however much the underlying network can support, whether that's 20Gbps or 200Gbps), and there's really only one approach that lets us do that: ![Linux Packet Processing Performance Comparison](/blog/posts/xdp-for-egress-traffic/xdp-process-light.png) In Linux, the gold standard for high-performance packet processing is [XDP (eXpress Data Path)](https://docs.ebpf.io/linux/program-type/BPF_PROG_TYPE_XDP/). By intercepting packets as soon as they arrive at the network driver (before reaching the kernel) XDP is able to achieve line-rate speeds in most environments. Our own benchmarks above show how easily we were able to reach line-rate with XDP, not to mention the fact that major companies like [Meta](https://engineering.fb.com/2018/05/22/open-source/open-sourcing-katran-a-scalable-network-load-balancer/), [Cloudflare](https://blog.cloudflare.com/l4drop-xdp-ebpf-based-ddos-mitigations/), and [GCore](https://gcore.com/blog/geoip-filtering) have already been using it for more than 5 years now to handle 10s of millions of packets per second. ## XDP's Main Limitation Unfortunately XDP has one fundamental flaw that everyone accepts as fact: **it only works for ingress (incoming) traffic**. This isn't a bug or an oversight - it's the entire identity of XDP, one of the main characteristics that define it. XDP *only* processes packets on ingress. Period. For routers and load balancers, this limitation is perfectly fine: every packet they handle arrives from an external interface, making it all ingress from the kernel's perspective. Our network plane, on the other hand, has to run on the same compute nodes as the workloads that we're live migrating. And when these workloads generate packets - initiating connections, sending responses, etc. - that's considered egress traffic by the host kernel. XDP simply does not work in this scenario. ## Why Traffic Control Isn't Good Enough A popular method for handling egress packets is [Traffic Control (TC)](https://docs.ebpf.io/linux/program-type/BPF_PROG_TYPE_SCHED_CLS/), another eBPF-based mechanism that allows for packet processing at both ingress *and egress*. TC is already commonly used for traffic shaping, queuing, filtering, and policing outbound traffic. In fact, it's the de facto standard in the Kubernetes ecosystem - CNIs like [Cilium](https://docs.cilium.io/en/stable/network/ebpf/intro/) and [Calico](https://docs.tigera.io/calico/latest/about/kubernetes-training/about-ebpf#architecture-overview) all rely on TC for egress control because, until now, XDP for egress simply wasn't possible. Given all of this, TC might seem like an obvious choice for our use case as well, but it has a fundamental flaw of its own: *Performance.* We haven't been able to process more than [21Gbps with TC on egress](#benchmarks) (or more than [23Gbps on ingress](#xdp-process)),which makes it a non-starter for our needs. The reason why TC suffers from this performance bottleneck is due to how (and more importantly *when*) the linux kernel runs the TC program: ![TC Program Flow Diagram](/blog/posts/xdp-for-egress-traffic/tc-light.png) As shown in the diagram above, TC programs operate quite late in the networking stack, after packets have already spent some time travelling through the linux kernel. By the time a packet reaches the TC hook, the kernel has already processed it through various subsystems for routing, firewalling, and even connection tracking. This means that we've wasted quite a few CPU cycles before our TC program even runs. Another major limitation of TC is that it works on socket buffers (called `struct sk_buff{:c}` [in the linux kernel](https://docs.kernel.org/networking/skbuff.html)) which are allocated per-packet. This structure - while necessary for linux's own packet handling - comes with a significant performance hit due to the allocations themselves as well as the additional memory copies required to populate it. This all becomes doubly problematic when you're trying to process millions of packets every second. XDP, on the other hand, not only operates directly on the raw packet memory (because it runs directly in the network drivers before the packet even reaches the linux kernel) but does so at the earliest point in the packet's lifecycle, meaning almost no CPU cycles have been spent by the time our XDP program starts running. All this results in zero-copy packet processing, meaning packets can be inspected, modified, and redirected with the absolute minimum overhead possible. For us XDP is a hard requirement, and while the industry seems to have accepted that this is impossible, we haven't. ## When Does an XDP Program Run? One of our core beliefs at Loophole Labs is that every so-called "limitation" imposed by modern infrastructure is really just a problem we haven't solved yet. In the spirit of this, we decided to go digging through the linux kernel source in an attempt to understand exactly why and how the kernel decides to classify a packet as "ingress" in the first place. As it turns out, linux doesn't actually classify the packet at all. When a packet arrives at a physical network interface, the network card writes the contents into an RX ring buffer - a memory region allocated by the device driver that the network card can write to directly via [DMA (Direct Memory Access)](https://docs.kernel.org/core-api/dma-api-howto.html). Next, the network card uses an interrupt to signal the device driver that there's a packet available for processing. The device driver then copies the packet from the ring buffer into its RX queue. And this is exactly when the XDP program runs: directly on the packet in the RX queue. This process is illustrated in the diagram below: ![XDP Program Flow Diagram](/blog/posts/xdp-for-egress-traffic/xdp-light.png) If this entire process makes one thing clear, it's that there is very little work being done in between the packet arriving at the physical interface and it being ready for the XDP program to run. The RX queue is the trigger that tells the linux kernel how to "classify" the packet as ingress and whether it should run the XDP hook. As we saw in [this](#tc) diagram, the RX queue is not used at all for egress packets, and this simple limitation is the cause of all our headaches. Now that we know all this, how can we get around it? As it turns out, *we don't have to*. ## Making Egress Look Like Ingress We were reading through the various linux interface docs, hoping to find some little insight into our predicament, when an interesting virtual interface caught our eye: **Virtual Ethernet**. A Virtual Ethernet (`veth{:bash}`) interface is a pair of network interfaces that act as a direct tunnel between each other. When a packet is transmitted from the TX queue of one side of the `veth{:bash}` pair, a pointer to the packet's memory is simply moved *to the RX queue of the other interface*. This makes the packet appear as if it were received by a physical network interface with very low overhead. Yep, you read that right - `veth{:bash}` interfaces *have an RX queue that's used when receiving a packet from the other side*. To illustrate this better, let's take an example setup like the one below. We have two applications running in their own network namespaces, with two `veth{:bash}` pairs (`veth0-A{:bash}` and `veth0-B{:bash}`) being used to route traffic out of the namespaces. ![XDP for Egress Traffic Flow Diagram](/blog/posts/xdp-for-egress-traffic/diagram-light.png) The key insight here is that if we send outgoing traffic through one end of the `veth{:bash}` pairs (`veth0-A{:bash}` in the diagram above), then from the perspective of the second interface (`veth1-A{:bash}`), the packet arrives at the RX queue of an interface, and is now considered ingress traffic. And, since XDP programs can be attached to any interface’s RX queue, our XDP hook will automatically run on that *egress* packet. Furthermore, if we run our XDP programs in native mode like in the diagram above, packets can be processed with zero-copy and will bypass the linux kernel entirely when we use `XDP_REDIRECT{:bash}` to route directly to the TX queue of the `eth0{:bash}` interface. What makes this discovery even more powerful is that modern container runtimes - Docker, Kubernetes, containerd - already use `veth{:bash}` pairs and network namespaces for container networking. Every container you're running right now is already connected through `veth{:bash}` interfaces, and it looks exactly like the diagram above. That's right - not only can we use XDP for egress traffic in any of these environments, but we can do it without having to change them in any way. ## If You Skip the Kernel, You Have to be the Kernel Unfortunately, while implementing this technique seemed straightforward at first, we quickly hit a snag while benchmarking. Our packets kept getting dropped after our XDP program ran, and at first we couldn't figure out why. We decided to run `tcpdump{:bash}` on the receiving host and realized the packets weren't even making it over the network. Next, we decided to run `tcpdump{:bash}` on the *switch* handling the packets, and that's when we realized what we'd missed. As it turns out, when you bypass the kernel's networking stack, you inherit its responsibilities. Normally, when a packet is sent out via the linux kernel, it handles the routing, checksumming, and ARP resolution for us. But we of course have bypassed the kernel's networking stack entirely, meaning now *we* have to take full responsibility for ensuring packets are properly formed and can actually reach their next hop. Our network plane already handles proper routing for us, but we'd missed both checksumming and ARP resolution. ### Checksum Calculations in XDP XDP programs unfortunately are not provided with the same [checksum helpers](https://docs.ebpf.io/linux/helper-function/bpf_csum_update/) that TC programs get. For NAT (Network Address Translation) or any other packet header modifications, you need to recalculate checksums manually - and when performance matters, the trick is to use incremental checksum updates rather than full recalculations: ```c static __always_inline __u16 csum16_add(__u16 csum, __u16 addend) { csum += addend; return csum + (csum < addend); } // Remove old IP from checksum csum = csum16_add(~tcp->check, ~old_ip_high); csum = csum16_add(csum, ~old_ip_low); // Add new IP to checksum csum = csum16_add(csum, new_ip_high); csum = csum16_add(csum, new_ip_low); tcp->check = ~csum; ``` ### ARP Resolution The linux kernel normally handles ARP to resolve IP addresses to MAC addresses and automatically sets the destination MAC address in the `ethernet{:bash}` layer of the outgoing packet. With XDP however, we need to maintain our own ARP table and pass in the destination MAC ourselves: ```c struct arp_entry { __u8 mac[ETH_ALEN]; }; struct { __uint(type, BPF_MAP_TYPE_HASH); __type(key, __be32); // IP address __type(value, struct arp_entry); __uint(max_entries, 65535); } arp_table SEC(".maps"); // In your XDP program, lookup the destination MAC struct arp_entry *entry = bpf_map_lookup_elem(&arp_table, &dest_ip); if (entry) { memcpy(eth->h_dest, entry->mac, ETH_ALEN); } ``` ## Benchmarks To validate our overall approach, we set up [iPerf3](https://github.com/esnet/iperf) containers between two 200Gbps-capable EC2 instances in the same AWS VPC. We purposely reduced the MTU to 1500 since traffic to the public internet generally [can't use jumbo frames in the first place](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/network_mtu.html#jumbo_frame_instances). **We used the exact same container networking setup for all three tests** - the same standard network namespaces with `veth{:bash}` pairs that Docker and every other container runtime uses by default. The only thing we changed was how packets were routed from the container's `veth{:bash}` interface to the host's physical interface: * **iptables**: The default that everyone uses today - `PREROUTING{:bash}` chains to move traffic out of the namespace * **Traffic Control**: Using a TC egress program on the `veth{:bash}` interfaces * **XDP**: Our technique - Using an XDP program attached to the `veth{:bash}` interfaces We also decided to benchmark both the generic and native XDP drivers implemented for `veth{:bash}` interfaces. We'll let the results speak for themselves: ![iPerf3 Benchmark With Various Routing Strategies](/blog/posts/xdp-for-egress-traffic/benchmarks-light.png) The first two results are exactly what we expect - iptables introduces the most overhead because it routes through the linux kernel, and traffic control performs better but still operates post-socket-buffer and can't come close to achieving line-rate. With the `generic{:bash}` XDP driver, however, we see something surprising: worse performance than our TC program. After a little digging we realized this actually makes sense. The `generic{:bash}` XDP driver does *not* run on the RX queue and instead, like TC, runs after the socket buffer has been allocated. The worse performance is the result of running the XDP program in the same place as TC but without any of the optimizations that TC benefits from. With the `native{:bash}` XDP driver (which is available in [linux 4.19+](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=638264dc90227cca00d20c26680171addce18e51)) we finally see the results we've been looking for - we're routing just shy of line-rate at about 194Gbps, 12.4 times the throughput of iptables and about 9.2x the throughput of TC. One final thing to note here are the error bars, which were significantly smaller with `native{:bash}` XDP. This makes sense since the bulk of our performance improvements come from bypassing the linux kernel *and doing less work*. iPerf3, iptables, and the linux kernel are all constantly fighting for the CPU which results in inconsistent throughput. ## A Drop-In Fast Path for Containers One of the most exciting aspects of this discovery is how immediately applicable it is. We setup our benchmarks to replicate how containers already use network namespaces and `veth{:bash}` pairs. This means we can dramatically accelerate container networking without changing how containers work or how they're orchestrated. Consider what happens every time a containerized application sends a packet today: it traverses through iptables rules, gets NAT'd, maybe goes through connection tracking, and finally makes it out to the network. All of this happens in the kernel, consuming precious CPU cycles that could be used by actual applications. With XDP on veth interfaces, we can bypass all of that overhead. The packet goes straight from the container's namespace through our XDP program to the physical interface. No iptables. No conntrack. Just pure, line-rate packet routing. ## What We're Building While our primary use case at Loophole Labs is live migration - where this technique enables us to transparently reroute connections at line rates during migrations - we recognize the broader impact this can have on container networking as a whole. That's why we're working on a Docker network plugin that implements this technique. It'll be a drop-in replacement for Docker's default bridge network driver, except it uses XDP instead of iptables for packet routing. For simpler container deployments that don't need the full complexity of Kubernetes networking (microservices, development environments, or edge computing nodes) this could mean: * Doubling network throughput without any hardware upgrades * Dramatically reducing CPU usage for network-heavy workloads * Eliminating iptables as a bottleneck in container-to-container communication We plan to open source this plugin soon, but the beauty of this technique is that you don't need to wait for us. Everything you need to implement this yourself is described in this post. The `veth{:bash}` pairs are already there, all that's left is writing the XDP programs to route your packets. ## The Bigger Picture Loophole Labs was built on a very simple premise: [Better Building Blocks = Better Applications](/about). This discovery - that XDP can process egress traffic by taking advantage of `veth{:bash}` interfaces - is the best representation of just that, a better building block that results in significantly better applications. While we'll be open-sourcing the Docker network plugin for those who want to take advantage of XDP's egress performance for themselves, this discovery also powers something much bigger: **Architect**, our live migration platform. Architect uses this XDP technique (as well as other breakthrough implementations for disk & memory checkpointing) to seamlessly live migrate your containers, VMs, and even active network connections between any clouds or regions - all without your users noticing. If you're interested in diving deeper into the technical details or implementing XDP egress in your own infrastructure, join our [Discord](https://loopholelabs.io/discord) where our engineering team hangs out and answers questions from the community. Trust me, we love talking about this stuff. ### Ready to Use Live Migration? Join our waitlist to be among the first to dramatically reduce your infrastructure costs while improving reliability: --- # Rethinking Spot Instances: Solving Preemption > Source: https://loopholelabs.io/blog/rethinking-spot-instances By Shivansh Vij, published 2025-09-08. Run any Kubernetes workload on Spot Instances with zero downtime and 75%+ cost savings - no code changes required. Architect eliminates service interruptions during preemption through seamless pod migration. #### TL;DR: *We've made spot instances practical for all Kubernetes workloads. Architect migrates pods to new nodes before spot instances are reclaimed - with zero downtime, zero client interruptions, and zero code changes. Get 75%+ cost savings on your entire Kubernetes infrastructure while maintaining production reliability.* [Join our waitlist](/waitlist) to run your production workloads on spot instances without any compromises. ## The Spot Instance Problem Spot instances are 75%+ cheaper than on-demand instances, but they can be reclaimed with just 30 seconds to 2 minutes notice. This makes them unusable for production Kubernetes workloads. When a spot instance is reclaimed, every pod on that node experiences: * Forced termination * Rescheduling to a new node (if capacity exists) * Full application restart and initialization * Load balancer re-registration * Client request failures during the transition Even "stateless" workloads aren't immune. Your users experience errors, timeouts, and degraded performance every time a node is preempted. The unpredictability makes it worse - preemption rates can swing from 3% to over 60% based on time, region, and instance type. This forces a stark technical trade-off: accept service interruptions and angry users, or pay 3-4x more for on-demand instances. Most production workloads stay on on-demand infrastructure, leaving significant cost savings on the table. ## Why Traditional Approaches Aren't Enough Kubernetes already provides some mechanisms to handle node failures - pod disruption budgets, graceful termination, and cluster autoscaling. But these are fundamentally reactive systems designed for failure recovery, not proactive migration. Each approach hits the same technical barrier: * **Extra replicas**: Maintaining 2-3x capacity eliminates the 70% cost savings * **Fast autoscaling**: New pods still require full initialization (pulling images, application startup, loading models, etc.) * **Graceful shutdown handlers**: Unable to prevent the pod from being terminated in the first place * **Mixed node pools**: Reduces the frequency of preemptions, but every preemption still causes downtime The core problem is that all these solutions accept that a pod must die and be recreated, and that comes with inherent downtime and service disruptions. ## Enter Architect: Seamless Pod Migrations We took a different approach. Instead of managing the chaos of preemption, we eliminate it entirely. Architect continuously snapshots your running pods without interrupting them. This results in Architect always having the complete state of your application on-hand before anything happens. When a spot instance preemption notice arrives, we migrate your pods to new nodes before the original instance terminates. Your applications keep running, connections stay alive, and clients never notice. This isn't a restart or a reschedule - it's true live migration for any Kubernetes workload. ## How It Works Under the hood Architect is a relatively simple state machine who's only job is to keep your application running under any circumstances. Architect fundamentally moves through four possible states: 1. **Normal Operation**: In this state Architect is continuously capturing pod state and syncing it across nodes. This is inherently why our migrations are instantaneous - we're amortizing the migrations ahead of time. During this time we're also carefully watching the resource utilization across the cluster to make sure we have enough resources to handle any sudden migrations that come up. 2. **Preemption**: When a preemption notice from the cloud provider arrives (between 30 seconds and 2 minutes depending on the provider) Architect goes to work live migrating the workloads away from the preempted nodes. 3. **Migrations**: This is where Architect earns its keep. Once we have the resources scheduled (ie. the pods themselves) have been rescheduled and all ancillary resources have been recreated (file descriptors, PVC attachments, GPU resources, etc.) it's time to move the state. This is relatively instantaneous thanks to all the work we did ahead of time. 4. **Rerouting**: Migrations aren't just supposed to be seamless for the application - they should be seamless for clients as well. To achieve this Architect gets to work updating our [XDP](/blog/xdp-for-egress-traffic.md)-based routing layer to transparently reroute connections. This is done in a way so that no packets are dropped and the client can't even tell that the workload has migrated. The magic happens in our continuous state capture mechanism. We've extended [CRIU (Checkpoint/Restore In Userspace)](https://github.com/checkpoint-restore/criu) to continuously serialize running processes to passive storage (PVCs, S3, etc.) without interrupting them or impacting performance. Think of it as streaming your application's entire memory state constantly to disk in the background. For GPU workloads, we go even further: we serialize the complete CUDA context and GPU memory state. This means your model weights, optimizer states, gradient buffers - everything stays exactly where it was. A training run that's 47 epochs deep doesn't restart from scratch; it continues from epoch 47 on the new node. We can even migrate GPU state independent from CPU state, enabling fascinating optimization strategies. When a preemption notice arrives (30 seconds to 2 minutes warning), we: 1. Provision the target node using standard Kubernetes scheduling 2. Stream the latest checkpoint to the new node from the passive storage layer as well as peer to peer from the previous node 3. Restore the process with all its state intact 4. Update our network layer to transparently redirect existing connections The end result is a completely seamless migration experience - neither your clients nor your application can tell that it's now running on a completely new node. The best part is that this entire process requires no code changes, just `helm install` and your infrastructure is spot-ready. ## See It In Action: Live Demo at KubeCon NA 2024 This isn't theoretical. At [KubeCon North America 2024](https://loophole.sh/kc2024), we demonstrated live pod migration across multiple cloud providers ([AWS](https://aws.amazon.com), [GCP](https://cloud.google.com), and [Azure](https://azure.microsoft.com)), all while maintaining active client connections: [Watch on YouTube](https://www.youtube.com/watch?v=kGUN7aUBGTU) ## Real Cost Impact The math for spot instances is actually quite straightforward: spot instances consistently cost 70% less than on-demand across AWS, GCP, and Azure. For a typical production cluster running 50 m5.xlarge instances: * **On-Demand**: \~$6,240/month * **With Architect on Spot Instances**: \~$1,872/month A monthly cost savings of $4,368 (\~70% reduction). Plus, Architect's overhead is minimal - the continuous checkpointing uses existing cluster storage, and migrations are primarily peer-to-peer with S3 acceleration only when needed. The system pays for itself if you're running more than a handful of instances. ## Inherently Compatible Architect is inherently compatible with any type of workload and fixes how your infrastructure handles spot preemptions and pod migrations: * **Web Services & APIs**: No 502 errors, no request timeouts, no user impact during preemptions * **Databases & Caches**: Maintain connections, preserve buffers, no rebalancing overhead * **ML/AI Workloads**: Models stay loaded, GPU state is preserved, training continues uninterrupted * **Streaming Systems**: Kafka consumers keep their positions, WebSocket connections survive * **Batch Jobs**: Complete without restart, no lost progress, no wasted compute With Architect, every pod in your cluster can now safely run on spot instances. ## The Power of Workload Mobility The same technology that enables spot instance migration also powers: * **Scale-to-Zero Without Cold Starts**: Hibernate idle pods, wake them in \<50ms * **Node Maintenance Without Downtime**: Evacuate nodes anytime without service impact * **Dynamic Cost Optimization**: Move workloads to the cheapest compute continuously and automatically Only when your pods can move without interruption does your infrastructure become truly elastic. ## Getting Started Architect is currently in early access. We're working with select teams to dramatically reduce their Kubernetes costs without compromising reliability. [Join our waitlist](/waitlist) to be among the first to deploy Architect in your cluster. We'll help you identify which workloads can benefit most and guide you through maximizing your spot instance savings. Spot instances have always been a compromise - great prices but unreliable for production. With Architect, that compromise finally disappears. --- # Reliable Cross-Cloud Kubernetes With Architect > Source: https://loopholelabs.io/blog/reliable-cross-cloud-kubernetes By Shivansh Vij, published 2025-05-07. Live migrate entire Kubernetes clusters - stateful workloads included - between AWS, Azure, GCP and bare metal with zero downtime. ### What if you could just pick up a Kubernetes cluster, and move it? At [KubeCon North America 2024](https://loophole.sh/kc2024) we demo'd just that: live-migrating a Kubernetes cluster (running a stateful [Valkey](https://valkey.io/) database) from **AWS to GCP to Azure** in *seconds* - and with **zero downtime or client interruptions**. [Felicitas Pojtinger](https://www.linkedin.com/in/pojntfx), who's the Head of R\&D at Loophole Labs (and our resident KVM whisperer), gave a talk that explored how to build Kubernetes clusters that leverage spot instances and then how to move that cluster in its entirety across cloud providers and even to on-premise infrastructure. A link to a video of the talk is [below](#building-reliable-cross-cloud-kubernetes-clusters-on-spot-instances) and we’ve also posted Felicitas’ [full notes](https://github.com/loopholelabs/architect-talks) from the talk to our github repo. ### Architect In order to migrate complex infrastructure environments like entire Kubernetes clusters, Architect employs a combination of open-source projects that, when used together, allow for seamless, zero-downtime live migrations. Felicitas explains how each project works and how they come together to deliver Architect, the underlying project used in the demo. At its core, Architect uses: * [PVM](https://github.com/loopholelabs/linux-pvm-ci) - Pagetable-based Virtual Machine (repo): This kernel patch delivers page table virtualization and enables running VMs within VMs without hardware acceleration. * [Firecracker](https://github.com/loopholelabs/firecracker): A hypervisor providing consistent VM behavior across different cloud platforms via CPU templates. Its snapshot and restore features are crucial for portability. * [Silo](https://github.com/loopholelabs/silo): An open-source storage engine optimized for tracking changes in VM snapshots, facilitating rapid migration between hosts. * [Drafter](https://github.com/loopholelabs/drafter): A framework that integrates PVM and Firecracker, offering tools for VM management, networking, health checks, snapshotting, and packaging. * [Conduit](https://github.com/loopholelabs/architect-networking): Enables live migration of network connections alongside VM state, ensuring zero downtime during migrations. ### Building Reliable Cross-Cloud Kubernetes Clusters on Spot Instances The talk is available on the [CNCF youtube channel](https://loophole.sh/kc2024): [Watch on YouTube](https://www.youtube.com/watch?v=kGUN7aUBGTU) If you want to know more please join our [Discord](https://loopholelabs.io/discord) or follow us on [LinkedIn](https://loopholelabs.io/linkedin), [Twitter](https://loopholelabs.io/x), and [GitHub](https://loopholelabs.io/github).