BuildKit

Sharing long-lived BuildKit instances across CI jobs cost us up to 4% of our container builds to unpredictable failures and noisy-neighbor contention. We replaced them with ephemeral, fully-isolated builders provisioned per job by a Kubernetes operator we’ve open-sourced, and build success now sits at 99.97%.

Every CI job used to get its own EC2 instance and its own Docker daemon, which made docker build easy and everything around it painful — polluted state between builds, tightly-coupled infrastructure, and multi-arch builds only via slow emulation. Here’s how we evaluated Podman, Kaniko, and Buildkit, why Buildkit won, and what it took to make it behave under Kubernetes autoscaling.