Kubernetes

Sharing long-lived BuildKit instances across CI jobs cost us up to 4% of our container builds to unpredictable failures and noisy-neighbor contention. We replaced them with ephemeral, fully-isolated builders provisioned per job by a Kubernetes operator we’ve open-sourced, and build success now sits at 99.97%.

Moving our GitLab CI runners onto Kubernetes fixed the wasted capacity and queue bottlenecks, but the real wins came afterwards. Distributed caching in S3 and ECR, HPA-driven autoscaling, dedicated NVMe node pools, and reserved idle capacity took average job queue time from 16 seconds to 2 and cut cost per job by 40%.

Every CI job used to get its own EC2 instance and its own Docker daemon, which made docker build easy and everything around it painful — polluted state between builds, tightly-coupled infrastructure, and multi-arch builds only via slow emulation. Here’s how we evaluated Podman, Kaniko, and Buildkit, why Buildkit won, and what it took to make it behave under Kubernetes autoscaling.