How fast does it take from a code change to an observable result in development?
The inner loop is loosely defined as a local build and deploy. Optimizing this loop is one of the keys to developer productivity. But it’s one of the things that developers rarely think of.
A good inner dev loop is both fast and correct. It’s easy in theory to get “correct”: work in an environment as close to production as possible and run a production build and deploy loop on every change. That’s usually too slow. “Correct” is so important because it prevents reproducibility bugs that are notoriously hard to track down and fix. “It works on my machine.”.
You can get “fast” by syncing files and doing incremental compilation tricks. Webpack dev server is probably the best example of this. You can get even faster if you hook the entire loop up to a file-watching trigger — automatically debouncing and triggering background compilation and deploys. However, an optimized runtime that’s completely different than a production environment loses much of its benefit. Most of these “fast” tools are specific to a framework or language, which limits their usefulness.
Why not have both, fast and correct?
This is the idea behind a developer tool I built at Google called skaffold. The idea was to make the inner development loop declarative and automated. It was built out of my own frustration with the slowness of development cycles. With a few tricks, you could get even faster development and a production-grade Kubernetes pipeline in development.
Here’s how it works:
First, it builds the dependency graph for code to be built and services to be deployed. Instead of having to do this manually, skaffold parses the dependencies out of a Dockerfile and images out of Kubernetes manifests.
With a file watcher, it knows when to trigger different paths in the dependency graph. If a build file changes, it rebuilds that container and redeploys dependent services. If a deployment configuration file changes, it knows just to redeploy.
For interpreted languages or languages and frameworks that have their own “fast” dev servers, skaffold handles a file sync between host and container. For these specified files, it can skip the build and deploy loop. This means you get the fastest dev experience across the spectrum: for static files, compiled files, and configuration files.
Finally, it streams and tags the logs from all deployed services. This greatly simplifies development when you’re dealing with multiple services — no more complicated tmux sessions.
Why does it work?
- Kubernetes provides APIs that are surprisingly useful in development, besides the obvious development/production parity. APIs to port-forward, exec, stream logs and deploy services.
- Docker and Kubernetes are language and framework agnostic. This means that improvements accrue across the board.
- If used correctly, Docker BuildKit can actually provide better cache management than some languages and frameworks. This is the piece of the puzzle that I never solved generically (although I have some ideas for you!). In the meantime, cache mounts and smart ordering.
- Some optimizations for local Kubernetes endpoints. I also built minikube and container tooling at Google, so I knew exactly how to optimize both tools together. Skipping push cycles, loading images, and other configuration shortcuts.