$ docker build -f mockerfile.yaml
In this blog post, I'll show you how to write your own Dockerfile syntax that works out of the box with any existing Docker installation. If you want to see it in action right away, here's a YAML file that is used in place of a Dockerfile.
curl https://raw.githubusercontent.com/r2d4/mockerfile/master/Mockerfile.yaml | DOCKER_BUILDKIT=1 docker build -f - .
The sample code for this post can be found on GitHub.
Background
Buildkit is a tool that can convert code to docker images. It's already integrated in Docker versions 18.09 and above.
Buildkit works by mapping a human-readable frontend (e.g. Dockerfile) to a set of Ops (ExecOp, CacheOp, SecretOp, CopyOp, SourceOp, etc.), collectively called low-level builders (LLB).
That LLB is then executed by either a runc or containerd worker and produces a docker image.
Design
Our demo frontend is going to be called Mockerfile. It's going to be a YAML based syntactic sugar for building ubuntu-based images. It will contain two keys: package
, which is some automation around apt-get
, and external
, which will fetch external dependencies concurrently.
#syntax=r2d4/mocker
apiVersion: v1alpha1
images:
- name: demo
from: ubuntu:16.04
package:
repo:
- deb [arch=amd64] http://storage.googleapis.com/bazel-apt stable jdk1.8
- deb [arch=amd64] https://download.docker.com/linux/ubuntu xenial edge
gpg:
- https://bazel.build/bazel-release.pub.gpg
- https://download.docker.com/linux/ubuntu/gpg
install:
- bazel
- python-dev
- ca-certificates
- curl
- build-essential
- git
- gcc
- python-setuptools
- lsb-release
- software-properties-common
- docker-ce=17.12.0~ce-0~ubuntu
external:
- src: https://storage.googleapis.com/kubernetes-release/release/v1.10.0/bin/linux/amd64/kubectl
dst: /usr/local/bin/kubectl
- src: https://github.com/kubernetes-sigs/kustomize/releases/download/v1.0.8/kustomize_1.0.8_linux_amd64
dst: /usr/local/bin/kustomize
sha256: b5066f7250beb023a3eb7511c5699be4dbff57637ac4a78ce63bde6e66c26ac4
- src: https://storage.googleapis.com/kubernetes-helm/helm-v2.10.0-linux-amd64.tar.gz
dst: /tmp/helm
install:
- install /tmp/helm/linux-amd64/helm /usr/local/bin/helm
- src: https://dl.google.com/dl/cloudsdk/channels/rapid/downloads/google-cloud-sdk-217.0.0-linux-x86_64.tar.gz
dst: /tmp
Code Walk-through
High level steps
- Write a conversion function from your configuration file format to LLB
- Write a build function that handles some extra tasks such as mounting the configuration file, secrets, or context.
- Use that build function in the frontend gRPC gateway
- Publish as a docker image
- Add the
#syntax=yourregistry/yourimage
directive to your top of your config file and setDOCKER_BUILDKIT=1
to build with any Docker installation.
Writing the Conversion Function
Here is my conversion function for Mockerfile. It takes my configuration struct and returns DAG called llb.State
.
Some interesting observations:
- You can start as many different concurrent paths as you want with
llb.Image
(Similar to aFROM
instruction), but those paths must be merged into a final image. - Merging is done with a copy helper function, which takes two
llb.State
, mounts src to dst, and copies the file over, producing a singlellb.State
. (Similar to aCOPY --from
multistage build)
The external files are downloaded in separate alpine images, and then use the copy helper to move them into the final image. It uses a small script to verify the checksums of the downloaded binaries s = s.Run(shf("echo \"%s %s\" | sha256sum -c -", e.Sha256, downloadDst)).Root()
. If the checksum does not match, the command fails, and the image build stops.
Writing the Build Function
Steps of the build function
- Get the Mockerfile/Dockerfile config and build context
- Convert config to LLB
- Solve the LLB
- Package the image and metadata
The configuration file itself must be mounted into the build contianer, for which we use llb.Local
. You can see this in action here. Mounting a build context would be done in a similar way.
Creating the gRPC gateway
We reuse the grpc client here. As long as your build function fits the interface type BuildFunc func(context.Context, Client) (*Result, error)
, things will work as expected.
Publish the image
Our image is quite simple, using the built binary as the entrypoint. The binary runs the grpc gateway we created in the last step. Here is an example.
Using it
- Add
# syntax=yourregistry/yourimage
to the top of your configuration file. Buildkit looks for that, and will pull and use that image as the solver. - Add
DOCKER_BUILDKIT=1
to yourdocker build
command to enable thebuildkit
backend.