images: slim OCI images of the swarm daemons from nix
This commit is contained in:
parent
ffbd2f4dea
commit
29ca381114
1 changed files with 218 additions and 6 deletions
75
docs/4-implementation/k8s/images.md
Normal file
75
docs/4-implementation/k8s/images.md
Normal file
|
|
@ -0,0 +1,75 @@
|
|||
# Service images
|
||||
|
||||
The swarm daemons that run in the k8s cluster ship as OCI images built by nix.
|
||||
`nix/images/default.nix` holds the one function that builds them,
|
||||
`mkServiceImage`, and the set of daemons that use it.
|
||||
|
||||
## What an image holds
|
||||
|
||||
One image runs one daemon. Its contents are:
|
||||
|
||||
- the daemon's per-bin package from `nix/packages/default.nix`,
|
||||
- the CA roots from `pkgs.cacert`, with `SSL_CERT_FILE` pointing at them,
|
||||
- the binary's own closure, which is glibc and libgcc.
|
||||
|
||||
The image has no shell and no init. The kubelet restarts a crashed container, so a
|
||||
service manager inside it would only disagree with the kubelet. A missing
|
||||
shell also leaves nothing to `exec` into. The entrypoint is `/bin/<bin>`, and
|
||||
the process runs as the `serviceUser` uid and gid that `nix/images/default.nix`
|
||||
defines. A pod's `securityContext.runAsUser` and `runAsGroup` read that binding
|
||||
too, so the number lives in one place.
|
||||
|
||||
`mkServiceImage` takes `name`, `package`, and optionally `bin` (the binary
|
||||
name, default `name`), `extraContents` and `env` (an attribute set added to
|
||||
the container environment).
|
||||
|
||||
## The image set
|
||||
|
||||
Each of these is a flake package named `image-<daemon>`, and the image name is
|
||||
`hyperhive/<daemon>`:
|
||||
|
||||
- `swarm-controller`
|
||||
- `swarm-nats-auth`
|
||||
- `swarm-logs-auth`
|
||||
- `swarm-metrics-auth`
|
||||
- `swarm-notify-router`
|
||||
- `swarm-ci-metrics`
|
||||
|
||||
`nix flake check` evaluates all six and builds none of them, so the workspace
|
||||
compile stays out of CI. `swarm-claude-login`, `swarm-stt` and
|
||||
`swarm-review-classifier` need claude-code, whisper or similar tools in their
|
||||
image and aren't in the set yet.
|
||||
|
||||
## Tags
|
||||
|
||||
An image has no fixed tag. The tag is the content hash, so a changed binary is
|
||||
a changed reference. A manifest reads the full reference from the package's
|
||||
`imageName` and `imageTag` attributes:
|
||||
|
||||
```sh
|
||||
nix eval --raw .#image-swarm-logs-auth.imageName
|
||||
nix eval --raw .#image-swarm-logs-auth.imageTag
|
||||
```
|
||||
|
||||
## Loading an image
|
||||
|
||||
`nix build` produces a `.tar.gz` that `podman load`, `skopeo` and the k3s image
|
||||
directory all read. Use `podman`, not docker:
|
||||
|
||||
```sh
|
||||
podman load -i "$(nix build --no-link --print-out-paths .#image-swarm-logs-auth)"
|
||||
```
|
||||
|
||||
For a containerd node, stream the file into the node's image store:
|
||||
|
||||
```sh
|
||||
k3s ctr images import - < "$(nix build --no-link --print-out-paths .#image-swarm-logs-auth)"
|
||||
```
|
||||
|
||||
## The `image-fn-smoke` check
|
||||
|
||||
`checks.<system>.image-fn-smoke` builds `mkServiceImage` around `hello` and
|
||||
reads the tarball back. It asserts the entrypoint, the user, the CA env var, a
|
||||
layer listing without `bash` or `systemd` and a size under 50 MB. The listing
|
||||
must also contain `hello` and `glibc`, so an empty listing can't pass the
|
||||
shell arm. The check runs without compiling any hyperhive binary.
|
||||
Loading…
Reference in a new issue