Why We Chose Buck2 Over Nix to Build Verifiable OS Images

Why Nix Wasn't the Answer for Amutable

When Amutable set out to build an operating system with cryptographically verifiable integrity, the build system had to provide that same integrity from the first action. Their team includes the creator of mkosi, so they evaluated familiar ground first. They still ended up with a Buck2-based system called tine, and the reason is worth studying because it exposes the hardest problem in OS image building: delta maintenance.

Nix solves the reproducibility half of the problem well. Every derivation is input-hashed, every closure lands in /nix/store/ under a path that encodes its full dependency graph, and two machines producing the same flake.lock get bit-identical output. That part is genuinely impressive.

The problem Amutable hit is upstream packaging. Distributions spend decades of thankless effort maintaining patches, security backports, and integration work for thousands of packages. Nix's model effectively asks you to re-declare that work in the Nix language, or maintain a permanent fork of it. A maintainer on the project, who spent over a decade packaging for (open)SUSE, put it bluntly: the most painful part of running a distribution is the deltas you carry forever after you hard-fork a package.

tine inverts that relationship. It imports upstream distribution artifacts — Fedora, Arch, and others — as first-class inputs, then lets you apply temporary deltas on top. You can react to a CVE without boiling the ocean, and you can diverge from an upstream packaging decision without owning that divergence for the next decade.

Buck2 is the wrong tool if your model of software is "repackage the world." It is the right tool if your model is "pin the world, then patch the parts that matter."

The trade-off is explicit: you accept a heavier dependency on upstream distributions staying buildable, in exchange for not rebuilding their packaging labor yourself. For a company shipping appliances rather than a general-purpose distro, that exchange is usually correct.

Action Hashing Beats Timestamp Builds

The foundation under tine is Buck2, Meta's open-source build engine. Its core guarantee is simple to state and hard to implement: an action is keyed by a hash of all its inputs — source files, tool binaries, build platform config, and the command line itself. Buck2 never consults file timestamps.

That distinction matters for images because image builds are where incremental correctness usually dies. A naive make graph sees a newer libfoo.so, rebuilds a layer, and silently produces a disk image whose contents no longer match its SBOM. With content-hashed actions, if the inputs did not change, the output cannot change — and Buck2 is free to pull the result from a shared cache instead of re-running it.

load("@tine//image:defs.bzl", "image")

image.rootfs_archive(
    name = "demo",
    package_manager = ":image.package-manager",
    packages = ["bash", "coreutils"],
    ops = [
        image.copy("marker.txt", "/etc/tine/marker"),
        image.run(
            ["/usr/bin/bash", "-c", "echo built-by-tine >> /etc/tine/marker"],
            chroot = True,
        ),
    ],
    format = "tar",
)

Every action runs in an unprivileged sandbox that sees only its declared inputs and has no network access. That is what makes the hash meaningful: if the graph claims an action depends on nothing else, the sandbox enforces it. Reproducibility becomes a property of the sandbox rather than a hopeful claim in a README.

The operational payoff is the "cheap world rebuild." A compiler bump invalidates a large slice of the graph, but a shared remote cache means a developer is rarely rebuilding from scratch. Lookups key off the same action digest used locally, so the local and remote results are the same artifact or nothing at all.

The cost: a content-addressed cache is only as good as its invalidation. If a rule forgets to declare an input, you get a stale hit that looks correct — the failure mode is silent, not loud. Hermetic sandboxes shrink that surface but do not eliminate it.

Boxes Are the Interesting Primitive

tine's most reusable idea is the box: a declared, pinned environment in which a build task runs. Think of a container or a distrobox environment, except it is declared natively in Buck2's language and inherits Buck2's caching and rebuild semantics.

The important consequence is that a box is itself an artifact. You do not docker pull a mutable base image and hope it is the same today as last month. The box declares its inputs — a distribution catalog, specific package versions, a compiler — and Buck2 hashes those inputs like everything else. Bump a package and the box gets a new action digest, which invalidates exactly the actions that used it and nothing more.

tine ships boxes for rpmbuild, Go, and Cargo work, plus a general-purpose fedora.rawhide.box that carries systemd-ukify and QEMU. A project can define its own. The pattern generalizes beyond OS images: anywhere you have a "toolchain that must be identical across CI and every developer laptop," an input-hashed box is a cleaner answer than a pinned Docker tag plus a README that says "please keep this up to date."

# The host needs only git, python3 (for bootstrap), and unprivileged user namespaces.
# On Ubuntu, allow unprivileged userns if apparmor restricts it:
#   sysctl kernel.apparmor_restrict_unprivileged_userns=0

git submodule add https://github.com/amutable-systems/tine tine
tine/bin/tine init            # writes tine.toml, .buckconfig, .gitignore
tine/bin/tine buck build //packages/...
tine/bin/tine buck targets //...   # enumerate every buildable target

Three host dependencies total. Everything else — the Buck2 binary, the compiler boxes, the package catalogs — is fetched and verified from pinned declarations on first use into ${XDG_CACHE_HOME:-~/.cache}/tine/buck2. A new engineer clones the repo and runs one command.

The trade-off: bootstrapping a from-scratch build environment means the first build is slow and network-dependent. Every subsequent build is fast and hermetic. You are trading a one-time cold-start cost for permanent incremental correctness, which is the right direction for a system an entire team rebuilds daily.

Building a Bootable Disk in One Target

The output path that makes tine interesting for infrastructure teams is native image building. A single Buck2 target can produce a bootable GPT disk with a unified kernel image (UKI) and a dm-verity protected /usr, signed either by a locally generated key or a hardware key through PKCS#11.

A UKI bundles the kernel, initramfs, kernel command line, and Secure Boot signatures into one signed PE binary that systemd-boot verifies in a single step. dm-verity places a Merkle hash tree over the read-only /usr, so every block the kernel reads is checked against the tree at boot. Together they collapse an entire class of boot-chain problems — unsigned initramfs, kernel/initramfs version skew, unverifiable userspace — into one signed artifact.

# Build a minimal tar image with a few packages and custom files:
tine/bin/tine buck build //examples/image:demo.fedora

# Build a bootable GPT disk: UKI + dm-verity-protected /usr
tine/bin/tine buck build //examples/image:boot-demo.fedora

# Boot it in an ephemeral VM (needs /dev/kvm; without it, builds still work)
tine/bin/tine buck run //examples/image:boot-demo-vm.fedora

Because the disk is a Buck2 target rather than a shell script, the SBOM comes for free. The same action graph that installed the packages knows the full, exact set of inputs, and tine emits a CycloneDX document from that graph:

tine/bin/tine buck build --out /tmp/demo.cdx.json :demo[sbom][cyclonedx]

This is the part most teams get wrong manually. Generating an SBOM by scanning a finished image with syft tells you what is present; deriving it from the build graph tells you what was intended. When the two disagree — a scanner finds a package the graph does not list — you have caught a supply-chain anomaly rather than a formatting quirk. The images remain compatible with standard scanner tooling like grype and trivy, so this is additive, not a replacement.

The lock-in cost is real: adopting tine means adopting Buck2 and Starlark. That is a real language and a real build graph model to learn. It is a smaller commitment than maintaining a bespoke distro, but a larger one than adding a Dockerfile. Choose it when immutable, verifiable images are a core requirement, not a side quest.

Sysext Turns the Image into a Base Layer

A verifiable base image solves provisioning. It does not solve the day-two problem: how do you ship a debug toolchain, an OEM agent, or a locally rebuilt library onto a fleet of immutable hosts without rebuilding and reflashing the image?

The answer that Flatcar, Kairos, and the systemd ecosystem converged on is systemd system extensions (sysext). A sysext is a signed, verity-protected image containing a /usr/lib/extension-release.d/extension-release.NAME identity file. At boot, systemd matches that identity against the host and overlays the extension's files onto /usr and /opt using OverlayFS. Unmerge, and the host tree is exactly as it was.

# /usr/lib/extension-release.d/extension-release.debugtools
ID=_any          # matches any host, any release
ARCHITECTURE=x86-64

# Or pin to a specific OS so the extension moves only when the OS moves:
# ID=fedora
# VERSION_ID=42

The distinction between the two identity strategies is the whole engineering decision. ID=_any means the extension runs on any host and must therefore bundle every library it needs statically. A version-pinned extension (ID=fedora plus VERSION_ID) can dynamically link against the host's /usr but is rebuilt with every OS release. Amutable chose Buck2 partly for this reason: their /usr layering model with sysexts does not compose cleanly with NixOS's /nix/store symlink model, which is an honest architectural constraint rather than a preference.

A sysext is not a package manager, and treating it as one is the classic mistake. There is no dependency solver. There are no scriptlets, no %post, no triggers. Files outside /usr and /opt are silently ignored, and cross-extension collisions are not enforced — the topmost layer simply wins. If you need dependency resolution or post-install logic, sysext is the wrong tool and you should say so rather than bend it.

The mutability trap: merging an extension onto a writable host freezes the overlaid base directory read-only for as long as the extension is merged. systemd 256 added a mutable mode — write routing through /var/lib/extensions.mutable/ — that piped /usr and /etc writes back to the real tree. If you enable it, verify the systemd version on your fleet first; systemd-sysext --version is the cheapest way to avoid a surprise on a production appliance.

When to Reach for This Stack

tine, Buck2, UKI, dm-verity, and sysext form a coherent stack, and the coherence is the point: each layer assumes the layer beneath it produces a cryptographically verifiable artifact. That is also the reason none of it is worth adopting piecemeal.

• Verifiability as a requirement. If your threat model includes tampered images or supply-chain integrity, generating the SBOM from the build graph and signing UKIs with a hardware key is the defensible path. If not, a Dockerfile and an image scan are cheaper and sufficient.
• Underlying delta maintenance. Importing upstream distro packages with temporary deltas — rather than forking them permanently or re-declaring them in a functional language — is the specific problem tine exists to solve. If you are not maintaining a distribution's worth of patches, the value is lower.
• Day-two delivery. If the fleet needs toolchain layers, OEM agents, or single-component rebuilds on immutable hosts, sysext is the standard answer and the ecosystem supports it.
• Willingness to learn Starlark and Buck2. This is the real cost. The build graph model is more powerful than Make or Meson, and it will be unfamiliar to most of your team.

The generalizable lesson is not "use Buck2." It is that reproducibility is a property of your input declarations, and correctness follows from enforcing those declarations in a sandbox. Whether you get there through Buck2, Nix, or a homegrown graph, the moment you stop trusting timestamps and start trusting content hashes, the failure modes change from silent corruption to loud cache misses. That is a fair trade every time.