muster, built: the parts, the build, and what it can do

The first article on muster explained why a small Linux fleet would want Active Directory’s core ideas and which of them muster refuses. The second walked through the running system one picture at a time and reported what the first rollout taught. This one is the reference piece. It lists the parts as they stand today, shows how to build a release from a clean checkout, and describes what a fleet can do with the result. Since the last article, host policy has grown from a curated set of severe rules into the full Ubuntu STIG and CIS Level 1, enforced ring by ring, so a good part of this piece is about that.

🔺 The parts

muster is three toolchains around one Rust workspace.

The Rust workspace: thirteen crates, described below.
The hub image toolchain: shell scripts that build the microVM the directory and the KDC run in.
The policy toolchain: Python on the admin workstation that turns upstream compliance content into one typed bundle.

Ansible roles install all of it, and a handful of operator scripts drive the stages of a rollout.

🔺 The workspace

Every crate forbids unsafe code, and every library keeps IO at its edges, so the rules that matter can be tested without a network.

Crate	What it does
`muster-common`	Entry types, validators, the security event model and the hash chain. No IO.
`muster-ldap`	muster’s own LDAPv3 client: SASL GSSAPI over the network, SASL EXTERNAL on a hub’s local socket, typed reads and writes.
`muster`	The admin CLI. Every directory write runs as the admin’s `/admin` principal.
`musterd`	The agent on every host: log shipper, WireGuard renderer loop, policy runs.
`muster-logsrc`	Log sources, parsers, redaction, the local spool, and the transport to the collectors.
`muster-collect`	The collector on each hub: chain storage, host binding, alert rules, push.
`muster-render`	The renderer framework and the WireGuard renderer, split into a pure planner and a small applier.
`muster-init`	PID 1 of the hub microVM.
`muster-policy`	Policy bundle types, the profile compiler, the pure evaluator, and what-if.
`muster-hostfacts`	Read-only fact collectors, one per fact type, fed through a runner that tests replace with fixtures.
`muster-facts`	Policy assignment and exceptions from the directory, the policy run, drift events, and the collectors’ fact store.
`muster-enforce`	Typed appliers with snapshots, confirm windows, health gates and rollback.
`muster-sshca`	The SSH host CA: trust list, signer, renewal and revocation.

The workspace is held to a line budget, per crate and in total. Today the total stands at 15,672 non-test lines against a ceiling of 15,700, and the gate fails any change that would cross it. The budget is not vanity. It forces a choice every time a feature arrives: either it earns its lines, or something else gives some back.

🔺 The hub image

The directory and the KDC run inside a Firecracker microVM with a read-only root filesystem, no shell and no package manager. Three scripts build it:

stage.sh downloads pinned Ubuntu packages and unpacks them with dpkg -x, checking each against a lock file of hashes. Nothing is installed on the workstation.
kernel.sh builds the guest kernel from a pinned kernel.org tarball, checked against both a pinned hash and kernel.org’s signed checksum list.
rootfs.sh builds muster-init as a static binary and assembles the root image and the directory configuration templates.

🔺 The policy toolchain

policy/build.sh fetches the ComplianceAsCode project at one pinned commit and runs five steps: select, lift, fact catalog, validate, bundle. The output is bundle.json, which is what every host reads. The build is deterministic. A gate rebuilds it in a temporary directory and fails unless the result is byte-identical, so a policy change shows up as a reviewable diff, never as an accident of build order.

🔺 Building it

🔺 Before the first build

muster ships for one realm. The Kerberos realm, the directory base, the mesh domain and the mesh address space are constants in the code, not configuration. To run another realm you change them in one tree before the first build and build everything from that tree. A realm cannot be renamed after its hubs are bootstrapped. Making those values runtime configuration would not remove that constraint; it would only move the failure somewhere harder to see.

The hubs need very little: Ubuntu 24.04 or 26.04 with systemd, /dev/kvm, a tsc clocksource so the VM can take its time from the host, chrony, and a WireGuard mesh already up. muster does not build the mesh on day one. It can render it later.

The admin workstation needs the Rust toolchain named in the repository, the musl target, Python with PyYAML, Ansible, and the usual tools for a kernel build. There is no Docker anywhere. The isolation muster needs comes from a VM boundary, and the build needs none at all.

🔺 A release, step by step

A release is a frozen directory of files with their hashes: the guest kernel, the root image, the directory copy template, the three binaries (muster, musterd, muster-collect) and the policy bundle. The roles refuse any file whose hash differs from the frozen one, on the controller and again on the host.

Stage, kernel, root image. Run the three image scripts. Each checks its inputs against pins before it uses them.
Binaries. Build the three binaries as static musl executables, with MIT Kerberos linked in statically from a source tarball whose hash and release signature are checked first. A static binary runs on any distribution of the right architecture, which matters more than it sounds. The first releases linked the workstation’s C library and would not start on a host one release older. Since the static build, the build script refuses an output that has a dynamic interpreter or a versioned C library symbol.
Policy. Run the policy build and its byte-identity gate, then copy bundle.json in.
Freeze. Copy everything into a release directory named after the date and the commit, write SHA256SUMS, and make every file read-only.
Prove it reproducible. Touch a source file, rebuild, and compare with the frozen bytes. Two builds from the same commit must be equal.

A release is built only from a clean commit, and a release for the public is cut on a separate branch with a single commit, so its history carries nothing but the code.

🔺 From release to running fleet

The first realm comes from one command, muster-live bootstrap. Run with --plan, it probes everything read-only and prints each step with what it would change. Run for real, it builds and freezes the release, starts the write hub, hands the directory to the second hub, issues the first admin tickets, and checks each stage before the next.

After that, hosts are enrolled one at a time, canary first, by muster-live onboard. It installs chrony where the host lacks it, adds the host to the inventory, runs the client role in check mode and then for real, verifies logins, and folds any local account with the same name into its directory identity.

Each stage closes with an acceptance script that runs against the real fleet. The login acceptance alone covers key, password and Kerberos logins, sudo rules arriving and leaving, access refused and revoked, break-glass from inside and outside the mesh, and logins with both hubs stopped. Nothing moves on until its gate passes.

🔺 Host policy grew up

The first two articles described host policy as 132 rules lifted from a curated selection of severe upstream controls. That scope was deliberately small. Today the content covers the Ubuntu 24.04 STIG and CIS Level 1 for servers, lifted from the benchmarks’ own control files.

Benchmark	Controls	Covered
STIG category I	16	16
STIG category II	161	119
STIG category III	17	17
CIS Level 1, server	232	175

From 603 upstream rule ids, the lift produces 271 typed rules. That there are fewer rules than ids is the point. Upstream often has several rules for one intent, one per distribution or per variant, and muster collapses them only when they share both the intent and the unit an exception would apply to. Individual sysctls and individual sshd directives stay separate rules, because a host may need a waiver for one and not the next.

The gaps in that table are not silence. Every control the benchmarks name but muster leaves out, 134 of them, is listed with a reason: a desktop setting on servers, a choice the fleet made the other way, or a check that cannot yet be typed safely. A benchmark claim without its list of exclusions is not a claim anyone can review.

🔺 Profiles, roles, exceptions

A host is assigned a profile in the directory. The fleet profile extends a layer that selects every benchmark tier, which extends the smaller layers beneath it, so a host can be held to less while it is being brought up.

Roles subtract. A directory copy runs slapd, a web server runs nginx, a mail server runs dovecot. Each of those breaks a rule that says “this daemon should not be installed”. A role on the host’s directory entry removes exactly those rules for exactly that host. It does not silence them anywhere else.

Exceptions are the third tool. Each one names a rule, the hosts, an owner, an approver, a justification and an expiry date. The evaluator ignores an exception that has expired or is missing a field. A waiver nobody has to renew is just a quieter way of turning a rule off.

🔺 What-if before anything ships

The collectors keep every host’s latest facts. That makes a what-if cheap: the deployed CLI evaluates a proposed bundle against the facts the fleet already reported and lists, per host, every outcome that would change. A new bundle ships only when that list holds no surprise. For the STIG release the list showed only the expected new rules and a handful of changed tailoring values. When a later change made the sshd rules not applicable on hosts without sshd, the what-if showed no change on any server and exactly the intended change on the one host without an SSH daemon.

🔺 Rings, soaks, and the vendor tool

Enforcement is still earned rule by rule. Of the 271 rules, 20 are currently promoted for enforcement. Each moves through four rings, canary, leaves, light hubs and hubs, with soak times of one, two, three and three days. A failed health gate or a new failure in a ring halts the rule’s rollout, and a host that has to roll a change back halts the rule everywhere.

The fleet’s Ubuntu 24.04 hosts also run Canonical’s own STIG tool, the Ubuntu Security Guide, which can both audit and fix. Two tools fixing the same setting to different values is a fight neither wins, so the two were aligned first: muster’s tailoring and the vendor tool agree on one value per setting, and each resolved disagreement is recorded. Each host then went through the same sequence. Arm a dead-man timer, apply the vendor tool’s fixes, and check that login, routing, WireGuard and the firewall all still work. Only then is the timer cancelled; if the checks fail, or the host cannot be reached at all, the timer puts the authentication, firewall and kernel settings back as they were before the fixes. Then re-audit, and only then turn on muster’s enforcement. On the five 24.04 servers the vendor audit went from between 69 and 147 failures per host to none on four of them and one on the fifth, a check that wants audit logs offloaded somewhere that host does not send them. No host lost a listening port or a running service on the way.

The 26.04 hosts have no vendor benchmark yet. There muster is the only tool, and the same content audits them.

🔺 What you can do with it

Here is what the finished system lets a small fleet do, roughly in the order it pays off.

Answer who can log in where, in one place. Access is a group membership, sudo is a directory rule, keys live on the person’s entry. Revoking someone is one change, and it reaches every host within one cache interval, by every login method.
Keep working when the directory cannot be reached. Either hub can serve logins alone, read-only copies serve hosts near them, and a host’s cache carries its logins through a full outage.
Prove the security log was not edited. Every host’s events form a hash chain stored by two independent collectors. muster log verify names the first record that was changed or removed.
See the fleet against a benchmark, continuously. Every host audits itself against the STIG and CIS content on a schedule, and a new failure raises an alert. No scanner logs in from outside, and facts leave the host as properties, never secrets.
Ask what a change would break before making it. What-if runs against the facts already collected, without touching a host.
Fix drift safely. A promoted rule is reapplied when a host drifts, with a snapshot, a confirm window and a rollback, and one host’s bad experience halts the rule everywhere.
Trust host keys without trusting first use. The SSH host CA signs each host’s names, with an offline master that signs only the list of signers.
Render the mesh from the directory. WireGuard peers for servers and road-warrior devices come from directory entries, planned by pure code that refuses rather than guesses.
Run on hosts nobody manages. A static .deb with no dependencies installs the agent on a distribution the roles do not cover.
Reach past Linux. The agent now runs on OpenBSD too. There the log source is authlog rather than the journal, privilege goes through doas, services are reloaded with rcctl, and Kerberos comes from the base system’s Heimdal. The router that joins the mesh from outside the fleet ships its logins and doas events to both collectors like any other host, and its chain verifies.

One more is under way, and it is the one Active Directory is best known for: signing in at a workstation’s own console with a directory password and getting a Kerberos ticket for the session. The servers already do this over SSH. On a workstation, the client role can now turn on directory sign-in for a host that has no SSH daemon, and a small script folds the local account into the directory one: a ZFS snapshot first, then the home directory re-owned to the directory uid, the local account removed, its local groups kept by name, and a separate local admin as the way back.

🔺 What the latest rollout taught

The first rollout found its surprises in the seams with the operating system. The policy rollout found most of its surprises in the benchmark content and the tooling around it.

An error is not a pass, and that held. One banner rule matched the operating system’s name case-insensitively. The agent’s regex build leaves out Unicode case folding to stay small, so the pattern never compiled, the collector skipped it, and the rule reported error rather than a quiet pass. A test now compiles every pattern in the bundle with the agent’s own build.
Not applicable is a real answer. A workstation reached through its VPN’s built-in SSH has no SSH daemon. Every sshd rule there reported error, because there was no daemon to ask. Those rules now apply only where the SSH server package is installed, and the what-if proved the change touched nothing else.
Names are code. The profile layer selecting every benchmark tier was first called after the STIG. The evaluator turns on ordered comparison of cipher lists when a profile’s lineage names the STIG, so the name alone changed outcomes. It was renamed.
Measure the canary. Before the bundle reached anyone else, its first run on the canary took 68 seconds and peaked at 22.6 MB, against 91 seconds and 25.5 MB for the smaller bundle it replaced. The numbers were the gate, not a curiosity.

None of these were found in production by a user. Each surfaced on the canary or in a gate, became a test, and went into the decision record, which now holds 158 entries.

🔺 The shape of the whole

Three articles in, the description has stopped changing much, which is a good sign. muster is still one directory, one KDC, one agent per host, two collectors and a CLI, built from stock daemons and a budgeted amount of Rust. What changed is how much of a host’s state it can vouch for. It started by answering who can log in. It now also answers whether each host is configured the way a published benchmark says it should be, with the exclusions listed. And the way back in, the recovery path muster is never allowed to touch, is still there.