I told you so.

muster: Active Directory lite for a small Linux fleet

One directory, Kerberos, and a small agent per host: identity, access, security logging and host policy for a handful of Linux servers.

Most small fleets start the same way. There are a handful of servers, one administrator, and a file of SSH keys copied to every machine. Each host has its own local accounts, its own sudoers file, and its own idea of who may log in. Logs stay where they are written. Revoking someone means remembering every place their key went.

This works until it does not. A key leaks and has to be pulled from eleven machines. A new host gets the wrong sudoers file. A failed login storm happens on a server nobody was watching. The question “who can log in where, and who did?” has no single answer.

In the Windows world, Active Directory answers it: one directory of people, groups and machines, Kerberos for authentication, and Group Policy to keep hosts in line. muster takes the useful core of that idea and rebuilds it for a Linux fleet from stock open-source parts plus a small amount of Rust. This article describes what it does, how it is put together, and why it was built the way it was. A companion article, muster in practice, walks through the same system one diagram at a time and reports what a real rollout taught.

🔺 The shape of the system

The fleet muster serves is a small mesh. Two full hubs and two light hubs are joined by WireGuard tunnels with BGP routing over a private IPv6 range. Road warrior laptops and phones attach to the full hubs.

muster puts four things on top of that mesh:

  1. A directory. OpenLDAP holds every person, group, host, sudo rule, SSH key, OpenPGP key and WireGuard peer. It is the single source of truth.
  2. Kerberos. An MIT KDC authenticates people and hosts. Its principals live on the same directory entries, so there is one record per identity, not two.
  3. An agent on every host. musterd turns directory data into local WireGuard configuration, ships security events, and runs host policy. SSSD, a stock component, handles login, sudo and SSH keys.
  4. Collectors. muster-collect runs on each hub. It receives the event streams, checks their integrity, raises alerts, and pushes them to a phone through a self-hosted ntfy server.

An admin drives it all with one CLI, muster, and enrolls hosts with Ansible roles. Here is what that looks like in practice:

  • Adding a person is one entry and one principal.
  • Granting access to a host is a group membership.
  • Revoking a key is one delete, and every host stops accepting it within SSSD’s cache window.
  • Seeing every root login in the fleet is one alert list.

🔺 Choosing what not to build

The first decisions in muster’s record (it keeps 127 of them, each with their reasons) are mostly refusals.

  • No Windows. muster does not speak the Active Directory protocols and never will. The name “AD lite” describes the idea, not the wire.
  • No general log collection. muster collects authentication and security events only: logins, sudo, directory writes, KDC activity, and its own actions. It is not a log platform.
  • No X.509. This is the most consequential refusal. Kerberos already gives every host and person a cryptographic identity. Adding X.509 would mean running a public-key infrastructure, protecting its root, and handling revocation, for no gain the KDC does not already give. What muster does run is a narrow SSH host CA, so that a client need not trust a host key the first time it sees one; it signs host names, never people, and the companion article describes it. The fleet’s existing SSH CA for root certificates is a different key altogether. It stays where it is, as the break-glass path, and muster never touches it.
  • No Docker. The isolation muster needs comes from a real VM boundary.
  • No reinvented daemons. OpenLDAP, MIT krb5, SSSD and WireGuard all run as their upstreams ship and test them. muster writes Rust only for the glue nothing else provides: the agent, the collector, the CLI, the VM’s init, and the policy engine.

These refusals keep the system small. The whole Rust workspace is held to a line budget of 13,300 non-test lines across twelve crates, and the gate fails a change that exceeds it.

🔺 The hubs: a realm inside a microVM

The directory and the KDC hold the fleet’s most valuable secrets: the Kerberos master key and every principal’s long-term key. The hubs that run them also serve public websites and DNS. A compromised web server must not be able to read the realm.

So each hub runs the directory and KDC inside a Firecracker microVM, started through Firecracker’s jailer. The first plan used Nanos unikernels. A short spike showed that slapd and the KDC wanted a real Linux userland, and a stock guest kernel was simpler and better tested. The VM is spare:

  • The root filesystem is read-only, assembled from pinned Ubuntu packages unpacked with dpkg -x, and checked against a lock file of hashes.
  • The kernel is built from kernel.org source. Its tarball’s checksum must match a pin and kernel.org’s signed checksum list, verified against a pinned signing key.
  • Nothing for an intruder to use: no shell, no sshd, no package manager.
  • State (the database, the config, the stash, the keytabs) lives on a second virtual disk. That disk is the realm.

PID 1 is muster-init, a few hundred lines of Rust. It mounts the filesystems, reads a plain-text plan of steps, starts slapd and the KDC in order, reaps children, and reboots the VM if a daemon dies. systemd on the host then restarts the VM.

Inside the VM, the KDC talks to slapd over a local Unix socket with SASL EXTERNAL: the kernel vouches for the caller’s uid. No password or TLS link between them exists to be stolen.

The VM reaches the network through a host-only bridge. nftables on the host forwards the LDAP and Kerberos ports from the hub’s mesh address, and only from the mesh interface. No rule mentions a public address.

🔺 Time is a security property

One of the more instructive problems came from the clock. The VM had no time source after boot, so each hub’s VM drifted a fraction of a second from its host. That sounds harmless.

The two hubs replicate in OpenLDAP’s mirror mode, and conflicting writes to an entry are ordered by a timestamp the writing hub stamps. With the hubs 0.3 seconds apart, a write on the slower hub that closely followed a write from the faster one carried an older stamp. The faster hub discarded it while the slower hub kept it. The hubs diverged silently. This was reproduced in testing before it ever reached production.

The fix gives each VM a precision time source with no network exposure: the ptp_kvm device, through which the guest reads the host’s clock. chrony inside the VM uses it as its only source, with no NTP port and no command socket, and slapd starts only after the clock is set. On every host, the enrollment role now refuses a clock that chrony does not keep, because Kerberos and replication both depend on it.

🔺 Replication and the read-only copies

The two hubs are equal peers for reads and Kerberos. Either KDC can issue tickets alone, so a hub can be down without anyone noticing. Writes go to one designated write hub, which alone runs kadmind for principal and password changes. The hubs replicate with delta-syncrepl, which ships individual changes from an access log rather than whole entries.

The light hubs hold read-only directory copies, so hosts near them keep resolving users and keys when a full hub is unreachable. The copies hold no Kerberos secrets at all. Their replication excludes those attributes, and the hubs’ access rules refuse them to the copies’ identities as well, so the copies never receive them even through the access log.

The copies, too, were changed by a found failure. Plain syncrepl copies whole entries. When the two hubs were cut off from each other but not from a copy, and each changed a different attribute of the same entry, the copy kept the later hub’s whole entry, losing the other hub’s change. When the partition healed, nothing told the copy to look again. The copies now use delta-syncrepl from both hubs, exactly as the hubs do with each other, and converge the same way.

🔺 Who is an admin

muster separates a person’s everyday identity from their administrative one. A user alice logs in as alice@REALM. Administrative work uses the separate principal alice/admin@REALM, which maps to its own directory entry under ou=admins.

Holding the admin principal is not enough. Every administrative rule in the directory’s access control pairs two conditions in the same clause:

  • the caller is bound as an /admin identity;
  • that identity’s user is a member of the admins group.

Removing either the principal or the membership ends admin rights at once.

An audit of the deployed system found that this held in the directory but not in kadmind. kadmind’s access file was a static */admin@REALM *, which trusted every admin principal regardless of group membership. The fix is a small supervisor, muster-kadmind:

  • Every ten seconds it regenerates the kadmind ACL from the directory, with one explicit line per qualifying admin and never a wildcard.
  • When the list changes, it restarts kadmind. It stops the old kadmind before rewriting the file, so a failed write leaves no kadmind running rather than one with stale, wider rights.
  • A directory read that fails produces an empty ACL. The design always fails toward refusing.

🔺 Hosts: login, keys, and sudo

On every Linux host, SSSD is the bridge to the directory:

  • Login is by Kerberos password through PAM, or by SSH key.
  • SSH keys come from the directory through sss_ssh_authorizedkeys.
  • Sudo rules are directory entries naming users and hosts.
  • Access is a group: access-all, or access-<host> for a single machine.
  • Offline: SSSD caches enough to keep working through a short directory outage.

Enrolling a host is an Ansible run, not a network service. The host’s keytab is generated on the admin workstation, copied over, and shredded locally. Nobody can enroll a machine by asking the network nicely.

Enrollment found its own traps in the seams between muster and the operating system. The companion article lists them.

Through all of it, one rule is absolute. muster never touches break-glass access. Before any change, the client role takes a digest of root’s authorized keys, the SSH daemon configuration, the local account files, and every CA file sshd names. After its last change it takes the digest again. Any difference fails the run. If muster breaks, root by key or by the fleet’s SSH certificate still works, because muster was never allowed near them.

🔺 Security events with a hash chain

musterd on each host follows the journal for sshd, sudo, su, login and SSSD. On the hubs it also follows the KDC, kadmind and the directory’s access log. It parses each line into a structured event:

  • who acted;
  • by what method, with which key fingerprint;
  • from where;
  • with what result.

A line it does not recognise is kept as an unparsed event, never dropped. Secrets are redacted before an event exists.

Each event becomes a record in a per-host hash chain:

  • records are numbered in order;
  • each one carries the previous record’s hash;
  • its own hash is blake3 over a canonical encoding.

Editing, removing or reordering any record breaks every hash after it.

Records are written to a local spool and fsynced before the agent advances its read position, so a crash can repeat an event but never lose one. A segment of the spool is deleted only once every collector has acknowledged it. If the spool fills, the agent stops reading sources and records exactly one event saying so. It does not silently discard old data.

🔺 A transport without TLS

Records travel to the collectors over TCP on the mesh, and the transport is plain GSSAPI. The agent authenticates with the host’s keytab, the collector with its own service principal, and every frame after the handshake is encrypted with the Kerberos session.

TLS was rejected, and not just because it would need a CA. On this mesh a source address proves nothing. Some WireGuard peers carry a wide catch-all route, so a peer can originate traffic from any mesh address. Identity therefore has to come from cryptography, and Kerberos already supplies it.

The collector adds a check of its own, host binding. A record is accepted only if the authenticated principal is the host the record claims to come from. A compromised host can lie about its own events but cannot forge another’s. A broken chain is stored exactly as sent and raises an alert. The evidence is preserved, not repaired.

🔺 Alerts

The collectors run a fixed set of rules:

  • directory writes;
  • root logins;
  • repeated sudo failures;
  • unknown principals;
  • host binding failures;
  • chain breaks;
  • a full spool;
  • refused WireGuard plans;
  • SSSD failing over to a directory copy;
  • host policy failures and enforcement rollbacks;
  • a write hub that has gone silent.

Root logins with the break-glass certificate are exempt, and every other root login alerts.

Both collectors compute alerts independently. An alert’s id is a hash of its host, sequence number and rule, so both collectors agree on it. Only the write hub’s collector pushes to ntfy. If the write hub falls silent, the other collector takes over and says so first.

A push carries only the rule, host, principal and alert id. The details stay on the collector, read with muster alerts, over the same authenticated transport.

🔺 WireGuard from the directory

The mesh itself is configured partly from the directory. Each server’s WireGuard public key and mesh prefix live on its host entry, and each road warrior device on its owner’s entry. musterd renders them into wg0 (the mesh) and, on hubs, wg1 (road warriors).

This is the most dangerous thing muster does. The directory is reached over the mesh that the renderer configures. A renderer that removes the wrong peer can cut a host off from the very directory that would fix it. So the renderer is built to fail toward doing nothing:

  • Pinned peers (the hubs, set in local configuration and identified by key) are never touched.
  • Removal happens only when a peer is explicitly tombstoned on every reachable hub. Missing data, an unreachable hub, or two hubs that disagree all mean no removal.
  • At most one removal per run. Mass deletion needs an explicit flag.
  • Validation before applying:
    • every allowed address is inside the mesh range;
    • no two peers overlap;
    • each road warrior gets exactly one /128.
  • Routes follow peers. WireGuard adds no kernel routes of its own.
  • Any error at all leaves the interface as it was.

The renderer is split so that every one of these rules lives in pure code with no IO, which is where the tests aim. Every run records its plan as an event, so the collectors see what each host intended, whether or not it applied.

🔺 Host policy: Group Policy without scripts

The last major piece is host policy, the Group Policy analogue. Its content comes from the ComplianceAsCode project: every high-severity rule, every STIG category I rule, and a curated set of high-impact moderate ones. In muster’s form, each rule is a typed declaration:

  • a resource type, such as a sysctl, a file mode, a package, or an sshd directive;
  • the parameters for it.

There are no scripts. Checks, fixes and rollbacks are derived from the declaration. Where something cannot be typed, a read-only probe pinned by digest and run sandboxed stands in for it, and is listed as a candidate to replace.

The content tooling is Python and runs on the admin workstation only. It compiles 303 upstream rules into 132 typed rules and one bundle.json, deterministically. A rebuild from a clean tree must be byte-identical.

On the host, three crates do the work:

  • Collectors gather only the facts the assigned rules need. They emit properties, never secrets: for GRUB’s boot password, a host reports whether one is set and whether it is hashed, never the password or its hash.
  • The evaluator is a pure function from facts to outcomes. The dependency graph is checked for IO crates. Its outcomes distinguish not applicable from pass, and an input it cannot verify is an error, never a pass.
  • What-if evaluates a proposed profile against the facts the collectors already hold, without contacting any host. An admin can see what a change would fail before making it.

Assignment lives in the directory, on hosts or their groups, along with exceptions, which carry a mandatory expiry.

🔺 Enforcement is earned, not assumed

Policy started audit-only and stays that way by default. A rule is enforced on a host only when it has been promoted for that host’s ring: named rollout stages from canary to leaves to the hubs, defined by directory groups, with soak times between them.

Risky changes need more:

  • a change that could lock someone out, stop a host booting, or break compatibility needs a human approval entry naming the host;
  • a change that reboots needs an open maintenance window;
  • a halt entry stops a rule everywhere.

Every applier follows the same discipline:

  • snapshot first;
  • prefer a drop-in file to editing in place;
  • never widen a file mode;
  • health checks before and after;
  • a confirmation window.

An unconfirmed change, a failed check, or any error rolls back. When a host does roll back, the collector writes a fleet-wide halt for that rule, so one host’s bad experience stops the rest.

And again, break-glass comes first. Before anything applies, a guard checks every promoted rule. If one would touch root’s key or certificate login or the CA files, the entire run is refused unless an unexpired exception covers it.

🔺 Writing an LDAP client

One decision shows the project’s attitude to dependencies. muster first used the ldap3 Rust crate. In testing, the hub’s access-log reader stalled on one particular cursor, while the command-line ldapsearch read the same entries fine.

The cause was in the crate’s handling of encrypted SASL buffers. If fewer than four bytes of the next buffer had arrived, it returned an error instead of waiting for more. It could also drop messages when one buffer held several or a message spanned two. Whether it failed depended only on where TCP happened to split the response, which made it repeatable for a given result set and invisible for most.

Rather than patch around it, muster replaced the dependency with its own LDAPv3 client. It is small: BER encoding for the subset LDAP uses, SASL GSSAPI with a mandatory confidentiality layer, and paged searches. Its tests take real responses captured from slapd and cut them at every byte boundary.

🔺 Invariants as tests

muster states twenty-three invariants. Each one has a test named after it that tries to break it, not merely to observe it. A few of them:

  • No private key is stored in the directory except the KDC’s own principal keys.
  • musterd listens on nothing; the collectors listen on the mesh only.
  • Hosts read what they need from the directory and write nothing.
  • Directory copies accept no writes and converge with the hubs.
  • Directory copies never hold a Kerberos secret.

The test layers build on each other:

  • unit and property tests in each crate;
  • a development stack of local VMs: two hubs, a client, a copy;
  • a lab of full Ubuntu VMs with systemd and nested KVM, where the real Ansible roles run against fresh machines on both 24.04 and 26.04;
  • an acceptance suite against the production hubs.

The production suite checks that:

  • stopping either hub’s VM leaves the other’s KDC serving;
  • replication runs in both directions within seconds;
  • removing an admin’s group membership ends their kadmind rights;
  • no muster port answers on a public address.

🔺 The record

Every one of muster’s decisions is written down with its reason, and many record what was rejected and why. Several of the most important were not planned but discovered: the clock skew, the copy divergence, the kadmind wildcard, and the LDAP decoder. Each was found by a test or an audit before it caused harm, written up, decided, and fixed with a test that keeps it fixed.

That is perhaps the real lesson of building infrastructure for a small fleet. Scale is not the hard part. Correctness under partial failure is: two hubs that cannot see each other, a clock a fraction of a second off, a renderer that loses its directory mid-run. muster’s answer is to keep the parts few and stock, push every safety rule into code that can be tested without a network, prefer doing nothing over doing the wrong thing, and leave a way back in that it is never allowed to touch.