remind.rsvp, or a lot of machinery for a casserole

remind.rsvp does one small job. Somebody posts an event, you say you’re going, and a reminder shows up before it starts. You don’t need an account, an app or a phone number. There are no texts, no tracking pixels and no photo feed. The pages are built to WCAG 2.2 AAA, so the text is 7:1 contrast and the buttons are big enough for a thumb in a mitten.

That’s the front of the house. This post is about the back, because the back is the fd53 network, and fd53 is where I keep all my bad ideas that turned out to work.

Audience: operators comfortable with Linux, systemd, DNS and WireGuard. Also anyone who’s wondered why an RSVP button needs BGP. (It doesn’t. It has it anyway.)

Our running example is the Whittier Garden Club’s Fall Seed Swap, Saturday at 2:00, Whittier Park Building. You’re bringing tomato seeds and, apparently, a casserole.

🔺 1. Overview

The master plan:

One Rust binary (rsvp), Axum and PostgreSQL, about 47,000 lines. It serves the pages, sends the mail and runs the CLI.
Four copies of it, called colors. Blue in Chicago, red in Toronto, green in Washington, gold in Los Angeles. Blue holds the database primary; the other three stream copies of it.
The fd53 mesh ties them together: WireGuard tunnels with BIRD running BGP over a private IPv6 range. Every color has its own /48 and its own private ASN, like a tiny ISP that only carries RSVPs.
Shared fd53 services do the rest: DNS, mail, sign-in, push notifications, backups and chat.

Here’s what that looks like when you click Going.

🔺 2. DNS finds you the nearest color

Your browser asks for remind.rsvp. The zone is served by ns-01 and ns-02, the fd53 nameservers, running PowerDNS with LUA records. The record has no fixed address. Instead it carries a small program and a list of colors, each with a geohash:

;include('_gh')include('_gn')return N{{ip,'dp3wjz'},{ip,'dpz83d'},{ip,'dqcjr0'},{ip,'9q5ctq'}}

dp3wjz is Chicago, dpz83d Toronto, dqcjr0 Washington, 9q5ctq Los Angeles. PowerDNS looks up where your resolver sits (using EDNS client subnet when it’s offered), decodes the geohashes, and answers with the nearest color. If it can’t place you, you get the first one in the list, which is the primary.

Two things bit me here, so they get to bite you first:

255 bytes. Every string in a LUA record has to fit in 255 bytes. Go over, and PowerDNS quietly splits it into two strings, which then fails as SERVFAIL. That’s why the helpers live in their own _gh and _gn records and the real record only includes them.
The free geo database has no accuracy radius. PowerDNS 4.8 drops any location without one, so every lookup came back as 0,0, somewhere off the coast of Africa, and “nearest” became “random.” A small tool rewrites the database with a radius added, and a timer on my workstation refreshes it on the 3rd of each month.

Something has to notice when a color falls over. rsvp-dns runs on FRA-01 (the PowerDNS hidden primary, in Germany) every 30 seconds. It checks each color’s /healthz, removes a dead one from the record, and pushes the zone to the secondaries. No load balancer is involved. I tried a NodeBalancer for less than a day, then deleted it. DNS and WireGuard do the job, with no extra box to pay for or to break.

Check it yourself:

dig +short remind.rsvp A
dig +short remind.rsvp A @ns-01.rns.sh +subnet=24.0.0.0/8

🔺 3. The page comes from close by, the yes goes to Chicago

You landed on, say, red in Toronto. Caddy there takes the TLS connection and hands it to the local app. If the local app is down, Caddy hands it to another color’s app over the mesh instead (lb_policy first). You never see the difference.

The event page is a read, and reads stay local. Red, green and gold each hold a streaming replica of blue’s database, and the app uses it as long as the replica’s heartbeat is under 30 seconds old. If replication stalls, the app quietly goes back to the primary. Showing you a stale headcount to save 10 ms isn’t worth it.

Your Going is a write, and every write goes to the primary in Chicago, across the color’s WireGuard tunnel. Every color has a direct tunnel to every other color, so it’s one hop over the mesh:

From blue (Chicago) to	Round trip
red (Toronto)	10.6 ms
green (Washington)	17 ms
gold (Los Angeles)	44 ms

Even from LA, that’s less time than it takes you to decide whether to tick “plus one.”

If Chicago itself goes away, switch.sh failover first powers off the old primary through the Linode API, then promotes a replica. The order matters. Two primaries both certain they’re in charge is how you lose data, so the old one gets fenced before the new one is crowned. A planned switch takes about ten seconds; in the last one, 7 of 1,895 requests failed.

Certificates are the funny part. At first every color ordered its own Let’s Encrypt certificate. Let’s Encrypt allows five certificates a week for the same set of names, and four colors plus one redeploy use that up by Wednesday afternoon. Now FRA-01 orders a single certificate over DNS-01 and copies it to every color, with ZeroSSL as the fallback.

🔺 4. The reminder: mail that knows its way home

Saying yes queues email. The queue is a table, the outbox, in the same database as your RSVP, so there’s no separate mail queue to lose. No color runs its own mail server. The sender hands each message over the mesh to mx-02, mail.remind.rsvp, which is remind.rsvp’s own MTA with its own address and DKIM key:

RSVP_SMTP_URL=smtp://[fd53:118:1000::10]:25

Every message gets its own envelope sender, bounce+<id>.<tag>@remind.rsvp. That’s VERP. When an address bounces, the bounce comes back to the exact message that caused it. mx-02 pipes it to the app’s /inbound/bounce, the app marks the address, and no more mail goes there. Caddy accepts /inbound only from the mail server’s own addresses, and only with a valid signature header, checked before the body is even read.

The reminder includes a calendar invitation. Tap Yes in your calendar app and the reply goes to invites@remind.rsvp, which mx-02 also pipes back to the app, and your RSVP updates. If you reply to that invitation with an actual sentence (“can I bring two casseroles”), it gets forwarded to the garden club’s own reply address. It only does that for replies to mail we sent, so it can’t be used as an open relay.

Every hour, a probe on mx-02 RSVPs to a test event, waits for the email to arrive, and complains if it doesn’t. It’s the closest thing remind.rsvp has to a canary. The probe’s RSVPs are deleted after a day.

🔺 5. Parsing untrusted input

Mail, uploaded pictures and import files (Eventbrite exports, Google Forms CSVs, .ics files) all arrive from people I don’t know. Parsers have bugs. A bug in a parser that runs inside the main process sits right next to the database login and every secret the app holds.

So the parsing moved out. rsvp parse is the same binary in a different mood. systemd starts a fresh copy for each job, over a socket, with nearly everything taken away:

DynamicUser=yes
PrivateNetwork=yes
IPAddressDeny=any
ProtectSystem=strict
InaccessiblePaths=-/etc/rsvp.env -/etc/rsvp.db.env -/opt/rsvp -/var/lib/postgresql
RestrictAddressFamilies=AF_UNIX
SystemCallFilter=~@privileged @resources @mount @clock socket socketpair connect bind listen accept accept4
MemoryMax=256M
RuntimeMaxSec=20

It has no network and no secrets, and the only place it can write is its own scratch space. It reads one job from the socket, writes one answer, and exits. The main process treats the answer as untrusted input anyway.

It also fails closed. If the worker doesn’t answer, the job fails; it never falls back to parsing in-process, because that fallback is exactly what a crash-the-worker attack would be after. Mail gets a 503, the mail server delivers it again later, and nothing is lost.

Passkey checks get the same treatment in a worker of their own, since anyone on the internet can try to sign in.

Guests never sign in. Hosts do, with a passkey. There are no passwords to leak, because there are no passwords.

Hosts who already have an FD53 account can use Sign in with FD53. That’s Keycloak at sso.itys.net, part of muster, the fd53 directory: LDAP, Kerberos, and an agent on every server. remind.rsvp is an ordinary OpenID Connect client using the code flow with PKCE. Keycloak’s own sign-in is also a passkey, so this route is no weaker than signing in directly.

There’s one exception. The actions that could take an account or a group away from you (changing your email, removing a passkey, changing a member’s role, and a few more) ask for your remind.rsvp passkey again, right then, however you signed in. A borrowed laptop or a stolen cookie can’t do them.

🔺 7. Event chat, on tern

A host can switch on chat for an event. Guests who are going and the hosts get a chat room, in the browser, with no account. The chat runs on tern, the fd53 IRC and XMPP network. Its relays are ns-01 and ns-02 (yes, the nameservers; they’re not busy), and its core is on FRA-01.

The trick is a signed ticket. remind.rsvp holds an Ed25519 key. When you open the chat, it signs you a ticket that’s good for 12 hours:

<room>.<name>.<gid>.<exp>.<sig>

tern’s core knows the matching public key (one guest_issuer line in its config) and admits the ticket to that one room and no other. Event rooms are all named rsvp-…, and regular tern accounts can’t see or join them. It’s a separate side door that only opens for guests.

tern is built around end-to-end encryption. These rooms don’t have it. A guest in a browser, with no account and no keys, can’t do end-to-end encryption, so the relays and the core can read an event chat the way any IRC server reads any channel. Rooms live in memory only, up to the last 500 messages or 7 days. The privacy statement says all of this wherever a host turns chat on. “The garden club is talking about seed potatoes” didn’t need the same protection as the rest of tern, but nobody gets told it’s private when it isn’t.

The chat window is gamja, a small web IRC client. It talks to wss://irc.fd53.net/socket. The same client, with a SCRAM-SHA-256 patch of ours, is what tern account holders use at https://irc.fd53.net.

🔺 8. Everything else on the mesh

Follow a group by email, Atom feed, or push. Push goes through ntfy at ntfy.itys.net (on BOOK-01). The app can publish to its topics but can’t read them, and you can read them but can’t publish.
Backups: the colors are backed up nightly with restic to B2. A backup you haven’t restored isn’t a backup, so the primary got a restore drill on a throwaway machine before it counted.
Deploys go one color at a time. Each run gets a throwaway test database for the test suite, and cargo-deny has to pass too. A color whose parse worker fails /healthz gets rolled back.
Every response carries X-Clacks-Overhead: GNU Terry Pratchett. A man is not dead while his name is still spoken, and every page remind.rsvp serves says his. GNU Terry Pratchett.

🔺 9. Why bother

You could put all of this on one VPS, and for a garden club you’d probably be fine. None of this was planned as a set. Each piece is boring on purpose and documented, and was added because something broke, or was about to: four cities once two servers in one Chicago data center looked like one outage from zero, the shared certificate after the rate limit, the parse worker after a dependency review listed every stranger’s file format the main process was reading.

Adjust the colors to taste. Mine are named after crayons.

If you have an event coming up, kick the tires at remind.rsvp. And if you RSVP to the seed swap, please bring the casserole.