badtime: what a bot meets on my web server
A trap in front of a hundred-odd web sites: tarpits, a maze that looks like a library, TLS fingerprints, a render beacon, every port open as old factory equipment, and a dashboard behind proof of work and a passkey.
One server of mine carries a little over a hundred web sites. Most of the
requests it gets are not from people. Some are search engines, which I want.
Many are scanners looking for a WordPress login that was never there, a
.env file with someone’s cloud keys in it, or a router admin page from
2014. A growing share claim to be an AI company’s crawler, and some of
those are telling the truth.
I wanted three things: to stop serving the scanners, to cost them something for asking, and to see clearly what was happening. The result is a program called badtime, plus a few pieces around it. This post is how it fits together and what each piece does.
One rule first, because everything below follows from it: badtime only decides what my own server says to a connection someone else opened. It never sends anything to an attacker’s machine, never scans back, never “strikes back.” The address on a malicious request is often forged or belongs to someone else’s compromised box, so hitting back usually means hitting a victim, and in most places it is also a crime. Everything here is a way of answering, or of not answering.
🔺 Where it sits
Traffic arrives at a small router on port 443 that reads only the name in the TLS handshake and passes the connection to the right place: most sites to Caddy, a few to a unikernel. Caddy holds the certificates and serves the sites.
badtime is a Rust program that sits beside Caddy rather than in front of
it. Caddy asks it about any request that already looks wrong: a scanner’s
user-agent, a path like /wp-login.php, an address that has been flagged.
badtime answers “fine, serve it” or takes the request over. Ordinary visitors
never touch it, so it can’t slow a real page down.
Caddy also sends badtime a copy of every access-log line over loopback UDP. Nothing waits on that copy. If badtime stops, the sites keep working and the copies fall on the floor. That copy is how badtime sees normal browsing, which several of the checks below need.
Once a minute, a small exporter reads badtime’s recent verdicts and turns them into firewall sets. Flagged addresses lose every port except the web ports, where badtime keeps them busy instead. Repeat offenders lose everything.
🔺 What it does to a bot
A request badtime takes over gets one of these, picked at random so the same address doesn’t see the same thing twice in a row:
- Tarpit. The response starts normally and then arrives one byte at a time, for minutes. The bot’s worker sits waiting for a page that never finishes. On my side it costs a timer and an open socket.
- The maze. A page of links to more pages of links, forever.
- Garbage. A slow stream of random bytes, which confuses anything trying to work out what software the server runs.
- A gzip bomb. About 2 MB on the wire that inflates to about 2 GB. It goes only to clients that said they accept gzip, because anything else would just be wasting my bandwidth.
A second hostile request within a few minutes ends the game: the address is dropped at the firewall on every port.
Bots that already know they’re bots get a TCP tarpit even earlier. For a flagged address, the router on 443 never reaches Caddy at all. It hands the connection to a tarpit that answers with a TLS handshake one byte at a time, for up to ten minutes.
🔺 How it decides
No single signal is good enough on its own, so there are several, and they catch different kinds of client.
- Things nobody legitimate asks for: honeypot paths, a
robots.txtthat forbids a trap path (anything that fetches it after reading that has announced itself), known scanner user-agents, exploit strings in the URL, bursts of 404s, and HTTP methods my sites don’t use. Anything other than GET, HEAD or POST is a near-certain tell. The few sites that do use other methods (an admin console, a git server) are listed as exceptions. - Browsers behave like browsers. A real browser fetches the favicon, the stylesheet, the fonts. A script fetches the HTML and leaves. Because badtime sees the whole access log, it can tell the difference without being in the request path.
- A render beacon. The favicon is a fixed URL, so a smart bot can fetch it just to look human. So Caddy now rewrites every HTML page on the way out and adds a hidden one-pixel image whose address is unique to that single response, and it logs that address. A browser that parses the page fetches it within a second. A scraper that only reads the HTML never does, and it can’t fetch it in advance, because the address doesn’t exist until the page is served.
- Reputation, by network. Every address that visits is counted against its network (its ASN) for thirty days, along with whether it earned a hostile verdict. A network whose visitors are almost all scanners gets less benefit of the doubt. The state is kept as hashes, not addresses.
- Shared blocklists. badtime acts as a bouncer for CrowdSec, whose community list adds about twenty thousand known-bad addresses.
- Search engines are checked, not trusted. A user-agent saying “Googlebot” is left alone only when it comes from Google’s network.
🔺 TLS fingerprints
The newest signal reads the TLS handshake itself. Every client’s first message lists the ciphers, extensions and protocols it supports, in a way that depends on the software, not on what the client claims to be. A JA4 fingerprint condenses that into a short string. Chrome has one shape, Firefox another, Safari another, and Python, Go and curl each have their own.
A small helper program reads a copy of incoming handshakes straight off the network interface. It is the only part with permission to capture packets, and it sits outside the traffic path entirely. It computes each connection’s JA4 and hands it to badtime with the client’s address and port. Caddy’s log line carries the same address and port, so every request can be matched with the handshake it arrived on.
Each verdict line now says what the client claimed to be and what its
handshake says it is. The first afternoon was instructive. A run of
“iPhone Safari” visitors turned out to be Go programs. A batch of “Chrome”
visitors were Rust programs. Of the caught requests that claimed to be a
browser, every one came from software that wasn’t. One scanner on a cloud
provider sent a few hundred requests for .env files from five addresses
with the same fingerprint. That fingerprint is now on a block list, which
works on every site, including the ones whose TLS badtime never sees inside.
The handshake check for browser claims is in log-only mode for now. It records what it would have done, so I can look for real people it would have caught before it starts acting.
🔺 A maze that looks like a library
The first version of the maze lived at /maze/, which is not subtle. It
now lives under a path that looks like an archive, and its pages look like
an archive’s pages: a breadcrumb trail, a title made of plausible words, a
paragraph, a few links, and “Older entries.” Nothing on them says maze.
Each link is the current path plus one more segment, so a crawler sees the
URLs of one site getting longer and more specific, as if it were drilling
into a real collection. The last segment ends in a token that looks random
but carries the page’s depth, scrambled and checked, so the logs can say
how far in each bot is. Old /maze/ links still work, so nothing that was
already inside falls out.
🔺 Every port open, as old equipment
Most scanners don’t stop at the web. Once an address is flagged, every other TCP port on the server is now “open” to it. The firewall quietly redirects those connections to a decoy listener, which asks the kernel which port was originally dialed and answers as that service would.
Each flagged address meets one device, chosen from its address, so its scan results stay consistent: a laser printer, a document scanner, an industrial controller from one of two vendors, or a serial-to-network gateway. Each port gives that device’s real opening handshake. SSH offers key exchange algorithms from 2006. Telnet does the right option negotiation, then shows the device’s own login prompt. The industrial ports answer the protocols those controllers speak (Modbus, S7, EtherNet/IP), including the identification queries scanners use to catalog them. Databases send their greeting and then refuse the login. RDP answers like Windows XP. Any other port answers whatever the client speaks: a TLS hello gets a TLS handshake-failure alert, an HTTP request gets the device’s web page.
Nothing ever succeeds. No login works, no file is accepted, and the mail port answers “try again later,” so a real sender caught by mistake simply retries. Each session is held open as long as the scanner is willing to wait.
I checked the decoy against nmap, whose service detection is a large database of what real software sends. It named every service correctly. Getting there taught me something. My first version dribbled its replies a byte at a time, as a tarpit should. nmap read the first three bytes of the RDP reply and decided it was a Lineage II game server, and read “220 “ of the mail banner and decided it was OpenBSD’s spamd. (spamd is itself a tarpit that answers a byte at a time, which is a good joke at my expense.) Real devices send a handshake in one piece. Now the decoy pauses the way a slow embedded device would, sends the reply whole, and does its time-wasting by holding the session open.
The decoy answers TCP only. Answering UDP (SNMP, BACnet) to an address that may be forged would turn the server into an amplifier aimed at whoever owns that address.
In its first half hour, the decoy answered about fifty addresses. Most were SSH clients, but there were also internet-wide scanners announcing themselves by name, a cryptocurrency miner trying to subscribe to a mining pool, a WebLogic exploit probe, and someone trying to log into it as a stock-trading system.
🔺 Watching it
badtime logs one JSON line per decision, which is complete and unreadable.
badtime watch turns that into one short colored line per verdict: time,
action, reason, address, site and path, network, the likely client software
and handshake, and the browser it pretended to be. Repeat hits from the
same source update their line in place with a counter, so a scanner
hammering away is one line, not a screenful. --summary totals any time
window. --sanitize blacks out my own hostnames so I can post screenshots.
There are also two pages. The server’s bare IP address serves a “no site here” page, and below it is a live version of the charts, rebuilt every minute as plain HTML and inline SVG. It names no site and no address.
🔺 A dashboard behind two locks
The fuller dashboard has its own name, and getting to it takes two steps.
First, proof of work, through Anubis. The browser has to find a number whose hash, with the challenge, starts with enough zeros. A real browser does this in well under a second. A script that can’t run JavaScript stops there. Anubis’s default policy is gentler than I wanted: it gave most browser-looking visitors only a short “wait a moment” check. The dashboard has its own policy, and every request has to do the work.
Second, a passkey. The sign-in goes through the network’s single sign-on, and only accounts in the admins group get through. There is also a guest view on the same sign-in page, with its own password, which shows the same charts. That password lives in a file only this one sign-in service reads. Otherwise the same program serves sign-ins for several other sites on the server, and a guest account on one would have been a guest account on all of them.
From inside my private network the dashboard has a second name that skips the proof of work and goes straight to the passkey. That name only answers connections from inside, and the public side cuts them off before saying anything.
🔺 What it runs on
- Caddy for the sites, rebuilt with a module that can rewrite response bodies, which is what the render beacon needs.
- nginx, built from source at the current stable release, as the port-443 router: it reads the TLS name, and it now also caps how many connections one address can hold, gives a client five seconds to say hello, and closes anything on 443 that isn’t TLS at all.
- nftables for the firewall sets and the decoy’s redirect.
- badtime itself, in Rust, as one static binary, plus the packet-reading helper, which is the only part with packet-capture permission.
- Anubis for proof of work, Keycloak for single sign-on with passkeys, and a small sign-in service in front of each protected site.
- Ansible for all of the above, so that rebuilding the server rebuilds the traps too.
The numbers will look different by the time you read this. The bots don’t stop, and neither, it turns out, do I.