Guix does make fixing this cheap: the whole system configuration is one Scheme file in git, and guix system roll-back takes seconds when a change breaks something. Over the last months I worked through the list, roughly in order of effort. This is what stuck, written so it applies to any Guix System install. Nothing here needs a custom kernel.

If you’re on a stock linux-libre kernel, one caveat up front: it ships no CPU microcode, and half of the Spectre-class mitigations live in microcode. On an affected part, /sys/devices/system/cpu/vulnerabilities/* will read “Vulnerable” no matter what you set below. The fix is the nonguix channel (kernel plus microcode-initrd), which is an unofficial channel with no security team of its own. I use it; decide for yourself.

sysctl

This block extends the two defaults rather than replacing them:

(use-modules (gnu services sysctl))

(modify-services %desktop-services
  (sysctl-service-type config =>
    (sysctl-configuration
     (inherit config)
     (settings
      (append (sysctl-configuration-settings config)
              '(("kernel.dmesg_restrict" . "1")
                ("kernel.kptr_restrict" . "2")
                ("kernel.yama.ptrace_scope" . "1")
                ("kernel.kexec_load_disabled" . "1")
                ("kernel.unprivileged_bpf_disabled" . "1")
                ("net.core.bpf_jit_harden" . "2")
                ("kernel.perf_event_paranoid" . "3")
                ("kernel.io_uring_disabled" . "2")
                ("kernel.sysrq" . "0")
                ("dev.tty.ldisc_autoload" . "0")
                ("fs.protected_fifos" . "2")
                ("fs.protected_regular" . "2")
                ;; Network
                ("net.ipv4.conf.all.rp_filter" . "1")
                ("net.ipv4.conf.default.rp_filter" . "1")
                ("net.ipv4.conf.all.accept_redirects" . "0")
                ("net.ipv4.conf.default.accept_redirects" . "0")
                ("net.ipv6.conf.all.accept_redirects" . "0")
                ("net.ipv6.conf.default.accept_redirects" . "0")
                ("net.ipv4.conf.all.send_redirects" . "0")
                ("net.ipv4.conf.default.send_redirects" . "0")
                ("net.ipv4.conf.all.accept_source_route" . "0")
                ("net.ipv4.conf.default.accept_source_route" . "0")
                ("net.ipv6.conf.all.accept_source_route" . "0")
                ("net.ipv4.tcp_syncookies" . "1")
                ("net.ipv4.icmp_echo_ignore_broadcasts" . "1")))))))

Two of these are worth explaining.

kernel.io_uring_disabled=2 has the best evidence behind it. Google’s kCTF numbers for the year to mid-2023: 60% of exploit submissions went through io_uring, and it featured in every submission that bypassed their mitigations. The sysctl has existed since 6.6; 2 means io_uring_setup fails with EPERM for everyone. On a desktop the only thing I found using it was libuv (Node, Electron), which falls back to its threadpool on that error. Recent PostgreSQL and QEMU builds can use it; if you run those natively, 1 plus kernel.io_uring_group is the softer setting.

kernel.yama.ptrace_scope=1 is the one you’ll notice: gdb -p and strace -p on a process that isn’t your descendant need sudo from now on. Chromium, Firefox and Wine already call prctl(PR_SET_PTRACER) for their own crash handlers, so they keep working.

Left out on purpose: anything that disables unprivileged user namespaces. On Guix that breaks the unprivileged daemon (see below).

Kernel arguments

The rest goes on the kernel command line. Keep %default-kernel-arguments at the end or you lose Guix’s module blacklist:

(kernel-arguments
 (cons* "slab_nomerge"
        "randomize_kstack_offset=on"
        "page_alloc.shuffle=1"
        "init_on_alloc=1"
        "proc_mem.force_override=never"
        %default-kernel-arguments))

init_on_alloc=1 is the only one with a measurable cost: upstream measured +7.75% sys time on hackbench and -0.13% wall on a kernel build, so it depends entirely on what you do. I left init_on_free=1 out - +8.38% wall and +24.42% sys on a kernel build, and Kees Cook’s own note is that catching a write between free and alloc is “pretty rare”. Kicksecure and secureblue run both; no mainstream distro does.

A few things you might expect here are already handled by the kernel config Guix ships, so there’s nothing to add: vsyscall=none (CONFIG_LEGACY_VSYSCALL_NONE), TIOCSTI compiled out, SLAB_FREELIST_RANDOM and _HARDENED, FORTIFY_SOURCE, HARDENED_USERCOPY, STRICT_DEVMEM. Check your own with zcat /proc/config.gz or the .config next to bzImage in the store, because the same file also tells you what’s not there: in the 7.2 config I looked at, CONFIG_SECURITY_LOCKDOWN_LSM and CONFIG_MODULE_SIG are both unset. So lockdown=integrity is silently ignored, and you’d need customize-linux and a local kernel build to get it.

A firewall

Guix’s example ruleset for iptables-service-type is called %iptables-accept-all-rules, and it does exactly that. Here’s a default-deny replacement. Open what you serve; I’ve left SSH in as the obvious example:

(use-modules (gnu services networking))

(define %firewall-ipv4
  (plain-file "iptables.rules" "*filter
:INPUT DROP [0:0]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [0:0]
-A INPUT -m conntrack --ctstate INVALID -j DROP
-A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -p tcp --dport 22 -j ACCEPT
COMMIT
"))

;; NDP and MLD are untracked by conntrack, so the ESTABLISHED,RELATED
;; accept never matches them. Without the ICMPv6 lines below, IPv6
;; simply never comes up - no neighbour ever resolves.
(define %firewall-ipv6
  (plain-file "ip6tables.rules" "*filter
:INPUT DROP [0:0]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [0:0]
-A INPUT -m conntrack --ctstate INVALID -j DROP
-A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -p ipv6-icmp -m icmp6 --icmpv6-type 1 -j ACCEPT
-A INPUT -p ipv6-icmp -m icmp6 --icmpv6-type 2 -j ACCEPT
-A INPUT -p ipv6-icmp -m icmp6 --icmpv6-type 3 -j ACCEPT
-A INPUT -p ipv6-icmp -m icmp6 --icmpv6-type 4 -j ACCEPT
-A INPUT -p ipv6-icmp -m icmp6 --icmpv6-type 130 -j ACCEPT
-A INPUT -p ipv6-icmp -m icmp6 --icmpv6-type 134 -j ACCEPT
-A INPUT -p ipv6-icmp -m icmp6 --icmpv6-type 135 -j ACCEPT
-A INPUT -p ipv6-icmp -m icmp6 --icmpv6-type 136 -j ACCEPT
-A INPUT -p tcp --dport 22 -j ACCEPT
COMMIT
"))

(service iptables-service-type
         (iptables-configuration
          (ipv4-rules %firewall-ipv4)
          (ipv6-rules %firewall-ipv6)))

The ICMPv6 types are the part of RFC 4890’s must-not-drop list a host actually receives: the four error types, MLD queries, router advertisements and neighbour solicitation/advertisement. Echo and redirect are left out on purpose (accept_redirects=0 above). I found the conntrack gap on a v6 network that looked configured and moved zero packets.

Fewer privileged programs

This is where Guix is ahead of other distros: the setuid list is a field in your config, not something you discover with find / -perm -4000. The default is thirteen entries: ping, ping6, passwd, chfn, sg, su, newgrp, newuidmap, newgidmap, sudo, sudoedit, fusermount, fusermount3. ping and ping6 are already capability-based rather than setuid, which is the right model.

I never use chfn, sg, newgrp or su (sudo covers it), and nothing mounts fuse2 here, so:

(use-modules (gnu system privilege)
             (gnu system setuid)
             (gnu packages admin)       ;; sudo, shadow, inetutils
             (gnu packages linux))      ;; fuse

(privileged-programs
 (cons* (privileged-program
         (program (file-append inetutils "/bin/ping"))
         (capabilities "cap_net_raw=ep"))
        (privileged-program
         (program (file-append inetutils "/bin/ping6"))
         (capabilities "cap_net_raw=ep"))
        (map file-like->setuid-program
             (list (file-append shadow "/bin/passwd")
                   (file-append shadow "/bin/newuidmap")
                   (file-append shadow "/bin/newgidmap")
                   (file-append sudo "/bin/sudo")
                   (file-append sudo "/bin/sudoedit")
                   (file-append fuse "/bin/fusermount3")))))

newuidmap and newgidmap stay if you run rootless Podman; it writes the subuid maps through them. Drop them too if you don’t.

%desktop-services adds more. mount.nfs and mount.ntfs-3g arrive setuid through a simple-service named mount-setuid-helpers, and because a simple-service wraps the target type in a fresh one, modify-services can’t delete it by type - filter by name instead:

(define %desktop-services-trimmed
  (filter (lambda (service)
            (not (eq? 'mount-setuid-helpers
                      (service-type-name (service-kind service)))))
          %desktop-services))

slock and xlock come setuid-root via screen-locker-service-type as well. If you’re on Wayland with swaylock, delete both and declare your own locker with (using-setuid? #f) and (using-pam? #t).

Four more you can’t remove: unix_chkpwd (PAM), dbus-daemon-launch-helper, and polkit’s polkit-agent-helper-1 and pkexec. The last one comes bundled with polkit’s extension, and polkit-service-type has no option to drop it.

Run guix-daemon unprivileged

Since 1.5, guix-daemon can run as its own user instead of root, with per-build UID isolation done through user namespaces instead of the guixbuilder01..10 pool:

(guix-service-type config =>
  (guix-configuration
   (inherit config)
   (privileged? #f)))

Every daemon-side root escalation of the last two years - CVE-2024-27297, CVE-2024-52867, the 2026 substitute bugs - lands in a plain user account instead. That’s a structural fix for a whole class of bugs, and it’s one line.

Three consequences, none of them obvious from the docs:

• The first reconfigure chowns the entire store. On my ~180 GB store that’s a blocking walk over ~90k items. Do it when you have time.
• Store files are now owned by guix-daemon, not root. Anything that insists on root-owned config - auditd, auditctl, pam_u2f’s authfile - will reject a file you deploy through etc-service-type, because that’s a symlink into the store. Copy such files out with an activation service and chown them (example below).
• Unprivileged user namespaces must stay on. No user.max_user_namespaces=0, no AppArmor userns restriction. That’s the one headline hardening move of the last few years (Ubuntu’s 24.04 default; ~44% of observed kernel exploit chains reportedly need userns) that’s off the table here, because the daemon uses exactly that mechanism for build isolation. I left a comment in the sysctl block so I don’t add it back by accident.

Delete services you don’t use

The cheapest attack-surface reduction is not running things. From %desktop-services, on a laptop:

(modify-services %desktop-services-trimmed
  ;; Hands browsers WiFi-derived street-level location via beacondb.net.
  (delete geoclue-service-type)
  ;; Announces the hostname on every LAN; nsswitch has no mdns anyway.
  (delete avahi-service-type)
  ;; Sends AT commands to any USB serial device; runs as root on insert.
  (delete modem-manager-service-type)
  (delete usb-modeswitch-service-type)
  ;; Replaced by chrony with NTS - authenticated time.
  (delete ntp-service-type))

NetworkManager depends on none of them. Check with herd status after a reboot what’s still running as root, and whether you need each one.

auditd

auditd-service-type is upstream. The value on a single-user machine is a log of who read your keys, and of every execve as root by a logged-in user (daemons carry auid -1, so they’re excluded):

-w /home/you/.ssh -p rwa -k sensitive-files
-w /home/you/.gnupg -p rwa -k sensitive-files
-w /home/you/.password-store -p rwa -k sensitive-files
-w /etc/sudoers -p wa -k privilege
-a always,exit -F arch=b64 -S execve -F euid=0 -F auid>=1000 -F auid!=-1 -k root-exec
-a always,exit -F arch=b32 -S execve -F euid=0 -F auid>=1000 -F auid!=-1 -k root-exec
-e 2

-e 2 locks the rules until reboot, so a root compromise can’t quietly drop a watch. It also means rule changes land at the next boot, not on reconfigure. Note -w <dir> is a recursive subtree watch; point it at secrets, not at a state directory with a 300 MB index in it.

Two traps, both caused by the unprivileged daemon above. auditd refuses a config file not owned by uid 0, so copy it out of the store:

(simple-service 'auditd-config activation-service-type
  #~(begin
      (mkdir-p "/etc/audit")
      (chown "/etc/audit" 0 0)
      (chmod "/etc/audit" #o700)
      (mkdir-p "/var/log/audit")
      (for-each (lambda (source target)
                  (false-if-exception (delete-file target))
                  (copy-file source target)
                  (chown target 0 0)
                  (chmod target #o600))
                (list #$%auditd-conf #$%audit-rules)
                (list "/etc/audit/auditd.conf" "/etc/audit/audit.rules"))))

(service auditd-service-type
         (auditd-configuration
          (configuration-directory "/etc/audit")))

And log into /var/log/audit/, not /var/log/audit.log directly: auditd chmods the log file’s parent directory to 0700 on every start and rotation. Pointed at /var/log it locks the whole directory, and the unprivileged daemon can no longer write its build logs under /var/log/guix.

The service doesn’t load rules itself; a one-shot Shepherd service running auditctl -R /etc/audit/audit.rules after auditd does. Once locked, auditctl -s reports enabled 2 and a reload is refused, so treat that as success or your reconfigure fails every time.

USBGuard

Not the biggest risk for most people. RHEL ships it disabled; it’s the one piece from there I wanted. The approach has two halves. The kernel argument usbcore.authorized_default=0 makes every USB device start unauthorized, closing the window between boot and the daemon coming up. USBGuard then authorizes devices matching a rule and leaves everything else blocked:

$ usbguard list-devices --blocked
$ usbguard allow-device -p <id>

Upstream Guix packages usbguard but has no service for it. I wrote one in my panther channel; if you’d rather not pull a channel for it, the service is small: a Shepherd service running usbguard-daemon with a config that sets ImplicitPolicyTarget=block, PresentDevicePolicy=apply-policy, AuthorizedDefault=none, and an IPC group so your user can run the CLI without sudo.

Two warnings. Check what your keyboard is. On my laptop it’s on i8042, so the LUKS prompt and greeter work with no USB at all. If yours is internal USB, authorized_default=0 locks you out at the disk prompt. And this guards against unknown hardware, not against a process already running as you - anyone in the IPC group can allow a device.

AppArmor, for one app at a time

Guix has no MAC service of any kind. The kernel side is there, though: linux-libre is built with AppArmor and Landlock, but its CONFIG_LSM string leaves Landlock out and puts SELinux, Smack and TOMOYO in (none of which you can use). So you need an explicit lsm= on the command line, an apparmor userspace, and something that mounts securityfs and runs apparmor_parser -r on your profiles at boot. Again, the service lives in panther; the same channel carries apparmor 5.0.2, because a profile that denies a unix socket by path needs network_v9 policy, which the 4.1.2 in Guix can’t emit.

"lsm=landlock,yama,apparmor"

The realistic use isn’t confining your whole session (Fedora doesn’t either; your desktop is unconfined_t there too). It’s containing the one app you trust least. Mine is an Electron chat client that I’d rather not have talking to NetworkManager and BlueZ for location-revealing scan results:

abi <abi/5.0>,
include <tunables/global>

profile chat /gnu/store/*-chat-desktop-*/{bin/chat,lib/chat/chat} {
  include <abstractions/base>

  userns,        # Electron's Chromium sandbox needs it
  file,
  capability,
  network,

  # The session bus at /run/user/*/bus stays open: tray, notifications
  # and portals keep working. Only the system bus is off limits.
  audit deny /run/dbus/system_bus_socket rw,
  audit deny /var/run/dbus/system_bus_socket rw,
  audit deny network netlink,
}

The store path changes on every upgrade, hence the glob. Two gotchas: deny rules are inert under flags=(complain), contrary to what aa-complain(8) suggests, so this has to run in enforce mode from day one. And without the abi <abi/5.0> line the socket deny stops open() but not connect(), so the rule looks like it works while the bus stays reachable. Denials show up in the same audit log as everything above: ausearch -m AVC.

To be clear about what this buys: MAC contains userspace lateral movement. It does nothing against a kernel exploit (Dirty COW ran straight through SELinux). It’s worth having; it’s not a guarantee.

Integrity, the Guix way

For the /usr equivalent you don’t need AIDE at all: guix gc --verify=contents rehashes the store against names the attacker can’t rewrite without also breaking every path. I still run AIDE daily, but only over /etc, /var and /boot - the parts that aren’t hash-addressed - via mcron-service-type, and Lynis weekly for the “did I regress something” check.

What I left out, and why

• Disabling unprivileged user namespaces. Breaks the unprivileged daemon. See above.
• lockdown=integrity and module signing. Not compiled into Guix’s kernel. Also, lockdown refuses hibernation the kernel didn’t encrypt itself, and I hibernate to a swapfile inside LUKS.
• Secure Boot. No signing path for Guix’s GRUB. With LUKS the kernel and initrd already live inside the encrypted root, so the exposed piece is grub.efi on the ESP, and that’s an evil-maid problem I accept.
• hardened_malloc. Kicksecure, whose ecosystem produced it, stopped enabling it by default: it breaks Firefox, Chromium and Electron, and cryptsetup runs ~6x slower. No published overhead number exists.
• nosmt, init_on_free. On a current AMD part every entry in /sys/devices/system/cpu/vulnerabilities reads “Not affected” or “Mitigation”. SMT off costs about 15% throughput, with nothing left to mitigate.
• noexec /tmp. /dev/shm is mounted exec, and one writable-plus-executable mount makes the other nearly moot. Also, guix-daemon builds under $TMPDIR, so you’d have to move that first.
• SELinux. Not in the kernel config, no service, and a labelling model that doesn’t fit an immutable store.

Where this lands

None of this is exotic; most of it is what Ubuntu or Arch already do out of the box, written by hand because Guix doesn’t ship it. The parts that are better than mainstream - a setuid list you can read, a build daemon that isn’t root, a store you can verify - come from how Guix is built. The parts that are worse - no MAC service, no firewall, two sysctls - are a few hundred lines in one file.

And the file is in git. The last time a change here broke something (that auditd chmod), I was back on the previous generation in under a minute.

There’s a small community keeping this device alive (Lineage-BBKey2, unofficial, actively maintained), and the recipe is well documented:

1. Unlock the bootloader with kibo, which exploits a fastboot buffer overflow
2. Reflash BlackBerry’s own signed firmware through their autoloader
3. Sideload LineageOS and GApps through recovery

None of that is new. The twist: I did it from a Guix machine, and Guix doesn’t play along with anything that expects a normal Linux filesystem.

Guix has no /lib64/ld-linux-x86-64.so.2

Google’s platform-tools and kibo’s prebuilt binary both expect /lib64/ld-linux-x86-64.so.2 and a handful of libraries at fixed paths. None of that exists on Guix. patchelf fixes it:

$ patchelf --set-interpreter $GLIBC/lib/ld-linux-x86-64.so.2 \
           --set-rpath "$GLIBC/lib:$LIBUSB/lib" kibo

Point the interpreter and rpath at the right store paths and the binary runs natively; no container, full USB access. Guix does package adb and fastboot, but they’re stuck at 7.1.2 from 2016. Google’s current 37.0.1 splits sparse images properly, which matters when you push a 3.2GB system.img against a 512MB per-transfer limit.

kibo’s device check was wrong

The unlock guide says to flash BlackBerry’s firmware twice (their A/B implementation is that broken). I did, and kibo still reported one of the two boot chains as INVALID and refused to unlock.

I wrote a small USB probe with pyusb to dump the raw response to the info query. Turns out kibo reads fixed-size buffers off the wire and never null-terminates them, so its strcmp() calls sometimes read past the real bytes into whatever is left on the stack. One buffer happened to sit next to a zero; the other didn’t.

kibo is GPL and three source files. Three lines to null-terminate each response at its actual length, then rebuild:

$ guix shell cmake make gcc-toolchain pkg-config libusb-next -- \
    cmake -B build && cmake --build build

Second run: Your device is vulnerable.

The result

The unlock went through, LineageOS 22.2 and MindTheGapps sideloaded without trouble, and the KEY2 booted Android 15 on a Snapdragon 660 from 2018. Gboard doesn’t reliably hide itself for the hardware keyboard on this build; K12KB, which the ROM ships for exactly that reason, does.

It’s a 2018 phone on an unofficial port. Dual SIM has a known bug (I only use one). But the keyboard clicks like new, and the phone that sat in a drawer for two years is back in my pocket. Good luck :)

Links

• Lineage-BBKey2 releases (KEY2) and Lineage-BBKey2LE releases (KEY2 LE)
• kibo - the bootloader unlock tool
• XDA: Unlock bootloader, KEY2 / KEY2 LE - firmware versions and step-by-step
• XDA: LineageOS 22.2 for the KEY2
• K12KB - hardware keyboard app

Talking it through with Claude, it turned out it can do the clicking itself. It needs three things: a display, a way to look at it, and a way to click.

Keep it off your desktop

The first try ran the app on my desktop and took a screenshot with grim. The Qt window had opened on another workspace, so the screenshot showed another application.

The setup

Everything runs as my user from a Guix shell; no root, nothing installed:

$ guix shell xorg-server -- Xvfb :99 -screen 0 1280x800x24 &
$ DISPLAY=:99 QT_QPA_PLATFORM=xcb ./build/app

Xvfb is an X server with no monitor behind it. Qt renders into it through the plain X11 backend, so the usual X tools work. From there it’s a loop of three commands:

$ guix shell imagemagick -- import -display :99 -window root shot.png   # look
$ guix shell xdotool -- xdotool mousemove 481 660 click 1               # click
$ guix shell zbar -- zbarimg -q --raw shot.png                          # read the QR

Claude reads the screenshot, picks the coordinates, clicks, waits, and takes the next screenshot. When the app draws a QR code, zbarimg decodes it from the screenshot. That’s the payload a phone would scan; a small script on the IdP side simulates the phone and approves the login. The next screenshot shows the status bar switch to “Logged-in”.

Longer flows run the same way. One took about a dozen rounds, including a picture from the real webcam (V4L2 works without a display).

Where it helps

A single happy-path login I can click through myself. What I don’t do by hand, every time, is the tedious part: wrong input, cancel halfway, an expired QR code, the same flow again after an error dialog. Claude runs through these as an adversarial review, with a screenshot for every step.

Limits

• No GPU. The app got past login, then crashed in its video layer. That still needs a real display.
• Coordinates drift. After an error dialog the window changed width, the button moved, and two clicks hit nothing. Every click needs a fresh screenshot.
• It’s slow. A few seconds of waiting plus a screenshot per round. Fine for a verification pass, not for every commit.
• Multiple windows. If the app opens more than one, xdotool search --name '<window title>' windowraise picks the right one, but you have to know the titles.

None of these tools are new; I’ve used Xvfb and xdotool before. The new part is a model in the loop that looks at the screenshot and decides where to click next.

The tedious testing is now something I can hand off, and the app is better for it.

$ uptime
 00:43:55  up 7 days,  8:58,  1 user,  load average: 4.54, 5.82, 8.75

Load of 8.75 over the last fifteen minutes, on twelve threads. That’s a guix build compiling a GTK app with -C lto -C codegen-units=1, a watchexec loop running cargo run on the backend I’m working on, and a second one rebuilding the frontend on every save. That’s what the machine looks like while I’m working.

So a cold release build of the Formshive backend (660 dependencies) gets whatever’s left over. It takes sixteen minutes, and slows down everything else while it runs, including the watch loops I need responsive.

Meet crunch

crunch is a drop-in cargo replacement: it rsyncs your project to a remote machine, runs the cargo command there, and copies the artifacts back. It’s packaged in Guix:

$ guix shell crunch -- crunch --exclude "target,.git" build --release

The remote needs a Debian-ish box with Rust on it, and an SSH host entry called crunch in your ~/.ssh/config. Set ControlMaster auto and ControlPersist 5m on that entry; without connection pooling, every rsync pays for a fresh SSH handshake.

crunch keeps the project in ~/crunch-builds on the remote. Your local target/ is excluded from the transfer, so the remote target/ is the only build cache.

Build times

I tried three sizes of Hetzner’s CCX line, the dedicated-vCPU plans with physical cores instead of a share. All three ran AMD EPYC Milan in Falkenstein. Same project, same commit, cold each time; I wiped ~/crunch-builds and the cargo registry between runs.

Going from two cores to four is 1.76x faster. Going from four to eight is only 1.29x; the critical path through the dependency graph and the serial link step don’t get faster with more cores. ccx43 costs twice as much as ccx33 and finishes 22% sooner, so ccx33 is my default.

The same build on my laptop took 964s, but that’s not a clean benchmark: a Guix source build was running at the same time. It’s the laptop as it is on a working evening, which is the number I care about, but it says nothing about the hardware.

Renting by the hour

Hetzner bills hourly, so I create a box when I need one and delete it when I’m done. The catch, straight from Hetzner’s docs: powering a server off does not stop billing. You pay until you delete it.

A small script, crunch-server, handles that:

$ crunch-server up      # create, provision, wire up ssh
$ crunch-server status  # what's running, and what it's cost so far
$ crunch-server down    # delete it (this is the part that stops billing)

up pulls the API token from pass, uploads my SSH key, creates the server with a cloud-init that installs the toolchain, waits for cloud-init to finish (not just for sshd to answer), and writes a Host crunch block with the pooling settings into ~/.ssh/config. If a box is already up, it reuses it; each fresh up starts a new billed hour.

Two things I got wrong the first time:

• Host keys. Every new box gets a recycled IP, so the main known_hosts throws a host-key warning every time. The script writes to a separate ~/.ssh/known_hosts.crunch and wipes it on teardown.
• Native dependencies. This project pulls pq-sys and openssl-sys, so libpq-dev and libssl-dev have to be in the cloud-init, or the build fails twelve minutes in.

Round trip

• Bring-up: 61 seconds, from nothing to a box that answers cargo --version
• Teardown: 18 seconds

I expected three or four minutes and was ready to build a base snapshot. At a minute, it’s not worth it.

Hetzner rounds usage up to the hour, though. Three short-lived boxes in an afternoon cost three hours; keep one and reuse it.

Caveats

• One project, one run per configuration. This shows the trend, not precise figures.
• The warm rebuild is a no-op; nothing changed between runs. It confirms the remote cache survives and that syncing 5 MB of source costs nothing, but it’s not the edit-compile loop. That’s the number I still need.
• The first ccx43 attempt failed with dedicated core limit exceeded. New projects have a cap on dedicated vCPUs; support raises it on request.
• crunch’s author recommends fewer fast cores over many slow ones, and runs a Hetzner AX102 (Ryzen 9 7950X3D). Milan EPYCs at around 2 GHz are the “many slow cores” option, and the weak scaling above four cores matches that.

The AX102 is next. For now the build runs somewhere else, my watch loops stay responsive, and the whole experiment cost about a euro.

A soft press lands as two clicks, or drops a selection halfway through a drag. Press it hard and it behaves. That’s a worn microswitch: the contact bounces open for a few tens of milliseconds right after it closes, and everything upstream reads the bounce as a real release.

libinput debounces buttons already, but only over a 12-25ms window it doesn’t expose. Mine bounces for longer, so the fix has to sit below libinput on the raw evdev stream.

Measure first

libinput’s own debouncing hides the fault, so read the evdev node directly. Being in the input group is enough; you don’t need root to look:

with open("/dev/input/event15", "rb") as device:
    seconds, microseconds, eventType, code, value = struct.unpack("llHHi", device.read(24))

A minute of normal clicking gave me this:

      1.228s  LEFT   press      1123.9 ms
      1.320s  LEFT   release      92.1 ms
      4.280s  LEFT   press      1256.0 ms
      4.340s  LEFT   release      60.0 ms   <-- bounce
      4.384s  LEFT   press        44.0 ms   <-- and back
      5.380s  LEFT   release     996.1 ms

That second click is one physical press: closed, opened again 60ms later, closed 44ms after that, then held for a second while I dragged. Depending on what’s listening, that’s two clicks or an interrupted drag.

What makes it fixable is that the bands don’t overlap:

A bounce always let go inside 60ms; a deliberate click always held for at least 92ms. That’s the whole filter.

Why not libinput

Its window is 12ms for a plain bounce, 25ms once a button has been flagged as bouncy, and the documentation is upfront that there’s no knob for it. Sensible default - a configurable debounce is a footgun aimed squarely at people who’d rather turn a dial than replace a switch.

The rule

A release arriving within --short-ms of its press is suspect, so the filter withholds it for --hold-ms. Press the button again inside that window and the release gets dropped along with the new press: as far as anything upstream is concerned, the button never came up. Otherwise the release goes out late.

Only short presses get withheld, so an ordinary 100ms click is never delayed. And since a dropped selection and a spurious double-click are the same fault - a phantom release - swallowing it fixes the drag case too.

The pipeline

Below libinput means evdev, and that means interception-tools. intercept grabs the device exclusively, the filter transforms the byte stream, uinput re-emits it on a clone that libinput picks up instead:

$ intercept -g /dev/input/event15 | mouse-debounce | uinput -d /dev/input/event15

Try that the first time on a laptop with a working touchpad. While the device is grabbed, the only pointer you have is the one coming out of the pipe.

It’s 255 lines of C: read, poll, write, and input_event out of <linux/input.h>. No allocation, no ownership graph, and input_event has time_t fields that C sizes straight off the header - 64-bit on x86-64, 32-bit on armv7.

Testing

The filter is timer-driven, so there’s nothing meaningful to unit test; the behaviour is “did the next press arrive within 70ms”. The suite spawns the real binary, writes real input_event structs down the pipe at real wall-clock spacing, and counts the clicks that come out:

case                                         clicks  want
recorded: bounce during a drag                    1     1
recorded: bounce turning one click into two       1     1
deliberate double-click, 80 ms gap                2     2
undebounced button is left alone                  2     2

The recorded cases are traces off my own mouse. Scheduling can stretch a 60ms bounce past the hold window, so each case gets three attempts before it counts as a failure. It runs in the Guix build container too, so the package won’t build if the timing behaviour has drifted.

Wiring it up

udevmon keeps the job on the device across reconnects. This one’s Bluetooth, so it gets a new event node every time it wakes up:

- JOB: "intercept -g $DEVNODE | mouse-debounce --hold-ms 90 --short-ms 80 | uinput -d $DEVNODE"
  DEVICE:
    NAME: "MX Anywhere 2S Mouse"
    EVENTS:
      EV_KEY: [BTN_LEFT]

NAME is a full-match regex, not a substring - udevmon uses std::regex with regex_match, defaulting to .*. The EVENTS clause is what pins the job to the right node: a Logitech mouse presents several under sibling names.

On Guix that’s a package and a service in my channel, so the config never gets typed into /etc:

(service px-mouse-debounce-service-type
         (px-mouse-debounce-configuration
          (device-name "MX Anywhere 2S Mouse")
          (hold-ms 90)
          (short-ms 80)))

The service generates the YAML above with store paths baked into the job line, so there’s no PATH to get wrong inside udevmon’s sh -c.

Thresholds

I started at --hold-ms 70, because the longest bounce gap I’d captured was 52ms. It still slipped, within an hour of real use: sixty seconds of deliberate clicking catches the typical bounce, not the tail. I’m on 90 now.

Raising it is cheap. Sweeping the suite:

case                                            50    70    90   110   want
recorded: bounce then a long hold                2     1     1     1   1
deliberate double-click, 80 ms gap               2     2     2     2   2

Nothing breaks at 90, or at 110. The 80ms double-click survives both because --short-ms gates the whole mechanism: that trace’s first click is 100ms long, so its release was never withheld in the first place. --hold-ms only bites when a double-click has both a short first click and a short gap. 50 is where it actually breaks and lets a 52ms bounce through.

Limits

Two genuine clicks, each shorter than --short-ms and less than --hold-ms apart, are the same waveform as a bounce, and the filter will merge them. With my numbers that’s two sub-80ms clicks less than 90ms apart. I can’t produce that deliberately; someone faster might.

And this is a workaround on a switch that’s mechanically failing. The bounce window widens as it wears - I’ve already moved once - and eventually it’ll overlap a real double-click, at which point no threshold separates them. The durable fix is a soldering iron and a new Omron - or a new mouse.

The code is at github.com/franzos/mouse-debounce if your switch is going the same way. Measure yours first.

Something had left the adapter running a permanent LE scan, and an active scan stops the host initiating LE connections - so the mouse and keyboard could advertise all they liked and nothing would ever answer. The headphones speak BR/EDR, classic Bluetooth, which pages the host instead. That path was never touched.

The first look

$ bluetoothctl show | grep -E 'Powered|Discovering'
	Powered: yes
	Discovering: yes

Discovering: yes, nothing connected, on a machine I hadn’t asked to scan anything. The device cache had about fifty entries with random addresses - 65:89:74:04:66:FA, 7D:BF:0F:DB:74:1C - which is what a long-running scan in a dense apartment block looks like.

bluetoothctl info on each of my three sorted them out:

The one device that kept working was the only one with a classic radio path. connect on either of the LE devices hung - no error, no refusal, twenty seconds and nothing came back.

Not a wedged controller

$ sudo herd restart bluetooth
$ bluetoothctl power off && bluetoothctl power on
$ bluetoothctl show | grep Discovering
	Discovering: yes

The scan was back within seconds, both times. That’s the useful result even though it fixed nothing: a stale session survives neither, and a power cycle clears adapter state outright. Something alive was re-arming the scan the moment the adapter came back.

Worth knowing on Guix: hciconfig, hcitool and btmgmt aren’t there to reach for. They’re deprecated upstream and Guix doesn’t compile them, so bluetoothctl power and rfkill are what you get.

Finding the client

BlueZ doesn’t expose which D-Bus client holds a discovery session, and dbus-monitor gets refused BecomeMonitor on this box. So work backwards: list every connection on the system bus, resolve each to a PID, and check whether that binary even knows the word StartDiscovery.

for n in $(dbus-send --system --dest=org.freedesktop.DBus --print-reply \
             /org/freedesktop/DBus org.freedesktop.DBus.ListNames |
           grep -oE '":[0-9.]+"' | tr -d '"'); do
  pid=$(dbus-send --system --dest=org.freedesktop.DBus --print-reply \
          /org/freedesktop/DBus org.freedesktop.DBus.GetConnectionUnixProcessID \
          string:"$n" | awk '/uint32/{print $2}')
  exe=$(readlink -f /proc/$pid/exe 2>/dev/null) || continue
  grep -aq StartDiscovery "$exe" 2>/dev/null && echo "$pid $(basename "$exe")"
done | sort -u

Two hits worth anything: chrome and slack, both Chromium underneath. Waybar came back clean, despite the Bluetooth module in my bar. One gotcha: bluetoothd matches too, since it’s the process providing StartDiscovery. Filter it out or you’ll chase your own daemon.

From there it’s a bisect. Quit Slack: still scanning. Quit Chrome:

$ bluetoothctl show | grep Discovering
	Discovering: no
$ bluetoothctl devices Connected
Device E2:7D:F3:EB:50:11 Air75 BT5.0
Device FD:B6:AB:99:E1:22 MX Anywhere 2S

Both devices back, immediately, without touching the adapter.

Starting Chrome again brought the scan straight back, but the mouse and keyboard stayed connected - which explains the timing. The scan doesn’t break a live link, it only blocks a new one from being established. It bites after the peripherals idle out on their own, at which point they’re stuck advertising into a host that can’t answer. Hence “broken overnight”.

Dead ends

A stored Web Bluetooth grant for banglejs.com - a watch I last touched six months ago - looked like the answer. Revoked it; Chrome carried on scanning. Extensions were the next guess, since three of mine bundle Web Bluetooth code for Ledger hardware wallets and a background service worker shows up in no tab and no manifest permission; --disable-extensions scanned too. No site, no extension, no open tab, which left Chrome itself.

The fix

Turn Web Bluetooth off at the policy layer. On Linux that’s one JSON file and no admin infrastructure:

/etc/opt/chrome/policies/managed/bluetooth.json
{ "DefaultWebBluetoothGuardSetting": 2 }

The path is easy to fluff - it’s /etc/opt/chrome/policies/managed, and I first created it without the chrome component, where nothing would ever read it. Root-owned and 0644 is what you want; the directory must not be writable by your own user, or anything running as you can set your browser policy for you.

Full quit, relaunch, and:

$ bluetoothctl show | grep Discovering
	Discovering: no

Chrome running, nothing scanning, all three devices connected. So it was Web Bluetooth after all, with nothing anywhere asking for it: Chrome opens the session on launch and holds it for as long as it runs. Why it does that when no site has requested Bluetooth, I still can’t tell you.

On Guix

The file is declared rather than typed into /etc by hand:

(simple-service 'chrome-web-bluetooth-policy etc-service-type
  (list `("opt/chrome/policies/managed/bluetooth.json"
          ,(plain-file "chrome-bluetooth-policy.json"
                       "{ \"DefaultWebBluetoothGuardSetting\": 2 }\n"))))

Two things to check before trusting that. etc-service-type symlinks top-level /etc entries, so /etc/opt becomes a link into the store - meaning the real /etc/opt I’d made by hand has to go first. activate-etc does rm-f then symlink, rm-f can’t delete a non-empty directory, and the symlink fails with EEXIST. Reconfigure dies, and the error doesn’t obviously point at the cause.

The second is whether Chrome accepts the file at all. guix-daemon runs unprivileged here, so store files are owned by the daemon rather than root, and that trap already bites two other things in my config: auditd refuses a config it doesn’t see as root-owned, and pam_u2f refuses a non-root authfile silently. Chrome doesn’t check. config_dir_policy_loader.cc enumerates the directory and deserializes whatever JSON it finds, with no ownership, permission or symlink check anywhere, and FileEnumerator uses stat() rather than lstat() unless you ask for SHOW_SYM_LINKS, so the store symlinks enumerate as plain files.

If this is you

1. bluetoothctl show | grep Discovering. If that says yes and you didn’t ask for it, you’ve found it.
2. Run the loop above to get your candidates.
3. Quit them one at a time. Don’t bother restarting bluetooth first, like I did - a live client just re-arms the scan.
4. If it’s Chrome, drop in the policy file. Quitting the browser every time your keyboard falls off is a workaround; the policy is the fix.
5. bluetoothctl trust <mac> on your input devices. Mine were untrusted, and with no Bluetooth agent running there’s nothing around to authorise an untrusted device’s reconnect. That wasn’t this bug, but it’s a second way to end up in the same place.

When one Bluetooth device works and another doesn’t, check what transport each is using before you start suspecting the radio. Mine was fine the entire time.

Tctl  61C  CRITICAL on CPU_IOWAIT (Min:8.9 Mean:30.5 Max:65.3): cargo, rustc, chrome

Chrome would stall mid-scroll. Bluetooth audio would drop out for a second and come back. Everything pointed at IO, and the processes named were the obvious suspects - cargo, rustc, rust-lld.

The short version: TRIM had never run on this machine, not once. The SSD believed all 1.8 TB were in use when 687 GiB were actually free, and it was paying for that on every single write.

It wasn’t memory

First thing worth ruling out, because “everything freezes” usually means swap:

$ free -h
               total        used        free      shared  buff/cache   available
Mem:            60Gi        10Gi       4.3Gi       972Mi        47Gi        49Gi

49 GiB available, and zram had 397 MB in a 24 GB device. Pressure stall info agreed - /proc/pressure/memory was flat at zero while /proc/pressure/io sat at avg300=10.53. So: not memory. Disk.

The tell

This is the measurement that made it obvious. /proc/diskstats tracks completed IOs and the milliseconds spent on them, so you can get an average latency per operation straight out of it:

$ awk '$3=="nvme0n1" {printf "%.2f ms/read  %.2f ms/write\n", $7/$4, $11/$8}' /proc/diskstats
0.18 ms/read  50.94 ms/write

Reads at 0.18 ms are perfectly healthy. Writes at 50 ms are not - that’s roughly what you’d expect from a mechanical drive, on an NVMe SSD that should be doing sub-millisecond.

That asymmetry is diagnostic. A dying drive is slow at everything. A drive that’s fast reading and terrible writing is a drive with no free blocks to write into. SMART confirmed it wasn’t wear:

Available Spare:    100%
Percentage Used:    4%
Data Units Written: 121,437,799 [62.1 TB]

4% used. The drive is fine. It just didn’t know where its free space was.

Why TRIM had never run

Two separate things had to be true, and both were.

First, TRIM has to pass through dm-crypt. On an encrypted root it doesn’t by default, and the option is on the mapped device, not the filesystem:

(mapped-devices
 (list (mapped-device
        (source (uuid "33d48354-afc2-428f-aa2a-0234984a04d8"))
        (target "cryptroot")
        (type luks-device-mapping)
        (arguments '(#:allow-discards? #t)))))

Worth knowing what you’re trading: allow-discards leaks a free/used block map to anyone holding the ciphertext. For a laptop-theft threat model that’s fine. For a drive you hand to someone else, think about it.

Second, and this is the one that actually bit me: I had a weekly mcron job to run fstrim, scheduled for Sunday at 04:00. My laptop is asleep at 04:00 on a Sunday. mcron has no catch-up for missed jobs - if the machine isn’t awake when the job is due, that run simply doesn’t happen. Ever. The job had been sitting in my config looking correct and had never once fired.

So the fix is two-part. Run it daily, so a miss costs a day rather than a week:

(define mcron-job-fstrim
  #~(job "0 4 * * *"
         #$(program-file "fstrim-daily"
             #~(execl #$(file-append util-linux "/sbin/fstrim")
                      "fstrim" "-v" "-m" "1M" "/"))))

And run one at boot as well, so a machine that’s always asleep at 4am still gets trimmed:

(define fstrim-at-boot-service
  (shepherd-service
   (provision '(fstrim-at-boot))
   (requirement '(file-systems))
   (respawn? #f)
   (documentation "Discard unused blocks on / shortly after boot.")
   (start #~(make-forkexec-constructor
             (list #$(program-file "fstrim-at-boot"
                       #~(begin
                           (sleep 300)
                           (execl #$(file-append util-linux "/sbin/fstrim")
                                  "fstrim" "-v" "-m" "1M" "/"))))))
   (stop #~(make-kill-destructor))))

The detail that matters there is what I didn’t write. Shepherd has a one-shot? flag, and using it here would have blocked boot until fstrim finished. Without it, make-forkexec-constructor forks and returns immediately, so boot isn’t delayed at all; the (sleep 300) then keeps the trim out of the login rush. respawn? #f stops shepherd restarting it when it exits.

Both get registered the usual way:

(simple-service 'fstrim-cron mcron-service-type
                (list mcron-job-fstrim))
(simple-service 'fstrim-at-boot shepherd-root-service-type
                (list fstrim-at-boot-service))

The result

The first run had eight months of debt to clear:

$ sudo fstrim -v /
/: 687.4 GiB (738038181888 bytes) trimmed

real    34m12.977s

34 minutes, and 840,000-odd discard commands. That’s a one-time cost - the drive now knows what’s free, and subsequent runs only have to deal with what’s been freed since.

Write latency, measured over a 15 second window before and after:

That middle column is the drive still chewing through the aftermath of a 34 minute discard storm. Left alone for a few hours it settled to 0.11 ms, which is where it should have been all along - call it 500x. The IO stalls stopped and Chrome went back to scrolling like a normal application.

A 43ms stall for 73 milliwatts

While I was in there I checked the drive’s power states, since “freezes after a moment of idle” has more than one cause:

$ sudo nvme id-ctrl /dev/nvme0 | grep -E '^ps '
ps 3 : mp:0.0800W non-operational enlat:1600 exlat:4600
ps 4 : mp:0.0070W non-operational enlat:1600 exlat:43000

Those exit latencies are microseconds. State 4 saves 73 milliwatts over state 3 and costs 43 ms to wake up from. On a laptop pulling several watts at idle, that’s not a trade I want. The kernel gates APST on exit latency alone, so a ceiling between the two blocks the deep state and keeps the shallow one:

"nvme_core.default_ps_max_latency_us=5000"

I’ll note that 43 ms sits right inside the range of write latencies I was measuring, so some of what I blamed on TRIM may have been this. I can’t separate them cleanly after the fact.

About that -m 1M

The -m 1M in those jobs tells fstrim to skip free extents smaller than 1 MiB, so a routine trim doesn’t turn into its own IO stall. I nearly talked myself out of it - the average extent in that first run was 845 KiB, below the floor, which looked like the daily job would skip most of what it should be trimming and quietly rebuild the backlog.

Measuring it says otherwise. Running both back to back, later the same evening:

The floor is about 6x faster per gigabyte, because the sub-1MiB extents are precisely where the cost is. Even a heavy day with 130 GiB to clear finishes in under a minute with the floor on. So it stays, and the boot job costs 8 seconds of background IO rather than anything you’d notice.

Granted, that’s not a clean A/B - the unfloored run went first, so it isn’t a fair measure of how much the floor skips. It’s a fair measure of what the floor costs, which was the thing I actually needed to decide.

What I still don’t know

The 500x is entirely down to TRIM. The APST and noatime changes need a reconfigure and a reboot, and hadn’t taken effect when I measured any of this. Whatever they’re worth is still to come.

If you’re running an encrypted root on Guix, the two-line version is: check that #:allow-discards? is actually set, then check that your trim job has ever run. Mine looked right in the config for months and had never executed once.

$ awk '$3=="nvme0n1" {printf "%.2f ms/read  %.2f ms/write\n", $7/$4, $11/$8}' /proc/diskstats

If the second number is much bigger than the first, you have the same problem I did.

Before I get to what we actually built, some ground rules, because they’re the reason the build looked the way it did.

It makes mistakes, and that’s not a bug you can file

An LLM is a non-deterministic process. So is OCR, and so is a human reading a scanned invoice. Nobody deploys OCR at scale without a correction step; everybody knows it’ll misread a 6 as an 8 often enough to matter. LLMs get deployed as if that step were optional.

It’s also a moving target. A new model behaves differently and brings its own quirks. Latency is fine on Tuesday and terrible on Thursday. Requests fail. Sometimes a model seems to shift in character between releases and you’re left re-tuning prompts that worked fine for months.

You can shape the thing without training it. If you need a specific voice, or you want the output constrained to a shape, you can get most of the way there with instructions, examples and validation. Treat all of it as best-effort. It’s a strong tendency, not a hard truth, and anything that depends on it being a hard truth needs a check behind it.

If you need better output, one option is to involve more than one model: one drafts, another reviews against the guidelines, and you look at what survives. That’s not a trick, it’s the same thing you’d do with people when the output actually matters. It helps. It doesn’t get you to zero.

And this part I’ll say flat: today, a professional with experience in their field, a reputation at stake, and some care about what leaves the building will beat the model on quality. Not on speed - it’ll lose that badly. On quality. That’s true as of today; I don’t know what tomorrow looks like.

None of this is an anti-AI position. Humans have the same failure modes; I wrote recently about a 1,534-line commit called “runs” that shipped a hardware wallet generating seeds from a non-cryptographic PRNG, and no model was anywhere near it. The difference is that a person can be told, and a person minds.

The calculator problem

Here’s the failure I actually worry about in a company: the output looks good, so people trust it.

My calculator tells me 1+1=2 and I trust it, correctly, forever. The LLM arrives on the same screen, on the same machine, in the same tab strip, and produces fluent, confident, well-formatted text. Everything about the packaging says “computer”, and everybody has thirty years of training that computers are right about this kind of thing. They’re not the same category of thing at all, and nothing in the interface tells you that.

Which makes deployment a human matter before it’s a technical one. Staff need training, and they need reminders, because the trust creeps back in. Don’t treat this like a calculator; treat it like something that makes mistakes.

The odds get worse in the directions you’d expect: the longer the output, and the tighter your expectations of it, the more likely you are to get something wrong. A one-line classification is fairly safe. Three pages that have to be right in every particular is a different proposition.

Outward-facing is where it gets expensive

That’s for internal processes. Point it at customers and the stakes change.

Think about how a call center actually works. People get trained before they take a call. They’re listened in on. They’re coached, corrected, and scored. The script is tight, and someone owns the script. Nobody hires an agent and then never checks on them again.

An LLM doing the same job gets the same treatment, minus every incentive the human had. Its job isn’t on the line. It has no reputation to protect, no interest in a promotion, no bad feeling about a call that went badly. It is, as it comes out. Set it and forget it isn’t a thing here; it isn’t a thing for the humans either, and we’ve known that for decades.

Somebody has to own it

So the question every AI project has to answer early: who watches this?

It’s either someone external, on a maintenance and QC arrangement, or someone internal you designate and give the time to actually do it. There’s no third option where nobody does it and it stays good.

What doesn’t work is the fifty-a-month subscription you sign up for and forget. That only holds if you genuinely don’t care about the outcome, and if you don’t care about the outcome, it’s not clear why you started. You want a measurable result. Does it bring in customers? Does support get better, or just faster? Does the process finish earlier? What’s the error rate, and is it going up or down? If you can’t answer those, you don’t have a project, you have a subscription.

What we actually did

With all that said, the build itself was boring, straight-forward stuff. Roughly:

1. Found the processes worth automating. Not all of them; the ones where more throughput, with oversight, genuinely helps. Time-consuming, repetitive, and cheap to check.
2. Wrote guidelines. How the system is to be used, and what to look out for. Think of the checklist a flight crew runs before departure: they’ve done it a thousand times and they still read it out.
3. Trained the assigned staff. The people who’d own it day to day, on their own workflows, not a generic demo.
4. Implemented the guidelines. Writing them down isn’t implementing them. They had to land in the actual process, with the actual checks in the actual places.

Using it is about as hard as using Gmail. The staff picked it up in ten minutes. The interface was never the hard part, and if your AI project’s difficulty is concentrated in the UI, something has gone sideways.

Why nobody pushed back

There was very little resistance, and I don’t think that’s down to change management. It made previously boring, time-consuming work easier, and it didn’t replace anyone. What it means in practice is that by Friday, the work is actually done.

That’s the whole pitch. It isn’t headcount, it’s the backlog that never quite cleared.

What happens next

I’ll check in weekly for a while, then every couple of weeks: collect feedback, look at what’s going wrong, fine-tune the guidelines. At some point it stops being the new thing and becomes how it’s always been done, which is the goal. If it keeps going well, we’ll pick the next process.

That last part is the bit I can’t shortcut and won’t pretend otherwise. AI is fast and it works at any hour, and that means there’s a lot more room for things to go wrong, quietly, at volume.

The short version: DCR made the authorization server hold state for every client that ever showed up. CIMD flips it - the client hosts a JSON file, the URL of that file is the client ID, and the AS fetches it on demand. Which is better, and also how you get a server-side request forgery hole in your authorization server.

Why DCR is going away

DCR (RFC 7591) arrived with MCP’s first authorization spec in 2025-03-26, which said implementations SHOULD support it and, further down, “strongly recommend” it. For eight months it was the only zero-touch option: an MCP client that’s never met your server POSTs its metadata to /register, gets back a client_id, and runs the normal authorization code flow.

CIMD arrived in 2025-11-25 (SEP-991), and the same revision dropped DCR to MAY, “included for backwards compatibility with earlier versions of the MCP authorization spec”. 2026-07-28 finished the job: the formal Deprecated.

The reasons were laid out well before that, in an August 2025 writeup on client registration by Paul Carleton - an exploratory post, not the announcement, but it’s where the problems are stated plainly:

• “Unbounded database growth: Every time a user connects a client to an MCP server, a new registration is created with the authorization server unless the client already has one.” Registrations are per-install, so the same app on your laptop and your desktop is two rows.
• “There’s no way to tell a client that its ID is invalid without creating an open redirect vulnerability.” So registrations effectively never expire.
• “An unauthenticated /register endpoint writes to a database within the authorization server, meaning that tenant admins now need to worry about rate limiting or policy controls.” That’s the part that kept me writing rate limiters.

The deprecation itself is in the 2026-07-28 changelog, and the same revision adopted a formal feature lifecycle policy with a twelve-month minimum deprecation window - shortenable to no less than ninety days, but only for a vulnerability with a published advisory or documented exploitation and no in-place mitigation. So DCR keeps working for at least a year, for backwards compatibility with authorization servers that don’t do CIMD yet. It just isn’t the answer anymore.

The two side by side

The document itself is simple:

{
  "client_id": "https://app.example.com/oauth/client-metadata.json",
  "client_name": "Example MCP Client",
  "redirect_uris": ["http://127.0.0.1:3000/callback"],
  "grant_types": ["authorization_code"],
  "response_types": ["code"],
  "token_endpoint_auth_method": "none"
}

client_id MUST equal the URL it was fetched from, exactly. Worth being precise about what that buys you: it stops me hosting a copy of your document at my URL. It does nothing to stop me handing the authorization server your URL and claiming to be you - that attack is alive and well, and I’ll come back to it.

The part that bites: SSRF

Here’s what actually changed for the server operator. Under DCR, an attacker sends you data. Under CIMD, an attacker sends you a URL, and your authorization server - which sits inside your network, holding your signing keys - goes and fetches it. That’s textbook SSRF, and the draft says so in section 6.5: “Authorization servers fetching the client metadata document and resolving URLs located in the metadata document should be aware of possible SSRF attacks.” It follows up with the actual instruction - servers SHOULD “avoid fetching any URLs using private or loopback addresses and consider network policies or other measures to prevent making requests to these addresses”, and SHOULD watch for non-HTTP URI schemes. (Section numbers are for -00, the revision MCP normatively cites; in the current -02 the security considerations moved to section 8.)

MCP’s own text wraps that in a MUST-consider (“authorization servers MUST consider the security implications detailed in [Section 6]”) while the SSRF bullet itself stays a SHOULD. So the obligation to think about it is hard; the specific defences are yours to pick. This is what Forseti’s fetcher does, and it’s roughly the list I’d want from anything else:

• HTTPS only, with the URL capped at 512 bytes. Plain http is allowed only when you explicitly flip allow_private_targets for local development. (The spec also requires the client_id URL to carry a path component; Forseti doesn’t enforce that one yet.)
• Reject loopback, private, and special-use IPs before the request goes out - and again at connect time through a guarded DNS resolver, because a DNS answer that’s public on the first lookup and 169.254.169.254 on the second is the whole rebinding trick.
• Don’t follow redirects at all. Forseti sets the redirect policy to none. Following one means re-running every check on the new target; not following is simpler and I haven’t missed it.
• Cap the body and the clock. 64 KiB and a 5s timeout. The draft suggests 5 KB; I went wider because logo and scope fields add up, and 64 KiB is still far below anything that hurts.
• Cache, with bounds. Respect HTTP cache headers but clamp the TTL to 60s..24h, single-flight concurrent fetches for the same URL, and serve a last-good document for up to an hour if the fetch fails. Never cache errors or malformed documents.
• Don’t let the client name itself. client_name is whatever the client wrote about itself, so on Forseti’s consent page the host of the client_id URL is the primary identity and the self-asserted name sits underneath it.

There’s an optional host allowlist ([oauth.cimd].allowed_client_hosts) for anyone who wants a closed set. Empty means open, gated by consent, which is the setting that makes MCP work the way it’s supposed to.

The failure mode of over-blocking is real too: Backstage hit it when a Kubernetes deployment’s own hostname resolved to a 10.x address and the private-IP check rejected a self-referential fetch. Granted, that’s an argument for a deliberate escape hatch, not for loosening the default.

The impersonation hole that’s left

That consent-screen design covers one attack and misses the more interesting one. Showing the client_id host is the right answer to a client lying in its client_name. It does nothing about the attack the spec spells out:

Client ID Metadata Documents cannot prevent localhost URL impersonation by themselves. An attacker can claim to be any client by: 1. Providing the legitimate client’s metadata URL as their client_id 2. Binding to any localhost port, and providing that address as the redirect_uri 3. Receiving the authorization code via the redirect when the user approves

Read that carefully. The client_id host in that attack is the legitimate one - it’s the real client’s document, fetched from the real client’s domain, and it passes every check in the list above. As the spec puts it: “The server will see the legitimate client’s metadata document and the user will see the legitimate client’s name, making attack detection difficult.” The only field that differs is the redirect URI, which is why the spec’s requirement there is a MUST: authorization servers “MUST clearly display the redirect URI hostname during authorization.”

Where my things stand

Forseti is the authorization-server side, so it took the whole change. Since 0.2.0, MCP clients connect with a URL client ID and no registration step; DCR and initial access tokens are removed rather than deprecated in place. On a first authorization request Forseti fetches and validates the document, then upserts a Hydra client row keyed to that URL, so Hydra keeps doing the OAuth work while the identity stays the client’s own URL. The scope set on that row is a ceiling, never a grant - consent is still the authorization.

Stackpit needed nothing, and that’s the more interesting half. It’s a resource server: its MCP endpoint is bearer-only, it publishes /.well-known/oauth-protected-resource/mcp per RFC 9728, and it validates audience, nbf, and scopes on the token. How the client got its client_id is entirely the authorization server’s problem. That separation is the quiet win of the RFC 9728 split - a registration mechanism can be deprecated out from under you and your resource servers don’t notice.

Should you move

If you run an authorization server for MCP clients: yes, but you’ve got a year, and the caveat is that not every client is there yet. Some still hard-require a registration_endpoint and fall over without one - Codex has an open issue exactly on that, and Claude Code has one for a related case where a pre-configured clientId still triggers a DCR attempt.

I dropped DCR from Forseti anyway because support is catching up fast.

If you run an MCP server that’s only a resource server, this isn’t your problem at all.

And note what you’re building on: an active Internet-Draft, currently at -02 (July 2026), while MCP still normatively cites -00. Section numbers have already moved between them, so check which one your implementation is reading. Nothing in CIMD attests that the name and logo in a document belong to whoever wrote them either - the draft has suggestions (prefetch the logo so you can moderate it, weigh domain age and reputation, warn on new domains), but they’re all things you build, not things the protocol gives you. It’s a better shape than DCR, not a solved problem.

Somebody wanted to swap MicroPython’s random number generator for their own. The build came back with duplicate symbol errors. Instead of untangling the collision, they turned the other one off:

#define MICROPY_HW_ENABLE_RNG (0)  // We have our own version of this code.

The error went away. So did the hardware RNG. The replacement was never actually wired up, so random.bytes(32) fell through to Yasmarang - a tiny, fast, thoroughly non-cryptographic PRNG that MicroPython keeps around for when you don’t have hardware entropy. That’s what generated seeds. Dusty Daemon’s write-up walks the whole call chain, and it’s worth reading in full.

What stuck with me wasn’t the bug. It was the commit: 1,534 lines changed, message runs. Five characters. There’s another one in there whose entire message is x.

Nobody’s AI did this

Worth saying plainly, because it would be easy to file this under vibe coding: no model wrote that line. A person did, years before any of us had an agent sitting in a terminal. Mistakes come from being rushed, from being sloppy, or from not knowing any better, and humans have had all three well covered for decades.

Which is exactly why I keep pushing back when AI gets sold as a different category of thing. It isn’t. It’s fallible in the same ways we are, just faster and with better grammar. It has no passion and no knowledge; it has statistics. And the comment in that line - “We have our own version of this code” - is the most human artifact in the whole story. Someone believed it. It wasn’t true. Nobody checked.

The bottleneck was never typing

AI has genuinely sped up how fast code appears. It has not moved the thing that actually protects you, which is somebody competent reading the code and understanding what it does on the real hardware.

A 1,534-line diff is a 1,534-line diff whether a model produced it in ninety seconds or a person produced it over a weekend. If nobody reads it, it’s a liability either way. The difference now is that the ninety-second version arrives more often, and review capacity didn’t grow to match.

So the rule I work by is boring: scale review to the stakes, not to the volume. A marketing site can ship on vibes. Something holding other people’s money or other people’s data gets a second set of eyes that actually knows the domain - not another model summarizing the diff for you, because the model shares the failure mode. It’ll tell you the change looks reasonable. It looked reasonable to the author too.

Go fast where being wrong is cheap. Slow down exactly where it isn’t. That’s most of the discipline, and it’s the same advice whether the code came from a person or a machine - which is sort of the point.

I wrote a while back that AI is the race car and I’m the driver. This is the other half of that: the car doesn’t decide which races are worth losing.