687 GiB My SSD Didn't Know Were Free
My laptop was freezing under Rust builds. The SSD was doing 50ms writes because TRIM had never actually run - here's the diagnosis and the Guix config that fixes it.
My monitoring started shouting at me while I was building a Rust project:
Tctl 61C CRITICAL on CPU_IOWAIT (Min:8.9 Mean:30.5 Max:65.3): cargo, rustc, chrome
Chrome would stall mid-scroll. Bluetooth audio would drop out for a second and come back. Everything pointed at IO, and the processes named were the obvious suspects - cargo, rustc, rust-lld.
The short version: TRIM had never run on this machine, not once. The SSD believed all 1.8 TB were in use when 687 GiB were actually free, and it was paying for that on every single write.
It wasn’t memory
First thing worth ruling out, because “everything freezes” usually means swap:
$ free -h
total used free shared buff/cache available
Mem: 60Gi 10Gi 4.3Gi 972Mi 47Gi 49Gi
49 GiB available, and zram had 397 MB in a 24 GB device. Pressure stall info agreed - /proc/pressure/memory was flat at zero while /proc/pressure/io sat at avg300=10.53. So: not memory. Disk.
The tell
This is the measurement that made it obvious. /proc/diskstats tracks completed IOs and the milliseconds spent on them, so you can get an average latency per operation straight out of it:
$ awk '$3=="nvme0n1" {printf "%.2f ms/read %.2f ms/write\n", $7/$4, $11/$8}' /proc/diskstats
0.18 ms/read 50.94 ms/write
Reads at 0.18 ms are perfectly healthy. Writes at 50 ms are not - that’s roughly what you’d expect from a mechanical drive, on an NVMe SSD that should be doing sub-millisecond.
That asymmetry is diagnostic. A dying drive is slow at everything. A drive that’s fast reading and terrible writing is a drive with no free blocks to write into. SMART confirmed it wasn’t wear:
Available Spare: 100%
Percentage Used: 4%
Data Units Written: 121,437,799 [62.1 TB]
4% used. The drive is fine. It just didn’t know where its free space was.
Why TRIM had never run
Two separate things had to be true, and both were.
First, TRIM has to pass through dm-crypt. On an encrypted root it doesn’t by default, and the option is on the mapped device, not the filesystem:
(mapped-devices
(list (mapped-device
(source (uuid "33d48354-afc2-428f-aa2a-0234984a04d8"))
(target "cryptroot")
(type luks-device-mapping)
(arguments '(#:allow-discards? #t)))))
Worth knowing what you’re trading: allow-discards leaks a free/used block map to anyone holding the ciphertext. For a laptop-theft threat model that’s fine. For a drive you hand to someone else, think about it.
Second, and this is the one that actually bit me: I had a weekly mcron job to run fstrim, scheduled for Sunday at 04:00. My laptop is asleep at 04:00 on a Sunday. mcron has no catch-up for missed jobs - if the machine isn’t awake when the job is due, that run simply doesn’t happen. Ever. The job had been sitting in my config looking correct and had never once fired.
So the fix is two-part. Run it daily, so a miss costs a day rather than a week:
(define mcron-job-fstrim
#~(job "0 4 * * *"
#$(program-file "fstrim-daily"
#~(execl #$(file-append util-linux "/sbin/fstrim")
"fstrim" "-v" "-m" "1M" "/"))))
And run one at boot as well, so a machine that’s always asleep at 4am still gets trimmed:
(define fstrim-at-boot-service
(shepherd-service
(provision '(fstrim-at-boot))
(requirement '(file-systems))
(respawn? #f)
(documentation "Discard unused blocks on / shortly after boot.")
(start #~(make-forkexec-constructor
(list #$(program-file "fstrim-at-boot"
#~(begin
(sleep 300)
(execl #$(file-append util-linux "/sbin/fstrim")
"fstrim" "-v" "-m" "1M" "/"))))))
(stop #~(make-kill-destructor))))
The detail that matters there is what I didn’t write. Shepherd has a one-shot? flag, and using it here would have blocked boot until fstrim finished. Without it, make-forkexec-constructor forks and returns immediately, so boot isn’t delayed at all; the (sleep 300) then keeps the trim out of the login rush. respawn? #f stops shepherd restarting it when it exits.
Both get registered the usual way:
(simple-service 'fstrim-cron mcron-service-type
(list mcron-job-fstrim))
(simple-service 'fstrim-at-boot shepherd-root-service-type
(list fstrim-at-boot-service))
The result
The first run had eight months of debt to clear:
$ sudo fstrim -v /
/: 687.4 GiB (738038181888 bytes) trimmed
real 34m12.977s
34 minutes, and 840,000-odd discard commands. That’s a one-time cost - the drive now knows what’s free, and subsequent runs only have to deal with what’s been freed since.
Write latency, measured over a 15 second window before and after:
| before | right after the trim | settled, a few hours later | |
|---|---|---|---|
| average write | 57.15 ms | 1.46 ms | 0.11 ms |
That middle column is the drive still chewing through the aftermath of a 34 minute discard storm. Left alone for a few hours it settled to 0.11 ms, which is where it should have been all along - call it 500x. The IO stalls stopped and Chrome went back to scrolling like a normal application.
A 43ms stall for 73 milliwatts
While I was in there I checked the drive’s power states, since “freezes after a moment of idle” has more than one cause:
$ sudo nvme id-ctrl /dev/nvme0 | grep -E '^ps '
ps 3 : mp:0.0800W non-operational enlat:1600 exlat:4600
ps 4 : mp:0.0070W non-operational enlat:1600 exlat:43000
Those exit latencies are microseconds. State 4 saves 73 milliwatts over state 3 and costs 43 ms to wake up from. On a laptop pulling several watts at idle, that’s not a trade I want. The kernel gates APST on exit latency alone, so a ceiling between the two blocks the deep state and keeps the shallow one:
"nvme_core.default_ps_max_latency_us=5000"
I’ll note that 43 ms sits right inside the range of write latencies I was measuring, so some of what I blamed on TRIM may have been this. I can’t separate them cleanly after the fact.
About that -m 1M
The -m 1M in those jobs tells fstrim to skip free extents smaller than 1 MiB, so a routine trim doesn’t turn into its own IO stall. I nearly talked myself out of it - the average extent in that first run was 845 KiB, below the floor, which looked like the daily job would skip most of what it should be trimming and quietly rebuild the backlog.
Measuring it says otherwise. Running both back to back, later the same evening:
| run | trimmed | time | rate |
|---|---|---|---|
| no floor | 131.5 GiB | 5m37s | 0.39 GiB/s |
-m 1M |
20.4 GiB | 8.5s | 2.4 GiB/s |
The floor is about 6x faster per gigabyte, because the sub-1MiB extents are precisely where the cost is. Even a heavy day with 130 GiB to clear finishes in under a minute with the floor on. So it stays, and the boot job costs 8 seconds of background IO rather than anything you’d notice.
Granted, that’s not a clean A/B - the unfloored run went first, so it isn’t a fair measure of how much the floor skips. It’s a fair measure of what the floor costs, which was the thing I actually needed to decide.
What I still don’t know
The 500x is entirely down to TRIM. The APST and noatime changes need a reconfigure and a reboot, and hadn’t taken effect when I measured any of this. Whatever they’re worth is still to come.
If you’re running an encrypted root on Guix, the two-line version is: check that #:allow-discards? is actually set, then check that your trim job has ever run. Mine looked right in the config for months and had never executed once.
$ awk '$3=="nvme0n1" {printf "%.2f ms/read %.2f ms/write\n", $7/$4, $11/$8}' /proc/diskstats
If the second number is much bigger than the first, you have the same problem I did.