# Testing GUI Apps With Claude (Claude Showed Me the Way)

Source: https://gofranz.com/blog/testing-gui-apps-with-claude/

I had Claude Code working through a long plan today: bring up a local stack of our services with one command, then prove that two native Qt apps (a KYC terminal and a nurse workstation) can log in against it. The last part has "done means verified" written into the spec, and I meant it; I'm tired of "should work" for anything with a login screen.

The catch: these are GUI apps. Login means pressing a button, a QR code shows up, a phone scans it, an approval comes back, and only then do you know. There's no headless mode. I expected Claude to stop there and hand the clicking back to me.

It didn't. Here's what it did instead.

### The first attempt went wrong in a useful way

It launched the app on my desktop and took a screenshot with `grim` to see what was going on. The screenshot was my Slack window; The Qt window had opened on another workspace. Claude noticed the mistake, said it wouldn't touch my session again, closed the app, and moved the whole thing to a display nobody looks at.

I liked that. Not because the screenshot mattered (it was a devops channel), but because it corrected course without me saying anything.

### The setup

Everything below runs as my user, from a Guix shell, with no root and nothing installed:

```sh
$ guix shell xorg-server -- Xvfb :99 -screen 0 1280x800x24 &
$ DISPLAY=:99 QT_QPA_PLATFORM=xcb ./build-dev/gui/kyc-gui -d -t
```

`Xvfb` is a framebuffer with no monitor behind it. Qt renders into it through the plain X11 backend, so all the old X tooling works. From here it's a loop of three commands:

```sh
$ guix shell imagemagick -- import -display :99 -window root shot.png   # look
$ guix shell xdotool -- xdotool mousemove 481 660 click 1               # click
$ guix shell zbar -- zbarimg -q --raw shot.png                          # read the QR
```

Claude reads the PNG with its image input, picks the coordinates off it, clicks, waits, and takes the next screenshot. When the app draws a QR code, `zbarimg` decodes it straight from the screenshot. That payload is what a phone would scan; a small script on the IdP side plays the phone and approves the login, and the next screenshot shows the status bar switch to "Logged-in".

It did the whole KYC enrollment that way: card read (faked), take the picture (real webcam, which works fine because V4L2 doesn't care about displays), create the account, decode the QR meant for the enrollee's phone, simulate that phone, and check on the IdP that a verified user came out the other end. About a dozen click-wait-look rounds.

### What it's not

- **No GPU.** The workstation got through login and started its QML main screen, then crashed in the video layer. That one still needs a real display.
- **Coordinates drift.** After an error dialog the KYC window changed width, the button moved, and two clicks landed on nothing. It only noticed by looking again. Every click needs a fresh screenshot; there's no shortcut.
- **It's slow.** Each round is a few seconds of sleeping plus a screenshot. Fine for a verification pass, not something you'd run on every commit.
- **Windows stack.** The workstation opens three fullscreen windows; `xdotool search --name 'VHH Workstation: Login' windowraise` picks the right one, but you have to know the titles.

### Why I'm writing this down

None of these tools are new. `Xvfb` and `xdotool` have been around forever, and I've used both. What I hadn't done is put a model in the loop that looks at the screenshot and decides where to click next, and then let it grind through a multi-step flow with a QR handoff in the middle, without me. It found the recipe itself; I only told it to keep its hands off my desktop.

Take this with a grain of salt: it worked on two apps, on one machine, on one afternoon. But it turned "I'll click through it later" into something that actually got checked. Good luck :)