Tools · claude

Testing GUI Apps With Claude

Letting Claude drive Qt apps on a virtual X server: screenshots, xdotool clicks and a QR reader.

By Franz Geffke 10 Sep 2026 3 min read #claude #qt #xvfb #testing

I’ve been working on a Qt app that logs in against an identity provider. Login means pressing a button, a QR code shows up, a phone scans it, and an approval comes back. There’s no headless mode, so every check ended with me clicking through it by hand.

Talking it through with Claude, it turned out it can do the clicking itself. It needs three things: a display, a way to look at it, and a way to click.

Keep it off your desktop

The first try ran the app on my desktop and took a screenshot with grim. The Qt window had opened on another workspace, so the screenshot showed another application.

The setup

Everything runs as my user from a Guix shell; no root, nothing installed:

$ guix shell xorg-server -- Xvfb :99 -screen 0 1280x800x24 &
$ DISPLAY=:99 QT_QPA_PLATFORM=xcb ./build/app

Xvfb is an X server with no monitor behind it. Qt renders into it through the plain X11 backend, so the usual X tools work. From there it’s a loop of three commands:

$ guix shell imagemagick -- import -display :99 -window root shot.png   # look
$ guix shell xdotool -- xdotool mousemove 481 660 click 1               # click
$ guix shell zbar -- zbarimg -q --raw shot.png                          # read the QR

Claude reads the screenshot, picks the coordinates, clicks, waits, and takes the next screenshot. When the app draws a QR code, zbarimg decodes it from the screenshot. That’s the payload a phone would scan; a small script on the IdP side simulates the phone and approves the login. The next screenshot shows the status bar switch to “Logged-in”.

Longer flows run the same way. One took about a dozen rounds, including a picture from the real webcam (V4L2 works without a display).

Where it helps

A single happy-path login I can click through myself. What I don’t do by hand, every time, is the tedious part: wrong input, cancel halfway, an expired QR code, the same flow again after an error dialog. Claude runs through these as an adversarial review, with a screenshot for every step.

Limits

• No GPU. The app got past login, then crashed in its video layer. That still needs a real display.
• Coordinates drift. After an error dialog the window changed width, the button moved, and two clicks hit nothing. Every click needs a fresh screenshot.
• It’s slow. A few seconds of waiting plus a screenshot per round. Fine for a verification pass, not for every commit.
• Multiple windows. If the app opens more than one, xdotool search --name '<window title>' windowraise picks the right one, but you have to know the titles.

None of these tools are new; I’ve used Xvfb and xdotool before. The new part is a model in the loop that looks at the screenshot and decides where to click next.

The tedious testing is now something I can hand off, and the app is better for it.