A very early look at the design principles for my Computer Use eval suite

Okay, so I think it’s safe to say I am now very much down the rabbit hole on Computer Use v1, my first eval suite focused on Computer Use, and I’ve already made a ton of mistakes. I like making these mistakes, and quickly, so I can learn from them, iterate, and then keep learning.

First, this is much harder than I thought it would be. Building eval suites is always hard, but Computer Use really does add an entirely new dimension that I’m honestly still wrapping my head around, lots of things I didn’t have to think about when building evals focused on agentic coding workflows.

So I think I’ll just go through, in a bit of a stream of consciousness way, the journey so far and what I’m looking at doing next. Oh and for anyone wondering, I’m focusing on MacOS to start because I think this is the operating system where most early adopters of ai assistants like Grok Bot and Muse will use.

The initial piece I wanted to get working was a core loop:

• a MacOS VM
• screen and input driven from the host
• a sample task graded with a reward

I have the core loop working, which kinda means nothing since it’s really just laying the foundation, but my guess is I’ll probably find some tweaks and optimizations I need to do with the VM I’m using, but we’ll see.

What I’ve been trying to zero in on is a couple key design principles, still thinking through these more but thought I’d just share them anyways as I go. For me, a lot of this is around making sure that I can setup a proper orchestration agent to build the eval suite with my own core principles in mind. Then I can test it, lots of stuff probably won’t work as expected, I can learn from that, maybe update one of my principles, and try again.

I have five principles I’m starting with, here they are:

Principle 1: the agent gets a screen, a mouse and a keyboard, nothing else

• The agent sees screenshots and sends input over VNC. It has no SSH, etc.

Principle 2: every trial starts from an identical world

• Each test starts from an instant APFS clone of a pre-seeded VM image…I think this will make resets cheap and precise.
• The clock is frozen and the base image is scrubbed of anything that can vary: first-run sheets, widgets, Siri suggestions, notifications, window restoration, network time.
• Lastly, a determinism check - two clones, two boots, should be the same screenshots.

Principle 3: grade the outcome, read straight from disk

• When the agent finishes, quit the apps, shut down, mount its disk read-only on the host, and copy out only the files the task declares.
• The verifier should only check the final app state, not the clicks: did the right reminders, events, contacts or files end up where the user asked?
• The result is a verifiable (mostly) binary reward at the end of a long episode.

Principle 4: a reward is only useful if it can’t be gamed

• Every task passes a validation pipeline before admission: doing nothing scores 0, the reference solution scores 1 three times in a row, and every known cheat scores 0.
• The “Terminal shortcut” probe: writing directly to an app’s database instead of using the app must fail. That’s reward hacking, and I want to make sure NOT to reward, reward hacking.
• The reference solutions drive the same VNC input path the agent uses, so validation exercises the real stack, not a shortcut.
• Every input event is logged as evidence of how the agent got there.

Principle 5: build in headroom from the start

• Benchmarks saturate quickly, this has been probably the biggest lesson I’ve learned w/VulcanBench…so every capability task must include at least one difficulty lever:
  • a long horizon (60+ actions)
  • state the agent has to carry across apps
  • dense perception (small text, many similar rows)
  • a change partway through the task
  • a Mac-specific interaction (sheets, contextual menus, dragging between apps, modifier-click selection)
• Three families: one app in depth, the system itself (System Settings, Spotlight, Spaces, shortcuts), and workflows across several apps.
• A target that keeps the benchmark honest: at least one task that both reference agents fail.
• Measure before scaling: a small pilot reads difficulty first, and the full set gets built only once the pilot confirms headroom.

Always open to feedback and suggestions, as you know, this is very new territory for me, I’m incredibly far from an expert, so expect me to be sharing all kinds of things that go wrong, and maybe all of these principles changing in the very near future.