X No. 13687 words
Coming up with a better way to compare different agentic coding tools and models

As a CTO today I’m on a mission to help my team access all the best agentic coding tools out there. And, yeah - that mission is a complex one because every week new tools and models are coming out, and it’s pretty crazy to stay on top of it all.
At the same time, at least for me, I’m finding it incredibly fun and exciting to try to stay on top of it all. Of course, the best way to stay on top of tools and new models is X, this is really where you can drink from the firehose. But the flip side is, it’s a lot to take in, and the place I’ve found myself struggling the most lately is comparing different agentic coding tools and models.
So this morning I sat down with Opus 4.7, which actually isn’t my favorite model for coding these days, but is my favorite for planning/ideating, and started to try to come up with a better way to test.
Since I know there are a lot of other founders and engineering leaders in a similar position to me right now, I thought, why no share what I learned, and how I’m planning on improving how I test and benchmark new agentic coding tools and models. So here ya go.
One good task beats five mediocre ones
Use a single fixed task you run against every tool. Variance between runs is real, you want apples-to-apples, and multiple tasks multiply evaluation cost. A well-designed single task discriminates better than people expect. SWE-bench and Aider’s polyglot benchmark exist for broader coverage, but for your own internal comparison, one well-instrumented task is enough.
The task should be in a real(ish) codebase (not greenfield, otherwise no navigation testing), have a hidden test suite for objective correctness, contain at least one subtle requirement that distinguishes careful from sloppy work, and take a senior dev roughly 30 to 90 minutes. That’s the sweet spot for discrimination.
Concrete structure
Take a small open-source library (~1000-3000 LOC), check it out at a specific commit, and pick a real closed bug:
-
Find an issue on the repo that landed with a test-first commit
-
Reset to just before the fix
-
Task prompt: “There’s a failing test in test_X.py, fix the underlying bug. Don’t change the test.”
-
Ground truth: the actual fix that shipped
-
Pass criteria: failing test passes, existing tests still pass, diff stays reasonably scoped
Real bugs have natural traps built in (the obvious fix is often wrong), which is what you want. Greenfield “build me a todo app” tasks don’t discriminate well because everything works the first time.
Measurement
Per run, log:
-
Wall clock time
-
Input tokens and output tokens separately (prices differ a lot)
-
Number of tool calls / turns
-
Pass/fail on hidden tests
-
Diff size vs. ground truth diff (proxy for scope discipline)
-
A blind 1-10 code quality score on the final diff
The useful efficiency metric is tokens-per-successful-task, not raw tokens. A tool that burns 50% more tokens but always succeeds beats a cheap one that fails half the time. Plot tokens against pass rate across multiple runs.
Run each tool minimum 3 times. Variance is significant and single runs lie.
One trap to avoid
Don’t conflate model and tool. Same model behaves differently in Cursor vs. Droid vs. Claude Code vs. raw API. Either fix the model and vary the tool, or vice versa. Not both, or you can’t tell what you measured.
I’m putting together my first really detailed test, still picking the library I want to use, and the exact bugfix I want to test against. And likely the first thing I try won’t be quite right, so I’ll need to iterate and try a few different things before I really nail it.
Once I do, I’ll probably put together another article to let you all know what I landed on. For now, I’m still in learning-mode, which is kinda my favorite mode to be in, esp. on a Saturday morning.
Okay, that’s all I got, more to come, happy Saturday, don’t forget to go outside, and get into nature if you can! 🧘♂️🌲