# I have become obsessed with building eval suites to challenge the frontier, and it's fucking hard

By Morgan Linton · Published 2026-10-07 · X No. 22 · 1,396 words · https://blog.morganlinton.com/on-x/i-have-become-obsessed-with-building-eval-suites-to-challenge-the-frontier-and/

![](../../assets/on-x/HUCH2jtbgAAaxDD.jpg)

Am I going to get in trouble for putting the F word in the title of my article? I think this is okay on X, but I guess we'll find out. For those who know me, I never swear, but this felt like the right time to do it, because seriously, it's really fucking hard to challenge the current frontier...but also one of the most fascinating and exciting things I have ever done in my life.

Hi, I'm Morgan, I'm the Cofounder and CTO of a software company, and I have gone down the rabbit hole of building eval suites to stump the latest frontier models. Oh and I built my own benchmarking tooling to do this, which I named [VulcanBench](https://www.vulcanbench.com/) 🖖

Like a lot of engineering teams, earlier this year we were struggling to figure out what model and effort levels we actually needed to do our regular work. I think a lot of leaders get lazy here (sorry but it has to be said), and just throw up their hands.

And then, everyone uses the latest frontier model, at High effort, sometimes higher, and they rack up huge bills, and spend a lot of time waiting.

I thought this was just too fucking inefficient of a path, so decided to start building my own eval suites, based on our own internal repos and actual PRs from my team.

What I learned is that the current frontier at that time, was overkill for a good chunk of routine daily tasks, and Low and Medium effort levels were all more than enough thinking depth. When you're using models with lower reasoning efforts (and getting the same accuracy) you are moving faster, and paying less.

This created a real shift in how we work with coding agents. We moved from the 2025 model of: give an agent a coding task, then go get coffee, wait, and probably wait some more. To pair programming with agents. It's a completely different modality imo, and it created what I can now say is a true addiction/obsession with building evals.

And that was earlier this year. Today, the frontier has moved even further forward, models are insanely smart, and lower thinking depth can provide the same accuracy as models did with Max effort just a few months ago.

I started sharing more public benchmarks, with eval suites not based on our internal repos/prs, but instead focused on truly challenging the current frontier, and holy shit is it getting harder.

But I like hard, in fact, as I've learned over the last couple of months, I can't stop thinking about how to build new eval suites, update how I run my benchmarks, and not just stump the frontier, but do it in a way that provides real signal that can be used for something.

So many benchmarks just show how models compare at Max effort. But I never use Max effort, my team never does, and a model that kicks ass at Max effort, might not be as good at High effort as another model half the cost and 3x faster.

Okay, so I really didn't expect this much preamble here, so I'll stop here and talk more about some of the challenges I've had building eval suites and what I've been doing to improve over time.

My wiring is, build something, get it out there, get a bunch of data and feedback, and use this to iterate, and tighten this loop as much as possible. I would like to be building new eval suites much faster, and I think as benchmarkers, we all need to move faster because the current frontier is not pacing itself, it's moving insanely fast.

Right now I'm working on the fifth iteration of my Frontier eval suite, and taking a bit of a new approach by building Frontier v5 and Frontier v5.1 in parallel. For all my previous eval suites I've used one coding agent, with Fable being my go-to because it's just such a sharp and thoughtful model.

For Frontier v5, I'm using Fable, and then in parallel, I'm using [Factory](https://www.factory.ai/) with their Auto Model orchestrator, with Opus 5.5 as a worker and GPT-6.1 Sol as the validator.

![](../../assets/on-x/HUB6vISb0AAuYks.png)

This isn't an article about setting up Factory, or their Auto Model router, so I won't go much deeper here. Instead, what I want to share is a new approach I'm taking to building out my eval suites, which is essentially looking at each coding agent as an employee/researcher, and pitting them against each other.

For the past four version of my Frontier eval suite, I have been the person that evaluates each decision, now I'm trying to see if I can have the models compete with each other, learn from each path, and in the end, produce a better result.

Here's the quick look at the Mission I setup in Factory:

![](../../assets/on-x/HUB6bqQbAAAlwXp.jpg)

As each coding agent reaches a new milestone, I will be sending what it has done to the other, so they can analyze each other's work, and (my hope) each improve as they build.

Both are also starting with a lot of lessons I've learned building my first four Frontier eval suites. I did a lot of experimenting with timeouts, at first they were too low, then I set them to 10 hours, and yeah, that was wayyyy too high. While it was fascinating to look at what happens when frontier models overthink like crazy, for ten hours straight (seriously interesting traces), that isn't actually what an engineer wants. Nobody wants their agents to spin its wheels, burning tokens for ten hours, especially if another model can actually complete the same task in 15 minutes.

I have also done a lot of research on sandboxing and determined there is one clear leader in this space, [harbor](https://www.harborframework.com/), which is made by the same team behind terminal-bench. I really like this diagram in their docs, and think it does a better job of visually explaining how harbor works a lot more than I could with a bunch of words:

![](../../assets/on-x/HUB8kHtaUAADY6y.jpg)

Additionally I've been thinking a lot about the spread of programming languages my eval suites include. Frontier v5 is going to have tasks in six different programming languages: Python, Javascript, C, C++, Java, and of course Rust. I love Rust 🦀

Frontier v4 focused on one task shape, I would give a model a spec, and a binary that had drifted from that spec, and ask it to write a program that matched the binary. I like this task shape a lot because it goes beyond just asking a coding agent to code, it really looks at how well it can think and problem solve, which imo is what makes engineering different from coding.

But, I don't like just having one task shape, so I'm adding more in, while still keeping the binary reconstruction for some percentage of the tasks.

There's a lot more bouncing around in my head, but I think I'll leave it here since this article has already turned into something much longer than I thought it would be.

What I love about this moment in history, and in my own life, is that I get to wake up every morning and look at truly novel results, at the edge of the frontier. I get to see how the newest models think, and where their thinking gets either totally wacky, or incredibly beautiful. I get to see my evals saturate, have moments of pure frustration, and then get to challenge myself even more.

This is the most alive I have ever felt, I am fucking obsessed and I love it. Some people say researchers building benchmarks is a waste of time, I strongly disagree. I think building and progressing independent benchmarks is one of the best things we can do to make an impact in the world.

Just look at something like [terminal-bench-science](https://www.tbench.ai/news/terminal-bench-science-0-1). This benchmark will likely be used by scientists who could find breakthrough treatments for certain kinds of cancers, alzheimer's, or dementia. Helping brilliant people pick the right model to solve truly life-changing problems is as big a purpose as I can think of.

And to help support independent benchmarks like VulcanBench, I'm not asking for money, I'm asking for feedback, for criticism, for anything you can do to help make my evals better. We can all make an impact together.

Live long and benchmark 🖖
