X No. 191,326 words
Sharing more about my mission with VulcanBench

The problem with benchmarks today, that benchmark coding tasks, is that they typically only benchmark models at one effort level, often the max effort level a model provides.
Engineering teams are more confused than ever now because they want to use the best models to write their code, but they also know that using the best frontier model on max is expensive and slow.
And the reality is, for daily work, routine stuff that engineering teams do daily, the most powerful frontier model at max is overkill. They can likely use a less powerful model, at a lower effort level, to get the exact same accuracy on their routine daily tasks, faster, and a lot cheaper.
But the challenge has always been, how can they figure this out?
If they look at a lot of the benchmarks out there today, they just see how models perform at Max effort. It would require a huge amount of experimentation for them to determine, oh, I can use that same model at Medium effort, for most of my daily tasks, and get the exact same accuracy, and I don’t have to leave and make coffee while I’m waiting, it just happens in front of me, fast.
That’s the core vision of VulcanBench and the problem that we solve. As a CTO, I was having a challenging time figuring out what models and effort levels my team should be using. If I just picked an Anthropic model, at high effort, and called it a day, we’d be paying over $300,000 a year.
Today we’re getting that same performance for under $30,000. But the only way I was able to do this was to both build eval suites that represent the kinds of tasks our team does daily, and run those across effort levels.
VulcanBench in many ways provide the actionable insights engineers and eng leaders need. Not, what model is the smartest at max effort, when compared to all the other frontier models at max effort. But instead, what model can produce the best accuracy, reliably, on the kind of engineering tasks they do daily, and what effort level they actually need to do that.
One of the core things I’m starting to see a lot of interest in right now is custom evals suites, i.e. evals based on the actual codebase and coding tasks for a specific company.
Here’s a good example. A company that writes mostly Rust code, in medium-sized codebases, can’t really use a lot of the benchmarks today to understand what model(s), and effort levels to use, for their engineers to write the best Rust code efficiently.
If they look at something like SWEBench, which is a really solid agentic coding benchmark, well it’s 100% Python tasks. But there’s SWE-bench Multilingual, which might seem like a better fit, and yes, it has Rust tasks in the eval suite, but they’re less than 10% of the tasks.
So maybe they look to something like TerminalBench, one of the most widely cited benchmarks out there, and another very good one. None of the TerminalBench 3.0 tasks are Rust.
And yes, different models, at different effort levels, are good at different kinds of tasks, on different codebase sizes, at different effort levels, and no, more effort doesn’t always mean more accuracy.
This is such a core point of confusion, and it’s also when so many engineering teams, and really people in general, overspend on models and end up constant either hitting usage limits, or paying a massive bill.
Reasoning effort controls how much thinking the model does before it answers. It’s an input knob only, adjusting one specific value, not a deterministic way to ensure high accuracy output. More effort just means more reasoning, on super complex architecture problems, or a really tricky bug in a large codebase, that can certainly be needed, and provide higher accuracy. But for most routine engineering tasks, all you’re doing is increasing latency and tokens, to get to the exact same answer.
Through my research with VulcanBench, and testing models across effort levels, on real coding tasks, across languages, I’ve also learned that most PRs, especially bigger ones, can be divided into two or three components.
Smaller PRs can be divided into an easy and medium segment, larger more complex PRs can be divided into an easy, medium, and hard segment. I’ve been experimenting with this a lot and came up with something I’m calling a Difficulty Aware, Multi-model Execution Layer.
By giving a smart and generally enjoyable model like Fable a task or PR, and asking it to divide it into two or three segments based on difficulty, I can then route each of those segments to a different model. The result very well could be the same as if I had sent it all to Fable High in the execution layer, but by using 2-3 models, I can get the same accuracy, much faster, and at a fraction of the cost.
While I can do this now with all the data I’m generating from VulcanBench, most engineering leaders are still just kinda scratching their heads, looking at benchmarks like TerminalBench, DeepSWE, etc. all solid benchmarks, but missing an actionable step for them to know how to use agentic coding models more efficiently with their teams.
There’s also one more very important layer, that gets overlooked too often, harness. The exact same model performs differently, and so differently that it impacts both accuracy and token use, in different harnesses. Using Opus 5 in Claude Code is different from using Opus 5 in Cursor, which is different from using Opus 5 in Pi.
Pretty much every benchmark today is benchmarking models at one effort level in one harness, and there’s a very good chance that that harness, and effort level, are not the exact harness you’re using, or effort level you actually need.
I have started to layer this into VulcanBench, testing not just the model itself in a neutral harness, but seeing how it performs in its native harness, and other harnesses optimized for agentic coding.
But once again, most benchmarks are looking at neutral or custom harnesses, and max effort, and often on tasks that don’t represent what real engineering teams are going to send to these models. It’s data, but it’s confusing, and not actionable.
So what do most people do when they don’t know what to do? Just stick with what they’ve been doing. Okay, everyone just codes with Opus, and yeah, we need good code, so set effort to High or greater!
I’ve talked to a bunch of engineering leaders at larger companies that spend six-figures a year on either Claude or ChatGPT, and 100% of them were using the same model, at an effort level of High or greater for everything.
So I think it’s pretty safe to say, if an Enterprise is using ChatGPT, they’re team uses GPT 5.6 Sol High or Extra High or Ultra for everything. If they’re using Claude, it’s Opus 5 High or Extra or Max, for everything. Same model for planning, single model execution layer, with that exact same model, at that exact same effort level.
This is the core problem I am dedicated to solving with VulcanBench, helping engineers and engineering teams know what models, at what effort levels, and in what harnesses, do they actually need. And the only way I think it can be solved is with eval suites based on real engineering tasks, and with more language and domain-specific evals, and benchmarks, lots and lots of benchmarks, across effort levels, not just at max.
My biggest barrier today is cost. Benchmarking models across effort levels isn’t cheap, but I’m continuing to find more labs that are supporting my mission, and coming up with more optimized ways to run benchmarks in a more cost effective way.
Still a lot more to learn, and a lot more tokens to burn, but I’m on a mission.
Live long and benchmark 🖖