Will We Lose the Ability to Select Model Effort Levels?

Right now we are living at such an exciting time, not just because AI allows us to move faster, and work alongside agentic teammates, but because we can tune it in really powerful ways. At the same time, yes, this creates more complexity and confusion. It’s why I created VulcanBench, and it saved my team over $300,000/year so yeah, I’m pretty happy about it.

At the same time, a lot of people still have the same workflow from 2025. Pick an agent, stick with their default effort level, and then use the same model/effort to plan, execute, and run code review. For most teams, this means using higher powered models than they need, for a good chunk of time, which means more tokens, and higher costs. For the labs, this pattern is the best thing ever when it comes to generating revenue.

Today the current frontier is insanely powerful, and as Matan from Factory has said, “if your daughter needs tutoring in algebra, you can probably find someone cheaper than Albert Einstein. Giving every task…”

And when you look at agentic coding, there are three core layers, each with high degrees of model/effort variability based on things like codebase size, problem complexity, and programming language. The three layers I’m talking about are planning, execution, and code review.

Here’s an example. If I have a small PR, in a medium sized codebase, in Python, I might be more than fine using say GPT-6.1 Sol Medium for planning, Opus Medium for execution, and Grok 4.7 Medium for code review. But a lot of teams today just say, well we use Claude Code, and Opus High feels like what we need to use, and they use Opus High for planning, execution, and code review.

And yes, it totally nails the PR, but it actually matches the accuracy of the multi-model path I shared above, it just takes longer and costs more. But so many people are wired to think of agentic coding as a process where they essentially pair program with an agent. This stage of agentic coding is actually still quite early imo, but who the heck am I, honestly, I’m no expert, I’m still learning a ton myself.

I do think Boris, the creator of Claude Code does have some good insights here, and while he outlines Stages 0 - 4, I’m going to just focus on Stages 1 - 3 as I think these are the stages most organizations are working through now:

Today, I think most eng orgs that are using agentic coding workflows today are in Stage 1. It’s one engineer, one agent, and they really use coding agents as a pair programmer. While this is definitely a step forward, it’s still a person and an agent, pair programming together, which is more of a 2025 pattern imo than a modern agentic coding workflow like we see teams adopting in 2026.

So going back to agents and effort levels, for teams in Stage 1, using the same coding agent, at the same effort level for this, means using a prompt and wait pattern. And the waiting is often unnecessary because the model is being used at a higher effort level than needed.

A lot of teams right now just accept this, and say, “that’s how it works, you prompt, get coffee, come back, but hey, an agent is working while I’m doing it!” If you scale up to Stage 2 and Stage 3 using this approach, it gets very expensive, and yup, a lot of waiting.

With the benchmarking I’ve done through VulcanBench, I have seen a lot of ways to optimize this workflow so that you’re not using an effort level that takes longer, and uses more tokens, just to get to the same result.

This is what model routers do today, and I think the companies with the most promising model routers are those that can use models from multiple labs, and at different effort levels. But there’s a bit of a wrinkle here. If you have a smaller team, or if you’re just one developer on an individual plan, you’re paying a deeply subsidized rate. Moving to a model router typically means moving to API pricing, which means you’re now paying let’s say 5x - 10x for the same tokens.

At the same time, yes, you’re saving time trying to figure out what model/effort level you and your team needs, and how this varies by team, i.e. your Data Engineering team probably needs a different combo of models and effort levels than your Frontend team. So model routers today solve for this, in some ways, but at a cost. If you weren’t paying API pricing, moving to a model router will very likely more than double your costs.

This is because model routers quote their savings based on API pricing. So sure, if you were on API pricing, using one model, at one effort level, for everything, you would save 30% - 40% or more using a model router. But if you’re like me and my eng team, that isn’t paying API pricing, and is paying subscription pricing, moving to a model router means a very steep cost increase.

So what can we do to optimize? We can look at benchmark data, build evals based on our own PRs, and figure out what models and effort levels we need based on the size of the PR, programming language, etc.

This used to be a lot of work for us to do, but now I have a pretty good flywheel setup, and it’s not much work, and we can change very quickly when a new model comes out.

But my biggest takeaway from this has been, if model companies remove our ability to select effort levels, we’ll be stuck using higher effort levels than we need. And maybe that’s the direction this all goes. The subscription pricing was just a window, a chance to use these models at a very competitive price, and that window closes, and API pricing is all there is.

In this world, then it becomes a battle of model routers, and engineering teams will then pay API+ pricing, i.e. API pricing to the labs, and a fee on top of that to the company that runs the model router. The labs of course, will also build their own routers, so with something like Claude Code, they would route to Haiku, Sonnet, and Opus and pick the effort level for you.

But I still am now convinced, with all the data I’ve seen with VulcanBench that in this world, where we can no longer pick model or effort level, our costs would go up, and our token use/waiting time would too. It would be a step backwards for teams like mine.

And yes, I get it, like I said above, a lot of teams don’t want to have to think about model and effort level, they’d happily pay more to not think about this. My point is, I think we all should have that choice.