Morgan's Blog

not an expert, always learning

X No. 17715 words

Stop evaluating model costs by input/output token prices, what matters is cost per task

Okay, so I feel like this needs to be said, because it’s getting a bit wacky out there when it comes to how people are evaluating model costs. And honestly, I think in many ways, the pricing of input/output tokens has become a marketing play more than a metric in some cases.

Here’s a good example. Take a look at this benchmark from @VulcanBench comparing GLM 5.2, Sonnet 5, and Fable 5.

Source: VulcanBench

Source: VulcanBench

First, all of these models scored on accuracy on this eval suite, so what we’re comparing against is specifically time, token, and cost. Also you’ll notice, Fable was run at low effort, it’s such a powerful model, it doesn’t need Medium or High effort to tie models like Sonnet and GLM. But that’s not the interesting part in this benchmark. Look at the cost.

GLM 5.2 came in at $15.81, Sonnet cost $8.41, Fable was only $2.27. This means GLM 5.2 is actually ~600% more expensive than Fable 5 in this case.

But when GLM 5.2 came out, everyone was touting how cheap it was, only $1.40/M input tokens and $4.4/M output tokens.

Compare this to Fable, which is $10/M input tokens and $50/M output tokens, and you probably couldn’t ever imagine Fable being cheaper than GLM 5.2, but it is.

As you can see from the benchmark results above, Fable used 0.14M tokens to complete the evals, GLM 5.2 used 10.82M tokens, big difference, and that means that GLM 5.2 cost was $1.98/task vs. Fable which was $0.28.

Another good example is a recent benchmark from Artificial Analysis.

Opus 4.8 costs more than Sonnet 5 when you look at input/output token costs, but it actually ends up about 15% cheaper than Sonnet 5 when you look at cost-per-task.

There are a ton of other examples like this, where a model has a low cost per input/output tokens, but because the model itself uses more tokens, it ends up costing a lot more than a model that might seem more expensive if you were look only at input/output token cost.

Also, I can’t really just stop here, because there’s one piece to the puzzle that keeps getting left out, and that impacts all of this: model effort.

Models like Fable 5 for example, can often solve a set of tasks, with the same level of accuracy at Low or High effort. But, with High effort, the cost could be significantly higher because you’re paying for greater thinking depth, which you don’t actually need.

There are a lot of ways to make a model look a lot more expensive than it is, which is why when benchmarking models, I think it’s so important to vary effort and compare across effort levels. If a model is going to get the same accuracy on an eval suite at low, medium, high, extra, and max effort, then showing costs at max doesn’t really make sense to me. All you’re showing is that if you over-provision a frontier model, it cost more, duh.

With this new class of models we now have, I think there’s two important things we have to look at before we can say a model is cheap, or expensive:

  1. Cost per task
  2. Ensure in calculating cost per task we aren’t over-provisioning the model, i.e. we need to see if a model can complete a task at Low or Medium effort, not just High, and if it completes the task at Low, mark that as the cost

Also, it’s important to remember that these are still the early days. The media is still trying to wrap their head around all of the advancements taking place in the LLM space, and it’s a lot more sensational to write about a new “cheap” model that beats a big frontier model, than to actually do a deep dive, and maybe find out that’s not the case, and the story just isn’t as compelling.

But I hope, with this article, and the research I’m doing into building better evals, and benchmarking across effort levels, I can help shed more light on what models truly are producing real breakthrough and cost savings. This will help so many engineers and engineering teams pick the most effective models to use vs. the model that has the best input/output token cost.


Further roving

  1. X No. 18
  2. X No. 16
  3. X No. 19