Morgan's Blog

not an expert, always learning

X No. 181,704 words

Some reflections on building evals and benchmarking frontier models

Building eval suites for VulcanBench is really fascinating, and also so much more challenging than I was expecting.

Yup, I knew it was going to be hard, but with multiple new models coming out weekly at this point, it’s probably the hardest it has ever been to build eval suites.

Right now I’m working on two new eval suites, CyberSecurity v1 and Python v1. I’m trying to go deeper into specific domains and programming languages because I think evals like these can both help people really understand if a model is good for the kind of work they do, and help labs build better models.

I’ve had a lot of people ask me about how I build my eval suites, and since I’m in the middle of these two, I thought now would be a good time to do it.

Also, I’m far from an expert, so take what I write with a grain of salt. I’m sharing my experience here, as a guy that’s been building evals for months, not years. But my process is continuing to evolve, and I’m learning a lot as I go, so I thought, why not share.

Generating tasks for my eval suites

The first step in building an eval suite, or at least my first step, is figuring out what tasks I want to include, and how many total tasks I want. If I had infinite money, I’d had thousands of tasks, but that would also be super expensive to run. So I’ve had to find some balance.

I decided that between 20 - 30 tasks is probably the sweet spot. It’s enough tasks to get real signal, but not so many that running across effort levels, on a single frontier model, is going to completely break the bank.

That being said, if you look at a new(ish) frontier model, and the cost to run one pass, across all effort levels, on 20 - 30 tasks, you are looking at somewhere in the $50 - $100 range. Luckily, competition is helping here, and I do see costs going down, not up, which means it should hopefully get cheaper to run these over time.

My lucky number is 23, so you’ll probably notice, all my eval suites have 23 tasks.

All of my eval suites are 100% focused on coding, and I want to make sure to replicate real world coding tasks, so I look for real repos on Github, and actually PRs that were submitted and accepted. This to me shows a clear validation path, and prevents creating tasks that might not truly represent the actual coding tasks someone might give a model in real life.

I also have to be aware of the date a given PR was submitted and make sure that it’s not in the model training data. At the same time, I know other benchmarks tell the model not to use something in their training data if it exactly matches the task…but I’m not sure how much I trust a model.

Now I wish I could just go out and find 23 tasks that look like normal stuff me and my eng team would use a model for and call it a day, but most frontier models are just too darn smart. And my challenge, in building eval suites, is finding tasks that truly can stump frontier models, within a given time boundary.

The time boundary piece is probably good to cover as it’s an area that I’m currently doing a deeper dive into.

When you’re benchmarking a model, kinda like when you’re giving a person a test, you need to give it some time limit. With infinite time, maybe I could get a perfect score on a test like the SAT, and maybe a lot of other people could too. But once you constrain the time, things change.

But, if you give everyone only 20 minutes to finish the SAT’s, you’re going to get a lot of low scores. You need to balance it and determine realistic timing. This is the same challenge I have with building eval suites. For the harder tasks in the suite, they need to be tasks that are first, of course, solvable, but hard enough to solve that not all models can solve them when given a generous(ish) amount of time.

One benchmark that I think does a good job with really thinking more deeply about timing is TerminalBench. Here’s the task timing that TerminalBench 3.0 uses for their tasks:

(TerminalBench 3.0 task timing)

(TerminalBench 3.0 task timing)

As you can see, TerminalBench 3.0 has a total of 74 tasks, and for most, models are given between 2 - 4 hours to complete them. I have developed a similar methodology, and essentially bucket my tasks into size/difficulty. Here’s the timing that I use in VulcanBench:

(VulcanBench eval suite 3 task timing)

(VulcanBench eval suite 3 task timing)

So when I build my eval suites, I try to find tasks across these five buckets, and with smaller tasks, timing is really important. Take a micro or small repo scale task. This is something that maybe any frontier model could solve, they aren’t crazy difficult, but solving it in under five minutes, or under ten minutes, that’s where you can really see how good a model is.

This is one of the areas I’m continuing to experiment with more and more because it’s one of the most important components in creating good eval suites. You want to truly challenge the models, but also give them tasks that cover a good range of difficulty to replicate things real people would do with the models.

Testing my eval suites

Once I have the first cut of my eval suite done, it’s rarely actually done. Now I need to test it and see if it’s hard enough, but not too hard. And I’ll be honest, this is an area where I’m definitely still learning a lot as task difficulty selection in eval suites is tricky, especially as models get more and more powerful.

Typically I will test my eval suite with a solid model from a previous generation first. So for my Python eval suite I’m building now, I tested with Grok 4.5 first. Grok 4.6 is the latest model, Grok 4.5 is the previous gen, so this eval suite should be a real challenge for Grok 4.5.

At the same time, and here’s where things get really tricky, it shouldn’t completely stump the model on all questions, and a previous gen model should be able to solve for maybe half the tasks in the eval suite, or at least that’s my current thinking.

But, and this is an important one. The tasks in my case, do need to really truly represent real tasks that real engineers would give to a model. So if I have an eval suite, that a previous gen model can solve for 50% - 60% of the tasks, and all those tasks are real tasks that you or I might give to the model in our own daily coding work, that’s fine.

But, if the tasks are just gimme’s, i.e. something that any model could solve and don’t represent things any of us would actually give to the model, well then those are a waste.

If it looks like my eval suite is hitting where I’d want it to with a previous gen model, then I’ll go back another generation, and sometimes switch to a different lab, so maybe I’d look at something like Opus 4.7 and see how it does.

My hope is that as I go back further in the model release timeline, the models score lower. Never zero, but lower. And as I said above, this is an area that I’m still refining and tinkering with constantly, and my thinking will likely change here more over time.

But let’s suppose that the eval suite is performing as I would expect with the previous generation of models, then I’ll test with current gen, and typically look at 3-4 different models, across effort levels, and see if I think those scores represent a real difficulty threshold that would make this eval suite a good candidate.

Usually, the answer is no, and I have to continue to refine tasks. This process goes on for a while, often weeks, and every day I swap in and out different tasks, test and retest until I’m satisfied.

Where I’m looking to innovate more

Right now building my eval suites is a pretty manual process, so I’m starting to build some tooling around all of these steps so that I can better automate the process. At the same time, I’m also trying to build some additional components into this tooling that can evaluate existing eval suites that are in the wild, both mine, and others, to determine if they are still relevant.

This I think is one of the biggest challenges in building eval suites and benchmarking models today. New models come out, and there’s a decent chance your eval suite is in the training data. And yes, you can instruct the model not to cheat, but at scale, a lot of benchmarking companies have a hard time actually verifying that the model didn’t cheat. Instead, they tell it not to cheat and trust that it didn’t.

I think there’s probably some interesting ways to better gauge how hard it is for a model to cheat, based on the eval suite, model training date, and a few other factors, and either update or augment the existing eval suite, or as we should probably be doing more of, declare a specific eval suite no longer valid.

Still early for me here, and not much more I can share meaningfully beyond this. Just continuing to iterate and learn as I go, and the AI labs are keeping me nice and challenged as they continue to come out with new models.

Okay, I’ll end it here for now, I feel like I could just keep writing for hours here, but I also know I gotta stop somewhere right!

If you liked this article, please share it, and if you don’t know, I also run The Model Routing Substack, which you can check out at modelrouting.substack.com

Oh and of course, if you have any questions, ask away, thanks for reading!


Further roving

  1. X No. 19
  2. X No. 17
  3. X No. 20