No. 7746 words
Update on building my next frontier eval suite for VulcanBench
I have been working on my next eval suite for VulcanBench for over a month now, it’s my fifth Frontier suite, so I have creatively named it, Frontier v5. What has been so interesting to me about building evals is that when I first create an eval suite, I feel like I’ve really found a unique and interesting way to challenge a model. Then when that suite gets saturated, which happens, to all eval suites, I always look back and go, what the heck, I could have done much better!?!
Maybe in many ways, this is something I like the most about building eval suites. As a researcher, you’re really just trying to build tools to do better research, and learn new things, and at the pace new models are coming out, this means constant change.
I like change, and I like the challenge that comes with it. Frontier v4 was focused on a specific task shape: give an agent a working binary, a spec that is supposed to describe that binary that has drifted (i.e. it says the binary does some things that it doesn’t), and the challenge of writing code to replicate the binary.
What I liked about this task shape is that it doesn’t just test how well a model can write code, it also tests problem solving abilities. A model isn’t just told to find bugs or write a program to do X, it’s asked to solve a problem, creatively, and to use code as the way it shows it solved it.
Now with Frontier v5, I have some new ideas I want to try with a broader range of task shape, more programming languages, and varying repo sizes. What I really want to move towards is a place where someone can say, I write code in Rust mostly, and mostly with Large repos, what model and effort combo is the right one for me?
One thing I’m also trying to do is reduce the amount of time it takes me to run a benchmark, because I would always like to be able benchmark more models…and safe to say, there is no shortage of models to benchmark these days.
It’s taking longer than I had expected to build Frontier v5, but that’s okay, I’m actually having a lot of fun in the process, and in many ways, I’m delaying it myself as I poke holes into different design ideas. Since VulcanBench is a completely independent benchmark with no business model, no revenue, no investors, just me, spending my own money, my own time, researching and learning, I can kinda just do it my way, which is nice.
All this being said, I have had a lot of people reach out to ask me how things are going with Frontier v5 and when it will be ready. And well, I don’t really have an answer to the when question, but I can share a bit more about how the current iteration is going.
First, Frontier v5 is not an evolution of Frontier v4, it is an entirely new eval suite, built from scratch, from the ground up, with nothing from Frontier v4 living on. I have moved my sandboxing over to Harbor Framework, and I’m loving it so far, super impressive stuff.
On the task shape, I will likely land on 3-4 different kinds of tasks, right now these are looking like things like bugs, small features, a performance target, and maybe a diagnosis. The agents will be challenged to solve these tasks across a range of programming languages, and with three different repo sizes: small, medium, and large.
I am also going to tighten the timing budget for these tasks because at the end of the day, I think time-per-task is starting to matter more and more, esp. with the current frontier. You can likely find two models that roughly tie in accuracy, but if one takes 3x the time to complete a task, you’d probably be okay being 5% less accurate, but significantly faster right?
The output format is going to be something pretty fun imo, I’m looking at a heatmap that can hopefully show how each model degrades (or not) as repo size grows.
And, that’s what I’m thinking now, and testing, and continuing to poke holes in. I do hope it will be done soon, but I’d rather it be done right, so I have no exact timeline on what it will be finished. But there’s my update, onward!