Morgan's Blog

not an expert, always learning

X No. 231,507 words

Some thoughts on ways to avoid hill climbing in eval suite design

The more time I spend building eval suites for @VulcanBench, the more I realize I don’t know about building eval suites. It is a pretty humbling experience, especially as the frontier continues to get smarter by the week.

I am currently building out the fifth version of my core eval suite, Frontier v5, and in the process, making a lot of mistakes, and trying my best to learn from them.

What I’ve been trying to evolve more with this eval suite is the task shape, because I’ve determined the way I have built my tasks in the past needs to change if I really want to truly challenge the current frontier.

The way I had planned to build this new eval suite was to give an agent a real open source project with some bugs added to it, then give the agent a description of what was going wrong and tell it to fix it without creating new bugs.

I started sourcing tasks, and tried to make some really challenging stuff that would stump the current frontier. And yeah, no dice. Models like Astra were able to ace six our of seven tasks in four to twelve minutes.

What I didn’t quite realize is that it wasn’t necessarily that my tasks weren’t hard, they were, but I had built hills, and agents can climb hills, really well.

So what do I mean by hills? I honestly didn’t know this was a thing until last week, so it’s new to me.

What hill-climbing means

I find that I learn the best with analogies, so I’ll use an analogy here before getting into the specifics of my eval suite design.

Let’s imagine I take you to a hill and blindfold you. I then tell you to try to climb the hill. Every step you take I yell, “higher!” or, “lower!”

If I keep doing this, you’ll make it to the top of the hill because I’m giving you hints every step of the way.

An AI agent essentially does the same thing itself, when a task is shaped in a way that it can quickly and easily check its progress.

Essentially, a test suite that runs quickly, is just like me yelling higher or lower. It’s a quick way for an agent to know if it’s going in the right direction.

And a working copy of the right answer, is an even more helpful. An AI agent can just make a small change, run tests, see some of them pass, and repeat. It never actually has to understand the problem and reason through it. It can essentially just hill climb it’s way to a perfect functional score.

The problem with binary parity tasks shapes

In my Frontier v4 eval suite, my tasks are designed around getting agents to match an existing binary. I give an agent a spec that has drifted from the actual program, then the agent has to build a program that matches the binary.

My thinking in using a binary parity task shape is that it would be a great way to really challenge models and make them reason more like a human would.

I thought it was a better way to build an eval suite that just giving a model a problem to solve because it required a more thoughtful approach to problem solving. The agent had to look at a spec that was wrong, figure out where it was wrong, and logic through a good solution to match something that was a complete black box to it.

But, yeah, turns out agents don’t quite work that way. While the binary itself was a black box, the agent could take it apart with a disassembler, read the machine code, and copy what it did.

METR shared some interesting analysis on hill climbing as well in a blog post they wrote earlier this year:

Consistent with our MirrorCode results, publicly reported anecdotes indicate that agents were exceptionally strong on problems that are easily “hill-climbable” — those where progress is cheap to verify and many approaches can be tried cheaply — including software reimplementation (as in MirrorCode), many optimization problems, some areas of mathematics, some forms of debugging, and some forms of vulnerability discovery. (Source: METR)

Of course no two models approach solving these kinds of tasks in exactly the same way. And I still got a lot of signal out of binary parity tasks. Some models would take ten hours, or more if they were given the time, to solve a task that another model could solve in under an hour.

In fact, I was having so many issues with some models taking way too long at higher effort levels that I reduced the time I gave models to solve my tasks from ten hours down to three.

And while I could write a whole article about determining the right amount of time to give a model to solve a task, that’s not what this article is about. But I will say I didn’t just randomly pick three, I did analysis to figure out what number was realistic given past model performance on my tasks.

But now let’s get to the more interesting stuff, what I learned and what I’m doing differently to prevent hill climbing as I design Frontier v5.

How to prevent hill climbing

So I’ve been doing a deeper dive in how to prevent hill climbing, and as you probably know by now after reading this, I’m far from an expert, so this is still a work in progress.

But I like sharing as I learn so here’s what I’ve learned so far that I’ll be incorporating into the design of my Frontier v5 eval suite. High-level the goal is to make it hard for an agent to do any “quick checks” that could allow it to hill climb.

1. Make the obvious fix break a visible test. Hill-climbing works because each check gives the agent a way to check if it’s getting closer. This kinda flips the script on it. Pick tasks where the first fix a sensible person would try breaks an existing test the agent can run. Now the agent’s own feedback says it’s going in the wrong direction when its trying to take a shortcut. Test it before using the task: write that obvious fix yourself, run it, and confirm it breaks something.

2. Hide the tests that check things working together. Let the agent see only small tests that check one thing at a time. Keep every test that combines several fixes hidden. This means that an agent can climb the visible tests think it aced it, but still fail because the hidden ones check cases it never got feedback on.

3. Never give the agent a runnable answer key. Anything the agent can run that already produces the right answer is a perfect “higher or lower” signal: feed it an input, compare, fix the difference, repeat. That covers a reference program, a compiled binary, or source code it can build.

4. Keep the real grading out of the agent’s reach. Run the real checks in a separate container after the agent is done, so it can never run them, read them, or see their results mid-task. Otherwise the grader itself becomes the agent’s altimeter. On its own this isn’t enough, because the agent can still climb whatever tests it can see, which is why it works best combined with these other components.

There are some other things I’m noodling on, but these four are the core techniques I’m focusing on in the design of my next eval suite.

I have no idea how long it will take me to build Frontier v5, and I’m actually okay with that. I have fun building these eval suites because I learn so much every time I do.

At the same time, I think it’s kinda like golf for many people. I end up being my own worst enemy, with nobody to blame but myself. Sometimes I’ll spend days refining a task and have to throw it out, it’s frustrating, and I often feel like I’m learning too slowly, but what I can I do but just continue to move forward.

We live at a time in history where it feels like everyone wants to be an expert, or at least calls themselves one. I feel like in many ways, I’m finding myself moving in the opposite direction. As I learn more about building eval suites, I feel like more and more of a complete noob at building eval suites.

The beauty of it all is, I love the process, I’m an insanely curious person and seeing how models think has become one of the greatest fascinations of my life. I wake up every morning to new, novel results, sometimes about a new frontier model, sometimes about a stupid task I made that I need to toss, either way, no morning is the same.

And on that note, it’s Sunday morning at 6:33am and starting to get light outside, time for a hike!


Further roving

  1. X No. 24
  2. X No. 22
  3. X No. 21