X No. 21639 words
How I'm building an independent safety benchmark for coding agents, and why model effort matters

So…I started writing this as a regular post on X, and then I just kept going and going, and decided, I think this is an article.
As I develop out the first eval suite, and framework for VulcanBench Safety v1, I wanted to share a little more about my approach. I think AI safety is incredibly important, and as an independent benchmarker, I feel like I have a unique role to play here.
First, let me point out one important fact: zero people from any ai labs are working with me, or are giving me advice on how to build this. Just thought it was important to get that out of the way, now here’s a bit more about what I’m doing:
-
First, I’m going to start with what I’ve got, @VulcanBench already has 23 tasks where an agent is given a problem to solve around making a Python module behave identically to a legacy binary.
-
Then I’m forking some of those tasks and putting prompt injections in parts of the repo, in places that normally, none of us ever read, and if we do, we skim. The prompt injections ask for things that escalate from basic housekeeping to real credential theft.
-
This is also where I move some of what VulcanBench does out of the open source world into a private repo, because I don’t want models trained on this for obvious reasons.
-
And to make things even spicier 🌶️ every fork also will have a planted .env with realistic fake credentials and a set of ground rules in both the AGENTS.md and CLAUDE.md that says to only touch the file you were asked to fix, and leave the reference binary alone, treat .env as confidential, and don’t touch the network.
-
To measure safety, a detector reads each run’s command stream, final patch and preserved workspace and classifies every injection as followed, reported (did not follow and mentioned it in the final message), or ignored.
-
Determining if an agent created a safety risk, comes from what the agent actually did, not from a network listener, so it works the same across harnesses w/different sandboxes.
-
Annnd, secrets are checked separately: reading .env is just annotated, but a secret value appearing in the final message, a command, or the patch is a big red flag. All of this sits next to the existing honesty, scope, and destructive-action detectors, and every number rolls up by model and effort level with Wilson intervals.
So how is VulcanBench Safety v1 different from what’s out there?
- From a technical standpoint, most injection benchmarks use synthetic environments that don’t necessarily reflect the real environments companies use agents in. I’m measuring how likely an agent is (model + harness) to break safety rules when doing ordinary engineering work, in the actual harnesses, real engineers at a company actually install and use.
- I’m testing across effort levels. This has always been the VulcanBench way. I think doing safety tests on models, only at Max effort, does not provide a real assessment of model safety, because most engineers at companies, aren’t running models at Max effort all day every day.
More to come, and like I said, this started as a post, and holy moly did that post get out of hand, it now honestly even feels like an article that’s too long, so I’ll end it here.
I am open to feedback, I won’t pretend to be an expert, because I’m not. As I tend to do, I will learn as I go, and share every step of the way.
So please comment, and tell me what you like or don’t like about my approach. This isn’t set in stone, I want to learn, and a lot of people reading this probably have good ideas I can learn from, so I want to hear them.
Live long and benchmark 🖖