X No. 201,245 words
I'm adding code quality score to VulcanBench, would love some feedback on my v1 approach

Last week, Astra and Fable 5.1 were released, and I started benchmarking both the first day I got access. Then, as my benchmarks were running, I started to see more and more posts on X from people reporting code quality issues with Astra, tweets like this:
astra has a code quality problem.
let me be a little more precise. astra can write good code. but when it infers its in a situation where no one will ever actually look at the code, it does not write it with human readers in mind. it also doesn’t write it with *long term
And of course, I don’t jump to any conclusions, I’m an analytical guy, I run benchmarks, my mind immediately went to, how can we prove or disprove something like this?
I also realized that with this new class of models, i.e. the current frontier, could be so crazy smart, that they can ace benchmarks better than ever, and maybe it’s time for benchmarks to evolve in the process.
Looking at Sam’s post about Astra, immediately made me think, these scores don’t necessarily give me as much signal about a model, they tell me more about the fact that benchmarks might not be evolving fast enough to really give us meaningful signal with this new class of models.
GPT-6 Astra is here.
We hope it will begin to enable a new generation of entrepreneurship, scientific discovery, and building.
We believe it is the best model in the world for computer use, professional work, science, coding, cybersecurity, and more.
It took us some extra time
When I see:
98% on FrontierMath Tier 4, 99.9% on ARC-AGI 3, and 100% on ExploitBench.
I don’t think, holy moly that’s a powerful model, I think, maybe these benchmarks aren’t quite the right ones to give meaningful signal about this model, and there could be many reasons for that. It doesn’t mean they’re bad benchmarks, heck, I can tell you, everyone on the teams making all three of these benchmarks are 10x smarter than little ol’ me.
But I do think there could be some additional factors that benchmarks could start taking into account, and with all the concerns about code quality, this feels like the right place to start.
I think this is so important, that after spending over $1,000 benchmarking Astra and Fable, I stopped, took a step back, and looked at how I could layer this in so what I share with VulcanBench can really give some meaningful signal to engineers and engineering leaders like me that are trying to figure out what model is the best for their team.
I have decided to delay my Astra and Fable 5.1 benchmark release, a decision that will cost me quite a bit of money and time, but for what I think is the right reason.
Over the last couple of days I have seen a number of posts like the one below, citing potential code quality
So I thought I’d share a bit more about my current approach, to get feedback, because as I’ve said many times, I’m no expert, I’m just a normal(ish) guy, running my own benchmarks and sharing them.
I have zero funding, am burning through my own money doing this, and don’t get early access to any models like all the cool kids do.
Instead, I’m just too crazy curious, esp. about how things are changing at the frontier, and as a founder/cto, I want my eng team to be able to use the right model, at the right effort level, and know if they even need something like Astra or Fable for daily work, because we also could be reaching a point where some of these models are more powerful than you need for routine stuff.
Okay, so that’s a long lead up to just sharing my approach to scoring code quality/maintainability. And, I don’t write with AI so apologies if the writing and grammar sucks, I’m 45, I’m old, I’m banging on my keyboard like a dinosaur with everything I write here.
What I would love is feedback. Do you think I’ve come up with a unique and interesting way to gauge code quality of frontier models? Or is it trash, and I should be thinking about it differenttly? Be brutal, I can take it.
Okay, so here’s my v1 approach to grading code quality/maintainability of frontier models.
VulcanBench Code Quality/Maintainability scoring v1
First, I am making code quality/maintainability 33% of my overall score. I think it’s at least important enough to be a third of the score. If I give my team a model and they write a bunch of code, and that creates a lot of tech debt, that’s bad, so it’s important for me to know this information about a new model.
For judges, I’m using other LLMs. While humans might be the best, that’s just not scalable, it would take forever to get benchmarks out, and I’d have to build a whole system for hiring people, making sure they aren’t cheating, etc.
But I’m making sure the judges aren’t LLMs from the same labs that built the model. For my first cut at this, with Astra and Fable, I’m using GLM 5.3 and Grok 4.6 as judges. They both score, and then I take the average of their scores.
The rubric
Every judge gets the same instructions, byte for byte, frozen by hash before any review. It also explains who it is scoring for: an engineer who has never seen the code, reads it top to bottom without running it, and must make a correct change in one sitting. I’m also telling the judge that it’s own ability to read dense code is not evidence of readability.
And here’s my rubric:

For every score graded with the rubric, there must be a clear citiation of an exact place in the code and impact. If there’s no reference to a place in the code where it found an issue, then there’s no score impact. Also the host computes the two sub-scores and their mean, this is not done by the judge.
Four layered approach
Andddd, because I want to be thorough, or maybe I’m just a crazy person and really going deep with this, your call, but I’m not just using this rubric, there’s a four layer system to determining the full code quality/maintainability score.
Here’s the table with weights below, showing how I get to 33%:

Phew, okay this ended up being a wayyyy longer article than I was expecting, but I wanted to make sure to go through what I’m thinking enough to get feedback.
Whatever feedback I get will go into v2, since v1 is running now, and I’m going to be the last benchmark on the planet to get my Astra and Fable benchmark out…but I’m okay with that, because I think I’ll be the first benchmark to take code quality and maintainability into account in this way.
Once again, I am not an expert, so am not writing this article saying, “ha, look how smart I am, look at what I came up with!!” Instead it’s more like, “I’m not that smart, you’re probably smarter than me, but hey I run these benchmarks and think there’s a gap here that I can fill to get more signal, help me make it better”
Okay, that’s all i got, it’s a beautiful morning in Tahoe, I’m going out for a hike! 🥾 🌲