Astra writes well (which is why I often like to work with it over Claude), but it tends to produce code golf that's borderline unreadable, like completely omitting line breaks. I thought I had something funky with my config, but it seems to be a known issue.
EDIT: You can search for "tenobrus astra" on X, but link removed.
The generated code certainly does what's asked of it, but it's as sloppy as anything else I've seen. I don't think this benchmark can control for that. Maybe if it had a metric for minimum diff to satisfy the new requirement with the existing constraints on dead code etc?
show comments
taspeotis
It's Opus specific, but personally I'd never bother with Opus above High, CursorBench shows a clear inflection: https://cursor.com/cursorbench
hedgehog
Is there a reasonable way to run a representative subset myself?
Edit:
> For folks who want to try to repro these findings: I used the pinned v1.0 catalog, the stock just-solve prompt, seed 42, the any-case runner pass policy, and Docker with Python 3.12 and uv.
I believe effort can mean both "solve harder problems" and "produce more stuff". I mostly don't want more stuff than I asked for, so a capable model at lower effort can indeed be the wiser choice. Faster response too.
show comments
visiondude
I can’t find this useful given they are trying to quantify wether the code ‘feels’ sloppy with some fuzzy math that can easily be manipulated. at least it’s an attempt, some feel it’s a good attempt, i feel it’s rather poor and misses the point. the best code is the one that's most easily understood and equally performant - some of the math here may go against that (given you can reduce
concentrations with helper functions and the like)
sheepscreek
Suggestion to include GPT 6.1 Sol (Low) since that is configured as the default. I’d be interested to see how well it fares against the others.
drivebyhooting
First of all, what happened to Max and Ultra?
Secondly, I’m mostly using Astra Ultra or Sol 6.1 when tight on quota. I pay $200/month and rarely if ever run out. Am I doing something wrong? Or perhaps I’m not that productive as you guys?
show comments
furyofantares
For most things I find medium to be the highest quality output. However if I want it to do spatial reasoning or if I want it be funny / write something (not code) of almost-passable quality, then xhigh is where it's at.
softwaredoug
IMO slop usually comes from overly defensive programming from LLMs. They’re loath to remove code and simplify, because unless we humans tell them some code / use case is unimportant, they have no reason to not care about it.
So agents can make these overly complex solutions trying to make bulletproof implementations and prevent regressions.
It requires a human to look at code, ask about complexity, and guide simplification by subtracting requirements implicit in the code.
show comments
theodorewiles
I have a simplify skill built from a rich hickey talk that basically solves the erosion and verbosity metrics SCB flagged (i.e. erosion and verbosity are way better and improve over time). However, after running the SCB benchmark I cannot find any evidence that it actually improves test pass rate trajectories. Food for thought on maintainability metrics.
thatslast
wondering how gemini 4 performs
jwpapi
Very interesting. If I’m honest with myself it could be very much a reality that I would be better off in only using for medium effor size tasked instead of trying to "offload" the necessary architecture thinking to "high" or even more.
They are good for inspiration though, but maybe I rather let me build a demo and then tell a medium model what I want from it.
giancarlostoro
GPT 5.5 on Medium is all I use lately, I don't understand why anyone would pay drastically more for slightly more, when Opus has been good enough for a year now to do most things.
show comments
mordae
This is interesting. Is nobody training LLMs to repeat resolving issues by patches from real projects where refactoring is required, judging the refactor similarity and not just tests passing?
0xbadcafebee
> The full benchmark is hard in a way that matters to me: the agents get the main behavior right much more often than they keep all tested behavior working as requirements change
So, it's like... software development? 90% of bugs come from unintended consequences of changes after the fact. There's a couple decades of design lessons people had to learn to keep this number low. The agents don't follow those lessons.
Good programmers rely on structure and reinforcement to prevent themselves from introducing bugs. Today that structure is a combination of software frameworks, SDLC gates, and system design patterns. For example, most any Python programmer can reliably churn out a Django app as a Microservice and be fairly sure it won't be that hard to make, or result in tons of unexpected bugs. The reason isn't that it's some perfect software design. It's that prescribed methods in themselves are bumper lanes, keeping you on track. A few of you may remember what programming was like before popular open source frameworks, or maybe you've worked on proprietary big-iron mega-projects. Left entirely to one's own devices, programmers can make a mess of things. Give them a formal structure and suddenly there's less to go wrong.
I wouldn't sit around waiting for models to become uber-programmers. Instead I'd treat 'em like humans, and enforce the structures we know have better results. You can try to impose good design as skills (https://github.com/ciembor/agent-rules-books) but the model will still do things its own way. That's why coding harnesses need to intervene and explicitly enforce the structure/framework/bumper lanes. Basically we need 1,000 different Pi extensions that will intervene as the agent codes in order to enforce the correct design. Otherwise we will forever be chasing some "perfect coding rlhf", and still the model will blunder into corners.
xlayn
what are the odds of as purely side effect of benchmaxing that the models get actually good because the benchmarkers the nittier they get the better the model has to become?
corrolary: just as cars evolved for worst because they were measured for "race numbers", what are the odds that the models just become "that uber nerd that degrades anyone who doesn't follow all his uber-rules for uber-code" and in the process start creating code that cannot be read or maintained?
Astra writes well (which is why I often like to work with it over Claude), but it tends to produce code golf that's borderline unreadable, like completely omitting line breaks. I thought I had something funky with my config, but it seems to be a known issue.
https://lucumr.pocoo.org/2026/9/7/astra-why/
EDIT: You can search for "tenobrus astra" on X, but link removed.
The generated code certainly does what's asked of it, but it's as sloppy as anything else I've seen. I don't think this benchmark can control for that. Maybe if it had a metric for minimum diff to satisfy the new requirement with the existing constraints on dead code etc?
It's Opus specific, but personally I'd never bother with Opus above High, CursorBench shows a clear inflection: https://cursor.com/cursorbench
Is there a reasonable way to run a representative subset myself?
Edit:
> For folks who want to try to repro these findings: I used the pinned v1.0 catalog, the stock just-solve prompt, seed 42, the any-case runner pass policy, and Docker with Python 3.12 and uv.
And https://github.com/gabeorlanski/scb-problems/
Great benchmark!
I believe effort can mean both "solve harder problems" and "produce more stuff". I mostly don't want more stuff than I asked for, so a capable model at lower effort can indeed be the wiser choice. Faster response too.
I can’t find this useful given they are trying to quantify wether the code ‘feels’ sloppy with some fuzzy math that can easily be manipulated. at least it’s an attempt, some feel it’s a good attempt, i feel it’s rather poor and misses the point. the best code is the one that's most easily understood and equally performant - some of the math here may go against that (given you can reduce concentrations with helper functions and the like)
Suggestion to include GPT 6.1 Sol (Low) since that is configured as the default. I’d be interested to see how well it fares against the others.
First of all, what happened to Max and Ultra?
Secondly, I’m mostly using Astra Ultra or Sol 6.1 when tight on quota. I pay $200/month and rarely if ever run out. Am I doing something wrong? Or perhaps I’m not that productive as you guys?
For most things I find medium to be the highest quality output. However if I want it to do spatial reasoning or if I want it be funny / write something (not code) of almost-passable quality, then xhigh is where it's at.
IMO slop usually comes from overly defensive programming from LLMs. They’re loath to remove code and simplify, because unless we humans tell them some code / use case is unimportant, they have no reason to not care about it.
So agents can make these overly complex solutions trying to make bulletproof implementations and prevent regressions.
It requires a human to look at code, ask about complexity, and guide simplification by subtracting requirements implicit in the code.
I have a simplify skill built from a rich hickey talk that basically solves the erosion and verbosity metrics SCB flagged (i.e. erosion and verbosity are way better and improve over time). However, after running the SCB benchmark I cannot find any evidence that it actually improves test pass rate trajectories. Food for thought on maintainability metrics.
wondering how gemini 4 performs
Very interesting. If I’m honest with myself it could be very much a reality that I would be better off in only using for medium effor size tasked instead of trying to "offload" the necessary architecture thinking to "high" or even more.
They are good for inspiration though, but maybe I rather let me build a demo and then tell a medium model what I want from it.
GPT 5.5 on Medium is all I use lately, I don't understand why anyone would pay drastically more for slightly more, when Opus has been good enough for a year now to do most things.
This is interesting. Is nobody training LLMs to repeat resolving issues by patches from real projects where refactoring is required, judging the refactor similarity and not just tests passing?
> The full benchmark is hard in a way that matters to me: the agents get the main behavior right much more often than they keep all tested behavior working as requirements change
So, it's like... software development? 90% of bugs come from unintended consequences of changes after the fact. There's a couple decades of design lessons people had to learn to keep this number low. The agents don't follow those lessons.
Good programmers rely on structure and reinforcement to prevent themselves from introducing bugs. Today that structure is a combination of software frameworks, SDLC gates, and system design patterns. For example, most any Python programmer can reliably churn out a Django app as a Microservice and be fairly sure it won't be that hard to make, or result in tons of unexpected bugs. The reason isn't that it's some perfect software design. It's that prescribed methods in themselves are bumper lanes, keeping you on track. A few of you may remember what programming was like before popular open source frameworks, or maybe you've worked on proprietary big-iron mega-projects. Left entirely to one's own devices, programmers can make a mess of things. Give them a formal structure and suddenly there's less to go wrong.
I wouldn't sit around waiting for models to become uber-programmers. Instead I'd treat 'em like humans, and enforce the structures we know have better results. You can try to impose good design as skills (https://github.com/ciembor/agent-rules-books) but the model will still do things its own way. That's why coding harnesses need to intervene and explicitly enforce the structure/framework/bumper lanes. Basically we need 1,000 different Pi extensions that will intervene as the agent codes in order to enforce the correct design. Otherwise we will forever be chasing some "perfect coding rlhf", and still the model will blunder into corners.
what are the odds of as purely side effect of benchmaxing that the models get actually good because the benchmarkers the nittier they get the better the model has to become?
corrolary: just as cars evolved for worst because they were measured for "race numbers", what are the odds that the models just become "that uber nerd that degrades anyone who doesn't follow all his uber-rules for uber-code" and in the process start creating code that cannot be read or maintained?
Related. Others?
Benchmarking Opus 5 on SlopCodeBench - https://news.ycombinator.com/item?id=49076391 - July 2026 (119 comments)
Wouldn't it be better to enforce pass via the harness and then measure cost/quality since that's more what people are likely to do?
Are the raw outputs published somewhere?
I am currently investigating SCB results and a larger, more representative dataset to work on would be really helpful!
Finally a serious benchmark!
why measure something you can just enforce with something like opcore which we have built? https://github.com/the-open-engine/opcore/
Opus did better on medium? Can anyone confirm this somehow? What’s the sweetspot in your opinion?
I wish a company named human layer won't write their reports with so much Claudisms.
not suprising. i think we also tend to do better at things when we only expend medium effort.
The irony is the article is very sloppy. Can we at least write it and review it properly..