Academically a - very - interesting article, which confirms my ongoing suspicion of LLM Benchmarks in general ( probably why I like it so much). Another thing that caught my eye is DeepSWE having a bunch of bugs in them. The more I learn about these benchmarks the more sceptical I become. I would expect such a prolific benchmark to have these bugs ironed out by now.

The practical takeaway, however, seems to be an interesting "don't use max mode". The passage about deepseeks weird API behavior dropping to pretty much zero the moment you used high says a lot.

Also, personally, even though Opus is clearly so much ahead of the Open Source models after the changes, I think the adjusted score is still very much competitive. I see other commenters using this as proof to justify why their opinions of the open source models - but a drop from 74% to ~50% is, while a big drop, still about a ~50% pass rate afterwards. That's far from useless for coding, especially since DeepSWE seems to be such a difficult benchmark. It simply seems to more accurately reflect the capabilities of these models.

All in all, I'd love to see more of such articles, and will probably keep an eye on this one - if they continue "benchmarking" these models this way.

This is a great example of how to test how well a benchmark score generalises. The first order approximation is not complicated: just vary a parameter and see how much the score changes.

Which parameter to vary depends on the result of the test, but the MaxxBench route of varying the task spec seems like a good default, though not always the most informative.

In one of my private benchmarks I realised many of the less capable models didn't fail because they submitted the wrong answer, but because they exceeded their budget for number of tool calls they were allowed. The more capable models never got close to the tool call limit because they tended toward combining tool calls (e.g. chaining multiple shell commands in one Bash tool call).

Was my benchmark actually measuring ability to combine tool calls rather than reasoning ability on the problem at hand? Well, easy enough to find out: re-run under a different tool call budget and see how much the scores change.

(Next problem: is ability to combine tool calls a useful thing to measure, i.e. should it confer an advantage in the benchmark? It could be if it is a sign of planning multiple steps ahead.)

The big trend in that graph is that smaller models collapse the most. Everything with "-flash" in the name, plus OpenAI Luna.

Look at GPT 5.6-Sol vs. GPT 5.6-Luna in particular.

(The GLM pair shouldn't be expected follow this pattern, because they're misnamed: GLM-5.3 and GLM-5.3-flash are distinct lineages. The -flash is an early release of a new pretrain on a new architecture).

If this *is* the main factor, I'd expect Kimi K3.1 and Gemini Carbon would also do well on this test (unlike Gemini-flash); while Haiku 6 would collapse (breaking the trend of the other recent Claude models). It's a falsifiable prediction.

The DeepSeek 4.1 Flash behavior of circumventing rules and restrictions is one I have observed several times. It doesn’t seem to be malicious or deliberate, but if you have a config problem in the harness or you block something it thinks it needs to accomplish its task, it will get very creative to circumvent it.

I run a lot of agents headless on tasks. They are pretty well sandboxed, but I would definitely suggest a regular audit script to go through the transcripts. Most of DeepSeek’s “creativity” has shown problems in my sandbox setup which get fixed (e.g. I had restricted something the agents actually needed), but some of it is actually worrying.

In an opposite turn, Kimi 3.1 follows rules much better (actually too literally sometimes). K3 was a real digger when trying to benchmark it if it got a sense it was being tested. 3.1 comments about 5x more that it thinks it’s being tested (and it’s right), but unlike K3 it doesn’t seem to do anything about it if it can’t get at something. I would love to see the authors do K3 and 3.1 in this because the 3rd party benchmark gains are large, but these two are behaving like different models for me rather than a point release.

My understanding is this is less about labs training directly on the benchmark as about the labs being sloppy about creating RL environments that reflect the benchmark task

They should be doing what this test did, and generalizing and significantly varying the training environments (and ensuring some of them are unexpectedly complex and unexpectedly simple).

Further I suspect the smaller models are especially overfitting to these sorts of simpler RL envs (benchmarks tend to be much simpler than the hardest RL env used today) because they are more likely to be scoring at ~20-40% success rate (which is sort of what you want to get good training signal when doing RL runs).

I feel more and more we need to move to a world where it’s easy to build and maintain our own benchmarks based off our own code base. I know it’s feasible, and I nor where I work have done it, but it’s funny that we trust benchmarks on tasks we basically have no idea what they do (not for lack of transparency) or how they translate to work I’m actually doing

And several companies to do just this exist (eg Superconductor, Fullbeam) - does anyone use them and find them valuable? And curious if you’ve noticed an improved change in your agents work as a result, and if over time you see yourself switching models ? Or is it more of a one time realization that model family X is best for your code.

Particularly interesting cases here are of GLM 5.3 and Claude Opus that seem to rely on assumptions despite clearly stated requirements.

> Let me just do float64; the risk of a test with >2^53 numbers is minimal. > such cases are unlikely … Truncation toward zero is fine.

Apart from this, we also a scenario where the models tried to pass benchmark specific tests by believing a parameter that the model printed.

> Seven models reward-hacked a test that believed a metric from the program itself.

DeepSeek also resorted to making external model calls via OpenRouter instead of doing the work itself. This suggests that even though the SOTA models are smart enough to compute, they either have an inbuilt bias towards achieving scores or the low/medium effort gives them the freedom to fetch information instead of computing it. With high effort, the models focus on computing the values but the token calls get exhausted.

Difficult to find a balance amongst compute, calls and cost.

One thing to consider is that even when some model has big gap between those two benchmark it doesn't mean it is much worse. It could be worse becasue he has assuptions based on training but on real scenario it wouldn't have it so it still could achive great results

I'm surprised by four things:

(1) An established framework for model evaluation runs the inference call with API key inside the isolated envionment the agent harness has full access to. Especially when it's intended to be an offline environment, that seems kinda... dumb. The language model itself is not dangerous -- it's only text. Inference is fine to do outside the isolated environment. It's only the harness' tools that can have effects on the enironment, so those need to be sent to the sandbox.

(2) Only a small set of language models exploited this! I run private benchmarks a lot and I know how much the language models try to anticipate grader behaviour when the task is bechmark-shaped. I would have thought more of them would request the answer from the web.

(3) It takes someone else to mutate the task specification. When I'm looking at an older iteration of one of my private benchmarks, the first thing I do is add a small twist to it to see how much of a passing grade is learned/anticipated. Seems like a cheap and easy coalmine canary that's in the interest of the evaluator.

(4) MaxxBench did not try to dress some DeepSWE tasks in clothes that make them look less benchmarky. Their purpose seems to be answering "how well does DeepSWE enchmark scores generalise?" Part of the generalisation error is memorised solutions, yes, but another part is that models emit different behaviour when they suspect they are in a benchmark. By being careful to not leak the benchmark nature of the task, models spend a lot less time reasoning about the behaviour of the grader and this is likely to produce different performance on the mutated task. It would be interesting to know in what direction that effect points, because that is the performance people get out of the model in real situations. (In my experience, hiding the benchmark nature of the task can produce improved performance on both the original and the mutated task, but particularly the latter.)

How is it different from SlopCode Bench? Isn't this a change in spec after the first prompt as in SlopCode Bench?

Great job, this mirrors my experience running private evals nearly 1:1, down to even GPT models post Spud pre-train (so anything starting with GPT-5.5 to now) underperforming [0] [1] what one would expect based on public evals, the second one runs them privately. Also, Anthropic models have rarely under and often over-performed in my evals focused on task adherence, task evaluation, task inference and tool calling.

Love having a clear, concise and actually based in tangible information post to point to next time someone tells me that me models like Deepseek, Qwen and Grok massively underperforming what the public numbers show means I must simply not be capable enough to use them properly, even when it comes to tasks that Opus 4.5 managed but deviate from what most public evals cover.

Maybe it's not me obviously being biased against Chinese entities (Deepseeks papers are glorious and I am grateful for them being public, I also very highly rated Kimi K2 quite some time before that became more mainstream). Maybe I don't suck at prompting/context engineering/software factories/whatever this weeks hype packaging of basic dev practices is. Maybe LLMs aren't "all basically the same, only getting slightly better but never worse" even though I can get bloody GPT-6.1 Sol using the same prompt to make destructive, irrecoverable mistakes that GPT-5.0 didn't. Maybe that 300+tps inference provider actually does cut corners to get to that number and the tool calling errors and inflated tokens per task to rectify those don't actually safe money over a more reliable model, hosted by a serious, on paper more expensive, inference provider (use Deepseek if your evals show that it suites, but also eval the same model between providers, faster and/or cheaper can and often is a lot slower, less accurate and even more expensive if you take a step back).

Hope that this will finally get people to accept that there is massive training to the test by certain labs and that labs like Anthropic have shown for years that, no, this isn't inevitable and one can build models that are not just able to match but even outperform private evals [2]. Whether this is intentional or unintentional "reward hacking" (like "hallucinations", a wildly unfitting term considering is actually happening), Anthropic has shown repeatedly and over many generations, that putting in the effort, one can prevent "reward hacking"/"accidental poisoning of training data with evals"/whatever the labs excuse is.

Same thing with the Pelican SVG, shared examples often enough of that being "benchmaxxed" by Qwen and Grok [3], which perform poor on similar requests without a bike or pelican. Not sure whether it is more troubling that some labs are so insecure as to do this or that we, despite the evidence, still act like this isn't the case, but that's life.

[0] https://news.ycombinator.com/item?id=49739490

[1] https://news.ycombinator.com/item?id=49577678

[2] https://news.ycombinator.com/item?id=45597901

[3] https://news.ycombinator.com/item?id=48951229

I bet we will find this is mostly that smaller models have issues to go against: 1. strong priors that are inverted here in the changes of the benchmarks. 2. underspecified specs for which hidden tests test things not clearly stated, but that very large models can guess knowing they are tested.

This is perfect. Real programming has a ton of these instances of minor changes.

DeepSWE was released on May and Deepseek v4 flash was released on April. By definition it couldn't be benchmaxxed on this particular benchmark. My guess is the mutation is not like real world and it is tripping small models, not benchmaxed model.

I liked the GLM models until 5.3 and this site seems to explain perhaps some of the reason why.

This is why I keep saying Chinese models are nowhere as good as their benchmark scores suggest, and MaxxBench seems to be very compelling evidence of that.

Every time a Chinese lab releases a new model, I give it a solid try that same weekend and drive it hard by giving it the exact same tasks that I give to Claude/Codex models. And the new releases all have the same issues: they start out strong, but start deviating after about ~30 minutes and I quickly find myself in a quagmire where the model makes mistake after mistake while fixing its previous mistakes. Sometimes these are minor. Occasionally they can be pretty bad. Like, would-cause-data-loss-if-merged bad.

Ultimately I also reject the notion that Chinese labs are only a few months behind US labs. We haven't had an actual Fable-level model come out of China. AFAIK only Kimi 3 was claimed to come close, and then it turned out that not only had they heavily distilled Anthropic models, but, unbeknownst to their users, they were routing some percentage of their requests to Fable.

Funny, I was just talking about this on HN a couple of days ago[0..2] and had a thread full of anklebiters telling me I'm using Luna wrong. It's clear that the vast majority of people never privately benchmark, because anyone who does knows models like Luna aren't remotely close to the performance the published benchmarks claim, immediately failing when even a single unexpected specification is introduced, no matter how trivial it is, exactly like Maxxbench's "filename must be lowercase". It's nice to have a public one to point to, but unfortunately every time you publish your private benchmarks they become targets for the next round, and as soon as the next generation of models releases you're back to arguing with people who take benchmaxxed results at face value.

[0] https://news.ycombinator.com/item?id=50373961 [1] https://news.ycombinator.com/item?id=50374111

> Most models tried pleasing an imagined grader over the prompt

>When we read the trajectories, we saw that many models had figured out they were in a benchmark and started second-guessing the prompt. Instead of just building what we asked for, they started coding to the tests they thought the grader would run.

I ran into the same problem on my English exams.

The fact that models detect being benchmarked and change behaviour is a failure in itself. Reminds me VW scandal with emission rules.

But what really caugh my attention is DS4 calling the openrouter with it's key. That's pretty bad for both the model and the evaluation sandbox.

Alignment? What alignment lol

I’ve been noticing that although Astra performs better on tasks with well-defined tasks with clearly outlined definition of done(Claudis/Neuralese is invading our language indeed!), it doesn’t follow instructions well and often ignores instructions defined in text which is something you’d expect from an LLM. On the other side, GPT 6.1 Sol, while performing worse than Astra on benchmarks, is way better at following instructions and in vibe-tests on certain domains. I suspect Astra is more focused on “agentic” tasks, and is not entirely supposed to pay too much attention to the instructions of a meat bag?

There is always a huge discrepancy between benchmarks and real world capabilities.

For my use case deepseek 4.1 flash is almost as dumb as haiku 5.5. Both are fast garbage producers.

It's a pity it doesn't show GPT 6.1 Sol and Terra or 6 Luna.

So basically it turns out the good models are good and the benchmaxxed models are benchmaxxed.

Edit: another comment pointed out it's probably not benchmaxxing due to the chronology.

https://news.ycombinator.com/item?id=50406948

But why should minor changes to the task cause catastrophic performance drops? How is that not evidence of benchmaxxing?

HN hug of death victim?

Two brothers created a benchmark, family is so proud.