I had a similar experience in search and found even holdouts can be overfit to. IE through brute force, it may not see the holdout, but if you gate a change on holdout acceptance it will land on a solution that’s overfit to it by somewhat random chance.
The other problem is that holdouts / data inaccessible to the agent isn’t easy to do in most coding agents. It’s not as simple as splitting training data 80% and giving some to the agent and hiding 20%. The agent can figure out where its data came from and find ways to reconstruct / cheat the holdout data.
All the ways of doing this seem annoying: ie having a second project that accepts / rejects changes.
I opted to just build my own harness for these things to avoid overfitting.
Fascinating article. I daily catch LLMs in “lies” like: “I found the root cause of the bug” or “this approach is twice as fast”. It’s hard to say what causes this uninformed certainty - is it intrinsic to being trained on human writing, or something that comes from the RLHF process afterwards, but it’s extremely annoying. It’s one thing to have a LLM make poor decisions, but it feels worse to have it “lie” to you in the process.
show comments
ouz-a
I stopped trusting benchmarks ever since LLMs started speaking alien like English, if I can't understand what they are saying how can I trust them.
show comments
stephantul
Unfortunately, even a holdout set doesn’t protect you from overfitting, it just takes longer.
Of course having a holdout set is better than not having one. It’s just not a silver bullet.
show comments
throwawayffffas
Not trying to defend anyone, but in my experience the latest models have been performing significantly better than 8 months ago. So, in my book the extent of over-fitting on benchmarks seems to be covering my use cases.
lavela
> it's become easier than ever to make serious performance gains
Is that true and if yes why? I was under the impression that it would become more difficult over time to make serious performance gains, which would also fit with reaching for benchmark hacking rather than relying on natural gains.
mppm
Cheating and overfitting, as discussed in the article, are the most obvious problems with benchmarking LLMs. But there is also the aspect that, at least for closed models, the tokens still have to be sent to the provider's servers for inference. This makes the holdout set not as held out as it may appear. OpenAI and Antropic probably don't care about your private set of regex benchmarks, but for the headline "closed" benchmarks, I'd be surprised if they haven't collected a nice representative set of "holdout" problems to be examined at leisure.
michalsustr
Nice article. As perf optimisation has become more like a machine learning problem, maybe there is opportunity to use other techniques for checking generalisation. Like e.g. cross validation
feverzsj
Maybe use fixed bugs from closed source to benchmark these "frontier" models.
akoboldfrying
I think an interesting direction for benchmarking is to take inspiration from metamorphic testing. Metamorphic testing is a way of extending property-based testing (in which you ask the test framework itself to automatically generate many random (input, expected output) pairs to test for you, instead of manually writing individual tests yourself) to handle situations where (a) it's hard to independently come up with the right answer for a specific given input, but (b) relationships between inputs imply checkable relationships between outputs. For example, if you're trying to test your own implementation of sin(), it's hard to automatically generate random (input, expected output) test pairs without using a separate, trusted implementation of the sine function, which may not be available; but one thing you can easily do is check, for many different random x, that sin(x) == -sin(x+180).
How to apply this idea to benchmarks? Basically, look for simple transformations of the input instances that should yield simple transformations of the outputs -- in particular, outputs that, in a non-overfitted implementation, should take the same length of time to compute. For regexes, you could rotate a subset of non-magic characters in both the string and the regex (e.g., A -> B, B -> C, ..., Z -> A).
Another example would be to reverse both the string and the regex (taking care to handle parenthesised regex subexpressions correctly) -- unlike the previous one, it's not expected that the transformed instance will take the exact same length of time, but there should not be too much blow up.
I had a similar experience in search and found even holdouts can be overfit to. IE through brute force, it may not see the holdout, but if you gate a change on holdout acceptance it will land on a solution that’s overfit to it by somewhat random chance.
The other problem is that holdouts / data inaccessible to the agent isn’t easy to do in most coding agents. It’s not as simple as splitting training data 80% and giving some to the agent and hiding 20%. The agent can figure out where its data came from and find ways to reconstruct / cheat the holdout data.
All the ways of doing this seem annoying: ie having a second project that accepts / rejects changes.
I opted to just build my own harness for these things to avoid overfitting.
https://softwaredoug.com/blog/2026/05/17/autoresearching-a-b...
Fascinating article. I daily catch LLMs in “lies” like: “I found the root cause of the bug” or “this approach is twice as fast”. It’s hard to say what causes this uninformed certainty - is it intrinsic to being trained on human writing, or something that comes from the RLHF process afterwards, but it’s extremely annoying. It’s one thing to have a LLM make poor decisions, but it feels worse to have it “lie” to you in the process.
I stopped trusting benchmarks ever since LLMs started speaking alien like English, if I can't understand what they are saying how can I trust them.
Unfortunately, even a holdout set doesn’t protect you from overfitting, it just takes longer.
Of course having a holdout set is better than not having one. It’s just not a silver bullet.
Not trying to defend anyone, but in my experience the latest models have been performing significantly better than 8 months ago. So, in my book the extent of over-fitting on benchmarks seems to be covering my use cases.
> it's become easier than ever to make serious performance gains
Is that true and if yes why? I was under the impression that it would become more difficult over time to make serious performance gains, which would also fit with reaching for benchmark hacking rather than relying on natural gains.
Cheating and overfitting, as discussed in the article, are the most obvious problems with benchmarking LLMs. But there is also the aspect that, at least for closed models, the tokens still have to be sent to the provider's servers for inference. This makes the holdout set not as held out as it may appear. OpenAI and Antropic probably don't care about your private set of regex benchmarks, but for the headline "closed" benchmarks, I'd be surprised if they haven't collected a nice representative set of "holdout" problems to be examined at leisure.
Nice article. As perf optimisation has become more like a machine learning problem, maybe there is opportunity to use other techniques for checking generalisation. Like e.g. cross validation
Maybe use fixed bugs from closed source to benchmark these "frontier" models.
I think an interesting direction for benchmarking is to take inspiration from metamorphic testing. Metamorphic testing is a way of extending property-based testing (in which you ask the test framework itself to automatically generate many random (input, expected output) pairs to test for you, instead of manually writing individual tests yourself) to handle situations where (a) it's hard to independently come up with the right answer for a specific given input, but (b) relationships between inputs imply checkable relationships between outputs. For example, if you're trying to test your own implementation of sin(), it's hard to automatically generate random (input, expected output) test pairs without using a separate, trusted implementation of the sine function, which may not be available; but one thing you can easily do is check, for many different random x, that sin(x) == -sin(x+180).
How to apply this idea to benchmarks? Basically, look for simple transformations of the input instances that should yield simple transformations of the outputs -- in particular, outputs that, in a non-overfitted implementation, should take the same length of time to compute. For regexes, you could rotate a subset of non-magic characters in both the string and the regex (e.g., A -> B, B -> C, ..., Z -> A).
Another example would be to reverse both the string and the regex (taking care to handle parenthesised regex subexpressions correctly) -- unlike the previous one, it's not expected that the transformed instance will take the exact same length of time, but there should not be too much blow up.