Somewhat related, the experiment ongoing at https://www.ecdsa.fail/ is fascinating: it's a competitive, leaderboard-style research challenge trying to optimise a quantum circuit for breaking ECDSA (specifically the elliptic-curve point addition in Shor's algorithm). It quickly surpassed a result announced by Google researchers last month. Now it's showing a 40% gain over Google's result.
show comments
cpard
I'm personally interested in this problem and it's a quite active research area right now.
My feeling is that the research is converging to what the paper claims, that the combination of two is the right way to do it and it's a matter of how you combine the two as part of the harness you built that makes the difference.
At the AID-Wild / ACM CAIS 2026 workshop that happened recently, there are plenty of examples in the accepted papers on that.
A great example is AI-PROPELLER: Warehouse-Scale Interprocedural Code Layout Optimization with AlphaEvolve. It uses AlphaEvolve and Vizier to evolve compiler code-layout heuristics. (https://arxiv.org/abs/2606.00131)
show comments
deerstalker
I have been doing some research on this topic, and found that for some budget regimes (really expensive objective function evaluations) and some applications (HPC code parameter autotuning), the frontier LLMs can even outperform classical optimizers. Even open-weight models can perform well on certain applications but one some they fail abysmally (Of course this is limited to a bunch of niche applications).
ekjhgkejhgk
Methodological flaw.
On Centaur (hybrid LLM + classic HPO) the LLM is only called to give its opinion a fraction r=0.3 of the time (the remaining is plain HPO). But that means that:
A) the compute used by Centaur is not directly comparable to the compute of the other methods. Centaur had the advantage the r was itself hyperparam-optimized with a cost that is not budgeted on the main graph. Centaur cheated by getting free compute under the table.
B) it's not even clear that the advantage of choosing r=0.3 is real and not noise. If you look at Figure 11, it's not clear that the stuff in between 0.1 and 0.5 isn't noise. It could well be noise. And if you believe the variation is noise and fit a line or a parabola to smooth out the noise, you'd conclude that the optimal is don't use an LLM, so it's not clear that the LLM contribution is even positive.
C) another reason why the LLM contribution doesn't look positive: again on Figure 11, how do you explain that r=0.8 is horrible? If the LLM is principled in some way, if it can reason through "I see such and such therefore I try such and such" then asking it more would mean that it can experiment more and exclude bad regions faster. And if there's no input for it to give, it could just accept "I'll use the optimizer's suggestion this time" over and over. Hybrid should always be strictly better than just classic, but in reality this is more false the larger the r.
Overall, I don't think the conclusion follows from the paper. However, as humans the idea that "reasoning + classic HPO should be classic HPO" is very appealing. I also like the idea of exposing the opimizer internals to the LLM.
show comments
woadwarrior01
Their centaur idea[1] is interesting and quite straightforward. It should be fairly easy to implement using a coding agent for the LLM and the ask-and-tell interface in pycma[2].
Honestly, the results kind of show the LLM is adding very marginal value. TPE crushes Karpathy’s autoresearch and it is neck and neck with the method in this paper, despite not needing to run any LLM inference at all.
I remember a few months ago people were fairly skeptical about autoresearch, but we didn’t have a ton of data to say it was better or worse. My own bias is to prefer cheaper methods unless the more expensive method is shown to be better.
ferreirafabio
Author here. Appreciate the interest in this line of work! Just wanted to share an extension of this work:
since the paper, I've extended the evaluation to more models (including newer Opus releases) and more seeds, and I'm posting ongoing results in a live tracker:
Somewhat related, the experiment ongoing at https://www.ecdsa.fail/ is fascinating: it's a competitive, leaderboard-style research challenge trying to optimise a quantum circuit for breaking ECDSA (specifically the elliptic-curve point addition in Shor's algorithm). It quickly surpassed a result announced by Google researchers last month. Now it's showing a 40% gain over Google's result.
I'm personally interested in this problem and it's a quite active research area right now.
My feeling is that the research is converging to what the paper claims, that the combination of two is the right way to do it and it's a matter of how you combine the two as part of the harness you built that makes the difference.
At the AID-Wild / ACM CAIS 2026 workshop that happened recently, there are plenty of examples in the accepted papers on that.
A great example is AI-PROPELLER: Warehouse-Scale Interprocedural Code Layout Optimization with AlphaEvolve. It uses AlphaEvolve and Vizier to evolve compiler code-layout heuristics. (https://arxiv.org/abs/2606.00131)
I have been doing some research on this topic, and found that for some budget regimes (really expensive objective function evaluations) and some applications (HPC code parameter autotuning), the frontier LLMs can even outperform classical optimizers. Even open-weight models can perform well on certain applications but one some they fail abysmally (Of course this is limited to a bunch of niche applications).
Methodological flaw.
On Centaur (hybrid LLM + classic HPO) the LLM is only called to give its opinion a fraction r=0.3 of the time (the remaining is plain HPO). But that means that:
A) the compute used by Centaur is not directly comparable to the compute of the other methods. Centaur had the advantage the r was itself hyperparam-optimized with a cost that is not budgeted on the main graph. Centaur cheated by getting free compute under the table.
B) it's not even clear that the advantage of choosing r=0.3 is real and not noise. If you look at Figure 11, it's not clear that the stuff in between 0.1 and 0.5 isn't noise. It could well be noise. And if you believe the variation is noise and fit a line or a parabola to smooth out the noise, you'd conclude that the optimal is don't use an LLM, so it's not clear that the LLM contribution is even positive.
C) another reason why the LLM contribution doesn't look positive: again on Figure 11, how do you explain that r=0.8 is horrible? If the LLM is principled in some way, if it can reason through "I see such and such therefore I try such and such" then asking it more would mean that it can experiment more and exclude bad regions faster. And if there's no input for it to give, it could just accept "I'll use the optimizer's suggestion this time" over and over. Hybrid should always be strictly better than just classic, but in reality this is more false the larger the r.
Overall, I don't think the conclusion follows from the paper. However, as humans the idea that "reasoning + classic HPO should be classic HPO" is very appealing. I also like the idea of exposing the opimizer internals to the LLM.
Their centaur idea[1] is interesting and quite straightforward. It should be fairly easy to implement using a coding agent for the LLM and the ask-and-tell interface in pycma[2].
[1]: https://github.com/ferreirafabio/autoresearch-automl/blob/ma...
[2]: https://github.com/CMA-ES/pycma
Honestly, the results kind of show the LLM is adding very marginal value. TPE crushes Karpathy’s autoresearch and it is neck and neck with the method in this paper, despite not needing to run any LLM inference at all.
I remember a few months ago people were fairly skeptical about autoresearch, but we didn’t have a ton of data to say it was better or worse. My own bias is to prefer cheaper methods unless the more expensive method is shown to be better.
Author here. Appreciate the interest in this line of work! Just wanted to share an extension of this work:
since the paper, I've extended the evaluation to more models (including newer Opus releases) and more seeds, and I'm posting ongoing results in a live tracker:
https://ferreirafabio.github.io/autoresearch-automl/#tab=tra...
TLDR so far: the centaur (LLM + classical optimizer) still wins.
[flagged]
[flagged]
[flagged]
TDLR: No.