Wow this is interesting to me from another angle. A friend of mine created a benchmark to evaluate LLMs on 9x9 Go (using KataGo opponents as Elo anchors) [1]. The relative rankings of this (StarSkirmish Bench v0.2) benchmark (GPT-6 Astra > GPT-6.1 Sol > Gemini 4 Argon > Claude Opus 5.5 > Claude Fable 5.1 > Claude Sonnet 5.5) matches the order of his GoBench rankings exactly, even though they are different problems at surface level.
Is there anything interesting to learn from the agent written bots? Or are they just inferior versions of the best bots because the time limit is so short? Is there any indication that they’re trained on the code from the human written bots (like, do they magically “guess” creating functionality that requires experimental feedback or domain experience?) How much are the bots on rails applying the same techniques every time?
I think those kinds of questions would be more interesting than the raw win/loss ratios.
show comments
djoldman
> ... runs each model in its lab's 1st party CLI harness at the highest reasoning setting it offers.
Anthropic (Claude models): Claude Code
OpenAI (GPT models): Codex
Google (Gemini models): Antigravity CLI
Meta (Muse models): Muse Code
> The CLIs' web search and web fetch tools are turned off, and the agent has no other internet access. The CLI is installed before the clock starts, and the hour counts from when the agent starts. There is no submit tool: the harness collects the bot when the time runs out.
One thing we're learning is that harness is HUGE. The difference in performance for the same underlying model can change dramatically across:
1. harnesses
2. allowed tools (no web search/fetch - how does the model refer to documentation?)
I thought Opus 5.5 performed worse across a lot of benchmarks on Max, would love to see it one notch down
Lucasoato
They don’t stand a chance against my cannon rush, I reached platinum with 10 APM and that strategy ahaha
BeaverGoose
They should benchmark it against Pluto. It's a artificial neutral network bot that can easily take games off the best broodwar players.
show comments
iugtmkbdfil834
I am not sure I like this benchmark, but mostly, because I am not sure what it is really evaluating. It is not an idle dismissal.. multiple factors seem to complicate what it is supposed to look at ( just top of my head ):
- code generation speed
- logic analysis
- understanding of the game platform and how use of C may influence it
- considered approaches, tactics and strategies
Don't get me wrong. It is is neat idea. I am just not sure what it measures.
show comments
t1amat
This is somewhat interesting to me as I’ve recently come to a different conclusion. I’ve been populating a ModernUO server with bots, and have found Opus 5.5 to be substantially better at understanding and building it. I actually started with 6.1 Sol and 6 Astra but pulled them after a week of little progress, had Opus 5.5 take over and it rebuilt everything properly in a night and has carried it forward quite a way since.
aiiotnoodle
Would love to see an LLM written bot play Red Alert 2/Red Alert.
I think with the abilities of AI improving, playing games would be a good benchmark as long as they're not trained on them.
m_w_
This is an interesting benchmark - I like that time is limited, as it often isn't in other benchmarks.
It would be interesting to see the reverse, how long does each model take to get >50% WR against bots of various tiers.
show comments
explodes
The video shows GLM but it's not in any graphs. Are there more complete charts anywhere else? Ones that include the 45 LLMs?
show comments
jstummbillig
How do they avoid the LLMs simply working with pre-training data? Which apparently they do, since the human written bots win. But it's a mystery, is what it is.
pelagicAustral
This kind of makes me think about making better bots for Counter Strike based on human top level players, which i think its a very undervalued category of software. You can even forget about about CS and name any FPS and you'll be hard pressed to find a game with seriously good bots.
show comments
qsort
It's beyond insane that Astra achieves above 50%. Are models allowed to "play against" the specific bot they're up against? What explains the fact that it drops to single-digits against strong bots?
show comments
donutdonot
I assume lots of bot code is part of the training data? So it's basically a version of how will they can improve upon those solutions public out there?
tweakimp
If you want to see human written bots in action or compete in the bot ladder yourself, try https://aiarena.net (SC2)
rdudek
Does anyone know if there was a project having LLMs play Cities Skylines and see how well it manages to design and build a city?
rafiyashaheen11
Oh my gosh! It feels so cool trying this that is played by LLMs. I am sooo eager to try argon 4 and build many robotic projects.
show comments
brap
What if we let the agents play directly instead of writing bots?
show comments
amishacreates
Benchmark vs intelligence which considers more?
caughtinthought
How do the best humans do against the LLMs?
show comments
xyst
No mention of actions per minute? Clearly the best indicator of performance missed in this experiment
0ddj0b
Any thoughts on building this for Starcraft 2?
aleksiy123
The site overall looks nice too. It’s fun not to see slop for a ui for this kind of project.
Cynddl
Once again, a page with a Pareto curve that is not a Pareto curve. Claude Haiku 5.5 is not on the curve, but inside. Seems like people don't really know what a Pareto curve is.
nozzlegear
This thing would beat me, but I'm ass at StarCraft.
Wow this is interesting to me from another angle. A friend of mine created a benchmark to evaluate LLMs on 9x9 Go (using KataGo opponents as Elo anchors) [1]. The relative rankings of this (StarSkirmish Bench v0.2) benchmark (GPT-6 Astra > GPT-6.1 Sol > Gemini 4 Argon > Claude Opus 5.5 > Claude Fable 5.1 > Claude Sonnet 5.5) matches the order of his GoBench rankings exactly, even though they are different problems at surface level.
[1] https://rolandgao.com/blog/gobench/
Is there anything interesting to learn from the agent written bots? Or are they just inferior versions of the best bots because the time limit is so short? Is there any indication that they’re trained on the code from the human written bots (like, do they magically “guess” creating functionality that requires experimental feedback or domain experience?) How much are the bots on rails applying the same techniques every time?
I think those kinds of questions would be more interesting than the raw win/loss ratios.
> ... runs each model in its lab's 1st party CLI harness at the highest reasoning setting it offers.
> The CLIs' web search and web fetch tools are turned off, and the agent has no other internet access. The CLI is installed before the clock starts, and the hour counts from when the agent starts. There is no submit tool: the harness collects the bot when the time runs out.One thing we're learning is that harness is HUGE. The difference in performance for the same underlying model can change dramatically across:
1. harnesses
2. allowed tools (no web search/fetch - how does the model refer to documentation?)
3. thinking levels
So this benchmark has the LLMs writing a bit to play StarCraft. Somewhat similar there was a post here a while back that had the LLMs playing StarCraft directly - https://broodbench.com/ and https://news.ycombinator.com/item?id=49766966
I thought Opus 5.5 performed worse across a lot of benchmarks on Max, would love to see it one notch down
They don’t stand a chance against my cannon rush, I reached platinum with 10 APM and that strategy ahaha
They should benchmark it against Pluto. It's a artificial neutral network bot that can easily take games off the best broodwar players.
I am not sure I like this benchmark, but mostly, because I am not sure what it is really evaluating. It is not an idle dismissal.. multiple factors seem to complicate what it is supposed to look at ( just top of my head ):
- code generation speed - logic analysis - understanding of the game platform and how use of C may influence it - considered approaches, tactics and strategies
Don't get me wrong. It is is neat idea. I am just not sure what it measures.
This is somewhat interesting to me as I’ve recently come to a different conclusion. I’ve been populating a ModernUO server with bots, and have found Opus 5.5 to be substantially better at understanding and building it. I actually started with 6.1 Sol and 6 Astra but pulled them after a week of little progress, had Opus 5.5 take over and it rebuilt everything properly in a night and has carried it forward quite a way since.
Would love to see an LLM written bot play Red Alert 2/Red Alert.
I think with the abilities of AI improving, playing games would be a good benchmark as long as they're not trained on them.
This is an interesting benchmark - I like that time is limited, as it often isn't in other benchmarks.
It would be interesting to see the reverse, how long does each model take to get >50% WR against bots of various tiers.
The video shows GLM but it's not in any graphs. Are there more complete charts anywhere else? Ones that include the 45 LLMs?
How do they avoid the LLMs simply working with pre-training data? Which apparently they do, since the human written bots win. But it's a mystery, is what it is.
This kind of makes me think about making better bots for Counter Strike based on human top level players, which i think its a very undervalued category of software. You can even forget about about CS and name any FPS and you'll be hard pressed to find a game with seriously good bots.
It's beyond insane that Astra achieves above 50%. Are models allowed to "play against" the specific bot they're up against? What explains the fact that it drops to single-digits against strong bots?
I assume lots of bot code is part of the training data? So it's basically a version of how will they can improve upon those solutions public out there?
If you want to see human written bots in action or compete in the bot ladder yourself, try https://aiarena.net (SC2)
Does anyone know if there was a project having LLMs play Cities Skylines and see how well it manages to design and build a city?
Oh my gosh! It feels so cool trying this that is played by LLMs. I am sooo eager to try argon 4 and build many robotic projects.
What if we let the agents play directly instead of writing bots?
Benchmark vs intelligence which considers more?
How do the best humans do against the LLMs?
No mention of actions per minute? Clearly the best indicator of performance missed in this experiment
Any thoughts on building this for Starcraft 2?
The site overall looks nice too. It’s fun not to see slop for a ui for this kind of project.
Once again, a page with a Pareto curve that is not a Pareto curve. Claude Haiku 5.5 is not on the curve, but inside. Seems like people don't really know what a Pareto curve is.
This thing would beat me, but I'm ass at StarCraft.
why protoss?
[flagged]