> We use a bash-tool-only agent harness to evaluate 89 Terminal-Bench 2.1 tasks from the official repository, where resources are capped at 6 CPU cores and 8GB RAM.
This disqualifies the results. Each terminal bench task has a cpu upper limit and RAM upper limit. Overriding either is disqualification.
For reference, in tbench-2.1,
1. 0 out of 89 task allow 6 cpu cores (highest is 4, and i think only 1 task)
2. 8 out of 89 tasks allow 8GB RAM
This kind of shady benchmarking (I was talking about it just yesterday in a different context https://news.ycombinator.com/item?id=48838212) takes all joy out of building a harness to improve benchmark performance of a model because no matter what you do, you won't beat the headline (cheating) number. This is presumably why this model is not in the official benchmark leaderboard https://www.tbench.ai/leaderboard/terminal-bench/2.1
As an ex Meta employee, this is a little sad but not massively surprising. 'Number go up' is the core performance evaluation metric until PSC is done and you move on.
show comments
simonw
I had a few days of preview access, which was long enough to put together a plugin for LLM. You can try the model out in the terminal like this:
uv tool install llm
llm install llm-meta-ai
llm keys set meta-ai
# paste API key here
llm -m meta-ai/muse-spark-1.1 "Generate an SVG of a pelican riding a bicycle"
Maybe Zuck should double down on his "spoiler" role with models rather than compete head-to-head.
He doesn't have to match Anthropic or OpenAI model revenue if he can deflate theirs by 99%.
All he has to do is keep spending a few billion dollars developing frontier models, release them as open weights, and turn coding models into a commodity. He also needs a good OSS reference harness to match. Very few people are in a position to do this and for it to make business sense.
That's quite likely where things are headed regardless, and he could speed it up significantly.
We should all hope models move from proprietary products to commodities the way compilers did.
This may be one of the best things Zuck could do for the world.
Tiberium
The pricing is insane: $1.25/$4.5 for 1M tokens, and $0.15 for cached input!
Interesting how the prevalent opinion until yesterday seems to have been that OpenAI & Anthropic are irreversibly ahead and now with xAI and Meta at least delivered something that's competitive with useful models and cheap too. Granted, the narrative that the two leading labs are ahead still holds with Fable (and perhaps an upcoming GPT6), but it's not as over as common knowledge by the opinion leaders would have us believe.
show comments
kilroy123
I personally do not like Meta, but I'll say this. The more competition, the better for regular consumers. (Enterprise too)
- Chinese models
- Grok
- Meta
- Google
- OpenAI
- Anthropic
I think this is a win. I'm building like crazy to take advantage of all these subsidized tokens while I can.
show comments
paxys
How is every company able to show itself at the top of every benchmark?
show comments
EgregiousCube
Their published benchmarks seem to indicate that it's pretty good at coding and multimodal, but VERY good at successful tool calls.
What kind of use case would be best for that shape?
show comments
fmind-dev
Glad to see Meta back on track! Users will benefit greatly from this competition.
bel8
It seems to trade blows with GPT 5.5 and Opus 4.8 in performance while being cheaper than GLM 5.2.
eugene3306
> Model API is not available in your region.
:(
Well, Vietnam is not in the list of restricted territories.
Anyway, what is "your region" ?
Is this where I am now, or is it where I activated my Oculus 2 five years ago ?
show comments
carimura
I missed the fact that Meta was developing and releasing closed-weights models... bummer. Would be great to see some more progress with American open-weights models.
arizen
I'm still confused is it available to public via some sort of subscription?
redox99
Very strong pricing, cheaper than Grok 4.5, particularly the cached reads. We'll have to wait to see if it's actually worth using (it's not on OpenRouter yet).
show comments
whinvik
Why are the plans and pricing for all these products so complicated.
I don't know where I need to sign up to try it out. What is pricing? Is it API or subscription, what?
I had the exact same experience with Grok 4.5 as well.
qpricjalcbeu
Yeah, no thanks. I cannot think of a worse company to trust with additional personal data.
show comments
zmmmmm
Good to see Meta finally back to releasing something at least worth evaluating. And it sounds like they did at least a bit skate to where the puck is going by focusing on tool and computer use.
Jcampuzano2
Competition for cheaper and efficient models is a good thing, regardless of if you don't like SpaceX, Meta, etc. Especially from US based labs
I for one am really glad to get competitive models that will push the major labs to bring prices down. While Chinese open source labs are also great, unfortunately when it comes to US/Western political pressure it won't often have as much of a bearing on labs bringing prices down, especially for enterprises.
Also if these numbers are true, this is truly breaking ground finally for Meta.
show comments
NitpickLawyer
How are people trying this? I don't see it on openrouter. Any ways of testing this without subscribing to meta stuff?
show comments
anthonypasq
Everyone has been loving to shit on the Alexander Wang acquisition but this seems legitimately impressive to me?
Meta's AI org when from a total mismanaged dumpster fire for multiple years to delivering a competitive model in less than a year on essentially their first try?
show comments
lnenad
Considering the DeepSWE result (imho if you're gonna give value to benchmarks this is one of the best) it's not good enough.
show comments
chvid
Interesting that neither meta nor xai chose to do open source given that they are both clearly behind Google, OpenAI and anthropic - and a serious us open source offering would give them a clear foothold.
show comments
jedisct1
Not opensource.
phillipcarter
My trust factor is gone with Meta right now. Has there been any independent analysis to confirm they didn't cheat on benchmarks again?
show comments
frangonf
Is this the model trained on Meta "draftees"? Are we seeing this in the jump on JobBench?
minraws
Tried to get access to the API, apparently the model API is not available in my region...
I have questions regarding if I should even care but I don't so Meta please keep enjoying the irrelevance. lmao
show comments
IshKebab
Haha their demo is AI spamming restaurants on Instagram. This is going to go really well.
guluarte
A lot of these benchmarks are unfamiliar. Are labs just choosing the ones that make them look best?
zb3
This is not open-weights, right?
show comments
greenavocado
Meta is back in the game, albeit not at the top. Impressive stuff, nonetheless.
Lot more details in the linked report https://ai.meta.com/static-resource/muse-spark-1-1-evaluatio...
From Terminal-bench-2.1 details,
> We use a bash-tool-only agent harness to evaluate 89 Terminal-Bench 2.1 tasks from the official repository, where resources are capped at 6 CPU cores and 8GB RAM.
This disqualifies the results. Each terminal bench task has a cpu upper limit and RAM upper limit. Overriding either is disqualification.
For reference, in tbench-2.1,
1. 0 out of 89 task allow 6 cpu cores (highest is 4, and i think only 1 task)
2. 8 out of 89 tasks allow 8GB RAM
This kind of shady benchmarking (I was talking about it just yesterday in a different context https://news.ycombinator.com/item?id=48838212) takes all joy out of building a harness to improve benchmark performance of a model because no matter what you do, you won't beat the headline (cheating) number. This is presumably why this model is not in the official benchmark leaderboard https://www.tbench.ai/leaderboard/terminal-bench/2.1
As an ex Meta employee, this is a little sad but not massively surprising. 'Number go up' is the core performance evaluation metric until PSC is done and you move on.
I had a few days of preview access, which was long enough to put together a plugin for LLM. You can try the model out in the terminal like this:
Here's the result: https://tools.simonwillison.net/markdown-svg-renderer#url=ht...For comparison, here's the pelican I got from Muse Spark 1: https://simonwillison.net/2026/Apr/8/muse-spark/
Maybe Zuck should double down on his "spoiler" role with models rather than compete head-to-head.
He doesn't have to match Anthropic or OpenAI model revenue if he can deflate theirs by 99%.
All he has to do is keep spending a few billion dollars developing frontier models, release them as open weights, and turn coding models into a commodity. He also needs a good OSS reference harness to match. Very few people are in a position to do this and for it to make business sense.
That's quite likely where things are headed regardless, and he could speed it up significantly.
We should all hope models move from proprietary products to commodities the way compilers did.
This may be one of the best things Zuck could do for the world.
The pricing is insane: $1.25/$4.5 for 1M tokens, and $0.15 for cached input!
https://dev.meta.ai/docs/getting-started/pricing-rate-limits
Interesting how the prevalent opinion until yesterday seems to have been that OpenAI & Anthropic are irreversibly ahead and now with xAI and Meta at least delivered something that's competitive with useful models and cheap too. Granted, the narrative that the two leading labs are ahead still holds with Fable (and perhaps an upcoming GPT6), but it's not as over as common knowledge by the opinion leaders would have us believe.
I personally do not like Meta, but I'll say this. The more competition, the better for regular consumers. (Enterprise too)
- Chinese models
- Grok
- Meta
- Google
- OpenAI
- Anthropic
I think this is a win. I'm building like crazy to take advantage of all these subsidized tokens while I can.
How is every company able to show itself at the top of every benchmark?
Their published benchmarks seem to indicate that it's pretty good at coding and multimodal, but VERY good at successful tool calls.
What kind of use case would be best for that shape?
Glad to see Meta back on track! Users will benefit greatly from this competition.
It seems to trade blows with GPT 5.5 and Opus 4.8 in performance while being cheaper than GLM 5.2.
> Model API is not available in your region.
:(
Well, Vietnam is not in the list of restricted territories.
Anyway, what is "your region" ?
Is this where I am now, or is it where I activated my Oculus 2 five years ago ?
I missed the fact that Meta was developing and releasing closed-weights models... bummer. Would be great to see some more progress with American open-weights models.
I'm still confused is it available to public via some sort of subscription?
Very strong pricing, cheaper than Grok 4.5, particularly the cached reads. We'll have to wait to see if it's actually worth using (it's not on OpenRouter yet).
Why are the plans and pricing for all these products so complicated.
I don't know where I need to sign up to try it out. What is pricing? Is it API or subscription, what?
I had the exact same experience with Grok 4.5 as well.
Yeah, no thanks. I cannot think of a worse company to trust with additional personal data.
Good to see Meta finally back to releasing something at least worth evaluating. And it sounds like they did at least a bit skate to where the puck is going by focusing on tool and computer use.
Competition for cheaper and efficient models is a good thing, regardless of if you don't like SpaceX, Meta, etc. Especially from US based labs
I for one am really glad to get competitive models that will push the major labs to bring prices down. While Chinese open source labs are also great, unfortunately when it comes to US/Western political pressure it won't often have as much of a bearing on labs bringing prices down, especially for enterprises.
Also if these numbers are true, this is truly breaking ground finally for Meta.
How are people trying this? I don't see it on openrouter. Any ways of testing this without subscribing to meta stuff?
Everyone has been loving to shit on the Alexander Wang acquisition but this seems legitimately impressive to me?
Meta's AI org when from a total mismanaged dumpster fire for multiple years to delivering a competitive model in less than a year on essentially their first try?
Considering the DeepSWE result (imho if you're gonna give value to benchmarks this is one of the best) it's not good enough.
Interesting that neither meta nor xai chose to do open source given that they are both clearly behind Google, OpenAI and anthropic - and a serious us open source offering would give them a clear foothold.
Not opensource.
My trust factor is gone with Meta right now. Has there been any independent analysis to confirm they didn't cheat on benchmarks again?
Is this the model trained on Meta "draftees"? Are we seeing this in the jump on JobBench?
Tried to get access to the API, apparently the model API is not available in my region...
I have questions regarding if I should even care but I don't so Meta please keep enjoying the irrelevance. lmao
Haha their demo is AI spamming restaurants on Instagram. This is going to go really well.
A lot of these benchmarks are unfamiliar. Are labs just choosing the ones that make them look best?
This is not open-weights, right?
Meta is back in the game, albeit not at the top. Impressive stuff, nonetheless.