This is bonkers good if these benchmarks hold up in real-world scenarios. To be at the level of Qwen3.8-27B with a looped transformer model is quite the flex. They just left this note on reddit:
Hope that folks find this model more amenable to local usage - the sizing was largely informed by what folks on this subreddit ask for the most haha!
There are two particularly interesting aspects of this model for local users:
We do our training end-to-end in nvfp4, and although we haven't exhaustively benchmarked this, folks like Unsloth have noticed that our models seem to respond very well to being quantized, maintaining 90% top-1 even at at 1-bit quantization.
This model is looped, which allows more depth at the same parameter count, which we particularly notice seems helpful for coding tasks. :)
This is our first model targeted at local users, so let us know how this model is and anything you want to see out of future iterations!
Seems pretty promising in a few quick coding tests. 27 tokens/s decode on Ryzen 395 and llama.cpp Vulkan. Besides the obvious unmerged PRs (32694, 32657, 32429, 32704) you'll want below tiny patch for about 30% speed boost.
Also, if anyone from the team is reading, if you have access for researchers I'd be interested in doing some higher volume tests & harness tuning than I have hardware for personally.
--- a/src/llama-graph.cpp
+++ b/src/llama-graph.cpp
@@ -2670,10 +2670,12 @@ ggml_tensor \* llm_graph_context::build_moe_ffn(
const float beta = hparams.situ_beta;
const float lb = hparams.situ_linear_beta;
- ggml_tensor \* act = ggml_scale(ctx0, ggml_tanh(ctx0, ggml_scale(ctx0, cur, 1.0f/beta)), beta);
+ // gate/up are strided views of the fused gate_up output; ggml-vulkan's SCALE
+ // needs a contiguous src, so without the cont both scales fall back to the CPU
+ ggml_tensor \* act = ggml_scale(ctx0, ggml_tanh(ctx0, ggml_scale(ctx0, ggml_cont(ctx0, cur), 1.0f/beta)), beta);
act = ggml_mul(ctx0, act, ggml_sigmoid(ctx0, cur));
if (lb > 0.0f) {
- up = ggml_scale(ctx0, ggml_tanh(ctx0, ggml_scale(ctx0, up, 1.0f/lb)), lb);
+ up = ggml_scale(ctx0, ggml_tanh(ctx0, ggml_scale(ctx0, ggml_cont(ctx0, up), 1.0f/lb)), lb);
}
cur = ggml_mul(ctx0, act, up);
cb(cur, "ffn_moe_situ", il);
show comments
lilytweed
Whoa, not just a model that acknowledges and embraces using a looped architecture (GPT Astra 6+ and Sol 6.1+ are only [heavily] rumored to be looped), but it's open weights, too?
I'm willing to wager that most open models will go looped in the wake of this.
Wonder what it means for safety and interpretability at this level of availability.
show comments
dgacmu
M1 Pro Macbook, 32GB: About 10t/s using q4 xl while running browsers/etc. and interacting using the local browser.
test 1: write a python program to sort a csv by lastname and print out fields.
Result: Nailed it. Nice python program. Nice explanation of output, if a little chatty (it gave an example output, which I didn't ask for).
test 2: my standard "identify the type and year of this coin" test. Failed miserably, I gave up after 12,000 tokens of it babbling incessantly about possible misinterpretations of a silver half-dollar. Unsurprising - most small models suck at this test - but slightly annoying it seemed to get in a loop.
Pretty good performance and result for the (very simple) code generation problem, though. One of the first models that seems OK running side-by-side with a browser on my older MBP.
show comments
ramshanker
I am getting Stable Diffusion level of breakthrough vibe here. And such looped transformer SMALL models are best suited to embed directly into silicon/transistor. So you get better than ASIC performance by just looping over the hard-coded silicon. Just like we had a breakthrough with "deep" neural network, soon we may have something similar with "deep" loop over the same set of weights. Imaging transistor level looping 128 times. Extreme power efficient, and god knows what level of intelligence is achievable. I could very well foresee purchasing USB Intelligence Devices with some standardised driver protocol. You just buy one off Amazon, throw away the old one and get an Intelligence Upgrade.
Quite an interesting model, but the second of my test prompts in "extra high" thinking in Unsloth really underscore what they say about knowledge here:
> While looping improves the model’s reasoning, it doesn’t enable its weights to store more facts—knowledge is mostly a function of total parameter count. Inkling 2 Mini’s score on AA-Omniscience, a measure of factuality, is in line with other models of the same size [...] but knowledge-intensive benchmarks doesn’t improve with more passes...
> Inkling 2 Mini overcomes the knowledge gap by efficiently using search and tools.
I tested the Unsloth Q4 version.
My prompt asked it to write a singleton class, single file plugin for WordPress that hooks user login and stores last login time as metadata. It has some code style requirements. The prompt also asked it to make no tool calls so it didn't search and only used its knowledge.
The model burned through 19200 thinking tokens on this task (which was the hard limit I had set) and didn't finish. Its thinking suggests it was going to make at least one mistake.
Both Qwen 3.6 35B and Gemma 4 26B can cope with this task, from memory, in far fewer tokens (even taking into account Qwen's tendency to loop in thinking).
Rerunning the same prompt with tool use allowed, it searched with browser use, pretty effectively, and got to a solution, but it still made the mistake the thinking traces suggested it would earlier (wrong hook name). Other than that it wrote good code.
But 20K-ish tokens in, I could then not get it to search to make the required correction — it would repeatedly EOT rather than actually run the tools it was saying it would call. Whether this is the model failing or Unsloth's harness and the model disagreeing, I don't know. Quitting and reloading the model to continue the conversation didn't help.
It worked through my usual MySQL-5.0-compatible max score per category prompt nicely, though it didn't volunteer the suggestion of adding an index for speed, but it doesn't outperform LFM 2.5-24B-A2B, which is a non-looping model of similar size. It does work through SQL in its thinking traces quite nicely, though.
Car wash problem got a correct answer quite quickly, albeit with a surreal caveat (why do they always seem to need to offer a caveat, no matter how weird)
So far, I think I would run Muse Glimmer or Nemotron Bolt over this; Muse Glimmer would output tokens more slowly but still get to a result much more quickly.
show comments
chriddyp
Very, very excited about models of this size. Small enough to run on a Mac Mini!
Ran my 43 question data analytics exam on it against OpenRouter & my M4 Pro 64GB Mac Mini via llama. It scored a "B" on the exam [1], same score as DeepSeek V4.1 Flash and Sonnet 5.
The main limitation that I saw was a "runaway loop" when asked to find a issues in a dataset: it worked for over 3 hours on my Mac Mini, taking over 180 tool calls to answer the question before I cut it off.
It's the most accurate U.S. open weight model that fits on a Mac mini that I've tested (other two are Muse Glimmer 30B and Gemma 4 31B). Against other models its size, the only model that got a better score was Qwen 3.8 27B(it got a B+).
On my Mac Mini, each question takes between 4min-30min to answer. On OpenRouter, the median was ~70s. So ~4x slower on desktop software. That should get faster when mlx support comes available.
For memory usage, Llama was taking about 29GB on my Mac Mini.
I think this model will be a crazy big deal, it might be the first model that can reliably run multi-agent reasoning harnesses on local hardware, as this is a MoE model and can have several agents each with their own KV caches that fit into VRAM.
Dethroning Qwen 3.8 is a big deal too, and I like their section about CoT monitorability. (This model beats Qwen on benchmarks - TML is not known to benchmaxx nearly as hard as Qwen.) I guess TML really are a bunch of rationalists who know what they're doing, sort of like Anthropic.
show comments
rsolva
Going to test this right away on my AMD 7900XTX and see how it does compared to Qwen 3.8 27B (or DaisyRose 27B that I'm currently running). Given that Inkling 2 Mini is a MoE, it should provide way higher decode speed, but with comparable performance. Exiting!
show comments
shahbazac
I don’t understand the business model of these labs which obviously spend a huge amount of money on compute then release their models. I definitely glad they do, but how does it make business sense?
I'm wondering when will these local models be trained for expert activation on a task-by-task basis, even in an implicit manner, by having a loss function that encourages expert coherence across token activations. The model size could be much, much higher as the bandwidth penalty would have to be paid only occasionally.
This would be an advantage that would only benefit local setups, which is why I guess we don't see it yet.
mvkel
Models like this are a product of resource constraints. It's great. It's why I am for a data center moratorium. Scarcity breeds innovation. If we're forced to work with what we have, incredible efficiencies are on the horizon.
show comments
Almondsetat
What are some models in this ballpark that are also good at conversation? They always show off their software capabilities, but can it talk naturally? make creative stories?
Idea being you can configure at run time how much thinking you let the model do per token.
jakswa
The demo video was 32 tok/s and did a 15X speedup when generating the code? Weird that they didn't publish token speeds for this laptop setup they recorded. I'm worried this tri-loop setup is going to be too slow. QFN is impressive but lately been feeling a little brain-fried by its prose.
show comments
causal
Big fan of open weights but those Terminal Bench numbers are pretty abysmal, and I find that tends to track "agent usefulness" more than most SWE benchmarks.
show comments
mattnewton
Benchmarks look good, and inkling 2 small felt better than the benchmarks suggested, so I am very excited to play with this.
Very interesting to see them publish a looped transformer. It's been reported that OpenAI are now using looped transformers, though of course not confirmed.
I think this may be the first public example of a looped transformer model good enough for real world use cases?
gregsadetsky
bonkers - I'm getting 20 tokens/s running the 4-bit version on my 2021 M1 Max MacBook pro with 64Gb...!!
The thing I like about looped transformers is it implies theres some kind of modularity learned with how each iteration works on the residual stream, which sounds like a lovely inductive bias.
Generated with Inkling-2-Mini-IQ4_NL.gguf and llama.cpp on a M5 MacBook Pro.
show comments
hatthew
Seems like this is the successor to Qwen 3.6 35B that we've been waiting for? I'm not aware of any other recent models of this size that can run well on mid-tier GPUs without severe quantization.
contentkraft
I tested the regular Inkling 2 and in my tasks it’s worse and more expensive than Hy4-flash. Those are ai agents tasks though so not sure how these benchmarks are scoring so differently
NiloCK
> a looped transformer, running the middle layers of the model three times for every token.
Completely unqualified take here, but unconditional re-runs of a fixed set of layers doesn't strike me as meaningfully 'loopy'. The exact performance profile is available in a more trad architecture by just inserting the layers after one another?
My presumption on loopy architectures was that they looped longer on token selections that were more difficult / higher stakes, with automatic kick-outs after some n loops to avoid machine analysis paralysis.
ALTHOUGH I guess that the presented architecture may also be useful insofar as it can fit in less memory than the same model with a tripled set of , and possibly that training over a looping architecture converges faster / truer than over naive architectures of the same size.
laybak
excited to see Thinking Machines accelerating their effort, but in their own way. loving this almost more indie flavor of AI research
nsingh2
I've been thinking about how small looped models are probably going to be the next big thing in local LLMs, and here it is. Exchanging a lower memory capacity requirement for more compute instead.
If the GPT-6.x models are looped, no doubt memory was a big reason.
samayashar
Inkling's benchmarks for testing their model are really amazing. There is no better way to demonstrate a model's capability than building a complete game/application powered by the model itself. For eg: the tiny pixel town where all the 20 villagers are powered by the model on a MacBook - is rock solid!
prometheus1992
This is pretty good but also from their hf model release;
>> Note: each extra pass through the looped layers adds about 1.0B effective active parameters. Since the model runs with three passes, there are 5.4B effective active parameters per token.
chr15m
This looks so good. Might replace Muse Glimmer for me for local tool calling workloads. As good as Qwen 27b for many tasks and no state mandated censorship - yes please! Super excited to try this out today.
frou_dh
IIRC Macs with 32GB unified memory have a default soft-cap of ⅔ for how much memory the graphics subsystem can use, so when they say it's targeted at 32GB Macs that effectively means targeting 21.3 GB VRAM?
show comments
coderz89
very promising with benchmark, I have used Inkling 2 small earlier and it was good with vision in the loop
relistan
Using this model on a MacBook Pro G4 Max with 36 GB… still not really worth using for much. I was hoping it would be a little more useful but the context has to be kept too small to fit in memory. But it’s progress.
neya
Wow, that coding demo was super impressive. Can't wait to try it out. Just need to wait for the hardware market to cool down a bit so we can run these on our own commodity hardware.
e44858
> 3.4B active
Does this mean it can run on a GPU with just 4GB of memory?
show comments
jakswa
The masses lying in wait for Qwen 4 cry out in joy. A savior? Could it be?!
show comments
xlayn
When you are sure enough about what you are doing to include Qwen 3.8 27B in your compare.. we know you are talking... let's test drive it!
stillpointlab
I'm guessing one of these local model companies will get picked by Apple sooner or later. It is just so within their business model and they have the hardware.
show comments
armcat
I didn't even know it was a thing to train using nvfp4 4-bit floats - I always thought it was a post-training quantisation thing. Fascinating!
show comments
capv
In my tests inkling-2-mini shows results similar to GPT 5.6 with light reasoning. Impressive ...
walrus01
Looking at the link to quantizations it looks like unsloth has already quantized it:
In terms of the active parameters and token/s on slow or less capable hardware it seems like it should be in the same general performance category as qwen 3.6 35B A3B.
show comments
mannyv
I've been experimenting with local agent committees, and this should allow me to get a couple of agents running in "realtime" instead of sequentially.
It's funny, you think you have enough RAM then you realize that what you need is more memory bandwidth (ugh).
show comments
mrboatshoepants
What coding harness would someone use with this, such as in the article’s demo video? The demo was impressive
show comments
opan
"Inkling" is strongly associated with Nintendo's Splatoon series these days, FYI.
lwarfield
Omg I've been working on the same thing! I was doing this off a hunch from David Noel ng's RYS posts!
When inkling first released I tested it out and thought it was the best prose from an LLM in my quick sampling -- easy-reading and easy to understand. Big hopes I can write code with one of these models one day.
show comments
vikmals
Nice model, i get 17 tokens/s on my AMD RX 9060 XT 16 GB. Testing coding with it now.
This is actually a big deal. I think probably by 2028 we will get a large number of daily tasks done reliably by laptop size local models.
nitroedge
What is the marketing pitch and the one-liner for what makes this model special?
sourcecodeplz
Even more talkative than Qwen
show comments
purplecats
hate these dark practices. they say click here to try it on our site. then u click. auth wall. fine you give it all your info. sunk cost is maximized. then they hit you with the credit card wall.
holistio
I'm starting to believe Gestalt is actually theirs. This is very nice.
show comments
RandyOrion
Finally another model for local LLM users.
To achieve better performance, compared to parameter scaling methods like mixture of more experts and more whatever embeddings which throw parameter efficiency out of window, layer reuse or looping show higher parameter efficiency and is much appreciated, which I think is enabled by attention residual?
The problem I observed is that in SWE workflows the output length of inkling is longer than qwen 3.8 27b xhigh, which is already notorious. Multilingual support except English is another pain point.
Anyway, congratulations to the inkling team for this release.
DeathArrow
Got between 130 and 152 tokens per second for Q4 using ninfer on a RTX 3090. Pretty good.
blurbleblurble
Yikes, the pelican was insane
hacker_88
Sensanova 9b is another one
sodowork
So the transformer launched in 2017 is still the foundation of these current models
show comments
jedisct1
Doesn't seem to run in MLX-Serve yet :(
lenerdenator
This looks like it could run on my M2 32GB MBP.
I honestly would rather have nice, focused-purpose local models than huge "let's make AGI" datacenter-hosted models. Will have to try it tonight when I get home.
show comments
christkv
I'm loving this trend. My framework desktop is looking like a better and better investment.
show comments
colordrops
Surprised they used a log scale for tokens per second. Seems like a missed opportunity to nail home the difference visually.
zkmon
Which inference engine works right now? https://github.com/ggml-org/llama.cpp/pull/32694 seems to have some comments saying it doesn't work well. Which one gives more than 200k context on 4090 for a Q4? Currently Qwen3.8-27B-Q4_K_M gives me that much space.
Also, why the model file size is only 20GB while UD quants are much larger for Q_8?
zergrush
pretty cool I guess I just don't know what i'd use it for other than being a nice toy. if it can fit on smaller embedded devices that could be useful
AndrewKemendo
Can anyone running one of these locally give any sense of how good they are?
mentalgear
Somewhat astonished that thinking machines would do this considering that looping transformers a.k.a. neuralese is safety wise way harder to keep track of what the LLM is actually doing. by the way, thinking machines CEO is one of the people that quit openAi over safety concerns.
kypro
> Efficient intelligence makes Inkling 2 Mini a great choice for inference where speed is paramount, or for running on local hardware.
Presumably v. good for offensive use cases too.
And a looped model?
This and GRH is +0.1% on my p-doom today.
The trends of capabilities and efficiency we've seen over the last few weeks should terrify anyone with any sense.
This will not end well but we'll have some nice AI generated dashboards until then I guess.
This is bonkers good if these benchmarks hold up in real-world scenarios. To be at the level of Qwen3.8-27B with a looped transformer model is quite the flex. They just left this note on reddit:
Hope that folks find this model more amenable to local usage - the sizing was largely informed by what folks on this subreddit ask for the most haha!
There are two particularly interesting aspects of this model for local users:
We do our training end-to-end in nvfp4, and although we haven't exhaustively benchmarked this, folks like Unsloth have noticed that our models seem to respond very well to being quantized, maintaining 90% top-1 even at at 1-bit quantization.
This model is looped, which allows more depth at the same parameter count, which we particularly notice seems helpful for coding tasks. :)
This is our first model targeted at local users, so let us know how this model is and anything you want to see out of future iterations!
https://www.reddit.com/r/LocalLLaMA/comments/1y1diuk/thinkin...
Seems pretty promising in a few quick coding tests. 27 tokens/s decode on Ryzen 395 and llama.cpp Vulkan. Besides the obvious unmerged PRs (32694, 32657, 32429, 32704) you'll want below tiny patch for about 30% speed boost.
Also, if anyone from the team is reading, if you have access for researchers I'd be interested in doing some higher volume tests & harness tuning than I have hardware for personally.
Whoa, not just a model that acknowledges and embraces using a looped architecture (GPT Astra 6+ and Sol 6.1+ are only [heavily] rumored to be looped), but it's open weights, too?
I'm willing to wager that most open models will go looped in the wake of this.
Wonder what it means for safety and interpretability at this level of availability.
M1 Pro Macbook, 32GB: About 10t/s using q4 xl while running browsers/etc. and interacting using the local browser.
test 1: write a python program to sort a csv by lastname and print out fields. Result: Nailed it. Nice python program. Nice explanation of output, if a little chatty (it gave an example output, which I didn't ask for).
test 2: my standard "identify the type and year of this coin" test. Failed miserably, I gave up after 12,000 tokens of it babbling incessantly about possible misinterpretations of a silver half-dollar. Unsurprising - most small models suck at this test - but slightly annoying it seemed to get in a loop.
Pretty good performance and result for the (very simple) code generation problem, though. One of the first models that seems OK running side-by-side with a browser on my older MBP.
I am getting Stable Diffusion level of breakthrough vibe here. And such looped transformer SMALL models are best suited to embed directly into silicon/transistor. So you get better than ASIC performance by just looping over the hard-coded silicon. Just like we had a breakthrough with "deep" neural network, soon we may have something similar with "deep" loop over the same set of weights. Imaging transistor level looping 128 times. Extreme power efficient, and god knows what level of intelligence is achievable. I could very well foresee purchasing USB Intelligence Devices with some standardised driver protocol. You just buy one off Amazon, throw away the old one and get an Intelligence Upgrade.
The GGUF version of this that can run on Ollama/Llama.cpp is: https://huggingface.co/unsloth/Inkling-2-Mini-GGUF
Quite an interesting model, but the second of my test prompts in "extra high" thinking in Unsloth really underscore what they say about knowledge here:
> While looping improves the model’s reasoning, it doesn’t enable its weights to store more facts—knowledge is mostly a function of total parameter count. Inkling 2 Mini’s score on AA-Omniscience, a measure of factuality, is in line with other models of the same size [...] but knowledge-intensive benchmarks doesn’t improve with more passes...
> Inkling 2 Mini overcomes the knowledge gap by efficiently using search and tools.
I tested the Unsloth Q4 version.
My prompt asked it to write a singleton class, single file plugin for WordPress that hooks user login and stores last login time as metadata. It has some code style requirements. The prompt also asked it to make no tool calls so it didn't search and only used its knowledge.
The model burned through 19200 thinking tokens on this task (which was the hard limit I had set) and didn't finish. Its thinking suggests it was going to make at least one mistake.
Both Qwen 3.6 35B and Gemma 4 26B can cope with this task, from memory, in far fewer tokens (even taking into account Qwen's tendency to loop in thinking).
Rerunning the same prompt with tool use allowed, it searched with browser use, pretty effectively, and got to a solution, but it still made the mistake the thinking traces suggested it would earlier (wrong hook name). Other than that it wrote good code.
But 20K-ish tokens in, I could then not get it to search to make the required correction — it would repeatedly EOT rather than actually run the tools it was saying it would call. Whether this is the model failing or Unsloth's harness and the model disagreeing, I don't know. Quitting and reloading the model to continue the conversation didn't help.
It worked through my usual MySQL-5.0-compatible max score per category prompt nicely, though it didn't volunteer the suggestion of adding an index for speed, but it doesn't outperform LFM 2.5-24B-A2B, which is a non-looping model of similar size. It does work through SQL in its thinking traces quite nicely, though.
Car wash problem got a correct answer quite quickly, albeit with a surreal caveat (why do they always seem to need to offer a caveat, no matter how weird)
So far, I think I would run Muse Glimmer or Nemotron Bolt over this; Muse Glimmer would output tokens more slowly but still get to a result much more quickly.
Very, very excited about models of this size. Small enough to run on a Mac Mini!
Ran my 43 question data analytics exam on it against OpenRouter & my M4 Pro 64GB Mac Mini via llama. It scored a "B" on the exam [1], same score as DeepSeek V4.1 Flash and Sonnet 5.
The main limitation that I saw was a "runaway loop" when asked to find a issues in a dataset: it worked for over 3 hours on my Mac Mini, taking over 180 tool calls to answer the question before I cut it off.
It's the most accurate U.S. open weight model that fits on a Mac mini that I've tested (other two are Muse Glimmer 30B and Gemma 4 31B). Against other models its size, the only model that got a better score was Qwen 3.8 27B(it got a B+).
On my Mac Mini, each question takes between 4min-30min to answer. On OpenRouter, the median was ~70s. So ~4x slower on desktop software. That should get faster when mlx support comes available.
For memory usage, Llama was taking about 29GB on my Mac Mini.
Very exciting overall!
[1] https://plotly.com/blog/inkling-2-mini-plotly-data-analytics...
Recently SensaNova also released a looping model with unbelievable numbers
https://www.sensetime.com/en/news/sensenova-loop-9b-release-...
I think this model will be a crazy big deal, it might be the first model that can reliably run multi-agent reasoning harnesses on local hardware, as this is a MoE model and can have several agents each with their own KV caches that fit into VRAM.
Dethroning Qwen 3.8 is a big deal too, and I like their section about CoT monitorability. (This model beats Qwen on benchmarks - TML is not known to benchmaxx nearly as hard as Qwen.) I guess TML really are a bunch of rationalists who know what they're doing, sort of like Anthropic.
Going to test this right away on my AMD 7900XTX and see how it does compared to Qwen 3.8 27B (or DaisyRose 27B that I'm currently running). Given that Inkling 2 Mini is a MoE, it should provide way higher decode speed, but with comparable performance. Exiting!
I don’t understand the business model of these labs which obviously spend a huge amount of money on compute then release their models. I definitely glad they do, but how does it make business sense?
This looks great and is getting a lot of fun.
Interesting cyberbench: https://x.com/ValsAI/status/2123874523938497007
Thinking machines just got up in Evaluation.
I'm wondering when will these local models be trained for expert activation on a task-by-task basis, even in an implicit manner, by having a loss function that encourages expert coherence across token activations. The model size could be much, much higher as the bandwidth penalty would have to be paid only occasionally.
This would be an advantage that would only benefit local setups, which is why I guess we don't see it yet.
Models like this are a product of resource constraints. It's great. It's why I am for a data center moratorium. Scarcity breeds innovation. If we're forced to work with what we have, incredible efficiencies are on the horizon.
What are some models in this ballpark that are also good at conversation? They always show off their software capabilities, but can it talk naturally? make creative stories?
Very cool, and the also released a research version with variable looping support: https://huggingface.co/thinkingmachines/TML-d24-varloop-rese...
Idea being you can configure at run time how much thinking you let the model do per token.
The demo video was 32 tok/s and did a 15X speedup when generating the code? Weird that they didn't publish token speeds for this laptop setup they recorded. I'm worried this tri-loop setup is going to be too slow. QFN is impressive but lately been feeling a little brain-fried by its prose.
Big fan of open weights but those Terminal Bench numbers are pretty abysmal, and I find that tends to track "agent usefulness" more than most SWE benchmarks.
Benchmarks look good, and inkling 2 small felt better than the benchmarks suggested, so I am very excited to play with this.
From a research perspective, I'm even more interested in the varlooped variant they're releasing https://thinkingmachines.ai/model-card/tml-d24-varloop-resea... I'm not aware of any other working open source models like that.
Very interesting to see them publish a looped transformer. It's been reported that OpenAI are now using looped transformers, though of course not confirmed.
I think this may be the first public example of a looped transformer model good enough for real world use cases?
bonkers - I'm getting 20 tokens/s running the 4-bit version on my 2021 M1 Max MacBook pro with 64Gb...!!
recipe: https://gist.github.com/gregsadetsky/85e8b8ea91430dc826611b9...
The thing I like about looped transformers is it implies theres some kind of modularity learned with how each iteration works on the residual stream, which sounds like a lovely inductive bias.
(all vibes, no data from me)
Pelicans aren't terrible (for the model size): https://tools.simonwillison.net/markdown-svg-renderer?url=ht...
Generated with Inkling-2-Mini-IQ4_NL.gguf and llama.cpp on a M5 MacBook Pro.
Seems like this is the successor to Qwen 3.6 35B that we've been waiting for? I'm not aware of any other recent models of this size that can run well on mid-tier GPUs without severe quantization.
I tested the regular Inkling 2 and in my tasks it’s worse and more expensive than Hy4-flash. Those are ai agents tasks though so not sure how these benchmarks are scoring so differently
> a looped transformer, running the middle layers of the model three times for every token.
Completely unqualified take here, but unconditional re-runs of a fixed set of layers doesn't strike me as meaningfully 'loopy'. The exact performance profile is available in a more trad architecture by just inserting the layers after one another?
My presumption on loopy architectures was that they looped longer on token selections that were more difficult / higher stakes, with automatic kick-outs after some n loops to avoid machine analysis paralysis.
ALTHOUGH I guess that the presented architecture may also be useful insofar as it can fit in less memory than the same model with a tripled set of , and possibly that training over a looping architecture converges faster / truer than over naive architectures of the same size.
excited to see Thinking Machines accelerating their effort, but in their own way. loving this almost more indie flavor of AI research
I've been thinking about how small looped models are probably going to be the next big thing in local LLMs, and here it is. Exchanging a lower memory capacity requirement for more compute instead.
If the GPT-6.x models are looped, no doubt memory was a big reason.
Inkling's benchmarks for testing their model are really amazing. There is no better way to demonstrate a model's capability than building a complete game/application powered by the model itself. For eg: the tiny pixel town where all the 20 villagers are powered by the model on a MacBook - is rock solid!
This is pretty good but also from their hf model release;
>> Note: each extra pass through the looped layers adds about 1.0B effective active parameters. Since the model runs with three passes, there are 5.4B effective active parameters per token.
This looks so good. Might replace Muse Glimmer for me for local tool calling workloads. As good as Qwen 27b for many tasks and no state mandated censorship - yes please! Super excited to try this out today.
IIRC Macs with 32GB unified memory have a default soft-cap of ⅔ for how much memory the graphics subsystem can use, so when they say it's targeted at 32GB Macs that effectively means targeting 21.3 GB VRAM?
very promising with benchmark, I have used Inkling 2 small earlier and it was good with vision in the loop
Using this model on a MacBook Pro G4 Max with 36 GB… still not really worth using for much. I was hoping it would be a little more useful but the context has to be kept too small to fit in memory. But it’s progress.
Wow, that coding demo was super impressive. Can't wait to try it out. Just need to wait for the hardware market to cool down a bit so we can run these on our own commodity hardware.
> 3.4B active
Does this mean it can run on a GPU with just 4GB of memory?
The masses lying in wait for Qwen 4 cry out in joy. A savior? Could it be?!
When you are sure enough about what you are doing to include Qwen 3.8 27B in your compare.. we know you are talking... let's test drive it!
I'm guessing one of these local model companies will get picked by Apple sooner or later. It is just so within their business model and they have the hardware.
I didn't even know it was a thing to train using nvfp4 4-bit floats - I always thought it was a post-training quantisation thing. Fascinating!
In my tests inkling-2-mini shows results similar to GPT 5.6 with light reasoning. Impressive ...
Looking at the link to quantizations it looks like unsloth has already quantized it:
https://huggingface.co/thinkingmachines/Inkling-2-Mini
https://huggingface.co/unsloth/Inkling-2-Mini-GGUF
In terms of the active parameters and token/s on slow or less capable hardware it seems like it should be in the same general performance category as qwen 3.6 35B A3B.
I've been experimenting with local agent committees, and this should allow me to get a couple of agents running in "realtime" instead of sequentially.
It's funny, you think you have enough RAM then you realize that what you need is more memory bandwidth (ugh).
What coding harness would someone use with this, such as in the article’s demo video? The demo was impressive
"Inkling" is strongly associated with Nintendo's Splatoon series these days, FYI.
Omg I've been working on the same thing! I was doing this off a hunch from David Noel ng's RYS posts!
https://dnhkng.github.io/posts/rys/
When inkling first released I tested it out and thought it was the best prose from an LLM in my quick sampling -- easy-reading and easy to understand. Big hopes I can write code with one of these models one day.
Nice model, i get 17 tokens/s on my AMD RX 9060 XT 16 GB. Testing coding with it now.
Thank you! Added to the Tiny League (https://tinyleague.nunodonato.com)
This is actually a big deal. I think probably by 2028 we will get a large number of daily tasks done reliably by laptop size local models.
What is the marketing pitch and the one-liner for what makes this model special?
Even more talkative than Qwen
hate these dark practices. they say click here to try it on our site. then u click. auth wall. fine you give it all your info. sunk cost is maximized. then they hit you with the credit card wall.
I'm starting to believe Gestalt is actually theirs. This is very nice.
Finally another model for local LLM users.
To achieve better performance, compared to parameter scaling methods like mixture of more experts and more whatever embeddings which throw parameter efficiency out of window, layer reuse or looping show higher parameter efficiency and is much appreciated, which I think is enabled by attention residual?
The problem I observed is that in SWE workflows the output length of inkling is longer than qwen 3.8 27b xhigh, which is already notorious. Multilingual support except English is another pain point.
Anyway, congratulations to the inkling team for this release.
Got between 130 and 152 tokens per second for Q4 using ninfer on a RTX 3090. Pretty good.
Yikes, the pelican was insane
Sensanova 9b is another one
So the transformer launched in 2017 is still the foundation of these current models
Doesn't seem to run in MLX-Serve yet :(
This looks like it could run on my M2 32GB MBP.
I honestly would rather have nice, focused-purpose local models than huge "let's make AGI" datacenter-hosted models. Will have to try it tonight when I get home.
I'm loving this trend. My framework desktop is looking like a better and better investment.
Surprised they used a log scale for tokens per second. Seems like a missed opportunity to nail home the difference visually.
Which inference engine works right now? https://github.com/ggml-org/llama.cpp/pull/32694 seems to have some comments saying it doesn't work well. Which one gives more than 200k context on 4090 for a Q4? Currently Qwen3.8-27B-Q4_K_M gives me that much space.
Also, why the model file size is only 20GB while UD quants are much larger for Q_8?
pretty cool I guess I just don't know what i'd use it for other than being a nice toy. if it can fit on smaller embedded devices that could be useful
Can anyone running one of these locally give any sense of how good they are?
Somewhat astonished that thinking machines would do this considering that looping transformers a.k.a. neuralese is safety wise way harder to keep track of what the LLM is actually doing. by the way, thinking machines CEO is one of the people that quit openAi over safety concerns.
> Efficient intelligence makes Inkling 2 Mini a great choice for inference where speed is paramount, or for running on local hardware.
Presumably v. good for offensive use cases too.
And a looped model?
This and GRH is +0.1% on my p-doom today.
The trends of capabilities and efficiency we've seen over the last few weeks should terrify anyone with any sense.
This will not end well but we'll have some nice AI generated dashboards until then I guess.