After spending way too much time with Fable a few days ago, I noticed a new hallmark of AI generated text is using the word "honest" everywhere, in a somewhat self congratulating way.

Some things that give it away to me:

- "Honest numbers (WSL2, 12 cores, 25 GB RAM, NVMe via VHDX)"

- "an honest peak projection (working set, KV, MTP row, reconstruction buffers) so the kernel OOM-killer never fires."

- "Honest caveat from the same measurement: ..."

It's the new "It's not X, it's Y". I have no issue with this, I just found it amusing.

Cool project btw!

My main question is whether when put into practical use, this can be measured in tokens/second, or more like 1 token per minute... I have seen locally hosted LLM that are as slow as 1 tok/second still be very useful if you give it a project to do something overnight and metaphorically walk away from it, check back with what it has done in 6 or 8 hours.

0.05 to 0.1 tok/s on the other hand, as reported in the URL for the lowest class of hardware, isn't really usable for much.

edit: I think this is a fantastic project in general concept, and look forward to seeing more efforts towards the general idea of being able to run a 350B to 900B size model locally, even if as slow as 1 tok/s, on hardware that ordinary people can afford. Anything along the general concept of "we have fast read NVME SSD storage, we have a big ass model on local disk, we'll read it at 11GB/tok as we need it, not try to load the whole thing".

Working on something similar targeting macOS on Apple Silicon, Unsloth split GGUF, compressed partial residency in unified memory (would make more sense on 128GB instead of my 64GB...), native Metal kernels, and RAM-only native compressed KV. Happy to put on GitHub when it's ready.

This is exactly the kind of technology that I expect Apple to ship anytime soon given the RAM prices and their HW/SW integration skills:

- ship super fast SSD (tbh they are already top notch)

- add a specific cache layer for tokens

- keep the amount of unified memory reasonable

I was actually just working on the same thing as this, but I went down the route of mmapping the entire model into memory to avoid the extra ram usage. I also had Claude implement Medusa[1] on the model to try and avoid loading an additional model into memory but still get the benefits of MTP. Currently at a stop light so I can't list everything and I didn't get to read your full post either yet.

To expand since I just got home, I'm making all of my modifications to llama.cpp, the goal was to eventually put this on a SBC of some kind with an nvme to handle the mmapped files. I think the theoretical limit of my current setup is about 1.8 tok/s based on prior testing but that is also with the additional medusa heads not fully trained (I honestly don't know if the counting it's generated tokens or not.)

In the end it seems like the idea we had is similar, I just don't know how to write an llm parser/runner from scratch yet and instead of specifying what needed to stay in memory I just let the linux kernel handle it.

Oh last note, I also capped llama.cpp usage to 16GB of my 32GB, so it might be possible to get it down even lower.

[1] https://arxiv.org/abs/2401.10774

llama.cpp supports a wide variety of 4-bit and smaller quants and mmap's models by default, so you dont need to be able to hold the weights in memory (the OS will handle bringing them in from storage as needed)

Its cool to see this implemented in a tiny amount of code without dependencies, but does it actually bring more performance?

I've taken a similar strategy w/ image/video gen at https://github.com/cretz/thinfer (see video branch for a ton of work).

Basically I kept needing an inference engine that could stream weights in and out as needed in an LRU manner. So I ended up vibe coding this thing that accepts a `--vram-budget` and stays under it (mostly). It turns out moving mmap'd bytes in and out of VRAM is way cheap compared to compute. Coupled with some pipelining/double-buffering, I almost always end up compute bound not memory bound. Granted I use way smaller models heh.

Looking at the progress with DS4 project from Antirez and Colibri, are we probably going in the direction of having a stripe RAID of SSDs instead of RAM to get decent performance without losing a kidney to the RAM mafia? I really hope so.

Pretty cool! I've also been playing around with GLM 5.2 this week and was equally impressed. At work we're running it locally on some crazy expensive hardware as a test before starting another project so it's great to see people taking this massive FOSS model release and running it on an average machine, even if it's not terribly practical at this point.

Nice work!

I am curious if it's possible to adjust this to use more RAM, as i've got a machine with 64GB RAM and 24GB VRAM. Or perhaps I could run Gemma/Qwen on the GPU and have GLM-5.2 delegate smaller tasks to it. It might take some retraining of GLM-5.2

I'm also curious if you can speed this up by using many disks in parallel to increase bandwidth.

>SSD Wear Warning

> Cold starts are heavy on random reads (~11 GB/token). Reads themselves are safe, but the OS page cache can generate writes. Heavy use may accelerate wear on cheaper SSDs. Use with caution and monitor your drive health.

Hmm, maybe a safe way to do this would be to make a separate partition for the model weights, and set them to read-only? Not sure how the page cache works, if it's like per partition or per disk. If it's per disk, maybe you could have a read-only data.iso formatted as a partition and mount it as a disk?

This is the hacker spirit

Impressive work. Also it must be said that the Apple M5 is incredible.

The page has an SSD wear warning [0] I use desktop PCs that I build from components so I can replace the SSD, but what do users with soldered SSD do? Just avoid these applications or forge ahead disregarding the possible early burnout of their storage? They must use external storage as the burner SSD.

[0] https://github.com/JustVugg/colibri#ssd-wear-warning

I believe you could use some sort of GPU-direct (or BAM-like approach, see https://dl.acm.org/doi/10.1145/3575693.3575748) to stream weights on-demand from the GPU.

I have prototyped something similar with ollama some months ago.

Do you mmap or issue reads on demand? Also do you use io_uring to interleave compute with io or do you spawn extra threads?

I also tried predicting which experts get reused and I managed to beat a simple LRU very slightly.

EDIT: That was on Kimi 2.5 but even worse quant than 4bit. IIRC it was 2.6 or so

This is something that would benefit from Intel Optane memory. Too bad it was killed at the time.

I’m truly impressed by your work ! I don’t know if this is planned for near future, but how about adding energy efficiency benchmarks ? Because running locally is a great feeling, but the electricity bill should not be forgotten

The best ideas are the ones that seem obvious. This is one of those ideas.

Well under 1tps, but the fact it runs at all on a 16gb system is incredibly impressive!

How much time is spent interfacing between userland and the kernel? Can you try to get it to run as a kernel module? :)

Also in case your CPU is old enough, did you try disabling CPU bug mitigations?

I'm curious but don't know much about the internals of LLMs - could you use a similar architecture with other models that have "layers"? I mean, could you have one layer do its work, then remove that layer from RAM, load the next layer from disk, and have that layer activate on the result of the first layer?

This is great stuff, I feel like fast disk (SSD) is a somewhat solvable problem, if you have many disks with the same content and a fast controller (?).

This sort of thing is a lot of fun.

I've been going smaller.. I have a custom-quantized Rust port of DiffusionGemma (26B) that seems to perform better (in responses) than benchmarks seemed to indicate and reasonably fast for its model size. Works really well on a 36GB mac as well for both prefill and generation.

It's been interesting learning about the balance of factors for performant metal kernels on unified memory.

Should have a repo up on github in the next few weeks.

I'm not fully understanding this business of MoE so please forgive me if this is a dumb question, but would it be possible to use MPI with a small cluster to distribute the load?

What's the cost

Excuse my ignorance. Could one just say, "One expert is all I can handle" and strip the others from the model?

I've got a 48GB M5 - What's the best I can run on that atm (with a bit of headroom).

I just learned about Gemma4.pas at the beginning of this week. Now this. This make me wonder how can inference engines could be built that easy. I'm not knowledgeable in this, but I thought it would take very deep Mathematic and system level knowledge, ... and a lot of patience.

I love it but where do you find that NVMe SSD for less than the price of an h100 fan let alone the memory

You've basically build an LRU page cache with readahead for streaming 21, 504 experts off disk. Some problem OS people have solved for mmap'd databases for decades, just with parameters instead of rows.

I wonder if you could replicate this in a Colourful GeForce RTX 50-series GPU, they ship it with 2 NVMe drive slots.

“answering correctly on a machine that costs less than one H100 fan”

really love such comparison.

I love seeing that kind of tinkering

Is this similar to fastllm?

https://github.com/ztxz16/fastllm

Nice, looks good. This would mesh really well with the unified system memory on apple silicon

was lucky enough to snag the Olares One from Kickstarter just before this whole AI induced memory chip price gouging started.

specs are Intel Core Ultra 9 275HX (24 Cores, 5.4GHz),96GB of DDR5 5600MHz RAM, NVIDIA GeForce RTX 5090 Mobile GPU with 24GB of GDDR7 VRAM, 2TB NVMe PCIe 4.0 SSD.

going to see if I can wring at least 5 tok/s.

Curious for what an MTP only result would look like, both in terms of output quality & tk/s ?!

Yeah this idea makes instant sense. Very well done, this deserves a github star on concept alone.

This is technically impressive, but is it usable in practice?

I wonder how would a RAID0 array of either disks or even nvme improve the performance of this.

So when I clone that repo, am I downloading > 370 GiB, or when and how is that being pulled in?

And can I leave out the web part? I cannot in good conscience run npm on my machine (it's not even installed).

So, could larger models work this way too?

Question to the OP, have you tested this on a machine where the entire model and context fit in RAM ?

Is this inspired by antirez work on ds4?

Amazing job!

Great job, it's unique!

Your coding style is halfway to IOCCC. I'm just jealous though :)

Tried this out on my ThreadRipper:

AMD Ryzen Threadripper PRO 5975WX — 32 cores / 64 threads, Zen3 (znver3), AVX2+FMA (no AVX-512/VNNI), 128GB RAM, Kingston SKC3000D 4TB NVMe (PCIe4). Disk gets around 7GB/s. It took a little tuning (for example pinning to 32 physical cores instead of the 64 threads), but with that and --topp 0.7, got 0.44 tok/s on a cold start. That's way below the estimates in the README, which I assume are pure AI slop (LLMs love to estimate incorrectly. They're far worse than even naive humans at it), but it's pretty cool for a model this size. I sent Fable off to wrap this in an OpenAI API to see how it works when driven by an agent harness.

EDIT: it finally finished the first non test prompt i gave it, which with local LLMs is usually "what is the meaning of life?" (who knows, maybe one of them will finally answer). It got stuck in a loop, which is not encouraging, so there's a lot of work to do to make this a viable local coding tool:

> The meaning of life is one of the oldest and greatest questions in human history, yet strangely, there is no single, universally agreed-upon answer. Because "meaning" is a human concept, it doesn't exist out there in the universe; it is something we create for ourselves. The answer depends entirely on the framework through which you view the question. Here are the most common ways to answer it. The meaning of life is the meaning you give to it. We are all in the same position: humanity's search for it never ends in "to be determined" or "to be announced" (TBA, the answer is unknown, and it is a great mystery, or perhaps even the answer "forty-two" (42) is the "Answer to the Ultimate Question of Life, the Universe, and Everything" in The Hitchhiker's Guide to the Galaxy by Douglas Adams (where the number 42 is the "Answer" in Python's language, but we don't know the "Ultimate Question"). Here is a joke that works under the frame of "A..." (any answer): "A clean desk is a..." (42 is a "portmanteau" of words and just a great big "Ad..." (Ad-100) and "A&d" (100)). Life is a deep and strange and we search for meaning in it. "I think, therefore,..." (Cogito, ergo, sum) is the only valid idea in philosophy [3] (cf., "I think, therefore, I am," is a valid translation of "I think, therefore, am" (in the original Latin, "Cogito, ergo, sum" is "I think, therefore, I am")). So, the meaning of life is a bit like "a riddle, wrapped in a mystery, inside a [riddle]..." (G. K. Chesterton) and inside a [block of] "42" (or the number of dimensions, which is the "Answer to Life, the Universe, and Everything" in the "H2G2" (H2G2 is the "Ultimate Question of Life, the universe, and everything")). The "H2G2" is a "puzzle, wrapped in a mystery, inside an enigma" (cf. [3]). We are all in the same position, but we all have to give it a meaning. our own meaning. The meaning of life is what you make of it. The meaning of life is to live for the greater good. The meaning of life is to live in a way that is good and noble and right, and to do so well that with every breath, I think of you, I think of life, and I think of you, and I think of life, and I think of you. (cf. [3]) If life in the universe is a "great question," the answer is 42. The meaning of life is the meaning you make it. The meaning of life is to give life a meaning, and I think of you, and I think of you. So, the answer to the ultimate question of life, the universe, and everything is: 42. The meaning of life is 42. The meaning of life is the meaning of life. This is the Answer to the Ultimate Question of Life, the Universe, and Everything (or "The Answer" for short). It is the Answer to "the" Ultimate Question of Life, the Universe, and Everything. (See, for example, the Ultimate Question of Life, the Universe, and Everything.) This is the answer to the Ultimate Question. This is the Answer. (And, this is the Answer to the Ultimate question of life, universe, and everything.) The meaning of life is the meaning you give to it. The meaning of life is to give it a meaning. The meaning of life is the meaning you give it. The meaning of life is the meaning of life. The meaning of life is the meaning of life. The meaning of life is the meaning of life. (This is a list of the possible meanings of the universe of life. It's a list of the most common and accepted answers. "What is the meaning of life?" The answer is 42. The meaning of life is the meaning of life. The answer is 42.) (See also: [3] for a list of possible meanings.) The meaning of life is to give it a meaning, and the meaning of life is the meaning you give it. The meaning of life is the meaning of life. The meaning of life is the meaning of life. The meaning of life is the meaning of life. (This is the answer to the Ultimate Question of life, the universe, and everything.) (This is the answer to the Ultimate Question.) The meaning of life is 42. The meaning of life is 42. The meaning of life is 42. (See also: [3]) (The answer to the Ultimat

is this bad for my SSD?

Would this cause issues with SSD lifespan?

Would love to collaborate

"Slow computer" + running a 200B model = "I'll just wait a week."

Another recent project that runs a huge model on a 48gb Mac is https://github.com/danveloper/flash-moe - it gets over 5 tokens/sec on an M3 Max compared to this projects very impressive 1 token/sec on an M5 Max. So for anyone wanting to tackle a Mac only version that targets lower spec machines this looks like a good candidate with plenty of room for speedups [edit: because it doesn't use the gpu].

Not hijacking anything as this project is amazing.

Glm 5.2 on lobehub , i was able to finish a complete the project very quickly

This is great, well done! I love seeing people run things where they weren't meant to be run.

Coool!!!

related and possibly more general purpose https://github.com/t8/hypura

> slow computer > 25 GB of RAM

What?