I feel like these types of benchmarks mostly test what the mainstream tells you is possible and or works. You rarely get to hear the different implementations of inference engines, different types of setups (e.g. hybrid ones or older hardware) and their pros and cons. Also it seems these were tested with very minimal token usage.

Currently on my 9 year old laptop with a GTX 1070 Max-Q 8GB vram, 16 gb ddr4 ram and i7 8750h. I am running a finetuned Qwen 3.5-35B-A3B (Tiel-Coder-35B-A3B-GGUF-MTP) with 4-bit quant on a llamacpp fork (Turboquant) with 131k context at around 16-19 tk/s and 350-500 tk/s prefil using Pi harness. It is amazing for writing scripts and doing medium coding tasks and summarizing text for me. After around 100k context it goes down to 9 tk/s but that is to be expected.

If you ever ask any frontier LLM or these types of benchmarks, they would probably say that the best model I can run on my system is a crappy 9B model or something, which is just completely false. So I really wonder how much more you can pull out from more powerful systems and much better open weight models.

It says 46 tok/sec for Qwen3.6-27b 4 bit on RTX5090, but I get 60-70 tok/sec on a small context like that (5 bit weights with 8 bit KV cache), although it does slow down to 40 tok/sec on large contexts. If I add ngram speculation, then it can increase to 100+ tok/sec on code.

There are many knobs to tweak to make it faster, which the site ignores.

By running vllm at work I learned that you will usually outperform benchmarks which you will find on the internet. Mostly by running proper parameter tuning. So you get impressive numbers. Still in reality those numbers do not tell the whole truth. We found that real usage is more 60k-140k prompt with a few k of output for realistic agentic coding workloads. Most of the time that will give you half to a third of decode throughput vs the benchmark. It also becomes much more about cache aware routing and kv cache management (how to offload and restore). Because prefilling 60-140k will simply take too long for interactive work.

Prices for everything are all in flux at this moment. Consumer cards like RTX 5090 are bonkers. Even 3090s have like doubled in the last six months. Meanwhile, prices for renting this class of hardware on vast.ai are spiking too, with rates increasing like 50% in September alone.

It's hard to know if we're at the beginning or the end of this trend. If manufacturing continues to focus on high end datacenter gpus at $40k a pop, and we continue to have supply constraints, this could last a while.

On the other hand, if Chinese models continue to nip at the tails of the frontier, cheaper by 20x, running on new Chinese chips, the whole bottom could fall out of the US market, and things could get very weird.

around this time, i was thinking about running a local LLM for the company i work for. i'm trying to figure out what kind of hardware i'd need and what level of performance i could expect. i tried a few things on your website but how much do the results reflect real world usage?

in my use case, the model might need to analyze a given task first then generate and execute a database query and finally send the results back to the client. to what extent could your setup handle this kind of workflow and how accurate are the results?

good job.

Depending on what hardware you're considering, you might rent a vast.ai box with similar specs for a month or two to get a feel for it.

This is great. It solves two of my issues:

  1. It is difficult to figure out if dropping $10k+ on hardware will make a meaningful difference for running local models.
  2. Downloading each model and each quantization of that model to see the speed seems rude to free services like hugging face, so now I can see things like "with my hardware, going to a Q8 only results in 3 more tokens per second"

> This combination won't run. Qwen 3.6/3.8 27B at FP16 needs 54.6 GB — the AMD RX 9070 XT 16GB has 14.9 GB usable. No quantization of this model fits; pick a smaller model or more memory.

While it is true that it won't fit in your GPU memory, it demonstrably does run on my hardware. Llama.cpp supports putting some layers on the GPU and running others on the CPU.

Also, that message says "No quantization of this model fits" but switching to a different quantization of that model (for example, Q8_0) makes the message go away?

It is nice to have simulated confirmation of the M4 Max 128gb limitations on Gemma4 31B vs Gemma4 26B A4B at Q4:

Time to first token (TTFT) on 5,687 tokens, like pasting a short code block into the prompt, takes 36 seconds vs 4.7 seconds. It also breaks down the seconds for each step like KV and prefilling.

Gemma4 31B TTFT on the M5 Max 128gb with same setup is showing as 19 seconds.

just go rent one for a few minutes and try it out ... it'll cost you well under $0.50 if you're just trying to get a feel for if it's right for you. There's a lot of things like context window that this doesn't capture.

Depending on who you sign up for, that's within the introductory credits (usually like 5-10$) so it's actually $0.

A random number I checked feels kinda optimistic, or there's something wrong with my setup; Gemma 26B-A4B on a 16GB 5060Ti is reported to get 21-88tok/s with Q4_K_M, meanwhile I can just barely eke out 70tok/s with Q3 (which fully fits in vram), and 40tok/s with Q4 (with recently-merged MoE cache on llama-cpp). I guess the average fits, but the peak feels implausible.

No clue about details, but Sebastian Raschka's site reports its kv cache usage at 210kB per token, which is like 6x more than this site claims ("0.273 GB per 8k"). Also, the site claims qwen 27b to use 2x more memory per token than gemma, which is the opposite from what I know.

How about context length? Isn't it still just a few thousand tokens on local models?

I would like to see in such simulators:

- multi GPU configuration (hetero and homo)

- PCIe gen chooser (3.0/4.0/5.0)

- is nvlink enabled (2 cards, 2 x 2 cards, 3 x 2 cards)

- is PCIe bifurcated or CPU native (threadripper pro for example)

Shame they are not including old crap like my RX580 which is doing some impressive stuff with Qwen 9B Models.

It would be great if numbers weren't off, would include optimised versions (ie. speculative decoding), context size, include allowed parallel inference (single agents use parallel inference a lot) etc.

I don't think in general people care about long list of models, they just care about the best one.

Also dual gpu is common setup, there should be ie. 2x 4090.

Note that this assumes a single chat (single cached context). As soon as you want to use it for several tasks in parallel or in frequent alternation, hardware requirements increase or performance drops.

A ton of RTX Spark (supposedly similar to DGX Spark) laptops just came out. We really need this for those.

Asus ProArt P14, Surface Ultra, Yoga Pro 9n, etc.

I am hoping the cheaper N1 models will also come out soon, but might be several months.

Also would be good to see a few more quant and model options.

Model list is a bit limited, and whole harness affects the speed, so the libs used in harness and their versions might be useful to show for each individual model setup.

Cool examples, this is interesting to test. I've noticed that OpenAI and Meta play a lot with animations and fades to mask the delays.

Just chiming in that the idea is cool, but the speeds are way wrong

Cool idea but I'd like to also be able to set a custom value of tok/s, ignoring all the other options

Doesn't seem to work? It just spits out a fixed answer about RAM. qwen3.8 27b

The hardware that you cant afford and will never pay for itself? Exciting!

No Qualcomm Snapdragon X Elite?

I don't know what basis these numbers are estimated on, but my M2 Ultra 128GB runs almost twice as fast as what's reported for the M3 Ultra 512GB.

I wished there was such a tool before picking my hardware, it's useful. But the estimates needs to be somewhat correct.

Missing: Dual Strix Halo, Dual Nvidia Spark

Would be nice if we could choose, say, multiple 4090s or otherwise customize the configuration (or maybe I just missed the button for that?)

Can't go below 12GB?

Please modify to include multiples of each as well. Most serious people aren't going to use a single PCIe card if they can help it.

This is a great idea but I'm far less concerned about output token speed and way more about the ability to parallelize prefill - for agentic coding that's pretty much way more important than output speed.

Why can't tools show a progress bar for the prefilling step?

It would be nice if it would give a feel for the quality too ...

Doesn't provide for hybrid setups RTX5070 + RTX5080 for example.

This is not accurate, I get like 80t/s on Q3.8 27b UD Q4_K_XL on my 4090 with 3 drafts and about 110K context window, sim shows mid 20s