> Put two machines on a desk, each about $2,000. One is a tower with an NVIDIA RTX 5090: 32GB of the fastest consumer memory ever shipped, 1,792 GB/s. The other is a mini PC the size of a paperback, an AMD Ryzen AI Max+ 395 "Strix Halo" box with 128GB of soldered memory at roughly 256 GB/s.
Doesn't change the conclusions of the article, but each of those machines is more like $4k+
"integrated graphics processor, using system memory" had its name dragged through the mud for decades.
So we had to rebadge it to "unified memory".
Curious if we'll ever see some old integrated graphics processor "hacked" to manage to handle 128 GB of allocated system RAM and be able to serve diffusion-LLMs at a decent rate on "old" hardware...
_davide_
I'm writing my own inference engine for Strix Halo and the same model. I already have 30%+ performance plus a more graceful decay over long contexts; that said, their point stands: memory bandwidth is what you really want.
cocodill
For some reason, this reminds me of my last shared memory system. It was an Athlon XP 1800+ with VIA ProSavage back around 2002. It was just barely able to run CS 1.6.
Havoc
Think future generations of AMD could get quite interesting. They’re no doubt seeing people whining about mem throughput specifically
danbruc
Why would a RTX 5090 with 32 GB not be able to deal with a 40 GB model? Is there anything preventing me from swapping the weights that do not fit into VRAM in and out of RAM? PCIe 5.0 x16 should max out around 64 GB/s, so slower than the unified memory machine, but at least it should be possible.
show comments
vkaku
I'm going to say this that we're not even close to the limits of what actually needs to be accomplished so at some point, memory will start needing better tiering for inference some day ....
_davide_
If compute is not the bottleneck, memory is easy-ish to produce (the hard part is mostly on the fab side); what stops a Chinese NVIDIA (huawei) from being 10x cheaper?
show comments
throwa356262
"Can't" is not really correct.
Nowadays, specially with MoE models you can run parts of the model on GPU and still get some speed up.
show comments
amelius
Do unified memory CPUs suffer from the same memory shortages as normal memory?
I guess they're just welding the memory to the CPU chip, but still curious.
show comments
lowbloodsugar
The current “big GPU” has 96gb of memory, but that’s not a “consumer GPU” apparently, while a $5000 Spark is a “consumer PC” I guess. In any case you’re probably better off running a large open weights model on the cloud.
erkt
Uhh the 5090 alone is double the cost of their quoted PC prices.
LoganDark
Can't really run it as well, though. My "mini PC" is an M4 Max with 128GB of unified memory and the memory bandwidth is still sorely lacking for inference (although it's far better than any non-unified consumer architecture!).
show comments
OutOfHere
Let's also ensure the SSD doesn't age prematurely.
> Put two machines on a desk, each about $2,000. One is a tower with an NVIDIA RTX 5090: 32GB of the fastest consumer memory ever shipped, 1,792 GB/s. The other is a mini PC the size of a paperback, an AMD Ryzen AI Max+ 395 "Strix Halo" box with 128GB of soldered memory at roughly 256 GB/s.
Doesn't change the conclusions of the article, but each of those machines is more like $4k+
https://www.microcenter.com/product/711961/amd-ryzen-ai-halo...
"integrated graphics processor, using system memory" had its name dragged through the mud for decades.
So we had to rebadge it to "unified memory".
Curious if we'll ever see some old integrated graphics processor "hacked" to manage to handle 128 GB of allocated system RAM and be able to serve diffusion-LLMs at a decent rate on "old" hardware...
I'm writing my own inference engine for Strix Halo and the same model. I already have 30%+ performance plus a more graceful decay over long contexts; that said, their point stands: memory bandwidth is what you really want.
For some reason, this reminds me of my last shared memory system. It was an Athlon XP 1800+ with VIA ProSavage back around 2002. It was just barely able to run CS 1.6.
Think future generations of AMD could get quite interesting. They’re no doubt seeing people whining about mem throughput specifically
Why would a RTX 5090 with 32 GB not be able to deal with a 40 GB model? Is there anything preventing me from swapping the weights that do not fit into VRAM in and out of RAM? PCIe 5.0 x16 should max out around 64 GB/s, so slower than the unified memory machine, but at least it should be possible.
I'm going to say this that we're not even close to the limits of what actually needs to be accomplished so at some point, memory will start needing better tiering for inference some day ....
If compute is not the bottleneck, memory is easy-ish to produce (the hard part is mostly on the fab side); what stops a Chinese NVIDIA (huawei) from being 10x cheaper?
"Can't" is not really correct.
Nowadays, specially with MoE models you can run parts of the model on GPU and still get some speed up.
Do unified memory CPUs suffer from the same memory shortages as normal memory?
I guess they're just welding the memory to the CPU chip, but still curious.
The current “big GPU” has 96gb of memory, but that’s not a “consumer GPU” apparently, while a $5000 Spark is a “consumer PC” I guess. In any case you’re probably better off running a large open weights model on the cloud.
Uhh the 5090 alone is double the cost of their quoted PC prices.
Can't really run it as well, though. My "mini PC" is an M4 Max with 128GB of unified memory and the memory bandwidth is still sorely lacking for inference (although it's far better than any non-unified consumer architecture!).
Let's also ensure the SSD doesn't age prematurely.