A good technical project, but honestly useless in like 90% of scenarios.
You want to use an NVidia GPU for LLM ? just buy a basic PC on second hand (the GPU is the primary cost anyway), you want to use Mac for good amount of VRAM ? Buy a Mac.
With this proposed solution you have an half-backed system, the GPU is limited by the Thunderbolt port and you don’t have access to all of NVidia tool and library, and on other hand you have a system who doesn’t have the integration of native solution like MLX and a risk of breakage in future macOS update.
I don't know how Apple has evaded regulatory scrutiny for their refusal to sign Nvidia's eGPU drivers since 2018.
show comments
mlfreeman
I followed the instructions link and read the scripts...although the TinyGPU app is not in source form on GitHub, this looks to me like the GPU is passed into the Linux VM underneath to use the real driver and then somehow passed back out to the Mac (which might be what the TinyGrad team actually got approved).
Or I could have totally misunderstood the role of Docker in this.
show comments
Keyframe
Such a shame both companies are big on vanity to make great things happen. Imagine where you could run Mac hardware with nvidia on linux. It's all there, and closed walls are what's not allowing it to happen. That's what we as customers lose when we forego control of what we purchase to those that sold us the goods.
show comments
tensor-fusion
As more people carry ARM laptops and keep the GPU somewhere else, I think the interesting UX question becomes whether the GPU can "follow" the local workflow instead of forcing the whole workflow to move to the GPU host. That's the problem we've been looking at with GPUGo / TensorFusion: local-first dev flow, remote GPU access when needed. Curious whether people here mostly want true attached-eGPU semantics, or just the lowest-friction way to access remote compute from a Mac without turning everything into a remote desktop / VM workflow.
show comments
arjie
Woah, this is exciting. I'm traveling but I have a 5090 lying around at home. I'm eager to give it a go. Docs are here: https://docs.tinygrad.org/tinygpu/
I hope it'll work on an M4 Mac Mini. Does anyone know what hardware to get? You'll need a full ATX PSU to supply power, right? And then tinygrad can do LLM inference on it?
show comments
eoskx
Interesting, but cannot run CUDA or more to the point `nvidia-smi`.
show comments
dd_xplore
Why does Apple need to make the drivers in a walled garden? Atleast they should support major device categories with official drivers.
show comments
wmf
Pretty misleading. This driver is only for compute not graphics.
show comments
vondur
If you could get Nvidia driver support on Mac’s I bet Apple would have sold more MacPro’s.
the__alchemist
I'm writing scientific software that has components (molecular dynamics) that are much faster on GPU. I'm using CUDA only, as it's the eaisiest to code for. I'd assumed this meant no-go on ARM Macs. Does this news make that false?
show comments
frankc
My main thought is would this allow me to speed up prompt process for large MoE models? That is the real bottleneck for m3ultra. The tokens per second is pretty good.
What are the limitations of USB4/Thunderbolt compared with a regular PCIe slot?
show comments
vegabook
now can they please approve the linux kernel
show comments
bigyabai
The opportunity cost of Apple refusing to sign Nvidia's OEM AArch64 drivers is probably reaching the trillion-dollar mark, now that Nvidia and ARM have their own server hardware.
A good technical project, but honestly useless in like 90% of scenarios.
You want to use an NVidia GPU for LLM ? just buy a basic PC on second hand (the GPU is the primary cost anyway), you want to use Mac for good amount of VRAM ? Buy a Mac.
With this proposed solution you have an half-backed system, the GPU is limited by the Thunderbolt port and you don’t have access to all of NVidia tool and library, and on other hand you have a system who doesn’t have the integration of native solution like MLX and a risk of breakage in future macOS update.
Idk why this doesn't link to the original source instead of this proxy source: https://x.com/__tinygrad__/status/2039213719155310736
From what I understand, only works with Tinygrad. Which is better than nothing but CUDA or Vulkan on pytorch isn’t going to work from this.
[1] https://docs.tinygrad.org/tinygpu/
I don't know how Apple has evaded regulatory scrutiny for their refusal to sign Nvidia's eGPU drivers since 2018.
I followed the instructions link and read the scripts...although the TinyGPU app is not in source form on GitHub, this looks to me like the GPU is passed into the Linux VM underneath to use the real driver and then somehow passed back out to the Mac (which might be what the TinyGrad team actually got approved).
Or I could have totally misunderstood the role of Docker in this.
Such a shame both companies are big on vanity to make great things happen. Imagine where you could run Mac hardware with nvidia on linux. It's all there, and closed walls are what's not allowing it to happen. That's what we as customers lose when we forego control of what we purchase to those that sold us the goods.
As more people carry ARM laptops and keep the GPU somewhere else, I think the interesting UX question becomes whether the GPU can "follow" the local workflow instead of forcing the whole workflow to move to the GPU host. That's the problem we've been looking at with GPUGo / TensorFusion: local-first dev flow, remote GPU access when needed. Curious whether people here mostly want true attached-eGPU semantics, or just the lowest-friction way to access remote compute from a Mac without turning everything into a remote desktop / VM workflow.
Woah, this is exciting. I'm traveling but I have a 5090 lying around at home. I'm eager to give it a go. Docs are here: https://docs.tinygrad.org/tinygpu/
I hope it'll work on an M4 Mac Mini. Does anyone know what hardware to get? You'll need a full ATX PSU to supply power, right? And then tinygrad can do LLM inference on it?
Interesting, but cannot run CUDA or more to the point `nvidia-smi`.
Why does Apple need to make the drivers in a walled garden? Atleast they should support major device categories with official drivers.
Pretty misleading. This driver is only for compute not graphics.
If you could get Nvidia driver support on Mac’s I bet Apple would have sold more MacPro’s.
I'm writing scientific software that has components (molecular dynamics) that are much faster on GPU. I'm using CUDA only, as it's the eaisiest to code for. I'd assumed this meant no-go on ARM Macs. Does this news make that false?
My main thought is would this allow me to speed up prompt process for large MoE models? That is the real bottleneck for m3ultra. The tokens per second is pretty good.
Apple should update this page for ARM macs, now runs tinygrad on eGPUs: https://support.apple.com/en-us/102363
What are the limitations of USB4/Thunderbolt compared with a regular PCIe slot?
now can they please approve the linux kernel
The opportunity cost of Apple refusing to sign Nvidia's OEM AArch64 drivers is probably reaching the trillion-dollar mark, now that Nvidia and ARM have their own server hardware.