I noticed the inference is routed through the gpu rather than the Apple neural engine. Google’s engineers likely gave up on trying to compile custom attention kernels for Apple’s proprietary tensor blocks iirc. While Metal is predictable and easy to port to, it drains the battery way faster than a dedicated NPU. Until they rewrite the backend for the ANE, this is just a flashy tech demo rather than a production-ready tool
show comments
temp7000
Is it me, or does the article sound like LLM output?
The pattern "It's not mere X — it's Y", occurs like 4 times in the text :v
show comments
blixt
I made this offline pocket vibe coder using Gemma 4 (works offline once model is downloaded) on an iPhone. It can technically run the 4B model but it will default to 2B because of memory constraints.
It writes a single TypeScript file (I tried multiple files but embedded Gemma 4 is just not smart enough) and compiles the code with oxc.
You need to build it yourself in Xcode because this probably wouldn't survive the App Store review process. Once you run it, there are two starting points included (React Native and Three.js), the UX is a bit obscure but edge-swipe left/right to switch between views.
show comments
codybontecou
Unfortunately Apple appears to be blocking the use of these llms within apps on their app store.
I've been trying to ship an app that contains local llms and have hit a brick wall with issue 2.5.2
Strangely, it is super fast on my 16 Plus, but with longer messages it can slow down a LOT, and not because of thermal throttling. I wish I could see some diagnostic data.
show comments
abc_lisper
Careful with using these small models. The other day, I asked it "Can dogs eat avocado" and answer was emphatic Yes.
This is not meant as a criticism, but people should be aware of their limitations.
show comments
conception
I’m pretty excited about the edge gallery ios app with gemma 4 on it but it seems like they hobbled it, not giving access to intents and you have to write custom plugins for web search, etc. Does anyone have a favorite way to run these usefully? ChatMCP works pretty well but only supports models via api.
Chrisszz
I just installed Google Ai Edge Gallery on my iPhone 16 pro, here are the results of the first benchmark with GPU, Prefill Tokens=256, Decode Tokens=256, Number of runs: 3. Prefill Speed=231t/s, Decode Speed=16t/s, Time to First Token=1.16s, First init time=20s
rich_sasha
Offline or not, I'm sure Google uploads every keystroke, phone orientation, photo, WiFi endpoints and your shoe size when you interact with it. To enhance your experience.
show comments
juancn
Gemma4 is still power hungry since it tends to activate pretty much every weight.
qwen3-coder-next uses a lot less since it seems to only activate ~3B parameters at a time.
My guess is that this is still close to tech demo, and a lot of performance is left on the table.
jimbokun
I feel like UX and API design are very under explored.
What are the possibilities of an Android or iOS device where the OS is centered around a locally running LLM with an API for accessing it from apps, along with tools the LLM can call to access data from locally running apps? What’s the equivalent of the original Mac OS?
Do apps disappear and there’s just a running dialog with the LLM generating graphical displays as needed on demand?
mistic92
It runs on Android too, with AI Core or even with llama.cpp
show comments
usmanshaikh06
ESET is blocking this site saying:
Threat found
This web page may contain dangerous content that can provide remote access to an infected device, leak sensitive data from the device or harm the targeted device.
Threat: JS/Agent.RDW trojan
show comments
politelemon
This is HN clickbait. No details or evidence, this is just generated for votes.
I think this should be flagged.
declan_roberts
I really hope this is a preview of the replacement for Siri that Google is creating bc these models are fantastic for their size!
show comments
deckar01
They still don’t render the markdown (or LaTeX) it outputs.
pabs3
> edge AI deployment
Isn't the "edge" meant to be computing near the user, but not on their devices?
show comments
bearjaws
Would love to see a show down of performance on iPhone vs Googles Tensor G5, which in my experience the G5 is 2 full generations behind performance wise.
DoctorOetker
does anyone know of a decent but low memory or low parameter count multilingual model (as many languages as possible), that can faithfully produce the detailed IPA transcription given a word in a sentence in some language?
I want to test a hypothesis for "uploading" neural network knowledge to a user's brain, by a reaction-speed game.
show comments
logicallee
For those who would like an example of its output, I'm currently working through creating a small, free (cc0, public domain) encyclopedia (just a couple of thousand entries) of core concepts in Biology and Health Sciences, Physical Sciences, and Technology. Each entry is being entirely written by Gemma 4:e4b (the 10 GB model.) I believe that this may be slightly larger than the size of the model that runs locally on phones, so perhaps this model is slightly better, but the output is similar. Here is an example entry:
I noticed the inference is routed through the gpu rather than the Apple neural engine. Google’s engineers likely gave up on trying to compile custom attention kernels for Apple’s proprietary tensor blocks iirc. While Metal is predictable and easy to port to, it drains the battery way faster than a dedicated NPU. Until they rewrite the backend for the ANE, this is just a flashy tech demo rather than a production-ready tool
Is it me, or does the article sound like LLM output?
The pattern "It's not mere X — it's Y", occurs like 4 times in the text :v
I made this offline pocket vibe coder using Gemma 4 (works offline once model is downloaded) on an iPhone. It can technically run the 4B model but it will default to 2B because of memory constraints.
https://github.com/blixt/pucky
It writes a single TypeScript file (I tried multiple files but embedded Gemma 4 is just not smart enough) and compiles the code with oxc.
You need to build it yourself in Xcode because this probably wouldn't survive the App Store review process. Once you run it, there are two starting points included (React Native and Three.js), the UX is a bit obscure but edge-swipe left/right to switch between views.
Unfortunately Apple appears to be blocking the use of these llms within apps on their app store. I've been trying to ship an app that contains local llms and have hit a brick wall with issue 2.5.2
Related: Gemma 4 on iPhone (254 comments) - https://news.ycombinator.com/item?id=47652561
Strangely, it is super fast on my 16 Plus, but with longer messages it can slow down a LOT, and not because of thermal throttling. I wish I could see some diagnostic data.
Careful with using these small models. The other day, I asked it "Can dogs eat avocado" and answer was emphatic Yes.
This is not meant as a criticism, but people should be aware of their limitations.
I’m pretty excited about the edge gallery ios app with gemma 4 on it but it seems like they hobbled it, not giving access to intents and you have to write custom plugins for web search, etc. Does anyone have a favorite way to run these usefully? ChatMCP works pretty well but only supports models via api.
I just installed Google Ai Edge Gallery on my iPhone 16 pro, here are the results of the first benchmark with GPU, Prefill Tokens=256, Decode Tokens=256, Number of runs: 3. Prefill Speed=231t/s, Decode Speed=16t/s, Time to First Token=1.16s, First init time=20s
Offline or not, I'm sure Google uploads every keystroke, phone orientation, photo, WiFi endpoints and your shoe size when you interact with it. To enhance your experience.
Gemma4 is still power hungry since it tends to activate pretty much every weight.
qwen3-coder-next uses a lot less since it seems to only activate ~3B parameters at a time.
My guess is that this is still close to tech demo, and a lot of performance is left on the table.
I feel like UX and API design are very under explored.
What are the possibilities of an Android or iOS device where the OS is centered around a locally running LLM with an API for accessing it from apps, along with tools the LLM can call to access data from locally running apps? What’s the equivalent of the original Mac OS?
Do apps disappear and there’s just a running dialog with the LLM generating graphical displays as needed on demand?
It runs on Android too, with AI Core or even with llama.cpp
ESET is blocking this site saying:
Threat found This web page may contain dangerous content that can provide remote access to an infected device, leak sensitive data from the device or harm the targeted device. Threat: JS/Agent.RDW trojan
This is HN clickbait. No details or evidence, this is just generated for votes.
I think this should be flagged.
I really hope this is a preview of the replacement for Siri that Google is creating bc these models are fantastic for their size!
They still don’t render the markdown (or LaTeX) it outputs.
> edge AI deployment
Isn't the "edge" meant to be computing near the user, but not on their devices?
Would love to see a show down of performance on iPhone vs Googles Tensor G5, which in my experience the G5 is 2 full generations behind performance wise.
does anyone know of a decent but low memory or low parameter count multilingual model (as many languages as possible), that can faithfully produce the detailed IPA transcription given a word in a sentence in some language?
I want to test a hypothesis for "uploading" neural network knowledge to a user's brain, by a reaction-speed game.
For those who would like an example of its output, I'm currently working through creating a small, free (cc0, public domain) encyclopedia (just a couple of thousand entries) of core concepts in Biology and Health Sciences, Physical Sciences, and Technology. Each entry is being entirely written by Gemma 4:e4b (the 10 GB model.) I believe that this may be slightly larger than the size of the model that runs locally on phones, so perhaps this model is slightly better, but the output is similar. Here is an example entry:
https://pastebin.com/ZfSKmfWp
Seems pretty good to me!
You are referring to the edge models, right? E2B and E4B, not the bigger ones (26B, 31B)...
There are many apps to run local LLMs on both iOS & Android
Do you know of a way of running these models on Android? Also, what does the thermal throttling look like?
is there a comparison of it running on iPhone vs. Android phones?
Is the output coherent though? I am yet to see a local model working on consumer grade hardware being actually useful.
I don't see the value in this post, are hacker news post being upvoted by bots?