Meanwhile:

> China’s Ministry of Commerce has led meetings over the past month with major AI companies, including Alibaba, ByteDance, and http://z.ai/, to discuss measures that would restrict overseas access to cutting-edge AI models, including models that have not yet been released.

> The discussions reportedly include not only closed-source models but also open-weight models.

> Future regulations could take the form of a tiered framework based on technological capability. Basic open-source AI models may be managed through a filing system, high-performance models may be subject to security reviews, and the most sensitive frontier models may be banned from public release or restricted to use within China

https://www.reuters.com/world/beijing-is-looking-curbing-ove...

I'm not convinced raw costs matter:

1. Compute costs collapsed since the advent of Cloud and yet hyperscalers still have fat margins.

2. Many open source office suites exist yet none compete with the ubiquity of gsuite or office. GitHub, Slack are similar examples.

3. Both Windows and macOS dominate the home desktop space despite free alternatives existing for a long time.

4. Many formerly open source infrastructure components like Redis and Elastic Search have Apache equivalents, but they still command healthy margins.

I understand the arguments for a margin collapse, but I don't see any historical analogues. It seems that enterprises will pay top dollar for service guarantees, integration, and someone they can sue.

It's nobody gets fired for buying IBM all over again.

I'll agree but from the other direction. AI continues to absorb my job as a senior systems software engineer (c/c++) and after a couple months I've only spent a few hundred dollars using gpt-5.5/5.6 and codex. I have no idea what people are doing to burn so many tokens but for me this is laughably cheap and every day I discover new capabilities. I don't care if costs go up or down, it's so cheap for what I get that I don't care.

Unlike the belief that frontier AI is expensive due to a high margin, and going to be expensive if there is no competition. My understanding is that, under certain circumstances (which is most likely true), the price will be driven down just because of profit seeking.

The frontier LLM labs run on a huge fixed cost and very low marginal cost. They need the economies of scale to make sense of the business (an incentive to expand their user base as large as possible). Imagine that you want to buy a few B300s to run GLM 5.2 and rent the service out to other people. How could this business be viable and sustainable in the first place? You need as many customers as possible. If you charge everyone $1000, you find fewer customers who can afford it. It rots the ROA if the servers are not utilized 100% (you would better buy less compute instead).

Also, the marginal cost for onboarding a new customer is low. And it's getting even lower when you have more customers. You wouldn't leave money on the table (especially for your competitors) if you want to maximize your profit.

By this logic, all frontier AI labs are incentivized to lower the price to maximize their customer base, profit, and ROA.

They have a vision MCP to make up for the model itself not having the capability natively: https://docs.z.ai/devpack/mcp/vision-mcp-server

I also found their web search to be mostly okay.

Furthermore, in case this is of interest to anyone, if you use their ZCode harness then you get bigger Coding Plan quotas: https://zcode.z.ai/en

Used it for a bit, it sits somewhere between OpenCode Desktop (still new but nice) and Claude Desktop (recent versions are good).

As for GLM 5.2 as a model - with max thinking it’s generally satisfactory, somewhere between Sonnet 5 and Opus 4.8, better than DeepSeek V4 Pro for sure.

Pricing wise, the subscription doesn’t seem as good as expected. I spent like 60% of the weekly limits of the Pro (50 USD) plan in one day, only because each 5 hour limit only gave me 20% to spend, otherwise it’d be 80-100%. Not even doing anything crazy, just parallel long form work on 2 projects with about 96% cache rate and at most 3 parallel code review sub-agents.

Their Max (100 USD) subscription would last me the whole week, but so does Anthropic for the same money and so would OpenAI. Off-peak is more palatable but I can’t just twiddle my thumbs at 9 AM to 1 PM local time.

Proper savings would show up with the Max plan and yearly billing, but that’s more of a tough sell.

Last month, I cancelled my Claude Pro subscription and instead used those 20$ to purchase Openrouter Credits. Most of my knowledge-seeking questions can be answered by Gemma4, for basic code editing, Qwen3.6 27b is enough, and for really difficult tasks, GLM5.2 doesn't leave me hanging. I'm by no means a heavy AI user, so I'm even saving money going the API Credit route and relying on the smallest possible model depending on the task complexity.

It’s important that none of these entities can collude to price fix. Having China be the competitor ensures that.

Basic microeconomics is still the easiest way to understand token economies. How is it not a competitive market (where profits go to zero?).

Anything A or O does to keep more margin, any competitor can copy or choose to undercut, and undercutting has the benefit of collecting training data. So what is going to stop gross profit of tokens going to zero except for collusion/price fixing?

> Of course, this was a hugely poor read of where the costs actually lie in AI. Training - while no doubt capex intensive - is a fixed, up-front cost. You spend hundreds of millions to train a model, then you are "done".

I don't understand this point that people make. If you're consistently needing[0] to train new models and the cost of training relative to the % improvement seems to go higher, isn't this just a constant cost that you continue to bear? The footnote seems to allude to this, but then sort of waves it away anyways. Also are there continuing incremental training costs to keep models relevant? Or do they only have knowledge of events up to the day they were trained?

[0] needing, because you have competitors and people expect more and more.

Metaphor i like is that it will be as cheap as electricty?

Do you know who is supplying your electricity or which factory it runs on? probably no, bc its a commodity and mostly settled and there is so many energy resources. some are alternative some are coal mines. And they all fight in the supply demand trade for energy which is happening real time ( think open router here)

And eventually the consumer wins bc of the abundance.

I think greatest example of abundance of cheap infinite intelligence will be not glm5.2 but DeepSeek V4 Pro max with $0.435 per 1M input tokens and $0.87 per 1M output tokens

>the least understood upcoming shift in AI economics.

Then proceeds to talk about something in the AI news every day. Hey, did you guys hear? Open source models are cheaper and their quality is increasing!

So, first, by no measure is GLM5.2 as good as Opus.

Second, yes, open source models will put pressure on margins...eventually. Everyone knows that. But do you think today's AI business model is the same as tomorrow's?

Seems like a pretty pointless post that still centers around output tokens.

In agentic coding, cached input tokens is 90% of the API "cost". It doesn't require GPU compute, and DeepSeek has shown that it can be done 50~100x cheaper with MLA/CSA/HCA, and a whole bunch of disks. This should collapse the margin.

I also think we’ll approach a point where increasing intelligence is not really going to suddenly improve most work tasks. I bet that’s already happened actually.

We’re oohing and aahing about models, when the ones a few versions ago did a good enough job for most of the dumb coding, etc we do

History is littered with the corpses of companies that had exceptional but expensive products that were replaced with cheap, good enough products.

There is mention of GLM 5.2's poor web search capabilities, but I see that as a harness responsibility.

I've set up my own SearXNG instance on my VPS and integrated it into Pi alongside the webfetch tool, and GLM 5.2 has so far been great at finding things. I asked it to give me the current news from an Austrian online newspaper that's difficult to parse because of its aggressive ad overlays. Both ChatGPT and Claude failed in their native chat apps. GLM 5.2 in Pi was clever enough to search for the RSS feed and gave me a detailed overview.

The lack of vision is a real shame, though. I've implemented workarounds in Pi that are okay, but they're not as good and the whole experience feels awkward.

I don’t see it. GLM 5.2 seems noticeably worse than Opus and especially GPT 5.5, the poor vision capabilities are also a massive strike against it since these are a huge improvement in the frontier models that can make all the difference when working on anything visual. Running it locally is its biggest advantage but for a lot of use cases that isn’t needed and is a burden to set up and maintain.

Companies like Amazon taking out loans to fund more AI infrastructure, coupled with AI companies that are massively overvalued and burning cash, coupled with companies saying there’s no ROI from AI investments, coupled with the margin from selling AI models trending to zero paints a nasty picture that can’t continue much longer.

The author mentions lack of good Web search. I've been using slightly modified crawl4ai and searXNG together with firebase for the rare sites that insist on throwing wrenches in the works of my LLMs.

I also have my fork of metamcp that replaces firebase MCP spec with my own that tells the model to use crawl4ai and SearXNG instead.

I've been using this wia Librechat with every commercial and open weight model I tested.

The search is way better than OpenAI and what ClaudeCode uses, but Gemini is way faster. That will change soon as I'm planning to put these instances in a DC with gigabit pipe.

Firebase is not cheap, but it retrieves everything, bypasses captchas and so on.... As long as one uses it for 1% of Web queries the cost is manageable.

I wonder if anyone has actually measured the difference in verification time between these models. A senior dev in a high cost of living area costs the company something like $200 an hour. If a cheaper model produces code that takes an extra 20 minutes to debug or verify because it missed a subtle edge case, you have already lost any savings from the lower API bill. It feels like the real moat for labs like Anthropic is the level of trust a reviewer can have in the output without reading every single line. Curious how much people trust GLM over something like Opus? Is there much of a marginal difference here?

An entertaining thread on just what a dismal rat race the life of a clanker shepherd has become.

"There is no doubt that using Z.ai's official API and subscription is almost certainly a non-starter, with their terms being at best weak and the deep connection to Mainland China."

This is the key statement in the article. I think people don't realize that these "open" weight models exist because giving away your product at a loss is a time honored marketing strategy. There's nothing guaranteeing that the next iterations will be open (remember "Open"AI?).

The Chinese labs are profit seeking companies. If they can't recoup their investment through API use, they won't be able to train more models. But if the argument is 'who cares, training models will be so cheap anyone will be able to do it ',then check the comment elsewhere on this comment section about free alternatives for consumer and enterprise software.

Oh... And the variation 'what we have today is already good enough for everyone' argument is just another incarnation of '640Kb should be enough for everyone'.

GLM5.2 is not the concern, it's that there's many good Chinese models and labs, glm, qwen, deepseek, kimik, mimo, longcat, minimax, hy3, Ernie, etc

They have figured out how to train, plenty of them and are consistently doing so

IMHO, cheaper inference means higher costs overall :) because everyone will use more thus driving up the investment required to stay current or to compete.

Switching models is also kind of easy but not plug-and-play. Most harnesses out there do very poor job with the open weight models. Unlike Opus, GLM 5.2 ends up in loops and hallucinates a lot more. If your harness is built on the expectation that the LLM will perform well, then switching to GLM 5.2 will be an uphill struggle. We had to refactor our harness and introduce more defences because of GLM.

The cost savings are substantial. Obviously it really depends on your workloads but it is noticeable cheaper for agentic work. Coding - I don't know. We do have some coding agents on GLM 5.2 and what I noticed with some landing page experiments that the results between GLM and Opus are identical - they might be using the same training data? Obviously Opus is still substantially better model. I don't think there is an argument to be made here but GLM 5.2 is cost effective and really good too.

Overall, we switched all of our internal agents to GLM 5.2 and because it is Open Weight we are in talks to get the model from certain geo locations giving us more freedom as well as extra protection.

Overall I think this industry will be in much better place because of GLM 5.2 and whatever open-weight models come next.

Agent systems only increase the gap between frontier and open models. Open models still experience more tool call failures, run longer loops, and get stuck more often. Until that's resolved (and it's obviously technically possible) people will be forking out for a better agent experience.

Agree with the web search point. (I would like for Perplexity to start to offer more models out of the box integrated, like they do now with OpenAI/Gemini models.)

The blog author complains of "lack of/poor web search capabilities" in GLM, but you can always use it against an MCP of which there are many. For applications where I am not concerned about my queries being passed through a US provider, I have had success with exa[1]

There are also other ways to give it context without web-search. For example the various MCPs that make `man` pages available.

I've also found GLM to be quite strong for coding tasks without the need for web search. So it also depends what you're doing.

[1] https://exa.ai/

People have been making claims about the commoditization of llms since chatGPT, and they've been wrong every time as quality and prices and differentiation have increased.

Somehow the blog post seems naive. Yes GLM 5.2 is good and cheaper per token, but margins are a result of supply and demand. Now demand for quality and quantity of tokens is increasing at least quadratic or cubic (more users * more tasks * more tokens per task). On the other side you have real infrastructure constraints on the supply side. Openai and Anthropic have large commitments and contracts that enable them to get access at a scale of compute that is not obviously going to be available for open source model hosts. And you see it, glm 5.2 inference is less stable and higher variance than any of the bigs labs.

Why is SpaceX not hosting glm 5.2? because they make more money with renting out to Anthropic and Google.

> I'd be very surprised if it wasn't more than 50% cheaper for nearly all workflows, for a very similar level of quality.

If your using pure API ... providers like neuralwatt cut that cost down even more by using energy as the actual cost. So GLM 5.2 is more expensive then GLM 5.1 on their service (those thinking tokens), compared to API costs, its dirt cheap. And way more tokens then the zai subscription delivers.

We are seeing a move towards more realistic pricing on actual consumption based usage. Be it DeepSeek, Xiaomi (MiMo), or zai's GLM via neuralwatt.

The main issue facing subscriptions a-la-carte usage, is that a lot of the heavy hitters really drain the resources. And that as a business model can not survive without ...

a) increasing the prices. b) everything goes to actual token/energy usage based billing but with more realistic pricing, and not the bloated API prices that are focused on companies.

We shall see what the future holds but things will change.

> It turns out that nearly every agentic session does a lot of web searching for looking up items

This is why Google will win the race over most of its competitors. They own search.

I run a 2 Claude setup (one architect, one coder) and have been using it extensively. But I don't like the fact that I have to pay $200/month to use it. I'm going to try GLM with my current setup.

Everyone is declaring GLM 5.2 as something that's really a big deal.

I don't know about that but based on my own experience with Deepseek v4 Lite alone (with high effort) I have no doubt in my mind that anyone claiming such great things about GLM 5.2 must be true because Deepseek v4 already is really awesome.

in cursor benchmark glm5.2 is on par with gpt 5.5 medium and sonnet for the same task from results and cost perspective.

The speed of generation for both gpt 5.5 medium and sonnet 5 will be dramatically faster. source : https://cursor.com/evals

I don't get the hype. It's near SOTA model that is not deepseek of this world. It an expensive to run model, and under certain tasks it is comparably cheap as closed source ones.

The problem is that the more AI eats labor, the more you can hike the costs until you pretty much can match the salary of the workers you replaced, with some margin, enough for the user to accept the cost. That’s what will happen in the next decade IMHO. price = base expense + what user accepts to pay

I am confused by this article landing on front page, it does not seem to contain any new insights but besides mostly reading ok falls apart when mixing up model and harness comparison in an amateurish way. Why would the subpar search tool zai provides be relevant for comparing models? They did not even mention the >capability< to use said tools but talk about MCP/ search providers as if thats not an implementation detail.

Where does the harness come in to play? I'd love to use GLM 5.2 for general chat, but I don't know of any harness that offers an experience close to ChatGPT or Claude (e.g., history, skills, projects) without requiring a PhD to set it up.

Inference margin is irrelevant.

That’s like a gas station saying they have 90% margin over pumps but still losing money.

I think the profits depend on how well they manage their fleet purchases (or possible sub-leasing?) to get high utilization without overloading or idle racks.

Because accelerators like H200, B300 etc. are highly parallel and designed to run like 200 or maybe 300 sequences at once (depends on the model, just guessing). I assume they finance the hardware and that cost per device or rack is the same whether each unit is handling 10 requests or 150 requests (aside from electricity).

And probably international customers factor into it to get good utilization over more of the night time. And it likely is something that they look at quarterly more seriously than monthly. The biggest risk to profits might be a downturn in business that causes some portion of the financed AI accelerators to go idle or get low utilization for some weeks (that they can't sublease).

Braintrust which is a really solid eval tool/platform just compared it to Opus 4.8 to see if it could preserve exact long context retrieval under prod serving constraints and it did really well. I think 6-12 months before OSS has Fable-esque models

Regarding the lack of vision part, if you are using Claude or opencode, I've made a skill[1] that let's you talk with any models in Claude/opencode mid-session. You ask "Have claude opus to look at this PDF for a second opinion" during a session of claude with GLM5.2 or opencode with GLM5.2

It doesn't need to pass whole conversation history as context (unlike /model), you can ask follow up to that forked model (which sub agents in claude doesn't support AFAIK), and you can ask models from opencode while using claude.

[1] https://github.com/kmcheung12/second-opinion

How long will that $4.40 rate persist? Until we know more about the real unit economics it will be damn near impossible to rely on steady inference costs or make them predictable at the enterprise level. Gonna be a wild ride for awhile.

How fast is glm 5.2 in western hosts? It's doing everything I want it to, but going through PRC host it takes like 5-10 times longer. Not sure if that is nature of modest or PRC computer infra/routing.

I don’t understand the argument here. The article doesn’t describe a collapse or the breadcrumbs for it. The only argument I can put together is companies hosting the open source models in house or use some service like Amazon that could potentially host them and so replace the frontier models. Data center and specifically infra to host llms is still the main sticking point given the security concerns about data going to china. The article doesn’t make these arguments coherently

the economics of this are a little counterintuitive.

is there a market saturation point for intelligence? how about for software? it seems like the more you have the more you want because you're trying to do more things.

as the models get smarter I get busier because I'm doing more things...

I don't think the writer has used top tier models very much. I have subscriptions to basically every provider, the difference between glm5.2 and opus is not even close, the gap is huge. raw benchmarks glm is impressive , but in practice these models are lacking so much. I had fable create a detailed implementation guide that explained how to implement everything in immense detail, it included all the libraries to use and versions. I then had deepseek v4 pro execute and it used old versions , different libraries and cut corners. Fable said about 80% was implemented wrong.

I had GLM 5.2 do the same, and it performed exceptionally better, but when it got stuck on something it would be trial and error mode going forward and have zero foresight for future issues that might occur due to fixes it was trying. the model severally lacks prompt understanding, and testing .

As long as the SOTA models are 'ahead' then there will be a big premium.

“Z.ai provides a replacement MCP for web search, but it's pretty awful and slow”

I’ve had good results with Tavily so far, might be worth checking as an alternative for agent search.

But what if they introduce a tarriff per token

I hope cheaper inference eventually means faster speeds at the lower tiers. I don't want to settle for 100 t/s, but I don't want to pay $10 per prompt either

GLM is the model that will sink the frontier labs.

Recall last year deepseek? And 18 month's later? What changed?

This article only promises to get into "the coming AI margin collapse" in a yet to be published "part two". This part only makes the point that GLM 5.2 is pretty good (no shit).

Is this AI written (or edited)? The word "genuinely" appears 4 times on the page.

Man, I hate how often people/LLMs use that word now. Maybe other people gloss over it but it's super distracting to me.

Disclosure - Fireworks kindly gave me some free credit to experiment with GLM to help write this article.

We'll keep saying the same things about every new free model just like we say the same things about every new frontier model but nothing really changes.

> I expect for most professional use the very lax terms around training and data retention will make this a difficult sell

I couldn't care less whether a chinese or american company reads my crap code.

I'm not working on state secrets but warehousing software for specific clients on a machine that has access to nothing but crap enterprise code.

how do massively negative margins "collapse"

If you use a good harness or add the right tools and plugins both the image and web search issues mentioned are non-issues.

oh-my-pi (omp.sh) handles images for text models out of the box - as long as you have any vision capable provider enabled, it will be used when you paste images to a text model. Rather than let it guess I configured it to use MiniMax M3 for this task (as well as other utility tasks like code exploration & library functions).

opencode has plugins that do the same thing, but I haven't used it since picking up omp and haven't tried them.

In open harnesses you can also configure your search provider(s) separately from the model provider - if you've got a ChatGPT sub you can use just their websearch for example. I've been using Kagi's API and found its cheap enough not to matter to me at all.

As for slowness, I'm not sure I'm really seeing that in terms of wall clock time. The author says GLM uses more tokens for reasoning but doesn't explain how they know that - frontier models don't provide nearly the entire reasoning trace. I have the suspicion that the author is not aware of that fact. I use Opus with Claude Code for work and I find it subjectively slower because I can't read its CoT trace. That is another HUGE benefit of GLM: I can't tell you how many times I've seen it start to go sideways in its CoT - usually due to something I didn't tell it - and I just stop it and give a course correction rather than wait a whole turn.

Overall I agree with the takes from the article and frankly its sad how much cope I see on Twitter (and even here) from people that think AI coding is busted once subscription subsidies are dropped. GLM is already good enough and cheap enough to use it at API rates - but it is MUCH more expensive than other open models that are also very nearly good enough.

In twelve months I'm confident you'll be able to get equivalent results at API rates for less than $1 per million output tokens, and more likely that will happen in six months. Deepseek v4 Pro is already almost there (and at only $0.85/MM) - and at least on benchmarks its already better than GLM 5.1 which I was happily using quite a lot before 5.2 dropped. I haven't tried Deepseek since I already have a z.ai pro sub that I locked in for $30 - at $72 its a lot less compelling.

Inference has been decreasing in cost by about 10x per year since 2023.

‘640kb of ram should be enough for anyone’

I would not be unsurprised if the US govt steps in to prevent this. They'll do anything to stop China getting ahead in the AI race.

There's the sanctions already implemented, next step might be giving these companies government funding, just like they do with military companies.

> Where it gets really scary for the frontier labs is how easy it is to migrate to open weights models. Both Z.ai and Fireworks offer both an OpenAI compatible and Anthropic compatible endpoint. This makes it absolutely trivial to use with Claude Code and Codex.

Yes the ease of switching is greatly appreciated.

Now the reason I tolerate Claude Code in my tmux sessions is because apparently Anthropic ain't playing nice with the subscription plans and other harnesses.

But I'm evaluating pi.dev atm and it looks amazing. To me being able to rid of that piece of vibe-coded underperforming, characters-modifying, turd that Claude Code is a big motivation to switch to GLM (I'll probably keep my OpenAI subscription as OpenAI repeatedly said they were cool with other harnesses).

It's also quite obvious that Claude Code is receiving new vibe-coded slop features after vibe-coded slop features in an attempt to lock you in.

To anyone thinking about switching to GLM: I'd say at least evaluate pi.dev and see if that wouldn't be an opportunity to kiss Claude Code and its "gameloop that converts characters from a headless browser to other characters to show in a terminal at 60 fps" goodbye once and for all.

Yes, margin on model inference is high with some providers. If you just wanted inference (at cost), you'd buy a GPU, or rent one from AWS or Microsoft. But you're not paying OpenAI/Anthropic for inference. You're paying them for a platform. Every feature OpenAI/Anthropic bake into their applications, models, online services, etc - anything that isn't pure LLM text generation - is a custom integrated add-on service that LLM weights do not include. Even if open weights became cheaper and better than OpenAI/Anthropic, most people would still pay for OpenAI/Anthropic, because they give you things the weights alone don't give you.

Comparing Z.ai GLM 5.2 to Claude Code w/Opus 4.8 is like comparing Linux Kernel 7.0 to Microsoft Windows 11. If you don't know much about computers, you'd say these are the same things. If you know a lot about computers, you know the latter has a thousand extra things that make a huge difference in what it does out of the box. Which one you use speaks to what kind of customer you are.

Sure, GLM 5.2 doesn't have vision; but an AI power user can plumb together any VLM with the text generation of GLM 5.2 in most AI harnesses, just like a Linux power user can combine the Linux kernel with KDE Desktop. Most people don't use Linux and KDE, because it's unpopular, difficult to use, hard to get support for. Instead they pay for Windows or Mac, because there's lots of support, with a giant company pouring money and effort into filling all the usability gaps, making it seamless.

Most people don't pay for the cheapest possible thing. They pay for the thing they can afford that improves their life while making it easier. An open weight alone is almost completely unusable by itself (like the Linux kernel), compared to an AI platform (a completely usable system). If you're constantly wondering about when open weights will reach parity with OpenAI/Anthropic, you're a Linux person. If you just pay $20/$50/$100 for OpenAI/Anthropic without thinking about it, you're a Windows/Mac person. There is nothing wrong with either of these groups, but they are fundamentally different, and always will be. An LLM weight is simply a different category of thing than an entire AI platform/provider.

I think OpenAI, Anthropic and SpaceX are going to envy the dinosaurs because there's not asteroid coming for them, there's three:

1. There will be no moat around frontier AI models in the future. China is going to make sure that happens. It's a national security interest for them. DeepSeek was the first shot across the bow for that but it won't end with them. There are other labs and there are non-Chinese actors too. The stratospheric valuations depend on there being that moat; and

2. Nobody seems to be considering what the next generation of AI hardware is going to do with current hyperscalar investments. We're about to go through this with the B100/200 move to R100/200 but a lot of the investments are probably slated for that next-gen. But what about 3 years from now when the hypothetical X100/200 comes out and doubles FLOPS and halves performance-per-watt. What will that do to existing investments? Some people are delusional and think that they'll get 10 years out of GPUs when 10 year old GPUs (eg V100) are sold for scrap and 5 year old GPUs (A100) cannot run DeepSeek v4 Pro. And people think the A100 is going to get another 5 years of use? No; and

3. Local LLMs are coming for remote usage. You can buy a 5090 PC for less than $5000 currently but you're limited to 32GB of VRAM, which will comfortably run 31B models but nothing really larger. Go to $12-13k to upgrade to an RTX 8000 Pro and you have 96GB of VRAM, which will run larger models (but certainly not, say, DS v4 Pro or even Flash). You have shared video memory products rapidly coming from NVidia's aggressive market segmentation. Things like Strix Halo and DGX Spark have severe limits on memory bandwidth (<300GB/s compared to 1.8TB/s for a 5090/6000 Pro and 3TB/s+ for server grade HBM3e/4 based GPUs). Macs could be real interesting in this space butr they lack the raw FLOPS with the M5 generation.

But what will this local hardware look like in 2-3 years? I think people will be shocked at how much better it will be with the Apple M7 Pro/Max generation (2028 expected) and the RTX 6000 cards at that time although I fully expect NVidia consumer GPUs to still top out at 32GB of VRAM to maintain that segmentation. And I look forward to what the next generation of the AMD Ryzen AI Halo platform will look like if they really try.

All of this adds up to these three companies needing to cash out before the music stops (IMHO).

I think the fixation on numbers of tokens and dollars per token is missing the point a bit. LLMs are quite useless without good tools. The article calls out search as one of them. And it's important. If you are coding, the tools are relatively easy: they are mostly open source and don't have a lot of authorization logic around them. Anyone with access to python and some access to a half decent AI model can pull together a decent agentic coding tool. There are many examples out there.

But if you look at the overall market, there's a rapid shift happening to non-coding tools and non programmer users starting to become very active. This kicked off beginning of the year with Claude Cowork. OpenAIs Codex and ChatGPT (they both have the same plugin infrastructure) is doing a lot of the same things. I've talked to a lot of non technical business users in recent months. There's a growing amount of people who definitely have zero interest in programming starting to use these tools and getting value out of them. This is going to rapidly scale to essentially most white collar users. Programming tools are becoming a side show to this market.

The difference here is that these people need connections to all their favorite protected data SAAS silos: MS office, Sales Force, Outlook, Gmail & GSuite, Calendar, SAP, Oracle, etc. The moat here is very different: it's mediated access to these silos in a compliant way. Anthropic announced a solution in the form of some MCP features. Those features boil down to getting access to all your favorite silos, if you sign in with the right identity provider. What's the right identity provider? The one that's whitelisted by the data silos you are locked into. Okta seems to have weaseled themselves into a position of power here. And it's all the other usual suspects. We'll see who is going to "win" that race but I bet it's going to be a pretty exclusive club with zero outsiders from China on that list. You can hack your way around some of those limitations. But doing so in a compliant way is going to be tricky.

And that's before you consider who's going to pay for this and what they are going to insist on. Corporate IT departments & data security policy compliance basically. What's the moat here? Secure & compliant access to all your favorite silos. Here in the EU that also includes data residency. The difference between sending all your data to Silicon Valley or Beijing is that of getting stabbed or getting shot. If it leaves the EU, you have a huge compliance issue. Most of the juicy corporate LLM usage is going to have to be fully compliant. I.e. hosted and controlled in the EU. This will be the same across the world. The least important choice right now is which model you use. The most important ones are about where those models run and what tools the models running there have access to and how that is governed.

On paper, OpenAI, Anthropic, MS, and Google are pretty well positioned here. Not necessarily in that order. Most others are still figuring it out. But they'll have a moat of data center ownership in the right regions + mediated tool access that works out of the box.

i would use glm 5.2 if the servers weren't in china

i mean i guess my employers wouldn't know the difference

but i'd like to play it safe and keep everything in america

The fact that these Chinese models are getting close to “Opus-grade” despite costing 6x-8x less is huge.

As the token bills start to come in, those economics will be harder to ignore (regardless of the origin of the LLM); especially as there will be many CIOs sweating over their quick and costly AI initiatives showing little ROI.

My hope is that the EU also steps up their own competition in the frontier model space so that it’s not just China v USA.