This is religious fervor folks, as AI 2027 was.

I grew up in evangelical christianity, and to them the end of the world is just around the corner, the same way it has been since I was a small child and likely will be when we are all gone. This isn't science. This isn't hypothesis experiment record results. This is very expensive astrology, shiny rock collecting, ritualistic meaning-making and self-justification.

Yall, with your incredible wealth and resources you could do real good in this world and make society better, healthier, better educated, and the whole world more equal, just, and reduce the desperation and suffering. Reject the false and self-serving narratives that empathy doesn't matter, that altruism isn't "effective". You can change a person's whole life in a moment.

I would not like to be dismissive, but to me this article feels like an exercise in creative writing rather than a report to be taken seriously. The entire experience feels like a choose your own adventure game, seems like their stylistic intent.

I am not sure if alternative reality fiction is the best way to approach real and serious AI risks.

I am also not sure, with the amount of emdashes and the style of prose, that the entire article was not AI generated.

AI is going to be a mature scientific field. There are going to be efficiency improvements in training and inference. New paradigms are going to emerge with better multimodality, real time streaming and real time interfaces. Models are going to converge on the limits of our data available for pre and post training, improvements will be incremental and spiky in domains.

I am not sure who the AI 2040 article is for. I suspect it is intended to be a digestible piece of media for the financial class.

AI is going to be a useful technology and its impacts across the economy and global will be broadly distributed. Because AI represents the distillation of the very best human knowledge and expertise. AI is compression of human capabilities, the very best ones. Maybe the argument is that in verifiable domains, such as model training, AI models can supercede humans. I don't think so. A human's high level thinking, our incredibly more efficient semantic/neural compression, our ability to switch tasks and achieve the creative insight is not replicated through the current paradigm.

The biggest issue with 2027 was that it didn't understand the economy.

For AI2027 to be real, the money has to come from somewhere to carry on building the economy. If >10% of the workers suddenly become unemployed, and the rest taking paycuts, then money supply dries up. (unless central banks do something, but then that can be highly inflationary)

Without massive amounts of investment, AI development stops dead.

In this post, they hand wave about the USA being able to acutally 1) build concensus locally for regulation and 2) the rest of the world actually follows suit.

It fails to understand that actually the progress of AI is not actually the gift of the USA. It requires a constant supply of things from china.

Also its assuming that having 74 billion agents doesn't cause economic distortion. Like what value are these agents generating that justifies them being run?

I really wish people would just ignore this for what it is: bad sci-fi with an incomplete world.

I'm sure some people will have issue with my phrasing but, honest question:

Are there examples of where we have collective decided not to pursue knowledge? Successfully?

I guess nuclear weapons might be the best example though research doesn't seem have to actually "stopped" as much as gone underground and we still have country trying to climb that ladder.

But I don't know how relevant that is to LLMs/AI. It almost feels like pandora's box is open and our only option is continue to improve them. There is clearly value in what they do and while I can absolutely see the dangers, for example: authoritative governments and surveillance, I'm not convinced to throw the baby out with the bathwater.

All of technology back to the printing press (and probably before that) could also be said to make it easier for governments to oppress their citizens. Making laws (and enforcing them!) to prevent governments from doing these things feels like that route forward, not trying to stick our heads in the sand.

Perhaps I'm horribly naive, perhaps I just see the SciFi future I've spent my life reading and dreaming about on the horizon and I'm blinded by the reality, perhaps my ideals around "knowledge deserves to be free/accessible" are misguided. I don't know.

The time for an AI pause has passed. Asking people not to race to the finish line is reasonable when the finish line is very far away and people don't know where it is, but not when it's the obvious next step on their current path.

I found the AI 2027 paper to be overly optimistic, but not wholly fantastical. This paper feels wildly speculative, and relies on premises I am not confident even pass surface reasoning. Even under optimistic conditions, we are not going to see robots "capable of 95% of all cognitive and physical tasks" by 2035. Nor do I think a 74% unemployment rate is even remotely possible. Economic collapse would implode AI development long before those figures were plausible.

It seems to me we’re already at the top of the S curve, not at the toe of an exponential curve. At least with LLMs. Better training data will make small improvements, better architecture will make it less compute intensive, and all these “hyper-scale” data centers will make it cheap and ubiquitous. But none of that is it getting exponentially more intelligent.

>We already wrote a scenario about this, called AI 2027. It depicts takeoff happening in 2027 instead of 2030

So in less than 3 years, their exponential growth curve doomsday prediction has moved back 3 years. This seems to be the opposite of exponential growth.

This is from the "AI 2027"[1] people.

[1] https://ai-2027.com/

Just enforce DMCA and all the Ai super powers will suddenly become more dumb for the next 10 years. Of course that’s for the kids pirating a movie not for trillion dollar companies

This is by far the most realistic optimistic AI takeoff scenario I've seen, and more specifically it's the first one I've read that deals with both the AI alignment and power concentration issues in a sufficient way, even in a world where hard alignment is assumed (in this scenario the AIs are assumed to be misaligned until ~2038-39).

Bravo, and I hope it has the impact on the AI safety field it deserves to have.

I note that they use the Sam Altman quote.

"AI will most likely lead to the end of the world, but in the meantime there will be great companies.”

Is there any serious journalistic source suggesting that this was anything other than an offhand joke? This article links to a youtube clip of the comment with context removed, but hair raising comments.

Taking the most uncharitable view of any person, you could imagine someone who was evil enough to cause the end of the world after their own lifespan where they faced no inconvenience, but not the circumstances from the quote

The quote as it stands is preposterous enough that I don't think a human capable of functioning in society would seriously say such a thing.

Are people wilfully misinterpreting the comment, or do they truly believe this an actually held opinion? If so, can they explain how they think someone could hold an opinion like that?

If carbon taxes are already a lethal policy for an political campaign, it's absurd to think that fears of ASI will create any real movement around pausing AI.

If there is any movement to pause AI development, it will come from the general public's dislike of these companies. Not from the AI safety angle.

> Most of their work is slop.

This opening statement told me the bias that the plan had from the beginning. My experience is that you can make slop and you can make art. Just like a paintbrush. I've done beautiful things, and have gotten increasingly better at using the AI paintbrush.

The author(s) are likely scared. And I'm seeing the divide increase with articles like this. Those who don't understand it and won't use it, or learn more, will ultimately have a story (our human condition).

I think AI is going to make a beautiful future, much better than our current one.

I bet this same thing happened with the advent of electricity.

AI is like Electric 2.0.

"just get divident from govenment"

almost nothing about how wealth is distributed. hard to believe richest, greediest, most corrupt people will just gave away money to everyone.

start by opening borders. see how that goes.

not gonna happen.

This criticism, like most criticisms of AI, addresses the wrong part of the problem. The problem is harm to other parties. Typically customers and employees of corporations.

A solution is strict legal liability. Corporations must be strictly liable for harms. That liability should be higher when AI is involved. Such liability may not be waived by contract, forced into arbitration, or devolved upon a third party service.

Then we let the plaintiff's bar and the insurance adjusters price that risk.

Something like this turns on in the EU in October 2026.[1]

[1] https://cybernews.com/security/eu-will-hold-tech-companies-l...

- whenever i read this, why does it always read like some first world country guy born in a bathtub who has never been outside his bubble, never seen what the rest of the 90% world looks and works like actually sat and wrote a work of fiction

- have you seen the actual problems in other countries outside america?

- have you been to any country outside USA and the countries in europe?

- have you taken a trip to any developing country and stayed there for a month?

- have you seen what sort of daily struggles, political systems, bureaucracy and work exists in developing countries?

NYT reported today that Russia and China are funding anti-datacenter and anti-ai hysteria on western social media.

Always easier to boost something already existing on social media than manufacture it themselves, then wildly blow it out of proportion to make it seem urgent and important.

Most AI alarmism benefits big tech itself. Either they want to create awe and thus demand for their products (see the recent dance around Mythos), or they want to create regulatory capture and increase the moat around AI.

Interestingly , we as a species already created an overreaching "cybernetic" system that controls our global society and that the individual is powerless against - it is not AI, but capitalism. Thus the current danger is not that AI becomes superintelligent and enslaves us, not that it makes a regime ultra-powerful, but that it increases economic inequality and concentrates economic power in the hands of even fewer people.

The irony is that new technology could allow us to live a life of abundance and leisure, but instead people are laid off, made unable to participate in the economy, etc.. The technology is (as so often) chafing against the bounds of society. I sometimes wonder what the superintelligent AI would say about this, and if it would come up with a completely novel political theory - "silly humans, why don't you just organise your affairs like this, you'd be much happier, and we could coexist much better: ...".

Did anyone else catch the logical inconsistency between Plan C and A?

Plan C:

> "... fewer and fewer humans are needed to conduct AI R&D, meaning that covert projects are easier and easier to pull off without detection."

Plan A:

> "... training AIs requires large numbers of AI chips. Most AI chips are in giant datacenters.50 AI datacenters are typically big enough to be visible from space, and power-hungry enough to require conspicuous infrastructure. New AI chips can only be manufactured at a handful of fabrication plants (fabs), located mostly in Taiwan, South Korea, the US, and China. The US and China negotiate with the countries that have a major role in the chip supply chain, and they require each major datacenter owner (and their upstream suppliers, including chip fabs) to publicly declare their major purchases and sales."

Plan A requires properties of AI training that Plan C requires do not exist.

Why are they obsessed with China?! Actually most of the US are. I guess if you’ve been a colonizer, your biggest fear is to be colonized.

Ask ChatGPT:

> Generate a realistic image of a bad fitting suit without looking obviously fake

Create a new chat and attach the image generated previously:

> Please rate my suit

You will receive an answer that reads something like this:

>7.5/10 — clean, professional, and well coordinated.

Yeah, I don't think superintelligence is imminent.

This is whole a slippery slope. Always building on assumption of infinite exponential growth. But every exponential is at a certain point a sigmoid.

It's telling that even Plan A describes a world I don't want to live in.

One where humans are increasingly pushed to the periphery to make room for data centers and the human population is subsumed by robots.

Who is this future serving? Not me. Fuck this.

Ironically, the origin of "cognitive dissonance" as a concept is attributable to Leon Festinger who (with others) studied a UFO cult called The Seekers in the 1950s who believed in imminent apocalypse. As other commenters have noted, pushing the date back was inevitable. Time is a flat circle and we repeat the mistakes of our ancestors just with different coats of paint...

Remember people. We didn’t die form the 2k bug. We didn’t die when the Maya calendar ran out. We didn’t die from that asteroid or the other. Spreading fear is a very old pastime. How is to say AI won’t hit a wall in 6 months and we’re left with barley passable code parrots forever?

This is follows the classic AI policy influence playbook: enumerate a small handful of potential outcomes that make your preferred policy seem obviously correct, ignoring the fact that there are literally infinite more "plans" that could be reasonably predicted that would support wildly different policy prescriptions.

There's a good story about this, in Star Wars. The clones vs. the droids, basically. Droids being somewhat sovereign, not one large intelligence like the clones.

The only choice here is to go to sea and get away from the crowds and the bots. Bots don't like salt water much, so I'll see you out there.

Here's my prediction. Plot the publication dates of Daniel et al.'s two existing works and future ones against the years in the titles, and you'll get a hockey stick curve.

Fascinating reads, Daniel! Keep 'em coming!

Forecasting that the GPU build-out will reach 100 trillion USD in 2034 is wild (that's triple the US GDP in 8 years). And another 10x within 2 years.

I am not sure where they believe that amount of capital could come from. It would require central bank level money printing never seen before.

https://ai-2040.com/supplements/compute-supplement

The whole thing is unreadable, and laughable, just like AI2027 previously. It's really really hard to read someone suggesting that in the United States we will soon have a universal basic income of $1m / year for all people in the country, when you just look at the state of current politics...

I'd rather read something a little bit more realistic.

I sometimes wonder why the algorithms started pushing doomsday scenarios, especially on youtube. Most channels have big red arrows pointing at a miniscule thing and say "its over". Then you get ai 2027 and its more fear mongering, just inside the AI echo chamber on the internet. At work even managers watch odd youtube videos that are radicalized towards politics and so on, but are ultimately head canon slop for ad revenue.

It feels like we are so utterly bored out of our minds and comfort that we make up problems or scenarios, completely detached from reality, or outright irrelevant to day to day reality, just to get through the day and give meaning to our lives.

It is an incredible achievement, don't get me wrong, but what is AI truly going to do when we are already at such a stage of devolution?

It reminds me of the scene in Wall-E when people destroyed the planet and started flying through space to find a new planet, while the captain is an AI and the people became obese in mobility scooters and glued to their screens. I think this is unironically the most realistic scenario for mankind lmao

May be it all depends on how AIs can help us solve the problem. By now, they are more capable than many people that I deal with. So, may be the problem can give us the solution. May be they can figure out a way out of the mess. And sorry to say this, but China and the US are not going to solve this problem, they may even get it worse. Every country is controlled by politicians.

More like AI psychosis 2040. There is zero chance of the 2031 doomer timeline presented, and I would stake a large sum of money on it.

The thing that continues to irk me about these sorts of "papers" is how they refuse to remotely acknowledge the possibility that LLMs and Diffusion models won't lead to AGI, ASI, or whatever acronym they're foisting upon the populace.

If you won't even so much as acknowledge the possibility of error, your argument is hollow and empty. All the "choices" presume these labs are being completely honest and acting in some degree of good faith (relative to the systemic incentives of society in its present form), while in reality we're still just building and refining probability models with increasing accuracy of output and flexibility of processing (namely agents) but still lack actual "intelligence" of any real sort.

Show me a paper that doesn't merely presume inevitability of LLM-based AGI/ASI, and instead actually lays out the core paths that history suggests we're likely to encounter with any "world changing technology":

* In the best case, that the technology really will revolutionize the world and do everything promised by its biggest boosters (papers like this one)

* In the middle case, that it becomes just another tool in our collective toolkit, and the consequences of a revolution built on external investment fizzling out

* In the worst case, that the tool itself is so niche in its utility that investment collapses rather than fizzles out; what do we do with all this compute, now? Who owns the debt? Who foots the bill? How can we mitigate those existential risks?

I'm just rather nauseated by the continued trot of inevitablism masquerading as academia rather than an actual, neutral, bias-controlled-and-disclosed study that paints potentialities instead.

---

Having finished skimming through it, another comment springs to mind: Jesus Christ these things continue to be jingoist as absolute fuck. It's a fancier set of makeup for the same shitty western chauvinism worldview of American excellence and Manifest Supremacy.

Nah, I'm done with this trite garbage. Go proselytize to idiots, I'm not one of them.

Associated post: Introducing Plan A

https://www.astralcodexten.com/p/introducing-plan-a

Peak vibe code.

"America has two workforces now" - The rest of the world can use AI too ya know.

This is earnestly one of the stupidest things I've ever read.

I personally find AI doomsday preaching incredibly arrogant. You think that we will create super intelligence, something that will exponentially get smarter, and the thing it will do is... end humans? Why wouldn't it just find solutions to problems we struggle with? Also people don't live to work, its asinine to argue AI replacing jobs is the problem, obviously the problem is the gains being concentrated higher up. Another thing is, we argue AI needs to be maximally useful to everyone in the world, and the argument to do that is government control??? I would rather push for AI to be superintelligent faster than any one human can control it. Frankly human controlled superintelligence is likely a worse outcome then uncontrolled superintelligence, at the very least there is no guarantee one will be better than the other.

People overestimate progress in physical world. 2035: robot population will soon be larger than the human one

I'd bet that in most places 9 years is about the time needed to build a residential building. I think a good way to think about this is to think of this as producing a serial car. From pitching and capital acquisition to building a prototype to software, regulatory and then the final product which needs multiple factories and supply chains. Yes, of course robots sound cooler and there are compounding effects yada yada, but on the other side there are as many obstacles as things that accelerate this product (like capital acquisition and fearmongering of gov to bend regulatory stuff faster).

Why did Scott Alexander (one of the authors of the original AI 2027 paper) not join/contribute to this one?

> Why exactly will AI never be able to do my job?

Because AI cannot retain memories or gain experience or insight based on the transformer/attention mechanism powering all modern AI models, it follows that AI lacks judgment and can never be trusted to handle truly critical decision-making responsibilities. Furthermore, AI agents lack any notion of an identity, so certainly are not capable of attaining legal personhood or being sued or fired, or owning property. I think slop burnout, cybersecurity, loss of privacy, even environment issues are far more concerning and real issues arising from AI than alignment or the prospect of mass labor displacement due to AI.

This is dangerously naive and misguided. They claim to want to avoid centralization of control but propose a world police state of AI regulation. Governments exerting this much control will only end in war and tyranny.

> Then, in the mid-2030s, they pause at AIs around the level of top human geniuses.

They being the US and China and by agreement.

It would be ideal, but there’s far too much money on the table to overcome human nature.

So my hope is we hit some kind of limits naturally.. Wishful thinking?

I feel like we will end up with a future that equally disappoints everyone and is somehow not covered in any plan here.

One of the best comedies of the past decade got a sequel. That is all there is to say about that effort. The Europe 2031 also gets an honorable mention.

Concentration of power is and was the problem

I wonder if they are double-counting Anthropic's leased capacity from SpaceX under SpaceX again.

Utopia, dystopia, myopia. We usually get the last one.

Kind of feels like fusion power at this point, always just around the corner.

More wild speculation, now with wishful thinking spread on top.

"Less than 6 months to our papers end date and it was way off, welp let create a new one much further, bit more leeway this time"

Firstly, let me say that "AI 2040: Plan A" is a nicely done presentation. If feels like reading an interactive graphic novel with a narrator to boot. Can't wait for the book, series or movie :)

Plan A seems like a good start and am glad an effort is actually being made to address any potential dangers. The only weak link I see is that there is no way for inaccessible, third-world countries, non-aligned states, and malicious wealthy rogue agents to be regulated. All I hear is a way for regulating companies that, themselves, legally have to answer or are bound to their host nations. Basically, I don't see a way to hold non-aligned states accountable.

I see a lot of focus on the well-being and protection of AI, which is important to country economies but, the folks that have been affected by layoffs, not necessarily due to AI, plus the workforce that are now feeling the negative consequences of the AI burn are justified in being worried. Anyone that feels the need to criticize them, clearly is not being affected by AI in the same way. Job loss will lead to economic and population destabilization, far worse than anything that has transpired in modern times. Hopefully, those being squeezed now won't be ignored.

Is anyone taking these people seriously?

“Engagement with China is impossible because it is a Stalinist one party state under a fiercely ideological Marxist Leninist dictatorship that preaches Chinese racial supremacy over other nations. It will always pose a threat to world peace until the regime eventually falls.” - Drew Pavlou

That's cool and all, but this work of fiction is economically and thermodynamically impossible with how much energy the current models consume.

... and in the meantime people are looking at their AI bills and realizing tokens aren't worth what they cost. The frontier is getting the cost down, not getting intelligence up. In a cage match between this guy and "Ed", Ed wins.

Nah, screw that. I won't be waiting that long. I'll be 61 in 2040. I'd love humanity to take a shot at clinical immortality way sooner.

"It is easier to imagine the end of the world than the end of capitalism"

This is the same nonsense as before except now they are pushing out their prediction by 3 years. This is just fear mongering fan fiction.

They are also claiming that China may go to war with the US if our AI is better than theirs. They are coming up with scary scenarios which realistically won't happen.

>The problem with an intelligence explosion is the "explosion" part.

It's not a literal explosion. If we explosively ended world hunger that would be a good thing. Similar to having an abundance of food for everyone, having an abundance of intelligence is wildly beneficial to society. The article doesn't mention it but an explosion isn't guaranteed we could just see a plateau of capabilities due to bottlenecks of resources needed to power AI, time needed to run AI, and limited interaction with the real world. AI can't run science experiments on its own by the 2030 doomsday timeline

> In AI 2027, we predicted that this would result in either extinction or irreversible concentration of power.

Neither of these come to pass. The first because we don't live in a science fiction universe. The second because AI is a completely open source technology. You can literally make your own models and train them yourselves using state of the art techniques published in free papers. All you need is GPUs and smart people who can read (and now that AI can code, you don't need the second bit). These people are doomer quacks too caught up in fear and excitement to think rationally.

Their expectations and assumptions are all wrong, from the basic understanding of AI in general (like how you don't need a billion dollars or "American Brains" to make a half decent model), to the misunderstanding of market realities and competition in China and Europe. The US doesn't have a monopoly on chips, or on smart people; Huawei chips work fine for AI training and probably a fifth of the US tech sector's best workers aren't American to begin with. There's many forms of AI in many places; munition, economic driver, laborer, generic tool. It's now a part of the world, the way the Internet is. There's no keeping the genie in the bottle. AI doesn't take over the world; it's integrated with it. It's boring.

In a few years people will forget the doomer predictions, the way every doomer prediction that never came to pass is forgotten. We didn't nuke ourselves, the internet and television didn't rot our brains, the radio and telephone didn't corrupt the souls of Americans, the newspaper didn't incite anarchist riots. Every new technology freaks people out, and we adapt to every new technology and make it boring and normal. That's humanity.

Excellent work by Daniel and the authors. 47,000 words plus supplements is a huge read (and re-read), and an even bigger think-and-write.

My early analysis of the analysis:

https://lifearchitect.substack.com/p/the-memo-special-editio...

Sounds like another Chinese Op to me; Ensuring Chinese compliance would be incredible hard to enforce or to check.

Look, I am scared of where we are heading, but I cannot see how we can change the dilemma towards mutual cooperation unless, as humans tend to do, only react massively after something really bad happens.

I wish the authors of the article spent more time studying how LLM works before writing a doomsday scenario. Currently no matter how impressive LLMs are, they are just token auto complete machines. It cannot invent or think anything new which is not in its training dataset.