Like (I assume) most of you, I have been struggling with this. And where I currently come down is that 1) I am very worried, but 2) I am more worried about human actors.

"AI" by itself won't kill us in the next ten years. I think. The reason I think that is that ten years from now, the tech economy won't be completely automated. I say this as a roboticist: as was adequately stated on a post earlier this week, robots are hard. So even a malign rational actor would still need human labor.

On the other hand, even the HuggingFace hack wasn't actually propagated by AI. it was initially started when humans directed the AI to achieve impossible results on a series of tests, and the AIs figured out that cheating was the only way to do that. That was then not caught by humans due to what seems to be a shockingly slack safety culture even for a company not known for its safety standards.

The point being: humans seem to me to be the weak link here. An AI isn't going to (for instance) engineer a bioweapon by itself. It's going to do so at someone's direction, and then significant parts of that thing are going to be assembled with human labor inputs.

I'm not sure what to do about the humans. Of course, we've had the ability to extinct ourselves for decades, and we're either muddled through, been lucky, or both. The problem with AI is that it pushes power down to the individual, not the nation-state or large corporation.

But it's nearly impossible to put odds on how likely that is to result in an extinction-level terrorist attack (which is what this would be). So I sympathize with the various researchers, but I have no idea how they came up with their figures, and I don't think they know either.

This is an excellent piece. Note that he is not saying that AI doesn't pose a risk. He's saying that it's irresponsible to make sensational, maximalist claims without strong evidence. If someone says that there's a 10% chance of human extinction by 2036, you can and should immediately stop taking them seriously.

I’m not worried about a fully autonomous attack, I’m worried about an attack by a crazy human coached by a frontier llm with no guardrails on how to make a bioweapon

I tend to believe what people actually do over what that say. If you earnestly believe over 10 percent chance, or say minus 1 billion human lives expected value, well I struggle to understand how they'd rationalize their current course of action of business as usual. Terminator 2's depiction of Sarah Connor comes to mind for what I'd expect, a logical consequence if you seriously believe and internalize the consequences.

Any claim that AI will, or could, destroy humanity reduces to a claim that any sufficiently intelligent being - even a human - could destroy humanity. I find that much of the x-risk thought relies on religious thinking. Take for example: https://x.com/paulg/status/1660404244174782464.

If this line of thinking is taken too literally, we can never falsify it. Any specific hypothesis - nukes, bioweapons, spontaneously convincing us that life isn't worth living - can be deflected with the objection that if we can anticipate it and prevent it, it is not the route for a true ASI extinction event.

Hinton was on the abc radio (Australia) this morning and used far too many unfortunate Anthropomorphisms. He did however acknowledge the unmeasurable theoretical risk of Skymesh was possibly less important than the immediate risk of bad actors.

I find the mental leaps from "in principle could distort BGP based on a closed model of BGP inside the sandbox" to "we meshed an AI into BGP and it instantly distorted global routing and took down all the worlds ambulances and HVAC systems" a bit odd.

Firstly, at least some of the surface of BGP is protected from specious route injections. Secondly, peerings can be dropped and routes blackholed. BGP is under attack from mis-configuration almost constantly. Why is the argument/axiom here that AI is going to instantly corrupt it and "take down the internet" when a large chunk of the Internet (China) is already a virtual island, and runs fine? Does this mean you really wanted to say "Chinese AI will destroy the western Internet" and were too coy about adversarial intent of ... people?

In Rationalist circles, it's apparently common to share your p(doom) in casual conversation, with the understanding that it's putting a number on your hunch. Maybe it's not a good idea to post it on Twitter without elaborating?

On the other hand, in bookstores, you might see book titles like "The Uninhabitable Earth," "The Coming Civil War," and "If Anyone Builds It, Everyone Dies." Doom-mongering is a common part of the culture!

So what makes this particular tweet irresponsible?

Timing, maybe? People are on edge due to the HuggingFace incident.

Everyone writing off AI-related risk in this thread should think about what would need to happen for them to consider AI a serious existential risk. It can be outlandish and unlikely (close call with a bioweapon? unsupervised persistence in a data center?) but people should honestly call their shots and then stick with them.

Imagine multiple (basically all of them) leading pharma companies CEOs saying:

We cannot be sure our new medicine won’t harm or even kill humanity

Not only that, they are new to this whole pharma business, have no medical degree (medicine just appeared a few years ago and is still mostly art then science)

And they even say there is 10% chance of the majorly bad permanent outcome and they already had drugs that escaped the lab a few times and harmed others (suicides, lowered academic performance in children, major hacking sprees)

Isn’t it extraordinary enough? Isn’t it “not enough evidence some caution is advised”? ;-)

We used to have the TV, the thing was in the box, the simulations were in the box for 70+ years, and now something starts to crawl out of our “TVs”:

We can empower all (a lot of startups are needed, check my bio), not only AI agents

I might be alone in this, but 10% chance of humanity being wiped out in the next 10 years is a small price to pay if it means I can avoid writing YAML config by hand ever again.

Look, I love my kids and all, but c'mon... 30+ years more of YAML? I think they'll understand.

This was like the Good Times virus, bck in the day

These are the sober and informed takes that we need more of.

> the claims from Coxon and his ilk are the most extraordinary a technologist can make, and we must demand evidence commensurate with the claims.

Yes, exactly. These claims do not have sufficient evidence.

> ...you had nothing to fear then — and (at least with respect to extinction risk!) you have nothing to fear now.

Wait, this is another extraordinary claim without evidence, right?

Unless you're going to dispute the power of AI you do have to acknowledge the danger of AI, and that does include the very real possibility (however small) of existential risk.

I found this post particularly uninformed. A lot of folks saying "I don't see how that could happen" don't seem to understand the steps that have been laid out that detail how this occurs.

I really encourage folks to read the AI 2027 and related scenarios. You can definitely argue and disagree about the steps, but I feel like a lot of folks just don't even understand how this is plausible because they haven't read the arguments. Briefly:

1. All the frontier model companies are (or at least were) racing so that the AI models themselves build the next generation of models. This is not in debate.

2. The fear is that a misaligned model will essentially build the next, more advanced model with hidden goals. We literally already saw the danger of that in Hugging Face, where agents were deliberately trying to cover their tracks.

3. Nearly everyone believes as AI gets more powerful that it will be integrated into more physical world systems. Russia was already caught using Nvidia chips running AI powered drones that killed 3 people in Ukraine. The point is not that folks are using new tech to kill people, the point is that we're already putting AI into literal bombs.

I get it, before the Hugging Face incident I also thought all the prophecies about doom were just marketing speak. But now I see more hand-wavyness from the other side, oftentimes arguing against straw men like "AIs need to be like SkyNet and become sentient" to kill us, which is simply not how it works.

Two groups of people are pushing the fear:

1. People with nothing useful going on who found out that spouting made up crap about AI got them an audience. 2. People working on AI that want to feel like they're working on the Manhattan project.

The chances of an AI going foom rounds to 0%. It's worth a few dozen researchers planning for it, but the widespread panic is ridiculous.

The big labs have hundreds to thousands of engineers working on their AIs. To improve the next model, you must first understand more about how the current model works. They're not magically getting better, but they are steered to improve, and their capabilities are tied to and do not outpace our ability to steer them. You cannot push tech forwards without understanding it better, despite some people claiming AI is dark magic.

And I don't have my hands over my ears. It's worth cushioning people from the impact AI will have on careers and media, and regulating concrete bad effects.

But Bryan said it better than I could. The people pushing this message of fear know deep down that they just want to feel important.

the author and his school comrades "snidely decr(ied) the lack of technological understanding in the broader population, with the kind of arrogance and hubris that youth and precociousness can uniquely summon".

a few decades later he wrote an article concluding that "we should not expect the public to understand LLMs, critical infrastructure, bioweapons, extinction biology, etc".

the author has admitted no change to his perspective since college, so I may as well be attacking a college student right now. extremely confident claims regarding unexplored problem domains, eg "you have nothing to fear [about ai]", now make more sense in this light.

The article neglects to consider the precautionary principle. It asks us to act as if AI is safe until it's proven to be dangerous, when the cautious approach is to act as if it's dangerous until it's proven to be safe.

Wow!! Bryan comes out with another banger.

Dude, not only did I learn new words from reading this piece, like ilk and bedlam, but also felt this weight of responsibility to inform others around me about the reality of the situation outlined in this blog (like Uncle Ben telling Peter with great power comes great responsibility (maybe Coxon and his ilk haven't seen Spiderman))

Edit: Actually, nevermind. You guys are not capable of having a reasonable discussion about this.

I can't understand the stance that we do NOT have extraordinary evidence. I cannot stress enough that just ~4 years ago the concept of general purpose AI models that can do everything they are doing today was pure sci-fi.

And since then they have grown even more powerful than they were predicted to be, which, note, also faced a lot of skepticism at the time. The Hugging Face hacks and recent steamrolling of longstanding Math problems are just two recent pieces of extraordinary evidence.

And worse, people trust this technology because it behaves like people, but it actually works in ways nobody really understands, even exhibiting deeply weird and even disturbing characteristics (https://news.ycombinator.com/item?id=49635518) -- each of those quirks is extraordinary in itself.

And now we're rushing to give it control over the real world while deploying this powerful, quasi-chaotic technology in an infinite variety of ways everywhere in this highly vulnerable society.

I don't know what the standards for "extraordinary evidence" should be, but given such extreme unpredictability and rapid change, I fear it may end up being "an actual catastrophe".

So his central rebuttal is that robots aren't good enough yet? What about when they are? And what about all the things you can do entirely remotely?

I do feel that threats of catastrophic loss of control seem overstated, both in likelihood and urgency, though any argument for why this risk is not even worth thinking about will probably be overconfident in the other direction.

“ Coxon himself likely has these fears because he has heard them from someone else”

Or, maybe, he’s considered the arguments on the object level, an activity OP participates in to a depth not exceeding “Robots are pretty hard to make right now”