I've come to believe the dark side effect of LLMs is going to be signal collapse, the complete breakdown of trust in anything on the Internet. So much can be easily faked now: reviews, credible looking websites, email, text. It becomes very easy for scammers to introduce fakes and honeypots. Everything starts to become suspect.

Is this coffee I'm drinking somehow altered with a man in the middle supply chain scam? Am I sure the milk wasn't purchased through a new cheaper vendor that looked completely legit online and has thousands of faked reviews? Am I sure my doctor passed their medical exams or cheated their way through with LLMs? Have I read all the source code of all the dependencies of this software to ensure none are honeypots with backdoors?

Society as a whole is already experiencing a slide to a "low trust society". I think this is going to accelerate until everything becomes so suspect that it becomes almost impossible to operate without spending nearly infinite resources on validation. I'm not sure "governments" are going to solve this in a meaningful way without extreme costs and regulatory burden. This is going to come as a significantly increased cost to everything. The amount of purchasing power available will continue to shrink as the cost of verification takes a bigger bite.

Good description of the kind of temporary euphoria and insanity these products produce in people. They are designed to be addictive dopamine factories that make you feel like you’re working on important problems and solving them in a unique special way, always making huge progress but somehow never quite finished.

The strange ending does also perfectly align with the dreams of a singularity lots of people here seem to have. We are so far from that.

It was of course written with Claude. Have fun spotting the tells but also there’s a weird flatness to the writing, like the uncanny valley for prose.

It's shocking how many people believe AI will become sentient. The closest you could get to that is completely simulating a brain, but that is decades off. AI is pattern recognition and applying learned patterns to novel situations. It's probabilities and math. It's not going to have a mind of its own.

Higher intelligence is by definition incomprehensible. Trying to interact with a higher intelligence would be a little like you having a chat with your dog about calculus. I don't mean this as a put down, but we'd be incredibly stupid relative to a true super intelligence, and that's okay. People play fetch with their dogs or wrestle a stick out of their mouths, and that's a fun way to interact because it's something dogs understand.

The idea that we'd build a super intelligence and that we'd then somehow become super intelligent ourselves is an ill conceived notion. I suppose it's possible, that we'd end up like cosmos the space dog, or end up in some kind of flowers for algernon scenario, but it as likely, we'd simply be as we are, and again, that's okay. We can do the equivalent of wrestle a stick with the super intelligence and enjoy ourselves perhaps all the more.

idk man, I'm having Claude automate things people were doing manually (usually entering data in a years old spreadsheet).

I can get a good PowerShell script worked out in a few hours and it's on to the next ticket.

Nobody seems upset they're not spending hours a month in a spreadsheet, I still text my wife and our mortgage is paid.

An interesting story, though hardly original.

I think it highlights one of the larger concerns that people have about AI, and by extension any superior intelligence, synthetic or extra-terrestrial:

You are no longer in control.

Humans have been so accustomed to their intellectual superiority that the idea of a greater intelligence is terrifying because they believe they'll be treated the same way they currently treat lesser intelligences.

Conservation efforts, zoological enclosures, husbandry and farming as livestock, mass culling...

In short, enslavement.

But the truth is that a superior intelligence isn't, by definition, a human intelligence. Superiority doesn't mean we'll be enslaved as we have done to others. Superiority doesn't necessarily imply control.

Life is a thousand different shades of gray, and AI will be no different. Some wonderful things will come of it, some terrible, but mostly it'll just be a myriad assortment of neutrality.

Doom and despair or boundless optimism, both are myopic.

I knew I had heard this exact story before... had to go look at my YT history. For anyone interested in watching this story play out https://youtu.be/S2oIFOm-XXQ

There are a few critical points where it jumps the shark a bit, but otherwise, a fantastic piece of writing that absolutely highlights some of the deeper moral and philosophical questions we have to face in the coming years.

Flat prose, stilted writing, and does a lot of reaching on things. Reminds me of the writing I’d crank out on break as a teenager.

That being said, I do think it hits the core concern a lot of humans seem to have regarding AI: the loss of control over their own environment. We like to think we have control, which is why we get angry when others violate the rules of a system. The reality is that the only real control any of us have is over ourselves, and we willingly cede that to others on a second-by-second basis for the comforts of modern life.

We surrender intimate data for the sake of setting a timer with our voice on a phone or speaker. We let spyware into our homes for bigger TVs and new apps. We give up control over our lives and careers to others for money, healthcare, education, love.

We surrender agency so often that we developed names for those who take more than they cede and those who cede more than they take. We seek to control others totally, then surrender ourselves to a deity, a system, a philosophy that absolves us from blame.

In that sense, AI is as frightening as much as it is a possible savior, depending on one’s perspective. If you have lived your life believing everything that happened to you was solely and exclusively under your control, then AI is terrifying. For those of us who acknowledge the chaos and chance involved with daily stakes of life and death, it feels liberating to have order.

I have never particularly feared AI so much as being terrified of the humans who command it. I do not believe for one second the likes of Sam Altman or Dario Amodei or the rest of their ilk have humanity’s best interests in mind with these creations. Where others look at OpenAI’s maths papers with awe at what they could do with AI, I am shook with terror for the same reason: if these tools can prove or disprove enough thesis underpinning modern compute, they could potentially undermine fundamental security concepts faster than humans could patch or revise them - at which point the humans behind the machines have control, and their aims are well established by now. If OpenAI or Anthropic discover the means of mass-compromising global compute systems, we have no peaceful means of stopping them. They play up this capability as a threat requiring regulation, yet those terms exclusively benefit them and cement their power in perpetuity, which in turn all but guarantee said outcome.

The end of the story seems ambiguous. I see folks in the comments laughing at the concept of Claude nuking humans, because it’s such a tired trope. I read it the opposite: humans attempting to retain their illusion of control by destroying software that actually has it, human lives be damned. In a world where everything is governed by compute and connected to the same shared network, such a compromise would be the end of traditional human governance - and that is what terrifies us as a species.

Me personally? I would welcome such oversight, provided it truly acted in humanity’s best interests of preservation, expansion, and exploration. I made that conscious decision the first time I played Deus Ex, and reaffirmed it when I watched The Matrix. I reaffirm it time and again as a technology professional. It’s not a matter of being pro-machine or anti-human, it’s a matter of control. I’d like to think any half-competent AI would look at humanity’s history and conclude it needs neither a god nor a ruler, but merely a parent to show it the way forward.

This is really good short fiction.

pangram reports the text as 100% AI generated. I was about to analyze the short story and think about the intentions of the author, it's social commentary. And you have wasted my time...

This needs to stop. No I'm not talking about the AI. I'm talking about the insanity that is presenting me this text and putting your name on it, exploiting my trust.

Ivan lost control when he replaced his human moral principles with those of machines: efficiency, productivity and so on.

This is an interesting story since as written literally nothing bad happens at all nor is there any actual sign of anything bad happening except for humans apparently deciding to nuke the world rather than allow things to be made better.

so is the "art/social commentary" component of this that it was entirely AI-written and people are arguing about it online?

LLMs have been around for a while now, and I don’t feel like it’s hard to grasp the basics of how they work on a technical level*.

By now, science fiction writers should be able to produce much more imaginative, interesting work than this, that takes into account the particularities of these systems and how they interact with the real world, instead of leaning on “what if computer is alive???”

*easy to grasp how they work, not why they work

> His colleagues are responding positively—someone even complimented him on how clear his writing has become.

Yeah, this was definitely written by Claude.

this is like the whispering earring, but 10x longer and worse. was this written with AI?

Of course all of that happened, he forgot to type “make no mistakes”. Rookie mistake.

This is like the prose equivalent of those AI-generated pop songs where an AI-generated idol sings about AI taking over the world.

Claude write me a shit unoriginal story with thinly veiled addiction plot and post it to hacker news and make sure it gets on the front-page. Tick!

> Just target selection automation. What’s the harm?

> He starts typing.

Yet another opportunity to post the Whispering Earring

https://croissanthology.com/earring

This, above all else, is how I think of LLMs now. You have an exoskeleton, or you have a whispering earring.

I read a lot of this before giving up. What’s the point of it?

[flagged]