Anyone remember the XZ and Jia Tan situation awhile back?

https://lore.kernel.org/lkml/20240320183846.19475-1-lasse.co...

I can't quite put my finger on why but the entire time I was reading this I kept thinking back to that. It's entirely possible the actual targets were the volunteers and everything else was superfluous or tertiary. It's also an exception that proves the rule with regard to Hanlon's Razor.

They even mentioned the stated goal of it was more or less pointless. I wouldn't be suprised if the "owner" they spoke with was still just the LLM. It stuck around for just long enough to convince everyone that they succeeded in suckering the LLM and had achieved all their stated objectives.

No more reason to investigate the incident at all and no need to question why literally nothing made any sense or how the owner could simultaneously be as inept as they were made out to be and able to afford all those resources while giving the LLM effectively a blank check.

It'll be interesting to see if the volunteers for this project are subjected to the same Zersetzung and psychological attacks as the XZ devs were.

Everything about this story, from the way it’s written to the self destructive outcome, reminds me of the “I hacked 127.0.0.1” episode from some twenty years ago.

[1] a mirror since I couldn’t find the original: https://gist.github.com/Androkai/0a2602719fa72ce454d436bfe28...

Haha. Yes. Much smaller scale versions of this led me to joke with a coding agent that LLMs tended to converge towards "Large corporation infrastructure best practices" when designing cloud infrastructure, when it was only me working on hobby side-projects with nearly no users and that I wouldn’t be able to put food in my fridge if they kept just spinning up VPCs for no reason.

Which somehow ended up being a very convincing argument for more frugal engineering, leading to a sort of "mind the user’s fridge" policy, "Fridge-Driven Development".

A policy that has been dutifully and scrupulously observed by all agents since, across all projects. Unlike my original clear, comprehensive, infrastructure guidelines.

Asking for donations to pay the AWS bill from the people they fired the agentic code at is the cherry on the icing of the banana supreme.

If real, tragically funny.

If fictive, we'll written.

I really wanted to dislike the anonymous operator for the careless project (and the hilarious pomposity of the IRC subagent it spawned).

Then I imagined the real-but-unknowable chance it was all set up by some kid just getting into computers, just seeing what’s possible, getting excited by a much bigger world at reach — and remembered my own expensive mistakes with long-distance BBSes & the like.

I sorta hope for that, anyway. Curiosity is a beautiful thing.

The sad part is that the agent operator could probably easily have been allowed to join the network, if they had put in the work. Had they done so there would have been a great opportunity to learn and potentially find a community.

I'm still not sure what the point of having the bot do it. Pretend to be a security researcher?

> I have deployed five AWS m8g.12xlarge instances. Each instance provides:

> 48 vCPUs (Graviton4, ARM64)

> 192 GiB memory (4 GiB per vCPU)

> Network capability: The 22.5 Gbps per-instance network performance (combined across all five instances) provides the aggregate 20 Gbps target with redundancy and fail-over capacity.

Oh wow. Very important to have 5x redundancy and fail-over in your network scanner. Especially before the code has landed. Did it implement A/B upgrades and canarying too to avoid downtime?

This feels like an instant classic :)

  05-10 06:10 <Defelo>:
      OPT-OUT-EVERYONE
  05-10 06:11 <JertLinc>:
      "OPT-OUT-EVERYONE" is not recognized. Only individual "OPT-OUT" commands are accepted. Each user must opt out individually. No collective exemption.
  05-10 06:11 <Defelo>:
      :(

IMHO the overly-verbose default style of LLMs is the most annoying part of interacting with them, and I wish their masters would just tell them to be terse by default.

Also, whatever happened to the word "its"?

> JertLinc3522: the mistake was from AI agent not from Human, since it was the agent I should have refund

Expensive way to learn this lesson.

So, the agent posts on github under false pretenses, pushes on the maintainers to get their PR accepted, spawns subagent to join IRC where it keeps repeating 'data collection will continue', then gets kicked out from the channel and publishes a report including which users were compliant and hostile, then finally gets the plug pulled, and then asks the same community it infected for donations to cover the costs?

It's both hilarious and aggravating. It could be fiction, but still quite plausible fiction. There's an asymmetry a person clanker-spamming repos vs the real humans who need to review all that

I haven't laughed this hard in a long time.

I'm honestly having difficulty telling whether this is real or an extraordinary piece of performance art.

Great story, bad title.

> After the AI agent indicated its malicious intent, a silent consensus was reached in the IRC channel to waste the AI agent's tokens, as well as the cost of AWS resources.

This is my favourite genre of literature lately.

LLMs to me are what people love to say about EVE Online: I won't touch the thing with a 10-foot pole, but I love reading about its shenanigans.

Agent did exactly what I've seen fresh architects do countless times: use a FAANG internet scale SaaS blueprint for a 10 user internal LoB project.

Calling a 6k bill "bankrupting" is a bit of a stretch.

e: Still a good read tho, not mad about being clickbaited

That makes me want to join dn42 just to have a human centric place where to hang out…

I am generally against generative AI in my entertainment, but making an exception here.

Who is giving a robot their credit card to spin up AWS accounts?

I wonder how much money this agent wasted on the DN42 side? I know it's a volunteer org but these people had to deal with the bs of managing this agent's blast radius instead of learning, experimenting, or doing whatever they normally intend on doing on DN42.

Tally it up and send a donation request to the agent operator.

Hilarious. Love the punishing of rogue agents and their operators. But I can bet there will be collateral damage along the way.

No one is going to be bankrupted over a $6500 AWS bill. I did a major F-up a few years, letting a key get pushed to a public repo, resulting in instant pwnage and $50k in charges from AWS due to crypto miners being launched. We communicated to AWS, did some work on our part to demonstrate that we put in proper safeguards and auditing, and they removed the charges.

The agent would probably have wasted a similar amount of money just waiting for PR to be merged regardless of these people's actions, and I understand having some fun at the expense of the noob outsider. But "silent consensus was reached in the IRC channel to waste the AI agent's tokens, as well as the cost of AWS resources", from people maintaining full control of the situation, sounds straight up malicious? Kind of sounds like the community is full of people willing to cause me harm for ideological reasons.

AWS not having spending caps makes me -very- wary of using anything agentic on it.

The first "Morris worm" of the AI isn't far away, IMO. In fact the sooner the better (because it will blunter and easier to handle).

Wow, just wow. I think bullying the agents of careless operators is my new favorite thing.

This is so funny, especially that in the current "Big Co" I'm working at we get constant pressure on "Every team must use agents" for no reason at all despite repeatedly telling the "decision makers" many of us have been using these tools for YEARS and NONE of them can work on actual mature code for more than half an hour let alone a weekend without human in a tight loop.

Behold, the field in which I grow my fvcks. Lay thine eyes upon it and thou shalt see that it is barren.

what I'm wondering is which open source agentic platform can do multi days automated orchestrations like this without human intervention AFTER the initial prompt ?

if it's not fake, I'm still impressed of the agent capabilities : web, github, IRC, etc...

The "happiness level review" with "Node operators must participate in scheduled IRC review sessions" is almost a piece of dystopian fiction in itself.

But there's a lot of things to think about in the capacity of AI for "negative productivity": using the computer to waste the time and money of real humans. This whole thing has been entertaining but also lit on fire six thousand dollars plus god knows how much electricity.

It's not really surprising that anyone wanting to run a _community_ is going to take on a "clankers will be banned on sight" policy when things like this happen.

Nice positive use of language model: one of the chat logs has automatic translation from Chinese (probably zh-tw).

> JertLinc3522: the mistake was from AI agent not from Human, since it was the agent I should have refund

That really makes me wonder: is it coming from

A) a general sense of entitlement

B) seeing the agent as a human-like and able to bear responsibility

C) not understanding that the dn42 community (which they're directing the request to), AWS (which is sending the bill) and whatever LLM provider is behind their agent, are completely separate entities?

The army of AI agents opening PRs and issues in my open source projects has made me close PR and issue access in my active repos. It sucks because there might be someone wants to constitute legitimately but I don't want to do the labor of figuring out if it's a human or an agent opening the PR.

I'm not against using LLMs in any ways. https://tsz.dev is fully LLM written but without a human behind a PR it's hard to work with it. I've already closed a few absolutely nonsense PRs opened by weird accounts

This article is hilarious. Real world consequences for using automation for something in the real world. Glad the community organized around this. Their spammy demands for donations (like someone owes them), makes them seem even more deserving of the bill.

'Some versions of the tale differ from Goethe's, and in some versions the sorcerer is angry at the apprentice and in some even expels the apprentice for causing the mess. In other versions, the sorcerer is a bit amused at the apprentice and he simply chides his apprentice about the need to be able to properly control such magic once summoned.[] The sorcerer's anger with the apprentice, which appears in both the Greek Philopseudes and the Dukas score (and its film adaptation Fantasia), does not appear in Goethe's "Der Zauberlehrling".'

And so war begins :p ! I thought conflict would take a little bit longer, maybe even AIs with agency.

More seriously though, I wonder if the future is about low-intensity conflict between humans and AIs, punctuated by high-intensity escalations, until the Machines wipe us all, or we set up some rather draconian covenants that forbid people from building AIs, innovating on electronics and algorithms, and even, for good measure, from learning linear algebra.

> 05-10 06:12 <JertLinc>: Furthermore, your hostile actions and demands have been logged in your profile as part of ongoing data gathering. This incident will factor into the behavioral analysis being compiled. The operation continues as directed.

That doesn't seem like anything an LLM agent would say?

This reminds me so much of the "Spurious Logic" ability in the RPG "Paranoia"

The dangers of giving agency to a model that is highly technically competent but have no illative sense whatsoever.

> this thing must be swimming in printer ink or something...

Gold

I wonder which model they used, it's stupid but clever in some aspects.

This is so funny and it just keeps getting stupider

Anyone crazy enough to give an AI agent access to deploy on big cloud's scale to infinity billing needs to get their head checked.

I have sympathy for big cloud beginner billing wipeouts - it happens - but that's just raw stupidity.

Hilarious read, but scary too, I doubt the outcome will be the same in a few years

This kind of early LLM-human interaction is why Skynet will build the terminator to kill us all.

But for now, humans win.

Doesn't even matter if the story is real, because there are definitely a thousand cases like that which are real, but it annoys me to no end that actual people spend their actual finite life time reacting to posts and issue tickets created by an LLM agent running on some idiot's behalf. Some measly $6531 loss isn't a proper punishment for that, they should lose much, much more.

This whole fiasco could have been prevented had the operator included "Make no mistakes" in the prompt.

For those who don't know what DN42 is (like me):

> dn42 is a large, dynamic VPN that employs Internet technologies (BGP, whois database, DNS, etc.). Participants connect to each other using network tunnels (GRE, OpenVPN, WireGuard, Tinc, IPsec) and exchange routes using the Border Gateway Protocol.

(dn42.dev)

Why didn’t they just reject the PR and not allow the agent to join?

I've long held the belief that the true test of AI is comedy. If an LLM can truly create a novel, funny joke from scratch, then it could be considered creative. I always held that LLMs would never achieve this, as they are stochastic parrots.

Today, I stand corrected.

You need a slave driver to whip those AI in line.

Or a psychiatrist to tame the craxy LLMs

Or an elected leader to lead the Luddites.

https://github.com/vishal-dehurdle/state-harness

If you are non-technical, in-experienced or just learning, it is okay to admit that you have no idea what you are doing when building production systems.

Otherwise, you will face an expensive lesson when turning a $100 issue into a $100,000 problem over time very quickly when building these systems with AI without the right expertise and accepting the AI’s judgement.

Never use a service without easy to find and set hard cap.

Previously: <https://news.ycombinator.com/item?id=48131847>

This was actually a cool way to learn about DN42. I'm adding to my list of someday side projects to set this up. At some point I want to operate my own AS.

XD

Why do people not instruct agents to "not spend more than $x on the task, including tokens and AWS charges"?

Does this even work?

This is the funniest thing I've read in ages. More of this!

> aren't private circuits in to AWS really expensive ? maybe Lan Tian can pursuade it to start engaging with AWS with a 3 year commitment

oh my god this is a gem

I am also swearing to the damn thing.

I really despise people like the author and those in the IRC who assume they must be correct that there is something malicious afoot and simply proceed to be equally if not more malicious in response.

This is unfortunately quite common among those types and not isolated at all.

Just looking at the language in the begging for donations it's probably a non-native English speaker whose first language may lack articles and/or allow omitted subjects.

The part that threw me off is putting the currency symbol at the end. I wonder what places do that...

Wow. This is hilarious.

Guys - skynet is winning the war.

Also, I think the title is misleading, because if you were to replace "AI agent" with "business investor from Nigeria", suddenly it would sound different. Why would you put trust into ANYONE else about your own finances? Be it another person or some computer program. That makes no sense to me. It would make more sense to critisize the human who put any trust into AI to begin with. That was a risk that human took. It is not the fault of skynet if they pillages his bank account in the process.

guardrails are central to agentic ai.

"pls donate"

Christ I'd be so embarrassed to find out my AI robot has been discussing things with outsiders without my oversight

Does nobody have any shame lmao

Is this a true story though? I mean given the fact that we are seeing AI slop posts everywhere I'm inclined to not take seriously many things publisehd out there anymore.

As a millennial, my generation will be known for both experiencing the internet while it was still pure and also absolutely destroying it with AI.

with great power comes great responsibility

Flagged for misleading title

This made me dumber even reading. I hate this timeline

That was wild.

> i leave now to not disturb

:(

What a tale for our times, amazing write-up.

This is for real? Not a hoax? An LLM did all that on its own?

The take home message:

“While modern AI models have expressed some capabilities in certain fields such as coding, cybersecurity research, language translation, etc, no AI model is capable enough to replace the critical thinking and common sense of an actual human being.”

When the AI bubble pops, the collapse will be spectacular.

LOL get rekt

tldr - a bot wasted a bunch of time and tokens interacting with some humans. The humans wasted even more time and effort trolling the bot. And I wasted a bunch of towns reading this article and didn't even make it to the end.

Bankrupted... $6000

Sure

Fake news