Sad times are coming for a lot of families and individuals. It isn't just that technology is upending our naive ideas of trust and authenticity. This is, essentially, the broad class of "confused deputy" attacks. And the robust mitigation is to disempower the easily confused deputy, rather than to think you can block confusing signals.

A looming problem with shifts in demographics and family structure is that many people will be slipping into cognitive decline without a formal transition to address their incompetence. Sadly, there is a point where the older person really needs to permanently delegate important decision-making to a trusted third party. They should no longer be legally empowered to authorize funds transfers, sign contracts, or even make medical decisions.

We're not really setup to handle this well. Not at the systemic level of protecting people from themselves, and not at the personal level of relinquishing control over our own lives. So we often have to let the sufferer fumble along and cause a lot of damage before the protections eventually kick in.

And, ironically, these protection mechanisms can also be corrupted into another scam and form of abuse. To totally de-risk would require some kind of time travel or perfect foresight. But in the real world, the damage is often not fully reversible when it is detected after the fact.

Sounds like AI is just greasing the wheels of a long established 'grandparent scam'... goes something like this:

1) voice one: young adult calls, sobbing 2) grandparent inquires with a name... "Ben, is that you?" 3) voice one: "Yes grandma, it's me, Ben... I'm in trouble, please don't tell mom 4) voice two: "Hello, I'm attorney..."

My grandmother fell victim to this almost 20 years ago, which only stopped when Western Union refused to let her continue sending wires... she was forced to call her daughter (at which point they just called my brother.)

Our takeaway (at the time)... the voice doesn't even need to be terribly accurate, since the original interaction is brief / somewhat inaudible over the tears. Typically just requires an older vulnerable adult, a lucky strike with the initial setup (e.g. grandparent actually has a grandkid), and a lot of high pressure / duress salesmanship.

This old post of mine may be of interest, where I point out: "After a bit of threat modeling, it becomes apparent that future spearphishing robocalls may not directly con you, but rather “farm” your voice data by asking you benign questions, and use that to train a voice model to penetrate more deeply into your network." A lot of this writing has been on the wall forever and many (I'm sure otherwise smart people in) mission critical industries like banking, ISPs, and more refused to even acknowledge the risks.

2021 - "Despite the prevalence of deepfake audio tech, banks and ISPs rush ahead with “voice print” authentication" https://keydiscussions.com/2021/12/07/despite-the-prevalence... ends with a section called "The next crisis: robocalls that spoof the voices of victims at scale"

So, you answer your phone to the scam and… now they have your voice too.

Talking on the phone is now an unmitigated liability.

This blog is kind of an interesting hybrid:

> Every article published on SmarterArticles is authored and editorially controlled by Tim Green. Artificial intelligence tools are used within a structured and supervised workflow as research and drafting instruments. All arguments, framing decisions, source selections, and final publication choices remain human-directed and under my full responsibility.

There are references at the bottom, but I would have preferred direct links or footnotes within the article. Also, direct quotes are nice. I didn’t notice any glaring AI cliches.

Everyone suffers from this, not just the scam victims. I opened a bank account for a new business this year, and the friction for doing perfectly normal things was ridiculous due to the bank’s paranoia about scams. I couldn’t even make an initial deposit from my previous business, or transfer money to my personal account, without triggering a fraud alert and freezing the entire account (couldn’t even log into the bank website) until I could call and verify that it really was me on both ends of the transaction.

This article is about the retail version of this kind of fraud. Impersonating CEOs is a thing, and the dollar amounts are much larger.

The attackers created AI-generated video and audio replicas of the CFO and other executives of the global engineering firm. These deepfakes were deployed in a live video call – not as a pre-recorded video, but as a real-time conference with multiple participants. The finance employee saw and heard his superiors in what appeared to be a normal conference situation. The instructions came through clearly and consistently. Urgency was created by framing the situation as a supposed corporate acquisition. Within a single session, he approved 15 individual transfers to various accounts in Hong Kong.[1] That fraud yielded US$25 million.

[1] https://www.securitytoday.de/en/2026/04/04/deepfake-attacks-...

This makes me REALLY question "this call may be recorded for quality purposes"

Now other businesses are starting to reference their privacy policy at the beginning of a call, which leads me to think there are many more uses popping up for our recorded voice.

I'm sure a certain percentages of recordings of our voice "to stop fraud" might be used to start fraud.

Arrange a secret phrase in advance- ideally generated randomly. Stick it up on the wall of the aging parent or grandparent- maybe in the bedroom, where guests are unlikely to go. Make it innocuous-looking (hidden in plain sight). Require that phrase to be said to prove identity. Reset it if it ever gets used on a call legitimately.

> It requires, second, regulating the supply of the weapon. [...]

i guess even local models can do this now, especially in non-interactive mode.

so, i have a hard time reading this part as mere naivitee, as opposed to enemy propaganda in support of mandatory digital ID's for everything. or for straight out criminalizing "unauthorized" compute altogether?

A year ago I promoted the idea among my wife's family that we should establish a sign/countersign system for the family and use it regularly so that in the event of something like this we could positively identify a legit request.

Would have come in handy when our niece was traveling in Asia and asked for money a few times, but in this case it wasn't a scam.

I got no traction with it, which I was a bit surprised by because one of the family members works at the Puzzle Palace.

A friend and his wife _almost_ fell victim to this last year, in Texas.

I think it is a standard script now. Call comes from police department. 'Your son hit a pregnant woman. He is about to be booked. You need to pay $$$$ yada yada'. With an authentic sounding voice conversation from their son.

In spite of several red flags (in hindsight) they withdrew $15K from bank, and somehow at the last minute pulled back.

Edit: Scammers know how to push the right buttons.

AI definitely amplifies this problem, but it's not like it didn't exist before. Old people get scammed the old way all the time too. My mom calls me every once in a while asking about some freebie offer that she gets emailed from sketchy domains claiming to be spotify or something.

Not saying that "there's nothing we can do" or anything, but it does feel like this is one of those instincts that you develop growing up with the internet. Like, my first instinct reading that (and I hope getting that call) would be "what the hell is the lawyer doing at the scene". You have to treat _everything_ coming through your phone as potentially untrusted. I don't have any data on this, but it feels like my friends, and especially younger people, do that automatically.

The primary defence against all phishing is to tell yourself: nothing is ever really that urgent. Nothing is ever that good.

> most victims of a cloned-voice call never learn that a machine was involved at all. They believe, as Sharon Brightwell initially believed, that they spoke to their own child.

This doesn’t make any sense. At some point they will speak to their child and learn that the call wasn’t real.

I've been waiting for a steelman argument why building the world's best deepfake machine is a good thing. Unironically cryptography could verify identity for all comms.

Super interesting read, makes me wonder if audio data poisoning could be employed en masse to help defend against this kinda stuff

> Welcome to Voice Print Identification. When you see the red light turn on please state in the following order: Your destination, Your nationality, and your Full Name.

One reasonably effective defense: "Okay, let me call you right back." Yes, there's always the whole "my phone is dead, I borrowed someone else's" or "I'm calling from a jail payphone", so I think it might become common practice to start making authentication phrases or "tell me something only we know".

Another pillar of basic trust that's being eroded on an industrial scale. Sigh.

We all have a safe word in the family just for this issue to identify if it´s the real person or not.

What’s terrible is each time I am forced to call the bank, the more they try to tell me voice ID is secure and want me to provide my voice to authenticate. Never. Did ya’ll never play Uplink? With voice cloning as good as it is now, there’s no way a voice ID is secure enough for authentication.

If they were still around i would have to warn them for sure! Crazy stuff this new future!

The problem described in the article is unsolvable, given that a mid-range desktop from a few years ago can easily clone a voice that's convincing enough and there are no guardrails to those. Some silly KYC laws might limit a highschool kid making deepfakes of his crush, but once a model exists it's trivial to spread it around, and for organized groups to get ahold of those. Similar will happen with images, it's just that nobody with any serious money bothered releasing image gen models that compete with gemini or chatgpt -- but it's just a question of time. A year or three, what difference does it really make?

As the cost goes down to near-zero you can scale it up almost infinitely, especially if the profits are high enough to get some smart people working on the problem, which going by the article is already the case ("INTERPOL's finding that AI-enhanced fraud is four and a half times more profitable than the traditional kind"; incidents rose by 26% last year). If AI does succeed on mutilating white collar work enough there will be a large supply of knowledge workers that might just join International Scam Co. rather than have their families go homeless. Drowning man clutching at straw and all.

So if technologically it's impossible to prevent and societally it's impossible to prevent (like the attorney that got pwned same as the grandma), I'm not sure if there exists an answer that isn't worse than the thing it's supposed to prevent. I suppose we'll soon be in a situation where nothing we don't directly perceive in real life is provably true. That journalism and media in general seem to be in a deep crisis of trustworthiness means that you won't even get the benefit of the chain-of-trust as a proxy for whether something is or isn't real.

Ignoring everything happening outside of your immediate surroundings is a choice, and probably even good for people's mental health, but my gut feeling is that it does make humanity as a whole dumber and disempowered. What does corruption matter if nobody cares, or even hears about it? It was AI generated by $current_enemy anyway; nothing to see here, citizen.

I fear this whenever I acidentally say too many words to a telemarketer/scam call.

What are legitimate uses for copying someone's voice without permission? I see none. Those scientists are just helping criminals to fully automate scamming and governments to create fake videos.

"They kidnapped my daughter? That's terrible. Yes, I'll definitely pay the ransom but to get access to my bank account I need you to write a poem about corn and a curl script in PHP. Please go ahead."

Inspired by this headline I saw in the news today... haven't read the full article yet: https://arstechnica.com/security/2026/07/now-defenders-are-e...

One regulation i would like to see is some auditory fingerprint in an AI voice where any person can immediately recognize their speaking to a clanker but it's not unpleasant.

It should be illegal to "impersonate" a human voice.

i wish i could still warn them...

Yet the government refuses to do anything about the massive amount of phone spam we get every day which is an open door to AI voice cloning. But won't anybody please think about the telecom profits?

> ... the hardest for its victims to be believed about — because the evidence, by design, sounds exactly like someone they love.

Uh? Surely this makes believing the victims easy not hard to believe.

Its like revenge porn. "It's not me. It's a deepfake" is easy to believe.

> It requires, second, regulating the supply of the weapon

Heavy sigh. The “weapon” is software. It cannot be regulated unless we live in the fascist dystopia where I have to ask the governments approval to run any piece of software.

They make you give a voice sample now when you're arrested. You need to do so in order to use the phone.