> Large language models generate text one word (token) at a time. Each word is assigned a probability score, based on how likely it is to be generated next. So for a sentence like “My favourite tropical fruits are mango and…”, the word “bananas” would have a higher probability score than the word “airplanes”.
> SynthID adjusts these probability scores to generate a watermark. It's not noticeable to the human eye, and doesn’t affect the quality of the output.
I think they need to be clearer about the constraints involved here. If I ask What is the capital of France? Just the answer, no extra information.” then there’s no room to vary the probability without harming the quality of the output. So clearly there is a lower bound beyond which this becomes ineffective. And presumably the longer the text, the more resilient it is to alterations. So what are the constraints?
I also think that this is self-interest dressed up as altruism. There’s always going to be generative AI that doesn’t include watermarks, so a watermarking scheme cannot tell you if something is genuine. It is, however, useful for determining that something came from a specific provider, which could be valuable to Google in all sorts of ways.
show comments
mingtianzhang
1. One-sample detection is impossible. These detection methods work at the distributional level—more like a two-sample test in statistics—which means you need to collect a large amount of generated text from the same model to make the test significant. Detecting based on a short piece of generated text is theoretically impossible. For example, imagine two different Gaussian distributions: you can never be 100% certain whether a single sample comes from one Gaussian or the other, since both share the same support.
2. Adding watermarks may reduce the ability of an LLM, which is why I don’t think they will be widely adopted.
3. Consider this simple task: ask an LLM to repeat exactly what you said. Is the resulting text authored by you, or by the AI?
show comments
daft_pink
Would you really use Google products to write your email if you knew that they were watermarking it like this?
I think this technology is gonna quickly get eliminated from the marketplace, cause people aren’t willing to use AI for many common tasks that are watermarked this way. It’s ultimately gonna cause Google to lose share.
This technology has a basic use dilemma problem where widely publishing it’s ability and existence will cause your AI to stop being used in some applications
Hey I made an open source version of this last week (albeit for different purposes). Check it out at: https://github.com/sutt/innocuous
There's lot of room for contributions here, and I think "fingerprinting layer" is an under-valued part of the LLM stack, not being explored by enough entrants.
utilize1808
I feel this is not the scalable/right way to approach this. The right way would be for human creators to apply their own digital signatures to the original pieces they created (specialised chips on camera/in software to inject hidden pixel patterns that are verifiable). If a piece of work lacks such signature, it should be considered AI-generated by default.
show comments
wenbin
I really hope SynthID becomes a widely adopted standard - at the very least, Google should implement it across its own products like NotebookLM.
The problem is becoming urgent: more and more so-called “podcasts” are entirely fake, generated by NotebookLM and pushed to every major platform purely to farm backlinks and run blackhat SEO campaigns.
Beyond SynthID or similar watermarking standards, we also need models trained specifically [0] to detect AI-generated audio. Otherwise, the damage compounds - people might waste 30 minutes listening to a meaningless AI-generated podcast, or worse, absorb and believe misleading or outright harmful information.
Could anybody explain how this isn't easily circumvented by using a competitor's model?
Also, if everything in the future has some touch of AI inside, for example cameras using AI to slightly improve the perceived picture quality, then "made with AI" won't be a categorization that anybody lifts an eyebrow about.
show comments
egeozcan
I guess this is the start of a new arms race on making generated content pass these checks undetected and detecting them anyway.
show comments
tiku
The whole NFT thing could be used to mark content, pictures and hashes of texts for example.
wiradikusuma
For image, what happen if I screenshot it? Will the watermark survive?
greatgib
Also, what will happen if you cut and paste some part or the whole image inside another bigger one, like traditional photo editing?
And if I scan the image or take a picture of the image on display.
Oras
OpenAI has been doing something similar for generated images using C2PA [0]
It is easy to alter by just saving to a different format or basic cropping.
I would love to see how SynthID is fixing this issue.
Would be nice if you guys open source the detection code, similar to the way C2PA is open
show comments
fertrevino
I wonder what exactly would prevent a developer from removing the signature from a generated file. One could remove arbitrary segments that signal that it is AI generated.
show comments
9dev
You can never be sure something has been generated by a model embedding one of these anyway, so it’s pretty moot.
doawoo
the beginning of walled garden “AI” tools has been interesting to follow
montag
"The watermarks are embedded across Google’s generative AI consumer products, and are imperceptible to humans."
I'd love to see the data behind this claim, especially on the audio side.
show comments
DrNosferatu
The first good use of blockchain comes to mind.
chii
i find the premise to be an invalid one personally - why is the property that a works from an AI model must be identified/identifiable?
show comments
DrNosferatu
If I slightly edit plain text watermarked with it, will the watermark identification be robust?
HighGoldstein
I wonder if, conversely, authentic media can be falsely watermarked as AI-generated.
show comments
peterkelly
Create the problem, sell the solution.
donperignon
I am not sure that text watermarking will be accurate, I foresee plenty of false positives.
show comments
pelasaco
looks like the same as anti-virus companies in the 80s? Write virus, Write anti-virus and profit!
> Large language models generate text one word (token) at a time. Each word is assigned a probability score, based on how likely it is to be generated next. So for a sentence like “My favourite tropical fruits are mango and…”, the word “bananas” would have a higher probability score than the word “airplanes”.
> SynthID adjusts these probability scores to generate a watermark. It's not noticeable to the human eye, and doesn’t affect the quality of the output.
I think they need to be clearer about the constraints involved here. If I ask What is the capital of France? Just the answer, no extra information.” then there’s no room to vary the probability without harming the quality of the output. So clearly there is a lower bound beyond which this becomes ineffective. And presumably the longer the text, the more resilient it is to alterations. So what are the constraints?
I also think that this is self-interest dressed up as altruism. There’s always going to be generative AI that doesn’t include watermarks, so a watermarking scheme cannot tell you if something is genuine. It is, however, useful for determining that something came from a specific provider, which could be valuable to Google in all sorts of ways.
1. One-sample detection is impossible. These detection methods work at the distributional level—more like a two-sample test in statistics—which means you need to collect a large amount of generated text from the same model to make the test significant. Detecting based on a short piece of generated text is theoretically impossible. For example, imagine two different Gaussian distributions: you can never be 100% certain whether a single sample comes from one Gaussian or the other, since both share the same support.
2. Adding watermarks may reduce the ability of an LLM, which is why I don’t think they will be widely adopted.
3. Consider this simple task: ask an LLM to repeat exactly what you said. Is the resulting text authored by you, or by the AI?
Would you really use Google products to write your email if you knew that they were watermarking it like this?
I think this technology is gonna quickly get eliminated from the marketplace, cause people aren’t willing to use AI for many common tasks that are watermarked this way. It’s ultimately gonna cause Google to lose share.
This technology has a basic use dilemma problem where widely publishing it’s ability and existence will cause your AI to stop being used in some applications
Here's the SynthID paper:
https://www.nature.com/articles/s41586-024-08025-4.pdf
https://www.nature.com/articles/s41586-024-08025-4
Hey I made an open source version of this last week (albeit for different purposes). Check it out at: https://github.com/sutt/innocuous
There's lot of room for contributions here, and I think "fingerprinting layer" is an under-valued part of the LLM stack, not being explored by enough entrants.
I feel this is not the scalable/right way to approach this. The right way would be for human creators to apply their own digital signatures to the original pieces they created (specialised chips on camera/in software to inject hidden pixel patterns that are verifiable). If a piece of work lacks such signature, it should be considered AI-generated by default.
I really hope SynthID becomes a widely adopted standard - at the very least, Google should implement it across its own products like NotebookLM.
The problem is becoming urgent: more and more so-called “podcasts” are entirely fake, generated by NotebookLM and pushed to every major platform purely to farm backlinks and run blackhat SEO campaigns.
Beyond SynthID or similar watermarking standards, we also need models trained specifically [0] to detect AI-generated audio. Otherwise, the damage compounds - people might waste 30 minutes listening to a meaningless AI-generated podcast, or worse, absorb and believe misleading or outright harmful information.
[0] 15,000+ ai generated fake podcasts https://www.kaggle.com/datasets/listennotes/ai-generated-fak...
Could anybody explain how this isn't easily circumvented by using a competitor's model?
Also, if everything in the future has some touch of AI inside, for example cameras using AI to slightly improve the perceived picture quality, then "made with AI" won't be a categorization that anybody lifts an eyebrow about.
I guess this is the start of a new arms race on making generated content pass these checks undetected and detecting them anyway.
The whole NFT thing could be used to mark content, pictures and hashes of texts for example.
For image, what happen if I screenshot it? Will the watermark survive?
Also, what will happen if you cut and paste some part or the whole image inside another bigger one, like traditional photo editing?
And if I scan the image or take a picture of the image on display.
OpenAI has been doing something similar for generated images using C2PA [0]
It is easy to alter by just saving to a different format or basic cropping.
I would love to see how SynthID is fixing this issue.
https://help.openai.com/en/articles/8912793-c2pa-in-chatgpt-...
Would be nice if you guys open source the detection code, similar to the way C2PA is open
I wonder what exactly would prevent a developer from removing the signature from a generated file. One could remove arbitrary segments that signal that it is AI generated.
You can never be sure something has been generated by a model embedding one of these anyway, so it’s pretty moot.
the beginning of walled garden “AI” tools has been interesting to follow
"The watermarks are embedded across Google’s generative AI consumer products, and are imperceptible to humans."
I'd love to see the data behind this claim, especially on the audio side.
The first good use of blockchain comes to mind.
i find the premise to be an invalid one personally - why is the property that a works from an AI model must be identified/identifiable?
If I slightly edit plain text watermarked with it, will the watermark identification be robust?
I wonder if, conversely, authentic media can be falsely watermarked as AI-generated.
Create the problem, sell the solution.
I am not sure that text watermarking will be accurate, I foresee plenty of false positives.
looks like the same as anti-virus companies in the 80s? Write virus, Write anti-virus and profit!
It only works across Google shit.