Please report success/failure after each test. Asking me to read and compare 30 writing samples to get any feedback at all means I won't finish. Telling me immediately when I got one wrong lets me recognize patterns and improve my guesses.

Interestingly, it seems like almost every set of three seems to follow a pattern: exactly one passage of the three will have a key word or phrase swapped in the first sentence. That is, for every set of 3 passages, two would start with ~identical sentences, and one would have a key word or token changed.

I caught onto this early and used it every time, and ended up getting 2/10, which I feel is meaningfully worse than random chance (which you'd expect given watermarking is, afaict, objectively lossless). I smell trickery!

I've read that watermarking should in theory be impossible to detect except by the entity that watermarked it. Which is sensible, and I mostly understand at a high level.

But what I don't know and don't understand is what happens if you watermark watermarked text. Does it test positive for both watermarks? Only the second? Indeterminate?

Or maybe I'm misunderstanding. Can you tell that it's watermarked, but only the entity who put the watermark in place can test if it's theirs? My confusion about watermarking multiple times still stands, though.

Regardless of what happens when you watermark multiple times, no matter the outcome, it weakens the watermark. Which, depending on the threat model, kind of makes it moot. I can't imagine a serious situation where a watermark can be weakened in any way and still be useful. Even "this came from an LLM" isn't a valid signal if you can just watermark ANY text through purely mechanical means.

It's also not clear to me how this will affect mainstream LLMs. If all output text is watermarked, there MUST be an escape hatch. Otherwise, JSON schemas will break (or provide holes where unwatermarked text can be exfiltrated through MCP), "return this text exactly with no changes" will be impossible, and writing diffs will break.

I feel like I must be missing something.

Utterly imperceptible, even when studied under the microscope in a way that LLM text very rarely is in practice.

It will be interesting to see whose concerns are assuaged (perhaps they genuinely though mistakenly believed it would degrade quality), and whose concerns are heightened (perhaps their real objection is that their AI-generated text will become detectable).

What is the meaning of the numbers?

> Weighted mean detector score 0.5307

Does it mean that that passage would be rated a 53% chance of being watermarked? So you would need a passage 10x as long to be reasonably sure of providence?

This is like giving you three outputs from md5sum and asking you to guess for which one the input ended in a "q". There's no way to tell unless you break the RNG.

Wow I actually got a 7/10. It was hard to tell at first but there are signs that tipped me off to which one probably had a higher score out of the multiple choice.

Could do with some context on how watermarking works. Objectively speaking it should be impossible to tell.

My sense of the concern here is that watermarking may somehow deprive someone or something of value regardless of whether or not they can tell, so I briefly pondered trying to rank these from best to worst and see if any set of those votes meaningfully deviated from ~average.

That said, I read the first triple and found all three tortured enough that I can't be bothered with the rest.

Call me persuaded, I guess.

I was never going to do very well on this. My ADHD was itching after the third one. I suspect it would have been sooner but I had a bit of extra focus from the suprise that it selected an answer for the first question when I tried to scroll.

To avoid that on the following questions I just held my finger on my phone to avoid a click. That eventually selected some text, and I instinctively tapped to deselect. That triggered another random pick, then I just tapped through to the end because I was fed up.

It feels all of them are terribly written. I don't know why.

Interesting, I did very badly, 3/10!

If SynthID is a google technology, then this is likely just us training their ai again, captcha all over again.

Google's SynthID page says they can watermark text, but it also says that it can only detect the watermark on "image, video or audio". Does that mean that the text watermarks can't actually be used as watermarks?

I had a moment I thought was concrete watermarking the other day. Claude wrote the sentence... "since the compute buffer estimate has some sl..." Now you'd think the right word would be slack, but Claude wrote... slop? Which does seem off but, those two letters could be tokens very close in probability.

I disliked the fact that the experiment only covered prose, which my eyes glossed over and made me actually do random entries to pass on and see the results. I'd love to see it on a more accurate output distribution like commented code

8/10. It was harder to distinguish than I expected. If they had applied something like a humanizer skill, it probably would have been nearly impossible to tell.

It takes a lot of patience to read this much slop voluntarily.

Thought there were only 2 options!

This is stupid --- way too long and wordy. Make your test worth taking. And even then it's theoretically impossible to detect the watermark so what even is the point? If it's to check whether the watermarking actually has that property, this is not at all a reliable way to do that.