I feel like most of these schemes to categorise data as "public but not really" are ultimately doomed to failure. Even if you could trust every AI company in the world to respect these terms, is there anything stopping someone else indexing the data and selling them the information? I know there's copyright law but they're apparently ignoring that anyway.
Ultimately this reminds me of those really early social media profiles (before people understood privacy settings if they even existed) which would say "If you're not my friend you're not allowed to read this page".
If you don't want your content to end up in some database/archive don't publish it for the whole world to see.
1vuio0pswjnm7
"Cloudflare classifies bots by behavior, and a single bot can exhibit more than one behavior."
Is that really true
CF classifies anyone not using a popular browser with Javascript enabled as a "bot"
CF fingerprints www users
As an example, look at CF's Permissions-Policy HTTP response header on a site with CF "bot protection", i.e., the "checking your browser" CAPTCHA nonsense (challenges.cloudflare.com). Then look at IA's Permissions-Policy response header. One CDN is advertiser-focused, the other is user-focused
IA = Internet Archive
show comments
OroPla
In a world where people's searches are already mostly answered by AI, is there any point to disallow the training?
I could totally see a future where Google just stops showing you links to actual web pages altogether and just gives you their chatbot.
nirmeetimthebes
"Accountable" is just a fancy word for "pinky promise, but with a label." Nothing stops the data from ending up in a training run once it's already been fetched.
show comments
qsbuilder
The irony is that search engines are AI companies now. Telling them 'index me for search but don't train your models' is asking them to split a brain that’s already fully merged.
show comments
skybrian
I didn't know websites could opt out of providing data to Google's AI training. Looks Google added support for this via 'Google-Extended' in robots.txt back in 2023:
What does this really mean though? You can use an LLM to search.
shark1
"Apple, Google, and Microsoft honor or have committed (in a specified time frame) to honor this setting."
Specified Time Frame ;)
show comments
kinduff
I maintain a cloud IP ranges database, and I'm going to test this out.
I have my doubts, though. A formal title like "Accountable" (capitalized) sounds deliberate, but I can't help imagining the renewal email:
"Hey, want to renew your Accountable™ license?
Just pinky promise again that you use your IPs for what you say you do."
mskalski
I wonder if protocols like Web Bot Auth [1] will see wider adoption. At least as a supported mechanism for those bots which identify themselves. The rest probably still have to be treated with Anubis. In my free time I've recently been experimenting with a Web Bot Auth implementation as an Envoy dynamic module [2] to have a way to define some additional policies for the traffic from bots.
> Accountable mixed-use crawlers remain allowed for search. Every other training crawler is blocked, including the training-only crawlers run by Amazon, Anthropic, Meta, and OpenAI — blocking those does not affect search.
> We also categorize the relevant crawlers from Amazon, Anthropic, Meta, and OpenAI as Accountable. These organizations separate their Search and Training crawlers, so Cloudflare can block the Training crawler without affecting search.
I find it difficult to trust that either Meta or OpenAI would use their separate search and training crawlers only for the respective purposes. Their pinky promises have no value, IMO. Both companies are premised on deceptive behaviors.
nicolodev
there have been a few links here on hn about content protection based on Markov’s chain. It might be interesting for Cloudflare to add damage for the scraper that tries to query the page, and not just blocking them.
gdiamos
I wouldn’t trust an AI company to honor this as far as I could throw them
fwlr
“Pre-label your most valuable data for us and we promise to not make it too obvious that we’re training on it”
GetSMS
This seems useful for my site. I don't really want competitors' AI systems use our research data to train their models.
dzhiurgis
If this admin is serious about AI growth they’d make anti-scrapping illegal.
show comments
gchamonlive
Cloudflare, enabling the problem and the solution since, how long has it been?
show comments
zergrush
what weirds me out is the analytics
theres no way my index.html page with nothing is getting 10000 hits a day
wtf?
show comments
gleezard
A bit too late honestly (?). With so many people who have shifted over to reading AI summaries as a primary search response, those with AI-enabled sites will win by attrition.
There is no going back from this. And the internet is a relatively new phenomenon. Recklessly, blindly applying ads to pages in hopes of generating revenue is a very silly thing to do. Technology with ad blockers and now AI summaries has taken that away. New business models, perhaps actually decent ones are required.
Death of ads everywhere? Good fucking riddance.
Posted from LibreWolf.
show comments
arm32
What does this new setting actually do? Does it block their IP ranges too, I hope? As if Meta, for instance, is actually going to respect Accountable, via themselves or their partners, quite frankly is eyebrow raising at best.
I feel like most of these schemes to categorise data as "public but not really" are ultimately doomed to failure. Even if you could trust every AI company in the world to respect these terms, is there anything stopping someone else indexing the data and selling them the information? I know there's copyright law but they're apparently ignoring that anyway.
Ultimately this reminds me of those really early social media profiles (before people understood privacy settings if they even existed) which would say "If you're not my friend you're not allowed to read this page".
If you don't want your content to end up in some database/archive don't publish it for the whole world to see.
"Cloudflare classifies bots by behavior, and a single bot can exhibit more than one behavior."
Is that really true
CF classifies anyone not using a popular browser with Javascript enabled as a "bot"
CF fingerprints www users
As an example, look at CF's Permissions-Policy HTTP response header on a site with CF "bot protection", i.e., the "checking your browser" CAPTCHA nonsense (challenges.cloudflare.com). Then look at IA's Permissions-Policy response header. One CDN is advertiser-focused, the other is user-focused
IA = Internet Archive
In a world where people's searches are already mostly answered by AI, is there any point to disallow the training?
I could totally see a future where Google just stops showing you links to actual web pages altogether and just gives you their chatbot.
"Accountable" is just a fancy word for "pinky promise, but with a label." Nothing stops the data from ending up in a training run once it's already been fetched.
The irony is that search engines are AI companies now. Telling them 'index me for search but don't train your models' is asking them to split a brain that’s already fully merged.
I didn't know websites could opt out of providing data to Google's AI training. Looks Google added support for this via 'Google-Extended' in robots.txt back in 2023:
https://blog.google/innovation-and-ai/products/an-update-on-...
What does this really mean though? You can use an LLM to search.
"Apple, Google, and Microsoft honor or have committed (in a specified time frame) to honor this setting."
Specified Time Frame ;)
I maintain a cloud IP ranges database, and I'm going to test this out.
I have my doubts, though. A formal title like "Accountable" (capitalized) sounds deliberate, but I can't help imagining the renewal email:
"Hey, want to renew your Accountable™ license? Just pinky promise again that you use your IPs for what you say you do."
I wonder if protocols like Web Bot Auth [1] will see wider adoption. At least as a supported mechanism for those bots which identify themselves. The rest probably still have to be treated with Anubis. In my free time I've recently been experimenting with a Web Bot Auth implementation as an Envoy dynamic module [2] to have a way to define some additional policies for the traffic from bots.
[1] https://datatracker.ietf.org/doc/draft-ietf-webbotauth-https... [2] https://github.com/michalskalski/envoy-web-bot-auth
> Accountable mixed-use crawlers remain allowed for search. Every other training crawler is blocked, including the training-only crawlers run by Amazon, Anthropic, Meta, and OpenAI — blocking those does not affect search.
> We also categorize the relevant crawlers from Amazon, Anthropic, Meta, and OpenAI as Accountable. These organizations separate their Search and Training crawlers, so Cloudflare can block the Training crawler without affecting search.
I find it difficult to trust that either Meta or OpenAI would use their separate search and training crawlers only for the respective purposes. Their pinky promises have no value, IMO. Both companies are premised on deceptive behaviors.
there have been a few links here on hn about content protection based on Markov’s chain. It might be interesting for Cloudflare to add damage for the scraper that tries to query the page, and not just blocking them.
I wouldn’t trust an AI company to honor this as far as I could throw them
“Pre-label your most valuable data for us and we promise to not make it too obvious that we’re training on it”
This seems useful for my site. I don't really want competitors' AI systems use our research data to train their models.
If this admin is serious about AI growth they’d make anti-scrapping illegal.
Cloudflare, enabling the problem and the solution since, how long has it been?
what weirds me out is the analytics
theres no way my index.html page with nothing is getting 10000 hits a day
wtf?
A bit too late honestly (?). With so many people who have shifted over to reading AI summaries as a primary search response, those with AI-enabled sites will win by attrition.
There is no going back from this. And the internet is a relatively new phenomenon. Recklessly, blindly applying ads to pages in hopes of generating revenue is a very silly thing to do. Technology with ad blockers and now AI summaries has taken that away. New business models, perhaps actually decent ones are required.
Death of ads everywhere? Good fucking riddance.
Posted from LibreWolf.
What does this new setting actually do? Does it block their IP ranges too, I hope? As if Meta, for instance, is actually going to respect Accountable, via themselves or their partners, quite frankly is eyebrow raising at best.