I think the problem is that when Archive.org has access to NYT and other publisher content, people can scrape NYT content at scale from Archive.org even when they cannot do so directly on NYT. If Archive.org blocks scrapers, maybe the publishers would make different choices and allow Archive.org access.
ctippett
Am I correct that this has come about because archive.org respects robots.txt and these sites have blocked their crawler from indexing their sites?
I'm not sure how to articulate my thoughts on this exactly, other than to say it's disappointing that doing the right thing (i.e. respecting robots.txt) is rewarded with the burden of soliciting responses to a petition while at the same time others are rewarded with profit for ignoring those same directives.
show comments
ajaimk
Idea: allow scraping but can’t publish for 1 year?
show comments
someperson
Maybe they should have an escrow like Financial Times is available on NewsBank service with a 30 day escrow
WarmWash
A bunch of people who have haven't ever loaded an ad or paid a subscription to those organizations are going to make a stand to demand they leave their backdoor open?
Need a cryptographically verifiable internet archive. This is probably not possible without something like web 3 or nostr or gpg pgp. Idk.
show comments
JumpCrisscross
I know a little about this debate on the Times and Atlantic sides. I’ll get some grief for this, but I asked a senior person at the former what they thought about the paywall workarounds that are frequent on HN—I was genuinely shocked to learn they hadn’t heard about it.
In the end, we settled on agreeing that making such stuff available after 30 days, and possibly with access restrictions (can’t be pulled more than N times a day, in case it becomes relevant in the future) struck the right balance.
To my knowledge, the Internet Archive hasn’t done any outreach on this issue. In addition to pressuring the publications, I’d put some pressure on them to negotiate.
show comments
crowcroft
Ok, but what about Meta and X etc.
eranation
I signed, but let’s be honest.
A pie chart showing the times I used the wayback machine to read an old NYT article vs the times I visited it due to a highly upvoted top HN comment linking to a relatively new article so we all can bypass the paywall is a solid circle.
show comments
karel-3d
There is still archive.today, too bad the owner is crazy
drivingmenuts
What is the advantage to those organizations to have their work preserved? If their work is stored in a public archive, they can’t charge for it and they lose money. If they make a mistake, then history is what they say it is and there is no external record to say otherwise.
show comments
shevy-java
We are kind of losing the world wide web here or at the least part of how we could use it in the past. More and more key services get knocked out; see the associated rise of age snifing and the campaign to destroy VPNs.
sublinear
After many years of these media outlets circling the drain, this is likely the clearest signal of their irrelevance. It's not like anyone is committing these rags to microfiche anymore.
show comments
kr108sdh
The petition should be to ban the AI theft. If it is on wayback, the bots could as well scrape the NYT directly.
The NYT is of course guilty itself. It did not investigate the possible murder of its star witness Suchir Balaji and is too reserved in examining the consequences of AI in general.
If they don't fulfill their journalistic and societal obligations, soon its own journalists will be replaced by AI bullet point slop like Axios.
WarmWash
Can we just go back to ads and normalize blocking people who ad-block?
I'm grown up now, I understand how things work, and I'd rather see Tide and Coke ads than pay $20/mo to 8 different orgs, while maintaining that ad free option for those who want it.
The children of the internet probably won't sign a truce, so let's just cut them out and let intellectually honest people have a decent internet.
show comments
righthand
Wouldn’t it be better to let these legacy news orgs (which aren’t really anything beyond advertising and data harvesting firms) block archive.org and thus no one will read their articles and they can go under? I’m struggling to think of a reason I need NY Times. I’ve never had a subscription and never seen writing that I thought benefited me as a citizen (they’re Very pro-war of any kind).
show comments
xyzzy_plugh
The title freaked me out. I thought this was about the Wayback Machine going away but no, it's just news publications blocking being archived.
I guess I don't really care. As soon as it becomes unworkable to view these publications through archivers I'll just stop viewing them altogether. I don't see this helping their bottom line though.
I think the problem is that when Archive.org has access to NYT and other publisher content, people can scrape NYT content at scale from Archive.org even when they cannot do so directly on NYT. If Archive.org blocks scrapers, maybe the publishers would make different choices and allow Archive.org access.
Am I correct that this has come about because archive.org respects robots.txt and these sites have blocked their crawler from indexing their sites?
I'm not sure how to articulate my thoughts on this exactly, other than to say it's disappointing that doing the right thing (i.e. respecting robots.txt) is rewarded with the burden of soliciting responses to a petition while at the same time others are rewarded with profit for ignoring those same directives.
Idea: allow scraping but can’t publish for 1 year?
Maybe they should have an escrow like Financial Times is available on NewsBank service with a 30 day escrow
A bunch of people who have haven't ever loaded an ad or paid a subscription to those organizations are going to make a stand to demand they leave their backdoor open?
I am looking forward to this (https://news.ycombinator.com/item?id=48070516)
Need a cryptographically verifiable internet archive. This is probably not possible without something like web 3 or nostr or gpg pgp. Idk.
I know a little about this debate on the Times and Atlantic sides. I’ll get some grief for this, but I asked a senior person at the former what they thought about the paywall workarounds that are frequent on HN—I was genuinely shocked to learn they hadn’t heard about it.
In the end, we settled on agreeing that making such stuff available after 30 days, and possibly with access restrictions (can’t be pulled more than N times a day, in case it becomes relevant in the future) struck the right balance.
To my knowledge, the Internet Archive hasn’t done any outreach on this issue. In addition to pressuring the publications, I’d put some pressure on them to negotiate.
Ok, but what about Meta and X etc.
I signed, but let’s be honest.
A pie chart showing the times I used the wayback machine to read an old NYT article vs the times I visited it due to a highly upvoted top HN comment linking to a relatively new article so we all can bypass the paywall is a solid circle.
There is still archive.today, too bad the owner is crazy
What is the advantage to those organizations to have their work preserved? If their work is stored in a public archive, they can’t charge for it and they lose money. If they make a mistake, then history is what they say it is and there is no external record to say otherwise.
We are kind of losing the world wide web here or at the least part of how we could use it in the past. More and more key services get knocked out; see the associated rise of age snifing and the campaign to destroy VPNs.
After many years of these media outlets circling the drain, this is likely the clearest signal of their irrelevance. It's not like anyone is committing these rags to microfiche anymore.
The petition should be to ban the AI theft. If it is on wayback, the bots could as well scrape the NYT directly.
The NYT is of course guilty itself. It did not investigate the possible murder of its star witness Suchir Balaji and is too reserved in examining the consequences of AI in general.
If they don't fulfill their journalistic and societal obligations, soon its own journalists will be replaced by AI bullet point slop like Axios.
Can we just go back to ads and normalize blocking people who ad-block?
I'm grown up now, I understand how things work, and I'd rather see Tide and Coke ads than pay $20/mo to 8 different orgs, while maintaining that ad free option for those who want it.
The children of the internet probably won't sign a truce, so let's just cut them out and let intellectually honest people have a decent internet.
Wouldn’t it be better to let these legacy news orgs (which aren’t really anything beyond advertising and data harvesting firms) block archive.org and thus no one will read their articles and they can go under? I’m struggling to think of a reason I need NY Times. I’ve never had a subscription and never seen writing that I thought benefited me as a citizen (they’re Very pro-war of any kind).
The title freaked me out. I thought this was about the Wayback Machine going away but no, it's just news publications blocking being archived.
I guess I don't really care. As soon as it becomes unworkable to view these publications through archivers I'll just stop viewing them altogether. I don't see this helping their bottom line though.