AWS’s US-East 1 continues to be the Achilles heel of the Internet.
And while yes building across multiple regions and AZs is a thing, AWS has had a string of issues where US-East 1 has broader impacts, which makes things far less redundant and resilient than AWS implies.
show comments
aurareturn
These things are dangerous. Someone who can take AWS down such as an employee can place a bet.
These bets aren’t as innocent as they seem because the bettors can often influence or change the outcome.
show comments
grendelt
There sure are a lot of eggs in that East basket.
fabian2k
I thought cooling was pretty much pre-planned in any data center, and you simply don't install more stuff than you can cool?
So did some cooling equipment fail here or was there an external reason for the overheating? Or does Amazon overbook the cooling in their data centers?
show comments
tornikeo
I wonder if hetzner had better uptime in EU than AWS this year.
It's always East 1... Jokes aside I don't understand how often east-1 is taken down compared to other regions. Like it should be pretty similar to other regions architecture wise.
show comments
fastest963
Coinbase claimed multiple AZs were down but the AWS statement was that only a single AZ was affected. Does anyone have more details?
show comments
whatever1
2/last 365 days down. My Ubuntu nas is 0/last 365days down.
Come and give me your cash if you want resilience.
show comments
Havoc
Could someone explain to me why they don't build these things near oceans? Like nuclear plants that need plenty cooling capacity too
Two loop cycle with heat exchanger to get rid of the heat
show comments
sitzkrieg
using aws since s3 came out and i’ve yet to see any major company do multi az failover in any capacity whatsoever. default region ftw
show comments
yomismoaqui
How many nines of are we at this year?
show comments
matt3210
Right, cooling.
nikcub
both realtime markets where multi-AZ is hard?
show comments
jeffbee
I don't see anything on downdetector suggesting this was particularly disruptive.
aussieguy1234
Once known for having super reliable services, I've heard this company is scrambling to re hire some of the engineers they overconfidently "replaced" with AI.
When customers pay for cloud services, they expect them to be maintained by competent engineers.
edit: Not sure why the downvotes. If you fire the engineers that have been keeping your systems running reliably for years, what do you expect to happen?
show comments
tcp_handshaker
I bet post-mortem will say vibe coding confused fahrenheit and celsius, we run too hot...
show comments
BugsJustFindMe
[flagged]
tailscaler2026
us-east-1 is down? shocking! stop putting SPOF services there. this location has had frequent issues for the past 15 years.
show comments
rswail
So in the comments here we have the usual about us-east-1, it's centralized, it's a SPOF for AWS, they should fix it, don't put your stuff there, etc.
This was one data centre in one zone of a multi-zone region.
Yes IAM/R53 and others are centralized there, yes, reworking those service to be decentralized and cross-region would be a Good Thing. But us-east-1 is already multi-zone (6 with a seventh marked as "coming in 2026") with multi DC within zones. From memory, when a global service like IAM is out, it's more likely to be bugs in the implementation or dependency than a "if this was cross-region it wouldn't have died" issue.
But this wasn't an outage of any AWS global service this time. The only one that seemed to have more impact was/is MSK. Which is likely to be more of an issue with Kafka than anything AWS related.
AWS’s US-East 1 continues to be the Achilles heel of the Internet.
And while yes building across multiple regions and AZs is a thing, AWS has had a string of issues where US-East 1 has broader impacts, which makes things far less redundant and resilient than AWS implies.
These things are dangerous. Someone who can take AWS down such as an employee can place a bet.
These bets aren’t as innocent as they seem because the bettors can often influence or change the outcome.
There sure are a lot of eggs in that East basket.
I thought cooling was pretty much pre-planned in any data center, and you simply don't install more stuff than you can cool?
So did some cooling equipment fail here or was there an external reason for the overheating? Or does Amazon overbook the cooling in their data centers?
I wonder if hetzner had better uptime in EU than AWS this year.
Related:
AWS EC2 outage in use1-az4 (us-east-1)
https://news.ycombinator.com/item?id=48057294
It's always East 1... Jokes aside I don't understand how often east-1 is taken down compared to other regions. Like it should be pretty similar to other regions architecture wise.
Coinbase claimed multiple AZs were down but the AWS statement was that only a single AZ was affected. Does anyone have more details?
2/last 365 days down. My Ubuntu nas is 0/last 365days down.
Come and give me your cash if you want resilience.
Could someone explain to me why they don't build these things near oceans? Like nuclear plants that need plenty cooling capacity too
Two loop cycle with heat exchanger to get rid of the heat
using aws since s3 came out and i’ve yet to see any major company do multi az failover in any capacity whatsoever. default region ftw
How many nines of are we at this year?
Right, cooling.
both realtime markets where multi-AZ is hard?
I don't see anything on downdetector suggesting this was particularly disruptive.
Once known for having super reliable services, I've heard this company is scrambling to re hire some of the engineers they overconfidently "replaced" with AI.
When customers pay for cloud services, they expect them to be maintained by competent engineers.
edit: Not sure why the downvotes. If you fire the engineers that have been keeping your systems running reliably for years, what do you expect to happen?
I bet post-mortem will say vibe coding confused fahrenheit and celsius, we run too hot...
[flagged]
us-east-1 is down? shocking! stop putting SPOF services there. this location has had frequent issues for the past 15 years.
So in the comments here we have the usual about us-east-1, it's centralized, it's a SPOF for AWS, they should fix it, don't put your stuff there, etc.
This was one data centre in one zone of a multi-zone region.
Yes IAM/R53 and others are centralized there, yes, reworking those service to be decentralized and cross-region would be a Good Thing. But us-east-1 is already multi-zone (6 with a seventh marked as "coming in 2026") with multi DC within zones. From memory, when a global service like IAM is out, it's more likely to be bugs in the implementation or dependency than a "if this was cross-region it wouldn't have died" issue.
But this wasn't an outage of any AWS global service this time. The only one that seemed to have more impact was/is MSK. Which is likely to be more of an issue with Kafka than anything AWS related.