It's 5:20 AM here in Europe and I just rotated a bunch of keys.
Only to read,
> Because deployments are paused, you will not be able to redeploy your services to pick up the new credentials until builds are restored. We will post here when builds resume with any additional details.
So now my service does not work. Which is worse: a stolen key or a broken product? I'm genuinely regretting following their advice on rotating keys, because logs showed no sign of incorrect usage, and our service would at least have remained online.
I hope that they automatically redeploy their customer services. I just worry that I won't have tested it. I mean: 5AM. Keys changed. Nothing more I can do. Should go back to sleep. No timeline to wake up to check deployments, so... I... wait?
show comments
vldszn
Crazy day for cybersecurity: PHP RCE, eMudhra certificates, and now Railway.
Yup, got my OpenRouter key. They managed to run up a whole $2 on Opus 5.5 and Fable. Well, time for a new hosting provider I guess...
Incidentally, I'm testing Muse as an email triage agent today - it buried "oh also, railway and anthropic say yours keys have been leaked and you need to rotate asap" under 50 routine deploy / admin emails notifications. Oh to live through such an AI powered software renaissance!
show comments
fbrncci
Right just as I take my first day off (for the entire year!) from work to fly out for a vacation. The mother of all security bombs starts dropping all around stacks and services that I am responsible for. How else could have taken time off go for me…
show comments
MattIPv4
Seen multiple folks in the Railway Discord confirming that their secrets stored in Railway have been used by third parties, so it definitely sounds like the attackers were able to read decrypted environment variables.
show comments
nosefrog
Railway was not honest about the reason for their last big outage.
martinald
I have a bad gut feeling that every day is going to look a bit like today with endless urgent patches and breaches.
Fwiw it sounds similar to the hack over the weekend that opencode suffered.
show comments
alpineman
Train wreck. People were writing on discord 3 hours ago, 10 hours after the incident started, that keys created just for railway were running up usage. Meanwhile builds have been down for more than 12 hours. Only NOW did they just send out an email to all customers. I wouldn't have found out until then if I wasn't checking the discord periodically.
show comments
averrous
Anyone can help?
I changed he api key in the env var as openrouter revoke the previous one based on this incident. However, as the redeploy cannot be done in my region, the env cannot be updated and now the product cannot be used. Any command in railway to fix this?
fyzix
How is it possible rotate keys if you disable deployments? You're basically telling us to take our services offline...
show comments
DrawTR
Jeez, what a rough past 24 hours for all security folk. Hang in there
show comments
AJRF
Do cloud providers have a lock-down mode where you can press 1 button and stop all spend?
show comments
jasona123
sighs
Thankfully I'm only using Railway for a hobby project (aptly, for a real-time transit dashboard) but still
minraws
how are the secrets decrypted, I would like some explanation so that I can have some understanding here, I am starting to feel like all these companies are just meme-ing with their security posture.
Like did someone vibe code too hard, how do env secrets leak for everyone...
show comments
1-6
It's time for competent programmers to get rehired to help patch massive exploits that are about to unfold before us.
cowlby
This is so fucked up. I was reviewing my account and I just saw this in the logs... Claude says database credentials and bucket credentials go in via env vars so likely they had access to database and bucket contents.
Audit Logs
Oct 21, 2026 05:54:13 PM CDT
BucketInstanceCredential.regenerated in my-project production by Railway Staff
Oct 21, 2026 05:54:13 PM CDT
BucketInstanceCredential.regenerated in my-project production by Railway Staff
Oct 21, 2026 05:54:13 PM CDT
BucketInstanceCredential.regenerated in my-project production by Railway Staff
paxy
It’s funny that every security breach disclosure now has a mandatory “the attacker used AI!” as if it absolves you from having to maintain a secure system. If anything it makes the situation more damning. If it was possible to find exploits using AI then your own red teaming should have found them before the attacker did.
show comments
tstrebitzer
meanwhile on railway.com:
Ship software peacefully
dylanz
I'm rotating production secrets all night... great. I confirmed that they've actually used many of my secret keys already.
show comments
arbayi
i revoked my api keys, changed with new ones but guess what? railway does not allow deployments at the moment, so my app is down.
gbnwl
The past two days have been awful as someone who uses Railway.
eecks
Huh, I've been using Railway tonight and this was the first I've seen of this. Just got an email about it now too at 23:36
If you're on Railway do not wait, ROTATE every API key at the source, and do not set new values because Railway is still compromised and not secured.
ksajadi
someone managed to get the environment variables AND the encryption keys for those? i feel for the secops engineers over there right now. brutal
wateralien
Is this our new AI daily reality?
show comments
shinycode
I’m lucky I moved away from it to host my own db with coolify
show comments
rpmisms
Aw shit. I just got done patching our PHP servers, now I gotta do the Railway too? I swear I'm gonna wake up to a Vercel SNAFU.
show comments
AndrewKemendo
Yeah I’m not sure what happened but there was a huge bot swarm the last few days
I had an open gitea host pulled into some botnet
Luckily digitaloceam sent an alert.
sergiotapia
Actually I will move my projects away from Railway. With AI what it is today, no reason to shy away from hosting on actual VPS.
What is the best VPS to use today? Is it still hetzner or ovh?
barelysapient
Ugh. Love Railway.
lijok
Damn I guess all those l33t hiring practices were for nothing
show comments
Keloran
Well this is certainly interesting, I wonder how their secrets were being stored, I assumed they were stored in vault and then injected with a sidecar, but I guess not
goforbg
the worst service I've signed up for has to be railway. What a terrible product.
esseph
Fuck, I thought this was about trains
torben-friis
>We believe this is part of a series of attacks recently targeting platform companies with the aim of compromising AI inference providers.
Is that an accusation? As in, major AI players taking out competitors ?
show comments
visch
This is exactly why we should own our deployment stacks. Going to push up the launch here as pdt is a railway alternative thttps://github.com/autoidm/pdt for scheduled jobs
AI makes this so much easier to do now, we should all own our own code, and deployment infrastructure (use a cloud instead of someone wrapping aws for you)
It's 5:20 AM here in Europe and I just rotated a bunch of keys.
Only to read,
> Because deployments are paused, you will not be able to redeploy your services to pick up the new credentials until builds are restored. We will post here when builds resume with any additional details.
So now my service does not work. Which is worse: a stolen key or a broken product? I'm genuinely regretting following their advice on rotating keys, because logs showed no sign of incorrect usage, and our service would at least have remained online.
I hope that they automatically redeploy their customer services. I just worry that I won't have tested it. I mean: 5AM. Keys changed. Nothing more I can do. Should go back to sleep. No timeline to wake up to check deployments, so... I... wait?
Crazy day for cybersecurity: PHP RCE, eMudhra certificates, and now Railway.
And it’s all just getting started imo.
https://blog.cloudflare.com/emudhra-certificate-compromise
https://www.wiz.io/blog/php2shell-php-rce
Postiz founder posted on X that his service is being brought down, his keys were already used and Railway was quiet about this until he spoke up:
https://x.com/wickedguro/status/2113015432974405658
Yup, got my OpenRouter key. They managed to run up a whole $2 on Opus 5.5 and Fable. Well, time for a new hosting provider I guess...
Incidentally, I'm testing Muse as an email triage agent today - it buried "oh also, railway and anthropic say yours keys have been leaked and you need to rotate asap" under 50 routine deploy / admin emails notifications. Oh to live through such an AI powered software renaissance!
Right just as I take my first day off (for the entire year!) from work to fly out for a vacation. The mother of all security bombs starts dropping all around stacks and services that I am responsible for. How else could have taken time off go for me…
Seen multiple folks in the Railway Discord confirming that their secrets stored in Railway have been used by third parties, so it definitely sounds like the attackers were able to read decrypted environment variables.
Railway was not honest about the reason for their last big outage.
I have a bad gut feeling that every day is going to look a bit like today with endless urgent patches and breaches.
Fwiw it sounds similar to the hack over the weekend that opencode suffered.
Train wreck. People were writing on discord 3 hours ago, 10 hours after the incident started, that keys created just for railway were running up usage. Meanwhile builds have been down for more than 12 hours. Only NOW did they just send out an email to all customers. I wouldn't have found out until then if I wasn't checking the discord periodically.
Anyone can help?
I changed he api key in the env var as openrouter revoke the previous one based on this incident. However, as the redeploy cannot be done in my region, the env cannot be updated and now the product cannot be used. Any command in railway to fix this?
How is it possible rotate keys if you disable deployments? You're basically telling us to take our services offline...
Jeez, what a rough past 24 hours for all security folk. Hang in there
Do cloud providers have a lock-down mode where you can press 1 button and stop all spend?
sighs
Thankfully I'm only using Railway for a hobby project (aptly, for a real-time transit dashboard) but still
how are the secrets decrypted, I would like some explanation so that I can have some understanding here, I am starting to feel like all these companies are just meme-ing with their security posture.
Like did someone vibe code too hard, how do env secrets leak for everyone...
It's time for competent programmers to get rehired to help patch massive exploits that are about to unfold before us.
This is so fucked up. I was reviewing my account and I just saw this in the logs... Claude says database credentials and bucket credentials go in via env vars so likely they had access to database and bucket contents.
It’s funny that every security breach disclosure now has a mandatory “the attacker used AI!” as if it absolves you from having to maintain a secure system. If anything it makes the situation more damning. If it was possible to find exploits using AI then your own red teaming should have found them before the attacker did.
meanwhile on railway.com:
Ship software peacefully
I'm rotating production secrets all night... great. I confirmed that they've actually used many of my secret keys already.
i revoked my api keys, changed with new ones but guess what? railway does not allow deployments at the moment, so my app is down.
The past two days have been awful as someone who uses Railway.
Huh, I've been using Railway tonight and this was the first I've seen of this. Just got an email about it now too at 23:36
Related: https://news.ycombinator.com/item?id=50177413
If you're on Railway do not wait, ROTATE every API key at the source, and do not set new values because Railway is still compromised and not secured.
someone managed to get the environment variables AND the encryption keys for those? i feel for the secops engineers over there right now. brutal
Is this our new AI daily reality?
I’m lucky I moved away from it to host my own db with coolify
Aw shit. I just got done patching our PHP servers, now I gotta do the Railway too? I swear I'm gonna wake up to a Vercel SNAFU.
Yeah I’m not sure what happened but there was a huge bot swarm the last few days
I had an open gitea host pulled into some botnet
Luckily digitaloceam sent an alert.
Actually I will move my projects away from Railway. With AI what it is today, no reason to shy away from hosting on actual VPS.
What is the best VPS to use today? Is it still hetzner or ovh?
Ugh. Love Railway.
Damn I guess all those l33t hiring practices were for nothing
Well this is certainly interesting, I wonder how their secrets were being stored, I assumed they were stored in vault and then injected with a sidecar, but I guess not
the worst service I've signed up for has to be railway. What a terrible product.
Fuck, I thought this was about trains
>We believe this is part of a series of attacks recently targeting platform companies with the aim of compromising AI inference providers.
Is that an accusation? As in, major AI players taking out competitors ?
This is exactly why we should own our deployment stacks. Going to push up the launch here as pdt is a railway alternative thttps://github.com/autoidm/pdt for scheduled jobs
AI makes this so much easier to do now, we should all own our own code, and deployment infrastructure (use a cloud instead of someone wrapping aws for you)