The article reminds me of when Hurricane Sandy took out a bunch of New York DCs with 3+ feet of water and people were carrying diesel up stairs to keep generators running.
“The superhuman folks at the data center are hauling 55 gallon drums of diesel fuel up 17 flights of stairs. Assuming they're able to sustain a rate of transfer that exceeds the rate of consumption, the data center should run until electricity is restored. The latest estimate from Con Edison for lower Manhattan is 4 days. (The New York Stock Exchange will be open tomorrow, also running on generator power.)”
In the article he talks about living within a few minutes walk to work and telling too many people so they made him the emergency office guy. Same thing happened to me, foolishly told everyone hey I can see my apartment from here so I would get panicked calls to run in and shut down servers because the information security guy got a tip of an impending exploit about to be released on a holiday weekend with no way to do it remotely. Everyone else lived a ferry ride or commuter train away.
show comments
rdl
Odd that they didn't have at minimum 1) a physical Emergency Power Off (EPO) button inside the datacenter, such that one button could de-energize everything -- usually required as part of building a datacenter due to code variance for the "low voltage" (<4000V in this context...) wiring. 2) EPO interlink to the smoke alarms, such that it would de-energize equipment automatically (maybe with delay) if smoke detected.
show comments
rainbowzootsuit
Interesting. The Sequence of Operations we write for the data halls I work on specifically do not shut down the CRACs on smoke alarm for this reason. The International Mechanical Code (IMC) specifically allows this where a air handler over 2000 cfm serves only one space because it's recognized that it will not spread smoke around a building. 2000 cfm is the threshold where you typically have to install smoke detectors, and typically they do shut down on smoke detection.
The CRAC won't feed a fire oxygen as it's just recirculating the same oxygen depleted air.
show comments
jddj
The episode of their podcast where Ron explores the facilities side of their data centers and offices is worth a listen too
Are there internet connected fire alarms that can be hacked to destroy the contents of some data centers? Like the European freezers?
andix
I did not know that data centers with sprinklers exist. I guess this is a very stupid idea in the first place.
If there is a fire the decision is: all equipment might be lost in a fire OR all equipment will be lost by water damage.
show comments
cesarb
> every minute, ask every server in each location directly for the temperature from its onboard sensors, take the average, and if it’s elevated, blast an alert out to everyone on the systems team to check for datacenter overheat.
I've once done the opposite: the office "server room" (a small room with three racks, two of them for the patch panels, one for the servers and switches) was cooled by a mini-split which was prone to locking up with the compressor enabled (the relay which controls the compressor got stuck) and freezing, which then stopped the air from circulating due to all the ice. I wrote a script which ran on one of the servers which read the ambient temperature through IPMI to detect when the room was too cold, email a warning, and power off (using ssh) the neighboring servers and then itself.
thibsrn
Great story, I love those in-the-trenches emergency stories. I wonder if it would have impacted Jane Street in any meaningful way, aside from a fat bill, if it ended with submarine servers. They could have diverged their compute to another datacenter surely.
We mostly hear about engineers from Jane Street here, and not much from their traders and finance people. I wish we could have some other perspective sometimes.
They publish a great podcast, Signal and Threads, I really recommend.
show comments
sciurus
I wonder why the answer was to turn off the servers instead of turn their cooling back on. Would it not be able to stabilize the temperature fast enough?
show comments
rhipitr
I thought these data centers have a way to pump in a gas that prevents fires? Remember touring a data center at some point that has this ability.
show comments
RomanKornev
Even though this happened in 2012, it's interesting that Jane Street is doubling down on compute and renting even more capacity from neoclouds. They are planning to expand 10x the amount of GPUs and are locking in tons of compute. On top of the $500M+ per MW of compute that they are paying Anthropic.
show comments
xhrpost
I specifically remember a TV commercial many years ago (15-20 maybe) that was trying to show the sophistication of the company with an anecdote of how when looking at some other company as a potential investment, considered if they used sprinklers in their data centers and determined that they did not. I could have sworn the company advertising was Jane Street.
show comments
mvcosta91
TIL: wabi-sabi. I’ve been trying to find a way to express the beauty of imperfection in the age of AI, and this fits perfectly.
show comments
liendolucas
Great post. I hope a big paycheck was given to the author for saving the DC and the company from disaster.
Many many years ago someone told me that your data is not safe if is not at least in 3 very different physical places. Today, I would probably add the requirement to be on different providers as well.
nunez
good to see you make front mbac; glad you're still holding it down!!!!
> the smoke alarm is wired to shut down the CRAC units
Why not just turn the CRAC back on?
> you don’t trust the automation unless there’s a big red-painted, mechanically connected lever you can pull to turn it off
Seems like this is one of those cases where you want to override the automation.
show comments
freddealmeida
I did not know Jane Street had a datacenter in NYC. Must of had it a long time ago.
arjgoel30
Wow crazy I didn't know they had such simple systems
amluto
I’m surprised by the pulling cords by hand part. Did they really not have an emergency power off wired up to a shunt trip or some other device to efficiently cut the power? Isn’t this mandatory, even? Especially in a datacenter in an office building?
green-salt
This would be the one time I get to hit the EPO button (which this DC seemed shockingly too ghetto to have) for our breakers and it not being a resume-generating event.
stevefan1999
Sounds like a good idea to hack in and blow up the infrastructure of the hedge funds. How you gonna hedge this suckers
philipwhiuk
Almost too close to 'Pool on the roof must have a leak' to be real ;)
show comments
caycep
Ah the apocryphal VAX story/scenario is alive and well
The article reminds me of when Hurricane Sandy took out a bunch of New York DCs with 3+ feet of water and people were carrying diesel up stairs to keep generators running.
“The superhuman folks at the data center are hauling 55 gallon drums of diesel fuel up 17 flights of stairs. Assuming they're able to sustain a rate of transfer that exceeds the rate of consumption, the data center should run until electricity is restored. The latest estimate from Con Edison for lower Manhattan is 4 days. (The New York Stock Exchange will be open tomorrow, also running on generator power.)”
https://news.ycombinator.com/item?id=4720388 (https://web.archive.org/web/20121101194314/http://status.fog...)
https://arstechnica.com/information-technology/2012/11/how-o...
In the article he talks about living within a few minutes walk to work and telling too many people so they made him the emergency office guy. Same thing happened to me, foolishly told everyone hey I can see my apartment from here so I would get panicked calls to run in and shut down servers because the information security guy got a tip of an impending exploit about to be released on a holiday weekend with no way to do it remotely. Everyone else lived a ferry ride or commuter train away.
Odd that they didn't have at minimum 1) a physical Emergency Power Off (EPO) button inside the datacenter, such that one button could de-energize everything -- usually required as part of building a datacenter due to code variance for the "low voltage" (<4000V in this context...) wiring. 2) EPO interlink to the smoke alarms, such that it would de-energize equipment automatically (maybe with delay) if smoke detected.
Interesting. The Sequence of Operations we write for the data halls I work on specifically do not shut down the CRACs on smoke alarm for this reason. The International Mechanical Code (IMC) specifically allows this where a air handler over 2000 cfm serves only one space because it's recognized that it will not spread smoke around a building. 2000 cfm is the threshold where you typically have to install smoke detectors, and typically they do shut down on smoke detection.
The CRAC won't feed a fire oxygen as it's just recirculating the same oxygen depleted air.
The episode of their podcast where Ron explores the facilities side of their data centers and offices is worth a listen too
https://signalsandthreads.com/the-thermodynamics-of-trading/
Are there internet connected fire alarms that can be hacked to destroy the contents of some data centers? Like the European freezers?
I did not know that data centers with sprinklers exist. I guess this is a very stupid idea in the first place.
If there is a fire the decision is: all equipment might be lost in a fire OR all equipment will be lost by water damage.
> every minute, ask every server in each location directly for the temperature from its onboard sensors, take the average, and if it’s elevated, blast an alert out to everyone on the systems team to check for datacenter overheat.
I've once done the opposite: the office "server room" (a small room with three racks, two of them for the patch panels, one for the servers and switches) was cooled by a mini-split which was prone to locking up with the compressor enabled (the relay which controls the compressor got stuck) and freezing, which then stopped the air from circulating due to all the ice. I wrote a script which ran on one of the servers which read the ambient temperature through IPMI to detect when the room was too cold, email a warning, and power off (using ssh) the neighboring servers and then itself.
Great story, I love those in-the-trenches emergency stories. I wonder if it would have impacted Jane Street in any meaningful way, aside from a fat bill, if it ended with submarine servers. They could have diverged their compute to another datacenter surely.
We mostly hear about engineers from Jane Street here, and not much from their traders and finance people. I wish we could have some other perspective sometimes.
They publish a great podcast, Signal and Threads, I really recommend.
I wonder why the answer was to turn off the servers instead of turn their cooling back on. Would it not be able to stabilize the temperature fast enough?
I thought these data centers have a way to pump in a gas that prevents fires? Remember touring a data center at some point that has this ability.
Even though this happened in 2012, it's interesting that Jane Street is doubling down on compute and renting even more capacity from neoclouds. They are planning to expand 10x the amount of GPUs and are locking in tons of compute. On top of the $500M+ per MW of compute that they are paying Anthropic.
I specifically remember a TV commercial many years ago (15-20 maybe) that was trying to show the sophistication of the company with an anecdote of how when looking at some other company as a potential investment, considered if they used sprinklers in their data centers and determined that they did not. I could have sworn the company advertising was Jane Street.
TIL: wabi-sabi. I’ve been trying to find a way to express the beauty of imperfection in the age of AI, and this fits perfectly.
Great post. I hope a big paycheck was given to the author for saving the DC and the company from disaster.
Many many years ago someone told me that your data is not safe if is not at least in 3 very different physical places. Today, I would probably add the requirement to be on different providers as well.
good to see you make front mbac; glad you're still holding it down!!!!
remember sandy? that was crazy
How is it the only datacenter?
33 Thomas St is basically one huge datacenter, that's why it doesn't have any windows. https://en.wikipedia.org/wiki/33_Thomas_Street
> the smoke alarm is wired to shut down the CRAC units
Why not just turn the CRAC back on?
> you don’t trust the automation unless there’s a big red-painted, mechanically connected lever you can pull to turn it off
Seems like this is one of those cases where you want to override the automation.
I did not know Jane Street had a datacenter in NYC. Must of had it a long time ago.
Wow crazy I didn't know they had such simple systems
I’m surprised by the pulling cords by hand part. Did they really not have an emergency power off wired up to a shunt trip or some other device to efficiently cut the power? Isn’t this mandatory, even? Especially in a datacenter in an office building?
This would be the one time I get to hit the EPO button (which this DC seemed shockingly too ghetto to have) for our breakers and it not being a resume-generating event.
Sounds like a good idea to hack in and blow up the infrastructure of the hedge funds. How you gonna hedge this suckers
Almost too close to 'Pool on the roof must have a leak' to be real ;)
Ah the apocryphal VAX story/scenario is alive and well
I am shocked to hear they do not use foam.
Related :) - https://news.ycombinator.com/item?id=50434310
He earned his bonus that year.
Man, I wish MOC were still alive. I’d love to know his take on this story.
Surprised they even had sprinklers set up. Historically, things that avoid moisture like Halon Gas [1] have been used.
[1] - https://en.wikipedia.org/wiki/Bromotrifluoromethane
> Everyone’s heard of how awesome Jane Street is.
I guess I’m the last one to hear it.
should be drowning tbh, less market manipulation
Alarm firmware wasn't programmed in OCaml
why don't they have a dry sprinkler system?
Datacenters on the coast never made sense to me.
Reminds me of DirectNic's datacenter flooding during Katrina (New Orleans), we all kept posting on Somethingawful until they finally shut it down.
> a false smoke alarm, a datacenter with no cooling, and a sprint to pull every power cord before the sprinklers went off
I smell AI