A false smoke alarm almost drowned Jane Street's only NYC datacenter

525 points · 254 comments on HN · read original →

Points and comments are a snapshot, not live.

A false smoke alarm nearly triggered sprinklers in Jane Street's only NYC datacenter.

A systems engineer recounts how contractors' dust set off a smoke alarm in Jane Street's office-building datacenter. The alarm shut down cooling CRAC units and pressurized sprinklers. Without cooling, room temperature rose toward the sprinkler activation threshold of about 150°F. The engineer sprinted from his nearby apartment and manually pulled power cords on every device until temperature stopped climbing at 143.6°F. The incident happened after markets closed, causing minimal downtime. Afterwards, the engineer built a temperature monitor in OCaml that averages each server's onboard sensor readings and alerts the team if elevated.

What commenters are saying

Commenters share sympathy for being the close-by employee called for emergencies. One recounts foolishly telling coworkers about his short commute, leading to panicked calls on weekends. Another warns against divulging personal details at work, citing a manager who exploited his known insomnia for late-night tasks. A subthread debates fire suppression choices: Halon and inert gas systems versus sprinklers, noting Halon is banned and alternatives exist. Some discuss building constraints, citing 33 Thomas Street as a dedicated datacenter building.