I don’t know if I’ve told this story before, but at a very large and well known financial company, the testing lab signed off on a new image that was to be pushed out to company desktops.
For whatever reason, the company (which I won’t name) pushing out the image added a piece of software to the image that was pushed out.
Blew up 1/3 of the desktops. The only reason all of the desktops weren’t blown is that the image was only pushed out to 1/3 of the computers.
1,000 computers were put out of commission.
The company pushing out the image had added a virus protection program to an image that already had a virus protection program. The financial company got the computers back online by disabling all virus protection.
I company I worked at was expanding their datacenter--not the physical space, because they had/have plenty of room. No, they needed more clusters, so they bought 13 more 70-node clusters from their vendor.
Their cooling system couldn't handle it as I began turning them on. They couldn't get any more big chillers that quickly, so they ended up renting a "portable" chiller for several months, just to keep this (very large) data center semi-cool.