It’s not fun when you can’t access data, whether it’s a spreadsheet you’ve been working on for an hour or something bigger like, say, Craigslist.

I spend a lot of time dealing with servers and web hosting safety, so Techyshit has invited me to round up three of my favourite - and perhaps most preventable - data losses.

1 - Mysterious Outage in San Fran

(Do people still call it San Fran, or did we drop that in the 90s? Anyhow.)

In 2007, Valleywag published a tip-off that a data centre employee “came in shitfaced drunk, got angry, went berserk, and fucked up a lot of stuff” - resulting in sites such as Craigslist, LiveJournal and Technorati going down.

What Really Happened?

The data center, 365 Main, denied any “employee misconduct” and the blogger admitted that their tipster may have been less then reliable. The problems were blamed on a nearby power outage, but there was no explanation as to why 5 of their 10 backup generators didn’t kick in to save the day. Whatever happened, the original story is more fun.

Learning Experience

Of course, you should test your backups once in a while, but whatever happened, the least embarrassing excuse for the company to make may have been “Yeah, Dave got bladdered and tried to upgrade the server room with a wrench.”

Perhaps all data centers could try tempting drunk employees away from the sensitive areas of the building - a few strategically placed kebabs with chilli sauce in the meeting rooms ought to be irresistible enough to do the job.

2 - The Bitch Broke My Server

Two weeks into her new job as receptionist at Action Tools, Lori Stint had a day off to take her dog to the vet. She dropped by the office to pick up her first paycheck, when all hell broke loose and the company’s servers went down. A few minutes later, Lori was unemployed again.

What Happened?

The dog, left to its own devices, saw the floor standing computer rack as an opportunity to relieve itself. When the boss saw the “puff of urine smelling smoke” that had previously been his files and client contacts, he fired Lori on the spot.

Learning Experience

Seriously, its a pretty simple list. Don’t take your pets to work. Train your pet not to piss wherever it wants to. Don’t hire employees who like dogs, ever.

3 - Third Times the Charm

TheDailyWTF reported on problems at the new data centre for the Georgia Institute of Technology. Some servers were installed whilst the interior was being finished, and after diligent monitoring, this didn’t seem to cause any trouble.

After a few days, one server (critical to the Uni’s information systems) had a massive, non-recoverable failure. The disk was identified as the issue, and it was replaced. A few days later, the same thing happened again, and the vendor sent a second replacement disk.

When the disk failed for a third time, a technician came to check the failed component. Cracking open the disk showed that the disk head had cut “deep jagged grooves right on the platter”. Unless the server had gained some level of Skynet-esque self awareness and tried to kill itself, there must have been an explanation, but inspection of the server gave no other clues.

What Happened?

It turns out that the builders fitting out the rack room were using a single saw horse to prop up the wood they were cutting to size. The other end of each plank was rested on this server’s disk enclosure, before taking to it with a saber saw.

Learning Experience

The easy answer here would be to not move into an unfinished server room. More usefully (and more entertainingly) we suggest taking apart each and every computer component that ever fails. If a drive looks like it has stabbed itself through the chest, then you might want to wonder why.

Thanks to Zoe from web hosting company UK2 for this week’s guest post.

Del.Icio.Us - Digg it! - StumbleUpon - Float - Mixx it up! - Reddit