Posted on 11/27/2012 8:52:35 AM PST by GangnamNerd
The Obama campaign's technologists were tense and tired. It was game day and everything was going wrong.
Josh Thayer, the lead engineer of Narwhal, had just been informed that they'd lost another one of the services powering their software. That was bad: Narwhal was the code name for the data platform that underpinned the campaign and let it track voters and volunteers. If it broke, so would everything else.
They were talking with people at Amazon Web Services, but all they knew was that they had packet loss. Earlier that day, they lost their databases, their East Coast servers, and their memcache clusters. Thayer was ready to kill Nick Hatch, a DevOps engineer who was the official bearer of bad news. Another of their vendors, PalominoDB, was fixing databases, but needed to rebuild the replicas. It was going to take time, Hatch said. They didn't have time.
They'd been working 14-hour days, six or seven days a week, trying to reelect the president, and now everything had been broken at just the wrong time. It was like someone had written a Murphy's Law algorithm and deployed it at scale.
And that was the point. "Game day" was October 21. The election was still 17 days away, and this was a live action role playing (LARPing!) exercise that the campaign's chief technology officer, Harper Reed, was inflicting on his team. "We worked through every possible disaster situation," Reed said. "We did three actual all-day sessions of destroying everything we had built."
(Excerpt) Read more at theatlantic.com ...
Born on 11/27/2012.
Intersting, but not necesarily relevant. We all started somewhere.
Everyone should read this. Aside from everything else, we also just got outsmarted technologically on November 6. There’s no excuse for that.
Absolutely no excuse. Calling what happened re: ORCA being "outsmarted technologically" is, I think, a massive understatement. The Romney campaign rolled out this piece of software on election day, without adequate testing and, most shockingly, without backup (my jaw hit the floor when I read that field offices didn't even have manual strike lists to use in the event of problems with ORCA). I just can't understand why they would put themselves in a position where one glitch could effectively render thousands of volunteers useless on Election day (not to mention the many more thousands of volunteers and volunteer man-hours that went into preparing for Election day GOTV).
That's not just a technical glitch, or being "outsmarted technically." That's just old-fashioned horrible planning. It's mind-boggling.
Disclaimer: Opinions posted on Free Republic are those of the individual posters and do not necessarily represent the opinion of Free Republic or its management. All materials posted herein are protected by copyright law and the exemption for fair use of copyrighted works.