Blog - fliiby.com : your file library ://

IT Hosting Companies - the other side of the medal

October 25, 2008 – 10:53 am

During last couple of days, as you may know, we experienced some down time. Whole thing started with Alphared Inc (http://alphared.com/) going down, since CCBill pulled funding from them. Our primary server was located at Alphared, and within minutes we decided to migrate to another IT hosting company… This is tremendous task, as moving terabytes of data is not something you do every day, so we started preparations and chose one and only Global IT Hosting Leader : The Planet (http://theplanet.com) to be Fliiby’s future infrastructure. ..

Server was ready within usually 72h, and migration process started. Whole Fliiby system worked fully, except upload - it was disabled. Copying of that much data, knows to give a  headaches, but we managed to move all our data from Alphared to the Planet with 0 seconds of downtime. DNS was changed and http://fliiby.com started serving users from the Planet - our new infrastructure. Latter on, we noticed that RAID5 level was not properly configured on our new server, so we asked the Planet to setup us a 2nd server with properly configured RAID5 level, and we would make another switch to it. As about us, it was better to make one more migration, then to let single HDD failure to damage the whole system.  The Planet agreed, and they started preparing 2nd server for us. And this is the part where Greg P. a.k.a. Natural disaster from the Planet came into the play. (not so) Dear Greg, managed to accidentally wipe out whole system from live server instead of 2nd. Within 4 minutes he destroyed 2 years of dedicated work and life of Fliiby team. Site was down.   After couple of hours, and talk with the Planet management, we did only thing we could : started another copying process from the Alphared to the 2nd server.

The problem was, after first migration, we slowly started removing copies of accounts and files from old server - Alphared site was down, cPanel licence expired, we could not make payment to them, and we did not want to leave whole copy of Fliiby on their hands. Whole 2nd copying process was pain in the ass, since the Planet accidentally put us on 10 Mbit port. And after the longest night, we managed to put whole system back to life, with downtime of ~48h and smaller data loss of ~2% (images and flash files).  We deeply apologies to our users who’s images and flash files are missing. We are already working on file replacement tool, so you could re-upload missing files to their original paths. And we took precautions that nothing like this ever happens again.

Whole this thing got us thinking how whole system is sensible, not just Fliiby, but the whole IT world…Maybe when people would do things with more love and more awake, situations like this could never occur. Till then, human factor remains number one vulnerability point in any system. As about the Planet, they apologize. They even gave us one month of free credit. But the loyalty of people, is something you just can’t buy with money…

Stay tuned
Fliiby

Tags: alpahared, backup, companies, data, disaster, down, FLIIBY, Hosting, human factor, IT, loss, loyalty, os reload, planet, theplanet, time, vulnerability poin