Posts Tagged with recovery

posted by brwyatt on April 22, 2013

Well... My server's primary system drive decided to develop a hugely large number of bad blocks. If you don't know what that is, the short answer is that a large part of the harddrive decided to simply stop working. There was dataloss. There was pain. There was frustration. It was terrible.

BUT! I managed to recover from a backup from 2012-12-20. The majority of important data was all on my RAID5 array, but this does mean that some configs were lost. Fortunately, this largely effects only me.

It looks like my BRGE and BadScience builds are intact (though probably pointed to the wrong repos), but my build for Bitcoin Access is gone. But email, MySQL, etc are all back up... and that's what matters for now. But I guess the rest of that goes on my "TODO" list... again.

On the bright side, I learned something totally awesome you can do with LVM2. PVMOVE. It didn't help in this case (the drive was too far dead and causing the CPU to get locked up with IOWaits), but holycrap. The whole concept of "oh, yeah, we can just copy all this data to another drive, sync them up, then stop using the old one and effectively move the partition to another drive" is just incredible. I love learning all the impossible things you can do on Linux that you just simply CAN'T do on Windows or OSX.

But anyway... I should probably get some sleep before REAL work tomorrow. But if anyone was wondering where the hell my server was this weekend... that should answer it.