Its 12:30AM (00:30 for you Euros). I am watching The Daily Show on Tivo. All is well. Then the phone beeps.
MySQL Main is critical
Digi console? no.
About a week ago, we had a mysqldump file that was corrupt. We cleaned it up. My worst fears came to my mind. We tried power cycling it. It did not come back.
While my coworker was dealing with the facilily people, I worked on the backup server. Had to ensure the last full backup was in place and apply the incremental data. Suddenly, my SSH connection dies. OMG. THAT DUMB A** GUY POWER CYCLED THE WRONG BOX!!! — FS corrupted. Damn you ReiserFS!
By now, it’s 4AM. Tech took an hour to get to the rack. It is 20 feet from his cubicle. I get in the car. I am two hours away from the facility. No sleep. It is still dark. I play loud music. I talk to myself. I curse the guy that power cycled the wrong box. The sun comes up and it is easier to drive.
I sit here, waiting on the OS to finish installing so I can restore the backup and incremental data again. Hours of content lost. The content team is hand writing HTML that other developers are rsyncing around to the servers.
The good news? All the work done in 2007 to separate our front end and backend worked. The front end works fine (99%). Just no new content. Well, except for the hand done HTML.
Note to self: Get that main database replication working again. ASAP.