Database nightmare

Its 12:30AM (00:30 for you Euros).  I am watching The Daily Show on Tivo.  All is well.  Then the phone beeps.

MySQL Main is critical

SSH?  no.

Digi console?  no.

About a week ago, we had a mysqldump file that was corrupt.  We cleaned it up.  My worst fears came to my mind.  We tried power cycling it.  It did not come back.

While my coworker was dealing with the facilily people, I worked on the backup server.  Had to ensure the last full backup was in place and apply the incremental data.  Suddenly, my SSH connection dies.  OMG.  THAT DUMB A** GUY POWER CYCLED THE WRONG BOX!!! — FS corrupted.  Damn you ReiserFS!

By now, it’s 4AM.  Tech took an hour to get to the rack.  It is 20 feet from his cubicle.  I get in the car.  I am two hours away from the facility.  No sleep.  It is still dark.  I play loud music.  I talk to myself.  I curse the guy that power cycled the wrong box. The sun comes up and it is easier to drive.

I sit here, waiting on the OS to finish installing so I can restore the backup and incremental data again.  Hours of content lost.  The content team is hand writing HTML that other developers are rsyncing around to the servers.

The good news?  All the work done in 2007 to separate our front end and backend worked.  The front end works fine (99%).  Just no new content.  Well, except for the hand done HTML.

Note to self: Get that main database replication working again.  ASAP.

Advertisements

4 Responses to Database nightmare

  1. x says:

    Using reiserfs is like playing russian rulette. xfs is the only solution…

  2. Brian Moon says:

    yeah, all going to xfs now.

  3. Jason cook says:

    Lights out cards like Dell’s Drac and pxe boot for install are wonderful things to have in a remote data center. I hate having to go there for a simple thing like a re-install.

  4. […] MySQL, Scalability by Brian Moon on the August 17, 2008 So, I wrote about the begining of our wild database issues. Since then, I have been fighting a cold, coaching little league football and trying to help out in […]

%d bloggers like this: