Jump to content

Major Server Problems Today


Casper

Recommended Posts

  • Replies 97
  • Created
  • Last Reply

Top Posters In This Topic

So did the system crash because of all the hits, what server are you running, and what do you use for backing up your data.

We just lost a server at work and had to have it back up and running that day. I feel your pain.

Oh yes, i have felt the server pain many days before...

Satan, if you need help with anything, let me know...

Link to comment
Share on other sites

My questions are these from an IT standpoint

1. Why the hell wasnt the OR host performing completion checks on backups. (Shitty host to me) Not to mention that they are not doing adequate backups. 2 fucking months, are you kidding me. They should be running full backups weekly and validating those and performing incrementals on another agreed upon basis. Sounds like they are doing full backups monthly and no incrementals at all. I would be getting my money back and find a different host.

2 Does the host ever do periodic restores to test their backup media for problems just like this.

3. What caused the crash in the first place.

Link to comment
Share on other sites

oh damn this sucks!! lol i know its not your fault ben, but it still sucks.

had to reset my password to sign in, then i lost all my friends and albums and avatar..lol it'll take me awhile to get it back where i had it.

sounds like we need a bigger server/host...

Link to comment
Share on other sites

To answer a few questions....

What happend?

Several really bad things happened all at the same time. I have a virtual dedicated server hosted out on the east coast. Apparently, they do not do backups of the virtual dedicated servers. I didn't know this. I've been slacking on verifying my own backups. I had it set to run a backup every night, then FTP the backup to another server. Apparently the SFTP server had a harddrive fill up to 75% and stop accepting incoming files. Had I been checking this, I would've known. It was my own fault. I'm still not 100% sure what happened to the database to begin with. While dropping a table, somehow I managed to drop the database. I went over the code a million times. I still have no idea how the database got dropped. However, usually this wouldn't be a big deal. I just restore to the last backup then roll forward the binary logs. All is good. Unfortunately, the backups were not working properly, and the binary log directory wasn't logging the correct database. Why not the correct database? Well back when I upgraded the vBulletin software in August, I had made a copy of the database to use for testing. I went ahead and used the test database for the production site. Somehow, the test database never got added to the list of MySQL databases in the my.cnf file to be logged. It should've been automatic, but I didn't check it. Had the server been here locally, I would've been able to pull the data off the harddrive with no problems. But since it's not here, and it's virtual, there was no way for me to do this. So, several bad things all led to one big boo boo.

How do we keep it from happening again?

For the immediate solution, I have written a shell script which is ran in CRON every day at 4a. The script does a MySQL dump of the database, compresses and encrypts it, makes two local copies of it, then SFTPs it to my remote server. My remote server then emails me when the file is received. For a long term solution, I'm looking into hosting the server here locally. Had the hardware been here, or at least not virtual, I would've been able to unmount the device, run LSOF, and pull the files off the disk. Unfortunately, I have no control over the hardware the way things are now. I'm looking into either a local co-lo or possibly hosting it myself.

What about my post count?

Everybody's posts from the past two months are gone. I can add to your post count, but the next time I rebuild/optimize the database your post count will go back down. Seems quite pointless to me. Just post more. Haha.

PMs, posts, pictures, etc all gone?

Yes. EVERYTHING from the past two months is gone. Name changes, private messages, profile messages, friend lists, social groups, etc...... I'm just thankful we didn't lose more than we did.

Cliff's Notes:

Shit's gone. Ain't coming back. I suck. Shouldn't happen again. I'm sorry.

Link to comment
Share on other sites

.... I suck....

That's what we like about you.....and what your wife LOVES about you!

If anyone's livelihood depended on the data, then you can beat yourself up about it. I mean, you aren't getting paid for OR, and it's being done in your spare time, so we get to rehash 2 months of political debates, and have to repost a bunch of porn.......life goes on.

Link to comment
Share on other sites

At work we have a poweredge 2650 if you are possibly interested in hosting locally. You would need to get bigger harddrives as it only has 5 16gb drives in it now. PM me if you would like to know more about it. It is a rack mount server though.

Thank you for getting the site back up.

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.


×
×
  • Create New...