Well, that was no fun


Derek Zeanah
March 19, 2008, 09:32 PM
I know the last 7 hours without THR have sucked. Believe me, they sucked more for me.

I tried defragmenting the THR database (which I've done countless times, and which used to be done nightly by a script) and things went to hell. Basically, the /tmp partition is apparently used by the database when optimizing tables, and I have it set to be 2 gigs max. Turns out the database is a bit big for a 2 gig /tmp partition, so the optimization attempt failed, the post table was marked bad, and I've been trying to make sure it was recovered without data loss.

Yay. Not.

So we're back up, without an optimized database. I'm taking the old server to the new datacenter this weekend (well, Friday if things go well) and I think I'll configure it as a SQL server without the limitations I've run into here. I believe I can do that securely provided I don't install web or mail services on it.

Anyway, that's the executive summary.

Sorry for the down-time.

If you enjoyed reading about "Well, that was no fun" here in TheHighRoad.org archive, you'll LOVE our community. Come join TheHighRoad.org today for the full version!
March 19, 2008, 09:33 PM
Thanks for all you do Derek. I don't think you're paid for your services. We all appreciate all you do to keep this big site alive on mysql.

March 19, 2008, 09:38 PM
Great going, you were missed.... Thanks for the notices of which I was able to see 2 , and for getting this site back on!

Thank you.... mac

March 19, 2008, 09:40 PM
techno-ignorant that i am you could have said the hamsters treadmill broke and i would believe you.
heart felt thanks for all your hardrwork

Larry Ashcraft
March 19, 2008, 09:42 PM
We appreciate everything you do, Derek.

AND, I actually got some work done this afternoon! :D

March 19, 2008, 09:47 PM
I don't understand any of that, but thank you for your hard work!

March 19, 2008, 10:16 PM
Derek, thanks for all your work to keep THR running for us. These happen from time to time. Really appreciate you posting that text message to let us know what was happening while the system was down.

Hope you kick back and have a cold brewski, decompress and have a good breather before getting into the new server.

You probably already know this, but I learned the hard way years ago: be sure the log files are on a different physical drive then the database and not on the root drive. It will work the other way, but if anything corrupts a drive then it can be hell to recover sometimes. Course if you are using RAID then that isn't such a big deal.

Thanks again, Derek!

March 19, 2008, 10:47 PM
Thanks for bringing the site back up!


March 19, 2008, 11:16 PM
Larry, hehe!

Thanks for the hard work, Derek. Despite what you may imagine, I don't pound on my keyboard and curse your name when the site doesn't come up. I've sat in that chair too often to do that. I understand, and it's alright.

March 19, 2008, 11:17 PM
Derek - Your efforts do not go unnoticed. Thanks.

March 20, 2008, 12:19 AM
Way to go Derek!!!!

I know what you put up with when working with computers.

I'm glad that you got this site back up. A lot sooner than I would have. LOL.

I too got a lot of work done around here,,,,,,well if you don't count running back to the computer every couple of minutes to see. LOL!!!!!!

Keep up the good work.

March 20, 2008, 12:20 AM
Derek, for what it's worth, if this helps, the Stickertramp thing, when asked to view a product category (or at times, a product) was getting errors that referenced something tmp also.

Did you take my advice of threatening it? Even if it doesn't work, it sure feels better.

Derek Zeanah
March 20, 2008, 12:31 AM

yeah - the original problem was the default place to drop temporary files was too small and getting filled up. So I changed it so that the temporary place was infinite(ish), and that was enough to convince the database server that the database files weren't corrupt.

Of course, it started erroring out that it couldn't write its temp files then, so I swapped back to the original setup.

As of this weekend I'm gonna keep regular folks on this MySQL server, and set THR and APS up on one of their own. That way impact will be minimized in the future.

Or at least, that's the current plan...

March 20, 2008, 01:21 AM
Well, at least upload a picture of a sledgehammer... You'll be surprised at how it seems to improve things. If things get really hairy, tell it that your favorite movie is Office Space...

March 20, 2008, 01:31 AM
Thanks for all the hard work, Derek! :)

March 20, 2008, 01:31 AM

Seriously, thanks for all the work you put into THR. I've learned so much from this site it's amazing!!!

By the way, my wife asks that you perform this "defrag/server move/optimize" thing every evening around 6:00pm. She thinks I'll do the dishes and play with the kids.

Cavalier Knight
March 20, 2008, 01:39 AM
Thanks for your hard work but next time send a text message to my phone or I'll jump up & down & throw things....

March 20, 2008, 09:57 AM
I'd be willing to help with the problem, with my vast experience working on computers, ( a monkey and a football come to mind here) I could really be an asset............:D

March 20, 2008, 10:29 AM
Thank you,Derek,for all that you do,and keep up the great work!!! You're the THR Superhero:)!

March 20, 2008, 10:36 AM
Thanks for your work. I am glad someone knows enough about this type stuff to keep this wonderful site up and running.

March 20, 2008, 12:11 PM
Well I for one am OUTRAGED!!!!! you should have seen me go through the 5 stages of grief. the Acceptance part was hard, but finally it came.... and then the site was back up, and I was engulfed with anger over the pain this site put me through.
But I'm putting it behind me. It's back, and we're still in love. Come back to bed THR..... it's been a long day.

*Elton John's Tiny Dancer Plays i nthe Background*

Good lord.... I am bored right now.
Thanks for the work you did! Good to have people that care about the board running well. thanks a ton.!

March 20, 2008, 04:30 PM
And, in the very recent past, Superman also found time to photograph my wedding ceremony...

March 20, 2008, 04:50 PM
Thanks to all for all your hard work !!!

March 21, 2008, 05:24 AM
Meh.. I didn't notice a thing. came home clicked the new posts button and it was only 25 pages instead of the usual 27 or 28 no big deal; . I do this once a day here.

If you enjoyed reading about "Well, that was no fun" here in TheHighRoad.org archive, you'll LOVE our community. Come join TheHighRoad.org today for the full version!