Pasting what I posted last night while it was up for a while and before it started acting sketchy again:
Sorry, and don't blame Huntsman, but I (along with about 6 various people at the server) did unwillingly spend the whole day (starting at 2 AM PST) fighting with the forum database. Some was accomplished but not much of what I wanted to get done at all. I expected to have the site down a couple hours in the middle of the night and repair the main 'post' table since I suspect it is part of the source of the slow queries lagging down the site. Unfortunately, neither I nor anyone available at the time knew that if you try a repair on a table bigger than your /tmp directory (2GB in our case) the repair will hang, forcing a restart of mySQL, and marking the table as 'crashed'. The temp files it creates are deleted upon doing this, so you won't find them there afterwards. Which means unless you know about this (it's buried in a sub-appendix of the mySQL docs
) mySQL will just sit there and spin looking like it's maybe kinda repairing something that should take quite a while to repair, while really it's doing nothing. And of course it doesn't tell you anything to the effect of 'temp space full, repair cannot proceed'. So at least, while waiting for this to happen (nothing was happening) I got a couple hours of sleep. Meanwhile I had them restore a backup snapshot from right before I started the repair. Yada yada, both ended up working but this copy is the original (more complete, shouldn't be missing any posts from last night) and we did finally get the repair to go through, so whatever was wonky before is hopefully fixed. I will still need to take it down again sometime to re-optimize the table but I am going to prune off as much data as I can get away with first to try to get it down to a more reasonable size. If anything seems missing, let me know, we can merge stuff in from the snapshot from before everything went crazy last night. Again, sorry for the withdrawals, hopefully everyone is fine and didn't accomplish too much today. If everything looks okay, it is now time for me to go drink.
We ended up restoring another copy of the backup from before all the mess started and that's what we're running now. Now as soon as everything looks okay, it will be time for me to go sleep for 16 hours or so.