Ok, for those of you wondering what happened to this site over the last few days, here’s the scoop.
On Monday while logged into the server I started getting error messages when I attempted to access any CGI programs. Hmmm . . . so I checked the error logs and it was reporting segmentation faults. That’s not good. Next thing you know I couldn’t see the root partition on the drive. It was gone. Doing “ls -la /” said that the partition couldn’t be found. That’s when the panic mode starts to set in. I can’t reboot because . . . “reboot” is on the root partition and it can’t find the command.
I own this box and co-locate it with a buddy who has a mini-datcenter in his garage so I call and email him. Unfortunately, he’s away at a conference so he isn’t able to get back to me until late that evening. When he does get to it and reboots it, he says things don’t look good. The hard drive is likely fried beyond recovery. At this point I have two major options:
a) Wait for him to get back, go buy a new SCSI drive, and attempt to recover.
b) Sign up with a new hosting provider, rent a dedicated server, and attempt to recover.
I selected option “b.” I’ve been hosting with my buddy for years but this is the second or third time we’ve had to do this hard drive swap business. I’m just tired of having to deal with it. Option “b” puts the responsibility for the hardware on the hosting provider’s plate and not mine so . . . .
Now, I run my own DNS and this server was the primary on most of the domains I host. I had to get a DNS server back up and running as quickly as possible. My second task was to get email back up and running. In retrospect, I should have held off on email because as soon as that came back up I started getting tons of emails from people asking me what happened to the server. Of course, none of the people who actually pay me for hosting even noticed. All the people using services that I host for them for free were up in arms though. That forced me into a realization that no good deed goes unpunished. Thus, I’m probably not going to be doing any free hosting in the future and I’m scaling back any free services offered that I don’t also get some benefit out of.
Now, on to other issues:
SPAM
I’ve always used Qmail and SpamAssassin. I’m still doing so but I was able to shoehorn in SpamCop’s spam checking before the server even accepts a message and my spam traffic dropped by over 70%. We’ll see if the numbers hold but I was very encouraged by the preliminary results.
I’m also going to change the primary email address that I use. Instead of bill, I’m using brini. I had a “friend” who decided to help me promote my consulting business so he put my email address on his website and I went from almost no spam to being overwhelmed by spam. I used that address for three years without incident and it was all blown in one act. I’m trying to retire the old address slowly but within the next few months all mail to bill will go straight into my spam box.
Comment Spamming
The new server wasn’t up one day before I got hit by a massive influx of comment spam. I had been using CHAPRA based method (image file with a code people have to input) of authenticating users but I had some problems installing the necessary Perl libraries on the new server so I had left it open. Big mistake. I’ve since gone to moderated mode so I have to approve new comments but even then I’m still getting a ton of comment spam so I’m in the process of considering forcing people to have a TypeKey account to post. I’ve noticed Blogger sites doing this more and more and LiveJournal as well so needing an authenticated account seems to be in the future.
Backups
Fortunately I had done a backup on 4/1 so pretty much everything was saved. Unfortunately, one of the things that did get hosed even with the backups was the LA Poker mailing list. It only had 6 or 7 people on it so I’m not sure it’s a major loss though. I’m probably not going to put it back up as it falls under the category of things that I was offering for free that I don’t really get much benefit for and have no desire to continue supporting.
RSS Feed Aggregator
I’ve been lazy but I do have a rough spec put together. I’m working on the high-level use cases and as soon as I get things back in place with the server I can return to that task. The big question though has been whether or not this falls into the free service category mentioned earlier. Either way, I think I should complete the specs and then put it out there to the people who offered to help and see if we can’t come up with an answer. I may end up supporting it or we may decide to host it elsewhere and simply ask for donations to cover the hosting costs . . . who knows.
That’s it for the moment. I still have a ton of stuff to do to get back to being totally right but that’s a longer term issue.