Stardate 20030803.1917

(Ship's log): You know, sometimes I find myself wondering if there really are gremlins in the world, infecting equipment.

Twice in the last two days my server ended up getting tied into knots, suffering what some refer to as a half-crash. In the two years I've been running this server, it's only happened twice before. Having it happen twice in such close succession was a bit worrying. The first time, I just used the console to force a reboot, but didn't really try to investigate.

The second time, early this morning, network-address-translation and the TCP/IP stack continued to work, so I was able to access the internet from my workstation, but Apache was wedged, and other things too, so that I couldn't access the server management package with a browser and couldn't telnet in and use the shell. Even more troubling, it was ignoring the console buttons, and that means something very deep in the kernel was wedged.

I had to power-cycle it to make it come back. Fortunately, the power button doesn't rely on software, else I'd have actually had to pop the cable from the power brick. That was about 7:30 this morning, and it seems to have been wedged for a couple of hours (while I was asleep) though I can't tell exactly.

Given that it happened twice in such a short period of time, and given that I've been remiss in applying updates, it occurred to me that some kind soul out there who disagrees with my politics might be doing me and the world the favor of taking advantage of some vulnerability I hadn't yet patched to prevent other people from being corrupted by my writing. So today I decided to catch up.

That's an easy process but a slow one. The server is a Qube-3, and it has many virtues. One of the reasons in particular that I decided to buy it is that the process of doing updates is extremely convenient. It's a lot like using Windows Update, in fact. But a lot of the patches require reboots, which take a while given how slow the CPU is. And of course while the server is rebooting, I can't reach the web.

Solution? Watch an anime DVD on my workstation while waiting. So I popped in the first volume of "Love Hina", which others had given high marks. And noticed that my workstation had started acting really strange.

I have two accounts. There's an administrator account, of course, but in normal use I have another which is a "power user". I do that deliberately because I don't like running as administrator routinely. It defeats all the protections that are or should be in place to prevent me from mistakenly screwing up the system, and preventing inimical programs from trying to do so deliberately.

I tried starting my DVD viewing program, which is Intervideo WinDVD Platinum 4.5, which hung. And after killing it off, I wasn't able to open the "My Computer" icon any longer. Clicking it didn't do anything.

Logging out and in again didn't fix it. That usually unwedges any strange program which may have seized a resource and refused to let it go; the process of logging off summarily resets everything in the API. When I logged in as Administrator, "My Computer" worked normally. When I then "changed user" to the normal account it worked fine, but when I logging all the way back out and logged in with the normal account I still couldn't get it to work. (At which point I began to hear the Twilight-Zone music off in the background.)

I'm aware that there are some vulnerabilities in WinXP which permit others to lay siege and break through the castle walls by taking advantage of various buffer overflows, but that isn't possible in my system. The Qube has two ethernet ports, and the cable modem is connected to one of them. The second one connects to my LAN, and all my other equipment (laptop, workstation, a couple of old computers I never turn on, the ReplayTV) are on the LAN, and they all use IPs in the range of 192.168.X.X. The Qube is the only one actually exposed to the outside world; there isn't any way for an attacker to even address my workstation. And I'm also not a fool, and I don't open or run attachments sent to me by the system administrator to tell me that my mail account is about to be cancelled. (Or any of the other social-engineering come-ons used by worm writers to try to convince people to run programs they should not.)

Rebooting the computer didn't make any difference, which was deeply disturbing. And despite my normal prudence regarding running software from unknown sources, I began to wonder if maybe I'd still managed to pick up a virus or worm. But neither Ad-Aware nor McAfee VirusScan found anything.

Like all good programmers, I understand that sometimes you simply have to cold-boot. I powered off the workstation, power-cycled all the external USB hard-drives, and power-cycled all the USB2 hubs, then turned the computer back on again, and it was back to normal. Whew!

Which made me two-for-two on having to use cold boots today to recover from mysterious failures. And zero-for-two on actually knowing why they happened.

The only thing I can guess is that something in the workstation hardware, probably in my DVD drive, got into a weird state so that a privileged user was able to do what it wanted, but a non-administrator ended up hung in a driver somewhere. Whatever it was didn't get corrected by the system reset signal from the front panel button. It took a power-cycle to clear it. (Or to make it go into hiding before emerging again later.)

Or maybe there actually is a gremlin here, having fun at my expense by inhabiting my computers...