2006/10/09 Well, after two months of stable production, things started to act up again on pillbox. The MySQL seems unstable again and we're having trouble pinning down the problem. We're trying to stabilise the situation.
This event has also triggered some other efforts again. Pillbox, 3v8.net and Joint Concepts have been static for a while now, waiting for me to decide te become serious, or leave them as my "playground". The judges are still out, but at least we're calling the jurors to court.
Pillbox will be replaced with a new machine soon, and though I don't know for sure if the problems we've been having are hardware-caused, it is also an opportunity to create a new software and OS "baseline" for the services. The current distribution has been accumulating patches for almost two years and it's time for a cleanup.
I'll try to put updates on the site a bit more regularly, also on decision made regarding software, hardware, services, partners, hosting providers, etc. In the mean time:
If you find anything that's still broken, or missing, please mail again.
2006/07/12 Well, after some searching it seems that the "default" kernel set to boot was broken and caused the system to become unstable after an unknown period of time. We've seen two weeks, two days and as little as a few hours.
For the time being we rolled back to the older kernel, which at least performed well for up to 250 days. We also tried to implement a safe-kernel-upgrade method and quick-tested it on-site, but the reliability of this method is uncertain.
If, for some reason, the old kernel becomes unstable as well, we'll not be taking the chances anymore and start hardware replacement.
If you find anything that's still broken, or missing, please mail again.
2006/06/27 During the weekend we didn't want to intervene with a shutdown, as this would probably hang on the blocked filesystems. Monday morning, with a little help from WideXS, we got the system to reboot.
First boot was agonizingly slow, but apparently a lot of recovery was going on, as after a while the system seemed to re-establish all services successfully. We performed some more checks and maintenance on low-level system libraries, then rebooted again for a full recovery/service establishment.
If you find anything that's still broken, or missing, please mail again.
2006/06/24 Tonight something went seriously wrong,
apparently with the /home filesystem. A number of processes started
spawning uncontrollably and none of them could get the privileges they
wanted from the filesystem. Unless I can isolate what's blocking the
system, I might need to reboot, which is a risky operation with a
hanging filesystem. Since this, of course, happens during a weekend,
I've chosen a reboot only as a last resort for monday. Websites and
applications running from /projects seem unaffected. The mailserver has
been stopped, so the fallback queue will activate on my server at home.
For the admins, only use direct-to-root sessions, as normal login will stat /home.
2006/10/09 Well, after two months of stable production, things started to act up again on pillbox. The MySQL seems unstable again and we're having trouble pinning down the problem. We're trying to stabilise the situation.