Debian breaks on Virtuozzo VPS

3 mins
, , , ,

Apologies if you are looking for any articles or trying to post a comment and you are getting 404 errors, or if you wondered where jbehave.org or behaviourdriven.org had gone. Over the last few days I’ve managed to completely hose my server running Debian testing distribution, and I’ve had to downgrade to Debian stable. This involved some rsync backups and a complete reinstall of the virtual server.

Apologies especially to the lovely people whose blogs I host and who have been extremely patient with me over the last few days.

Because stable is about a year out of date, it is several versions behind fast-moving projects like WordPress, so I have some more fiddling to do before permalinks or comments will be working again.

If you are running Debian/testing in a Virtuozzo VPS, check your kernel version. It seems Virtuozzo uses a broken RedHat 2.6.9 kernel which simply doesn’t work with current Debian/testing packages. If your VPS provider uses a broken kernel, DO NOT UPGRADE libc6 to the current version (2.7) otherwise your system will simply die on its feet. It’s incredible—/bin/ls stops working, so does ssh (so you can’t log in to fix it!)—in fact it seems a huge weakness in the Debian setup. I would have expected the core admin packages and system binaries to be statically linked to avoid exactly this problem. A statically-linked busybox helped me enormously here while I monkey-patched it enough to get it to boot.

For the record, here’s my experience so far:

Places to back up

  • /etc (all the system settings)
  • /home (also contains /home/vmail where all the virtual mailboxes live
  • /var/lib/mysql (lots of WordPress databases)
  • /usr/share/wordpress/wp-content (mostly for themes and plugins)
  • /var/www (wikis)

I’m quite pleased things were this well organised. I thought I might be looking for files all over the place.

Things that worked

  • openssh with my existing config and keys. Hurrah!
  • postfix and dovecot. Although that’s hardly surprising—they are both rock solid and quite stable. I just rsync’ed the mail directories and settings back and they both started working. After a recommendation by Steve Purcell I’ve been using dovecot as my imap server and authentication daemon with postfix and they are a joy.
  • apache2. Enough said.
  • mysql 5. I’m really starting to like mysql.
  • moinmoin. Needed some love. Actually I needed to clear out all the caches before it would work, and install packages moinmoin-common and python-moinmoin, and of course libapache2-mod-python.

Things that don’t work

  • Apache2 mod-fcgi. I was having problems with this before which went away when I upgraded to Debian/testing, so I’m assuming it just doesn’t work properly in Debian/stable.
  • Apache2 mod-svn. It just doesn’t exist in stable, and neither does mod_dav, so I can’t do funky internet filesystem things for my Windows friends.

Things I’m still struggling with

  • Links in WordPress. This is bound to be something really obvious that I’ll work out after a good night’s sleep. It seems ok for some blogs and not for others, so it might be to do with the themes. Update: it turns out I needed a /usr/share/wordpress/.htaccess owned by the apache user with the appropriate rewrite rules in it—thanks Ben Coleman for sorting that out.

So far my VPS hosting company, Solar VPS, have been great with any support requests I’ve made. I really hope they come through with an up-to-date kernel so I can dist-upgrade everything back to Debian/testing. Update: Virtuozzo has no plans to upgrade the kernel before its next major release, so I’m stuck with Debian Stable (Etch).