Tag Archives: debian

Posted on
by

I have a hetzner server (the server that hosts this blog, as it happens) which runs Debian. While waiting to give a presentation at the day job over zoom, I decided this was a good moment to upgrade the system from Debian Bullseye (now EOL) to Bookworm.

This is something I’ve done before, so I didn’t anticipate this to be a problem.

I was wrong.

Anyway, after performing the upgrade in the usual way, I rebooted and was greeted with… well… nothing.

The server failed to restart.

I logged in to hetzner robot and booted the rescue image, but I couldn’t find anything of note in the server logs. Indeed, the logs appeared to be completely untouched. To me, this pointed to a problem with the boot loader.

Thankfully, Hetzner supports a vKVM, so I booted into that. Again, nothing.

Hmm…

On playing around, however, I did notice that if you boot the KVM (which automatically starts the rescue image), and then trigger a soft reset from within the kvm itself, the kvm will remain attached during the boot process, and allowed me to see… a grub error.

Joy. But at least I had identified the fault.

The error in question was complaining about not being able to find normal.mod, which is fairly critical. I poked around the recovery console, mounting the various filesystems in the RAID, but couldn’t find the file. So I attempted to load the kernel manually using insmod… to be treated to another error complaining about linux.mod not being found.

So, grub was completely b0rked.

This, however, gave us the answer….

The fix

The problem is that, for whatever reason, grub (the bootloader) has got messed up. So, we need rebuild it. Since we can’t boot the system, we need to do this from the rescue system. Hetzner does have a installimage tool, but I felt a little wary about running that since my understanding was that this would wipe everything… a bit of a nuclear option.

Thankfully, there was a lower impact solution we could try first.

  1. Confirm your software raid is working by taking a look at /proc/mdstat, and listing the structure using lsblk. Your raid should already be assembled, but if it isn’t, you can run mdadm --assemble --scan
  2. Next, find your mount points, for me:
    • md0 = swap (ignore)
    • md1 = /boot
    • md2 = /
    • md3 = is your home directory, so leave this alone
  3. Now you’re ready to rebuild your filesystem in chroot.
    • Mount your root drive (md2) to /mnt/ – mount /dev/md2 /mnt
    • Mount your boot drive (md1) inside – mount /dev/md1 /mnt/boot
    • Bind various system drives
      • mount --bind /dev /mnt/dev
      • mount --bind /proc /mnt/proc
      • mount --bind /sys /mnt/sys
    • Finally, create your chroot: chroot /mnt
  4. Now, rebuild and reinstall grub
    • grub-install /dev/sda (I took a guess this was where my boot loader is, usually the case)
    • update-grub
  5. Exit, unmount, and reboot
    • umount /mnt/dev
    • umount /mnt/proc
    • umount /mnt/sys
    • umount /mnt/boot  
    • umount /mnt
    • reboot

All being well, your server should be back up and running. For me, however, this wasn’t quite the end of the story.

After rebooting, my server was still inaccessible. I repeated the vKVM trick and fully expected to see a grub error, however the server was booting normally.

Using the root password, I logged in to the console and sure enough my server was running, however there was no network connectivity.

A bit of poking around shows that for some reason the network interface name had changed, and the server was hard coded in /etc/network/interfaces to use the incorrect one.

I used ip link show to find the correct network interface address, modified interfaces and restarted.

Boom, server back up… and now I can tell you about it here!

Hope this is of use to someone.

Posted on
by

So, I’ve been using ownCloud for a while now as a convenient way to share certain files between all of my various devices. I’ve even learnt on How to Make a Minecraft Server, which has been pretty fun. The server is a PHP application, so it’s pretty easy to set up.

Anyway, I updated my server to use PHP 7.3, in order to run the latest Known code, among other things. PHP 7.3 is the latest stable code, and so is what everyone should be running, really.

This presented a bit of a problem, as ownCloud would only run on PHP version up to and including 7.2! The next version of ownCloud will apparently support PHP 7.3, however release schedules are slow, and I really needed to get my syncing up and running again.

The obvious solution would be to run PHP 7.2 for the ownCloud server, and then PHP 7.3 for everything else.

Installing PHP-FPM

If you’re running the old school mod_php apache module, the first thing you need to do is install PHP-FPM.

I had been meaning to do this anyway, as this is essentially the modern way of running PHP. It’s faster, gives you many more options for performance, and crucially for my purposes, decouples Apache from PHP meaning you have the option of running multiple versions.

On Debian based servers (mine is Debian, with a third party PHP 7.3 apt repository set up), this turns out to be incredibly easy:

apt-get install php7.3-fpm php7.2-fpm

You’ll also want to install all the PHP modules you need (pdo, gd, etc), but I’ll leave that as an exercise for the reader.

Next, you want to switch over your config:

a2dismod php7.3
a2enmod proxy_fcgi setenvif
a2enconf php7.3-fpm
systemctl restart apache2

Two things to note here. Firstly, replace the a2dismod statement with whatever php version you currently have installed. Secondly, you’ll notice I didn’t enable the PHP 7.2 FPM config. This is because I want PHP 7.3 to be the default for all sites, but to be able to selectively enable PHP 7.2 on selective virtual hosts.

Checking phpinfo() should now show you something like this:

Note the PHP version and the Server API.

If you look at your server processes, you’ll also see both PHP 7.3 and PHP 7.2 FPM servers running:

# ps aux | grep php
root      1437  0.0  0.2 601224 34420 ?        Ss   Sep14   0:03 php-fpm: master process (/etc/php/7.3/fpm/php-fpm.conf)
www-data  1936  0.2  0.7 685708 113000 ?       S    15:05   0:19 php-fpm: pool www
www-data  6650  0.2  0.7 688600 122332 ?       S    09:04   1:08 php-fpm: pool www
www-data  6657  0.2  0.7 687132 125016 ?       S    09:04   1:12 php-fpm: pool www
root      7658  0.0  0.2 591404 32916 ?        Ss   Sep14   0:03 php-fpm: master process (/etc/php/7.2/fpm/php-fpm.conf)
www-data 19281  0.1  0.3 673936 51792 ?        S    Sep14   2:16 php-fpm: pool www
www-data 19289  0.1  0.3 673836 49044 ?        S    Sep14   2:17 php-fpm: pool www
www-data 19290  0.1  0.3 673936 49760 ?        S    Sep14   2:18 php-fpm: pool www
root     21084  0.0  0.0 132340   924 pts/0    S+   17:01   0:00 grep php

Configuring ownCloud’s VirtualHost to use PHP 7.2

So, now you need to modify your ownCloud VirtualHost to use the PHP 7.2 fast CGI server.

Again, this is really really easy, and is pretty much a cut and paste from the php7.2-fpm.conf file you’ll find in your /etc/apache2/conf-available directory.

Place the following somewhere in your ownCloud virtual host definition:

<FilesMatch ".+\.ph(ar|p|tml)$">
SetHandler "proxy:unix:/run/php/php7.2-fpm.sock|fcgi://localhost"
</FilesMatch>

Now, when you run a phpinfo() on your ownCloud domain, you should see it running PHP 7.2!

Now I can get back to syncing my files, while running the latest PHP version for other domains.

This is a useful feature, and obviously can be used to get more than just slow to update software up and running. For a start, this technique will let me run a bleeding edge version of PHP like PHP 7.4 against, for example, my development version of Known, but keep my blog running the stable release.

Anyway, I thought this was cool. Hopefully you’ll find it useful too!

Posted on
by

I hit a number of gotchas when upgrading my home and business web server from jessie to stretch, here they are in no particular order. Hopefully will save you some hair pulling…

Broken MariaDB install

Debian now ships with MariaDB by default, but when I upgraded mariadb-server would not install, meaning their were loads of broken dependencies. Dpkg exited with an error status, but with no indication as to what the actual error actually was.

Fixes suggested elsewhere (purging and reinstalling, moving /var/lib/mysql away and reinstalling, etc) did not help.

Eventually, I was able to manually execute mysql_install_db, which actually gave me some output. For me, the problem seemed to have been caused by the slow query logging entries, which are either unsupported in MariaDB or are named something else (I’ve not had a chance to check).

I commented out the following lines as follows:

# log_slow_queries  = /var/log/mysql/mysql-slow.log
# long_query_time = 2
# log-queries-not-using-indexes

… and apt-get was able to install the package.

Isolated /tmp

The version of systemd shipping with Debian 9 includes some security enhancements, including PrivateTmp, which isolates the temporary directory from users.

So, if you use your tmp directory to store e.g. cache data when developing websites, you’re going to need to store this somewhere else, otherwise file_exists and other file functions will not be able to read or write to them.

PHP 7

Ooooooo… boy.

Biggest hitter by far for me was that Debian 9 now ships with PHP7. Usefully, 5.6 is still available, so you have to switch to 7 manually (which means installing all the appropriate module again). Gotcha here is the mysql extension has been entirely removed, good thing too… however, if you’ve been running your server for a while like I have, you’re going to have a metric shittonne of things that need to be upgraded in order to work. Biggest pain in the bum was my ownCloud 8 server (made harder by the fact you can’t cross major versions in an upgrade, and the releases for those versions were no longer available until I nudged someone on IRC, also, pro tip, do the upgrade on PHP 5).

For scripts that either don’t have newer versions, or legacy stuff you don’t have time right now to allocate significant dev resources to, there is a mysql->mysqli shim available. This seems to work quite well in most cases, although of course it should be fairly high priority for you to migrate to PDO or similar!

Elgg and PHP 7

If you’re building sites on the 1.x.x branch of Elgg, you’re either going to have to upgrade to Elgg 2.x to run on PHP7, or use the shim.

I only have development sites running on PHP 7 at the moment, all of my clients that use < Elgg 2 are running on older PHP releases for now, but the shim works well in development and until I can manage those upgrades. If you use the shim you may need to comment out the following lines in executeQuery() in engine/classes/Elgg/Database.php:

if (!is_resource($dblink)) {
    throw new \DatabaseException("Connection to database was lost.");
}

…since the resource returned by the shim is a different type than expected.

That’s all so far, hope this will save you some stress!