I have a hetzner server (the server that hosts this blog, as it happens) which runs Debian. While waiting to give a presentation at the day job over zoom, I decided this was a good moment to upgrade the system from Debian Bullseye (now EOL) to Bookworm.

This is something I’ve done before, so I didn’t anticipate this to be a problem.

I was wrong.

Anyway, after performing the upgrade in the usual way, I rebooted and was greeted with… well… nothing.

The server failed to restart.

I logged in to hetzner robot and booted the rescue image, but I couldn’t find anything of note in the server logs. Indeed, the logs appeared to be completely untouched. To me, this pointed to a problem with the boot loader.

Thankfully, Hetzner supports a vKVM, so I booted into that. Again, nothing.

Hmm…

On playing around, however, I did notice that if you boot the KVM (which automatically starts the rescue image), and then trigger a soft reset from within the kvm itself, the kvm will remain attached during the boot process, and allowed me to see… a grub error.

Joy. But at least I had identified the fault.

The error in question was complaining about not being able to find normal.mod, which is fairly critical. I poked around the recovery console, mounting the various filesystems in the RAID, but couldn’t find the file. So I attempted to load the kernel manually using insmod… to be treated to another error complaining about linux.mod not being found.

So, grub was completely b0rked.

This, however, gave us the answer….

The fix

The problem is that, for whatever reason, grub (the bootloader) has got messed up. So, we need rebuild it. Since we can’t boot the system, we need to do this from the rescue system. Hetzner does have a installimage tool, but I felt a little wary about running that since my understanding was that this would wipe everything… a bit of a nuclear option.

Thankfully, there was a lower impact solution we could try first.

  1. Confirm your software raid is working by taking a look at /proc/mdstat, and listing the structure using lsblk. Your raid should already be assembled, but if it isn’t, you can run mdadm --assemble --scan
  2. Next, find your mount points, for me:
    • md0 = swap (ignore)
    • md1 = /boot
    • md2 = /
    • md3 = is your home directory, so leave this alone
  3. Now you’re ready to rebuild your filesystem in chroot.
    • Mount your root drive (md2) to /mnt/ mount /dev/md2 /mnt
    • Mount your boot drive (md1) inside – mount /dev/md1 /mnt/boot
    • Bind various system drives
      • mount --bind /dev /mnt/dev
      • mount --bind /proc /mnt/proc
      • mount --bind /sys /mnt/sys
    • Finally, create your chroot: chroot /mnt
  4. Now, rebuild and reinstall grub
    • grub-install /dev/sda (I took a guess this was where my boot loader is, usually the case)
    • update-grub
  5. Exit, unmount, and reboot
    • umount /mnt/dev
    • umount /mnt/proc
    • umount /mnt/sys
    • umount /mnt/boot  
    • umount /mnt
    • reboot

All being well, your server should be back up and running. For me, however, this wasn’t quite the end of the story.

After rebooting, my server was still inaccessible. I repeated the vKVM trick and fully expected to see a grub error, however the server was booting normally.

Using the root password, I logged in to the console and sure enough my server was running, however there was no network connectivity.

A bit of poking around shows that for some reason the network interface name had changed, and the server was hard coded in /etc/network/interfaces to use the incorrect one.

I used ip link show to find the correct network interface address, modified interfaces and restarted.

Boom, server back up… and now I can tell you about it here!

Hope this is of use to someone.

In Linux, it is possible to expand an existing filesystem over multiple disk drives. I recently had to do this for a few VPS as part of my day job, since they were running out of disk space and causing some instability with one of our core services.

Here’s how to do it, mainly for my own benefit, for when I inevitably have to do it again and have forgotten how…

  1. First, after you have connected your new drive (either physically or virtually), you need to create a new logical partition on that drive using cfdisk. You can use fdisk -l to find out the drive’s name (e.g. /dev/sdc)
  2. Create a new physical volume for this partition using pvcreate, e.g. pvcreate /dev/sdc5
  3. Extend the existing volume group to include this, e.g. vgextend VOLUMEGROUP /dev/sdc5, where VOLUMEGROUP is the name of the volume group. You can find out what the volume group is called by using the command vgdisplay.
  4. Extend the logical volume to include this extra space, e.g. lvextend -l +100%FREE /dev/VOLUMEGROUP/root
  5. Resize the filesystem. The command for this varies depending on what filesystem is currently installed, so for ext4 this would be resize2fs, and in my case for xfs this is xfs_growfs. E.g. xfs_growfs /dev/VOLUMEGROUP/root

After which, you should be able to type df -h and see increased disk space available to you.

There has been a quiet revolution taking place over the last few years. AI (Artificial Intelligence) has always been a goal in the IT world – a utopia in which we are watched over by machines of loving grace. Perhaps I’m late to the party, but in 2022 I feel it has really taken giant leaps forward in terms of practical application.

By now you’ve probably heard of DALL-E, OpenAI’s tool for generating “original” artworks based on natural language input (used to put together the cover art for this post, btw), and over the last few days I have been playing around with their Chat GPT tool.

Chat GPT – which is kind of an Eliza bot combined with a search engine (although this rather flippant description is in no way doing the tool justice) – allows for natural language conversations, to which the bot will answer your queries with “original” content… and it is, as Elon Musk puts it… “Scarily good”.

In the AI’s own words: I am an artificial intelligence trained to assist with a wide range of tasks, including answering questions and providing information on a variety of topics.

You can still tell it’s a machine (for now), but that’s not the point. It is not an attempt to pass a Turing test, it is a learning intelligence and automation tool, which is generationally more advanced and flexible than Siri or Alexa.

You’ll also note I put the word “original” in scare quotes – these models are trained on data that’s out there on the internet, so true creativity is currently beyond them. They can remix, and reuse, but not necessarily create, at least not yet… although you could argue that this all humans do anyway. As an aside, I wonder whether having access to a large body of software code for an AI to digest played any part in Microsoft’s decision to buy GitHub a few years ago?

While my fellow software engineers chew that over, take a look at this brief chat and consider how safe your job is from automation:

Pretty good already, and my point is that this is still in the early stages. So if you think your job isn’t at risk from AI… you’re probably wrong.

AI is coming for your job.

Yes, probably even yours.

Adapt

Change is a comin’, but personally I’m really excited.

The more I play with these tools, the more certain I am that there is major opportunity here for those who chose to embrace it. Or, to put it another way, like it our not you better find a way to make yourself hard to automate by creatively using these new tools, and to position yourself properly, because time only moves one way.

  1. Keep learning: Stay up to date with the latest developments in your field and consider getting additional training or education. This will make you more valuable to your employer and can help you stay ahead of the curve.
  2. Be flexible: Consider learning new skills that are in high demand and that may not be easily automated, such as leadership, communication, and problem-solving.
  3. Embrace change: AI and automation are transforming many industries, so be open to new ways of working and be willing to adapt to changes in your role or company.
  4. Create value: Focus on creating value for your employer and customers, rather than just completing tasks. This can help you stand out and become more valuable to your organization.
  5. Communicate: Talk to your employer about your concerns and try to understand their perspective on the use of AI in the workplace. This can help you find ways to work together and address any concerns you may have.

The above list was generated entirely by AI, when I asked it “How can I avoid losing my job to AI?”. Not bad at all, even if it does read a little like an ESL listicle you might find on Buzzfeed (bye bye Buzzfeed journalists, I guess). Point two is something I very much agree with – technically aware people skills and leadership (in my industry at least) is something that is going to be hard to automate away, at least until the Singularity (at which point everyone is screwed), and is something that is a high value add.

…but it does present a bit of a problem if you’re only now starting to cut your teeth as an entry level coder. Humans are still going to be necessary… just fewer of them. Reference the self service tills at your local supermarket.

I believe there are some major labour saving advantages for individuals willing to take advantage. I personally have already used chat GPT to produce an initial draft for an article or two, reply to a few awkward emails, write a profile for a dating app, and even come up with a rather tasty curry recipe for leftover chicken.

Could I have done all these myself? Yes. Did I want to? No, and having 80-90% of the job automated away meant I could get on with doing other things. I had to tweak a few things here or there, and maybe rephrase things so they sounded a bit more like me, but the AI almost always produced a workable draft. No more writer’s block.

Questioning Reality

This person does not exist

How do you know what is real?

The “Dead Internet Theory” is a hypothetical scenario in which the internet becomes inhabited solely by automated systems or bots, and no human users remain. This could potentially occur if all human users of the internet were to suddenly disappear or become unable to access the internet for some reason. In such a scenario, the internet would continue to function, with bots carrying out tasks and interacting with each other, but there would be no human users to use or appreciate the internet.

I am starting to think this may actually be happening before our eyes.

With more and more of our interactions conducted through screens, and with AI getting more and more sophisticated, we are fast approaching the point where unless you actually experience it yourself in meat space you can never be sure something actually happened.

Yes, even if you see a video of it on the news.

It goes so much further than selectively editing a video to fit a particular narrative, with AI powered deep fakes, you could generate entirely new videos of entirely fake events which look real (predicted decades ago by Running Man).

If it isn’t already happening, I think it is only a matter of time before you start seeing these technologies applied to media, news, and especially interactions on social media. I wonder how much of the Twitter hellscape of the past few years has been caused by nation state actor AIs battling to control narratives? I’m not even going to touch on how big data analysis demonstrates just how predictable the actions of large groups of people are, and how on aggregate free will is pretty much an illusion.

Zoom meeting culture really took off in the pandemic, but with AI and deep fakes becoming more convincing and widespread, I suspect that in the future high level meetings will have to be conducted in person… out of necessity.

As individuals we are all going to have to be far more sophisticated when interacting online, and maintain a healthy level of scepticism around absolutely everything we see or are told is happening. How we do this without tipping into a nihilistic state where absolutely nothing can be relied on as objective truth, I’m not sure.

I guess it will become even more important to turn off the screens, and cultivate those real in person human to human relationships.