Posted on
by

Like many people out there who have servers, I use cross site backups for the purposes of disaster recovery.

While I also use “proper” backup software as well (which maintains a historic delta of changes, among other things), I’m a big fan of redundancy, so like many other server administrators out there, I also perform regular site to site copies using Rsync over SSH.

Rsync, for those who aren’t familiar, is a file copy tool, which, after the first copy, will only send changes during subsequent updates. This makes it a very efficient tool, especially when used over an internet connection. SSH stands for Secure Shell, and is an encrypted connection between two computers.

Anyway, to enable rsync from server A to server B, it is common to perform the login via key. This means that on Server A you’d generate a SSH keypair for your backup user, then copy the public key that was generated into the ~/.ssh/authorized_keys file for your backup user on Server B. There’s a good guide to doing that here.

Because rsync is going to be executed automatically via cron script, it is necessary to create the key file without a password.

Backups are running, so what’s the big?

Ok, so the key for your backup user isn’t password protected, but that doesn’t mean that anybody can log in as that user, they’d still need to have a copy of the cryptographic key.

If someone is able to break into Server A and steal your backup user’s key, they will be able to log in to Server B as that user. If that user is a non privileged user, that isn’t absolutely terrible (although they could, for example, read your /etc/passwd file, or anything else you’ve not locked down) and you can revoke this key at any time. However, since strength in depth and compartmentalisation are two concepts you should always remember when building a secure system, it’d be nice if it was possible to do something about it.

Thankfully there is.

Jail time

The solution is to configure Server B’s ssh client to place the backup user’s sessions into what’s called a “Jail” using a Linux tool called chroot. When the backup user logs in, they will not have access to any other parts of the system, other than what’s in their jail cell.

It’s a little fiddly to set up, but most of the hard work is done by SSH.

  • First, install chroot: “`apt-get install dchroot debootstrap“`
  • Configure your SSH server
    • Open up “`/etc/ssh/sshd_config“`
    • Make sure there’s a line that says 
      Subsystem sftp internal-sftp
    • At the end of the file, tell SSH to create a chroot jail for your backup user: 
      Match User backup-user
        ChrootDirectory /home/backup-user
        AllowTcpForwarding no

      Note, because of the way chroot works, you’ll need to make sure the chroot directory is owned by ROOT, even if it’s actually the home directory of your backup user.

    • Save, and restart your SSH server.

This gets you part of the way, you should now be able to SFTP into Server B using your backup user, and a normal SSH session will be refused. When connected, you will be restricted to the location set in ChrootDirectory.

Unfortunately, rsync needs more than this, and in order to copy files it’ll need access to the shell (I’m assuming bash), as well as the rsync application itself, together with whatever libraries are required.

Therefore, it becomes necessary to create a partial chroot image in the backup user’s chroot directory. You could do this the traditional way (e.g. by using something like debootstrap), which will create a mirror of your base operating system files in the chroot jail. However, this generally takes a few hundred megabytes at least, and if all you want is to copy some files, you don’t want to give access to more than you need.

Instead, I opted to create a skeleton chroot jail by hand.

The goal here is to mirror the filesystem of your server inside the chroot jail, so that if a file exists in /foo/bar, then you need to copy it to /home/backup-user/foo/bar, and make sure it’s owned by root.

  • Copy bash from “`/bin/bash“` to the directory “`/home/backup-user/bin/“`
  • Copy rsync (on my system this was in “`/usr/bin“`)
  • Next, you need to copy the symbolic link libraries to which these files are linked against. You can use the tool “`ldd“` to interrogate the executable and get a list of files to copy, e.g: 
    root@server-b:/home/backup-user# ldd /bin/bash 
    linux-vdso.so.1 =>  (0x00007fff52bff000)
    libtinfo.so.5 => /lib/x86_64-linux-gnu/libtinfo.so.5 (0x00007f412810a000)
    libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f4127f06000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f4127b79000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f4128340000)

    Copy the files which have directories into the appropriate locations, e.g. “`/lib/x86_64-linux-gnu/libtinfo.so.5“` should go into “`/home/backup-user/lib/x86_64-linux-gnu/“`

  • Do the same for “`/usr/bin/rsync“`

All being well, your rsync backups from Server A should now function as before, however the backup user will not be able to touch anything outside of the chroot jail.

Hope that’s useful to you!

Posted on
by

I manage a whole number of device and servers, which are monitored by various utilities, including Nagios. I also have clients who do the same, as well as using other tools that produce notifications – build systems etc.

Nagios is the thing that tells me when my web server is unavailable, or the database has fallen over, or, more often, when my internet connection dies. I have similar setups in various client networks that I maintain.

It logs to the system log, sends me emails, and in some urgent cases, sends a ping to my phone. All very handy, but isn’t very handy for other casual users who may just want to see if things are running properly. For those users, who are somewhat non-technical, it’s a bit much to ask them to read logs, and emails often get lost.

For one of my clients we had a need to be able to collect these status updates from different sources together, make it more persistent, and make it visible in a much more accessible way than log messages (which has a very poor signal to noise ratio) or email alerts (which only go to specific people).

“Known” issues

A solution I came up with was to create a Known site for the network which can be used to log these notifications in a user friendly, chronological and searchable form.

I created an account for my Nagios process, and then, using my Known command line tools, I extended the Nagios script to use my Known site as a notification mechanism.

In commands.cfg:

define command {
        command_name host-notify-by-known
        command_line echo "$HOSTNAME$: $HOSTSTATE$" | /etc/nagios/known_nagios_notify.sh
}
define command {
        command_name service-notify-by-known
        command_line echo "$HOSTNAME$ – $SERVICEDESC$ : $SERVICESTATE$. Additional info: '$SERVICEOUTPUT$'" | /etc/nagios/known_nagios_notify.sh
}

Then in conf.d/contacts.cfg I extended my “Root” contact:

define contact{
        contact_name                    root
        alias                           Root
        service_notification_period     24x7
        host_notification_period        24x7
        service_notification_options    w,u,c,r
        host_notification_options       d,r
        service_notification_commands   notify-service-by-email, service-notify-by-known
        host_notification_commands      notify-host-by-email, host-notify-by-known
        email                           root@localhost
        }

Finally, the script itself, which serves as a wrapper around the api tools and sets the appropriate path etc:

#!/bin/bash

PATH=/path/to/BashKnown:"${PATH}"

status.sh https://my.status.server nagios *YOURAPICODE* >/dev/null

exit 0

Consolidating rich logs

Of course, this is only just the beginning of what’s possible.

For my client, I’ve already modified their build system to post on successful builds, or build errors, with a link to the appropriate logs. This particular client was already using Known for internal communication, so this improvement was logical.

The rich content types that Known supports also raises the possibility of richer logging from a number of devices, here’s a few thoughts of some things I’ve got on my list to play with:

  • Post an image to the channel when motion is detected by a webcam pointed at the bird feeders (again, trivial to hook up – the software triggers a script when motion is detected, and all I have to do is take the resultant image and CURL it to the API)
  • Post an audio message when a voicemail is left (although that’d require me to actually set up asterisk, which has been on my list for a while now)
  • Attach debugging info & a core dump to automated test results

I might get to those at some point, but I guess my point is that APIs are cool.

Posted on
by

I needed some tools for talking to the Known API from the command line in order to play around with a few ideas I’ve been having.

So, I put together a few BASH shell scripts.

Installation

  • Install the prerequisites: curl php_cli python openssl base64
  • Check out the repository and add it to your system path.

Note, due to this bug, you’ll need to be running the latest version of Known if you want to use the syndication functionality.

Talking to Known

The first thing you’ll need (other than a Known account of course) is to get your API key, you can find this in your settings page under “Tools and Apps”.

You can then use those as parameters to known.sh. For example, to make a status update you’d type:

echo "body=my+data" | known.sh https://mysite.com/status/edit *username* *apikey*

Of course, you might want to use one of the wrapper scripts like status.sh, which also supports syndication e.g:

echo "my tweet" | status.sh https://mysite.com *username* *apikey* twitter::username

If successful, the scripts will output a JSON representation of what the API says.

Have fun!

» Visit the project on Github...