Fail2Ban is a simple, but powerful, open source intrusion detection and prevention system which can run on most POSIX compliant operating systems. It works by monitoring various system logs for signs of intrusion attempts (failed logins etc), and on finding them, executes a preconfigured action.

Typically, this action is to block further access attempts from the remote host, using local firewall rules.

Out of the box, Fail2Ban comes configured to monitor SSH for signs of intrusion. However, since it works by monitoring log files, Fail2Ban can be configured to monitor many other services. I figured it would be pretty cool if you could also use it to protect Elgg sites as well.

Elgg already has a per user account lockout on login, however it is not without its limitations. It is pretty basic, and while it protects against access to specific accounts, it does not protect against dictionary attacks against multiple or non-existent accounts. Using Fail2Ban, you can protect against multiple access attempts from the same IP address easily, and the cut them off at the network level, frustrating the attack.

Installing Fail2Ban

The first step to getting this all working is to install Fail2Ban.

This is covered in detail elsewhere, but on Debian/Ubuntu it was a simple matter of pulling it from the apt repo:

sudo apt-get install fail2ban

Out of the box Fail2Ban will block using IPTables, but if you use shorewall, as I do, you’ll need to modify the actions to use that.

Getting Elgg to log access

It is an omission (quite possibly on my part), but the default Elgg login action does not explicitly log login attempts and login errors. While it is quite probable that you could hack together some regexp to parse the apache error logs, these are often quite noisy, highly changeable, often stored in odd locations, and, more often than not, are turned off in production environments.

I thought I’d make things a little easier on myself, and so I wrote a tiny Elgg plugin which overrides the default login action and outputs explicit error messages to the system auth.log, on both success and failure.

Once installed, you should begin to see logging messages start to appear in your server’s auth log (usually /var/log/auth.log) along the lines of this:

Mar 22 18:24:43 web elgg(web.example.com)[16483]: Authentication failure for fakeuser from 111.222.333.444
Mar 22 18:25:05 web elgg(web.example.com)[16483]: Accepted password for admin from 111.222.333.444

Again, to keep things simple, and to avoid getting a regular expression headache, I kept the authentication messages similar to those used by the SSH filter.

Monitoring the log with Fail2Ban

Finally, you need to configure fail2ban to look out for the Elgg messages in the auth.log.

  • Copy the elgg.conf into your fail2ban filters directory, on Debian this is in /etc/fail2ban/filters.d/
  • Create a jail.local in /etc/fail2ban/ if you have not already done so, and then create a rule, along the lines of the following:

    [elgg]
    enabled = true
    filter = elgg
    logpath = /var/log/auth.log
    port = all

Restart Fail2Ban, and you should be up and running! To test, attempt to log in (using a machine on a different machine if at all possible) and try a few failed logins.

A future enhancement of this that you could consider, especially if running in a production environment, is to modify the block action to redirect queries from the offender’s IP to a place-holder page explaining why they have been banned. This could probably be done quite easily using a REDIRECT rule, although I’ve not tried it yet.

Anyway, code, as always, is on github. Have a play!

» Visit the project on Github…

Helping out a friend and colleague, as well as stretching my programming muscles with a language I don’t often get to play with these days, I’d like to introduce LoveNote Server.

LoveNote is a simple abstract message queue server which lets you pass an message payload to one of a pool of endpoints and receive a webhook callback with the result. You can specify that this message be delivered ASAP, but crucially you can also specify a date and time for the delivery.

How it works

It works by POSTing a bit of JSON to a webhook provided by the server that contains a delivery time, an array of servers to try, the payload and an optional callback URL.

When received by the server, the message is queued. When the delivery time is reached the list of servers is randomised and the payload POSTed in sequence until all servers fail, or delivery is achieved. If a callback is specified, a report is then POSTed back to the callback as a JSON blob.

Why this is useful

Simply, this provides a common message passing framework with a unified event driven API, simplifying your architecture somewhat. It is especially handy if you wanted a message to be delivered some time in the future, for example for a credit card renewal or email reminder, where beforehand you’d probably have to write a dedicated server process.

All you now have to do is listen to webhook pings.

What still needs to be done

This is an early version and was written to help out a friend with a specific need, and although I’ve gone on to use it in a couple of client projects, there are still a fair amount of enhancements that could be made.

Some obvious ones are:

  • Message IDs: Currently messages in the queue are anonymous. It’d be handy to have message IDs since this would allow more sophisticated process control of scheduled messages.
  • Cancel Control: Basic message control to cancel future queued messages.
  • Make message queues persistent: Currently the queue is held in memory, which is simple and fast, but far from ideal. We should periodically flush the queue to a persistent storage so that no messages are lost if a server goes down.

Get involved and send your pull requests to the usual place!

» Visit the project on Github…

StatsD, created by Etsy, is a simple Node.JS server that provides powerful way to collect numerous statistics about your web application, and to do so simply and quickly.

At Etsy, they graph everything, which they use to great effect.

Collecting statistics is cool because it gives you hard data about how your software is performing. This in turn gives you powerful analytical tools to dig deep into your code, and find out what’s really going on (especially true when combined with a visualisation tool such as Graphite). It let you see the impact of code or infrastructure changes, and to quickly identify problems. Perhaps most importantly, it is only when armed kind of hard data that you can even begin to grasp at the nettles of code optimisation, growth and scalability.

Introducing StatsD for Elgg

With this in mind, and because I needed to collect some stats for a couple of client projects, I’ve put up on Github the first version of an Elgg statsD module. When installed and configured, this module will interface with a statsd server and collect a whole bunch of statistics from your running Elgg site.

Out of the box the current version of the plugin can log:

  • Events & Hooks (which in turn give you things like user signup events and object creation)
  • Exceptions
  • PHP Errors, Notices and Warnings
  • Elgg popups (system messages and error messages)
  • Database calls
  • Script execution time

In addition you can record your own statistics by making a call to wrapper functions contained in the plugin itself.

All data will be logged into a custom “bucket”, which is by default derived from your Elgg site name. This lets you log statistics from multiple different sites to a common statsd server.

Installation is pretty straightforward once you’ve installed the base infrastructure. Follow the instructions for installing graphite, nodeJS and statsd from the various sites around the internet, and then upload the elgg-statsd plugin to your Elgg site’s mod directory.

Once activated, you can specify the statsd server you wish to log to and configure what statistics you want to record.

Have fun!

» Visit the project on Github…

P.S. If you try and set this up and are seeing errors in your graphite log along the lines of “create() takes at most 5 arguments (6 given)“, then you are likely falling foul of this bug.

My solution was to build Whisper from the latest code in master rather than the stable 0.9.x branch. This worked for me, but of course YMMV.