Mod_security is a plugin for the popular Apache web server that lets you block malicious traffic on your web applications based on rules you define. Essentially, it acts as a firewall for web apps, blocking suspicious traffic and malformed requests. It is highly configurable, and comes with a good set of default rules to get you started.

I had need to configure it for a particularly security sensitive client site recently, and while I was at it I thought it’d be cool to fire it up on my personal server. I installed the module, reloaded Apache and began watching the output of the audit log.

Holy Pingback spam Batman!

Pingback, is a way of notifying a site that you’ve written about them in your blog. The receiving blog then usually renders this out as a link in the comments section of the post, allowing visitors to read the expanded discussion.

Watching the mod_security audit log was an eye opener, because I found that my site was being hit by a metric shitload of bogus pingback requests (the order of a couple every minute), all from different sources. None of these messages had made it as far as appearing on my site of course, seemingly they have been blocked by something in WordPress itself (probably Akismet), so I honestly didn’t realise that this was a thing.

ModSecurity was flagging them up because the body of the message was malformed XML, so was kicking them out (legitimate wordpress to wordpress pings were being accepted, at least when I tested it), and closed the connection with a 400 response. In every case the request had similar signatures; it was chunk encoded, it was always linking to a real site (in most cases some poor schmo’s pwned wiki, which didn’t mention the post that was pinging), and the User-Agent was always “PHP/5.2.10”, so we’re clearly dealing with a script kiddy.

One thing I noticed was that, although I was getting pingback spam from multiple sources, each IP would retry every couple of minutes. This meant that my web server was having to spool up to handle each request, even if the spam did not make it through. Negligible in the grand scheme of things, but irritating nonetheless.

Since I am a firm believer in both a strength in depth approach to security, and I like quiet logs, I wrote a fail2ban script to catch these messages. As before, because I’m operating behind a reverse proxy I’m keying off the squid logs (until I can work out how to change mod_security’s log to spit out X-Forwarded-For anyway).

After a couple of hours, this is what my munin graph looked like:

Holy crap.