Posted on
by

StatsD, created by Etsy, is a simple Node.JS server that provides powerful way to collect numerous statistics about your web application, and to do so simply and quickly.

At Etsy, they graph everything, which they use to great effect.

Collecting statistics is cool because it gives you hard data about how your software is performing. This in turn gives you powerful analytical tools to dig deep into your code, and find out what’s really going on (especially true when combined with a visualisation tool such as Graphite). It let you see the impact of code or infrastructure changes, and to quickly identify problems. Perhaps most importantly, it is only when armed kind of hard data that you can even begin to grasp at the nettles of code optimisation, growth and scalability.

Introducing StatsD for Elgg

With this in mind, and because I needed to collect some stats for a couple of client projects, I’ve put up on Github the first version of an Elgg statsD module. When installed and configured, this module will interface with a statsd server and collect a whole bunch of statistics from your running Elgg site.

Out of the box the current version of the plugin can log:

  • Events & Hooks (which in turn give you things like user signup events and object creation)
  • Exceptions
  • PHP Errors, Notices and Warnings
  • Elgg popups (system messages and error messages)
  • Database calls
  • Script execution time

In addition you can record your own statistics by making a call to wrapper functions contained in the plugin itself.

All data will be logged into a custom “bucket”, which is by default derived from your Elgg site name. This lets you log statistics from multiple different sites to a common statsd server.

Installation is pretty straightforward once you’ve installed the base infrastructure. Follow the instructions for installing graphite, nodeJS and statsd from the various sites around the internet, and then upload the elgg-statsd plugin to your Elgg site’s mod directory.

Once activated, you can specify the statsd server you wish to log to and configure what statistics you want to record.

Have fun!

» Visit the project on Github…

P.S. If you try and set this up and are seeing errors in your graphite log along the lines of “create() takes at most 5 arguments (6 given)“, then you are likely falling foul of this bug.

My solution was to build Whisper from the latest code in master rather than the stable 0.9.x branch. This worked for me, but of course YMMV.

Posted on
by

iCal is a simple format for sharing calendar, event and todo list information between applications. It is widely supported by many many applications.

I have recently been hacking on a couple of projects which required me to implement the format, and during this I hit on a couple of less well documented gotchas.

So, I’ve jotted them down here in order to hopefully save you some time…

  • Line endings: The specification says that line endings should be CRLF only, and some less tolerant clients puke if they’re not.
  • No blank lines: Ensure that your output contains no blank lines, again some clients with puke or interpret these as the end of the file.
  • End in .ics (the biggy): This is important if you dynamically generate an iCal feed for use with Google Calendar. You should set header type appropriately, e.g.

    header("Content-Type: text/calendar");
    header('Content-Disposition: attachment; filename="calendar.ics"');

    But critically, if all your entries are showing up only as “Busy”, you must also end the URL you pass to Google calendar as http://foo.com/bar/whatever.ics

You may find one of these validator tools useful too.

Hope that helps!

Image “Calendar Logo” by Quadmod

Posted on
by

As I’ve blogged before, IFTTT.com (short for “If This Then That”) is a popular service which lets you trigger actions based on certain events that occur around the internet.

One of the most requested features, by programmers at least, is to add WebHooks support. Webhooks are a very simple way of pinging API information about the internet between web services. It uses JSON to send an arbitrary payload over HTTP via a POST request to a specified endpoint. This is about as simple as you can get, which is of course the beauty of it.

Support for Webhooks is an obvious extension to IFTTT, and would allow people to build on the service, connecting together more than just the hand picked menu of channels on the IFTTT dashboard. Quite why this is so slow coming is a mystery; some have speculated that it was a business decision on their part, others that it is hard to build a slick interface for something of such a highly technical nature, others that they simply haven’t gotten around to it just yet.

Free Software to the Rescue!

Until IFTTT implement a WebHooks channel, you can use the following workaround.

Abhay Rana, in a project over on GitHub, has built a some code which provides a WebHooks bridge for IFTTT and other services. I have extend to add some extra functionality you might find useful.

Currently, the IFTTT wordpress channel uses that software’s RPC endpoint to make posts, so the software works by providing a little bit of middleware, that you install on an internet facing machine, which pretends to be an installation of WordPress. You then point the wordpress IFTTT channel at this installation, passing it some special parameters.

You specify the final endpoint URL as a tag, and by default the contents of the post, title, and categories get JSON encoded and relayed to this endpoint.

My extensions

Different Webhook endpoints need different fields, however, so my extension lets you provide service specific plugins. These plugins can manipulate the data sent to the upstream endpoint further, providing different fields and encoding options. This lets you support multiple webhook endpoints from a single installation, and without having to set up multiple IFTTT accounts.

To use, create the appropriate plugin in your installation’s plugins directory containing your extension of the Plugin class, then pass a special category “plugin:NameOfClass“.

I have included an example class and a JSON payload class. The latter assumes that the post body is valid JSON, validates it, and then sends it to the upstream endpoint. This lets you send specially crafted messages to upstream webhook endpoints.

My extension also includes much more debug logging, as well as some extra validation code and graceful failures.

Why?

IFTTT is a fantastically useful service, but unfortunately you are limited to using the services they choose to (or have time to) write connectors for. I find this limiting, and at times frustrating, since as well as excluding some great existing services, it places limits on my ability to hack my own stuff together!

Previously, I’d often used twitter to glue things together. However, since Twitter are currently making concerted efforts to turn their service into just another ad serving platform, rather than the communication platform and messaging service it was growing into, it was necessary to come up with an alternative method.

WebHooks seem a better solution anyway.

Thanks again to Abhay Rana for the original code, and I hope people find my additions useful!

» Visit the project on Github…