So, unless you have been living under a rock for the last few days, you would have heard about Heartbleed, the recently discovered OpenSSL security bug (go over to XKCD for the best explanation I’ve seen for how it works).

It was bad, really bad. As Bruce Schneier put it:

“Catastrophic” is the right word. On the scale of 1 to 10, this is an 11.

…And it was, very bad, but it could have been much much worse.

As it was, within a day or two of the bug being declared, the vast majority of affected servers were patched, and service users notified. We don’t know how often, if at all, this bug has been exploited, and we likely never will, but we can easily tell if it’s been fixed.

Bugs happen, but the only reason we know about Heartbleed at all is because OpenSSL is Open Source software. Open source allows you, and other third parties, to independently review and audit the code, something not possible with software from proprietary vendors.

Does software from Apple, Google or Microsoft contain similar ticking timebombs? Who knows, and we have no way of finding out. This is one of the many reasons why you should never trust closed source or proprietary security products. Ever.

Bluntly, anything where you can’t see the code can’t be audited, so can’t be trusted not to do something malicious, whether by accident or design. Trusting a product from a manufacturer purely on brand is a genetic fallacy.

What next

OpenSSL is an incredible project, and the community do a bang up job at keeping our communications safe from prying eyes. The fact that this bug, a simple buffer overflow error (a mistake every C programmer has made countless times in their career), was not spotted for almost two years is a little troubling, but those of us outside the project can’t really comment usefully on that.

Bugs happen, sometimes they’re serious, but the responsible action was followed and internet security is stronger for it. Patch and move on.

Talking specifically about OpenSSL, and open source in general, my thinking is that Heartbleed highlights a resource problem common with many projects which fall under the umbrella of being “infrastructure”. In that, it is largely built by volunteers (although also with large contributions from engineers paid to work on the project by their day job), and so sometimes there’s not enough bandwidth available to do everything, but by being “infrastructure”, doesn’t get so much of the attention as the latest wizzbang project on Hacker news.

My hope is that this bug will see more stakeholders taking an active interest, and so see a growth in the numbers of contributors and auditors. If every company who relied on OpenSSL paid one of their engineers to spend one day a month trying to break it, or audit the latest code, how many more person-hours this would give the project?

Another issue Heartbleed illustrates painfully well, is the danger of homogeneity. When the vast majority of the world use the same bit of software, it presents a single point of failure, and one error can have a massive impact.

OpenSSL has become the de-facto standard SSL/TLS implementation in use through merit, and while there is a very good case for not rolling your own crypto, I wonder if there is a case for encouraging and maturing different open source implementations of the protocol? Pros and cons on either side, but single points of failure should always be avoided…

Image “How the Heartbleed bug works” by XKCD

CVE-2014-0160, better known as the Heartbleed bug, it a critical, easily exploited, and widespread bug in the OpenSSL library that powers many HTTPS implementations around the world.

It is a memory leak exploit that can be used to potentially expose server keys, and any amount of other private information, so, it’s hard to stress exactly how important it is that you patch your servers now!

Because it may help others, here’s what I did:

Diagnosis

First step is to confirm that your servers are vulnerable. The vulnerability exists in OpenSSL libraries newer than 0.9.8 and persists until fixed in 1.0.1g. This means it is present in both Debian stable and Ubuntu systems, among others.

You can confirm whether your systems are vulnerable with this handy python program (mirror on gist), written by Jared Stafford, which attempts to exploit this bug.

Fixing

  • Debian and Ubuntu users should apt-get update; apt-get upgrade as soon as possible, since the security team has already patched the vulnerability. Other distros are likely to do the same, and I wouldn’t be surprised if patches were already released.
  • If your distro hasn’t patched yet, or you’ve compiled your own code, you should update to 1.0.1g now, or recompile your current OpenSSL binary with the -DOPENSSL_NO_HEARTBEATS. option set.

Thankfully, I run Debian, so my fix was easy. However, it is important to highlight that after updating the library you must RESTART any services that make use of it, notably Apache in my case.

This seems obvious, but until you do this your server is still vulnerable, so it’s worth underlining, and it caught me out!

Verification

Finally, it is important to verify your fix (how I spotted the restart requirement!). Use the same python tool above, and you should see something like:

marcus@dushka:~$ python ssltest.py marcus-povey.co.uk
Connecting...
Sending Client Hello...
Waiting for Server Hello...
 ... received message: type = 22, ver = 0302, length = 58
 ... received message: type = 22, ver = 0302, length = 3527
 ... received message: type = 22, ver = 0302, length = 781
 ... received message: type = 22, ver = 0302, length = 4
Sending heartbeat request...
Unexpected EOF receiving record header - server closed connection
No heartbeat response received, server likely not vulnerable

Happy patching!

Update: seems that lots of people must be after that python script, as we seem to have killed the poor guy’s server. I’ve stuck a copy of it on Gist, hopefully Jared won’t mind!

GCHQoogle: so much for "Don't be evil"Given what we now know about the mass surveillance, and attack on the infrastructure of the internet, conducted by Britain’s GCHQ and America’s NSA (as well as their Chinese, Russian, German, etc counterparts).

Given that we now know, for a fact, that almost every byte of non-encrypted traffic is recorded and analysed, shouldn’t we now make a concerted effort to finally deprecate vanilla HTTP in favour of HTTP over TLS (HTTPS)?

When you use HTTP, it is a trivial matter for an attacker to see the content of the pages you visit, when, and how often you visit them. When using HTTP, there is also no guarantee that the content of the page hasn’t been modified without your knowledge, exposing you to all kinds of attacks.

Encryption, by and large, removes these problems, as well as massively increasing the cost of mass surveillance. Is it not time for all of us, as well as standards organisation like the IETF, push to make HTTPS the default? Even during my time I’ve seen insecure protocols like telnet and FTP go from widespread use to being almost completely replaced by secure alternatives (ssh and scp), so could we not do the same with HTTP?

Certificate authorities

Ok, there is one big difference between HTTPS and ssh (ok, many many, but one I care about here), and that is that HTTPS relies on certificate authorities. These are necessary in order to distribute trust, so that browsers can know to automatically accept a certificate and verify the server it is connecting to is who it says it is.

This is much nicer for the average user than, say, manually verifying the server’s fingerprint (as you have to do with SSH), but comes with some pretty serious problems if we were to make it default:

  • Every site owner would have to get a certificate, and these can only be obtained by a certificate authority if you don’t want browsers to pop up a big red warning, meaning we further bake these guys in to the Internet’s DNA.
  • Certificate authorities can be directly pressured by governments, so, a government attacker could MITM you on a secure connection and present you with a certificate that your browser accepts as valid, and so will give you no warning (of course, this is much more costly than the blanked mass surveillance that is currently going on).
  • Getting a certificate either costs money, and/or has restrictions placed on their use (for example, no commercial use, in the case of StartCom). This is really bad, since it essentially requires permission from a third party to launch a site.

It is this last causes me most concern, since it essentially provides an easy way of suppressing minority views.

Imagine that we lived in a world where HTTP had been deprecated, and browsers no longer supported unencrypted HTTP, or could, but you had to request it specifically (essentially the reverse of what we currently have). You wanted to launch a site that expressed a minority view – perhaps you were critical of your government, or you wanted to leak some information about crimes being committed, is it not inconceivable that you could have trouble obtaining a certificate? Given that certificate authorities are companies who worry about their bottom line, and are a convenient point for the bad guys to apply pressure?

If you couldn’t get a certificate in this environment, it could dramatically reduce the audience that would see your site.

So, perhaps before we move to deprecate HTTP, we must first find a better way than certificate authorities to distribute trust? How could we accomplish this? Perhaps we could take advantage of the fact that most people’s browsers automatically update, and so we could distribute browsers with expected certificates for sites hard coded into them (giving an added advantage that we could pin certificates)?

Anyway, its complicated, and I’m thinking aloud here… what are your thoughts?