Writing this here, since it caused some issues at the Day Job, and took a little bit of work to debug. Hopefully this will save some of you some time, and will jog my memory should something like this happen again.

Anyway, last week, one of my team wanted to deploy a new release of our Access Management System (ARIA), which involved deploying a bunch of containers. The release procedure worked fine, however after the deploy, the main web app container was entirely unable to talk to our API layer.

My team did a little bit of debug, but it was at this point that it got escalated to me. I rolled the live environment back, and began debugging the problem.

On the face of it, it seemed that networking, or at least name resolution, was no longer working from within the container. A curl call from the command line produced:

curl: (6) getaddrinfo() thread failed to start

However, a connection to an IP address would work. So, I began looking at networking / name resolution. The next step was to see what the name servers were doing… however, nslookup gave me:

isc_thread_create(): fatal error: pthread_create(): Operation not permitted

Interesting… so something was blocking creating new threads within the container. Likely the security model that docker was running… not sure why this would change, but I confirmed this by redeploying with SECCOMP turned off:

security_opts:
      - "apparmor:unconfined"
      - "seccomp:unconfined"

Confirmed, networking was working.

Not sure what’s changed, but it would appear that somewhere down the line the base Apache Linux image has updated, and is now using a different system call for starting threads. Likely a new version of GLIBC has been rolled into the container somewhere.

The final fix was to update the various containers to make sure they were all running on the newer base image, and then to redeploy docker on our estate so that it was running the latest version.

Boom.

Everything back to normal, hope this saves you some time!

I was recently trying to develop some Known schema changes, and had the need to start kicking about Known in a Postgres environment.

Known supports Postgres out of the box, but I admit it’s a less trodden path. I don’t routinely use Postgres (I tend to favour mysql/mariadb, purely because that’s what I’m used to), and it’s usually a pain to switch environments.

I previously built a docker development environment, so I took the obvious (and as it turned out trivial) next step to build a Postgres version.

Installation

  • Download and install docker
  • Add this docker image to known using composer composer require mapkyca/mapkyca-known-docker-postgres --dev

This will create a docker environment in /vendor/mapkyca/mapkyca-known-docker-postgres/

Usage

  • cd /vendor/mapkyca/mapkyca-known-docker-postgres/
  • docker-compose up
  • Point your browser at localhost:8089 and install in the usual way

» Visit the project on Github...

Even though we’re all under house arrest, work never stops!

However, because of all this excitement, I’ve had to switch about on machines and development environments a lot recently.

Previously, I’ve used my Known vagrant build to do development, but that started to prove a little bit heavy on a number of setups I’ve been using. So, since I’ve been playing with docker more recently, I thought it’d be nice to have a development container folks could use to quickly get up and running.

I’ve created a docker image that you can use in order to set up a quick development environment for your Known, and is installable via composer.

Installation

  • Download and install docker
  • Add this docker image to known using composer composer require mapkyca/mapkyca-known-docker --dev

This will create a docker environment in /vendor/mapkyca/mapkyca-known-docker/

Usage

  • cd /vendor/mapkyca/mapkyca-known-docker/
  • docker-compose up
  • Point your browser at localhost:8088 and install in the usual way

Data storage

Files you upload will be stored in your Known install’s “Uploads” directory.

Your database will be stored in /vendor/mapkyca/mapkyca-known-docker/db/run/.

IMPORTANT

This is designed to be used for temporary development, not production. I make no promises as to what happens to the database directory when doing composer updates.

If you care, back this up regularly!

» Visit the project on Github...