Building Docker images on EC2 instances

Building Docker images on EC2 instances rather than on a local machine makes a lot of sense.

You get the benefits of a consistent build environment, as well as a typically much-faster network connection than you often enjoy locally.

However, there are some gotchas, and we look at these below.

Determining the Docker path

If you've been using Docker for Mac, then you've probably been using Docker at:

/usr/local/bin/docker

However, when moving to a Linux box, Docker instead is to be found at:

/usr/bin/docker

Setting Docker storage on EC2

Another particular problem that I've run across is ensuring that there's enough space for building large Docker images.

We use an ansible role to start an EC2 instance, and then push, pull and build Docker images using this EC2 instance.

Using our ansible role, we spin up an m3.large instance, particularly because we want the 32GB SSD that it comes with.

However, when we install Docker on the instance, the default setting for storage is in /var/lib/docker, whereas our 32GB SDD storage is mounted at /mnt.

On an m3.large the /dev/xvda1 mount / has a size of 7.8GB, which is not enough for some of the images we'll be building: ideally, Docker images should be kept as small as possible, however, we need to build some rather large monolithic applications, so we really need the space at /mnt.

This is not a problem when running a container from an image, as we can simply map the volume:

-v /mnt/docker/data:/root/monolithic-app-storage \

However, when building an image, although we can specify the volume in the Dockerfile, we can't map it to a location on disk, and therefore are reliant on there being enough disk space where the Docker images are stored.

Theoretically, we can simply change the $DOCKER_OPTS and restart docker:

mkdir -p /mnt/docker/data
export DOCKER_OPTS="-g /mnt/docker"
service docker restart

If this worked, we should see our $DOCKER_OPTS when we do the following:

ps -aux | grep docker

However, we don't, and this has been a problem that others have run across, such as here and here.

One answer (though possibly not the best), is to edit /lib/systemd/system/docker.service and add our desired storage location to the ExecStart variable:

ExecStart=/usr/bin/docker daemon -H fd:// $DOCKER_OPTS --graph /mnt/docker

When we look, we then see our mount point used for Docker:

ps -aux | grep docker

root     16939 11.0  0.5 540896 42388 ?        Ssl  04:18   2:40 /usr/bin/docker daemon -H fd:// --graph /mnt/docker

Note that if you've already pulled or built images, you'll either need to re-do this, or save and then load them.