Automating Elastic Beanstalk

thumb image-post

Good practice and scalability

In this post we're going to demonstrate running Docker containers using Amazon's Elastic Beanstalk.

When I was figuring out to use Docker containers on Amazon's Elastic Beanstalk, I seemed to find plenty of articles detailing how to manually setup Elastic Beanstalk using the AWS dashboard, but I really wanted a solution that was automatable / repeatable.

Specifically, with this project we want to be able to:

  • Install and use Docker on Mac OSX (though other host platforms are fine)
  • Build a custom apache-php5 Docker image using a Dockerfile, fully provisioned and ready for our app
  • Build a php-app Docker image containing our custom app on top of this apache-php5 Docker image
  • Push our Docker images to
  • Run up our application locally from the Docker images using Elastic Beanstalk's eb local run
  • Run up our application remotely on AWS using Elastic Beanstalk's eb create / eb deploy
  • Do all of the above in an automated fashion, using a Makefile

We want to follow good practice, and this involves splitting our database and application into separate containers. This will then alow us to scale up application containers and database containers independently.

For the local install, we'll deploy both the application and database container on the same host, but when deploying for production, we'll use RDS to serve our database, and an EC2 instance dedicated to each application container.

Presuming that our application is PHP heavy but database light (say, because it only uses the database for login / personal profile storage), we can probably get away with a single
database container / RDS instance, while scaling up more EC2 instances of our application container when demand increases.

Our app will do nothing more arduous than displaying the information generated by phpinfo() (on the php-app container) and show that it can connect to a MySQL database (on the mysql container if local or the using RDS if deployed on AWS).

However, you should easily be able to build out a much more complicated PHP application after
reading through these notes.

Note that we build an apache-php5 Docker image first, and then another php-app image containing our app on top of this image. This allows us to re-use the existing apache-php5 Docker image whenever our app changes, and thus speeds up our builds.

All code for this project can be found at


We're doing this on OSX, but it should work on typical linux variants.
We're presuming that you've got installed:

We're also presuming that you're reasonably comfortable with the principles behind Docker, and have probably already had some exposure to AWS.


We'll use a local shell script, .env_local, not versiond in git, to store our custom / sensitive details and implement them as environment variables.
This should be the only place we have to regularly make changes, and is used by our Makefile when running build steps:


export EB_APP=local-elasticbeanstalk
export EB_SCALE_MIN=2

export DOCKER_MACHINE=default
export DOCKER_USER=mebooks
export DOCKER_PASSWORD=bailter

DOCKERED=eval "$(docker-machine env $DOCKER_MACHINE)"

Set execute permissions on this file:

chmod a+g .env_local

Note that running the .env_local loads the environment so we can use docker (once installed) from the local OSX terminal, and does the same as:

eval "$(docker-machine env default)"


We use docker-machine on OSX via the Docker ToolBox.

The easiest way to do this is:

brew cask install dockertoolbox

As we're using Max OSX, this means that we'll end up with a Virtualbox Linux VM that will be used to run Docker.

Start our docker-machine:

docker-machine start default

Setup docker so we can use it from our current shell

Port-forward in VirtualBox, so we can access port 80 transparently:

VBoxManage list vms
VBoxManage modifyvm "defaut" --natpf1 "guestnginx,tcp,,80,,80"

Once installed, we can see that docker is at version 10.1:

docker -v
$ Docker version 1.10.2, build c3959b1

If we apply our environment variables:

. ./.env_local

we should then see our environment values based on docker-machine config default plus any extra that we've added in .env_local:

env | grep DOCKER


and we should be able to access docker directly

docker info

Install awsebcli

We install the awsebcli using homebrew:

brew install awsebcli

However, there's a problem with the version compatibility check, meaning that awsebcli thinks that docker 1.10.2 is < docker 1.6, and we receive the following message:

"You must install Docker version 1.6.0 to continue. If you are using Mac OS X, ensure you have boot2docker version 1.6.0. Currently, "eb local" does not support Windows."

To rectify this, currently we must edit /usr/local/Cellar/aws-elasticbeanstalk/3.7.3/libexec/lib/python2.7/site-packages/ebcli/containers/ as follows:

def supported_docker_installed():
    Return whether proper Docker version is installed.
    :return: bool

        #return commands.version() >= SUPPORTED_DOCKER_V
        return True
    # OSError = Not installed
    # CommandError = docker versions less than 1.5 give exit code 1
    # with 'docker --version'.
    except (OSError, CommandError):
        return False

Install Composer

curl -sS | sudo php -- --install-dir=/usr/local/bin --filename=composer

Use composer to install the dependencies for our php-app

cd php-app
composer install

App environment variables

We need to make certain environment variables available to our PHP scripts, particularly
those to do with connecting to our MySQL container.

To do this, we create a php-app/.env file, with placeholders for the expected
environment variables:

# php-app/.env
# The variables below are replaced during container startup by

# If we're using a local mysql container, the MYSQL variables are populated

# If we;re using an AWS RDS instance, the RDS  variables are populated

We then use our script to read the environment variables during the initialisation of our php-app container, and replace the placeholders in php-app/.env with the environment variables values:

# Make a copy of our .env file, as we don't want to pollute the original
cp /var/www/html/.env /tmp/

# Update the app configuration to make the service environment
# variables available.
function setEnvironmentVariable() {
    if [ -z "$2" ]; then
        echo "Environment variable '$1' not set."

    # Check whether variable already exists
    if grep -q "\${$1}" /tmp/.env; then
        # Reset variable
        sed -i "s/\${$1}/$2/g" /tmp/.env

# Grep for variables that look like MySQL (for local deployments)
# or RDS (for remote deployments).
for _curVar in `env | grep 'MYSQL\|RDS' | awk -F = '{print $1}'`;do
    # awk has split them by the equals sign
    # Pass the name and value to our function
    setEnvironmentVariable ${_curVar} ${!_curVar}

# Now that /tmp/.env is populated, we can start/restart apache
# and let our PHP scripts access them.
service apache2 restart

Create our Makefile

We use a Makefile to make our builds slightly easier:

include .env_local
VERSION=`git describe --tags`

all: build-base prepare

base: build-base push-base

app: prepare-app build-app push-app

environment: create-environment

# Our base image tasks
    docker build -t $(BASE_IMAGE):$(VERSION) docker/base

    docker login --username=$(DOCKER_USER) --email=$(DOCKER_EMAIL) --password=$(DOCKER_PASSWORD)
    docker push $(BASE_IMAGE)

# Our app image tasks
    # Update with the current image version
    sed -i '' "s~${APP_IMAGE}\:[^\"]*~${APP_IMAGE}\:$(VERSION)~g"
    git archive --format tgz HEAD $(APP) > docker/app/$(APP).tgz

    docker build -t $(APP_IMAGE):$(VERSION) docker/app

    docker login --username=$(DOCKER_USER) --email=$(DOCKER_EMAIL) --password=$(DOCKER_PASSWORD)
    docker push $(APP_IMAGE)

# Our Elastic Beanstalk tasks
    eb create -v \
        --cfg $(EB_APP) \
        --scale $(EB_SCALE_MIN) \
        --cname $(EB_ENVIRONMENT) \

Build the docker images

Commit and tag our changes, e.g.:

git tag 2.3.0

The use of git describe --tags in the Makefile means that we'll be using the latest git tag to tag our Docker images.
Note that, if you've made commits since your last git tag, we'll end up using a tag value which is a combination of the tag and the last commit hash:

git describe --tags

Build our mebooks/apache-php5 base Docker image

# Create the `mebooks/apache-php5` Docker image
make base

# Check that the image was created
docker images

Edit docker/app/Dockerfile and ensure that our php-app is refering to the same version as that that we just built:

FROM mebooks/apache-php5:2.3.0

Our mebooks/apache-php5 Docker image should only need updating when we want to update the packages in the distribution, such as when there are security vulnerabilities.

Build our app docker image

Note that our app docker image wil be tagged with the current version of the repo and our file will be updated accordingly.

# Create our php-app image and push it to
make app

# Check that we updated the image version in our ``
cat | grep 'Name'
    "Name": "mebooks/php-app:2.4.0-8-gb0eef33",

As we've just tagged our file with the current version of the repo,
we need to commit changes, otherwise the eb create / eb deploy (which makes use of git archive) will use the previous version of the file.

Note that, as eb create / eb deploy uses git archive to create a zipfile of our application for deployment, this means that:

  • we can use .gitignore to specify files that shouldn't be under version control
  • we can use .gitattributes to specify files that should be under version control but should be deployed in the app bundle.

eb create / eb deploy can also make use of a .ebignore file.

Check that our containers function as expected

# Set our environment variable
VERSION=`git describe --tags` && echo $VERSION

# Start just the PHP container
docker run -tid -p 80:80 \
    --name=php-app \

# Start both containers linked
docker run -p 3306:3306 \
    -e MYSQL_USERNAME=root \
    -e MYSQL_ROOT_PASSWORD=password \
    -e MYSQL_DATABASE=my_db \
    -d \
    --name mysqlserver \

docker run -tid -p 80:80 \
    --name=php-app \
    --link mysqlserver:mysqldb \

We should now be able to see the PHP Info details at the address reported by docker-machine ip, e.g.:

docker-machine ip

If we've started the mysql container, we should also be able to see a simple example of connecting to our MySQL database at

Push our images to hub.docker

Although our Makefile handles the pushing of our containers to hub.docker, we'll cover it here as we need to know about AWS using the associated config file to gain access to pull our images from hub.docker.

Login to our docker account:

# Enter the username, password and email when prompted
docker login

# Alternatively, specify username, email and password
# If any of these parameters are not supplied, you'll be prompted for them
docker login --username=mebooks --password=WHATEVER

The login will create a config file at ~/.docker/config.json.

AWS currently uses an older format of the docker config for authentication to hub.docker,
so we need to change our current ~/.docker/config.json to the required format by removing the auths wrapper:

# ~/.docker/config.json
    "auths": {
        "": {
            "auth": "WHATEVER",
            "email": ""


# ~/.docker/.dockercfg.json
        "": {
            "auth": "WHATEVER",
            "email": ""

We now need to push this to a suitable s3 bucket so Elastic Beanstalk can use it to download our images from our private repository.

s3cmd put ~/.docker/.dockercfg.json s3://elasticbeanstalk-ap-southeast-2-<aws_account_id>

Pushing our images to hub.docker is as simple as:

docker push mebooks/apache-php5
docker push mebooks/php-app

Note that the make app task automatically does the docker push mebooks/php-app, while the make base task automatically does the docker push mebooks/apache-php5,

Once pushed, we should be able to see our images on hub.docker:

Create our ECR respositories

Alternatively, in this section we'll look at using Amazon's Elastic Container Registry (ECR) as a place to store our images instead of using hub.docker.

Authenticate Docker to an Amazon ECR registry with get-login:

aws ecr --profile oregon get-login

    docker login -u AWS -p BIGLONGNUMBERAPPEARSHERE -e none https://<aws_account_id>

Note that is the URL for our container registry.

Copy and paste the docker login command into a terminal to authenticate your Docker CLI to the registry. This command provides an authorization token that is valid for the specified registry for 12 hours.

Create our repositories:

aws ecr create-repository --profile oregon --repository-name mebooks/apache-php5

    registryId: <aws_account_id>
    repositoryArn: arn:aws:ecr:us-west-2:<aws_account_id>:repository/mebooks/apache-php5
    repositoryName: mebooks/apache-php5

aws ecr create-repository --profile oregon --repository-name mebooks/php-app

    registryId: <aws_account_id>
    repositoryArn: arn:aws:ecr:us-west-2:<aws_account_id>:repository/mebooks/php-app
    repositoryName: mebooks/php-app

Tag our images and push them

docker tag mebooks/apache-php5:latest <aws_account_id>

docker push <aws_account_id>

docker tag mebooks/php-app:2.4.0-8-gb0eef33 <aws_account_id>

docker push <aws_account_id>

Create our (multi-container version)

We can use a multi-container if we want to use eb local run to spin up both our php-app container and an associated mysql container.

    "AWSEBDockerrunVersion": 2,
    "containerDefinitions": [
            "name": "mysql",
            "image": "mysql:5.6",
            "essential": true,
            "portMappings": [
                    "hostPort": 3306,
                    "containerPort": 3306
            "environment": [
                    "name": "MYSQL_USERNAME",
                    "value": "root"
                    "name": "MYSQL_PASSWORD",
                    "value": "password"
                    "name": "MYSQL_DB_NAME",
                    "value": "my_db"
            "name": "php-app",
            "image": "mebooks/php-app",
            "essential": true,
            "memory": 128,
            "portMappings": [
                    "hostPort": 80,
                    "containerPort": 80
            "links": [

Note that we specify a (very) weak MYSQL_ROOT_PASSWORD in our — you'll want to change this and ideally not have it under version control.

Run our containers locally

Finally, use eb local to create our docker containers locally:

eb local run

In a second terminal:

eb local status
docker ps

Accessing the containers locally

Find the appropriate container id, and start a bash shell on it:

docker ps

$ 832af3ff45d8        mebooks/apache-php5:latest
$ fc6a9553583f        mysql:5.6

# Access our PHP container
docker exec -it 832af3ff45d8 bash

Alternatively, we can use eb to find the details, including the human-readable container names:

eb local status

$ Platform: 64bit Amazon Linux 2015.09 v2.0.8 running Multi-container Docker 1.9.1 (Generic)
$ Container name: elasticbeanstalk_mysql_1
$ Container ip:
$ Container running: True
$ Exposed host port(s): 3306
$ Full local URL(s):

$ Container name: elasticbeanstalk_phpapache_1
$ Container ip:
$ Container running: True
$ Exposed host port(s): 80
$ Full local URL(s):

Access our PHP container:

docker exec -it elasticbeanstalk_phpapache_1 bash

# check our apache config
apachectl configtest

# view our apache config
cat /etc/apache2/sites-enabled/vhost.conf

# Find the ip of our MySQL container
env | grep MYSQL_1_PORT_3306_TCP_ADDR


# Login to mysql
mysql -u root -h  -p

Access our MySQL container:

docker exec -it elasticbeanstalk_mysql_1 bash

# Display our databases -- we should see my_db
mysql -u root -p -e "show databases;"

Create our (single container version)

Although the multi-container version above is useful for testing locally, in production we'll be using RDS to host MySQL, so we'll use a single container to deploy our php-app container, and use an Amazon RDS instance for our database.

    "AWSEBDockerrunVersion": 1,
    "Image": {
        "Name": "mebooks/php-app:2.4.0-8-gb0eef33",
        "Update": "true"
    "Authentication": {
        "Bucket": "elasticbeanstalk-ap-southeast-2-<aws_account_id>",
        "Key": ".dockercfg.json"
    "Ports": [
            "ContainerPort": "80"
    "Logging": "/var/log/apache2"

Initialise our Elastic Beanstalk app

Now that we're happy with the running the containers up locally, we need to initialise our .elasticbeanstalk/config.yml so we can deploy our containers to AWS:

eb init

More info about the eb cli tool is to be found on the Amazon site.

We'll need to customise our configuration, so create a standard config as a starting point:

# This creates `.elasticbeanstalk/local-elasticbeanstalk.env.yml`
eb config

Ensure we have settings as we want, e.g. the MinSize for autoscaling:

    Availability Zones: Any
    Cooldown: '360'
    Custom Availability Zones: 'ap-southeast-2'
    MaxSize: '4'
    MinSize: '2'

Create our Elastic Beanstalk application environment


# Ensure `.env_local` specifies the correct `EB_ENVIRONMENT`:
export EB_ENVIRONMENT=dev-local-elasticbeanstalk

# Create our new environment on Elastic Beanstalk
make environment

make environment effectively runs something like the following, depending on your environment variable settings in .env_local:

eb create -v \
    --scale 2
    --cname dev-local-elasticbeanstalk \

We should then see output at our terminal like the following:

    INFO: Creating new application version using project code
    WARNING: You have uncommitted changes.
    INFO: Getting version label from git with git-describe
    Creating application version archive "app-8215-160402_175219".
    INFO: creating zip using git archive HEAD
    INFO: git archive output:
    INFO: Uploading archive to s3 location: local-elasticbeanstalk-php-demo/
    Uploading local-elasticbeanstalk-php-demo/ to S3. This may take a while.
    Upload Complete.
    INFO: Creating AppVersion app-8215-160402_175219
    INFO: Creating new environment
    Environment details for: dev-local-elasticbeanstalk
      Application name: local-elasticbeanstalk-php-demo
      Region: ap-southeast-2
      Deployed Version: app-8215-160402_175219
      Environment ID: e-myq9pzyjxg
      Platform: 64bit Amazon Linux 2015.09 v2.0.8 running Docker 1.9.1
      Tier: WebServer-Standard
      Updated: 2016-04-02 04:52:24.369000+00:00
    Printing Status:
    INFO: createEnvironment is starting.
    INFO: Using elasticbeanstalk-ap-southeast-2-<aws_account_id> as Amazon S3 storage bucket for environment data.
     -- Events -- (safe to Ctrl+C)

# CTRL-C when prompted, and follow progress:
eb status

If we want to reference an existing config, we can do this during creation:

eb create -v --cfg local-elasticbeanstalk

Note that eb create will use the settings from .gitattributes to export-ignore files in the zip file that it creates and uploads to s3.
As such, we have to be careful that the file is included in the root level of our zip file.

Creating application version archive "app-4bbe-160402_140739".
Uploading local-elasticbeanstalk-php-demo/ to S3. This may take a while.

If we wish, we can easily download this zip file and inspect the contents:

s3cmd get s3://elasticbeanstalk-ap-southeast-2-<aws_account_id>/local-elasticbeanstalk-php-demo/

Once created (and even during creation), we will see our envionment in our AWS dashboard:


We now need to create our RDS instance.
For this project we'll use:

  • T2 Small
  • Multi-AZ
  • Allocated storage: 5GB
  • DB Instance Identifier: mebooks-mysql-dbinstance
  • Master Username: root

Using Multi-AZ ensures that we mitigate the chance of failure by having our main RDS instance in one availability zone, and a RDS failover in another availability zone.
Ensure that the security group used for the RDS instance allows connections to and from port 3306, and that the EC2 instances will be also using this security group.

Importantly, we should ensure that our RDS instance is private, but set to use the same VPC as our EC2 instances.

Once setup, we should be able to connect from an EC2 instance in the same group as follows (depending on the actual assigned RDS endpoint):

mysql -h -u root -p

As per this answer, there's currently no easy way to automate the attachment of an RDS database existing outside of an Elastic Beanstalk environment to the environment during creation.
Amazon seem to assume that you'll be wanting to create an RDS instance inside the Elastic Beanstalk environment, which means that your database becomes less independent, and may disappear should anything go wrong with your Elastic Beanstalk environment.

Instead, if you want the RDS instance to exist outside of the environment you can simply provide the connection parameters as environment variables via the EB Console: Configuration -> Web Layer -> Software Configuration:

RDS_DB_NAME : my_db

These environment variables can then be accessed from inside your PHP app.

This does mean that our application will not be able to connect to our database on the initial deployment during the eb create, as we won't have yet had the chance to set these RDS
environment variables in the EB dashboard.

Note that with this method, as the RDS_PASSWORD will be in plain sight on the AWS dashboard, it's probably wise to change this regularly. When you make changes to your environment variables on the EB dashboard and click apply, Elastic Beanstalk will then re-provision your existing EC2 instances with the new values.

Accessing our application

To see our application in the browser, we can use:

# opens a browser tab with something like:
eb open

We should see our PHP Info page, and be able to see our database connection details at /mysql.php.

Once our Elastic Beanstalk cluster is running , we can ssh into our load balancer, and then telnet to our instances:

# once sshed into the load balancer, you'll be presented with a choice of the
# EC2 instances if there are more than one.
eb ssh
sudo -s
yum install telnet

Once we know the public DNS of our ec2 instance, we can also directly ssh in as ec2-user:

ssh -i ~/.ssh/aws-eb

# Once logged in, elevate to root to be able to acces docker:
sudo -s


Presuming that we're only updating our custom php-app code, successive deployments should be a matter of the following:

# create the image and push it to hub.docker
make app

As make app tagged our file with the current version of the repo, we need to commit the changes, otherwise the eb create / eb deploy (which uses git archive) will use the previously-committed version of the file.

git add
git commit -m "Updated Docker image tag in"

eb deploy -v local-elasticbeanstalk


Once we've finished with a particular environment, we can terminate it:

eb terminate local-elasticbeanstalk

Cleaning up

Once we're finished, we can remove our containers locally, either by id or by name

docker rm 832af3ff45d8 fc6a9553583f

If we're finished with our image, we can delete it:

docker rmi mebooks/php-app
docker rmi mebooks/apache-php5


Hopefuly the above is well-enough detailed to help you get to grips with Elastic Beanstalk.

It's a great mechanism for easily deploying a scalable Docker-based application, and all the better when we can automate the creation and deployment of our environments and applications.

In case using a Makefile as we did here becomes a bit limiting, you may want to look at using something like Ansible to automate the creation and deployment tasks, and there's
a great example of this at

Elastic Beankstalk is a very handy service, but should you want to create a more enterprise-level cloud application, you might also want to look at something like RedHat's OpenShift Origin.

However, being a developer rather than operations, I found the method outlined above
suited me, as it was more Dev-ops (with a big 'D') rather than dev-Ops (with a big 'O').

Further reading

In putting together the above, I found these posts of help: