Fast HTTPS using Pound, Varnish, LetsEncrypt and Pagespeed

Image courtesy of Jay Mantri.

Making https fast

Let’s Encrypt is a great initiative allowing anyone to easily generate and install SSL certificates so that traffic can be securely served over https.

As well as gaining the benefits of secure traffic, by using https we're also setting ourselves up to benefit from HTTP/2, which will allow a number of performance benefits, but is only available over https.

However, generating and installing SSL certificates using Let’s Encrypt doesn't inherently make our site faster, and actually adds some overhead in terms of extra negotiation between the client and the server, as well as requiring the client to check with the certificate authorities to determine whether a certificate has been revoked (at least until we configure the server to take over this task using OCSP stapling.)

Two great measures for improving website performance are Varnish and Google's Pagespeed Module for Apache and Nginx.

Below, we'll look at how to generate and install SSL certificates with Let’s Encrypt, and then how to configure our site so that we can still make use of Varnish and the Pagespeed Module to optimise our site performance.

We're assuming here that you've already got a site up and running using Varnish, and Apache/Nginx, serving http traffic on port 80.

Let’s Encrypt

Let’s Encrypt is now in public beta, meaning that it's available for anyone to use, though you'll need to be aware of the rate-limiting to prevent abuse of the service, the most relevant parts of which are:

• Names/Certificate is the limit on how many domain names you can include in a single certificate. This is currently limited to 100 names, or websites, per certificate issued.

• Certificates/Domain you could run into through repeated re-issuance. This limit measures certificates issued for a given combination of Public Suffix + Domain (a "registered domain"). This is limited to 5 certificates per domain per week.

To install and use Let’s Encrypt, we'll do the following (presuming you're running Ubunutu or Debian):

# Create a directory for letsencrypt
mkdir /usr/local/letsencrypt

# Clone letsencrypt from the repo.
# You may want to adjust the release tag
cd /usr/local/letsencrypt
git clone https://github.com/letsencrypt/letsencrypt.git#v0.4.2

# Stop any webservers
service apache2 stop
service nginx stop

# Run letsencrypt to create our certificate.
./letsencrypt-auto certonly --email whoever@gmail.com --agree-tos \
    -d whatever.nz -d www.whatever.nz \
    -d whatever.co.nz -d www.whatever.co.nz

Note that we stop our webservers before running letsencrypt, as it needs to use ports 80 and 443 for communication with the certificate authorities.

We're also using --agree-tos to automatically agree to the terms of service, and --email to provide our email so that letsencrypt can create our account. We actually use Ansible configuration management scripts for provisioning our webservers and so need these scripts to run automatically, but in the interests of simplicity, we're not showing that here.

Letsencrypt can include a number of domains on a given certificate (currently up to a 100), though it can't produce wildcard certificates, and we use the -d option above to specify each domain.

Note also that we're using the certonly function; letsencrypt can automatically update webserver configuration files to use the newly generated certificate, but in our case we're going to use Pound to handle https, and our webservers will therefore only be dealing with http traffic.

More information about running letsencrypt can be found on the letsencrypt site.

Letsencypt certificates are currently valid for 90 days, and in the interests of making life easier for ourselves, we'll set up a cron job to automatically attempt to renew the certificate on a weekly basis. Our bash script, /usr/local/letsencrypt/letsencrypt-auto-renew.sh, is as follows:

#!/bin/bash

# letsencrypt needs access to port 80
monit stop pound

# To simulate a dry-run near the end of the certificate term:
# /usr/local/letsencrypt/letsencrypt-auto renew --dry-run --email mebooks.support@gmail.com --agree-tos
if ! /usr/local/letsencrypt/letsencrypt-auto renew --email mebooks.support@gmail.com --agree-tos > /var/log/le-renew.log 2>&1; then
    echo Automated renewal failed:
    cat /var/log/le-renew.log
    exit 1
fi

# Restart pound
monit start pound

# Cat the privkey and fullchain for pound
cat /etc/letsencrypt/live/whatever.co.nz/privkey.pem /etc/letsencrypt/live/whatever.co.nz/fullchain.pem > /etc/letsencrypt/archive/whatever.co.nz/privkey_fullchain.pem

# Recreate the symlink
unlink /etc/letsencrypt/live/whatever.co.nz/privkey_fullchain.pem
ln -s /etc/letsencrypt/archive/whatever.co.nz/privkey_fullchain.pem /etc/letsencrypt/live/whatever.co.nz/privkey_fullchain.pem

We can ensure that this runs on a weekly frequency by entering a line in our cron file, using crontab -e:

# m h  dom mon dow   command
 30 2  *   *   1     /usr/local/letsencrypt/letsencrypt-auto-renew.sh >> /var/log/le-renew.log

So, we've now got a newly-generated certificate installed, but we actually need to make some use of it.

Varnish

Varnish provides a great http caching layer for websites, and is probably the single most important mechanism for ensuring that your site can serve traffic quickly to a large audience.

However, Varnish only serves traffic via http, and this comment from the maintainer makes it pretty clear it's unlikely it will ever have support for https:

Would I be able to write a better stand-alone SSL proxy process than the many which already exists ?

Probably not, unless I also write my own SSL implementation library, including support for hardware crypto engines and the works.

That is not one of the things I dreamt about doing as a kid and if I dream about it now I call it a nightmare.

So, if we want to use https, we have to use another service in front of Varnish to proxy https traffic as http to and from Varnish.

If you are using Varnish to directly cache traffic on port 80, your /etc/default/varnish would probably have a line that looks like:

DAEMON_OPTS="-a :80 \

However, as we'll now use Pound to receive traffic in front of Varnish, we'll ensure that Varnish is receiving traffic on its default port of 6081:

DAEMON_OPTS="-a :6081 \

Enter Pound

Pound is a great project, and works very nicely in front of Varnish as reverse proxy, accepting http and/or https traffic as configured, and passing it through to Varnish as http.

In order to be able to disable the SSLv3 protocol to cope with the POODLE vulnerability, we'll need to use Pound 2.7f, as outlined here. Currently, the standard Ubuntu repositories only make Pound 2.6 available.

# Add the PPA
sudo add-apt-repository 'deb https://mslinn-ppa.s3.amazonaws.com stable main'

# Update our apt-get cache
sudo apt-get update

# Check that we see: Candidate:2.7f-0ubuntu1
sudo apt-cache policy pound

# Install Pound
sudo apt-get install pound

Once installed, we'll use the following /etc/pound/pound.cfg:

## Minimal sample pound.cfg
##
## see pound(8) for details
######################################################################
## global options:

User            "www-data"
Group           "www-data"
#RootJail       "/chroot/pound"

## Logging: (goes to syslog by default)
##      0       no logging
##      1       normal
##      2       extended
##      3       Apache-style (common log format)
LogLevel        1

## check backend every X secs:
Alive           30

## use hardware-accelleration card supported by openssl(1):
#SSLEngine      "<hw>"

# poundctl control socket
Control "/var/run/pound/poundctl.socket"
######################################################################
## listen, redirect and ... to:

ListenHTTP
    # 0.0.0.0 below should be your public IP address
    Address  0.0.0.0
    Port     80
    # This part makes sure you redirect all HTTP traffic to HTTPS
    Service
        HeadRequire "Host: whatever.co.nz"
        Redirect 301 "https://whatever.co.nz"
    End
End

# As per https://milos.jakovljevic.me/howto-lets-encrypt-ssl-with-varnish-and-pound-on-ubuntu-server/
ListenHTTPS
        HeadRemove "X-Forwarded-Proto"
        AddHeader  "X-Forwarded-Proto: https"
        Address    {{ public_ip }}
        Port       443
        Cert       "/etc/letsencrypt/live/whatever.co.nz/privkey.pem"
        # http://permalink.gmane.org/gmane.comp.web.pound.general/7489
        Disable SSLv2
        Disable SSLv3
        SSLAllowClientRenegotiation 0
        SSLHonorCipherOrder 1
        # We want to get an A on the Qualys SSL Test (https://www.ssllabs.com/ssltest)
        # https://scotthelme.co.uk/a-plus-rating-qualys-ssl-test/
        Ciphers     "ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS"
        # Ensure pound doesn't rewrite location headers, as this can cause a redirect loop
        RewriteLocation 0
        Service
                BackEnd
                        Address 127.0.0.1
                        Port    6081
                End
        End
End

# Get pound to do our hostname redirects (whether they come in as HTTP or HTTPS)
Service
        HeadRequire "^Host: www.whatever.co.nz$"
        Redirect 301 "https://whatever.nz"
End

Note the following:

• We're using the ListenHTTP block to listen for http traffic on port 80, and redirect it (permanently) to port 443.
• We use the ListenHTTPS block to listen for https traffic on port 443, and direct it to Varnish on port 6081
• We disable the SSLv2 and SSLv3 protocols using Disable SSLv2 and Disable SSLv3
• We disallow client renegotiation of the SSL protocol using SSLAllowClientRenegotiation, in order to stop attackers trying to demote a secure connection to a less secure connection
• We use SSLHonorCipherOrder to ensure our ciphers are applied in the order that we specify them; this allows us to ensure that the browsers will use the most secure first, if they have support for it
• We specify our allowed ciphers using Ciphers
• We use RewriteLocation to stop Pound rewriting location headers, as this can lead to redirect loops
• We use Pound to do our hostname redirects (whether they come in as HTTP or HTTPS), redirecting www.whatever.co.nz to http://whatever.co.nz.

Note that, in order to avoid the POODLE vulnerability, we're disabling the SSLv3 protocol, and not just the SSLv3 cipher. There appears to be some confusion, with some thinking that disabling the SSLv3 cipher is enough to avoid POODLE, but that's not the case.

Our Ciphers is a list of reasonably modern ciphers, ordered from most-secure to least-secure:

ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS

We don't provide support for ciphers such as RC4, as this is considered too weak. This means people using Internet Explorer 6 won't be able to access our site, but that's not really a problem these days.

We're using Scott Helme's article on Getting an A+ rating on the Qualys SSL Test to ensure that we follow good practice regarding security. The Qualsys SSLTest is a great resource to determine how secure your site is, and using the above ciphers and SSLv2/SSLv3 migitations, we're able to acheive a solid A grade.

As security good practice evolves, you'll want to revisit the Qualsys SSLTest and ensure that you're continuing to provide a secure site.

As we're going to be using Pound to listen for traffic on port 443, we need to ensure that our webserver doesn't bind to this port.

For Apache, our /etc/apache2/ports.conf will look like:

# Varnish serves traffic to Apache on port 8080
Listen 8080

<IfModule ssl_module>
    # We'll park Apache's SSL to listen on 8443
    Listen 8443
</IfModule>

<IfModule mod_gnutls.c>
    # We'll park Apache's SSL to listen on 8443
    Listen 8443
</IfModule>

You'll want to ensure that your Apache config changes are valid before restarting Apache:

apache2ctl configtest

In order to activate Pound, we need to enable it:

# Check that our pound.cfg is valid
pound -c

# Remove the pound default startup state in /etc/default/pound
sed -i '/startup=0/startup=1/' /etc/default/pound

# Start pound 
service pound start

Pagespeed module

We've now got pound enforcing that all traffic is https, and working nicely with Varnish, which continues to cache our site content.

We've not had to make any changes to our Apache or Nginx site vhost configs, as they're still seeing traffic as http.

Google's Pagespeed Module for Apache and Nginx is a great drop-in webserver module for speeding up our site by making a number of on-the-fly optimisations. It's a topic in itself, and it's worth reading through the documentation to understand how it works and how it can benefit you.

sudo dpkg -i mod-pagespeed-*.deb
sudo apt-get -f install

This results in the Pagespeed module being installed, and since we're using Apache, we'll find the Pagespeed configuration at /etc/apache2/mods-enabled/pagespeed.conf.

We'll want to make the following changes in /etc/apache2/mods-enabled/pagespeed.conf:

• Ensure it's on: ModPagespeed on
• Respect X-Forwarded-Proto headers: ModPagespeedRespectXForwardedProto on

As Pagespeed is seeing all traffic coming from Varnish as http, it will normally serve any optimisations accordingly as http. However, this will cause Mixed content warnings in our browser, as http assets are being served on an https site.

Using ModPagespeedRespectXForwardedProto on ensures that Pagespeed checks for the X-Forwarded-Proto: https header added by Pound, and serves any associated optimisations as https.

Summary

We've glossed over a few things above, as it's presumed that you've already installed and are using Varnish and a web-server such as Apache or Nginx to serve traffic on port 80.

However, the above is hopefully easy enough to follow, and will help you to serve your site securely using Let’s Encrypt and Pound, and fast, using Varnish and Google's Pagespeed Module for Apache and Nginx.

More notes about configuring Pound and Varnish can be found at the following:

• Web Server Performance Part II: Varnish and Pound
• Let's Encrypt Certificates and Pound Load Balancer