Fast HTTPS using Pound, Varnish, LetsEncrypt and Pagespeed
Image courtesy of Jay Mantri.
Making https fast
Let’s Encrypt is a great initiative allowing anyone to easily generate and install SSL certificates so that traffic can be securely served over https.
As well as gaining the benefits of secure traffic, by using https we're also setting ourselves up to benefit from HTTP/2, which will allow a number of performance benefits, but is only available over https.
However, generating and installing SSL certificates using Let’s Encrypt doesn't inherently make our site faster, and actually adds some overhead in terms of extra negotiation between the client and the server, as well as requiring the client to check with the certificate authorities to determine whether a certificate has been revoked (at least until we configure the server to take over this task using OCSP stapling.)
Two great measures for improving website performance are Varnish and Google's Pagespeed Module for Apache and Nginx.
Below, we'll look at how to generate and install SSL certificates with Let’s Encrypt, and then how to configure our site so that we can still make use of Varnish and the Pagespeed Module to optimise our site performance.
We're assuming here that you've already got a site up and running using Varnish, and Apache/Nginx, serving http traffic on port 80.
Let’s Encrypt
Let’s Encrypt is now in public beta, meaning that it's available for anyone to use, though you'll need to be aware of the rate-limiting to prevent abuse of the service, the most relevant parts of which are:
-
Names/Certificate is the limit on how many domain names you can include in a single certificate. This is currently limited to 100 names, or websites, per certificate issued.
-
Certificates/Domain you could run into through repeated re-issuance. This limit measures certificates issued for a given combination of Public Suffix + Domain (a "registered domain"). This is limited to 5 certificates per domain per week.
To install and use Let’s Encrypt, we'll do the following (presuming you're running Ubunutu or Debian):
# Create a directory for letsencrypt
mkdir /usr/local/letsencrypt
# Clone letsencrypt from the repo.
# You may want to adjust the release tag
cd /usr/local/letsencrypt
git clone https://github.com/letsencrypt/letsencrypt.git#v0.4.2
# Stop any webservers
service apache2 stop
service nginx stop
# Run letsencrypt to create our certificate.
./letsencrypt-auto certonly --email whoever@gmail.com --agree-tos \
-d whatever.nz -d www.whatever.nz \
-d whatever.co.nz -d www.whatever.co.nz
Note that we stop our webservers before running letsencrypt, as it needs to use ports 80 and 443 for communication with the certificate authorities.
We're also using --agree-tos
to automatically agree to the terms of service, and --email
to provide our email so that letsencrypt can create our account. We actually use Ansible configuration management scripts for provisioning our webservers and so need these scripts to run automatically, but in the interests of simplicity, we're not showing that here.
Letsencrypt can include a number of domains on a given certificate (currently up to a 100), though it can't produce wildcard certificates, and we use the -d
option above to specify each domain.
Note also that we're using the certonly
function; letsencrypt can automatically update webserver configuration files to use the newly generated certificate, but in our case we're going to use Pound to handle https, and our webservers will therefore only be dealing with http traffic.
More information about running letsencrypt can be found on the letsencrypt site.
Letsencypt certificates are currently valid for 90 days, and in the interests of making life easier for ourselves, we'll set up a cron job to automatically attempt to renew the certificate on a weekly basis. Our bash script, /usr/local/letsencrypt/letsencrypt-auto-renew.sh
, is as follows:
#!/bin/bash
# letsencrypt needs access to port 80
monit stop pound
# To simulate a dry-run near the end of the certificate term:
# /usr/local/letsencrypt/letsencrypt-auto renew --dry-run --email mebooks.support@gmail.com --agree-tos
if ! /usr/local/letsencrypt/letsencrypt-auto renew --email mebooks.support@gmail.com --agree-tos > /var/log/le-renew.log 2>&1; then
echo Automated renewal failed:
cat /var/log/le-renew.log
exit 1
fi
# Restart pound
monit start pound
# Cat the privkey and fullchain for pound
cat /etc/letsencrypt/live/whatever.co.nz/privkey.pem /etc/letsencrypt/live/whatever.co.nz/fullchain.pem > /etc/letsencrypt/archive/whatever.co.nz/privkey_fullchain.pem
# Recreate the symlink
unlink /etc/letsencrypt/live/whatever.co.nz/privkey_fullchain.pem
ln -s /etc/letsencrypt/archive/whatever.co.nz/privkey_fullchain.pem /etc/letsencrypt/live/whatever.co.nz/privkey_fullchain.pem
We can ensure that this runs on a weekly frequency by entering a line in our cron file, using crontab -e
:
# m h dom mon dow command
30 2 * * 1 /usr/local/letsencrypt/letsencrypt-auto-renew.sh >> /var/log/le-renew.log
So, we've now got a newly-generated certificate installed, but we actually need to make some use of it.
Varnish
Varnish provides a great http caching layer for websites, and is probably the single most important mechanism for ensuring that your site can serve traffic quickly to a large audience.
However, Varnish only serves traffic via http, and this comment from the maintainer makes it pretty clear it's unlikely it will ever have support for https:
Would I be able to write a better stand-alone SSL proxy process than the many which already exists ?
Probably not, unless I also write my own SSL implementation library, including support for hardware crypto engines and the works.
That is not one of the things I dreamt about doing as a kid and if I dream about it now I call it a nightmare.
So, if we want to use https, we have to use another service in front of Varnish to proxy https traffic as http to and from Varnish.
If you are using Varnish to directly cache traffic on port 80, your /etc/default/varnish
would probably have a line that looks like:
DAEMON_OPTS="-a :80 \
However, as we'll now use Pound to receive traffic in front of Varnish, we'll ensure that Varnish is receiving traffic on its default port of 6081:
DAEMON_OPTS="-a :6081 \
Enter Pound
Pound is a great project, and works very nicely in front of Varnish as reverse proxy, accepting http and/or https traffic as configured, and passing it through to Varnish as http.
In order to be able to disable the SSLv3 protocol to cope with the POODLE vulnerability, we'll need to use Pound 2.7f, as outlined here. Currently, the standard Ubuntu repositories only make Pound 2.6 available.
# Add the PPA
sudo add-apt-repository 'deb https://mslinn-ppa.s3.amazonaws.com stable main'
# Update our apt-get cache
sudo apt-get update
# Check that we see: Candidate:2.7f-0ubuntu1
sudo apt-cache policy pound
# Install Pound
sudo apt-get install pound
Once installed, we'll use the following /etc/pound/pound.cfg
:
## Minimal sample pound.cfg
##
## see pound(8) for details
######################################################################
## global options:
User "www-data"
Group "www-data"
#RootJail "/chroot/pound"
## Logging: (goes to syslog by default)
## 0 no logging
## 1 normal
## 2 extended
## 3 Apache-style (common log format)
LogLevel 1
## check backend every X secs:
Alive 30
## use hardware-accelleration card supported by openssl(1):
#SSLEngine "<hw>"
# poundctl control socket
Control "/var/run/pound/poundctl.socket"
######################################################################
## listen, redirect and ... to:
ListenHTTP
# 0.0.0.0 below should be your public IP address
Address 0.0.0.0
Port 80
# This part makes sure you redirect all HTTP traffic to HTTPS
Service
HeadRequire "Host: whatever.co.nz"
Redirect 301 "https://whatever.co.nz"
End
End
# As per https://milos.jakovljevic.me/howto-lets-encrypt-ssl-with-varnish-and-pound-on-ubuntu-server/
ListenHTTPS
HeadRemove "X-Forwarded-Proto"
AddHeader "X-Forwarded-Proto: https"
Address {{ public_ip }}
Port 443
Cert "/etc/letsencrypt/live/whatever.co.nz/privkey.pem"
# http://permalink.gmane.org/gmane.comp.web.pound.general/7489
Disable SSLv2
Disable SSLv3
SSLAllowClientRenegotiation 0
SSLHonorCipherOrder 1
# We want to get an A on the Qualys SSL Test (https://www.ssllabs.com/ssltest)
# https://scotthelme.co.uk/a-plus-rating-qualys-ssl-test/
Ciphers "ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS"
# Ensure pound doesn't rewrite location headers, as this can cause a redirect loop
RewriteLocation 0
Service
BackEnd
Address 127.0.0.1
Port 6081
End
End
End
# Get pound to do our hostname redirects (whether they come in as HTTP or HTTPS)
Service
HeadRequire "^Host: www.whatever.co.nz$"
Redirect 301 "https://whatever.nz"
End
Note the following:
- We're using the
ListenHTTP
block to listen for http traffic on port 80, and redirect it (permanently) to port 443. - We use the
ListenHTTPS
block to listen for https traffic on port 443, and direct it to Varnish on port 6081 - We disable the SSLv2 and SSLv3 protocols using
Disable SSLv2
andDisable SSLv3
- We disallow client renegotiation of the SSL protocol using
SSLAllowClientRenegotiation
, in order to stop attackers trying to demote a secure connection to a less secure connection - We use
SSLHonorCipherOrder
to ensure our ciphers are applied in the order that we specify them; this allows us to ensure that the browsers will use the most secure first, if they have support for it - We specify our allowed ciphers using
Ciphers
- We use
RewriteLocation
to stop Pound rewriting location headers, as this can lead to redirect loops - We use Pound to do our hostname redirects (whether they come in as HTTP or HTTPS), redirecting www.whatever.co.nz to http://whatever.co.nz.
Note that, in order to avoid the POODLE vulnerability, we're disabling the SSLv3 protocol, and not just the SSLv3 cipher. There appears to be some confusion, with some thinking that disabling the SSLv3 cipher is enough to avoid POODLE, but that's not the case.
Our Ciphers
is a list of reasonably modern ciphers, ordered from most-secure to least-secure:
ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS
We don't provide support for ciphers such as RC4, as this is considered too weak. This means people using Internet Explorer 6 won't be able to access our site, but that's not really a problem these days.
We're using Scott Helme's article on Getting an A+ rating on the Qualys SSL Test to ensure that we follow good practice regarding security. The Qualsys SSLTest is a great resource to determine how secure your site is, and using the above ciphers and SSLv2/SSLv3 migitations, we're able to acheive a solid A grade.
As security good practice evolves, you'll want to revisit the Qualsys SSLTest and ensure that you're continuing to provide a secure site.
As we're going to be using Pound to listen for traffic on port 443, we need to ensure that our webserver doesn't bind to this port.
For Apache, our /etc/apache2/ports.conf
will look like:
# Varnish serves traffic to Apache on port 8080
Listen 8080
<IfModule ssl_module>
# We'll park Apache's SSL to listen on 8443
Listen 8443
</IfModule>
<IfModule mod_gnutls.c>
# We'll park Apache's SSL to listen on 8443
Listen 8443
</IfModule>
You'll want to ensure that your Apache config changes are valid before restarting Apache:
apache2ctl configtest
In order to activate Pound, we need to enable it:
# Check that our pound.cfg is valid
pound -c
# Remove the pound default startup state in /etc/default/pound
sed -i '/startup=0/startup=1/' /etc/default/pound
# Start pound
service pound start
Pagespeed module
We've now got pound enforcing that all traffic is https, and working nicely with Varnish, which continues to cache our site content.
We've not had to make any changes to our Apache or Nginx site vhost configs, as they're still seeing traffic as http.
Google's Pagespeed Module for Apache and Nginx is a great drop-in webserver module for speeding up our site by making a number of on-the-fly optimisations. It's a topic in itself, and it's worth reading through the documentation to understand how it works and how it can benefit you.
sudo dpkg -i mod-pagespeed-*.deb
sudo apt-get -f install
This results in the Pagespeed module being installed, and since we're using Apache, we'll find the Pagespeed configuration at /etc/apache2/mods-enabled/pagespeed.conf
.
We'll want to make the following changes in /etc/apache2/mods-enabled/pagespeed.conf
:
- Ensure it's on:
ModPagespeed on
- Respect X-Forwarded-Proto headers:
ModPagespeedRespectXForwardedProto on
As Pagespeed is seeing all traffic coming from Varnish as http, it will normally serve any optimisations accordingly as http. However, this will cause Mixed content
warnings in our browser, as http assets are being served on an https site.
Using ModPagespeedRespectXForwardedProto on
ensures that Pagespeed checks for the X-Forwarded-Proto: https
header added by Pound, and serves any associated optimisations as https.
Summary
We've glossed over a few things above, as it's presumed that you've already installed and are using Varnish and a web-server such as Apache or Nginx to serve traffic on port 80.
However, the above is hopefully easy enough to follow, and will help you to serve your site securely using Let’s Encrypt and Pound, and fast, using Varnish and Google's Pagespeed Module for Apache and Nginx.
More notes about configuring Pound and Varnish can be found at the following: