Capturing Errors with Sentry

When deploying applications into production, the challenge then becomes keeping on top of any errors that may be thrown during day-to-day operation.

In the bad old days, you might put in some error trapping to log to a disk file (and then remember to check this file regularly) or send yourself an email.

Thankfully, we've largely outgrown this approach, and these days there's a number of services which provide online error tracking, such as Raygun; Paul Irish has a list of similar services.

If you want to install an error tracking service on your own hardware, then Sentry is a pretty good choice, being a Python application that is available in both SAAS and OpenSource versions.

Sentry is the product of David Cramer (an engineer at Dropbox) and Chris Jennings (a designer at Github) and was started while they were at Disqus. In an interview on LeanStack they mention that they consider themselves fortunate enough to have good day jobs such that Sentry can thrive as a OpenSource project:

Yeah, so we open-sourced it because we wanted people to use it. Not because we wanted to create this thing and make money off of it. And it's still the same thing today. A lot of people don't pay for Sentry. I would say 90% of people just host it themselves...But those 90% of people who aren't paying us are directly making Sentry better for the people who are paying us so it works.

thumb
The Sentry dashboard, showing a number of different errors

Sentry provides an HTTP API to allow your application to capture and send it information about errors, typically achieved using the officially sanctioned Raven libraries, and Raven provides error collectors for languages such as PHP, Python, Ruby, Node, and client-side JavaScript (a full list can be found in the Sentry docs).

Installation

Although I initially tried to install Sentry using pip and easy_install, I ran into some problems with reported errors not making it onto the dashboard, so ended up installing from source.

The following notes were developed through deploying Sentry on an Ubuntu VPS, but should be broadly similar for any Linux-like environment.

As I'm a fan of Ansible, I include some of the steps below as Ansible scripts; if you're not conversant with Ansible, don't worry as you should be able to figure out the equivalent shell commands from the scripts without too much problem.

Our main control file looks like:

---
# roles/sentry/main.yml
# This Playbook deploys sentry!

# http://cavaencoreparlerdebits.fr/blog/2014/04/install-sentry-to-catch-and-manage-your-errors-softwares
# http://blog.jesse-obrien.ca/post/add-beautiful-and-effective-exception-handling-to-laravel-with-sentry

- hosts: droplets
  user: root
  vars_files:
    - vars/main.yml
  handlers:
   - include: handlers/main.yml

  tasks:
  - include: tasks/sentry_packages.yml
  - include: tasks/sentry_user.yml
  - include: tasks/sentry_env.yml
  - include: tasks/sentry_mysql.yml
  - include: tasks/sentry_install.yml
  - include: tasks/sentry_nginx.yml

We also have some variables that are referenced by the scripts:

# roles/sentry/vars/main.yml
sentry:
  user: sentry
  password: whatever
  hostname: whatever
  listen_addr: 127.0.0.1
  listen_port: 9000
  db: sentry
  db_username: sentry
  db_password: whatever
  url: http://whatever
  env: sentry-env
  email_host: whatever
  email_host_user: whoever@wherever
  email_host_password: whatever
  email_port: whichever
  email_use_tls: True

Install the requisite packages

# roles/sentry/tasks/sentry_packages.yml
# ===============================================================
# packages: install the necessary packages
# ===============================================================

# Add the officially-sanctioned nodejs ppa
- name: add-apt-repository ppa:chris-lea/node.js
  action: template 
    src=files/chris-lea-node_js-wheezy.list
    dest=/etc/apt/sources.list.d/chris-lea-node_js-wheezy.list
    owner=root group=root mode=0644
  tags: packages

- name: update apt cache
  apt: update_cache=yes cache_valid_time=3600
  sudo: yes
  tags: packages

- name: install packages
  apt: pkg={{ item }} state=latest
  sudo: yes
  with_items:
    - nginx
    - python-software-properties
    - python-pip
    - python-dev
    - build-essential
  tags: packages

- name: force install packages (because we're using a ppa)
  apt: pkg={{ item }} state=latest force=yes
  sudo: yes
  with_items:
    - nodejs
  tags: packages

- name: install python packages using pip
  pip: name={{ item }}
  with_items:
    - virtualenv
    - virtualenvwrapper
    - python-memcached
  tags: packages

The above Ansible script can be run using the --tags option:

# Install the packages
ansible-playbook -l droplets roles/sentry/main.yml -i hosts --tags=packages

Here we install a number of python-based packages, as well as Node from the officially-sanctioned PPA. Note that we had to change wheezy to lucid in the PPA definition as it appears the PPA is not yet supporting wheezy:

# roles/sentry/files/chris-lea-node_js-wheezy.list
deb http://ppa.launchpad.net/chris-lea/node.js/ubuntu lucid main
deb-src http://ppa.launchpad.net/chris-lea/node.js/ubuntu lucid main

Create our user

We need to create a user to run Sentry under:

# roles/sentry/tasks/sentry_user.yml
# ===============================================================
# user: create the sentry user
# ===============================================================

- name: Create sentry user
  user: name={{ sentry.user }}
        password={{ sentry.password }}
        createhome=yes
        shell=/bin/bash
        state=present
  tags: user

- name: Create the SSH directory
  file: state=directory path=/home/{{ sentry.user }}/.ssh

- name: Add authorized key for the user
  authorized_key: user={{ sentry.user }} key='{{ item }}'
  with_file:
    - ~/.ssh/id_rsa.pub

- name: Backup sudoers file
  command: cp -f /etc/sudoers /etc/sudoers.bak
  tags: user

- name: Add sentry user to sudoers
  action: lineinfile
        dest=/etc/sudoers
        regexp='{{ sentry.user }} ALL'
        line='{{ sentry.user }} ALL=(ALL) ALL'
        state=present
  tags: user

- name: Check sudoers file syntax
  shell: visudo -q -c -f /etc/sudoers
  register: result
  ignore_errors: True
  tags: user

- name: Rolling back - restoring backed-up sudoers file
  action: cp -f /etc/sudoers.bak /etc/sudoers
  when: result|failed
  tags: user

The above Ansible script can be run using the --tags option:

# Create the sentry user and home directory
ansible-playbook -l droplets roles/sentry/main.yml -i hosts --tags=user

Create our virtual environment

We want to use a python virtual environment to isolate our Sentry python dependencies, and ensure that we can run different versions of python for other applications.

We also install our Sentry conf — normally this is installed using sentry init once Sentry has been installed, however we want to template our Sentry conf for use with Ansible and customise a few of the directives, such as using MySQL rather than the default SQLite. More details about the Sentry configuration can be found on the Sentry site.

The important part of the sentry.conf is the SENTRY_URL_PREFIX, which should be set to your domain:

SENTRY_URL_PREFIX = 'example.com'  # No trailing slash!

# ===============================================================
# env: initialise the sentry environment
# ===============================================================

- name: include virtualenvwrapper in our shell
  lineinfile: >
    destfile=/home/{{ sentry.user }}/.bashrc
    regexp="^source /usr/local/bin/virtualenvwrapper\.sh"
    line="source $HOME/.virtualenvs/{{ sentry.env}}/bin/activate"
    state=present
  remote_user: "{{ sentry.user }}"
  tags: env

- name: create our virtual environment
  shell: >
    executable=/bin/bash
    source `which virtualenvwrapper.sh` && mkvirtualenv {{ sentry.env }}
  register: create_virtualenv
  remote_user: "{{ sentry.user }}"
  environment:
    HOME: /home/{{ sentry.user }}
  tags: env

- debug: var=create_virtualenv.stdout_lines
  tags: env

- name: install the sentry conf
  template: >
    src=templates/sentry.conf.py.j2
    dest=~/sentry.conf.py
    owner="{{ sentry.user }}" group="{{ sentry.user }}" mode=0644
  remote_user: "{{ sentry.user }}"
  environment:
    HOME: /home/{{ sentry.user }}
  tags: env

- name: install python packages inside the virtual env using pip
  pip: name={{ item }} virtualenv={{ sentry.env }}
  with_items:
    - MySQL-python
  tags: env

The above Ansible script can be run using the --tags option:

# Create our virtual environment
ansible-playbook -l droplets roles/sentry/main.yml -i hosts --tags=env

Importantly, the above script will create our virtual environment (in /home/sentry/.virtualenvs/sentry-env) and will add the following line to our .bashrc, which ensures that the virtual environment is loaded when we run Sentry under our sentry user:

source $HOME/.virtualenvs/{{ sentry.env}}/bin/activate

Installing Sentry

Now that we've done the hard work of setting up the environment, the actual install of Sentry is quite easy:

# Clone the sentry project
cd /home/sentry
git clone git://github.com/getsentry/sentry.git sentry

# Make sentry.
# This will be created in /home/sentry/.virtualenvs/sentry-env
cd /home/sentry/sentry/
make

# Create our db and user
ansible-playbook -l droplets roles/sentry/main.yml -i hosts --tags=mysql

# Run the database migrations
sentry --config=/home/sentry/sentry.conf.py upgrade

Here we're using the following Ansible script to setup our MySQL environment as we'd like it, prior to running Sentry's database migrations:

# roles/sentry/tasks/sentry_mysql.yml
#     ===============================================================
# mysql: install sentry database
# ===============================================================

- name: install mysql dependencies
  action: apt pkg={{item}} state=installed
  with_items:
    - mysql-server 
    - mysql-client 
    - libmysqlclient-dev
    - python-mysqldb
  tags: mysql


# 'localhost' needs to be the last item for idempotency, see
# http://ansible.cc/docs/modules.html#mysql-user
- name: determine whether .my.cnf exists
  shell: ls -la ~/.my.cnf
  ignore_errors: True
  register: ls_my_cnf
  tags: mysql

- name: copy .my.cnf file with empty root password credentials
  template: src=.my.cnf.orig.j2 dest=/root/.my.cnf owner=root mode=0600
  when: ls_my_cnf.stdout.find("cannot access") == 1
  tags: mysql

- name: update mysql root password for all root accounts
  mysql_user: name={{ mysql.user }} host={{ item }} password={{ mysql.password }} priv=*.*:ALL,GRANT
  with_items:
    - 127.0.0.1
    - localhost
  tags: mysql

- name: update mysql sentry password for all sentry accounts
  mysql_user: name={{ sentry.db_username }} host={{ item }} password={{ sentry.db_password }} priv={{ sentry.db }}.*:ALL,GRANT
  with_items:
    - localhost
  tags: mysql

- name: copy .my.cnf file with root password credentials
  template: src=.my.cnf.j2 dest=/root/.my.cnf owner=root mode=0600
  tags: mysql

- name: ensure database sentry is present
  mysql_db: name={{ sentry.db }}
        collation=utf8_unicode_ci
        encoding=utf8
        state=present
  tags: mysql

- name: ensure database user for sentry is present and has necessary privileges
  mysql_user: name={{ sentry.db_username }}
        host={{ item }}
        password={{ sentry.db_password }}
        priv={{ sentry.db }}.*:ALL,GRANT
        state=present
  with_items:
    - localhost
  tags: mysql

We then need to create the administrator account:

sentry --config=/home/sentry/sentry.conf.py createsuperuser

And following this, run the repair

sentry --config=/homee/sentry/sentry.conf.py repair --owner=<superusername>

Nginx

We want to reverse-proxy Sentry to the outside world using Nginx:

server {
    listen 80;
   server_name {{ sentry.hostname }};

    access_log /var/log/nginx/{{ sentry.hostname }}.access.log;
    error_log /var/log/nginx/{{ sentry.hostname }}.error.log;

    # keepalive + raven.js is a disaster
    keepalive_timeout 0;

    location / {
        # proxy_pass http://127.0.0.1:9000;
        proxy_pass http://{{ sentry.listen_addr }}:{{ sentry.listen_port }};
        proxy_redirect off;

        proxy_set_header   Host              $host;
        proxy_set_header   X-Real-IP         $remote_addr;
        proxy_set_header   X-Forwarded-For   $proxy_add_x_forwarded_for;
        proxy_set_header   X-Forwarded-Proto $scheme;

        error_page 500 502 503 504 /500.html;
    }
}

The above Ansible script can be run using the --tags option:

# Create our virtual environment
ansible-playbook -l droplets roles/sentry/main.yml -i hosts --tags=env

nginx reload

Supervisor

We'll use Supervisor to look after our Sentry process, and this means creating the following as /etc/supervisor/conf.d/sentry.conf:

[program:sentry-web]
directory=/home/{{ sentry.user }}/.virtualenvs/{{ sentry.env }}/
command=/home/{{ sentry.user }}/.virtualenvs/{{ sentry.env }}/bin/sentry --config=/home/{{ sentry.user }}/sentry.conf.py start
autostart=true
autorestart=true
redirect_stderr=true

[program:sentry-worker]
directory=/home/{{ sentry.user }}/.virtualenvs/{{ sentry.env }}/
command=/home/{{ sentry.user }}/.virtualenvs/{{ sentry.env }}/bin/sentry celery worker -B
autostart=true
autorestart=true
redirect_stderr=true

We need to tell Supervisor about the new configuration file:

sudo supervisorctl restart sentry-web

Now that Sentry's installed, you should be able to visit the Dashboard and login using the superuser credentials you set earlier.

You'll want to obtain the DSN, which can be found at: http://<your domain>/sentry-internal/sentry/keys/

Capturing errors to Sentry

Having Sentry installed is nice, but it's a little bit redundant until we start sending it errors.

The Raven PHP client has some good notes about creating a simple application to post to Sentry.

We'll create a simple PHP script to generate a few errors, and capture them to our Sentry installation using Raven.

<?php 
// sentry-test.php

// Use Composer's autoloader
require('vendor/autoload.php');
Raven_Autoloader::register();
 
$client = new Raven_Client(
    // Importantly, we define below the DSN.
    // This is made up of a <public key> and a <secret>, which can be found at
    // http://<your domain>/sentry-internal/sentry/keys/
    // while <your domain> will be the value defined for SENTRY_URL_PREFIX in sentry.conf.
    'http://<public key>:<secret>@<your domain>/1'
    , array(
        'tags' => array(
            'php_version' => phpversion()
    ),
));
 
echo "Capture a debug message\n";
$client->captureMessage('Test debug message %s', array('foo'), array(
    'level' => Raven_Client::DEBUG,
    'extra' => array('foo' => 'bar')
));

echo "Capture an info message\n";
$client->captureMessage('Test info message %s', array('foo'), array(
    'level' => Raven_Client::INFO,
    'extra' => array('foo' => 'bar')
));

echo "Capture a warning message and obtain the Sentry reference\n";
$event_id = $client->getIdent($client->captureMessage('Test warning message %s', array('foo'), array(
    'level' => Raven_Client::WARNING,
    'extra' => array('foo' => 'bar')
)));
echo "Your reference ID is " . $event_id . "\n";
 
try {
    echo "Capture an exception and obtain the Sentry reference\n";
    throw new Exception('Uh oh!');
}
catch (Exception $e) {
    $event_id = $client->getIdent($client->captureException($e));
    echo "Your reference ID is " . $event_id . "\n";
}
 
// optionally install a default error handler to catch all exceptions
$error_handler = new Raven_ErrorHandler($client);
 
// Register error handler callbacks
set_error_handler(array($error_handler, 'handleError'));
set_exception_handler(array($error_handler, 'handleException'));

We'll also need to install the Raven PHP library, and as we also want to make use of Composer's autoloader, we'll use Composer:

{
    "name": "you/sentry-test",
    "description": "Test an newly-installed sentry installation",
    "require": {
        "raven/raven": "dev-master"
    },
    "license": "whatever",
    "authors": [
        {
            "name": "your name",
            "email": "your email"
        }
    ],
    "minimum-stability": "dev"
}

To install the composer dependencies:

# Install composer if necessary
curl -sS https://getcomposer.org/installer | php

# Install the dependencies for this project
composer install

Running the following should generate some messages and errors, which should appear in quick fashion on your Sentry dashboard:

php sentry-test.php