Capturing Errors with Sentry
When deploying applications into production, the challenge then becomes keeping on top of any errors that may be thrown during day-to-day operation.
In the bad old days, you might put in some error trapping to log to a disk file (and then remember to check this file regularly) or send yourself an email.
Thankfully, we've largely outgrown this approach, and these days there's a number of services which provide online error tracking, such as Raygun; Paul Irish has a list of similar services.
If you want to install an error tracking service on your own hardware, then Sentry is a pretty good choice, being a Python application that is available in both SAAS and OpenSource versions.
Sentry is the product of David Cramer (an engineer at Dropbox) and Chris Jennings (a designer at Github) and was started while they were at Disqus. In an interview on LeanStack they mention that they consider themselves fortunate enough to have good day jobs such that Sentry can thrive as a OpenSource project:
Yeah, so we open-sourced it because we wanted people to use it. Not because we wanted to create this thing and make money off of it. And it's still the same thing today. A lot of people don't pay for Sentry. I would say 90% of people just host it themselves...But those 90% of people who aren't paying us are directly making Sentry better for the people who are paying us so it works.
The Sentry dashboard, showing a number of different errors
Sentry provides an HTTP API to allow your application to capture and send it information about errors, typically achieved using the officially sanctioned Raven libraries, and Raven provides error collectors for languages such as PHP, Python, Ruby, Node, and client-side JavaScript (a full list can be found in the Sentry docs).
Installation
Although I initially tried to install Sentry using pip and easy_install, I ran into some problems with reported errors not making it onto the dashboard, so ended up installing from source.
The following notes were developed through deploying Sentry on an Ubuntu VPS, but should be broadly similar for any Linux-like environment.
As I'm a fan of Ansible, I include some of the steps below as Ansible scripts; if you're not conversant with Ansible, don't worry as you should be able to figure out the equivalent shell commands from the scripts without too much problem.
Our main control file looks like:
---
# roles/sentry/main.yml
# This Playbook deploys sentry!
# http://cavaencoreparlerdebits.fr/blog/2014/04/install-sentry-to-catch-and-manage-your-errors-softwares
# http://blog.jesse-obrien.ca/post/add-beautiful-and-effective-exception-handling-to-laravel-with-sentry
- hosts: droplets
user: root
vars_files:
- vars/main.yml
handlers:
- include: handlers/main.yml
tasks:
- include: tasks/sentry_packages.yml
- include: tasks/sentry_user.yml
- include: tasks/sentry_env.yml
- include: tasks/sentry_mysql.yml
- include: tasks/sentry_install.yml
- include: tasks/sentry_nginx.yml
We also have some variables that are referenced by the scripts:
# roles/sentry/vars/main.yml
sentry:
user: sentry
password: whatever
hostname: whatever
listen_addr: 127.0.0.1
listen_port: 9000
db: sentry
db_username: sentry
db_password: whatever
url: http://whatever
env: sentry-env
email_host: whatever
email_host_user: whoever@wherever
email_host_password: whatever
email_port: whichever
email_use_tls: True
Install the requisite packages
# roles/sentry/tasks/sentry_packages.yml
# ===============================================================
# packages: install the necessary packages
# ===============================================================
# Add the officially-sanctioned nodejs ppa
- name: add-apt-repository ppa:chris-lea/node.js
action: template
src=files/chris-lea-node_js-wheezy.list
dest=/etc/apt/sources.list.d/chris-lea-node_js-wheezy.list
owner=root group=root mode=0644
tags: packages
- name: update apt cache
apt: update_cache=yes cache_valid_time=3600
sudo: yes
tags: packages
- name: install packages
apt: pkg={{ item }} state=latest
sudo: yes
with_items:
- nginx
- python-software-properties
- python-pip
- python-dev
- build-essential
tags: packages
- name: force install packages (because we're using a ppa)
apt: pkg={{ item }} state=latest force=yes
sudo: yes
with_items:
- nodejs
tags: packages
- name: install python packages using pip
pip: name={{ item }}
with_items:
- virtualenv
- virtualenvwrapper
- python-memcached
tags: packages
The above Ansible script can be run using the --tags option:
# Install the packages
ansible-playbook -l droplets roles/sentry/main.yml -i hosts --tags=packages
Here we install a number of python-based packages, as well as Node from the officially-sanctioned PPA. Note that we had to change wheezy to lucid in the PPA definition as it appears the PPA is not yet supporting wheezy:
# roles/sentry/files/chris-lea-node_js-wheezy.list
deb http://ppa.launchpad.net/chris-lea/node.js/ubuntu lucid main
deb-src http://ppa.launchpad.net/chris-lea/node.js/ubuntu lucid main
Create our user
We need to create a user to run Sentry under:
# roles/sentry/tasks/sentry_user.yml
# ===============================================================
# user: create the sentry user
# ===============================================================
- name: Create sentry user
user: name={{ sentry.user }}
password={{ sentry.password }}
createhome=yes
shell=/bin/bash
state=present
tags: user
- name: Create the SSH directory
file: state=directory path=/home/{{ sentry.user }}/.ssh
- name: Add authorized key for the user
authorized_key: user={{ sentry.user }} key='{{ item }}'
with_file:
- ~/.ssh/id_rsa.pub
- name: Backup sudoers file
command: cp -f /etc/sudoers /etc/sudoers.bak
tags: user
- name: Add sentry user to sudoers
action: lineinfile
dest=/etc/sudoers
regexp='{{ sentry.user }} ALL'
line='{{ sentry.user }} ALL=(ALL) ALL'
state=present
tags: user
- name: Check sudoers file syntax
shell: visudo -q -c -f /etc/sudoers
register: result
ignore_errors: True
tags: user
- name: Rolling back - restoring backed-up sudoers file
action: cp -f /etc/sudoers.bak /etc/sudoers
when: result|failed
tags: user
The above Ansible script can be run using the --tags option:
# Create the sentry user and home directory
ansible-playbook -l droplets roles/sentry/main.yml -i hosts --tags=user
Create our virtual environment
We want to use a python virtual environment to isolate our Sentry python dependencies, and ensure that we can run different versions of python for other applications.
We also install our Sentry conf — normally this is installed using sentry init once Sentry has been installed, however we want to template our Sentry conf for use with Ansible and customise a few of the directives, such as using MySQL rather than the default SQLite. More details about the Sentry configuration can be found on the Sentry site.
The important part of the sentry.conf is the SENTRY_URL_PREFIX, which should be set to your domain:
SENTRY_URL_PREFIX = 'example.com' # No trailing slash!
# ===============================================================
# env: initialise the sentry environment
# ===============================================================
- name: include virtualenvwrapper in our shell
lineinfile: >
destfile=/home/{{ sentry.user }}/.bashrc
regexp="^source /usr/local/bin/virtualenvwrapper\.sh"
line="source $HOME/.virtualenvs/{{ sentry.env}}/bin/activate"
state=present
remote_user: "{{ sentry.user }}"
tags: env
- name: create our virtual environment
shell: >
executable=/bin/bash
source `which virtualenvwrapper.sh` && mkvirtualenv {{ sentry.env }}
register: create_virtualenv
remote_user: "{{ sentry.user }}"
environment:
HOME: /home/{{ sentry.user }}
tags: env
- debug: var=create_virtualenv.stdout_lines
tags: env
- name: install the sentry conf
template: >
src=templates/sentry.conf.py.j2
dest=~/sentry.conf.py
owner="{{ sentry.user }}" group="{{ sentry.user }}" mode=0644
remote_user: "{{ sentry.user }}"
environment:
HOME: /home/{{ sentry.user }}
tags: env
- name: install python packages inside the virtual env using pip
pip: name={{ item }} virtualenv={{ sentry.env }}
with_items:
- MySQL-python
tags: env
The above Ansible script can be run using the --tags option:
# Create our virtual environment
ansible-playbook -l droplets roles/sentry/main.yml -i hosts --tags=env
Importantly, the above script will create our virtual environment (in /home/sentry/.virtualenvs/sentry-env) and will add the following line to our .bashrc, which ensures that the virtual environment is loaded when we run Sentry under our sentry user:
source $HOME/.virtualenvs/{{ sentry.env}}/bin/activate
Installing Sentry
Now that we've done the hard work of setting up the environment, the actual install of Sentry is quite easy:
# Clone the sentry project
cd /home/sentry
git clone git://github.com/getsentry/sentry.git sentry
# Make sentry.
# This will be created in /home/sentry/.virtualenvs/sentry-env
cd /home/sentry/sentry/
make
# Create our db and user
ansible-playbook -l droplets roles/sentry/main.yml -i hosts --tags=mysql
# Run the database migrations
sentry --config=/home/sentry/sentry.conf.py upgrade
Here we're using the following Ansible script to setup our MySQL environment as we'd like it, prior to running Sentry's database migrations:
# roles/sentry/tasks/sentry_mysql.yml
# ===============================================================
# mysql: install sentry database
# ===============================================================
- name: install mysql dependencies
action: apt pkg={{item}} state=installed
with_items:
- mysql-server
- mysql-client
- libmysqlclient-dev
- python-mysqldb
tags: mysql
# 'localhost' needs to be the last item for idempotency, see
# http://ansible.cc/docs/modules.html#mysql-user
- name: determine whether .my.cnf exists
shell: ls -la ~/.my.cnf
ignore_errors: True
register: ls_my_cnf
tags: mysql
- name: copy .my.cnf file with empty root password credentials
template: src=.my.cnf.orig.j2 dest=/root/.my.cnf owner=root mode=0600
when: ls_my_cnf.stdout.find("cannot access") == 1
tags: mysql
- name: update mysql root password for all root accounts
mysql_user: name={{ mysql.user }} host={{ item }} password={{ mysql.password }} priv=*.*:ALL,GRANT
with_items:
- 127.0.0.1
- localhost
tags: mysql
- name: update mysql sentry password for all sentry accounts
mysql_user: name={{ sentry.db_username }} host={{ item }} password={{ sentry.db_password }} priv={{ sentry.db }}.*:ALL,GRANT
with_items:
- localhost
tags: mysql
- name: copy .my.cnf file with root password credentials
template: src=.my.cnf.j2 dest=/root/.my.cnf owner=root mode=0600
tags: mysql
- name: ensure database sentry is present
mysql_db: name={{ sentry.db }}
collation=utf8_unicode_ci
encoding=utf8
state=present
tags: mysql
- name: ensure database user for sentry is present and has necessary privileges
mysql_user: name={{ sentry.db_username }}
host={{ item }}
password={{ sentry.db_password }}
priv={{ sentry.db }}.*:ALL,GRANT
state=present
with_items:
- localhost
tags: mysql
We then need to create the administrator account:
sentry --config=/home/sentry/sentry.conf.py createsuperuser
And following this, run the repair
sentry --config=/homee/sentry/sentry.conf.py repair --owner=<superusername>
Nginx
We want to reverse-proxy Sentry to the outside world using Nginx:
server {
listen 80;
server_name {{ sentry.hostname }};
access_log /var/log/nginx/{{ sentry.hostname }}.access.log;
error_log /var/log/nginx/{{ sentry.hostname }}.error.log;
# keepalive + raven.js is a disaster
keepalive_timeout 0;
location / {
# proxy_pass http://127.0.0.1:9000;
proxy_pass http://{{ sentry.listen_addr }}:{{ sentry.listen_port }};
proxy_redirect off;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
error_page 500 502 503 504 /500.html;
}
}
The above Ansible script can be run using the --tags option:
# Create our virtual environment
ansible-playbook -l droplets roles/sentry/main.yml -i hosts --tags=env
nginx reload
Supervisor
We'll use Supervisor to look after our Sentry process, and this means creating the following as /etc/supervisor/conf.d/sentry.conf:
[program:sentry-web]
directory=/home/{{ sentry.user }}/.virtualenvs/{{ sentry.env }}/
command=/home/{{ sentry.user }}/.virtualenvs/{{ sentry.env }}/bin/sentry --config=/home/{{ sentry.user }}/sentry.conf.py start
autostart=true
autorestart=true
redirect_stderr=true
[program:sentry-worker]
directory=/home/{{ sentry.user }}/.virtualenvs/{{ sentry.env }}/
command=/home/{{ sentry.user }}/.virtualenvs/{{ sentry.env }}/bin/sentry celery worker -B
autostart=true
autorestart=true
redirect_stderr=true
We need to tell Supervisor about the new configuration file:
sudo supervisorctl restart sentry-web
Now that Sentry's installed, you should be able to visit the Dashboard and login using the superuser credentials you set earlier.
You'll want to obtain the DSN, which can be found at: http://<your domain>/sentry-internal/sentry/keys/
Capturing errors to Sentry
Having Sentry installed is nice, but it's a little bit redundant until we start sending it errors.
The Raven PHP client has some good notes about creating a simple application to post to Sentry.
We'll create a simple PHP script to generate a few errors, and capture them to our Sentry installation using Raven.
<?php
// sentry-test.php
// Use Composer's autoloader
require('vendor/autoload.php');
Raven_Autoloader::register();
$client = new Raven_Client(
// Importantly, we define below the DSN.
// This is made up of a <public key> and a <secret>, which can be found at
// http://<your domain>/sentry-internal/sentry/keys/
// while <your domain> will be the value defined for SENTRY_URL_PREFIX in sentry.conf.
'http://<public key>:<secret>@<your domain>/1'
, array(
'tags' => array(
'php_version' => phpversion()
),
));
echo "Capture a debug message\n";
$client->captureMessage('Test debug message %s', array('foo'), array(
'level' => Raven_Client::DEBUG,
'extra' => array('foo' => 'bar')
));
echo "Capture an info message\n";
$client->captureMessage('Test info message %s', array('foo'), array(
'level' => Raven_Client::INFO,
'extra' => array('foo' => 'bar')
));
echo "Capture a warning message and obtain the Sentry reference\n";
$event_id = $client->getIdent($client->captureMessage('Test warning message %s', array('foo'), array(
'level' => Raven_Client::WARNING,
'extra' => array('foo' => 'bar')
)));
echo "Your reference ID is " . $event_id . "\n";
try {
echo "Capture an exception and obtain the Sentry reference\n";
throw new Exception('Uh oh!');
}
catch (Exception $e) {
$event_id = $client->getIdent($client->captureException($e));
echo "Your reference ID is " . $event_id . "\n";
}
// optionally install a default error handler to catch all exceptions
$error_handler = new Raven_ErrorHandler($client);
// Register error handler callbacks
set_error_handler(array($error_handler, 'handleError'));
set_exception_handler(array($error_handler, 'handleException'));
We'll also need to install the Raven PHP library, and as we also want to make use of Composer's autoloader, we'll use Composer:
{
"name": "you/sentry-test",
"description": "Test an newly-installed sentry installation",
"require": {
"raven/raven": "dev-master"
},
"license": "whatever",
"authors": [
{
"name": "your name",
"email": "your email"
}
],
"minimum-stability": "dev"
}
To install the composer dependencies:
# Install composer if necessary
curl -sS https://getcomposer.org/installer | php
# Install the dependencies for this project
composer install
Running the following should generate some messages and errors, which should appear in quick fashion on your Sentry dashboard:
php sentry-test.php