Using Splunk with Traefik
In this post we'll look at what it takes to set up Splunk with the Traefik reverse-proxy, such that we can send log events using Splunks HTTP Event Collector, and see and query the results in Splunk's admin dashboard.
Splunk is a great tool for capturing and analysing application logs.
Although Splunk is reasonably pricey, they do allow you to use Splunk Enterprise self-hosted for up to 500MB of data per day.
Given that Splunk Enterprise is available as a Docker image, and I've been using Traefik as a reverse proxy for my projects recently, I was curious if I could get the two to work nicely together. Looking for clues as to how others had done this in Google, I came up empty-handed, hence this post.
Splunk configuration
Given that I'm developing on a Mac (though using Linux in production), I ran into the issue that Splunk doesn't yet have robust support for Mac OS High Sierra.
Thankfully there's a workaround that involves adding the OPTIMISTIC_ABOUT_FILE_LOCKING
setting to the splunk-launch.conf
.
Therefore, when we run Splunk, we'll share an edited version of the splunk-launch.conf
via a volume.
Set up the local directory containing an empty splunk-launch.conf
:
mkdir -p /opt/splunk
touch /opt/splunk/splunk-launch.conf
The splunk-launch.conf
will look like:
# Version 7.2.4
# Modify the following line to suit the location of your Splunk install.
# If unset, Splunk will use the parent of the directory containing the splunk
# CLI executable.
#
# SPLUNK_HOME=/opt/splunk-home
# By default, Splunk stores its indexes under SPLUNK_HOME in the
# var/lib/splunk subdirectory. This can be overridden
# here:
#
# SPLUNK_DB=/opt/splunk-home/var/lib/splunk
# Splunkd daemon name
SPLUNK_SERVER_NAME=Splunkd
# Splunkweb daemon name
SPLUNK_WEB_NAME=splunkweb
# If SPLUNK_OS_USER is set, then Splunk service will only start
# if the 'splunk [re]start [splunkd]' command is invoked by a user who
# is, or can effectively become via setuid(2), $SPLUNK_OS_USER.
# (This setting can be specified as username or as UID.)
#
# SPLUNK_OS_USER
# https://answers.splunk.com/answers/306998/why-am-i-getting-homepathoptsplunkvarlibsplunkaudi.html?childToView=578312#answer-578312
OPTIMISTIC_ABOUT_FILE_LOCKING=1
Running a Splunk docker container
Before we setup our docker-compose.yml
, we'll run a Splunk container directly, as we need to check that it works, and also set up the HTTP Event Collector.
In order to use Splunk to capture our logs, splunk exposes the HTTP Event Collector
service via an https endpoint, which we can ping with our log data.
Setting up the the HTTP Event Collector
basically involves generating an authorization token in the Splunk instance, which can then be used to authenticate each request to send data to Splunk.
# Create the local folders
mkdir -p /opt/splunk/etc/
mkdir -p /opt/splunk/var/
# Run the Splunk docker container
docker run \
-p 8000:8000 \
-p 8088:8088 \
-e 'SPLUNK_START_ARGS=--accept-license --no-prompt --answer-yes' \
-e 'SPLUNK_USERNAME=admin' \
-e 'SPLUNK_PASSWORD=CHANGEMENOW' \
-v /opt/splunk/splunk-launch.conf:/opt/splunk/etc/splunk-launch.conf \
-v /opt/splunk/etc:/opt/splunk/etc \
-v /opt/splunk/var:/opt/splunk/var \
splunk/splunk:latest
Presumably you'll want to change the password from CHANGEMENOW
to something more appropriate -- note that Splunk enforces a password policy of a length of at least eight ASCII characters.
Given that this is a test, and given that Splunk takes some time to start up, I've not deamonised the above command, so I can follow the progress easily at the terminal. You'll now that Splunk is ready when you see the following:
Ansible playbook complete, will begin streaming var/log/splunk/splunkd_stderr.log
You should now be able to see the Splunk admin interface at http://localhost:8000:
Setting up the HTTP Event Collector
We need to set up the HTTP Event Collector as per the Splunk documentation.
Once done, we should have a Token Value
at http://localhost:8000/en-US/manager/search/http-eventcollector:
We can test that this works by pinging the Splunk endpoint with some example data:
curl -ki https://localhost:8088/services/collector/event \
-H "Authorization: Splunk a6cb2c21-7dd1-4028-bbee-257f5a5e17db" \
-d '{"event": "hello splunk"}'
HTTP/1.1 200 OK
Content-Length: 27
Content-Type: application/json; charset=UTF-8
Date: Sun, 24 Mar 2019 00:13:53 GMT
Server: Splunkd
Vary: Authorization
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
{"text":"Success","code":0}
Note a couple of things about the call:
- We need to use
https
- Because we probably don't have a valid cert, we need to specify the
-k
Curl option.
Traefik
As Splunk demands that we call the HTTP Event Collector using an https endpoint, we need to ensure that we set up the Splunk backend service in Traefik as HTTPS.
However, this leads to the problem that Traefik will not be able to call the Splunk backend service via HTTPS as it doesn't have access to the cert.
We could extract the certs from the Splunk container (they live at $SPLUNK_HOME/etc/auth/
) and supply these to Traefik, but for a non-production set up, we can use the Traefik insecureSkipVerify
global configuration value.
We'll create a local directory in which our traefik.toml
will live:
mkdir -p /opt/etc/traefik
touch /opt/etc/traefik/traefik.toml
At a minimum, our traefik.toml
will look something like:
################################################################
# Global configuration
################################################################
defaultEntryPoints = ["http", "https"]
insecureSkipVerify = true
################################################################
# Entrypoints configuration
################################################################
# Entrypoints definition
#
# Optional
# Default:
[entryPoints]
# http should be redirected to https
[entryPoints.http]
address = ":80"
[entryPoints.http.redirect]
entryPoint = "https"
# https is now the default
[entryPoints.https]
address = ":443"
# Traefik status page
[entryPoints.traefik]
address = ":8080"
[entryPoints.traefik.auth.basic]
usersFile = "/usr/local/share/.auth"
# Enable ACME (Let's Encrypt): automatic SSL
# https://docs.traefik.io/user-guide/examples/#basic-example-with-http-challenge
[acme]
storage = "/etc/traefik/acme.json"
caServer = "https://acme-v02.api.letsencrypt.org/directory"
entryPoint = "https"
acmeLogging = true
onDemand = false
onHostRule = true
[acme.tlsChallenge]
################################################################
# Traefik logs configuration
################################################################
[traefikLog]
filePath = "/var/log/traefik/traefik.log"
################################################################
# Access logs configuration
################################################################
[accessLog]
filePath = "/var/log/traefik/access.log"
docker-compose.yml
We'll assume the following environment variables:
BASIC_AUTH_USERNAME=admin
BASIC_AUTH_PASSWORD=whatever
DOMAIN=localhost.example.com
EMAIL=whoever@example.com
SPLUNK_USERNAME=admin
SPLUNK_PASSWORD=changemenow
CERT=example.com.pem
KEY=example.com-key.pem
We'll assume that we've added the following entries to our /etc/hosts
file:
127.0.0.1 splunk.localhost.example.com
127.0.0.1 splunk-api.localhost.example.com
We'll also assume that we've generated a cert for *.localhost.example.com
-- you can use something like mkcert to generate this cert and key, and store the results at /opt/etc/traefik/
.
We'll add basic auth to Traefik by using a Dockerfile
based on traefik:alpine
:
# Dockerfile
FROM traefik:alpine
ARG BASIC_AUTH_USERNAME
ARG BASIC_AUTH_PASSWORD
ENV BASIC_AUTH_USERNAME $BASIC_AUTH_USERNAME
ENV BASIC_AUTH_PASSWORD $BASIC_AUTH_PASSWORD
RUN apk add --update apache2-utils \
&& rm -rf /var/cache/apk/* \
&& mkdir -p /var/log/traefik \
&& mkdir -p /etc/traefik \
&& touch /etc/traefik/acme.json \
&& chmod 0600 /etc/traefik/acme.json \
&& htpasswd -cBb /usr/local/share/.auth $BASIC_AUTH_USERNAME $BASIC_AUTH_PASSWORD
Our docker-compose.yml
ends up looking like the following:
version: '2'
services:
traefik:
build:
context: .
args:
- BASIC_AUTH_USERNAME=$BASIC_AUTH_USERNAME
- BASIC_AUTH_PASSWORD=$BASIC_AUTH_PASSWORD
container_name: "traefik.${DOMAIN}"
restart: unless-stopped
command: -c /dev/null --api --docker --logLevel=DEBUG --acme.email=$EMAIL \
--configFile=/etc/traefik/traefik.toml \
--entryPoints='Name:https Address::443 TLS:/etc/traefik/${CERT},/etc/traefik/${KEY}'
ports:
- 80:80
- 443:443
- 8025:8025
expose:
- 8080
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- /opt/etc/traefik:/etc/traefik
labels:
traefik.enable: "true"
traefik.backend: traefik
traefik.frontend.rule: "Host:traefik.${DOMAIN}"
traefik.port: "8080"
splunk:
# https://www.splunk.com/blog/2018/10/24/announcing-splunk-on-docker.html
image: splunk/splunk:latest
container_name: "splunk.${DOMAIN}"
environment:
SPLUNK_START_ARGS: --accept-license --no-prompt --answer-yes
SPLUNK_USERNAME: $SPLUNK_USERNAME
SPLUNK_PASSWORD: $SPLUNK_PASSWORD
restart: unless-stopped
labels:
# Use the following yaml formatting to allow our basic auth
# variables to be properly escaped
- "traefik.ui.backend=splunk"
- "traefik.ui.port=8000"
- "traefik.ui.frontend.rule=Host:splunk.${DOMAIN}"
- 'traefik.ui.frontend.auth.basic=${BASIC_AUTH_USERNAME}:${BASIC_AUTH_PASSWORD_ENCRYPTED}'
- "traefik.api.backend=splunk-api"
- "traefik.api.protocol=https"
- "traefik.api.port=8088"
- "traefik.api.frontend.rule=Host:splunk-api.${DOMAIN}"
volumes:
# We need to override splunk-launch.conf on Mac OS Sierra:
# https://answers.splunk.com/answers/306998/why-am-i-getting-homepathoptsplunkvarlibsplunkaudi.html?childToView=578312#answer-578312
- /opt/splunk/splunk-launch.conf:/opt/splunk/etc/splunk-launch.conf
- /opt/splunk/etc:/opt/splunk/etc
- /opt/splunk/var:/opt/splunk/var
expose:
- 8000
- 8088
depends_on:
- traefik
We should now be able to run up our stack. Again, we don't daemonise, so we can easily see the logs messages in the terminal, given that Splunk takes some time to start:
docker-compose up --build
We should then be able to see our admin at https://splunk.localhost.example.com.
We should also be able to ping the HTTP Event Collector:
curl -ki https://splunk-api.localhost.example.com/services/collector/event -H "Authorization: Splunk a6cb2c21-7dd1-4028-bbee-257f5a5e17db" -d '{"event": "hello splunk"}'
HTTP/1.1 200 OK
Content-Length: 27
Content-Type: application/json; charset=UTF-8
Date: Sun, 24 Mar 2019 00:13:53 GMT
Server: Splunkd
Vary: Authorization
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
{"text":"Success","code":0}
Using the Splunk service
In order to easily send logs to Splunk from docker containers, we'll want to install the docker-logging-plugin:
docker plugin install splunk/docker-logging-plugin:latest --alias splunk-logging-plugin
docker plugin enable splunk-logging-plugin
If we want another service in our docker-compose.yml
to direct logs to Splunk, we'll need to ensure it contains an appropriate logging
section, specifying the splunk-token
and splunk-url
we've set up previously:
logging:
driver: splunk-logging-plugin
options:
splunk-token: a6cb2c21-7dd1-4028-bbee-257f5a5e17db
splunk-url: https://splunk-api.${DOMAIN}
splunk-insecureskipverify: 'true'
labels: whatever
And that should be all we need to do to get Splunk running with Traefik in Docker, at least on a local Mac OS machine.
A production Splunk instance will require significantly more setup than this though, as you'll want to ensure Splunk is secure and robust.