Using Splunk with Traefik

In this post we'll look at what it takes to set up Splunk with the Traefik reverse-proxy, such that we can send log events using Splunks HTTP Event Collector, and see and query the results in Splunk's admin dashboard.

Splunk is a great tool for capturing and analysing application logs.
Although Splunk is reasonably pricey, they do allow you to use Splunk Enterprise self-hosted for up to 500MB of data per day.

Given that Splunk Enterprise is available as a Docker image, and I've been using Traefik as a reverse proxy for my projects recently, I was curious if I could get the two to work nicely together. Looking for clues as to how others had done this in Google, I came up empty-handed, hence this post.

Splunk configuration

Given that I'm developing on a Mac (though using Linux in production), I ran into the issue that Splunk doesn't yet have robust support for Mac OS High Sierra.
Thankfully there's a workaround that involves adding the OPTIMISTIC_ABOUT_FILE_LOCKING setting to the splunk-launch.conf.

Therefore, when we run Splunk, we'll share an edited version of the splunk-launch.conf via a volume.

Set up the local directory containing an empty splunk-launch.conf:

mkdir -p /opt/splunk
touch /opt/splunk/splunk-launch.conf

The splunk-launch.conf will look like:

#   Version 7.2.4

# Modify the following line to suit the location of your Splunk install.
# If unset, Splunk will use the parent of the directory containing the splunk
# CLI executable.
#
# SPLUNK_HOME=/opt/splunk-home

# By default, Splunk stores its indexes under SPLUNK_HOME in the
# var/lib/splunk subdirectory.  This can be overridden
# here:
#
# SPLUNK_DB=/opt/splunk-home/var/lib/splunk
# Splunkd daemon name
SPLUNK_SERVER_NAME=Splunkd

# Splunkweb daemon name
SPLUNK_WEB_NAME=splunkweb

# If SPLUNK_OS_USER is set, then Splunk service will only start
# if the 'splunk [re]start [splunkd]' command is invoked by a user who
# is, or can effectively become via setuid(2), $SPLUNK_OS_USER.
# (This setting can be specified as username or as UID.)
#
# SPLUNK_OS_USER
# https://answers.splunk.com/answers/306998/why-am-i-getting-homepathoptsplunkvarlibsplunkaudi.html?childToView=578312#answer-578312
OPTIMISTIC_ABOUT_FILE_LOCKING=1

Running a Splunk docker container

Before we setup our docker-compose.yml, we'll run a Splunk container directly, as we need to check that it works, and also set up the HTTP Event Collector.

In order to use Splunk to capture our logs, splunk exposes the HTTP Event Collector service via an https endpoint, which we can ping with our log data.

Setting up the the HTTP Event Collector basically involves generating an authorization token in the Splunk instance, which can then be used to authenticate each request to send data to Splunk.

# Create the local folders
mkdir -p /opt/splunk/etc/
mkdir -p /opt/splunk/var/

# Run the Splunk docker container
docker run \
    -p 8000:8000 \
    -p 8088:8088 \
    -e 'SPLUNK_START_ARGS=--accept-license --no-prompt --answer-yes' \
    -e 'SPLUNK_USERNAME=admin' \
    -e 'SPLUNK_PASSWORD=CHANGEMENOW' \
    -v /opt/splunk/splunk-launch.conf:/opt/splunk/etc/splunk-launch.conf \
    -v /opt/splunk/etc:/opt/splunk/etc \
    -v /opt/splunk/var:/opt/splunk/var \
    splunk/splunk:latest

Presumably you'll want to change the password from CHANGEMENOW to something more appropriate -- note that Splunk enforces a password policy of a length of at least eight ASCII characters.

Given that this is a test, and given that Splunk takes some time to start up, I've not deamonised the above command, so I can follow the progress easily at the terminal. You'll now that Splunk is ready when you see the following:

Ansible playbook complete, will begin streaming var/log/splunk/splunkd_stderr.log

You should now be able to see the Splunk admin interface at http://localhost:8000:

splunk-admin

Setting up the HTTP Event Collector

We need to set up the HTTP Event Collector as per the Splunk documentation.

Once done, we should have a Token Value at http://localhost:8000/en-US/manager/search/http-eventcollector:

splunk-http-event-collector

We can test that this works by pinging the Splunk endpoint with some example data:

curl -ki  https://localhost:8088/services/collector/event \
    -H "Authorization: Splunk a6cb2c21-7dd1-4028-bbee-257f5a5e17db" \
    -d '{"event": "hello splunk"}'
    
HTTP/1.1 200 OK
Content-Length: 27
Content-Type: application/json; charset=UTF-8
Date: Sun, 24 Mar 2019 00:13:53 GMT
Server: Splunkd
Vary: Authorization
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN

{"text":"Success","code":0}

Note a couple of things about the call:

  • We need to use https
  • Because we probably don't have a valid cert, we need to specify the -k Curl option.

Traefik

As Splunk demands that we call the HTTP Event Collector using an https endpoint, we need to ensure that we set up the Splunk backend service in Traefik as HTTPS.

However, this leads to the problem that Traefik will not be able to call the Splunk backend service via HTTPS as it doesn't have access to the cert.

We could extract the certs from the Splunk container (they live at $SPLUNK_HOME/etc/auth/) and supply these to Traefik, but for a non-production set up, we can use the Traefik insecureSkipVerify global configuration value.

We'll create a local directory in which our traefik.toml will live:

mkdir -p /opt/etc/traefik
touch /opt/etc/traefik/traefik.toml

At a minimum, our traefik.toml will look something like:

################################################################
# Global configuration
################################################################

defaultEntryPoints = ["http", "https"]
insecureSkipVerify = true

################################################################
# Entrypoints configuration
################################################################

# Entrypoints definition
#
# Optional
# Default:
[entryPoints]
  # http should be redirected to https
  [entryPoints.http]
    address = ":80"
    [entryPoints.http.redirect]
      entryPoint = "https"

  # https is now the default
  [entryPoints.https]
    address = ":443"

  # Traefik status page
  [entryPoints.traefik]
     address = ":8080"
     [entryPoints.traefik.auth.basic]
        usersFile = "/usr/local/share/.auth"

# Enable ACME (Let's Encrypt): automatic SSL
# https://docs.traefik.io/user-guide/examples/#basic-example-with-http-challenge
[acme]
  storage = "/etc/traefik/acme.json"
  caServer = "https://acme-v02.api.letsencrypt.org/directory"
  entryPoint = "https"
  acmeLogging = true
  onDemand = false
  onHostRule = true
  [acme.tlsChallenge]

################################################################
# Traefik logs configuration
################################################################

[traefikLog]

filePath = "/var/log/traefik/traefik.log"

################################################################
# Access logs configuration
################################################################

[accessLog]

filePath = "/var/log/traefik/access.log"

docker-compose.yml

We'll assume the following environment variables:

BASIC_AUTH_USERNAME=admin
BASIC_AUTH_PASSWORD=whatever
DOMAIN=localhost.example.com
EMAIL=whoever@example.com
SPLUNK_USERNAME=admin
SPLUNK_PASSWORD=changemenow
CERT=example.com.pem
KEY=example.com-key.pem

We'll assume that we've added the following entries to our /etc/hosts file:

127.0.0.1	splunk.localhost.example.com
127.0.0.1	splunk-api.localhost.example.com

We'll also assume that we've generated a cert for *.localhost.example.com -- you can use something like mkcert to generate this cert and key, and store the results at /opt/etc/traefik/.

We'll add basic auth to Traefik by using a Dockerfile based on traefik:alpine:

# Dockerfile
FROM traefik:alpine

ARG BASIC_AUTH_USERNAME
ARG BASIC_AUTH_PASSWORD

ENV BASIC_AUTH_USERNAME $BASIC_AUTH_USERNAME
ENV BASIC_AUTH_PASSWORD $BASIC_AUTH_PASSWORD

RUN apk add --update apache2-utils \
  && rm -rf /var/cache/apk/* \
  && mkdir -p /var/log/traefik \
  && mkdir -p /etc/traefik \
  && touch /etc/traefik/acme.json \
  && chmod 0600 /etc/traefik/acme.json \
  && htpasswd -cBb /usr/local/share/.auth $BASIC_AUTH_USERNAME $BASIC_AUTH_PASSWORD

Our docker-compose.yml ends up looking like the following:

version: '2'

services:
  traefik:
    build:
      context: .
      args:
        - BASIC_AUTH_USERNAME=$BASIC_AUTH_USERNAME
        - BASIC_AUTH_PASSWORD=$BASIC_AUTH_PASSWORD
    container_name: "traefik.${DOMAIN}"
    restart: unless-stopped
    command: -c /dev/null --api --docker --logLevel=DEBUG --acme.email=$EMAIL \
      --configFile=/etc/traefik/traefik.toml \
      --entryPoints='Name:https Address::443 TLS:/etc/traefik/${CERT},/etc/traefik/${KEY}'
    ports:
      - 80:80
      - 443:443
      - 8025:8025
    expose:
      - 8080
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - /opt/etc/traefik:/etc/traefik
    labels:
      traefik.enable: "true"
      traefik.backend: traefik
      traefik.frontend.rule: "Host:traefik.${DOMAIN}"
      traefik.port: "8080"

  splunk:
    # https://www.splunk.com/blog/2018/10/24/announcing-splunk-on-docker.html
    image: splunk/splunk:latest
    container_name: "splunk.${DOMAIN}"
    environment:
      SPLUNK_START_ARGS: --accept-license --no-prompt --answer-yes
      SPLUNK_USERNAME: $SPLUNK_USERNAME
      SPLUNK_PASSWORD: $SPLUNK_PASSWORD
    restart: unless-stopped
    labels:
      # Use the following yaml formatting to allow our basic auth
      # variables to be properly escaped
      - "traefik.ui.backend=splunk"
      - "traefik.ui.port=8000"
      - "traefik.ui.frontend.rule=Host:splunk.${DOMAIN}"
      - 'traefik.ui.frontend.auth.basic=${BASIC_AUTH_USERNAME}:${BASIC_AUTH_PASSWORD_ENCRYPTED}'
      - "traefik.api.backend=splunk-api"
      - "traefik.api.protocol=https"
      - "traefik.api.port=8088"
      - "traefik.api.frontend.rule=Host:splunk-api.${DOMAIN}"
    volumes:
      # We need to override splunk-launch.conf on Mac OS Sierra:
      # https://answers.splunk.com/answers/306998/why-am-i-getting-homepathoptsplunkvarlibsplunkaudi.html?childToView=578312#answer-578312
      - /opt/splunk/splunk-launch.conf:/opt/splunk/etc/splunk-launch.conf
      - /opt/splunk/etc:/opt/splunk/etc
      - /opt/splunk/var:/opt/splunk/var
    expose:
      - 8000
      - 8088
    depends_on:
      - traefik

We should now be able to run up our stack. Again, we don't daemonise, so we can easily see the logs messages in the terminal, given that Splunk takes some time to start:

docker-compose up --build

We should then be able to see our admin at https://splunk.localhost.example.com.

We should also be able to ping the HTTP Event Collector:

curl -ki  https://splunk-api.localhost.example.com/services/collector/event -H "Authorization: Splunk a6cb2c21-7dd1-4028-bbee-257f5a5e17db" -d '{"event": "hello splunk"}'

HTTP/1.1 200 OK
Content-Length: 27
Content-Type: application/json; charset=UTF-8
Date: Sun, 24 Mar 2019 00:13:53 GMT
Server: Splunkd
Vary: Authorization
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN

{"text":"Success","code":0}

Using the Splunk service

In order to easily send logs to Splunk from docker containers, we'll want to install the docker-logging-plugin:

docker plugin install splunk/docker-logging-plugin:latest --alias splunk-logging-plugin
docker plugin enable splunk-logging-plugin

If we want another service in our docker-compose.yml to direct logs to Splunk, we'll need to ensure it contains an appropriate logging section, specifying the splunk-token and splunk-url we've set up previously:

    logging:
      driver: splunk-logging-plugin
      options:
        splunk-token: a6cb2c21-7dd1-4028-bbee-257f5a5e17db
        splunk-url: https://splunk-api.${DOMAIN}
        splunk-insecureskipverify: 'true'
        labels: whatever

And that should be all we need to do to get Splunk running with Traefik in Docker, at least on a local Mac OS machine.

A production Splunk instance will require significantly more setup than this though, as you'll want to ensure Splunk is secure and robust.