> ## Content Index > Fetch the complete content index at: https://blog.mebooks.co.nz/llms.txt > Use this file to discover other available public pages before exploring further. # Using Splunk with Traefik - URL: https://blog.mebooks.co.nz/using-splunk-with-traefik/ - Published: 2019-03-24T01:15:57.000Z - Updated: 2019-03-24T01:40:26.000Z - Author: Jason Darwin - Tags: #Import 2026-09-23 10:11 In this post we'll look at what it takes to set up Splunk with the Traefik reverse-proxy, such that we can send log events using Splunks HTTP Event Collector, and see and query the results in Splunk's admin dashboard. Splunk is a great tool for capturing and analysing application logs. Although Splunk is reasonably pricey, they do allow you to use Splunk Enterprise self-hosted for up to 500MB of data per day. Given that [Splunk Enterprise is available as a Docker image](https://docs.splunk.com/Documentation/Splunk/7.2.4/Installation/DeployandrunSplunkEnterpriseinsideDockercontainers?ref=blog.mebooks.co.nz), and I've been using [Traefik](https://traefik.io/?ref=blog.mebooks.co.nz) as a reverse proxy for my projects recently, I was curious if I could get the two to work nicely together. Looking for clues as to how others had done this in Google, I came up empty-handed, hence this post. ## Splunk configuration Given that I'm developing on a Mac (though using Linux in production), I ran into the [issue that Splunk doesn't yet have robust support for Mac OS High Sierra](https://answers.splunk.com/answers/306998/why-am-i-getting-homepathoptsplunkvarlibsplunkaudi.html?childToView=578312&ref=blog.mebooks.co.nz#answer-578312). Thankfully there's a workaround that involves adding the `OPTIMISTIC_ABOUT_FILE_LOCKING` setting to the `splunk-launch.conf`. Therefore, when we run Splunk, we'll share an edited version of the `splunk-launch.conf` via a volume. Set up the local directory containing an empty `splunk-launch.conf`: ``` mkdir -p /opt/splunk touch /opt/splunk/splunk-launch.conf ``` The `splunk-launch.conf` will look like: ``` # Version 7.2.4 # Modify the following line to suit the location of your Splunk install. # If unset, Splunk will use the parent of the directory containing the splunk # CLI executable. # # SPLUNK_HOME=/opt/splunk-home # By default, Splunk stores its indexes under SPLUNK_HOME in the # var/lib/splunk subdirectory. This can be overridden # here: # # SPLUNK_DB=/opt/splunk-home/var/lib/splunk # Splunkd daemon name SPLUNK_SERVER_NAME=Splunkd # Splunkweb daemon name SPLUNK_WEB_NAME=splunkweb # If SPLUNK_OS_USER is set, then Splunk service will only start # if the 'splunk [re]start [splunkd]' command is invoked by a user who # is, or can effectively become via setuid(2), $SPLUNK_OS_USER. # (This setting can be specified as username or as UID.) # # SPLUNK_OS_USER # https://answers.splunk.com/answers/306998/why-am-i-getting-homepathoptsplunkvarlibsplunkaudi.html?childToView=578312#answer-578312 OPTIMISTIC_ABOUT_FILE_LOCKING=1 ``` ## Running a Splunk docker container Before we setup our `docker-compose.yml`, we'll run a Splunk container directly, as we need to check that it works, and also set up the [HTTP Event Collector](https://docs.splunk.com/Documentation/Splunk/latest/Data/UsetheHTTPEventCollector?ref=blog.mebooks.co.nz). In order to use Splunk to capture our logs, splunk exposes the `HTTP Event Collector` service via an https endpoint, which we can ping with our log data. Setting up the the `HTTP Event Collector` basically involves generating an authorization token in the Splunk instance, which can then be used to authenticate each request to send data to Splunk. ``` # Create the local folders mkdir -p /opt/splunk/etc/ mkdir -p /opt/splunk/var/ # Run the Splunk docker container docker run \ -p 8000:8000 \ -p 8088:8088 \ -e 'SPLUNK_START_ARGS=--accept-license --no-prompt --answer-yes' \ -e 'SPLUNK_USERNAME=admin' \ -e 'SPLUNK_PASSWORD=CHANGEMENOW' \ -v /opt/splunk/splunk-launch.conf:/opt/splunk/etc/splunk-launch.conf \ -v /opt/splunk/etc:/opt/splunk/etc \ -v /opt/splunk/var:/opt/splunk/var \ splunk/splunk:latest ``` Presumably you'll want to change the password from `CHANGEMENOW` to something more appropriate -- note that Splunk enforces a password policy of a length of at least eight ASCII characters. Given that this is a test, and given that Splunk takes some time to start up, I've not deamonised the above command, so I can follow the progress easily at the terminal. You'll now that Splunk is ready when you see the following: ``` Ansible playbook complete, will begin streaming var/log/splunk/splunkd_stderr.log ``` You should now be able to see the Splunk admin interface at [http://localhost:8000](http://localhost:8000/?ref=blog.mebooks.co.nz): ![splunk-admin](https://blog.mebooks.co.nz/content/images/2019/03/splunk-admin.png) ## Setting up the HTTP Event Collector We need to [set up the HTTP Event Collector as per the Splunk documentation](https://docs.splunk.com/Documentation/Splunk/latest/Data/UsetheHTTPEventCollector?ref=blog.mebooks.co.nz). Once done, we should have a `Token Value` at [http://localhost:8000/en-US/manager/search/http-eventcollector:](http://localhost:8000/en-US/manager/search/http-eventcollector:?ref=blog.mebooks.co.nz) ![splunk-http-event-collector](https://blog.mebooks.co.nz/content/images/2019/03/splunk-http-event-collector.png) We can test that this works by pinging the Splunk endpoint with some example data: ``` curl -ki https://localhost:8088/services/collector/event \ -H "Authorization: Splunk a6cb2c21-7dd1-4028-bbee-257f5a5e17db" \ -d '{"event": "hello splunk"}' HTTP/1.1 200 OK Content-Length: 27 Content-Type: application/json; charset=UTF-8 Date: Sun, 24 Mar 2019 00:13:53 GMT Server: Splunkd Vary: Authorization X-Content-Type-Options: nosniff X-Frame-Options: SAMEORIGIN {"text":"Success","code":0} ``` Note a couple of things about the call: - We need to use `https` - Because we probably don't have a valid cert, we need to specify the `-k` Curl option. ## Traefik As Splunk demands that we call the HTTP Event Collector using an https endpoint, we need to ensure that we set up the Splunk backend service in Traefik as HTTPS. However, this leads to the problem that Traefik will not be able to call the Splunk backend service via HTTPS as it doesn't have access to the cert. We could extract the certs from the Splunk container (they live at `$SPLUNK_HOME/etc/auth/`) and supply these to Traefik, but for a non-production set up, we can use the Traefik `insecureSkipVerify` global configuration value. We'll create a local directory in which our `traefik.toml` will live: ``` mkdir -p /opt/etc/traefik touch /opt/etc/traefik/traefik.toml ``` At a minimum, our `traefik.toml` will look something like: ``` ################################################################ # Global configuration ################################################################ defaultEntryPoints = ["http", "https"] insecureSkipVerify = true ################################################################ # Entrypoints configuration ################################################################ # Entrypoints definition # # Optional # Default: [entryPoints] # http should be redirected to https [entryPoints.http] address = ":80" [entryPoints.http.redirect] entryPoint = "https" # https is now the default [entryPoints.https] address = ":443" # Traefik status page [entryPoints.traefik] address = ":8080" [entryPoints.traefik.auth.basic] usersFile = "/usr/local/share/.auth" # Enable ACME (Let's Encrypt): automatic SSL # https://docs.traefik.io/user-guide/examples/#basic-example-with-http-challenge [acme] storage = "/etc/traefik/acme.json" caServer = "https://acme-v02.api.letsencrypt.org/directory" entryPoint = "https" acmeLogging = true onDemand = false onHostRule = true [acme.tlsChallenge] ################################################################ # Traefik logs configuration ################################################################ [traefikLog] filePath = "/var/log/traefik/traefik.log" ################################################################ # Access logs configuration ################################################################ [accessLog] filePath = "/var/log/traefik/access.log" ``` ## docker-compose.yml We'll assume the following environment variables: ``` BASIC_AUTH_USERNAME=admin BASIC_AUTH_PASSWORD=whatever DOMAIN=localhost.example.com EMAIL=whoever@example.com SPLUNK_USERNAME=admin SPLUNK_PASSWORD=changemenow CERT=example.com.pem KEY=example.com-key.pem ``` We'll assume that we've added the following entries to our `/etc/hosts` file: ``` 127.0.0.1 splunk.localhost.example.com 127.0.0.1 splunk-api.localhost.example.com ``` We'll also assume that we've generated a cert for `*.localhost.example.com` \-- you can use something like [mkcert](https://blog.filippo.io/mkcert-valid-https-certificates-for-localhost/?ref=blog.mebooks.co.nz) to generate this cert and key, and store the results at `/opt/etc/traefik/`. We'll add basic auth to Traefik by using a `Dockerfile` based on `traefik:alpine`: ``` # Dockerfile FROM traefik:alpine ARG BASIC_AUTH_USERNAME ARG BASIC_AUTH_PASSWORD ENV BASIC_AUTH_USERNAME $BASIC_AUTH_USERNAME ENV BASIC_AUTH_PASSWORD $BASIC_AUTH_PASSWORD RUN apk add --update apache2-utils \ && rm -rf /var/cache/apk/* \ && mkdir -p /var/log/traefik \ && mkdir -p /etc/traefik \ && touch /etc/traefik/acme.json \ && chmod 0600 /etc/traefik/acme.json \ && htpasswd -cBb /usr/local/share/.auth $BASIC_AUTH_USERNAME $BASIC_AUTH_PASSWORD ``` Our `docker-compose.yml` ends up looking like the following: ``` version: '2' services: traefik: build: context: . args: - BASIC_AUTH_USERNAME=$BASIC_AUTH_USERNAME - BASIC_AUTH_PASSWORD=$BASIC_AUTH_PASSWORD container_name: "traefik.${DOMAIN}" restart: unless-stopped command: -c /dev/null --api --docker --logLevel=DEBUG --acme.email=$EMAIL \ --configFile=/etc/traefik/traefik.toml \ --entryPoints='Name:https Address::443 TLS:/etc/traefik/${CERT},/etc/traefik/${KEY}' ports: - 80:80 - 443:443 - 8025:8025 expose: - 8080 volumes: - /var/run/docker.sock:/var/run/docker.sock - /opt/etc/traefik:/etc/traefik labels: traefik.enable: "true" traefik.backend: traefik traefik.frontend.rule: "Host:traefik.${DOMAIN}" traefik.port: "8080" splunk: # https://www.splunk.com/blog/2018/10/24/announcing-splunk-on-docker.html image: splunk/splunk:latest container_name: "splunk.${DOMAIN}" environment: SPLUNK_START_ARGS: --accept-license --no-prompt --answer-yes SPLUNK_USERNAME: $SPLUNK_USERNAME SPLUNK_PASSWORD: $SPLUNK_PASSWORD restart: unless-stopped labels: # Use the following yaml formatting to allow our basic auth # variables to be properly escaped - "traefik.ui.backend=splunk" - "traefik.ui.port=8000" - "traefik.ui.frontend.rule=Host:splunk.${DOMAIN}" - 'traefik.ui.frontend.auth.basic=${BASIC_AUTH_USERNAME}:${BASIC_AUTH_PASSWORD_ENCRYPTED}' - "traefik.api.backend=splunk-api" - "traefik.api.protocol=https" - "traefik.api.port=8088" - "traefik.api.frontend.rule=Host:splunk-api.${DOMAIN}" volumes: # We need to override splunk-launch.conf on Mac OS Sierra: # https://answers.splunk.com/answers/306998/why-am-i-getting-homepathoptsplunkvarlibsplunkaudi.html?childToView=578312#answer-578312 - /opt/splunk/splunk-launch.conf:/opt/splunk/etc/splunk-launch.conf - /opt/splunk/etc:/opt/splunk/etc - /opt/splunk/var:/opt/splunk/var expose: - 8000 - 8088 depends_on: - traefik ``` We should now be able to run up our stack. Again, we don't daemonise, so we can easily see the logs messages in the terminal, given that Splunk takes some time to start: ``` docker-compose up --build ``` We should then be able to see our admin at [https://splunk.localhost.example.com](https://splunk.localhost.example.com/?ref=blog.mebooks.co.nz). We should also be able to ping the HTTP Event Collector: ``` curl -ki https://splunk-api.localhost.example.com/services/collector/event -H "Authorization: Splunk a6cb2c21-7dd1-4028-bbee-257f5a5e17db" -d '{"event": "hello splunk"}' HTTP/1.1 200 OK Content-Length: 27 Content-Type: application/json; charset=UTF-8 Date: Sun, 24 Mar 2019 00:13:53 GMT Server: Splunkd Vary: Authorization X-Content-Type-Options: nosniff X-Frame-Options: SAMEORIGIN {"text":"Success","code":0} ``` ## Using the Splunk service In order to easily send logs to Splunk from docker containers, we'll want to install the [docker-logging-plugin](https://github.com/splunk/docker-logging-plugin?ref=blog.mebooks.co.nz#install-the-plugin-from-docker-store): ``` docker plugin install splunk/docker-logging-plugin:latest --alias splunk-logging-plugin docker plugin enable splunk-logging-plugin ``` If we want another service in our `docker-compose.yml` to direct logs to Splunk, we'll need to ensure it contains an appropriate `logging` section, specifying the `splunk-token` and `splunk-url` we've set up previously: ``` logging: driver: splunk-logging-plugin options: splunk-token: a6cb2c21-7dd1-4028-bbee-257f5a5e17db splunk-url: https://splunk-api.${DOMAIN} splunk-insecureskipverify: 'true' labels: whatever ``` And that should be all we need to do to get Splunk running with Traefik in Docker, at least on a local Mac OS machine. A production Splunk instance will require significantly more setup than this though, as you'll want to ensure Splunk is secure and robust.