Deploying the Telegraf InfluxDB Grafana (TIG) Stack with Docker to Monitor a Linux Host

A screenshot of the "Host Dashboard" Grafana dashboard.

So I set up an old Dell Optiplex 7010 to be my home Linux development server and decided to use that as an opportunity to start learning Docker so I used that to deploy an IRC bouncer onto my home server.

Obviously, the next thing to do would be to deploy monitoring software so that I can monitor my server with a neat web interface just for fun.

Prerequisites

Before beginning, you’ll need to install Docker.

docker-compose.yaml

Since we’ve got three layers to our stack, we’re going to be dealing with three containers. I decided this would be a good time to learn about docker-compose as that makes it convenient to organize a multi-container setup very quickly.

version: '3'
services:
    grafana:
        container_name: grafana
        image: grafana/grafana
        ports:
            - '3000:3000'
        volumes:
            - grafana-storage:/var/lib/grafana
        links:
            - influxdb
    influxdb:
        container_name: influxdb
        image: influxdb:1.7.8
        ports:
            - "8083:8083"
            - "8086:8086"
        volumes:
            - influxdb-storage:/var/lib/influxdb
    telegraf:
        container_name: telegraf
        image: telegraf:1.12.3
        volumes:
            - /var/run/docker.sock:/var/run/docker.sock
            - /proc:/host/proc
            - ${PWD}/conf/telegraf/telegraf.conf:/etc/telegraf/telegraf.conf
volumes:
    influxdb-storage:
    grafana-storage:

First, here’s an explanation of some of the more common fields we see here:

  • container_name – this is the name of the container! We can use this to refer it with commands like docker restart grafana.
  • image – this is the Docker image we are creating a container from.
  • ports – this is the port combination used by our host and container respectively. When you see a port mapping like 8080:80, that means the host computer will route traffic on port 8080 to the container’s port 80.
  • volumes – what Docker volumes or bind mounts do we use? This is how we maintain persistent storage as containers are killed, removed, restarted, etc.
  • links – a way to make another container accessible by its alias, such as http://influxdb:8063, which will make more sense later
  • volumes – what, again? The top-level volumes field are the actual Docker volumes used by this composition.

First time setup

Telegraf

Telegraf will need a configuration file so it knows what inputs (system information) to send to what outputs (InfluxDB). Inside the folder where you put your docker-compose.yaml, let’s have Telegraf generate one for us:

$ mkdir -p conf/telegraf
$ docker exec telegraf:1.12.3 telegraf config > ./conf/telegraf/telegraf.conf

Now open telegraf.conf with your favorite text editor because we have some options to enable.

You’ll want to scroll down until you find the section labelled with [[outputs.influxdb]].

Do some uncommenting and editing surgery until you have something that resembles this:

[[outputs.influxdb]]
 urls = ["http://influxdb:8086"]
 database = "telegraf"
 skip_database_creation = true
 timeout = "5s"
 username = "telegraf"
 password = "YOURPASSWORDHEREFROMINFLUXDB"

  ## HTTP User-Agent
 user_agent = "telegraf"

  ## UDP payload size is the maximum packet size to send.
 udp_payload = "512B"

Launch the containers

Now, let’s bring the containers to life with docker-compose -d up.

InfluxDB

We need to create the database for Telegraf to write to and we also have to create a user account for Telegraf to use. The $ indicates a shell prompt and the > indicates the Influx database prompt.

$ docker exec -it influxdb influx
> create database telegraf
> create user telegraf with password 'YOURPASSWORD'

Grafana

In your web browser, navigate to grafana‘s address. If you’re doing this all on your local computer, it’ll be something like http://localhost:3000. Go ahead and sign in.

The default account you must use is admin and the password is admin. Change the password :-)

Go to Settings > Data Sources > Add Datasource. Then select “InfluxDB” from the listing you’re brought to.

Here are the fields you’ll have to configure:

  • ‘Name’ – I named my ‘InfluxDB’. You can name yours whatever you want.
  • ‘URL’ – http://influxdb:8086
  • ‘Database’ – We named the database ‘telegraf‘ during first time setup
  • ‘User’ – We also made the user account’s name ‘telegraf‘ for InfluxDB
  • ‘Password’ – Set this to the InfluxDB password you made.

Click Save and Test. You should get a green popup saying that the data source is working!

Adding a Dashboard

You can create your own dashboard if you want to. However, there are so many dials here that no one will blame you if you import one. You can import someone else’s dashboard if you go to Dashboards > Manage and click the “Import” button.

I imported the Host Dashboard by ichasco.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s