Chapter 15. Monitoring Tungsten Clusters Using Prometheus and Grafana
Prev			Next

Chapter 15. Monitoring Tungsten Clusters Using Prometheus and Grafana

Table of Contents

15.1. Monitoring Tungsten Clusters Using Prometheus

15.1.1. Example Prometheus Installation Procedure
15.1.2. Example Prometheus Configuration Procedure
15.1.3. Example Prometheus Boot Configuration Procedures
15.1.4. Example Prometheus Test Procedure

15.2. Monitoring Tungsten Clusters Using Grafana

15.2.1. Example Grafana Installation Procedure
15.2.2. Example Grafana Configuration Procedure
15.2.3. Example Grafana Boot Configuration Procedure
15.2.4. Example Grafana Test Procedure
15.2.5. Example Grafana Setup and Usage

Tungsten Dashboard has introduced basic support for using Prometheus and Grafana to monitor Tungsten Clusters. As of Tungsten Clustering software v6.1.4, key Prometheus exporters have been added to the distribution. These exporters will allow a Prometheus server to gather metrics for:

the MySQL server and the underlying node "hardware" using external binaries added to the distribution
the Tungsten Manager, Replicator and Connector using new built-in functionality

A new script has been included to assist with the management and testing of the exporters called tmonitor.

To learn more about the tmonitor command and the included exporters, please visit ??? for more information.

IMPORTANT: To get the most benefit out of the exporters along with ensuring both ease of configuration and security, Continuent requires that both the Prometheus and Grafana servers be installed onto the same instance hosting the Dashboard web server when using the Prometheus and Grafana integration with Dashboard.

15.1. Monitoring Tungsten Clusters Using Prometheus

15.1.1. Example Prometheus Installation Procedure
15.1.2. Example Prometheus Configuration Procedure
15.1.3. Example Prometheus Boot Configuration Procedures
15.1.4. Example Prometheus Test Procedure

The below example procedure is designed to help you get Prometheus installed and working with the goal of monitoring Tungsten Clusters through the Dashboard.

This section of the documentation is a summary guide for how to install an external software product, Prometheus. The usual caveats apply, and as always, your mileage may vary.

For more information about getting started with Prometheus, please visit the Prometheus website at https://prometheus.io/docs/introduction/first_steps/

15.1.1. Example Prometheus Installation Procedure

First, download the tarball from https://prometheus.io/download/

Next, go to the install directory, normally /usr/local, and extract the tarball. Complete the Prometheus software installation by creating a symbolic link for convenience when upgrading.

shell> cd /usr/local
shell> sudo tar xvzf {tarball_fullpath_here}
shell> sudo ln -s {extracted_dir} prometheus

In this step, create the Prometheus user and directories you will need, along with setting proper ownership. Be sure to modify the examples to match your environment.

shell> sudo useradd -rs /bin/false prometheus
shell> sudo mkdir -p ~prometheus/data/
shell> sudo mkdir -p ~prometheus/var/
shell> sudo touch ~prometheus/var/prometheus.log
shell> sudo chown -R prometheus: /usr/local/prometheus* ~prometheus

15.1.2. Example Prometheus Configuration Procedure

The below example shows three (3) 3-node clusters for a total of nine (9) nodes.

Each node has 5 available exporters (name:port) - node:9400, mysqld:9404, replicator:8091, manager:8092 and connector:8093.

Create anew or edit the existing Prometheus configuration file, normally /usr/local/prometheus/prometheus.yml, and adjust the file to match your specific needs.

shell> sudo vi /usr/local/prometheus/prometheus.yml

# sample config for monitoring Tungsten Clusters
global:
  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

# Alertmanager configuration
alerting:
  alertmanagers:
  - static_configs:
    - targets:
      # - alertmanager:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
    - targets: ['localhost:9090']

  - job_name: 'node'
    scrape_interval: 5s
    static_configs:
      - targets: ['db1:9100','db2:9100','db3:9100','db4:9100','db5:9100','db6:9100','db7:9100','db8:9100','db9:9100']

  - job_name: 'mysqld'
    scrape_interval: 5s
    static_configs:
      - targets: ['db1:9104','db2:9104','db3:9104','db4:9104','db5:9104','db6:9104','db7:9104','db8:9104','db9:9104']


  - job_name: 'tungsten_replicator'
    scrape_interval: 5s
    static_configs:
    - targets: ["db1:8091", "db2:8091", "db3:8091", "db4:8091", "db5:8091", "db6:8091",'db7:8091','db8:8091','db9:8091']

  - job_name: 'tungsten_manager'
    scrape_interval: 5s
    static_configs:
    - targets: ["db1:8092", "db2:8092", "db3:8092", "db4:8092", "db5:8092", "db6:8092",'db7:8092','db8:8092','db9:8092']

  - job_name: 'tungsten_connector'
    scrape_interval: 5s
    static_configs:
    - targets: ["db1:8093", "db2:8093", "db3:8093", "db4:8093", "db5:8093", "db6:8093",'db7:8093','db8:8093','db9:8093']

15.1.3. Example Prometheus Boot Configuration Procedures

init.d-based procedure

Create the prometheus boot script for init.d:

shell> sudo vi /etc/init.d/prometheus

#!/bin/bash
#
# /etc/rc.d/init.d/prometheus
#
# Prometheus monitoring server
#
#  chkconfig: 2345 20 80 Read
#  description: Prometheus monitoring server
#  processname: prometheus

# Source function library.
. /etc/rc.d/init.d/functions

PROGNAME=prometheus
RETENTION=3d
HOMEDIR="/home"
INSTALLDIR="/usr/local"

PROG=$INSTALLDIR/$PROGNAME/$PROGNAME
CONFIG_FILE=$INSTALLDIR/$PROGNAME/$PROGNAME.yml

USER=$PROGNAME
DATADIR=$HOMEDIR/$USER/data
LOGFILE=$HOMEDIR/$USER/var/$PROGNAME.log
LOCKFILE=$HOMEDIR/$USER/var/$PROGNAME.pid

start() {
    echo -n "Starting $PROGNAME: "
    daemon --user $USER --pidfile="$LOCKFILE" "$PROG --config.file=$CONFIG_FILE --storage.tsdb.path=$DATADIR --storage.tsdb.retention=$RETENTION --web.enable-admin-api &>$LOGFILE &"
    echo $(pidofproc $PROGNAME) >$LOCKFILE
    echo
}

stop() {
    echo -n "Shutting down $PROGNAME: "
    killproc $PROGNAME
    rm -f $LOCKFILE
    echo
}


case "$1" in
    start)
    start
    ;;
    stop)
    stop
    ;;
    status)
    status $PROGNAME
    ;;
    restart)
    stop
    start
    ;;
    reload)
    echo "Sending SIGHUP to $PROGNAME"
    kill -SIGHUP $(pidofproc $PROGNAME)
    ;;
    *)
        echo "Usage: <servicename> {start|stop|status|reload|restart}"
        exit 1
    ;;
esac

Enable the prometheus service to start at boot time via chkconfig, and then start it using service:

shell> sudo chkconfig --add prometheus
shell> sudo chkconfig --list | grep prometheus
shell> sudo service prometheus start
shell> sudo service prometheus status

systemd-based procedure

Create the prometheus.service boot script for systemd:

shell> sudo vi /etc/systemd/system/prometheus.service

[Unit]
Description=Prometheus
After=network.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
    --config.file=/usr/local/prometheus/prometheus.yml \
    --storage.tsdb.path=/home/prometheus/data \
    --web.enable-admin-api

[Install]
WantedBy=multi-user.target

Use systemctl to reload the boot config, enable the prometheus service to start at boot time, and then start Prometheus:

shell> sudo systemctl daemon-reload
shell> sudo systemctl enable prometheus
shell> sudo systemctl start prometheus

15.1.4. Example Prometheus Test Procedure

Once the Prometheus server has been started, you may test that it is running via browser URL http://{yourServer}:9090/graph

Prometheus may now be enabled in the Tungsten Dashboard one of two ways, either via the browser Dashboard settings panel or manually by editing the config.json file in the Dashboard WEBROOT directory. Add the configuration option "enablePrometheus":1 and refresh the Dashboard page in the browser to see the additional button in the top navigation bar.

For more information about next steps with Prometheus, please visit the Prometheus website at https://prometheus.io/docs/introduction/first_steps/

15.2. Monitoring Tungsten Clusters Using Grafana

15.2.1. Example Grafana Installation Procedure
15.2.2. Example Grafana Configuration Procedure
15.2.3. Example Grafana Boot Configuration Procedure
15.2.4. Example Grafana Test Procedure
15.2.5. Example Grafana Setup and Usage

The below example procedure is designed to help you get Grafana installed and working with the goal of monitoring Tungsten Clusters through the Dashboard.

This section of the documentation is a summary guide for how to install an external software product, Grafana. The usual caveats apply, and as always, your mileage may vary.

For more information about getting started with Grafana, please visit the Grafana website at https://grafana.com/docs/grafana/latest/guides/getting_started/

15.2.1. Example Grafana Installation Procedure

This procedure example uses the YUM-based method. For other ways to install Grafana, please visit the Grafana install page at https://grafana.com/docs/grafana/latest/installation/

First, create the YUM repository configuration file for Grafana:

shell> sudo vi /etc/yum.repos.d/grafana.repo

[grafana]
name=grafana
baseurl=https://packages.grafana.com/oss/rpm
repo_gpgcheck=1
enabled=1
gpgcheck=1
gpgkey=https://packages.grafana.com/gpg.key
sslverify=1
sslcacert=/etc/pki/tls/certs/ca-bundle.crt

Install Grafana using yum:

shell> sudo yum install grafana

15.2.2. Example Grafana Configuration Procedure

REQUIRED STEP - Configure Embedding

In order to use the Grafana integration with the Tungsten Dashboard, one line needs to be added to the [security] stanza in the Grafana configuration file (normally /etc/grafana/grafana.ini). This setting is usually commented out and set to false, so just add a new line under the commented one:

shell> sudo vi /etc/grafana/grafana.ini

...
[security]
;allow_embedding = false
allow_embedding = true
...

OPTIONAL STEP - Configure Anonymous Auth

The embedded Grafana panel will require a login and password. To disable this requirement, and allow the panels to be shown without authentication via limited read-only access, add two lines under the [auth.anonymous] stanza in the Grafana configuration file (normally /etc/grafana/grafana.ini). These settings are usually commented out, so just add the new lines under the commented ones:

shell> sudo vi /etc/grafana/grafana.ini

...
[auth.anonymous]
;enabled = false
enabled = true

;org_name = Main Org.
org_name = {Your Exact Organization Name In Grafana}
...

OPTIONAL STEP - Configure HTTPS

The embedded Grafana panel will be called on whatever transport is used to load the main Tungsten Dashboard page, i.e. http or https. To configure Grafana to use https, add four lines to the [server] stanza in the Grafana configuration file (normally /etc/grafana/grafana.ini). These settings are usually commented out, so just add the new lines under the commented ones:

shell> sudo vi /etc/grafana/grafana.ini

...
[server]
;protocol = http
protocol = https

;domain = localhost
domain = dashboard.yourdomain.com

;cert_file =
cert_file = /etc/letsencrypt/archive/dashboard.yourdomain.com/fullchain1.pem

;cert_key =
cert_key = /etc/letsencrypt/archive/dashboard.yourdomain.com/privkey1.pem
...

Important

It is critical that the domain = value be the same FQDN as the one the Tungsten Dashboard web service answers to, and that the same keys are in use.

Our example shows Let's Encrypt certificates that are shared with the Dashboard web server instance. For this to work, the permissions of the key files must allow for Grafana to access them. Velow is an example of how you could allow access to the needed certificate files:

shell> sudo chgrp grafana /etc/letsencrypt/archive/dashboard.yourdomain.com/privkey1.pem
shell> sudo chmod g+r /etc/letsencrypt/archive/dashboard.yourdomain.com/privkey1.pem

Please remember to restart Grafana when the certificates expire and get renewed!

Below is a sample Let's Encrypt command to get a cert for dashboard.yourdomain.com, assuming that there is real DNS for that domain, that it resolves for the world, and that web server is reachable by the world:

shell> certbot certonly \
--webroot \
--renew-by-default \
--agree-tos \
-v \
--debug \
--email you@yourdomain.com \
-w /volumes/data/tungsten/html \
-d dashboard.yourdomain.com \
--dry-run

Please remember to remove the --dry-run argument at the end and re-run to get the real certs!

All examples provided for convenience. As always, YMMV, and supporting Grafana and/or certificates is outside Continuent's scope.

15.2.3. Example Grafana Boot Configuration Procedure

The YUM-based install automatically creates the grafana user, along with the systemd and init.d boot scripts. This means you do not have to create the boot scripts by hand!

init.d-based procedure

Enable the grafana-server service to start at boot time via chkconfig, and then start it using service:

shell> sudo chkconfig --add grafana-server
shell> sudo chkconfig --list | grep grafana-server
shell> sudo service grafana-server start
shell> sudo service grafana-server status

systemd-based procedure
Use systemctl to reload the boot config, enable the grafana-server service to start at boot time, and then start Grafana:
```
shell> sudo systemctl daemon-reload
shell> sudo systemctl enable grafana-server
shell> sudo systemctl start grafana-server
```

15.2.4. Example Grafana Test Procedure

Once the Grafana server has been started, you may test that it is running via browser URL http://{yourServer}:3000

Grafana may now be added to the Dashboard via the config.json file in the Dashboard WEBROOT directory. Add the configuration option "enableGrafana":1 and refresh the Dashboard page in the browser to see the additional button in the top navigation bar.

For more information about next steps with Grafana, please visit the Grafana website at https://grafana.com/docs/grafana/latest/guides/getting_started/

15.2.5. Example Grafana Setup and Usage

Once logged into the Grafana server as admin, you may configure a data source and import the dashboards.

Create a data source using Prometheus
Click the Configuration cog on the left nav bar, then click "Add data source", Choose prometheus, then add 'http://localhost:9090' to the HTTP URL field, then click Save & Test at the bottom. This should create a new data source named 'Prometheus for use a few steps below.
Optional Step: Import the Included Prometheus and Grafana Dashboards
If you want to use the built-in metrics for Prometheus and/or Grafana, import the included dashboards as desired.
Click on the Dashboards Tab in the center window to the right of the Settings Tab, then click on the blue Import button for each of Prometheus Stats, Prometheus 2.0 Stats and Grafana metrics.
Import the Continuent Tungsten Dashboard
Hover over the Dashboards icon in the left nav bar, then select Manage from the sub-menu. Click the Import link to the right of the green New Dashboard button.
In the Grafana.com Dashboard field, enter 12760 for the Continuent Tungsten dashboard, then click Load.
Select the Prometheus data source, then click the green Import button.
If you have Prometheus setup correctly and running, you should see results instantly.
Save this Dashboard by clicking the 3.5 inch floppy icon in the upper-right corner, then click the green Save button.
Click the star in the upper-right corner to make this dashboard a favorite. This makes finding the dashboard MUCH easier.
Import the Node Exporter Full Dashboard
Hover over the Dashboards icon in the left nav bar, then select Manage from the sub-menu. Click the Import link to the right of the green New Dashboard button.
In the Grafana.com Dashboard field, enter 1860 for the Node Exporter Full dashboard, then click Load.
Select the Prometheus data source, then click the green Import button.
If you have Prometheus setup correctly and running, you should see results instantly.
Save this Dashboard by clicking the 3.5 inch floppy icon in the upper-right corner, then click the green Save button.
Click the star in the upper-right corner to make this dashboard a favorite. This makes finding the dashboard MUCH easier.
Import the Percona MySQL Dashboard
Hover over the Dashboards icon in the left nav bar, then select Manage from the sub-menu. Click the Import link to the right of the green New Dashboard button.
In the Grafana.com Dashboard field, enter 7362 for the Percona MySQL dashboard, then click Load.
Select the Prometheus data source, then click the green Import button.
If you have Prometheus setup correctly and running, you should see results instantly.
Save this Dashboard by clicking the 3.5 inch floppy icon in the upper-right corner, then click the green Save button.
Click the star in the upper-right corner to make this dashboard a favorite. This makes finding the dashboard MUCH easier.