Monitoring Tungsten Clusters Using Prometheus

The below example procedure is designed to help you get Prometheus installed and working with the goal of monitoring Tungsten Clusters through the Dashboard.

This section of the documentation is a summary guide for how to install an external software product, Prometheus. The usual caveats apply, and as always, your mileage may vary.

For more information about getting started with Prometheus, please visit the Prometheus website

Example Prometheus Installation Procedure

First, download the tarball from https://prometheus.io/download/

Next, go to the install directory, normally /usr/local, and extract the tarball. Complete the Prometheus software installation by creating a symbolic link for convenience when upgrading.

shell> cd /usr/local
shell> sudo tar xvzf {tarball_fullpath_here}
shell> sudo ln -s {extracted_dir} prometheus

In this step, create the Prometheus user and directories you will need, along with setting proper ownership. Be sure to modify the examples to match your environment.

shell> sudo useradd -rs /bin/false prometheus
shell> sudo mkdir -p ~prometheus/data/
shell> sudo mkdir -p ~prometheus/var/
shell> sudo touch ~prometheus/var/prometheus.log
shell> sudo chown -R prometheus: /usr/local/prometheus* ~prometheus

Example Prometheus Configuration Procedure

The below example shows three (3) 3-node clusters for a total of nine (9) nodes.

Each node has 5 available exporters (name:port) - node:9400, mysqld:9404, replicator:8091, manager:8092 and connector:8093.

Create a new or edit the existing Prometheus configuration file, normally /usr/local/prometheus/prometheus.yml, and adjust the file to match your specific needs.

shell> sudo vi /usr/local/prometheus/prometheus.yml
# sample config for monitoring Tungsten Clusters
global:
  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

# Alertmanager configuration
alerting:
  alertmanagers:
  - static_configs:
    - targets:
      # - alertmanager:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
# The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
- job_name: 'prometheus'

# metrics_path defaults to '/metrics'
# scheme defaults to 'http'.

static_configs:
- targets: ['localhost:9090']

- job_name: 'node'
scrape_interval: 5s
static_configs:
  - targets: ['db1:9100','db2:9100','db3:9100','db4:9100','db5:9100','db6:9100','db7:9100','db8:9100','db9:9100']

- job_name: 'mysqld'
scrape_interval: 5s
static_configs:
  - targets: ['db1:9104','db2:9104','db3:9104','db4:9104','db5:9104','db6:9104','db7:9104','db8:9104','db9:9104']


- job_name: 'tungsten_replicator'
scrape_interval: 5s
static_configs:
- targets: ["db1:8091", "db2:8091", "db3:8091", "db4:8091", "db5:8091", "db6:8091",'db7:8091','db8:8091','db9:8091']

- job_name: 'tungsten_manager'
scrape_interval: 5s
static_configs:
- targets: ["db1:8092", "db2:8092", "db3:8092", "db4:8092", "db5:8092", "db6:8092",'db7:8092','db8:8092','db9:8092']

- job_name: 'tungsten_connector'
scrape_interval: 5s
static_configs:
- targets: ["db1:8093", "db2:8093", "db3:8093", "db4:8093", "db5:8093", "db6:8093",'db7:8093','db8:8093','db9:8093']

Example Prometheus Boot Configuration Procedures

• init.d-based procedure  

  Create the prometheus boot script for init.d:

  shell> sudo vi /etc/init.d/prometheus
  #!/bin/bash
  #
  # /etc/rc.d/init.d/prometheus
  #
  # Prometheus monitoring server
  #
  #  chkconfig: 2345 20 80 Read
  #  description: Prometheus monitoring server
  #  processname: prometheus
  
  # Source function library.
  . /etc/rc.d/init.d/functions
  
  PROGNAME=prometheus
  RETENTION=3d
  HOMEDIR="/home"
  INSTALLDIR="/usr/local"
  
  PROG=$INSTALLDIR/$PROGNAME/$PROGNAME
  CONFIG_FILE=$INSTALLDIR/$PROGNAME/$PROGNAME.yml
  
  USER=$PROGNAME
  DATADIR=$HOMEDIR/$USER/data
  LOGFILE=$HOMEDIR/$USER/var/$PROGNAME.log
  LOCKFILE=$HOMEDIR/$USER/var/$PROGNAME.pid
  
  start() {
  echo -n "Starting $PROGNAME: "
  daemon --user $USER --pidfile="$LOCKFILE" "$PROG --config.file=$CONFIG_FILE --storage.tsdb.path=$DATADIR --storage.tsdb.retention=$RETENTION --web.enable-admin-api &>$LOGFILE &"
  echo $(pidofproc $PROGNAME) >$LOCKFILE
  echo
  }
  
  stop() {
  echo -n "Shutting down $PROGNAME: "
  killproc $PROGNAME
  rm -f $LOCKFILE
  echo
  }
  
  
  case "$1" in
  start)
  start
  ;;
  stop)
  stop
  ;;
  status)
  status $PROGNAME
  ;;
  restart)
  stop
  start
  ;;
  reload)
  echo "Sending SIGHUP to $PROGNAME"
  kill -SIGHUP $(pidofproc $PROGNAME)
  ;;
  *)
      echo "Usage: <servicename> {start|stop|status|reload|restart}"
      exit 1
  ;;
  esac

  Enable the prometheus service to start at boot time via chkconfig, and then start it using service:

  shell> sudo chkconfig --add prometheus
  shell> sudo chkconfig --list | grep prometheus
  shell> sudo service prometheus start
  shell> sudo service prometheus status

• systemd-based procedure

  Create the prometheus.service boot script for systemd:

  shell> sudo vi /etc/systemd/system/prometheus.service
  [Unit]
  Description=Prometheus
  After=network.target
  
  [Service]
  User=prometheus
  Group=prometheus
  Type=simple
  ExecStart=/usr/local/bin/prometheus --config.file=/usr/local/prometheus/prometheus.yml --storage.tsdb.path=/home/prometheus/data --web.enable-admin-api
  
  [Install]
  WantedBy=multi-user.target
  ]]>
  
  </CONSOLE>
  
          <p>
            Use <command>systemctl</command> to reload the boot config, enable
            the <code>prometheus</code> service to start at boot time,
            and then start Prometheus:
          </p>
  
  <CONSOLE interpolate showLineNumbers language="bash">
  shell> sudo systemctl daemon-reload
  shell> sudo systemctl enable prometheus
  shell> sudo systemctl start prometheus

Example Prometheus Test Procedure

Once the Prometheus server has been started, you may test that it is running via browser URL http://{yourServer}:9090/graph

Prometheus may now be enabled in the &dashboard_name; one of two ways, either via the browser Dashboard settings panel or manually by editing the config.json file in the Dashboard WEBROOT directory. Add the configuration option "enablePrometheus":1 and refresh the Dashboard page in the browser to see the additional button in the top navigation bar.

For more information about next steps with Prometheus, please visit the Prometheus website