10.5.2.6. tpm cert: Getting Started - Functional Database Cert Rotation Example

The philosophy here is that the cert rotation work is done on a single cluster node. We will call this the "work node".

The secret to getting certs to work with a cluster is to make sure that you copy all of the MySQL database certs and Tungsten security files from the work node to the rest of the nodes at the correct point in the process. If all the cluster and database cert files are not the same across all nodes, the cluster will fail.

In the following functional example, we demonstrate the actual steps to rotate the database certs in a standalone 5-node Tungsten cluster.

When doing a database cert rotation, multiple files must be regenerated:

  • Five (5) required MySQL security files (may be more created than needed):

    1. ca.pem

    2. client-cert.pem

    3. client-key.pem

    4. server-cert.pem

    5. server-key.pem

  • One (1) .p12 file to represent the database client cert:

    1. client-cert.p12

  • Four (4) Tungsten-specific files:

    1. tungsten_keystore.jks

    2. tungsten_truststore.ts

    3. tungsten_truststore.ts

    4. tungsten_connector_truststore.ts

Next, confirm that both MySQL and Tungsten are configured to use the proper files:

  • /etc/my.cnf entries:

    [mysqld]
    ssl-ca=/etc/mysql/certs/ca.pem
    ssl-cert=/etc/mysql/certs/server-cert.pem
    ssl-key=/etc/mysql/certs/server-key.pem
    require_secure_transport=ON

    Important

    This example assumes that the database certs are located in /etc/mysql/certs on all cluster database nodes.

  • /etc/tungsten/tungsten.ini entries:

    datasource-mysql-ssl-ca=/etc/mysql/certs/ca.pem
    datasource-mysql-ssl-cert=/etc/mysql/certs/client-cert.pem
    datasource-mysql-ssl-key=/etc/mysql/certs/client-key.pem

    Important

    This example assumes that no /opt/continuent/share/tungsten.env file exists.

*** FUNCTIONAL PROCEDURE EXAMPLE STARTS HERE ***

** On a single node (db1) as the tungsten OS user **
  • Backup the old database certs from /etc/mysql/certs/ to /opt/continuent/backups/:

    tpm cert backup mysql
  • Clean out any old database certs so new certs are generated:

    sudo rm /etc/mysql/certs/*.pem
  • Verify all old database certs are gone:

    ls -l /etc/mysql/certs/*.pem
  • Generate new mysql certs:

    tpm cert gen mysqlcerts -x
  • Set proper ownership and permissions for Tungsten access:

    sudo chown -R mysql: /etc/mysql/certs/
    sudo chmod -R g+r /etc/mysql/certs/
  • Verify new database certs:

    ls -l /etc/mysql/certs/*.pem
  • Copy new database certs to all other database nodes:

    for i in 2 3 4 5; do sudo rsync -avc --delete /etc/mysql/certs/ db$i:/etc/mysql/certs/; done
  • Backup any previously-generated cluster certs to /opt/continuent/backups/:

    tpm cert backup gen
  • Backup the running cluster certs to /opt/continuent/backups/:

    tpm cert backup share
  • Regenerate the MySQL .p12 file and the needed Tungsten security files, using --livetls to specify the running TLS cert file (/opt/continuent/share/tungsten_tls_keystore.jks):

    tpm cert gen p12,ke,ts,ck,ct --livetls
  • Examine the new files:

    tpm cert info p12,ke,ts,ck,ct
  • Copy new files to ALL other cluster nodes:

    tpm copy --gen 
      ~OR~
    for i in 2 3 4 5; do rsync -avc --delete /opt/continuent/generated/ db$i:/opt/continuent/generated/; done
  • Set the cluster policy to MAINENANCE

    tpm policy -m
** On all cluster nodes as the tungsten OS user **
  • Stop the cluster processes:

    stopall
  • Restart the database server process:

    sudo systemctl restart mysqld
      ~OR~
    sudo service mysqld restart
  • Identify the Tungsten software staging directory:

    cd `tpm query staging| cut -d: -f2`
  • Update the Tungsten software to use the new certs, which will restart all Tungsten processes:

    tools/tpm update -i --replace-release
  • When all updates have completed, start the cluster software:

    startall
  • When all nodes have been updated and started, wait 30 seconds, then test:

    echo ls | cctrl
    tpm connector
** On a single node as the tungsten OS user **
  • When all of the above tests are ok, set the cluster policy to AUTOMATIC:

    tpm policy -a
  • Test again:

    echo ls | cctrl
    tpm connector

*** FUNCTIONAL PROCEDURE EXAMPLE ENDS HERE ***