The philosophy here is that the cert rotation work is done on a single cluster node. We will call this the "work node".
The secret to getting certs to work with a cluster is to make sure that you copy all of the MySQL database certs and Tungsten security files from the work node to the rest of the nodes at the correct point in the process. If all the cluster and database cert files are not the same across all nodes, the cluster will fail.
In the following functional example, we demonstrate the actual steps to rotate the database certs in a standalone 5-node Tungsten cluster.
When doing a database cert rotation, multiple files must be regenerated:
Five (5) required MySQL security files (may be more created than needed):
ca.pem
client-cert.pem
client-key.pem
server-cert.pem
server-key.pem
One (1) .p12 file to represent the database client cert:
client-cert.p12
Four (4) Tungsten-specific files:
tungsten_keystore.jks
tungsten_truststore.ts
tungsten_truststore.ts
tungsten_connector_truststore.ts
Next, confirm that both MySQL and Tungsten are configured to use the proper files:
/etc/my.cnf entries:
[mysqld] ssl-ca=/etc/mysql/certs/ca.pem ssl-cert=/etc/mysql/certs/server-cert.pem ssl-key=/etc/mysql/certs/server-key.pem require_secure_transport=ON
This example assumes that the database certs are located in
/etc/mysql/certs on all cluster database nodes.
/etc/tungsten/tungsten.ini entries:
datasource-mysql-ssl-ca=/etc/mysql/certs/ca.pem datasource-mysql-ssl-cert=/etc/mysql/certs/client-cert.pem datasource-mysql-ssl-key=/etc/mysql/certs/client-key.pem
This example assumes that no
/opt/continuent/share/tungsten.env file exists.
*** FUNCTIONAL PROCEDURE EXAMPLE STARTS HERE ***
Backup the old database certs from
/etc/mysql/certs/ to
/opt/continuent/backups/:
tpm cert backup mysql
Clean out any old database certs so new certs are generated:
sudo rm /etc/mysql/certs/*.pem
Verify all old database certs are gone:
ls -l /etc/mysql/certs/*.pem
Generate new mysql certs:
tpm cert gen mysqlcerts -x
Set proper ownership and permissions for Tungsten access:
sudo chown -R mysql: /etc/mysql/certs/ sudo chmod -R g+r /etc/mysql/certs/
Verify new database certs:
ls -l /etc/mysql/certs/*.pem
Copy new database certs to all other database nodes:
for i in 2 3 4 5; do sudo rsync -avc --delete /etc/mysql/certs/ db$i:/etc/mysql/certs/; done
Backup any previously-generated cluster certs to
/opt/continuent/backups/:
tpm cert backup gen
Backup the running cluster certs to
/opt/continuent/backups/:
tpm cert backup share
Regenerate the MySQL .p12 file and the needed Tungsten security files,
using --livetls to specify the running TLS cert file
(/opt/continuent/share/tungsten_tls_keystore.jks):
tpm cert gen p12,ke,ts,ck,ct --livetls
Examine the new files:
tpm cert info p12,ke,ts,ck,ct
Copy new files to ALL other cluster nodes:
tpm copy --gen ~OR~ for i in 2 3 4 5; do rsync -avc --delete /opt/continuent/generated/ db$i:/opt/continuent/generated/; done
Set the cluster policy to MAINENANCE
tpm policy -m
Stop the cluster processes:
stopall
Restart the database server process:
sudo systemctl restart mysqld ~OR~ sudo service mysqld restart
Identify the Tungsten software staging directory:
cd `tpm query staging| cut -d: -f2`
Update the Tungsten software to use the new certs, which will restart all Tungsten processes:
tools/tpm update -i --replace-release
When all updates have completed, start the cluster software:
startall
When all nodes have been updated and started, wait 30 seconds, then test:
echo ls | cctrl tpm connector
When all of the above tests are ok, set the cluster policy to AUTOMATIC:
tpm policy -a
Test again:
echo ls | cctrl tpm connector
*** FUNCTIONAL PROCEDURE EXAMPLE ENDS HERE ***