v7 is a major release with many changes, specifically to security. At this time, upgrading directly to v7 is only supported from v5 onwards. If security is NOT enabled in your installation, then upgrading from an older release may work, however any issues encountered will not be addressed and upgrading to v6 first will be the advised route.
Whilst every care has been taken to ensure upgrades are as smooth and easy as possible, ALWAYS ensure full backups are taken before proceeding, and if possible, test the upgrade on a non-Production environment first.
Prior to v7, Tungsten came with security turned OFF through the tpm flag
disable-security-controls set to
true by default. This flag, when set to
false would translate to the following settings being
applied:
file-protection-level=0027rmi-ssl=truethl-ssl=truermi-authentication=truejgroups-ssl=true
This would enable SSL communication between Tungsten components. However, connection to the database remained unencrypted, which would translate to the following settings being applied:
datasource-enable-ssl=falseconnector-ssl=false
Setting these to true is possible, however there are many more manual steps that would have been required.
v7 enables full security by default, so the
disable-security-controls flag will
default to false when not specified.
In addition to the default value changing,
disable-security-controls now enables
encrypted communication to the database. Setting this value to
false, now translates to the following settings being
applied:
file-protection-level=0027rmi-ssl=truethl-ssl=truermi-authentication=truejgroups-ssl=truedatasource-enable-ssl=trueconnector-ssl=true
In summary, this change in behavior means that upgrades need to be handled with care and appropriate decisions being made, both by the tpm process, and by the "human" to decide on what end result is desired. The various options and examples are outlined in the following sections of this document.
This is the easiest and smoothest approach. tpm will
process your configuration and do its best to maintain the same level of
security. In order to achieve that, tpm will
dynamically update your configuration (either the
tungsten.ini file for INI installs, or the
deploy.cfg for staging installs) with additional
properties to adjust the level of security to match.
The properties that tpm will add to your configuration will be some or all of the following depending on the initial starting point of your configuration:
disable-security-controlsconnector-rest-api-sslmanager-rest-api-sslreplicator-rest-api-ssldatasource-enable-sslenable-connector-ssl
You can now proceed with the upgrade, refer to Section 4.4.7.7, “Steps to upgrade using tpm” for the required steps
The following security setting levels can be enabled, and will require user action prior to upgrading. These are:
Internal Encryption and Authentication
Tungsten to Database Encryption
Application (Connector) to Database Encryption
API SSL
Applying all of the above steps will bring full security, equivalent to the default v7 configuration.
The steps to enable will depend on what (if any) security is enabled in your existing installation. The following sections outline the steps required to be performed to enable security for each of the various layers. To understand whether you have configured any of the various layers of security, the following summary will help to understand your configuration:
No Security
If no security has been configured, the installation that you are
starting from will have
disable-security-controls=true (or it
will not supplied at all) and no additional securoty properties will be
supplied.
Partial Security
The installation that you are starting from will have partial security in place. This could be a combination of any of the following:
Internal encryption is configured
(disable-security-controls=false),
and/or
Connector encryption is enabled
(enable-connector-ssl=true and/or
Cluster to the database encryption is enabled
(datasource-enable-ssl=true or
repl-datasource-enable-ssl=true)
To upgrade and enable security, you should follow one or more of the following steps based on your requirements. At a minimum, the first step should always be included, the remaining steps are optional.
Prior to running the upgrade, you need to manually create the keystore, to do this follow these steps on one host, and then copy the files to all other hosts in your topology:
db1>mkdir /etc/tungsten/securedb1>keytool -genseckey -alias jgroups -validity 3650 -keyalg Blowfish -keysize 56 \ -keystore /etc/tungsten/secure/jgroups.jceks -storepass tungsten -keypass tungsten -storetype JCEKS
If you have an INI based install, and this is the only level of security you plan on configuring you should now copy these new keystores to all other hosts in your topology. If you plan to enable SSL at the other remaining layers, or you use a Staging based install, then skip this copy step.
db1> for host in db2 db3 db4 db5 db6; do
ssh ${host} mkdir /etc/tungsten/secure
scp /etc/tungsten/secure/*.jceks ${host}:/etc/tungsten/secure
doneEnabling internal encryption and authentication will also enable API SSL by default.
If you need to enable encryption to the underlying database, now proceed to the next step Section 4.4.7.4, “Enable Tungsten to Database Encryption” before running the upgrade, otherwise you can then start the upgrade by following the steps in Section 4.4.7.7, “Steps to upgrade using tpm”.
The following additional configuration properties will need adding to your existing configuration. The suggested process based on an INI or Staging based install are outlined in the final upgrade steps referenced above.
disable-security-controls=falseconnector-rest-api-ssl=truemanager-rest-api-ssl=truereplicator-rest-api-ssl=truejava-jgroups-keystore-path=/etc/tungsten/secure/jgroups.jceks
The following prerequisite steps must be performed before continuing with this step
In this step, you pre-create the various keystores required and register
the MySQL certificates for Tungsten. Execute all of the following steps on
a single host, for example, db1. In the example below it is assumed that
the mysql certificates reside in /etc/mysql/certs. If
you use the example syntax below, you will also need to ensure the
following directory exists: /etc/tungsten/secure
These commands will import the MySQL certificates into the required Tungsten truststores.
db1>keytool -importkeystore -srckeystore /etc/mysql/certs/client-cert.p12 -srcstoretype PKCS12 \ -destkeystore /etc/tungsten/secure/keystore.jks -deststorepass tungsten -srcstorepass tungstendb1>keytool -import -alias mysql -file /etc/mysql/certs/ca.pem -keystore /etc/tungsten/secure/truststore.ts \ -storepass tungsten -noprompt
If you have an INI based install, and you do not intend to configure SSL for your applications (via Connectors), or if your connectors reside on remote, dedicated hosts, you should now copy all of the generated keystores and truststores to all of the other hosts. If you use a Staging based install, then skip this copy step.
db1> for host in db2 db3 db4 db5 db6; do
ssh ${host} mkdir /etc/tungsten/secure
scp /etc/tungsten/secure/*.jceks ${host}:/etc/tungsten/secure
scp /etc/tungsten/secure/*.jks ${host}:/etc/tungsten/secure
scp /etc/tungsten/secure/*.ts ${host}:/etc/tungsten/secure
doneIf you need to enable encryption to the underlying database from the connectors, now proceed to Section 4.4.7.5, “Enable Connector to Database Encryption” before running the upgrade, alternatively you can now follow the steps outlined in Section 4.4.7.7, “Steps to upgrade using tpm”
The following additional configuration properties will need adding to your existing configuration. The suggested process based on an INI or Staging based install are outlined in the final upgrade steps referenced above.
datasource-enable-ssl=truejava-truststore-path=/etc/tungsten/secure/truststore.tsjava-truststore-password=tungstenjava-keystore-path=/etc/tungsten/secure/keystore.jksjava-keystore-password=tungstendatasource-mysql-ssl-cert=/etc/mysql/certs/client-cert.pemdatasource-mysql-ssl-key=/etc/mysql/certs/client-key.pemdatasource-mysql-ssl-ca=/etc/mysql/certs/ca.pem
The steps outlined in this section will need to be performed on all nodes where the Connector has been installed
If you are also enabling Internal Encryption, you would have followed the
steps in Section 4.4.7.3, “Setup internal encryption and authentication” and you would have a
number of files already in /etc/tungsten/secure. This next step will
pre-create the keystore and truststore, and register the MySQL
certificates for the Connectors. Execute all of the following steps on a
single host, in this example, db1. In the example below it is assumed the
mysql certificates reside in /etc/mysql/certs. If you
use the example syntax below, you will need to ensure the following
directory also exists: /etc/tungsten/secure
db1>keytool -importkeystore -srckeystore /etc/mysql/certs/client-cert.p12 \ -srcstoretype PKCS12 -destkeystore /etc/tungsten/secure/tungsten_connector_keystore.jks \ -deststorepass tungsten -srcstorepass tungstendb1>keytool -import -alias mysql -file /etc/mysql/certs/ca.pem \ -keystore /etc/tungsten/secure/tungsten_connector_truststore.ts -storepass tungsten -noprompt
Now that all of the necessary steps have been taken to create the various keystores, and if you use an INI based install, you now need to copy all of these files to all other hosts in your topology. If you are using a Staging based installation, then skip this copy step.
db1> for host in db2 db3 db4 db5 db6; do
ssh ${host} mkdir /etc/tungsten/secure
scp /etc/tungsten/secure/*.jceks ${host}:/etc/tungsten/secure
scp /etc/tungsten/secure/*.jks ${host}:/etc/tungsten/secure
scp /etc/tungsten/secure/*.ts ${host}:/etc/tungsten/secure
doneOnce the steps above have been performed, you can then continue with the upgrade, following the steps outlined in Section 4.4.7.7, “Steps to upgrade using tpm”
The following additional configuration properties will need adding to your existing configuration. The suggested process based on an INI or Staging based install are outlined in the final upgrade steps referenced above.
enable-connector-ssl=truejava-connector-keystore-path=/etc/tungsten/secure/tungsten_connector_keystore.jksjava-connector-keystore-password=tungstenjava-connector-truststore-path=/etc/tungsten/secure/tungsten_connector_truststore.tsjava-connector-truststore-password=tungsten
A prerequisite to enabling full security, is to enable SSL within your database if this isn't already configured. To do this, we can use the mysql_ssl_rsa_setup tool supplied with most distributions of MySQL. If you do not have this tool, or require more detail, you can refer to Section 5.13.1, “Enabling Database SSL”. The steps below summarise the process using the mysql_ssl_rsa_setup
The first step is to setup the directories for the certs, perform this on ALL hosts in your topology:
shell>sudo mkdir -p /etc/mysql/certsshell>sudo chown -R tungsten: /etc/mysql/certs/
NB: The ownership is temporarily set to tungsten so that the subsequent scp will work between hosts.
This next step should be performed on just one single host, for the purpose of this example we will use db1 as the host:
db1>mysql_ssl_rsa_setup -d /etc/mysql/certs/db1>openssl pkcs12 -export -inkey /etc/mysql/certs/client-key.pem \ -name mysql -in /etc/mysql/certs/client-cert.pem -out /etc/mysql/certs/client-cert.p12 \ -passout pass:tungsten
When using OpenSSL 3.0 with Java 1.8, you
MUST add the
-legacy option to the openssl
command.
db1> for host in db2 db3 db4 db5 db6; do
scp /etc/mysql/certs/* ${host}:/etc/mysql/certs
doneNext, on every host we need to reset the directory ownership
shell>sudo chown -R mysql: /etc/mysql/certs/shell>sudo chmod g+r /etc/mysql/certs/client-*
Now on every host, we need to reconfigure MySQL. Add the following
properties into your my.cnf
[mysqld] ssl-ca=/etc/mysql/certs/ca.pem ssl-cert=/etc/mysql/certs/server-cert.pem ssl-key=/etc/mysql/certs/server-key.pem [client] ssl-cert=/etc/mysql/certs/client-cert.pem ssl-key=/etc/mysql/certs/client-key.pem ssl-ca=/etc/mysql/certs/ca.pem
Next, place your cluster(s) into MAINTENANCE mode
shell> cctrl cctrl> set policy maintenance
Restart MySQL for the new settings to take effect
shell> sudo service mysqld restart
Finally, return your cluster(s) into AUTOMATIC mode
shell> cctrl cctrl> set policy automatic
When you are ready to perform the upgrade, the following steps should be followed:
Ensure you place your cluster(s) into MAINTENANCE
mode
If no additional steps taken, and you wish to maintain the same level of security, skip Step 3, and proceed directly to Step 4.
Update your tungsten.ini and include some, or
all, of the options below depending on which steps you took earlier.
All entries should be placed within the
[defaults] stanza.
disable-security-controls=falseconnector-rest-api-ssl=truemanager-rest-api-ssl=truereplicator-rest-api-ssl=truejava-jgroups-keystore-path=/etc/tungsten/secure/jgroups.jceks
If "Tungsten to Database Encryption" IS configured, also add:
datasource-enable-ssl=truejava-truststore-path=/etc/tungsten/secure/truststore.tsjava-truststore-password=tungstenjava-keystore-path=/etc/tungsten/secure/keystore.jksjava-keystore-password=tungstendatasource-mysql-ssl-cert=/etc/mysql/certs/client-cert.pemdatasource-mysql-ssl-key=/etc/mysql/certs/client-key.pemdatasource-mysql-ssl-ca=/etc/mysql/certs/ca.pem
If "Tungsten to Database Encryption" IS NOT configured, also add:
datasource-enable-ssl=falseIf "Application (Connector) to Database Encryption" IS configured, also add:
enable-connector-ssl=truejava-connector-keystore-path=/etc/tungsten/secure/tungsten_connector_keystore.jksjava-connector-keystore-password=tungstenjava-connector-truststore-path=/etc/tungsten/secure/tungsten_connector_truststore.tsjava-connector-truststore-password=tungsten
If "Application (Connector) to Database Encryption" IS NOT configured, also add:
enable-connector-ssl=false
If start-and-report=true, remove
this value or set to false
Obtain the TAR or RPM package for your installation. If using a TAR
file unpack this into your software staging tree, typically
/opt/continuent/software. If you use the INI
install method, this needs to be performed on every host. For
staging install, this applies to the staging host only.
Change into the directory for the software
shell> cd /opt/continuent/software/tungsten-clustering-6.1.25-6Issue the following command on all hosts.
shell> tools/tpm update --replace-release
When upgrading the connectors, you could include the optional
--no-connectors option if you wish to control the
restart of the connectors manually
For Multi-Site/Active-Active topologies, you will also need to repeat the steps for the cross-site replicators
Finally, before returning the cluster(s) to
AUTOMATIC, you will need to sync the new
certificates, created by the upgrade, to all hosts. This step will
be required even if you have disabled security as these files will
be used by the API and also, if you choose to enable it, THL
Encryption.
From one host, copy the certificate and keystore files to ALL other hosts in your topology. The following scp command is an example assuming you are issuing from db1, and the install directory is /opt/continuent:
db1> for host in db2 db3 db4 db5 db6; do
scp /opt/continuent/share/[jpt]* ${host}:/opt/continuent/share
scp /opt/continuent/share/.[jpt]* ${host}:/opt/continuent/share
doneThe examples assume you have the ability to scp between hosts as the tungsten OS user. If your security restrictions do not permit this, you will need to use alternative procedures appropriate to your environment to ensure these files are in sync across all hosts before continuing.
If the files are not in sync between hosts, the software will fail to start!
You will also need to repeat this if you have a Multi-Site/Active-Active topology for the cross-site replicators:
db1> for host in db2 db3 db4 db5 db6; do
scp /opt/replicator/share[jpt]* ${host}:/opt/replicator/share
scp /opt/replicator/share.[jpt]* ${host}:/opt/replicator/share
doneRestart all tungsten components, one host at a time
shell>manager restartshell>replicator restartshell>connector restart
Return the cluster(s) to AUTOMATIC mode
Ensure you place your cluster(s) into MAINTENANCE
mode
Obtain the TAR or RPM package for your installation. If using a TAR
file unpack this into your software staging tree, typically
/opt/continuent/software. If you use the INI
install method, this needs to be performed on every host. For
staging install, this applies to the staging host only.
Change into the directory for the software and fetch the configuration, e.g
shell>cd /opt/continuent/software/tungsten-clustering-6.1.25-6shell>tpm reverse > deploy.sh
If no additional steps taken, and you wish to maintain the same level of security, skip Step 5, and proceed directly to Step 6.
Edit the deploy.sh file just created, and include some, or all, of
the options below depending on which steps you took earlier (They
should be placed within the defaults.
--disable-security-controls=false--connector-rest-api-ssl=true--manager-rest-api-ssl=true--replicator-rest-api-ssl=true--java-jgroups-keystore-path=/etc/tungsten/secure/jgroups.jceks
If "Tungsten to Database Encryption" IS configured, also add:
--datasource-enable-ssl=true--java-truststore-path=/etc/tungsten/secure/truststore.ts--java-truststore-password=tungsten--java-keystore-path=/etc/tungsten/secure/keystore.jks--java-keystore-password=tungsten--datasource-mysql-ssl-cert=/etc/mysql/certs/client-cert.pem--datasource-mysql-ssl-key=/etc/mysql/certs/client-key.pem--datasource-mysql-ssl-ca=/etc/mysql/certs/ca.pem
If "Tungsten to Database Encryption" IS NOT configured, also add:
--datasource-enable-ssl=falseIf "Application (Connector) to Database Encryption" IS configured, also add:
--enable-connector-ssl=true--java-connector-keystore-path=/etc/tungsten/secure/tungsten_connector_keystore.jks--java-connector-keystore-password=tungsten--java-connector-truststore-path=/etc/tungsten/secure/tungsten_connector_truststore.ts--java-connector-truststore-password=tungsten
If "Application (Connector) to Database Encryption" IS NOT configured, also add:
--enable-connector-ssl=false
If start-and-report=true, remove
this value or set to false
An example of a BEFORE and AFTER edit including all options:
shell> cat deploy.sh
# BEFORE
tools/tpm configure defaults \
--reset \
--application-password=secret \
--application-port=3306 \
--application-user=app_user \
--disable-security-controls=true \
--install-directory=/opt/continuent \
--mysql-allow-intensive-checks=true \
--profile-script=/home/tungsten/.bash_profile \
--replication-password=secret \
--replication-user=tungsten \
--start-and-report=true \
--user=tungsten
# Options for the nyc data service
tools/tpm configure nyc \
--connectors=db1,db2,db3 \
--master=db1 \
--slaves=db2,db3 \
--topology=clusteredshell> cat deploy.sh
# AFTER
tools/tpm configure defaults \
--reset \
--application-password=secret \
--application-port=3306 \
--application-user=app_user \
--install-directory=/opt/continuent \
--mysql-allow-intensive-checks=true \
--profile-script=/home/tungsten/.bash_profile \
--replication-password=secret \
--replication-user=tungsten \
--user=tungsten \
--start-and-report=false \
--disable-security-controls=false \
--connector-rest-api-ssl=true \
--manager-rest-api-ssl=true \
--replicator-rest-api-ssl=true \
--datasource-enable-ssl=true \
--java-jgroups-keystore-path=/etc/tungsten/secure/jgroups.jceks \
--java-truststore-path=/etc/tungsten/secure/truststore.ts \
--java-truststore-password=tungsten \
--java-keystore-path=/etc/tungsten/secure/keystore.jks \
--java-keystore-password=tungsten \
--enable-connector-ssl=true \
--java-connector-keystore-path=/etc/tungsten/secure/tungsten_connector_keystore.jks \
--java-connector-keystore-password=tungsten \
--java-connector-truststore-path=/etc/tungsten/secure/tungsten_connector_truststore.ts \
--java-connector-truststore-password=tungsten \
--datasource-mysql-ssl-cert=/etc/mysql/certs/client-cert.pem \
--datasource-mysql-ssl-key=/etc/mysql/certs/client-key.pem \
--datasource-mysql-ssl-ca=/etc/mysql/certs/ca.pem
# Options for the nyc data service
tools/tpm configure nyc \
--connectors=db1,db2,db3 \
--master=db1 \
--slaves=db2,db3 \
--topology=clusteredNext, source the file to load the configuration and then execute the update:
shell>source deploy.shshell>tools/tpm update --replace-release
You may wish to include the optional
--no-connectors option if you wish to control the
restart of the connectors manually
For Multi-Site/Active-Active topologies, you will also need to repeat the steps for the cross-site replicators
Finally, before returning the cluster(s) to
AUTOMATIC, you will need to sync the new
certificates, created by the upgrade, to all hosts. This step will
be required even if you have disabled security as these files will
be used by the API and also, if you choose to enable it, THL
Encryption.
From one host, copy the certificate and keystore files to ALL other hosts in your topology. The following scp command is an example assuming you are issuing from db1, and the install directory is /opt/continuent:
db1> for host in db2 db3 db4 db5 db6; do
scp /opt/continuent/share/[jpt]* ${host}:/opt/continuent/share
scp /opt/continuent/share/.[jpt]* ${host}:/opt/continuent/share
doneThe examples assume you have the ability to scp between hosts as the tungsten OS user. If your security restrictions do not permit this, you will need to use alternative procedures appropriate to your environment to ensure these files are in sync across all hosts before continuing.
If the files are not in sync between hosts, the software will fail to start!
You will also need to repeat this if you have a Multi-Site/Active-Active topology for the cross-site replicators:
db1> for host in db2 db3 db4 db5 db6; do
scp /opt/replicator/share[jpt]* ${host}:/opt/replicator/share
scp /opt/replicator/share.[jpt]* ${host}:/opt/replicator/share
doneRestart all tungsten components, one host at a time
shell>manager restartshell>replicator restartshell>connector restart
Return the cluster(s) to AUTOMATIC mode
Once the upgrade has been completed, if you plan on using the API you will need to complete a few extra steps before you can use it. By default, after installation the API will only allow the ping method and the createAdminUser method.
To open up the API and access all of its features, you will need to configure the API User. To do this, execute the following on all hosts (Setting the value of pass to your preferred password):
shell> curl -k -H 'Content-type: application/json' --request POST 'https://127.0.0.1:8096/api/v2/createAdminUser?i-am-sure=true' \
> --data-raw '{
> "payloadType": "credentials",
> "user":"tungsten",
> "pass":"security"
> }'For more information on using the new API, please refer to ???