4.4.7. Upgrading to v7.0.0+

Warning

v7 is a major release with many changes, specifically to security. At this time, upgrading directly to v7 is only supported from v5 onwards. If security is NOT enabled in your installation, then upgrading from an older release may work, however any issues encountered will not be addressed and upgrading to v6 first will be the advised route.

Warning

Whilst every care has been taken to ensure upgrades are as smooth and easy as possible, ALWAYS ensure full backups are taken before proceeding, and if possible, test the upgrade on a non-Production environment first.

4.4.7.1. Background

4.4.7.1.1. v6 (and earlier) behavior

Prior to v7, Tungsten came with security turned OFF through the tpm flag disable-security-controls set to true by default. This flag, when set to false would translate to the following settings being applied:

file-protection-level=0027
rmi-ssl=true
thl-ssl=true
rmi-authentication=true
jgroups-ssl=true

This would enable SSL communication between Tungsten components. However, connection to the database remained unencrypted, which would translate to the following settings being applied:

datasource-enable-ssl=false
connector-ssl=false

Setting these to true is possible, however there are many more manual steps that would have been required.

4.4.7.1.2. New behavior in v7

v7 enables full security by default, so the disable-security-controls flag will default to false when not specified.

In addition to the default value changing, disable-security-controls now enables encrypted communication to the database. Setting this value to false, now translates to the following settings being applied:

file-protection-level=0027
rmi-ssl=true
thl-ssl=true
rmi-authentication=true
jgroups-ssl=true
datasource-enable-ssl=true 
connector-ssl=true
4.4.7.1.3. Summary

In summary, this change in behavior means that upgrades need to be handled with care and appropriate decisions being made, both by the tpm process, and by the "human" to decide on what end result is desired. The various options and examples are outlined in the following sections of this document.

4.4.7.2. Upgrade Decisions

4.4.7.2.1. Keep existing level of security

This is the easiest and smoothest approach. tpm will process your configuration and do its best to maintain the same level of security. In order to achieve that, tpm will dynamically update your configuration (either the tungsten.ini file for INI installs, or the deploy.cfg for staging installs) with additional properties to adjust the level of security to match.

The properties that tpm will add to your configuration will be some or all of the following depending on the initial starting point of your configuration:

disable-security-controls
connector-rest-api-ssl
manager-rest-api-ssl
replicator-rest-api-ssl
datasource-enable-ssl
enable-connector-ssl

You can now proceed with the upgrade, refer to Section 4.4.7.7, “Steps to upgrade using tpm” for the required steps

4.4.7.2.2. Apply new recommendations and setup security

The following security setting levels can be enabled, and will require user action prior to upgrading. These are:

  1. Internal Encryption and Authentication

  2. Tungsten to Database Encryption

  3. Application (Connector) to Database Encryption

  4. API SSL

Applying all of the above steps will bring full security, equivalent to the default v7 configuration.

The steps to enable will depend on what (if any) security is enabled in your existing installation. The following sections outline the steps required to be performed to enable security for each of the various layers. To understand whether you have configured any of the various layers of security, the following summary will help to understand your configuration:

No Security

If no security has been configured, the installation that you are starting from will have disable-security-controls=true (or it will not supplied at all) and no additional securoty properties will be supplied.

Partial Security

The installation that you are starting from will have partial security in place. This could be a combination of any of the following:

To upgrade and enable security, you should follow one or more of the following steps based on your requirements. At a minimum, the first step should always be included, the remaining steps are optional.

4.4.7.3. Setup internal encryption and authentication

Prior to running the upgrade, you need to manually create the keystore, to do this follow these steps on one host, and then copy the files to all other hosts in your topology:

db1> mkdir /etc/tungsten/secure
db1> keytool -genseckey -alias jgroups -validity 3650 -keyalg Blowfish -keysize 56 \
-keystore /etc/tungsten/secure/jgroups.jceks -storepass tungsten -keypass tungsten -storetype JCEKS

If you have an INI based install, and this is the only level of security you plan on configuring you should now copy these new keystores to all other hosts in your topology. If you plan to enable SSL at the other remaining layers, or you use a Staging based install, then skip this copy step.

db1> for host in db2 db3 db4 db5 db6; do 
ssh ${host} mkdir /etc/tungsten/secure
scp /etc/tungsten/secure/*.jceks ${host}:/etc/tungsten/secure
done

Enabling internal encryption and authentication will also enable API SSL by default.

If you need to enable encryption to the underlying database, now proceed to the next step Section 4.4.7.4, “Enable Tungsten to Database Encryption” before running the upgrade, otherwise you can then start the upgrade by following the steps in Section 4.4.7.7, “Steps to upgrade using tpm”.

The following additional configuration properties will need adding to your existing configuration. The suggested process based on an INI or Staging based install are outlined in the final upgrade steps referenced above.

disable-security-controls=false
connector-rest-api-ssl=true
manager-rest-api-ssl=true
replicator-rest-api-ssl=true
java-jgroups-keystore-path=/etc/tungsten/secure/jgroups.jceks

4.4.7.4. Enable Tungsten to Database Encryption

The following prerequisite steps must be performed before continuing with this step

In this step, you pre-create the various keystores required and register the MySQL certificates for Tungsten. Execute all of the following steps on a single host, for example, db1. In the example below it is assumed that the mysql certificates reside in /etc/mysql/certs. If you use the example syntax below, you will also need to ensure the following directory exists: /etc/tungsten/secure

These commands will import the MySQL certificates into the required Tungsten truststores.

db1> keytool -importkeystore -srckeystore /etc/mysql/certs/client-cert.p12 -srcstoretype PKCS12 \
-destkeystore /etc/tungsten/secure/keystore.jks -deststorepass tungsten -srcstorepass tungsten

db1> keytool -import -alias mysql -file /etc/mysql/certs/ca.pem -keystore /etc/tungsten/secure/truststore.ts \
-storepass tungsten -noprompt

If you have an INI based install, and you do not intend to configure SSL for your applications (via Connectors), or if your connectors reside on remote, dedicated hosts, you should now copy all of the generated keystores and truststores to all of the other hosts. If you use a Staging based install, then skip this copy step.

db1> for host in db2 db3 db4 db5 db6; do 
ssh ${host} mkdir /etc/tungsten/secure
scp /etc/tungsten/secure/*.jceks ${host}:/etc/tungsten/secure 
scp /etc/tungsten/secure/*.jks ${host}:/etc/tungsten/secure
scp /etc/tungsten/secure/*.ts ${host}:/etc/tungsten/secure
done

If you need to enable encryption to the underlying database from the connectors, now proceed to Section 4.4.7.5, “Enable Connector to Database Encryption” before running the upgrade, alternatively you can now follow the steps outlined in Section 4.4.7.7, “Steps to upgrade using tpm”

The following additional configuration properties will need adding to your existing configuration. The suggested process based on an INI or Staging based install are outlined in the final upgrade steps referenced above.

datasource-enable-ssl=true
java-truststore-path=/etc/tungsten/secure/truststore.ts
java-truststore-password=tungsten
java-keystore-path=/etc/tungsten/secure/keystore.jks
java-keystore-password=tungsten
datasource-mysql-ssl-cert=/etc/mysql/certs/client-cert.pem
datasource-mysql-ssl-key=/etc/mysql/certs/client-key.pem
datasource-mysql-ssl-ca=/etc/mysql/certs/ca.pem

4.4.7.5. Enable Connector to Database Encryption

The steps outlined in this section will need to be performed on all nodes where the Connector has been installed

If you are also enabling Internal Encryption, you would have followed the steps in Section 4.4.7.3, “Setup internal encryption and authentication” and you would have a number of files already in /etc/tungsten/secure. This next step will pre-create the keystore and truststore, and register the MySQL certificates for the Connectors. Execute all of the following steps on a single host, in this example, db1. In the example below it is assumed the mysql certificates reside in /etc/mysql/certs. If you use the example syntax below, you will need to ensure the following directory also exists: /etc/tungsten/secure

db1> keytool -importkeystore -srckeystore /etc/mysql/certs/client-cert.p12 \
-srcstoretype PKCS12 -destkeystore /etc/tungsten/secure/tungsten_connector_keystore.jks \
-deststorepass tungsten -srcstorepass tungsten

db1> keytool -import -alias mysql -file /etc/mysql/certs/ca.pem \
-keystore /etc/tungsten/secure/tungsten_connector_truststore.ts -storepass tungsten -noprompt

Now that all of the necessary steps have been taken to create the various keystores, and if you use an INI based install, you now need to copy all of these files to all other hosts in your topology. If you are using a Staging based installation, then skip this copy step.

db1> for host in db2 db3 db4 db5 db6; do 
ssh ${host} mkdir /etc/tungsten/secure
scp /etc/tungsten/secure/*.jceks ${host}:/etc/tungsten/secure
scp /etc/tungsten/secure/*.jks ${host}:/etc/tungsten/secure 
scp /etc/tungsten/secure/*.ts ${host}:/etc/tungsten/secure
done

Once the steps above have been performed, you can then continue with the upgrade, following the steps outlined in Section 4.4.7.7, “Steps to upgrade using tpm”

The following additional configuration properties will need adding to your existing configuration. The suggested process based on an INI or Staging based install are outlined in the final upgrade steps referenced above.

enable-connector-ssl=true
java-connector-keystore-path=/etc/tungsten/secure/tungsten_connector_keystore.jks
java-connector-keystore-password=tungsten
java-connector-truststore-path=/etc/tungsten/secure/tungsten_connector_truststore.ts
java-connector-truststore-password=tungsten

4.4.7.6. Enable MySQL SSL

A prerequisite to enabling full security, is to enable SSL within your database if this isn't already configured. To do this, we can use the mysql_ssl_rsa_setup tool supplied with most distributions of MySQL. If you do not have this tool, or require more detail, you can refer to Section 5.13.1, “Enabling Database SSL”. The steps below summarise the process using the mysql_ssl_rsa_setup

  1. The first step is to setup the directories for the certs, perform this on ALL hosts in your topology:

    shell> sudo mkdir -p /etc/mysql/certs
    shell> sudo chown -R tungsten: /etc/mysql/certs/

    NB: The ownership is temporarily set to tungsten so that the subsequent scp will work between hosts.

  2. This next step should be performed on just one single host, for the purpose of this example we will use db1 as the host:

    db1> mysql_ssl_rsa_setup -d /etc/mysql/certs/
    db1> openssl pkcs12 -export -inkey /etc/mysql/certs/client-key.pem \
    -name mysql -in /etc/mysql/certs/client-cert.pem -out /etc/mysql/certs/client-cert.p12 \
    -passout pass:tungsten

    Important

    When using OpenSSL 3.0 with Java 1.8, you MUST add the -legacy option to the openssl command.

    db1> for host in db2 db3 db4 db5 db6; do 
    scp /etc/mysql/certs/* ${host}:/etc/mysql/certs 
    done
  3. Next, on every host we need to reset the directory ownership

    shell> sudo chown -R mysql: /etc/mysql/certs/
    shell> sudo chmod g+r /etc/mysql/certs/client-*
  4. Now on every host, we need to reconfigure MySQL. Add the following properties into your my.cnf

    [mysqld]
    ssl-ca=/etc/mysql/certs/ca.pem
    ssl-cert=/etc/mysql/certs/server-cert.pem
    ssl-key=/etc/mysql/certs/server-key.pem
    
    [client]
    ssl-cert=/etc/mysql/certs/client-cert.pem
    ssl-key=/etc/mysql/certs/client-key.pem
    ssl-ca=/etc/mysql/certs/ca.pem
  5. Next, place your cluster(s) into MAINTENANCE mode

    shell> cctrl
    cctrl> set policy maintenance
  6. Restart MySQL for the new settings to take effect

    shell> sudo service mysqld restart
  7. Finally, return your cluster(s) into AUTOMATIC mode

    shell> cctrl
    cctrl> set policy automatic

4.4.7.7. Steps to upgrade using tpm

When you are ready to perform the upgrade, the following steps should be followed:

4.4.7.7.1. Steps for INI Based Installations
  1. Ensure you place your cluster(s) into MAINTENANCE mode

  2. If no additional steps taken, and you wish to maintain the same level of security, skip Step 3, and proceed directly to Step 4.

  3. Update your tungsten.ini and include some, or all, of the options below depending on which steps you took earlier. All entries should be placed within the [defaults] stanza.

    disable-security-controls=false
    connector-rest-api-ssl=true
    manager-rest-api-ssl=true
    replicator-rest-api-ssl=true
    java-jgroups-keystore-path=/etc/tungsten/secure/jgroups.jceks

    If "Tungsten to Database Encryption" IS configured, also add:

    datasource-enable-ssl=true
    java-truststore-path=/etc/tungsten/secure/truststore.ts
    java-truststore-password=tungsten
    java-keystore-path=/etc/tungsten/secure/keystore.jks
    java-keystore-password=tungsten
    datasource-mysql-ssl-cert=/etc/mysql/certs/client-cert.pem
    datasource-mysql-ssl-key=/etc/mysql/certs/client-key.pem
    datasource-mysql-ssl-ca=/etc/mysql/certs/ca.pem

    If "Tungsten to Database Encryption" IS NOT configured, also add:

    datasource-enable-ssl=false

    If "Application (Connector) to Database Encryption" IS configured, also add:

    enable-connector-ssl=true
    java-connector-keystore-path=/etc/tungsten/secure/tungsten_connector_keystore.jks
    java-connector-keystore-password=tungsten
    java-connector-truststore-path=/etc/tungsten/secure/tungsten_connector_truststore.ts
    java-connector-truststore-password=tungsten

    If "Application (Connector) to Database Encryption" IS NOT configured, also add:

    enable-connector-ssl=false

    Important

    If start-and-report=true, remove this value or set to false

  4. Obtain the TAR or RPM package for your installation. If using a TAR file unpack this into your software staging tree, typically /opt/continuent/software. If you use the INI install method, this needs to be performed on every host. For staging install, this applies to the staging host only.

  5. Change into the directory for the software

    shell> cd /opt/continuent/software/tungsten-clustering-6.1.25-6
  6. Issue the following command on all hosts.

    shell> tools/tpm update --replace-release

    When upgrading the connectors, you could include the optional --no-connectors option if you wish to control the restart of the connectors manually

  7. For Multi-Site/Active-Active topologies, you will also need to repeat the steps for the cross-site replicators

  8. Finally, before returning the cluster(s) to AUTOMATIC, you will need to sync the new certificates, created by the upgrade, to all hosts. This step will be required even if you have disabled security as these files will be used by the API and also, if you choose to enable it, THL Encryption.

    From one host, copy the certificate and keystore files to ALL other hosts in your topology. The following scp command is an example assuming you are issuing from db1, and the install directory is /opt/continuent:

    db1> for host in db2 db3 db4 db5 db6; do
    scp /opt/continuent/share/[jpt]* ${host}:/opt/continuent/share
    scp /opt/continuent/share/.[jpt]* ${host}:/opt/continuent/share
    done

    Note

    The examples assume you have the ability to scp between hosts as the tungsten OS user. If your security restrictions do not permit this, you will need to use alternative procedures appropriate to your environment to ensure these files are in sync across all hosts before continuing.

    If the files are not in sync between hosts, the software will fail to start!

  9. You will also need to repeat this if you have a Multi-Site/Active-Active topology for the cross-site replicators:

    db1> for host in db2 db3 db4 db5 db6; do
    scp /opt/replicator/share[jpt]* ${host}:/opt/replicator/share
    scp /opt/replicator/share.[jpt]* ${host}:/opt/replicator/share
    done
  10. Restart all tungsten components, one host at a time

    shell> manager restart
    shell> replicator restart
    shell> connector restart
  11. Return the cluster(s) to AUTOMATIC mode

4.4.7.7.2. Steps for Staging Based Installations
  1. Ensure you place your cluster(s) into MAINTENANCE mode

  2. Obtain the TAR or RPM package for your installation. If using a TAR file unpack this into your software staging tree, typically /opt/continuent/software. If you use the INI install method, this needs to be performed on every host. For staging install, this applies to the staging host only.

  3. Change into the directory for the software and fetch the configuration, e.g

    shell> cd /opt/continuent/software/tungsten-clustering-6.1.25-6
    shell> tpm reverse > deploy.sh
  4. If no additional steps taken, and you wish to maintain the same level of security, skip Step 5, and proceed directly to Step 6.

  5. Edit the deploy.sh file just created, and include some, or all, of the options below depending on which steps you took earlier (They should be placed within the defaults.

    --disable-security-controls=false
    --connector-rest-api-ssl=true
    --manager-rest-api-ssl=true
    --replicator-rest-api-ssl=true
    --java-jgroups-keystore-path=/etc/tungsten/secure/jgroups.jceks

    If "Tungsten to Database Encryption" IS configured, also add:

    --datasource-enable-ssl=true
    --java-truststore-path=/etc/tungsten/secure/truststore.ts
    --java-truststore-password=tungsten
    --java-keystore-path=/etc/tungsten/secure/keystore.jks
    --java-keystore-password=tungsten
    --datasource-mysql-ssl-cert=/etc/mysql/certs/client-cert.pem
    --datasource-mysql-ssl-key=/etc/mysql/certs/client-key.pem
    --datasource-mysql-ssl-ca=/etc/mysql/certs/ca.pem

    If "Tungsten to Database Encryption" IS NOT configured, also add:

    --datasource-enable-ssl=false

    If "Application (Connector) to Database Encryption" IS configured, also add:

    --enable-connector-ssl=true
    --java-connector-keystore-path=/etc/tungsten/secure/tungsten_connector_keystore.jks
    --java-connector-keystore-password=tungsten
    --java-connector-truststore-path=/etc/tungsten/secure/tungsten_connector_truststore.ts
    --java-connector-truststore-password=tungsten

    If "Application (Connector) to Database Encryption" IS NOT configured, also add:

    --enable-connector-ssl=false

    Important

    If start-and-report=true, remove this value or set to false

    An example of a BEFORE and AFTER edit including all options:

    shell> cat deploy.sh
    # BEFORE
    tools/tpm configure defaults \
    --reset \
    --application-password=secret \
    --application-port=3306 \
    --application-user=app_user \
    --disable-security-controls=true \
    --install-directory=/opt/continuent \
    --mysql-allow-intensive-checks=true \
    --profile-script=/home/tungsten/.bash_profile \
    --replication-password=secret \
    --replication-user=tungsten \
    --start-and-report=true \
    --user=tungsten
    # Options for the nyc data service
    tools/tpm configure nyc \
    --connectors=db1,db2,db3 \
    --master=db1 \
    --slaves=db2,db3 \
    --topology=clustered
    shell> cat deploy.sh
    # AFTER 
    tools/tpm configure defaults \
    --reset \
    --application-password=secret \
    --application-port=3306 \
    --application-user=app_user \
    --install-directory=/opt/continuent \
    --mysql-allow-intensive-checks=true \
    --profile-script=/home/tungsten/.bash_profile \
    --replication-password=secret \
    --replication-user=tungsten \
    --user=tungsten \
    --start-and-report=false \
    --disable-security-controls=false \
    --connector-rest-api-ssl=true \
    --manager-rest-api-ssl=true \
    --replicator-rest-api-ssl=true \
    --datasource-enable-ssl=true \
    --java-jgroups-keystore-path=/etc/tungsten/secure/jgroups.jceks \
    --java-truststore-path=/etc/tungsten/secure/truststore.ts \
    --java-truststore-password=tungsten \
    --java-keystore-path=/etc/tungsten/secure/keystore.jks \
    --java-keystore-password=tungsten \
    --enable-connector-ssl=true \
    --java-connector-keystore-path=/etc/tungsten/secure/tungsten_connector_keystore.jks \
    --java-connector-keystore-password=tungsten \
    --java-connector-truststore-path=/etc/tungsten/secure/tungsten_connector_truststore.ts \
    --java-connector-truststore-password=tungsten \
    --datasource-mysql-ssl-cert=/etc/mysql/certs/client-cert.pem \
    --datasource-mysql-ssl-key=/etc/mysql/certs/client-key.pem \
    --datasource-mysql-ssl-ca=/etc/mysql/certs/ca.pem
    
    # Options for the nyc data service
    tools/tpm configure nyc \
    --connectors=db1,db2,db3 \
    --master=db1 \
    --slaves=db2,db3 \
    --topology=clustered
  6. Next, source the file to load the configuration and then execute the update:

    shell> source deploy.sh
    shell> tools/tpm update --replace-release

    You may wish to include the optional --no-connectors option if you wish to control the restart of the connectors manually

  7. For Multi-Site/Active-Active topologies, you will also need to repeat the steps for the cross-site replicators

  8. Finally, before returning the cluster(s) to AUTOMATIC, you will need to sync the new certificates, created by the upgrade, to all hosts. This step will be required even if you have disabled security as these files will be used by the API and also, if you choose to enable it, THL Encryption.

    From one host, copy the certificate and keystore files to ALL other hosts in your topology. The following scp command is an example assuming you are issuing from db1, and the install directory is /opt/continuent:

    db1> for host in db2 db3 db4 db5 db6; do
    scp /opt/continuent/share/[jpt]* ${host}:/opt/continuent/share
    scp /opt/continuent/share/.[jpt]* ${host}:/opt/continuent/share
    done

    Note

    The examples assume you have the ability to scp between hosts as the tungsten OS user. If your security restrictions do not permit this, you will need to use alternative procedures appropriate to your environment to ensure these files are in sync across all hosts before continuing.

    If the files are not in sync between hosts, the software will fail to start!

  9. You will also need to repeat this if you have a Multi-Site/Active-Active topology for the cross-site replicators:

    db1> for host in db2 db3 db4 db5 db6; do
    scp /opt/replicator/share[jpt]* ${host}:/opt/replicator/share
    scp /opt/replicator/share.[jpt]* ${host}:/opt/replicator/share
    done
  10. Restart all tungsten components, one host at a time

    shell> manager restart
    shell> replicator restart
    shell> connector restart
  11. Return the cluster(s) to AUTOMATIC mode

4.4.7.8. Optional Post-Upgrade steps to configure API

Once the upgrade has been completed, if you plan on using the API you will need to complete a few extra steps before you can use it. By default, after installation the API will only allow the ping method and the createAdminUser method.

To open up the API and access all of its features, you will need to configure the API User. To do this, execute the following on all hosts (Setting the value of pass to your preferred password):

shell> curl -k -H 'Content-type: application/json' --request POST 'https://127.0.0.1:8096/api/v2/createAdminUser?i-am-sure=true' \
> --data-raw '{
>   "payloadType": "credentials",
>   "user":"tungsten",
>   "pass":"security"
> }'

For more information on using the new API, please refer to ???