1.17. Tungsten Replicator 5.0.0 GA (7 December 2015)

Version End of Life. 30 June 2018

VMware Continuent for Replication 5.0.0 is a major release that incorporates the following changes:

  • The software release has been renamed. For most users of VMware Continuent for Replication, the filename will start with vmware-continuent-replication. If you are using an Oracle DBMS as the source and have purchased support for the latest version, the filename will start with vmware-continuent-replication-oracle-source.

    The documentation has not been updated to reflect this change. While reading these examples you will see references to tungsten-replicator which will apply to your software release.

  • New Oracle Extraction module that reads the Oracle Redo logs provided faster, more compatible, and more efficient method for extracting data from Oracle databases. For more information, see Oracle Replication using Redo Reader (in [Tungsten Replicator 5.0 Manual]).

  • Security, including file permissions and TLS/SSL is now enabled by default. For more information, see Deployment Security (in [Tungsten Replicator 5.0 Manual]).

  • License keys are now required during installation. For more information, see Deploy License Keys (in [Tungsten Replicator 5.0 Manual]).

  • Support for RHEL 7 and CentOS 7.

  • Basic support for MySQL 5.7.

  • Cleaner and simpler directory layout.

Upgrading from previous versions should be fully tested before attempted in a production environment. The changes listed below affect tpm (in [Tungsten Replicator 5.0 Manual]) output and the requirements for operation.

Behavior Changes

The following changes have been made to Continuent Replicator and may affect existing scripts and integration tools. Any scripts or environment which make use of these tools should check and update for the new configuration:

  • Tungsten Replicator now requires license keys in order to operate.

    License keys are provided to all customers with an active support contract. Login to my.vmware.com to identify your support contract and the associated license keys. After collecting the license keys, they should be placed into /etc/tungsten/continuent.licenses or /opt/continuent/share/continuent.licenses. The /opt/continuent (in [Tungsten Replicator 5.0 Manual]) path should be replaced with your value for --install-directory (in [Tungsten Replicator 5.0 Manual]). Place each license on a new line in the file and make sure it is readable by the tungsten system user.

    If you are testing VMware Continuent or don't have your license key, talk with your sales contact for assistance. You may enable a trial-mode by using the license key TRIAL. This will not affect the runtime operation of VMware Continuent but may impact your ability to get rapid support.

    The tpm (in [Tungsten Replicator 5.0 Manual]) script will display a warning if license keys are not provided or if the provided license keys are not valid.

  • Tungsten Replicator now enables security by default. Security includes:

    • Authentication between command-line tools (trepctl (in [Tungsten Replicator 5.0 Manual])) and background services.

    • SSL/TLS between command-line tools and background services.

    • SSL/TLS between Tungsten Replicator and datasources.

    • File permissions and access by all components.

    The security changes require a certificate file to be generated prior to operation. The tpm (in [Tungsten Replicator 5.0 Manual]) command can do that during upgrade if you are using a staging directory. Alternatively, you can create the certificate (in [Tungsten Replicator 5.0 Manual]) and update your configuration with the corresponding argument. This is required if you are installing from an INI file. See Installing from a Staging Host with Manually Generated Certificates (in [Tungsten Replicator 5.0 Manual]) or Installing via INI File with Manually Generated Certificates (in [Tungsten Replicator 5.0 Manual]) for more information. This functionality may be disabled by adding --disable-security-controls (in [Tungsten Replicator 5.0 Manual]) to your configuration.

    If you would like tpm (in [Tungsten Replicator 5.0 Manual]) to generate the necessary certificate from the staging directory. Run tpm update (in [Tungsten Replicator 5.0 Manual]) with the --replace-tls-certificate (in [Tungsten Replicator 5.0 Manual]) option.

    staging-shell> ./tools/tpm update --replace-tls-certificate

    For more information, see Deployment Security (in [Tungsten Replicator 5.0 Manual]).

  • For compatibility with MySQL 5.7, the tpm (in [Tungsten Replicator 5.0 Manual]) command will now check for the super_read_only setting and warn if this setting is enabled.

    Issues: CONT-1039

  • For compatibility with MySQL 5.7, the tpm (in [Tungsten Replicator 5.0 Manual]) command will use the authentication_string field for validating passwords.

    Issues: CONT-1058

  • For compatibility with MySQL 5.7, the tpm (in [Tungsten Replicator 5.0 Manual]) command will now ignore the sys schema.

    Issues: CONT-1059

  • Tungsten Replicator now includes RELEASE_NOTES in the package and displays a warning if they have not been reviewed.

    During some tpm (in [Tungsten Replicator 5.0 Manual]) commands, the script will check to see if the release notes have been reviewed and accepted. This may be done by running tools/accept_release_notes from the staging directory. The script will display the information and prompt the user for acceptance. A hidden file will be created on the staging server to mark the release notes have been accepted and the warning will not be displayed.

    This process may be automated by calling tools/accept_release_notes -y prior to installation. The script will mark the release notes as accepted and the warning will not be displayed.

    Issues: CONT-1122

Improvements, new features and functionality

  • Installation and Deployment

    • During installation, tpm (in [Tungsten Replicator 5.0 Manual]) writes the configuration log to /tmp/tungsten-configure.log. If the file exists, but is owned by a separate user the operation will fail with a Permission Denied error. The operation has now been updated to create a directory within /tmp (in [Tungsten Replicator 5.0 Manual]) with the name of the current user where the configuration log will be stored. For example, if the user is tungsten, the log will be written to /tmp/tungsten/tungsten-configure.log.

      Issues: CONT-1402

  • Oracle Replication

    • The replicator will automatically determine if the Oracle JDBC driver is available within $ORACLE_HOME/jdbc/lib or the current path, and will copy it into the distribution directory during installation if available.

      Issues: CONT-1344

Bug Fixes

  • Installation and Deployment

    • During installation, a failed installation by tpm (in [Tungsten Replicator 5.0 Manual]), running tpm uninstall could also fail. The command now correctly uninstalls even a partial installation.

      Issues: CONT-1359

Known Issues

  • Oracle Replication

    • The user configuration for Oracle users required when enabling Oracle extraction has a number of rules that must be followed to ensure valid replication:

      The replication user (configured with --replication-user (in [Tungsten Replicator 5.0 Manual])) has the following rules

      • The user should not contain data that will be replicated to other hosts.

      • If the user contains replicated data and filters are used, the results of replication cannot be guaranteed.

      • A different replication user must be used for each service extracting from the same Oracle Database.

      Issues: CONT-1403

Tungsten Replicator 5.0.0 Includes the following changes made in Tungsten Replicator 5.0.0

Behavior Changes

The following changes have been made to Release Notes and may affect existing scripts and integration tools. Any scripts or environment which make use of these tools should check and update for the new configuration:

  • The Bristlecone load generator toolkit is no longer included with Release Notes by default.

    Issues: CONT-903

  • The scripts previously located within the scripts directory have now been relocated to the standard bin directory. This does not affect their availability if the env.sh (in [Tungsten Replicator 5.0 Manual]) script has been used to update your path. This includes, but is not limited to, the following commands:

    • ebs_snapshot.sh

    • file_copy_snapshot.sh

    • multi_trepctl

    • tungsten_get_position

    • tungsten_provision_slave

    • tungsten_provision_thl

    • tungsten_read_master_events

    • tungsten_set_position

    • xtrabackup.sh

    • xtrabackup_to_slave

    Issues: CONT-904

  • The backup (in [Tungsten Replicator 5.0 Manual]) and restore (in [Tungsten Replicator 5.0 Manual]) functionality in trepctl (in [Tungsten Replicator 5.0 Manual]) has been deprecated and will be removed in a future release.

    Issues: CONT-906

  • The batch loading scripts used by HP Vertica, Hadoop and Amazon Redshift appliers have been moved to the appliers/batch directory.

    Issues: CONT-907

  • The location of the JavaScript filters has been moved to new location in keeping with the rest of the configuration:

    • samples/extensions/javascript has moved to support/filters-javascript

    • samples/scripts/javascript-advanced has moved to support/filters-javascript

    The use of these filters has not changed but the default location for some filter configuration files has moved to support/filters-config. Check your current configuration before upgrading.

    Issues: CONT-908

  • The ddlscan (in [Tungsten Replicator 5.0 Manual]) templates have been moved to the support/ddlscan directory.

    Issues: CONT-909

  • For compatibility with MySQL 5.7, the tpm (in [Tungsten Replicator 5.0 Manual]) command will now check for the super_read_only setting and warn if this setting is enabled.

    Issues: CONT-1039

  • For compatibility with MySQL 5.7, the tpm (in [Tungsten Replicator 5.0 Manual]) command will use the authentication_string field for validating passwords.

    Issues: CONT-1058

  • For compatibility with MySQL 5.7, the tpm (in [Tungsten Replicator 5.0 Manual]) command will now ignore the sys schema.

    Issues: CONT-1059

  • The Vertica applier should write exceptions to a temporary file during replication.

    The applier statements will include the EXCEPTIONS attribute in each statement to assist in debugging. Review the replicator log or trepctl status (in [Tungsten Replicator 5.0 Manual]) output for more details.

    Issues: CONT-1169

Known Issues

The following issues may affect the operation of Continuent Tungsten and should be taken into account when deploying or updating to this release.

  • Core Replicator

    • Use of LOAD DATA commands requires the correct permissions to be given to the mysql user. One of the following must be done.

      • The tungsten system user must have the same default group as the mysql system user.

      • The mysql system user must be a member of the default group for tungsten system user.

      • The --file-protection-level (in [Tungsten Replicator 5.0 Manual]) option must be set to none to allow full visibility to all temporary files.

    • The replicator can hit a MySQL lock wait timeout when processing large transactions.

      Issues: CONT-1106

    • The replicator can run into OutOfMemory when handling very large Row-Based replication events. This can be avoided by setting --optimize-row-events=false (in [Tungsten Replicator 5.0 Manual]).

      Issues: CONT-1115

    • The replicator can fail during LOAD DATA commands or Vertica loading if the system permissions are not set correctly. If this is encountered, make sure the MySQL or Vertica system users are a member of the Tungsten system group. The issue may also be avoided by removing system file protections with --file-protection-level=none (in [Tungsten Replicator 5.0 Manual]).

      Issues: CONT-1460

Improvements, new features and functionality

  • Command-line Tools

    • The dsctl (in [Tungsten Replicator 5.0 Manual]) has been updated to provide help output when specifically requested with the -h or -help options.

      Issues: CONT-1003

      For more information, see dsctl help Command (in [Tungsten Replicator 5.0 Manual]).

Bug Fixes

  • Core Replicator

    • A master replicator could fail to finish extracting a fragmented transaction if disconnected during processing.

      Issues: CONT-1163

    • A slave replicator could fail to come ONLINE (in [Tungsten Replicator 6.0 Manual]) if the last THL file is empty.

      Issues: CONT-1164

    • The replicator applier and filters may fail with ORA-955 because the replicator did not check for metadata tables using uppercase table names.

      Issues: CONT-1375

    • The replicator incorrectly assigns LOAD DATA statements to the #UNKNOWN shard. This can happen when the entire length is above 200 characters.

      Issues: CONT-1431