A.2. Tungsten Replicator 5.1.0 GA (26 Apr 2017)

Tungsten Replicator 5.1.0 is a minor feature release and constains some significant improvements in the compatiblity and stability for Hadoop loading, JavaScript filters, heterogeneous filter compatibility and important bug fixes.

Improvements, new features and functionality

  • Installation and Deployment

    • The list of supported Ruby versions has been updating to support Ruby up to and including Ruby 2.4.0.

      Issues: CT-138

  • Heterogeneous Replication

    • The support for loading into Hadoop has been improved with better compatibility for recent Hadoop releases from the major Hadoop distributions.

      • MapR 5.2

      • Cloudera 5.8

      In addition to ensuring the basic compatibility of these tools, the continuent-tools-hadoop has been updated to support the use of the beeline as well as the hive command.

      Issues: CT-153, CT-155

      For more information, see Section 9.8, “The load-reduce-check Tool”.

    • The replicator and load-reduce-check command that is part of the continuent-tools-hadoop repository has been updated so that it can support loading and replication into Hadoop from Oracle. This includes creating suitable DDL templates and support for accessing Oracle via JDBC to load DDL information.

      Issues: CT-168

  • Filters

    • The JavaScript environment has been updated to include a standardized set of filter functionality. This is proivided and loaded as standard into all JavaScript filters. The core utilities are provided in the coreutils.js file.

      The current file provides three functions:

      • load — which loads an external JavaScript file.

      • readJSONFile — which loads an external JSON file into a variable.

      • JSON — provides JSON class including the ability to dump a JavaScript variable into a JSON string.

      Issues: CT-99

    • The thl has been improved to support -from and -to options for selecting the range. These act as synonyms for the existing -low and -high options and can be used with all commands.

      Issues: CT-111

    • A number of filters have been updated so that the THL metadata for the transaction includes whether a specific filter has been applied to the transaction in question. This is designed to make it easier to determine whether the filter has been applied, particularly in heterogeneous replication, and also to determine whether the incoming transaction are suitable to be applied to a targert that requires them. Currently the metadata is only added to the transactions and no enforcement is made.

      The following filters add this information:

      The format of the metadata is tungsten_filter_NAME=true.

      Issues: CT-157

Bug Fixes

  • Installation and Deployment

    • The rubygems extension to Ruby was loaded correctly causing some tools to fail to load correctly, or fail to use the Net/SSH tools correctly.

      Issues: CT-143

    • One of the checks built into tpm, MySQLUnsupportedDataTypesCheck was spelt incorrectly, which meant that it was difficult to bypass and ultimately did not always correctly run or get ignored.

      Issues: CT-147

    • The tpm update command could fail when using Ruby 1.8.7.

      Issues: CT-165

  • Command-line Tools

    • The tungsten_provision_slave could fail if the innodb_log_home_dir and innodb_data_home_dir were set to a value different to the datadir option, and the --direct was used.

      Issues: CT-83, CT-141

  • Heterogeneous Replication

    • The Hadoop loader would previously load CSV files directly into the /users/tungsten within HDFS, completely ignoring the setting of thr replication user within the replicator. This has been corrected so that data can be loaded into the configured replication user.

      Issues: CT-134

    • By default the the Hadoop loader would default to use a directory structure that matched the SERVICENAME/SCHEMANAME/TABLENAME. This cause problems with the default DDL templates and the continuent-tools-hadoop tools which used only the schema and table name.

      Issues: CT-135