1.14. Tungsten Replicator 5.2.0 GA (19 July 2017)

Version End of Life. 31 January 2019

Tungsten Replicator 5.2.0 is a new feature release that contains a combination of new features, specifically new replicator applier targets:

This release also provides improvements to the trepctl (in [Tungsten Replicator 5.2 Manual]) and thl (in [Tungsten Replicator 5.2 Manual]) commands, and bug fixes to improve stability.

Improvements, new features and functionality

  • Command-line Tools

    • The trepctl (in [Tungsten Replicator 5.2 Manual]) command has been updated to provide clearer and more detailed information on certain aspects of it's operation. Two new commands have been added, trepctl qs (in [Tungsten Replicator 5.2 Manual]) and trepctl perf (in [Tungsten Replicator 5.2 Manual]):

      • The trepctl (in [Tungsten Replicator 5.2 Manual]) command has been updated to provide a simplified status output that provides an easier to understand status, using the qs (in [Tungsten Replicator 5.2 Manual]) command. For example:

        shell> trepctl qs
        State: alpha Online for 1172.724s, running for 124280.671s
        Latency: 0.71s from source DB commit time on thl://ubuntuheterosrc:2112/ into target database
         7564.198s since last source commit
        Sequence: 4860 last applied, 0 transactions behind (0-4860 stored) estimate 0.00s before synchronization
      • The trepctl perf (in [Tungsten Replicator 5.2 Manual]) command provides detailed performance information on the operation and status of the replicator and individual stages. This can be useful to identify where any additional latency or performance issues lie:

        shell> trepctl perf
        Statistics since last put online 1360.141s ago
        Stage | Seqno | Latency | Events | Extraction | Filtering | Applying | Other | Total
        remote-to-thl | 4860 | 0.475s | 70 | 116713.145s | 0.000s | 2.920s | 0.000s | 116716.065s
         Avg time per Event | 1667.331s | 0.000s | 0.000s | 0.042s | 1667.372s
        thl-to-q | 4860 | 0.527s | 3180 | 113842.933s | 0.011s | 2873.039s | 0.102s | 116716.085s
         Avg time per Event | 35.800s | 0.000s | 0.000s | 0.903s | 36.703s
        q-to-dbms | 4860 | 0.536s | 3180 | 112989.667s | 0.010s | 3701.035s | 25.554s | 116716.266s
         Avg time per Event | 35.531s | 0.000s | 0.008s | 1.164s | 36.703s

      Issues: CT-29

    • A number of improvements have been made to the identification of long running transactions within the replicator:

      • A new field has been added to the output of trepctl status -name tasks (in [Tungsten Replicator 5.2 Manual]):

        timeInCurrentEvent : 6571.462

        This shows the time that the replictor has been processing the current event. For a long-running event, it helps to indicate that the replicator is still processing the curent event. Note that this is a just a counter for how low the current event has been running. For a replicator that is idle, this will show the time the replicator has spent both processing the original event and waiting to process the new event.

      • The thl list (in [Tungsten Replicator 5.2 Manual]) has been expanded to provide simple and detailed THL size information so that large transactions can be identified. Using the -sizes (in [Tungsten Replicator 5.2 Manual]) and -sizesdetail (in [Tungsten Replicator 5.2 Manual]) displays detailed information about the size of the SQL, number of rows, or both for each stored event. For example:

        shell> thl list -sizes
        SEQ# Frag# Tstamp
        ...
        12 0 2017-06-28 13:21:11.0 Event total: 1 chunks 73 bytes in SQL statements 0 rows
        13 0 2017-06-28 13:21:10.0 Event total: 1645 chunks 0 bytes in SQL statements 1645 rows
        14 0 2017-06-28 13:21:11.0 Event total: 1 chunks 36 bytes in SQL statements 0 rows

        For more information, see thl list -sizes Command (in [Tungsten Replicator 5.2 Manual]) and thl list -sizesdetail Command (in [Tungsten Replicator 5.2 Manual]).

      • The trepctl (in [Tungsten Replicator 5.2 Manual]) command has been updated to provide more detailed information on the performance of the replicator, see trepctl perf (in [Tungsten Replicator 5.2 Manual]).

      • For easier navigation and selection of THL events, the thl (in [Tungsten Replicator 5.2 Manual]) has had two further command-line options added, -first (in [Tungsten Replicator 5.2 Manual]) and -last (in [Tungsten Replicator 5.2 Manual]) to select the first and last events in the THL. Both also take an optional number that shows the first N or last N events.

      Issues: CT-34

    • A new command, tungsten_send_diag (in [Tungsten Replicator 5.2 Manual]), has been added that provides a simplified method for sending a tpm diag (in [Tungsten Replicator 5.2 Manual]) output automatically through to the support team. The new command uploads the diagnostic information directly in Amazon S3 without requiring a separate upload to Zendesk.

      Issues: CT-158

    • A new command, clean_release_directory (in [Tungsten Replicator 5.2 Manual]) has been added to the distribution. This command removes old releases from the installation directory that have been created during either upgrades or configuration updates. The command removes all old entries except the current active one, and the last five entries.

      Issues: CT-204

  • Heterogeneous Replication

    • A new applier has been added to Tungsten Replicator that applies data directly into Cassandra. Data is loaded using a batch applier that writes the data through staging tables into Cassandra.

      Issues: CT-43

      For more information, see Deploying MySQL to Cassandra Replication (in [Tungsten Replicator 5.2 Manual]).

    • A new applier has been added to Tungsten Replicator that applies data directly into Kafka. Incoming row data is converted into a JSON document which is then embedded into a Kafka message and sent on a topic using the schema and table name.

      Issues: CT-101

      For more information, see Deploying MySQL to Kafka Replication (in [Tungsten Replicator 5.2 Manual]).

    • Tungsten Replicator has been certified compatible with Vertica 8 using the existing vertica6.js (in [Tungsten Replicator 5.2 Manual]) batch-loading script.

      Issues: CT-152

    • A new applier has been added to Tungsten Replicator that applies data directly into Elasticsearch. Incoming row data is converted into a JSON document and then uploaded directly into an Elasticsearch index and type according either to explicitly settings, or based automatically on the schema and table name.

      Issues: CT-220

      For more information, see Deploying MySQL to Elasticsearch Replication (in [Tungsten Replicator 5.2 Manual]).

  • Filters

    • The filter functionality has been improved and standardised as a continuing effort to make the filters more usable. At the moment, the effect is embedded into the new filters in this release (SkipEventByType (in [Tungsten Replicator 5.2 Manual]) and ConvertStringFromMySQLFilter (in [Tungsten Replicator 5.2 Manual])). These new filters do make use of a new configuration file system and format based on JSON that will eventually become the standard method to configure all filters.

      Issues: CT-214

      For more information, see ConvertStringFromMySQL Filter (in [Tungsten Replicator 5.2 Manual]), SkipEventByType Filter (in [Tungsten Replicator 5.2 Manual]).

    • A new filter, SkipEventByType (in [Tungsten Replicator 5.2 Manual]), has been added. This allows for events to be skipped based on their operation type (INSERT, UPDATE, DELETE). This can be applied on a schema and/or table basis, alongside a default option that will be applied to all schema/table combinations not explicitly specified.

      Issues: CT-216

      For more information, see SkipEventByType Filter (in [Tungsten Replicator 5.2 Manual]).

    • A new filter, ConvertStringFromMySQLFilter (in [Tungsten Replicator 5.2 Manual]), has been added. This allows for conversion of data extracted and stored in the native MySQL environment (where --mysql-use-bytes-for-string=false (in [Tungsten Replicator 5.2 Manual])). This is particularly useful in situations where data is being replicated out of an existing cluster (where bytes are used by default), but the data is being replicated to a heterogeneous target.

      Issues: CT-217

      For more information, see ConvertStringFromMySQL Filter (in [Tungsten Replicator 5.2 Manual]).

  • Documentation

    • The documentation has been updated to make the use of the --property (in [Tungsten Replicator 5.2 Manual]) option to tpm (in [Tungsten Replicator 5.2 Manual]).

      Issues: CT-180

Bug Fixes

  • Command-line Tools

    • The tungsten_provision_slave (in [Tungsten Replicator 5.2 Manual]) command could hang during the execution of an external command which could cause the entire process to fail to complete properly.

      Issues: CT-82

    • When a replicator has been configured a cluster slave, the masterListenUri (in [Tungsten Replicator 5.2 Manual]) would be blank. This was because a pure cluster-slave configuration did not correctly configure the necessary pipelines.

      Issues: CT-197

    • The query (in [Tungsten Replicator 5.2 Manual]) tool has been updated to provide better error handling and messages during an error. This particularly affects tools which embed the use of this command, such as tungsten_provision_slave (in [Tungsten Replicator 5.2 Manual]).

      Issues: CT-203

    • An auto-refresh option has been added to certain commands within trepctl (in [Tungsten Replicator 5.2 Manual]). By adding the -r (in [Tungsten Replicator 5.2 Manual]) option and the number of seconds to either trepctl status (in [Tungsten Replicator 5.2 Manual]), trepctl qs (in [Tungsten Replicator 5.2 Manual]), or trepctl perf (in [Tungsten Replicator 5.2 Manual]) commands. For example, trepctl qs -r 5 (in [Tungsten Replicator 5.2 Manual]) would refresh the quick status command every 5 seconds.

      Issues: CT-209