B.1. Tungsten Replicator 5.2.1 Daily Builds (23 Sep 2017)

Builds after 23 Sep 2017 are based on the 5.2.1 commercial release of Tungsten Replicator. This includes a number of improvements, including to the trepctl and thl tools, and support for applying data into Cassandra.

Improvements, new features and functionality

  • Installation and Deployment

    • The autocomplete information in env.sh has been updated to support newer trepctl and thl commands.

      Issues: CT-292

  • Command-line Tools

    • The trepctl command has been updated to provide clearer and more detailed information on certain aspects of it's operation. Two new commands have been added, trepctl qs and trepctl perf:

      • The trepctl command has been updated to provide a simplified status output that provides an easier to understand status, using the qs command. For example:

        shell> trepctl qs
        State: alpha Online for 1172.724s, running for 124280.671s
        Latency: 0.71s from source DB commit time on thl://ubuntuheterosrc:2112/ into target database
         7564.198s since last source commit
        Sequence: 4860 last applied, 0 transactions behind (0-4860 stored) estimate 0.00s before synchronization
      • The trepctl perf command provides detailed performance information on the operation and status of the replicator and individual stages. This can be useful to identify where any additional latency or performance issues lie:

        shell> trepctl perf
        Statistics since last put online 1360.141s ago
        Stage | Seqno | Latency | Events | Extraction | Filtering | Applying | Other | Total
        remote-to-thl | 4860 | 0.475s | 70 | 116713.145s | 0.000s | 2.920s | 0.000s | 116716.065s
         Avg time per Event | 1667.331s | 0.000s | 0.000s | 0.042s | 1667.372s
        thl-to-q | 4860 | 0.527s | 3180 | 113842.933s | 0.011s | 2873.039s | 0.102s | 116716.085s
         Avg time per Event | 35.800s | 0.000s | 0.000s | 0.903s | 36.703s
        q-to-dbms | 4860 | 0.536s | 3180 | 112989.667s | 0.010s | 3701.035s | 25.554s | 116716.266s
         Avg time per Event | 35.531s | 0.000s | 0.008s | 1.164s | 36.703s

      Issues: CT-29

    • A number of improvements have been made to the identification of long running transactions within the replicator:

      • A new field has been added to the output of trepctl status -name tasks:

        timeInCurrentEvent : 6571.462

        This shows the time that the replictor has been processing the current event. For a long-running event, it helps to indicate that the replicator is still processing the curent event. Note that this is a just a counter for how low the current event has been running. For a replicator that is idle, this will show the time the replicator has spent both processing the original event and waiting to process the new event.

      • The thl list has been expanded to provide simple and detailed THL size information so that large transactions can be identified. Using the -sizes and -sizesdetail displays detailed information about the size of the SQL, number of rows, or both for each stored event. For example:

        shell> thl list -sizes
        SEQ# Frag# Tstamp
        12 0 2017-06-28 13:21:11.0 Event total: 1 chunks 73 bytes in SQL statements 0 rows
        13 0 2017-06-28 13:21:10.0 Event total: 1645 chunks 0 bytes in SQL statements 1645 rows
        14 0 2017-06-28 13:21:11.0 Event total: 1 chunks 36 bytes in SQL statements 0 rows

        For more information, see ??? and ???.

      • The trepctl command has been updated to provide more detailed information on the performance of the replicator, see trepctl perf.

      • For easier navigation and selection of THL events, the thl has had two further command-line options added, -first and -last to select the first and last events in the THL. Both also take an optional number that shows the first N or last N events.

      Issues: CT-34

  • Heterogeneous Replication

    • A new applier has been added to Tungsten Replicator that applies data directly into Cassandra. Data is loaded using a batch applier that writes the data through staging tables into Cassandra.

      Issues: CT-43

      For more information, see ???.

  • Filters

    • The thl has been improved to support -from and -to options for selecting the range. These act as synonyms for the existing -low and -high options and can be used with all commands.

      Issues: CT-111

Bug Fixes

  • Command-line Tools

    • The dsctl command has been updated:

      • The -ascmd option has been added to output the current position as a command that you can use verbatim to reset the status. For example:

        shell> dsctl get -ascmd
        dsctl set -seqno 17 -epoch 11 -event-id "mysql-bin.000082:0000000014031577;-1" -source-id "ubuntu"
      • The -reset option has been added so that the current position can be reset and then set using dsctl set -reset without having to run two separate commands.

      Issues: CT-24

    • An auto-refresh option has been added to certain commands within trepctl. By adding the -r option and the number of seconds to either trepctl status, trepctl qs, or trepctl perf commands. For example, trepctl qs -r 5 would refresh the quick status command every 5 seconds.

      Issues: CT-209

    • The Redshift applier where the relative directory used for the AWS configuration reference would look in the wrong location.

      Issues: CT-375