Tungsten Replicator

Tungsten Replicator 7.0.0

Build: 848

Release Date: 29 Mar 2022

End of Life Date: 28 Jun 2026

Release 7.0.0 is a major release introducing many new features including a fully documented API. There are a large number of bug fixes and improvements in all areas of the product, and a number of key behavior changes, most significantly being Security is now enabled by default.

Behavior Changes (15)

The following changes may affect existing scripts and integration tools. Any scripts or environment which make use of these tools should check and update for the new configuration:

Installation and Deployment (1)

An RPM update now issues --replace-release
Issue: CT-1708

Command-line Tools (8)

Changed output of thl purge command when no lower and upper bounds are given from 'Deleting events where' to 'Deleting all events'.
Issue: CT-1738
The tpm diag command now gathers the /etc/os-release file when located. Also now using the ip command on systems where ifconfig and/or route is deprecated.
Issue: CT-1496
The tpm diag command now uses the ss (socket status) command in place of netstat on SUSE and other operating systems that have deprecated netstat.
Issue: CT-1483
tpm now accepts chrony as a valid time synchronization software
Issue: CT-1462
The tungsten_find_orphaned script now creates a log file every time it runs which is stored in the configured temporary directory (/tmp/ by default, tpm query values temp_directory). This is to allow for easier troubleshooting and visibility during automatic execution.
Issue: CT-1447
The tungsten_post_process script functionality has been merged into the tpm post-process command. The tungsten_post_process script remains as a shell wrapper for tpm post-process.
Issue: CT-1314
- tungsten_set_position has been deprecated and no longer available in this release. dsctl should be used instead.
- tungsten_provision_slave has now been renamed to tprovision
Issue: CT-1302
The tpm diag command now uses tar czf instead of the zip command to compress the gathered files. The zip command is no longer a prerequisite for tpm diag.
Issue: CT-1253

Backup and Restore (3)

Additional messaging has been added to the output displayed when running tprovision.
Issue: CT-1689
tungsten_provision_slave has now been renamed to tprovision.
Issue: CT-1436
From v8 of xtrabackup, the --stream=tar option was removed, meaning that backups could fail if using the newer release of the Percona tools
In this release, the backups will now create the backup and then manually compress it
Warning
This change will increase the required disk space for backups to allow the post-backup compression to complete.
Issue: CT-1346

Core Replicator (1)

Replicator will now check after extracting a mysql STOP_EVENT whether a new binlog file was created, actually handling this event as a ROTATE EVENT.
This decreases the time that would be needed after STOP_EVENT to extract a new event.
Issue: CT-1349

Other Issues (1)

Log files for each component now have the same date and time stamp format.
Issue: CT-1669

API (1)

API calls triggering configuration changes are protected by a flag, i-am-sure=true, in order to avoid unwanted, potentially dramatic, configuration changes. This applies to:
- configuration/module/servicesmap
- reset
- offline
- online
- onhold
- addDataService
- addDataSource
Issue: CT-1317

Known Issue (1)

The following issues are known within this release but not considered critical, nor impact operation. They will be addressed in a subsequent patch release.

Installation and Deployment (1)

After starting up Tungsten components, a defunct process for each running component can be found in the process listing.
Whilst this does not cause any issues, it could generate unnecessary alerts for customers monitoring.
The cause has been identified and affects version 7.0.0 and 7.0.1. This will be fixed in the next 7.0.2 release.
Issue: CT-1876

Improvements, new features and functionality (32)

Installation and Deployment (3)

Support has been added for Java 17 LTS
Issue: CT-1706
Support now included for MariaDB 10.3+
Issues: CT-1433, CT-1276
Two new tpm options have been included as part of the new API in this release.
The options are used for setting the API admin user credentials and are as follows:
- rest-api-admin-user
- rest-api-admin-pass
Issue: CT-1327

Command-line Tools (11)

A new command (error) has been added to trepctl to output a full stack trace of the last error, if any.

shell> trepctl -service <serviceName> error

Event application failed: seqno=10 fragno=0 message=Table hr.regions not found in database. Unable to generate a valid statement.
com.continuent.tungsten.replicator.applier.ApplierException: Table hr.regions not found in database. Unable to generate a valid statement.
at com.continuent.tungsten.replicator.applier.JdbcApplier.getTableMetadata(JdbcApplier.java:582)
at com.continuent.tungsten.replicator.applier.JdbcApplier.fillColumnNames(JdbcApplier.java:494)
at com.continuent.tungsten.replicator.applier.JdbcApplier.getColumnInformation(JdbcApplier.java:1236)
at com.continuent.tungsten.replicator.applier.MySQLApplier.applyOneRowChangePrepared(MySQLApplier.java:418)
at com.continuent.tungsten.replicator.applier.JdbcApplier.applyRowChangeData(JdbcApplier.java:1460)
at com.continuent.tungsten.replicator.applier.JdbcApplier.apply(JdbcApplier.java:1576)
at com.continuent.tungsten.replicator.applier.ApplierWrapper.apply(ApplierWrapper.java:100)
at com.continuent.tungsten.replicator.pipeline.SingleThreadStageTask.apply(SingleThreadStageTask.java:871)
at com.continuent.tungsten.replicator.pipeline.SingleThreadStageTask.runTask(SingleThreadStageTask.java:601)
at com.continuent.tungsten.replicator.pipeline.SingleThreadStageTask.run(SingleThreadStageTask.java:185)
at java.base/java.lang.Thread.run(Thread.java:834)

Issue: CT-1747

The tapi command now supports the --affinity argument which will display all Connector-specific affinity settings, along with --connectorstatus to show all.
Issue: CT-1700
A new option to print the merged logs to STDOUT has been added to tungsten_merge_logs (--stdout|-O)
The tpm command suite now properly supports the --profile argument to specify a Tungsten json configuration file in place of the installed tungsten.cfg.
Issue: CT-1680
The tmonitor command now has better help text and more options to ease usage, including --filter to allow easy viewing of the tmonitor test output.
Issue: CT-1585
The tungsten_purge_thl command is now a wrapper for the tpm purge-thl command.
Issue: CT-1488
With the release of APIv2, a new cli tool has been introduced to allow easier access called tapi.
In addition, the vast majority of Tungsten cli tools have been updated to optionally use the APIv2 interface when desired.
The Nagios and Zabbix checks are also available via APIv2 using the tapi tool.
Issue: CT-1454
tungsten_send_diag now supports a new command-line argument, --cleanup, which will cause the removal of the diagnostic archive file generated using the --diag argument.
Issue: CT-1360
tpm diag now collects routing table information via route -n , and has two new command-line arguments: --include and --groups.
--include specifies a comma-separated list of subroutines to include. Any gather subroutine not listed will be skipped.
--groups specifies a comma-separated list of subroutine groups to include. Any group not listed will be skipped.
Issue: CT-1322
A new tpm option purge-thl and a new script tungsten_purge_thlhave been added to allow easier and more intelligent THL purging across all nodes in a topology.
This allow you to purge THL files based on the following rules:
- Gather the last applied seqno from all Replica nodes and take the lowest one
- Find the current THL file which contains that seqno, then locate the previous one
- Construct a thl purge command to remove thl thru the last seqno in the prev file
The default behavior is to display the needed commands for the admin to execute manually.
Issue: CT-1273
Prometheus exporters mysqld_exporter and node_exporter are now included with the distribution packages.
A new command line tool tmonitor is now available for the management and testing of external Prometheus exporters (node and mysqld), and for the testing of internal exporters (Manager, Connector and Replicator).
Issue: CT-960
A new tpm option delete-service is now available to simplify the removal of clusters and/or replicator services.
Issues: CT-210, CT-327, CT-1275

Backup and Restore (2)

Fixes an issue where tprovision could remove the SSL certs for MySQL on reprovision
Issue: CT-1323
Added support for mariabackup using the following configurable tpm options:
```
backup-method=mariabackup
or
backup-method=mariabackup-incremental
```
Issue: CT-1100

Core Replicator (7)

In-Flight THL Compression is now available.
For full details on enabling this feature, refer to "THL Encryption and Compression"
Issue: CT-1420
The replicator will now be able to handle new SQL_MODES available in later releases of MySQL and MariaDB, these are as follows:
- MySQL: TIME_TRUNCATE_FRACTIONAL
- MariaDB: TIME_ROUND_FRACTIONAL, SIMULTANEOUS_ASSIGNMENT
Issue: CT-1362
It is now possible to compress and/or encrypt THL on disk. For more information on using these features see"THL Encryption and Compression"
Issue: CT-630
Added a new property to allow configuration of character set used during batch loading into MySQL targets.
Default is utf8mb4. Override using the following:
```
property=replicator.applier.dbms.loadCharset=utf8
```
Issue: CT-1647
Added ability to the Batch Applier, to apply to a remote MySQL instance using LOAD DATA LOCAL INFILE.
This is disabled by default, and can be enabled by the following property:
```
property=replicator.applier.dbms.useLoadDataLocalInfile=true
```
Issue: CT-1622
When using the MySQL Batch Applier, you can now choose to log rows into an error table that would otherwise have generated a duplicate key error.
This is disabled by default, but can be enabled and configured by using the following two new properties:
```
property=replicator.applier.dbms.errorTablePrefix=error_xxx_ 
property=replicator.applier.dbms.batchLogDuplicateRows=true
```
In addition, if your tables contain Foreign Keys, you can choose to disable them during the data loading process using the following property:
```
property=replicator.applier.dbms.disableForeignKeys=true
```
Issue: CT-1589
When using the MySQL Batch Applier, you can now choose to drop all deletes by specifying the following tpm property:
```
property=replicator.applier.dbms.skipDeletes=true
```
This is disabled (false) by default.
Issue: CT-1588

PostgreSQL Replication (1)

Templates have been included for ddlscan to enable MySQL to Postgres Schema translation
Issue: CT-1736

Amazon Redshift Replication (1)

Adds a new option redshiftCopyOptions that can be supplied in the s3.json file to include valid options that can be included in the Redshift COPY command when loading csv data from S3 into Redshift staging tables. See example below.
```
{
  "awsS3Path" : "s3://your-bucket-for-redshift/redshift-test",
  "awsIAMRole" : "arn:iam-role",
  "redshiftCopyOptions" : "ACCEPTANYDATE ACCEPTINVCHARS TRUNCATECOLUMNS",
  "cleanUpS3Files" : "true"
}
```
Issue: CT-1298

Security (2)

Tungsten software now undergoes a rigorous security scan during QA. We also check included open-source/3rd-party software.
Issue: CT-1579
Tungsten can now install on default CentOS and RedHat 8 with tightened security settings thanks to support of TLSv1.3
Issue: CT-1359

Monitoring (1)

A number of new metrics have been added to the Prometheus exporters.
Issues: CT-1615, CT-1266

Platform Specific Deployments (1)

ARM 64 bit processor support added (linux aarch64)
Note
Note that at time of release, there is currently no xtrabackup binary available for ARM.
Issues: CT-1619, CT-1620

Other Issues (2)

IPv6 host addresses are now fully supported.
Can be enabled with the following configuration property:
```
prefer-ip-stack="6"
```
By default, IPv4 is enabled, which equates to the value of "4" in the above property.
Issue: CT-1537
General improvements in MySQL 8.0 support.
Issue: CT-1346

connector (1)

Added support for TLSv1.3.
Note that TLS 1.3 was introduced in jdk 8u272. So tpm will only set it by default for java 8u272 and later versions. It is still possible to force the protocol to be used via tpm flag tls-enabled-protocols=TLSv1.xxx
Issue: CT-1367

Bug Fixes (44)

Installation and Deployment (2)

When services are deployed with systemd and MySQL could not start due to an error, tpm would not be able to later start MySQL
Issue: CT-1734
Default systemd configuration files for Tungsten components no longer specifies the tungsten group to execute the command. This will prevent file access issues when tungsten user belongs to several groups.
Issue: CT-1550

Command-line Tools (24)

The tpm diag command now handles zero-length mysqld.log files gracefully.
Issue: CT-1740
tungsten_monitor.rb script no longer uses sudo to send emails if the configuration doesn't allow it.
Issue: CT-1737
The tpm command now communicates properly when there is no INI configuration file or staging-method deploy.cfg configuration defined.
Issue: CT-1712
tpm now parses the MYSQL SSL related setting correctly.
Issue: CT-1662
When MySQL services were badly installed, some distribution could show a “not-found” status within systemctl, confusing tpm
Issue: CT-1677
Fixes an issue where the deployall command would create a root owned wrapper.log in the ./tools directory.
Issue: CT-1664
The tpm ask command no longer calls Data::Dumper when it is not available.
Issue: CT-1626
The tungsten_find_orphaned command no longer fails with an 'Can't exec "/bin/sh": Argument list too long' error when there are too many THL files to parse.
Issue: CT-1545
The tpm update command now handles updates/upgrades more gracefully when the previous version did not have the latest tpm framework.
Issue: CT-1506
Database monitoring logs are now reporting the correct error number and SQL state when database errors occur.
Issue: CT-1497
The tpm diag command now properly collects the system information file on Debian systems.
Issue: CT-1492
The tpm purge-thl command now handles command-line arguments more intelligently.
Issue: CT-1489
All tpm sub-commands now handle command-line arguments more intelligently.
Issue: CT-1487
The tmonitor command no longer fails on Debian 9 and Ubuntu.
Issue: CT-1485
tprovision (formerly tungsten_provision_slave) may fail to provision if the MySQL data directory was not accessible to the tungsten user.
Issue: CT-1475
Fixes a monitoring bug with users using caching_sha2_password.
Issue: CT-1406
The tpm command no longer aborts with a Use of uninitialized value errorwhen a stray tungsten.cfg file exists under $CONTINUENT_ROOT
Issue: CT-1394
The tpm command no longer prints an error when run with no other command-line arguments.
Issue: CT-1373
tungsten_send_diag no longer prints an error about Use of uninitialized value $diagArgs in concatenation.
Issue: CT-1354
Fixes a security issue within the tpm diag command.
Issue: CT-1295
The tpm update command will now exit with an error if any files not owned by the configured Tungsten OS user are found in the Tungsten installation directory.
For example, if the OS user is tungsten and the installation directory is /opt/continuent, containing the file /opt/continuent/thl/archived_thl.zip owned by root would cause something like the following error to be produced, and tpm update would exit:
```
Foreign-owned files found!

Located files in the Tungsten installed directory /opt/continuent
that are not owned by the Tungsten OS user (tungsten):

/opt/continuent/thl/archived_thl.zip

Please change the ownership of these files to OS user "tungsten"
using the chown command as root via sudo, then rerun the `tpm update` command.

For example:
shell> sudo chown -R tungsten:tungsten /opt/continuent
```
Issue: CT-1260
Fixes a bug in tprovision when using xtrabackup version 8, due to changes in xtrabackup binaries.
Issue: CT-1248
In certain cases, tprovision would not be able to find the binary log position of the backup when taken from a primary. This has been fixed.
Issue: CT-1085
The tpm command now checks the local host’s shortname in addition to the FQDN returned by the hostname command upon initialization.
This fixes a bug that prevented tpm mysql from working on applier hosts running MySQL.
Issue: CT-1414

Backup and Restore (1)

Fixed an issue where an xtrabackup generated by the replicator would fail to be restored using trepctl restore command.
Issue: CT-1575

Core Replicator (10)

Added more debug information for detecting possible hanging connections while a THL client connects to the THL server. Also, added socket timeout for the connection initialization
Issue: CT-1760
Fixed an issue where trepctl was leaving JMX connections opened.
Issue: CT-1752
Fixed an issue when using Parallel apply that would show a NullPointerException in case an event could either not be found or be corrupted in THL. This will now display a correct message Missing or corrupted event from storage
Issue: CT-1722
When connecting to a THL server, a client will now connect to the next available host in its THL URI, if the first does not have the sequence number that the client requires. The client will then fail only if none of the hosts from the URI can provide the needed sequence number.
Issue: CT-1558
For row based events, SQL modes were not displayed in the THL output. This is now fixed.
Issue: CT-1440
Fixes occurrences of NullPointerException that would occur when bringing the replicator online before MySQL was started.
Issue: CT-1348
A change in the way MySQL logs CREATE TABLE AS SELECT in the Binary Logs from v8.0.20 onwards, meant these transactions would previously fail.
Warning
Whilst these statements will now replicate, it must be noted that in the event of a failure during the data load, the initial CREATE statement won't be rolled back, and therefore care must be taken when using this type of DDL.
Note
This only affects customers using MySQL v8.0.20+ running with ROW based replication. An alternative workaround to ensure correct rollback on failure, would be to run the statement with STATEMENT based replication for the session. This will also provide better performance for larger tables.
Issue: CT-1301
When using parallel apply, the replicator would error with a Foreign Key constraint error if statements were issued against two or more objects that shared the same name, but with different case sensitivity, for example:
```
mysql> create table testtable;
mysql> drop table testtable;
mysql> create table TestTable;
```
Issue: CT-1259
The replicator metadata cache will now correctly handle table names whenlower_case_table_names=1 is set in the MySQL configuration.
Issue: CT-651
Fixed an issue with the batch loader where DATETIME and TIME datatypes with milliseconds would be replicated as .000 instead of the real millisecond value.
Issue: CT-1739

Heterogeneous Replication (1)

Fixes an issue where a heterogeneous (non-MySQL) applier would try to run MySQL syntax to set the timezone in use.
Issue: CT-1324

Filters (2)

The BidiRemoteSlaveFilter could fail to correctly flag fragmented events in unprivileged environments (Aurora, for example) In such an environment (multi-active, unprivileged database access), a new setting was introduced to force extraction process to read ahead to the last fragment to detect the service name (false by default). Enabled with repl-svc-extractor-multi-frag-service-detection=true
Issue: CT-1351
Includes previously missing template file to enable easy configuration of the dbrename filter.
Issue: CT-1350

Security (1)

log4j libraries updated to v2.17.1 specifically to mitigate risk of exposure to the 0-day vulnerability detected in log4j v2.14
Issue: CT-1703

oracle (2)

Fixes a bug in the MySQL to Oracle ddlscan templates that prevented them from executing
Issues: CT-1735, CT-1756
Fixes an issue causing an error when using the ddl-mysql-oracle.vm template with ddlscan to convert MySQL schema DDL into Oracle DDL
Issue: CT-1326

cmdlie (1)

Issuing dsctl set [--reset] against Aurora source and targets would fail with Access Denied errors. This is due to the inability to disable Binary Logging for specific transactions which dsctl issues. The errors did not prevent dsctl from working, however they were misleading.
Issue: CT-1040