The tpm diag command now gathers the mount and hostname commands.

Issues: CT-2314

Added the -last and -first options to thl dsctl command

Issues: CT-2318

The tpm diag command now gathers the output of crontab -l.

Issues: CT-2344

The tungsten_find_orphaned command now accepts the --sql option when using the --epoch option for Scenario 2. This will output the full SQL statements instead of just the headers. For use with --epoch only. Not available with ROW-based replication, only with STATEMENT or MIXED.

Issues: CT-2345

The tpm diag command now gathers service_logs/tprovision.log and service_logs/trestore.log, if they exist

Issues: CT-2385

The tpm diag command now gathers show global status; from the MySQL server.

Issues: CT-2387

A new maskData filter has been added, allowing basic level data obsfucation capabilities.

Issues: CT-849

The tpm check ini command now properly handles property= INI entries.

Issues: CT-2352

The tungsten_mysql_ssl_setup command no longer tries to ls any .p12 file unless --p12 has been given on the cli

Issues: CT-2357

The tpm copy --doini command now functions properly from the staging directory before the initial installation of Tungsten.

Issues: CT-2362

The tpm cert command now easily imports certificate bundles into various files, and correctly reads a file name as the proper typeSpec in various subcommands.

Issues: CT-2369

The tpm check ini command now honors the --short and --valid options

Issues: CT-2373

Fixed a bug preventing the auto-recovery feature to put the service back online if it failed to connect to database during a retry.

Issues: CT-2377

Fixed an issue where the replicator was not able to build a valid DELETE statement, when enable-heterogeneous-service was set to true.

Issues: CT-2379

Fixed a bug in latency calculations that in some circumstances could appear as though the latency on the applier stages was "flapping".

Issues: CT-2386

Added support for Ruby 3.2.

Issues: CT-2296

Oracle Linux 9 has been certified for Tungsten products from version 7.1.3.

Issues: CT-2370

Improved help text in the tpm cert command

Issues: CT-2222

A new command has been added, tpm check ini, that will validate all options present in a tungsten.ini file. This command can be run prior to installation.

Issues: CT-2240

tpasswd utility gets a new switch -C or --compare.to allowing password store file comparison for tpm update purpose.

Issues: CT-2278

Added new tpm check ports command which will check port connectivity for standard Tungsten ports to a specified host

Issues: CT-2292

In proxy mode, kill queries are now intercepted by the connector in order to replace the given thread_id by the underlying database connection thread_id.

Issues: CT-2298

The tpm update command now passes the verbose and debug flags properly to tpm delete-service.

Issues: CT-2299

The tpm cert and tpm copy commands now work from the staging directory before Tungsten is installed.

Issues: CT-2181

All tpm commands now use the --force and -f arguments consistently.

Issues: CT-2224

Fixes a regression in tpm cert that prevented BASE_DIR in tungsten.env from being used properly.

Issues: CT-2237

The tpm cert command now works correctly with the share/tungsten.env configuration file prior to installation.

Issues: CT-2255

The tpm report command now displays channel 2 output properly when in Bridge mode.

Issues: CT-2294

The zabbix_tungsten_online monitoring tool now properly supports active witnesses.

Issues: CT-2300

The check_tungsten_progress command now behaves properly on witness nodes by exiting with an error CRITICAL: Replicator is not installed on Witness nodes.

Issues: CT-2301

A new INI option has been added rest-api-admin-password which acts as an alias for the existing rest-api-admin-pass option.

Issues: CT-2246

A new command, tungsten_mysql_ssl_setup, has been introduced to create all needed security files for the MySQL database server. tungsten_mysql_ssl_setup will act as a direct replacement for the mysql_ssl_rsa_setup command which is not included with either Percona Server or MariaDB. This command will now be called by tpm cert gen mysqlcerts instead of mysql_ssl_rsa_setup.

Issues: CT-2188

The tpm diag command now captures the output of sestatus and getenforce.

Issues: CT-2243

A new filter, binarystringconversionfilter, was added to provide a way to convert varchar data from one charset into another.

Issues: CT-981

The tpm update command now properly recognizes the INI option dataservice-composite-datasources as an alias for composite-datasources.

Issues: CT-2241

Fixed an issue where the shard name of an event could be badly identified, which could potentially lead to deadlocks when parallel applier was enabled.

Issues: CT-2266

Fixed an issue caused by a MySQL behavior change in how a DROP TABLE is logged in the binlog, that made the replicator unable to identify correctly if the query had tungsten metadata in it.

Issues: CT-2269

Event extraction can sometimes generate an empty fragment (for a multi fragment transaction). This in turn can lead to bad shard detection. While empty fragment is expected, shard detection will now use the last non-empty fragment.

Issues: CT-2284

Fixed an issue where some applications won't be able to connect to the connector under bridge mode and ip filtering in place through authorized_hosts.

Issues: CT-2261

Fixed a bug where any new connection to the connector would trigger a call to hostname, potentially exhausting CPU resources.

Issues: CT-2285

Fixes an issue with data source monitoring when client and server don't share the same default authentication plugins.

Issues: CT-2233

The tpm diag command no longer errors out on witness hosts installed with the Staging method.

Issues: CT-2262

Fixes a bug that would cause a switch to fail if the cluster had been idle for more than 15 minutes.

Issues: CT-2263

The tapi command now supports the ?no-connectors=1 argument in APIv2 via the new cli argument --no-connectors

Issues: CT-2174

The tpm mysql command now accepts the --force argument to function on Witness nodes that have a running database.

Issues: CT-2180

Installations will now succeed on hosts running Ruby 3.x

Issues: CT-2196

The tpm report command now correctly displays security info tpm options as blank when none exists.

Issues: CT-2176

The tpm update command no longer fails to display foreign-owned dot-files and directories.

Issues: CT-2190

The tpm mysql command now properly accepts the -e argument as a shortcut for the --execute mysql cli client argument.

Issues: CT-2217

Fixed a thread synchronization issue that can cause memory leaks.

Issues: CT-2197

v6 API configuration properties within the INI file, are now also recognised by v7.

Issues: CT-2231

Fixed a bug when upgrading from 6.x to 7.x that prevented switches/failovers from working. Also the change of datasource-group-id in the INI will now be correctly reflected after a tpm update.

Issues: CT-2236

Running tpm uninstall will now save all of the Tungsten database tracking schemas for later use. There is also a new tpm keep command, which allows the tracking schemas to be saved to disk at any time in multiple formats (.json, .dmp and .cmd)

Issues: CT-2131

Added tpm flag deploy-systemd as a more meaningful alias to install

Issues: CT-2152

Added a new option --preserve-schema to the tpm uninstall command in order to leave the tracking schema in the database.

Issues: CT-561

A new command tpm cert has been added to aid in the creation, rotation and management of certificates for all areas of Tungsten.

Known limitations: Percona 5.6, and all versions of MariaDB do not provide the mysql_ssl_rsa_setup command required by tpm cert gen mysqlcerts.

Issues: CT-2085

The tpm report command now displays the security-specific information for each channel shown, including file paths and tpm options. Security information and tpm options for each channel will also be shown when --extra is used with --list.

Also added the tpm ask certs command with expiry and sha2566 info per alias, along with the tpm ask certtpm and tpm ask certlocations commands with reference information about the security.properties file.

Added help text for tpm ask --long which shows the key/variable name along with the value. The tmonitor -t test command now shows only the actual Tungsten-specific metrics lines.

Added tmonitor -T test to show metrics help and headers along with actual tungsten_* metrics lines.

Issues: CT-2088

tungsten show processlist could error with a disconnection message listing connections disconnected on the mysql server side.

Issues: CT-2112

The tpm diag command now captures the output of cctrl> cluster topology validate

Issues: CT-2115

The tpm diag command now supports the --skipsudo and --nosudo arguments to prevent operations from using the sudo command. Using this option may result in tpm diag skipping/failing various gathers due to a lack of access.

Issues: CT-2146

The tungsten_send_diag command has a new argument --all which will tell the tpm diag command to gather all hosts with -a, and this replaces the previous method for gathering all hosts for tungsten_send_diag, --args '--all'

Issues: CT-2150

MySQL clone can now be used as an option for recovery using tprovision.

Issues: CT-1417

tungsten_get_mysql_datadir can now return additional mysql database directories.

Issues: CT-1985

tprovision will now add a heartbeat after restore. This will ensure the replicator can be put back online when there is no load on the cluster.

Issues: CT-2005

The tprovision script will sleep for 5 seconds by default when using the rsync method after issuing a flush logs. The sleep value is configurable as a command line option --flush-after-sleep.

Issues: CT-2101

A new shardbyrules filter has been added that will allow rule based sharding of replication based on user confgurable rules that would allow sharding at table level, whereas previoulsy sharding would only be handled at schema level.

For more information, see shardbyrules Filter Documentation

Issues: CT-2164

Tungsten Connector now supports dual passwords, mirroring the MySQL v8.0.14+ functionality. When changing a user password, the previous password can be retained as long as needed in order to allow changing account passwords with no downtime.

Issues: CT-2127

Added a flag to avoid generating and sending EOF packets to client applications when CLIENT_DEPRECATE_EOF is not set on both client and mysql server sides. This fixes an issue with Go-MySQL-Driver and prepared statements.

When using Go MySQL Driver, flag --connector-generate-eof=false should be specified. Default is set to true for backwards compatibility.

Issues: CT-2177

Upgraded JGroups library to 4.2.22

Issues: CT-2011

There is a new datasource-group-id TPM option. In a single cluster the nodes with the same datasource-group-id will form a Distributed Datasource Group (DDG).

Issues: CT-2051

Prometheus exporters now provide the ssl cert expiration date as an epoch value in addition to the label.

Issues: CT-2099

Added Prometheus exporter metrics for composite parent and sub-services.

Issues: CT-2121

Prometheus libraries have been upgraded from version 0.8.1 to 0.16.0

Issues: CT-2166

The ability to configure the Java Virtual Machine (JVM) settings for the manager has been made easier, by the use of the manager_java_settings.conf file. For more information see Adjusting JVM Settings for the Manager

Issues: CT-2167

Fixed RPM package script to run tpm install instead of tpm update when installing the rpm

Issues: CT-2130

The tpm command now handles situations when the Manager process is not running.

Issues: CT-2103

tpm uninstall would sometimes print "ERROR >> db1 >> undefined method '+' for nil:NilClass"

Issues: CT-2104

The tungsten_find_orphaned command now handles some edge cases more gracefully.

Issues: CT-2107

The cctrl cluster topology validate command now checks, in a CAA setup, if the relay in the subservice is on the same host as the primary, and reports if there is a mismatch.

Issues: CT-2114

The tpm command now searches more places to locate shell commands that are called, especially useful when $CONTINUENT_ROOT/share/env.sh is not sourced.

Issues: CT-2182

tprovision would produce errors if the local hostname were different from the hostname used in the Tungsten install (short vs long names).

Issues: CT-1363

tprovision will now print an error message and exit if the MySQL datadir does not exit.

Issues: CT-1901

tprovision would accept bogus options and not produce an error. This has now been fixed.

Issues: CT-2045

tprovision will now timeout if ssh is blocked from the target to the source host.

Issues: CT-2139

Using mysqldump for tprovision could incorrectly create a new SSL key pair.

Issues: CT-2142

Improved a query that is run by Tungsten when fetching tables metadata (column names, datatypes, etc). While it is not generally needed, the unoptimized query can run badly (especially) against old mysql versions with a lot of databases / tables. For now, the new optimized query is not used by default, but this could change in some future version.

This can be enabled by using the following property :

property=replicator.datasource.global.connectionSpec.usingOptimizedMetadataQuery=true

Issues: CT-2077

Fixed an issue while processing geometry data with SRID 4326 that would swap longitude and latitude. This applies only to MySQL 8, as prior MySQL versions do not allow specifying the order when applying a WKB (Well-known binary) to MySQL

Issues: CT-2172

Fixed NullPointerExceptions when reading packets from disconnected client connections.

Issues: CT-2132

Fixed a log4j configuration issue where the connector-audit.log and connector-api.log files that have been rotated will end up in cluster-home/bin/directory instead of tungsten-connector/log/.

Issues: CT-2137

Fixed an issue with topology validation when running tpm promote-connector on a witness host

Issues: CT-2161

Fixed an issue where a failover could hang in rare cases when security and Proxy mode are enabled. SSL connections to a failed data source could enter a deadlock and block failover for up to 15 minutes.

Issues: CT-2183, CT-2187

In a Composite Active/Active topology, issuing SHUN/DRAIN or WELCOME on a node in cctrl would only affect the node in the main cluster service. This action will now also be applied to the same node within the x_from_y sub-service.

Issues: CT-2145

A set of changes to improve cctrl responsiveness

Issues: CT-1795

A new thl tail command has been added, allowing you to view the live THL changes as they are generated.

Issues: CT-405

A new command, tpm copy-keys, has been added to assist with the distribution of security-related files after an INI-based installation. This tool requires ssh access between the cluster nodes for the copy to work.

Issues: CT-1910

The tungsten_get_ports command now supports the /opt/continuent/tungsten/tungsten-connector/conf/interfaces.json file, if found.

Issues: CT-2003

The tungsten_get_ports command now use values based on values obtained from tpm query values when possible for the greatest output accuracy.

Issues: CT-2008

The tpm ask summary command has two new keys, isSmartScale and isDirect, which are also available individually on the command line.

Issues: CT-2035

When in Connector Proxy mode, the tpm report command now displays the Direct and SmartScale status via Channel 2: Connector to Database

Issues: CT-2037

Shell command tab completion for Tungsten-specific commands has been updated to support recent additions and changes.

Issues: CT-2057

THL transfer protocol can now be changed dynamically. This can be done using the trepctl command tool

shell> trepctl setdynamic -property replicator.thl.protocol.client.serialization -value DEFLATE 
Property replicator.thl.protocol.client.serialization successfully set to DEFLATE

This can also be done through API v2

curl --user tungsten:secret --insecure --request POST 'https://127.0.0.1:8097/api/v2/replicator/service/alpha/thl/inflight-protocol' -d 
'{
 "payloadType":"StringPayload",
 "payloadVersion":"1",
 "payload":
 {
 "value":"legacy"
 }
}' 

{ 
 "payloadType": "StringPayload", 
 "payloadVersion": "1", 
 "payload": 
 { 
 "value": "THL inflight protocol is now set to legacy" 
 } 
}

Issues: CT-1862

Improved smart scale handling of session and global changes so they're all played against connected replica or in case of reconnection. As an example, SET SESSION TRANSACTION READ ONLY will be sent to both primary and replica connections so that a subsequent SELECT @@SESSION.transaction_isolation will return the expected value, even if executed on a replica.

Issues: CT-1600

Added user-based authentication to existing host-based authentication to restrict connections per host and per user.

For more details, see Host-based Authentication

Issues: CT-1978

A new advanced listeners feature has been added to the connector for even great control of client connectivity.

For more details, see Advanced Listeners

Issues: CT-2000

Fixed rsync availability check in TPM to error out early when the executable is not found in the path

Issues: CT-1028

The tungsten_get_status command now handles shunned nodes properly.

Issues: CT-2026

The tpm ask keys command now lists all available keys properly.

Issues: CT-2041

The tpm update command no longer aborts with a Dataservice was NOT removed error when service names with a hyphen are defined in the configuration. We strongly discourage the use of hyphens (-) in service names and recommend underscores (_) instead.

Issues: CT-2052

The tpm validate-update command now performs the delete-service checks properly.

Issues: CT-2054

The tungsten_find_events command no longer fails with error Undefined subroutine &main::info

Issues: CT-2058

tpm now builds affinity the same way in bridge and proxy modes.

Issues: CT-2060

Fixes an issue when tprovision would delete the MySQL PEM files in the datadir.

Issues: CT-2067

The tpm diag command will now locate and kill any remaining tpm diag child processes on the calling host.

Issues: CT-2068

Suppressed wrong tpm errors with autodetect hosts.

Issues: CT-2072

Fixed a tpm issue trying to restore an empty policy.

Issues: CT-2074

Fixed an issue that was occurring when an ALTER USER is applied on a mysql instance that uses lower case identifiers.

Issues: CT-2024

Fixed an issue where an extracted JSON field could be invalid, with doubled commas in JSON arrays.

Issues: CT-2049

Fixed an issue where status would fail if issued while a service was reconfiguring (in the middle of a setrole command for example).

Issues: CT-2081

Fixed an issue with CLIENT_QUERY_ATTRIBUTES and/or CLIENT_DEPRECATE_EOF flags that would prevent fallback bridge mode to handle proxy connections properly.

Issues: CT-2020

Fixed an issue in Dynamic Active-Active environments where smartScale would always read from the site with write affinity, even if a different site is specified for read-affinity.

Issues: CT-2040

rsync is now an option in tprovision, in addition to using xtrabackup and mysqldump. To use rsync, specify -m rsync.

Using rsync by default will provision a replica in 2 passes:

Issues: CT-338

A new -c option is now available with some trepctl commands that can be used in conjunction with the -r option to indicate the number of times to refresh before automatically terminating. For example, the following command:

shell> trepctl perf -r 3 -c 10

Will refresh the output every 3 seconds, 10 times.

Issues: CT-679

The trepctl status will now show the last known applied seqno and latency.

This information is stored on disk at regular intervals (10s minimum) so as not to overload the replicator, therefore the value can be shown as slightly old dependant on when the status command was issued.

By default, this feature is disabled. It can be enabled by setting the following parameter in the configuration :

svc-applier-last-applied-write-interval=20

This will write current position to disk every 20 seconds. This information is also exported by the Prometheus exporter.

If the service is online, it will display the current value (the same as appliedLastSeqno and appliedLatency)

shell> trepctl status 
Processing status command... 
NAME VALUE 
---- ----- 
appliedLastEventId : mysql-bin.000017:0000000151329854;70 
appliedLastSeqno : 999 
appliedLatency : 347707.0 
... 
lastKnownAppliedLatency: 347707.0 
lastKnownAppliedSeqno : 999 
...

Issues: CT-1823

The tungsten_merge_logs command now supports the --before TIMESTAMP and --after TIMESTAMP filters

Issues: CT-1869

Added a new log file (tungsten-replicator/log/data-drift.log) for data drift messages, i.e. :

Issues: CT-1873

The tpm ask summary command now provides the coordinator host and the isCoordinator boolean if the Manager is running on that node.

Also, tpm ask now supports direct calls to coordinator, \{isCoordinator|iscoordinator} and \{isBridgeMode|isBridge|bridge|isbridge|isbridgemode}.

Issues: CT-1874

The tungsten_generate_haproxy_for_api and tpm generate-haproxy-for-api commands now support using connector hosts in the backend definitions via -c, and extra backend flags to the backend hosts lines using -f.

Issues: CT-1909

The tungsten_generate_haproxy_for_api and tpm generate-haproxy-for-api commands no longer call the Perl Data::Dumper module.

Issues: CT-1915

The tungsten_reset_manager command now supports the ability to simply print out the path or paths to be cleared, one per line via the -l or --list arguments.

Issues: CT-1917

The tmonitor command now accepts cli args to specify the ports and will auto-configure the ports if they have been changed via the Tungsten configuration.

Issues: CT-1919

The tpm command calls to glob have been improved to be more strict and compliant.

Issues: CT-1940

The tpm ask stages and tpm ask allstages commands have been added to display the Replicator stages for the current node (stages) and the stages for each role (allstages).

Issues: CT-1943

The tpm ask command has five new variables available: dsrole & dsstate for the current datasource, and trrole & trstate for the current replicator, and nodeinfo which displays all 4 of the new variables.

Issues: CT-1944

A new standalone status script has been added called tungsten_get_status that shows the datasources and replicators for all nodes in all services along with seqno and latency.

Issues: CT-1962

The tpm report command now prints the hostname and listener ports where available when using the --extra|-x option or the new --ports option.

Issues: CT-1969

A new dsctl option has been added to the thl command, and a new -event option added to thl list

Full details and examples of these new options can be found here

Issues: CT-2012

Added a new feature that enables pausing a replicator stage for some amount of time.

This will pause the given stage for 100 seconds.:

trepctl pause -stage thl-to-q -time 100

This will pause the stage indefinitely (or until restart, etc) Add -y to avoid the prompt message whether you are sure.

trepctl pause -stage thl-to-q

For the previous 2 commands, running a pause command again will override the previous command.

This will resume the suspended stage (Note that if the stage is not paused, this will have no effect):

trepctl resume -stage thl-to-q

Issues: CT-1912

Per-service tuning of the replicator thl directory is now possible for multi-service replicator-only installs as well as for clustering. The given value should be the base directory, to which tungsten will add the service name. For example, the following entry in the tungsten.ini:

[alpha]
...
...
thl-directory=/drv1/thl
...

Would result in the THL being placed in /drv1/thl/alpha

Issues: CT-1927

A new replicator role (thl-applier) has been added to allow a replicator service to apply its locally available THL, without pulling from a remote host

Issues: CT-1936

Added a way to configure the maximum number of rows that can be grouped together when applying row based events for multiple insert or delete statements.

For these properties to be in effect, you must ensure that optimize-row-events=true is either explicitly set in your configuration, or not present (since it will be enabled by default)

For example, the following settings will limit the number of inserted or deleted rows applied at once to 10:

optimize-row-events-limit-insert-rows=10 
optimize-row-events-limit-delete-rows=10

The default values if not specified will be 50 for inserts and 100 for deletes. Note that for deletes to be optimized, the affected table MUST have a single column PK.

Issues: CT-1980

The connector graceful-stop command now supports systemd service manager properly. The connector stop command now takes an optional argument that will make it a graceful stop. If connector stop is run without the parameter, it will stop the connector immediately. If a positive number of seconds is passed, it will wait, at most, this timeout for connections to disconnect (refusing new connections), after which it will force close all connections and shutdown the connector. connector graceful-stop behavior is unchanged: without the parameter, the connector will wait "forever" for connections to disconnect. A positive timeout in seconds can be passed to sever connections after the given delay

Issues: CT-1921

Two new commands have ben introduced to cctrl to provide better insight and control of the connectors. These are:

For more detils on the usage and options, see cctrl datasource Command

Issues: CT-1949

Added logging configuration example to print load balancers activity

Issues: CT-1966

Added a new option to TPM manager-replicator-offline-timeout=<timeout_in_sec> that configures the timeout for the manager to wait until the replicator goes offline. When parallel applier is in use the default timeout was too low, so it’s now user configurable so that it can be adjusted to suit different topologies. If not supplied, the default is 180 (3 minutes). This value should be sufficent in most use cases.

Issues: CT-1892

New logs files have been added for the REST API, these are as follows:

Issues: CT-1983

ddlscan, dsctl and tungsten_send_diag are now added to the aliases.sh script.

Issues: CT-813

Fixes issues where fixed properties and filters passed to tpm in service stanzas were not being configured correctly

Issues: CT-1463

No longer using Tanuki wrapper functionality to print jvm version, which was creating defunct java processes at startup, now using internal code.

Issues: CT-1876

The tpm install and tpm update commands now properly support the --thl-ports option for cross-site subservices.

Issues: CT-1953

The tungsten_skip_seqno command no longer fails when -i is specified, and now properly filters using --filter when there is a long error message.

Issues: CT-1877

The tpm command now allows any case for section entries (i.e. [alpha_FROM_beta]) in the INI files.

Issues: CT-1879

The tpm diag command now passes when the nodename defined in the tungsten.ini is the shortname, and DNS returns the FQDN.

Issues: CT-1908

The tpm diag command now gathers the mysql.log file when SSL is enabled in the server.

Issues: CT-1920

The tpm command checks for the existence of the mysql command-line client when installing/upgrading. The process will no longer abort with an error on non-MySQL targets such as heterogeneous replicator appliers, or Active-Witness hosts.

Issues: CT-1924, CT-2018

Fixes an issue that prevented dsctl from connecting to MySQL if SSL was enabled.

Issues: CT-1928

The tpm mysql command will now gracefully handle being run on a non-database node.

Issues: CT-1946

The tpm mysql command no longer aborts with an access denied error on CentOS 6.

Issues: CT-1977

Both TungstenAPI and tpasswd now properly update .passwords.store.orig backup file so that proper manipulation of passwords won't trigger tpm update failure

Issues: CT-1981

The cluster_backup script will no longer backup a replica if the replicator is in an ERROR state.

Issues: CT-1036

Fixes an issue that would prevent a service from going offline at a specified time (trepctl online -until-time) when parallel apply is enabled. This is a rework of CT-1243.

Issues: CT-1684

Fixed a possible issue when recovering an old primary as a replica after failover when parallel apply is enabled, that could lead the replica to be unable to come online and require a reprovisioning of this replica.

Issues: CT-1890

Fixes an issue that prevented geometry datatypes with SRID from being replicated.

Issues: CT-1904

Fixed an issue where filtered events would trigger a useless update to the service trep_commit_seqno table while it is overwritten anyway once the last statement of the applied event is done, just prior to committing the whole block.

Issues: CT-1931

Fixed an issue where the replicator would hang after applying a DROP TABLE event, that originally failed on the primary, but got logged into the binlog.

Issues: CT-1973

Fixed a parsing issue that would prevent the replicator from correctly detecting a CREATE TABLE statement with START TRANSACTION

Issues: CT-1987

Fixed an issue where the dropsqlmodes filter would fail to remove invalid sql modes from a multi-statement event

Issues: CT-1993

Connector now auto detects default authentication plugin by retrieving MySQL data source variable default_authentication_plugin rather than just using MySQL server version

Issues: CT-1926

No longer printing warning "SequenceException: Parents differ" when canceling connections of a composite data service

Issues: CT-1964

Fixed connector logging configuration to show hostname and class printing logs

Issues: CT-1965

The cctrl command datasource <ds> slave now sets the replicator role correctly. Previously, only the datasource role would change.

Issues: CT-1882

A bug has been fixed that, in a few very rare cases, would allow replicas to continue to pull and apply THL from a failed primary whilst a failover was in the process of electing a new primary. This resulted in failovers being unable to complete fully. Whilst the new primary would be online and functioning, existing replicas in the cluster could experience errors due to THL discrepencies between the old and new primary nodes.

Issues: CT-1986

Fixed REST API /api/v2/manager/control/service/{service}/switch call. It will not switch anymore to a shunned node.

Issues: CT-796

Calls to /api/v2/manager/cluster/status now return properly when a peer cluster is fully offline or unreachable.

Issues: CT-1945

The REST API call /api/v2/manager/control/service/{service}/datasource/{datasource}/slave now sets the role of the replicator correctly.

Issues: CT-1975

Added the ability to turn auto recovery on or off dynamically, removing the need to run tpm update.

This is done by running the following command:

shell> trepctl -service servicename setdynamic -property replicator.autoRecoveryMaxAttempts -value <number>

Issues: CT-1088

The new command tungsten_upgrade_manager will help clear specific cached SSL values, to be used only when specified by Continuent Support.

Issues: CT-1788

A new tpm report sub-command has been added. By default, tpm report will generate a security report for all available communication channels on a per-node basis. The purpose of tpm report is to provide easy access to all of the settings that pertain to a specific topic. The default (and only) topic at this time is the security stance. More topics will be added over time.

Issues: CT-1793, CT-1857

The tapi command now supports the creation and modification of multiple APIv2 admin users. Two new cli args have been added: --create-user and --create-pass.

Issues: CT-1801

The tapi command now auto-detects the protocol, either http or https

Issues: CT-1803

Added connector<->manager link encryption status in the output of router * status command in cctrl

Issues: CT-1811

Added JMX encryption and authentication status to trepctl -verbose and connector command outputs

Issues: CT-1813

The tapi command will now output service names with the new option --includeServiceName, or if there is more than one service name found.

Issues: CT-1860

The thl list command now displays an approximative field size in bytes for row-based replication.

Issues: CT-1847

Added a connector mode command to print which mode the connector is running in, either "bridge" or "proxy"

Issues: CT-1799

Replaced official log4j library with a secured version in which all vulnerable classes have been removed. This prevents exposing the software following a (user) misconfiguration of the log4j properties

Issues: CT-1810

Added a way to retrieve service properties through the API, for example, to check a specific service parameter, run the following API call :

shell> curl https://127.0.0.1:8097/api/v2/replicator/service/alpha/properties?filter=replicator.global.db.sslEnabled
{ 
 "payloadType": "PropertiesListPayload", 
 "payloadVersion": "1", 
 "payload": { 
 "replicator.global.db.sslEnabled": "false" 
 } 
}

Issues: CT-1802

Added new top level REST API call to display RMI SSL status, run the following API call :

shell> curl https://127.0.0.1:8097/api/v2/rmiSecurity

Issues: CT-1812

The tpm update command now properly handles v7 -> v7 upgrades.

Issues: CT-1815

deployall script now properly displays the executable prefix for restarting services, typically mm_treplicator in Multi-Site/Active-Active replicator-only installs, as well as the full path to component binaries

Issues: CT-1835

tpm would fail on keystores containing special characters in alias names.

Issues: CT-1852

Worked around a ruby issue where DNS name resolution was sometimes preferred to file-based resolution in TPM. The following logic now applies: first trying file-based host name resolution, upon failure try DNS, then through ping command

Issues: CT-1653

The tpm update command now returns clusters to the proper mode (i.e. maintenance or automatic) for staging-method upgrades (INI not affected).

Issues: CT-1784

The tpm policy command now returns the proper policy in a composite cluster. There is also a new cli argument, --all, to show the policy for all cluster services.

Issues: CT-1787

The tpm update command no longer aborts during a staging deployment when the actual hostname does not match what is configured for the node names in tpm

Issues: CT-1791

Improves the tpm diag command so that it waits 2 seconds for replicator thread dump to complete.

Issues: CT-1792

Fixed an `Undefined subroutine` call error in tapi --create

Issues: CT-1798

The tpm connector command now properly passes the -e arguments to the mysql command.

Issues: CT-1816

tpm now properly reports errors upon timeout executing commands. Typically, when a host is down, when upgrading or installing, tpm will now properly report that pinging this host failed.

Issues: CT-1819

The tpm update command now properly determines services to be deleted.

Issues: CT-1825

TPM would fail to validate data sources when using ssl-capath [directory] rather than ssl-ca [file] in my.cnf.

This fixes only the validation part, automated truststore creation by TPM will not be possible at this point, a full tungsten-truststore.ts will have to be passed through --java-truststore-path

Issues: CT-1826

The tapi command now auto-switches protocol (http or https) properly.

Issues: CT-1829

replicator start offline would not properly pass the offline argument to systemd scripts when installed through deployall.

Issues: CT-1836

undeployall script would not properly uninstall systemd services on some distributions including amazon linux.

Issues: CT-1845

Fixed an issue with latest xtrabackup 8.0.28

Issues: CT-1838

During the backup/restore process the MySQL certificates are kept intact.

Issues: CT-1841

Restore will now succeed even if only one incremental backup has been created.

Issues: CT-1842

Incremental backups can now be restored more then once.

Issues: CT-1850

Fixed the EnumToString and pkey filters to renew their database connections (every hour by default). This can be changed with the following properties:

property=replicator.filter.enumtostring.reconnectTimeout=3600
property=replicator.filter.pkey.reconnectTimeout=3600

Issues: CT-1786

tpm update would fail with message "The host is configured to be '' but it is currently operating as a 'master'" when a dynamic replicator configuration file is found

Issues: CT-1848

Fixes issues with PrimaryKeyFilter:

Issues: CT-1861

Fixed a NullPointerException that could happen if a replicator was put offline during startup. This had mostly no consequence except the exception being logged.

Issues: CT-1863

Fixed the output of connector cluster-status by adjusting log4 appender.

Issues: CT-1778

An exception could occur while forcing shutdown of router gateways.

Issues: CT-1779

Fixed an issue where the command system summary issued in cctrl would hang the managers in the cluster.

Issues: CT-1849

Two new tpm options have been included as part of the new API in this release.

The options are used for setting the API admin user credentials and are as follows:

Issues: CT-1327

Support now included for MariaDB 10.3+

Issues: CT-1276, CT-1433

Support has been added for Java 17 LTS

Issues: CT-1706

A new tpm option delete-service is now available to simplify the removal of clusters and/or replicator services.

Issues: CT-210, CT-327, CT-1275

Prometheus exporters mysqld_exporter and node_exporter are now included with the distribution packages.

A new command line tool tmonitor is now available for the management and testing of external Prometheus exporters (node and mysqld), and for the testing of internal exporters (Manager, Connector and Replicator).

Issues: CT-960

A new tpm option purge-thl and a new script tungsten_purge_thl have been added to allow easier and more intelligent THL purging across all nodes in a topology.

This allow you to purge THL files based on the following rules:

The default behavior is to display the needed commands for the admin to execute manually.

Issues: CT-1273

tpm diag now collects routing table information via route -n , and has two new command-line arguments: --include and --groups.

--include specifies a comma-separated list of subroutines to include. Any gather subroutine not listed will be skipped.

--groups specifies a comma-separated list of subroutine groups to include. Any group not listed will be skipped.

Issues: CT-1322

A new sub-command has been added, tpm generate-haproxy-for-api. This read-only action will read all available INI files and dump out corresponding haproxy.cfg entries with properly incrementing ports; the composite parent will come first, followed by the composite children in alphabetical order.

The tungsten_generate_haproxy_for_api script functionality has been merged into the tpm generate-haproxy-for-api command. The tungsten_generate_haproxy_for_api script remains as a shell wrapper for tpm generate-haproxy-for-api.

Issues: CT-1342

tungsten_send_diag now supports a new command-line argument, --cleanup, which will cause the removal of the diagnostic archive file generated using the --diag argument.

Issues: CT-1360

The tungsten_reset_manager command is now able to restart the Manager process when the --start or -s argument is passed in.

Issues: CT-1401

With the release of APIv2, a new cli tool has been introduced to allow easier access called tapi.

In addition, the vast majority of Tungsten cli tools have been updated to optionally use the APIv2 interface when desired.

The Nagios and Zabbix checks are also available via APIv2 using the tapi tool.

Issues: CT-1454

The tungsten_purge_thl command is now a wrapper for the tpm purge-thl command.

Issues: CT-1488

The tmonitor command now has better help text and more options to ease usage, including --filter to allow easy viewing of the tmonitor test output.

Issues: CT-1585

A new option to print the merged logs to STDOUT has been added to tungsten_merge_logs (--stdout|-O).

The tpm command suite now properly supports the --profile argument to specify a Tungsten json configuration file in place of the installed tungsten.cfg.

Issues: CT-1680

The tapi command now supports the --affinity argument which will display all Connector-specific affinity settings, along with --connectorstatus to show all.

Issues: CT-1700

The cctrl.log file is now accessible from the $CONTINUENT_ROOT/service_logs directory

Issues: CT-1727

A new command (error) has been added to trepctl to output a full stack trace of the last error, if any.

shell> trepctl -service <serviceName> error

Event application failed: seqno=10 fragno=0 message=Table hr.regions not found in database. Unable to generate a valid statement.
com.continuent.tungsten.replicator.applier.ApplierException: Table hr.regions not found in database. Unable to generate a valid statement.
at com.continuent.tungsten.replicator.applier.JdbcApplier.getTableMetadata(JdbcApplier.java:582)
at com.continuent.tungsten.replicator.applier.JdbcApplier.fillColumnNames(JdbcApplier.java:494)
at com.continuent.tungsten.replicator.applier.JdbcApplier.getColumnInformation(JdbcApplier.java:1236)
at com.continuent.tungsten.replicator.applier.MySQLApplier.applyOneRowChangePrepared(MySQLApplier.java:418)
at com.continuent.tungsten.replicator.applier.JdbcApplier.applyRowChangeData(JdbcApplier.java:1460)
at com.continuent.tungsten.replicator.applier.JdbcApplier.apply(JdbcApplier.java:1576)
at com.continuent.tungsten.replicator.applier.ApplierWrapper.apply(ApplierWrapper.java:100)
at com.continuent.tungsten.replicator.pipeline.SingleThreadStageTask.apply(SingleThreadStageTask.java:871)
at com.continuent.tungsten.replicator.pipeline.SingleThreadStageTask.runTask(SingleThreadStageTask.java:601)
at com.continuent.tungsten.replicator.pipeline.SingleThreadStageTask.run(SingleThreadStageTask.java:185)
at java.base/java.lang.Thread.run(Thread.java:834)

Issues: CT-1747

Added support for mariabackup. Configurable tpm option:

backup-method=mariabackup
or
backup-method=mariabackup-incremental

Issues: CT-1100

Fixes an issue where tprovision could remove the SSL certs for MySQL on reprovision

Issues: CT-1323

It is now possible to compress and/or encrypt THL on disk. For more information on using these features see https://docs.continuent.com/tungsten-clustering-7.0/thl-compress-encrypt.html

Issues: CT-630

The replicator will now be able to handle new SQL_MODES available in later releases of MySQL and MariaDB, these are as follows:

Issues: CT-1362

In-Flight THL Compression is now available.

For full details on enabling this feature, refer to this page

Issues: CT-1420

Audit Logging is now available in the Connector: it allows for logging data transferred between client application and MySQL servers to a file, database or socket.

For more details and steps to enable, see https://docs.continuent.com/tungsten-clustering-7.0/connector-advanced-audit-logging.html

Issues: CT-78

Added support for TLSv1.3.

Note that TLS 1.3 was introduced in jdk 8u272. So tpm will only set it by default for java 8u272 and later versions. It is still possible to force the protocol to be used via tpm flag --tls-enabled-protocols=TLSv1.xxx

Issues: CT-1367

A new Dynamic Active/Active feature has been added within the Proxy.

This new Proxy mode is for Composite Active/Active clusters only, and allows you to specifically configure writes to be directed to a single cluster by the use of the tpm connector-write-affinity flag.

For full details on this feature, and configuring it, see [link to be added]

Issues: CT-1540

connector drain [opt-timeout] has been introduced as an alias to existing connector graceful-stop command and will prevent new connections, then shutdown the connector after an optional delay.

The opt-timeout parameter (seconds) can be specified to limit the wait before stopping the connector. Not passing this parameter implies infinite wait.

Issues: CT-1644

connector-reset-when-affinity-back is now available with proxy mode.

Issues: CT-1763

Tungsten can now install on default CentOS and RedHat 8 with tightened security settings thanks to support of TLSv1.3

Issues: CT-1359

Tungsten software now undergoes a rigorous security scan during QA. We also check included open-source/3rd-party software.

Issues: CT-1579

A number of new metrics have been added to the Prometheus exporters.

Issues: CT-1266, CT-1615

ARM 64 bit processor support added (linux aarch64)

Issues: CT-1619, CT-1620

General improvements in MySQL 8.0 support.

Issues: CT-1346

IPv6 host addresses are now fully supported.

Can be enabled with the following configuration property:

prefer-ip-stack="6"

By default, IPv4 is enabled, which equates to the value of "4" in the above property.

Issues: CT-1537

Default systemd configuration files for Tungsten components no longer specifies the tungsten group to execute the command. This will prevent file access issues when tungsten user belongs to several groups.

Issues: CT-1550

When services are deployed with systemd and MySQL could not start due to an error, tpm would not be able to later start MySQL

Issues: CT-1734

In certain cases, tprovision would not be able to find the binary log position of the backup when taken from a primary. This has been fixed.

Issues: CT-1085

Fixes a bug in tprovision when using xtrabackup version 8, due to changes in xtrabackup binaries.

Issues: CT-1248

The tpm connector command now handles special characters in the password string.

Issues: CT-1258

The tpm update command will now exit with an error if any files not owned by the configured Tungsten OS user are found in the Tungsten installation directory.

For example, if the OS user is tungsten and the installation directory is /opt/continuent, containing the file /opt/continuent/thl/archived_thl.zip owned by root would cause something like the following error to be produced, and tpm update would exit:

Foreign-owned files found!

Located files in the Tungsten installed directory /opt/continuent
that are not owned by the Tungsten OS user (tungsten):

/opt/continuent/thl/archived_thl.zip

Please change the ownership of these files to OS user "tungsten"
using the chown command as root via sudo, then rerun the `tpm update` command.

For example:
shell> sudo chown -R tungsten:tungsten /opt/continuent

Issues: CT-1260

The tpm diag command now behaves correctly on Connector-only nodes, where previously it would try to gather Manager and Replicator-specific items.

Issues: CT-1284

Fixes a security issue within the tpm diag command.

Issues: CT-1295

The number of created and active connections could be incorrect when listed from a composite data service

Issues: CT-1312

tungsten_send_diag no longer prints an error about Use of uninitialized value $diagArgs in concatenation.

Issues: CT-1354

The tpm command no longer prints an error when run with no other command-line arguments.

Issues: CT-1373

The tpm command no longer aborts with a Use of uninitialized value error when a stray tungsten.cfg file exists under $CONTINUENT_ROOT

Issues: CT-1394

Fixes a monitoring bug with users using caching_sha2_password.

Issues: CT-1406

tprovision (formerly tungsten_provision_slave) may fail to provision if the MySQL data directory was not accessible to the tungsten user.

Issues: CT-1475

The tpm generate-haproxy-for-api command no longer fails on CentOS 8.

Issues: CT-1484

The tmonitor command no longer fails on Debian 9 and Ubuntu.

Issues: CT-1485

All tpm sub-commands now handle command-line arguments more intelligently.

Issues: CT-1487

The tpm purge-thl command now handles command-line arguments more intelligently.

Issues: CT-1489

The tpm diag command now properly collects the system information file on Debian systems.

Issues: CT-1492

Database monitoring logs are now reporting the correct error number and SQL state when database errors occur.

Issues: CT-1497

The tpm update command now handles updates/upgrades more gracefully when the previous version did not have the latest tpm framework.

Issues: CT-1506

The tpm update command will now properly remove composite services.

Issues: CT-1519

The tungsten_find_orphaned command no longer fails with an 'Can't exec "/bin/sh": Argument list too long' error when there are too many THL files to parse.

Issues: CT-1545

The tpm ask command no longer calls Data::Dumper when it is not available.

Issues: CT-1626

tpm now parses the MYSQL SSL related setting correctly.

Issues: CT-1662

Fixes an issue where the deployall command would create a root owned wrapper.log in the ./tools directory.

Issues: CT-1664

When MySql services where badly installed, some distribution could show a “not-found” status within systemctl, confusing tpm

Issues: CT-1677

The tpm command now communicates properly when there is no INI configuration file or staging-method deploy.cfg configuration defined.

Issues: CT-1712

The tpm diag command now handles Multi-Site/Active-Active topologies better.

Issues: CT-1718

tungsten_monitor.rb script no longer uses sudo to send emails if the configuration doesn't allow it.

Issues: CT-1737

The tpm diag command now handles zero-length mysqld.log files gracefully.

Issues: CT-1740

When running the cluster_backup in a Active/Active environment, when setting require_master_backup to false, the script would still attempt to backup the Primary as it would scan the wrong sub-service and incorrectly identify the Relay node as a candidate.

Issues: CT-1280

Fixed an issue where an xtrabackup generated by the replicator would fail to be restored using trepctl restore command.

Issues: CT-1575

The replicator metadata cache will now correctly handle table names when lower_case_table_names=1 is set in the MySQL configuration.

Issues: CT-651

When using parallel apply, the replicator would error with a Foreign Key constraint error if statements were issued against two or more objects that shared the same name, but with different case sensitivity, for example:

mysql> create table testtable;
mysql> drop table testtable;
mysql> create table TestTable;

Issues: CT-1259

A change in the way MySQL logs CREATE TABLE AS SELECT in the Binary Logs from v8.0.20 onwards, meant these transactions would previously fail.

Issues: CT-1301

Fixes occurences of NullPointerException that would occur when bringing the replicator online before MySQL was started.

Issues: CT-1348

For row based events, SQL modes were not displayed in the THL output. This is now fixed.

Issues: CT-1440

When connecting to a THL server, a client will now connect to the next available host in its THL uri, if the first does not have the sequence number that the client requires. The client will then fail only if none of the hosts from the uri can provide the needed sequence number.

Issues: CT-1558

Fixed an issue when using Parallel apply that would show a NullPointerException in case an event could either not be found or be corrupted in THL. This will now display a correct message Missing or corrupted event from storage

Issues: CT-1722

Fixed an issue where trepctl was leaving JMX connections opened.

Issues: CT-1752

Added more debug information for detecting possible hanging connections while a THL client connects to the THL server. Also, added socket timeout for the connection initialization

Issues: CT-1760

Includes previously missing template file to enable easy configuration of the dbrename filter.

Issues: CT-1350

The BidiRemoteSlaveFilter could fail to correctly flag fragmented events in unprivileged environments (Aurora, for example) In such an environment (multi-active, unprivileged database access), a new setting was introduced to force extraction process to read ahead to the last fragment to detect the service name (false by default). Enabled with repl_svc_extractor_multi_frag_service_detection=true

Issues: CT-1351

By default, the connector will no longer transparently reconnect underlying connections to database servers when the data service changes.

This will prevent the following case: in Composite Active/Active topologies, a given connection starts to write data to a site. The site fails, connection gets reconnected the other site and resumes writing. However the data written to the 1st site has not reached the 2nd site, thus data will not be consistent.

Default is to reject reconnections that follow a write operation (RW_STRICT connection or SmartScale after a write) and to allow reconnection after a read operation (RO_RELAXED or SmartScale after a read) which translates to --connector-allow-cross-site-reconnects-for-writes=false and --connector-allow-cross-site-reconnects-for-reads=true

It is still possible to get the previous behavior (reconnecting transparently connections cross-site) by specifying both --connector-allow-cross-site-reconnects-for-writes=true and --connector-allow-cross-site-reconnects-for-reads=true, at your own risk

Issues: CT-1265

With @direct r/w splitting in active/active configurations, the connector was not correctly redirecting connections to an active site after re-joining the cluster following a failure.

Issues: CT-1400

The Connector now properly retries specific MySQL commands when possible: INIT_DB, CHANGE_USER, STATISTICS and prepared statement functions.

Issues: CT-1480

Connector now forbids non-ssl connections when mysql server has require_secure_transport=ON

Issues: CT-1666

Connector now mirrors the MySQL default connect_timeout by retrieving it from the primary when starting up. This timeout will apply to all connections made from the connector to MySQL servers.

This setting can be over-ridden by using the following tpm property

property=connectTimeout=VALUE

If VALUE set to autodetect, this value will mirror the MySQL connect_timeout system variable. Set to 0 for infinite timeout.

Issues: CT-1726

Connector no longer requests cross-site services information from manager. In addition to removing extra network traffic, this cures a problem of connections counted twice in data source connection statistics

Issues: CT-1775

Fixes a NullPointerException error (NPE) in the manager logs.

Issues: CT-1132

Fixes an edge case bug that would allow a Composite Active/Active cluster to contain 2 Primary nodes.

Issues: CT-1474

Fixes a manager-internal connection check during recovery that was not properly using SSL when required.

Issues: CT-1661

Early initialisation of the REST API sometimes caused the manager to hang and fail to startup correctly on new installations.

Issues: CT-1725

Fixes an issue when the manager would wait longer than the timeout for replicator purge on a failover.

Issues: CT-1733

Fixes an issue where datasource <datasourcename> welcome would fail to welcome a manually failed composite datasource.

Issues: CT-1771

log4j libraries updated to v2.17.1 specifically to mitigate risk of exposure to the 0-day vulnerbaility detected in log4j v2.14

Issues: CT-1703

In proxy mode, kill queries are now intercepted by the connector in order to replace the given thread_id by the underlying database connection thread_id.

Issues: CT-2298

Fix for an issue in ColumnNameFilter, EnumToStringFilter and TableMetadataCache where an ALTER USER statement could trigger a NullPointerException.

Issues: CT-2321

Fixes an issue when the manager would wait longer than the timeout for replicator purge on a failover.

Issues: CT-1733

Fixed an issue where a failover could hang in rare cases when security and Proxy mode are enabled. SSL connections to a failed data source could enter a deadlock and block failover for up to 15 minutes.

Issues: CT-2183, CT-2187

Fixed RPM package script to run tpm install instead of tpm update when installing the rpm

Issues: CT-2130

A change in the way MySQL logs CREATE TABLE AS SELECT in the Binary Logs from v8.0.20 onwards, meant these transactions would previously fail.

Issues: CT-1301

Fixed an issue where an extracted JSON field could be invalid, with doubled commas in JSON arrays.

Issues: CT-2049

Improved a query that is run by Tungsten when fetching tables metadata (column names, datatypes, etc). While it is not generally needed, the unoptimized query can run badly (especially) against old mysql versions with a lot of databases / tables. For now, the new optimized query is not used by default, but this could change in some future version.

This can be enabled by using the following property :

property=replicator.datasource.global.connectionSpec.usingOptimizedMetadataQuery=true

Issues: CT-2077

Note, this bug only affects v6.1.21

Removed extra logging for each "SET xxx" statement when in proxy mode. On top of polluting the logs, this extra logging can overwhelm the connector, making it slow to respond and have client threads pile up, up to the point where that could exhaust memory

Issues: CT-2165

tungsten show processlist could error with a disconnection message listing connections disconnected on the mysql server side.

Issues: CT-2112

The tpm diag command now captures the output of cctrl> cluster topology validate

Issues: CT-2115

Improved smart scale handling of session and global changes so they're all played against connected replica or in case of reconnection. As an example, SET SESSION TRANSACTION READ ONLY will be sent to both primary and replica connections so that a subsequent SELECT @@SESSION.transaction_isolation will return the expected value, even if executed on a replica.

Issues: CT-1600

ARM 64 bit processor support added (linux aarch64)

Issues: CT-1619, CT-1620

The tungsten_get_status command now handles shunned nodes properly.

Issues: CT-2026

The tpm ask keys command now lists all available keys properly.

Issues: CT-2041

A new command has been added called tungsten_get_ports, which will display the running Tungsten processes and the associated TCP ports that those processes are listening on.

Issues: CT-1970

The tungsten_get_ports command now supports the /opt/continuent/tungsten/tungsten-connector/conf/interfaces.json file, if found.

Issues: CT-2003

The tungsten_get_ports command now use values based on values obtained from tpm query values when possible for the greatest output accuracy.

Issues: CT-2008

Worked around a ruby issue where DNS name resolution was sometimes preferred to file-based resolution in TPM. The following logic now applies: first trying file-based host name resolution, upon failure try DNS, then through ping command

Issues: CT-1653

When running the cluster_backup in a Active/Active environment, when setting require_master_backup to false, the script would still attempt to backup the Primary as it would scan the wrong sub-service and incorrectly identify the Relay node as a candidate.

Issues: CT-1280

Fixed an issue where the dropsqlmodes filter would fail to remove invalid sql modes from a multi-statement event

Issues: CT-1993

The cctrl command datasource <ds> slave now sets the replicator role correctly. Previously, only the datasource role would change.

Issues: CT-1882

A bug has been fixed that, in a few very rare cases, would allow replicas to continue to pull and apply THL from a failed primary whilst a failover was in the process of electing a new primary. This resulted in failovers being unable to complete fully. Whilst the new primary would be online and functioning, existing replicas in the cluster could experience errors due to THL discrepencies between the old and new primary nodes.

Issues: CT-1986

The tpm command calls to glob have been improved to be more strict and compliant.

Issues: CT-1940

The tpm ask stages and tpm ask allstages commands have been added to display the Replicator stages for the current node (stages) and the stages for each role (allstages).

Issues: CT-1943

The tpm ask command has five new variables available: dsrole & dsstate for the current datasource, and trrole & trstate for the current replicator, and nodeinfo which displays all 4 of the new variables.

Issues: CT-1944

A new standalone status script has been added called tungsten_get_status that shows the datasources and replicators for all nodes in all services along with seqno and latency.

Issues: CT-1962

Added a connector mode command to print which mode the connector is running in, either "bridge" or "proxy"

Issues: CT-1799

The connector graceful-stop command now supports systemd service manager properly. The connector stop command now takes an optional argument that will make it a graceful stop. If connector stop is run without the parameter, it will stop the connector immediately. If a positive number of seconds is passed, it will wait, at most, this timeout for connections to disconnect (refusing new connections), after which it will force close all connections and shutdown the connector. connector graceful-stop behavior is unchanged: without the parameter, the connector will wait "forever" for connections to disconnect. A positive timeout in seconds can be passed to sever connections after the given delay

Issues: CT-1921

The tpm install and tpm update commands now properly support the --thl-ports option for cross-site subservices.

Issues: CT-1953

Fixes an issue where the pause state would be badly displayed under no load, when the pause reaches the defined time. Note this is only a display issue : once an event is received, the pause state and displayed remaining time will be reseted correctly.

Issues: CT-1942

The tpm mysql command will now gracefully handle being run on a non-database node.

Issues: CT-1946

Fixes an issue that would leave a transaction uncommitted longer than necessary.

Issues: CT-1958

Fixed an issue where the shard_id stored in the trep_commit_seqno table would show an incorrect shard after processing a skipped event (an event from another channel)

Issues: CT-1967

Fixed an issue where the replicator would hang after applying a DROP TABLE event, that originally failed on the primary, but got logged into the binlog.

Issues: CT-1973

The BidiRemoteSlaveFilter could fail to correctly flag fragmented events in unprivileged environments (Aurora, for example) In such an environment (multi-active, unprivileged database access), a new setting was introduced to force extraction process to read ahead to the last fragment to detect the service name (false by default). Enabled with repl_svc_extractor_multi_frag_service_detection=true

Issues: CT-1351

Connector now auto detects default authentication plugin by retrieving MySQL data source variable default_authentication_plugin rather than just using MySQL server version

Issues: CT-1926

The tungsten_merge_logs command now supports the --before TIMESTAMP and --after TIMESTAMP filters

Issues: CT-1869

The tungsten_generate_haproxy_for_api and tpm generate-haproxy-for-api commands now support using connector hosts in the backend definitions via -c, and extra backend flags to the backend hosts lines using -f.

Issues: CT-1909

The tungsten_generate_haproxy_for_api and tpm generate-haproxy-for-api commands no longer call the Perl Data::Dumper module.

Issues: CT-1915

The tpm ask isBridge and tpm ask isCoordinator commands have been back-ported to v6.

Issues: CT-1916

The tungsten_reset_manager command now supports the ability to simply print out the path or paths to be cleared, one per line via the -l or --list arguments.

Issues: CT-1917

The thl list command now displays an approximative field size in bytes for row-based replication.

Issues: CT-1847

Added a new feature that enables pausing a replicator stage for some amount of time.

This will pause the given stage for 100 seconds.:

trepctl pause -stage thl-to-q -time 100

This will pause the stage indefinitely (or until restart, etc) Add -y to avoid the prompt message whether you are sure.

trepctl pause -stage thl-to-q

For the previous 2 commands, running a pause command again will override the previous command.

This will resume the suspended stage (Note that if the stage is not paused, this will have no effect):

trepctl resume -stage thl-to-q

Issues: CT-1912

Added a new option to TPM manager-replicator-offline-timeout=<timeout_in_sec> that configures the timeout for the manager to wait until the replicator goes offline. When parallel applier is in use the default timeout was too low, so it’s now user configurable so that it can be adjusted to suit different topologies. If not supplied, the default is 180 (3 minutes). This value should be sufficent in most use cases.

Issues: CT-1892

Fixes issues where fixed properties and filters passed to tpm in service stanzas were not being configured correctly

Issues: CT-1463

replicator start offline would not properly pass the offline argument to systemd scripts when installed through deployall.

Issues: CT-1836

undeployall script would not properly uninstall systemd services on some distributions including amazon linux.

Issues: CT-1845

The tpm diag command now passes when the nodename defined in the tungsten.ini is the shortname, and DNS returns the FQDN.

Issues: CT-1908

The tpm diag command now gathers the mysql.log file when SSL is enabled in the server

Issues: CT-1933

The tpm diag command now behaves properly when the MySQL server is logging to STDERR.

Issues: CT-1935

Fixed an issue where an xtrabackup generated by the replicator would fail to be restored using trepctl restore command.

Issues: CT-1575

During the backup/restore process the MySQL certificates are kept intact.

Issues: CT-1841

Restore will now succeed even if only one incremental backup has been created.

Issues: CT-1842

Incremental backups can now be restored more then once.

Issues: CT-1850

Fixes an issue that would prevent a service from going offline at a specified time (trepctl online -until-time) when parallel apply is enabled. This is a rework of CT-1243.

Issues: CT-1684

Fixed a possible issue when recovering an old primary as a replica after failover when parallel apply is enabled, that could lead the replica to be unable to come online and require a reprovisioning of this replica.

Issues: CT-1890

Fixes an issue that prevented geometry datatypes with SRID from being replicated.

Issues: CT-1904

Fixed an issue where filtered events would trigger a useless update to the service trep_commit_seqno table while it is overwritten anyway once the last statement of the applied event is done, just prior to committing the whole block.

Issues: CT-1931

Added the ability to turn auto recovery on or off dynamically, removing the need to run tpm update.

This is done by running the following command:

shell> trepctl -service servicename setdynamic -property replicator.autoRecoveryMaxAttempts -value <number>

Issues: CT-1088

The cctrl.log file is now accessible from the $CONTINUENT_ROOT/service_logs directory

Issues: CT-1727

connector-reset-when-affinity-back is now available with proxy mode.

Issues: CT-1763

Replaced official log4j library with a secured version in which all vulnerable classes have been removed. This prevents exposing the software following a (user) misconfiguration of the log4j properties

Issues: CT-1810

IPv6 host addresses are now fully supported.

Can be enabled with the following configuration property:

prefer-ip-stack="6"

By default, IPv4 is enabled, which equates to the value of "4" in the above property.

Issues: CT-1537

When services are deployed with systemd and MySQL could not start due to an error, tpm would not be able to later start MySQL

Issues: CT-1734

deployall script now properly displays the executable prefix for restarting services, typically mm_treplicator in Multi-Site/Active-Active replicator-only installs, as well as the full path to component binaries

Issues: CT-1835

When MySql services where badly installed, some distribution could show a “not-found” status within systemctl, confusing tpm

Issues: CT-1677

The tpm command now communicates properly when there is no INI configuration file or staging-method deploy.cfg configuration defined.

Issues: CT-1712

The tpm diag command now handles Multi-Site/Active-Active topologies better.

Issues: CT-1718

tungsten_monitor.rb script no longer uses sudo to send emails if the configuration doesn't allow it.

Issues: CT-1737

The tpm update command no longer aborts during a staging deployment when the actual hostname does not match what is configured for the node names in tpm

Issues: CT-1791

Improves the tpm diag command so that it waits 2 seconds for replicator thread dump to complete.

Issues: CT-1792

The tpm connector command now properly passes the -e arguments to the mysql command.

Issues: CT-1816

tpm now properly reports errors upon timeout executing commands. Typically, when a host is down, when upgrading or installing, tpm will now properly report that pinging this host failed.

Issues: CT-1819

Fixed an issue with latest xtrabackup 8.0.28

Issues: CT-1838

When connecting to a THL server, a client will now connect to the next available host in its THL uri, if the first does not have the sequence number that the client requires. The client will then fail only if none of the hosts from the uri can provide the needed sequence number.

Issues: CT-1558

Fixed an issue when using Parallel apply that would show a NullPointerException in case an event could either not be found or be corrupted in THL. This will now display a correct message Missing or corrupted event from storage

Issues: CT-1722

Fixed an issue where trepctl was leaving JMX connections opened.

Issues: CT-1752

Added more debug information for detecting possible hanging connections while a THL client connects to the THL server. Also, added socket timeout for the connection initialization

Issues: CT-1760

Fixed the EnumToString and pkey filters to renew their database connections (every hour by default). This can be changed with the following properties:

property=replicator.filter.enumtostring.reconnectTimeout=3600
property=replicator.filter.pkey.reconnectTimeout=3600

Issues: CT-1786

By default, the connector will no longer transparently reconnect underlying connections to database servers when the data service changes.

This will prevent the following case: in Composite Active/Active topologies, a given connection starts to write data to a site. The site fails, connection gets reconnected the other site and resumes writing. However the data written to the 1st site has not reached the 2nd site, thus data will not be consistent.

Default is to reject reconnections that follow a write operation (RW_STRICT connection or SmartScale after a write) and to allow reconnection after a read operation (RO_RELAXED or SmartScale after a read) which translates to --connector-allow-cross-site-reconnects-for-writes=false and --connector-allow-cross-site-reconnects-for-reads=true

It is still possible to get the previous behavior (reconnecting transparently connections cross-site) by specifying both --connector-allow-cross-site-reconnects-for-writes=true and --connector-allow-cross-site-reconnects-for-reads=true, at your own risk

Issues: CT-1265

Connector now mirrors the MySQL default connect_timeout by retrieving it from the primary when starting up. This timeout will apply to all connections made from the connector to MySQL servers.

This setting can be over-ridden by using the following tpm property

property=connectTimeout=VALUE

If VALUE set to autodetect, this value will mirror the MySQL connect_timeout system variable. Set to 0 for infinite timeout.

Issues: CT-1726

A new command, tungsten_merge_logs has been added which will gather Manager, Replicator and Connector logs into one file sorted by timestamp for easier troubleshooting.

Issues: CT-1667

A new command tungsten_get_mysql_datadir has been added which will gather and display the actual running value for the mysql data directory (datadir) with any sym-links to the physical destination resolved.

Issues: CT-1690

Now properly adds executable-prefix (when specified) to systemd boot scripts for the replicator.

Issues: CT-1663

The cluster-specific check_tungsten_online tool now properly checks the Replicator and validates the service name if provided.

Issues: CT-1674

The tpm ask tool now works properly on all supported Linux distros.

Issues: CT-1682

The tpm diag command now uses sudo to read the mysqld.log file.

Issues: CT-1685

The tpm --help text now includes the --ini argument.

Issues: CT-1693

Fixes an issue where the replicator would loop forever if a wrong startup option was provided, e.g replicator start wrong

Now, the replicator will stop with a fatal error message logged Unrecognized option: wrong - Shutting down

Issues: CT-1672

Fixes a replication issue with some JSON fields that would contain a single quote within the JSON text

Issues: CT-1691

connector reconfigure, which allows seamless reconfiguration of the Tungsten Connector during tpm update, now properly reloads load balancers.

Issues: CT-1655

The documentation for the tpm command now describes the use of the --ini argument fully.

Issues: CT-1692

The tpm update command now handles the --replace-release argument properly when --ini is also specified.

Issues: CT-1623

Added missing connector capability flag CLIENT_PS_MULTI_RESULTS allowing multiple result sets to be retrieved when executing multiple select prepared statements.

Issues: CT-1624

Improved Perl-based command-line tools to use routines with no external dependencies in place of Data::Dumper

Issues: CT-1630

tungsten_provision_slave would fail using mysqldump with MySQL version 8.0.26+ due to warning of using --master-data=2

Issues: CT-1612

Improved error reporting when the replicator is not able to open a file. This was causing a NullPointerException to be raised.

Issues: CT-1618

Fixed an issue with the ColumnNameFilter that was not resetting its internal connection to the database.

This would then fail if the database connection would time out (e.g. MySQL wait_timeout reached). The connection is now renewed every 1 hour by default.

This can be reconfigured to a different value by adding the following tpm property:

property=replicator.filter.colnames.reconnectTimeout=<time in seconds>

Issues: CT-1646

The show slave status hook could use the wrong tracking schema in Composite Active/Passive clusters.

Issues: CT-1625

Tungsten Connector upgrades are now performed with minimal downtime thanks to graceful-shutdown which is now also used when systemd services are installed.

systemd definition file has been updated to prevent unwanted systemd-driven control of Tungsten components. 2 new settings have been added to definition files:

Note that Tungsten has its own service manager (Tanuki service wrapper), systemd is only used for system startup launch of Tungsten components.

Issues: CT-1538

Added tpm support for tuning maxDelayInterval, used to tune parallel apply. Usage example:

property=replicator.store.parallel-queue.maxDelayInterval=120

This sets the max delay interval to 2 minutes

Issues: CT-1541

A new command-line tool tungsten_show_processlist has been added. This is a read-only script which will show the output from tungsten show processlist; on all available Connectors.

Issues: CT-1542

3 new tpm flags have been added:

These flags can be used to provide a custom systemd configuration that will be used in place of the default, generated one.

This is typically useful when ExecStartPre or ExecStartPost commands are needed

Issues: CT-1555

Improved the tpm command's internal logic and introduced a new sub-command, tpm ask.

Issues: CT-1573

Improved the check_tungsten_online command:

Issues: CT-1578

New ddlscan template (ddl-mysql-staging.vm) available to generate ddl required for staging tables when deploying the batch applier on a MySQL target.

Issues: CT-1587

A new tpm option has been added --backup-options

This property can be used to pass options directly to the backup binary (Xtrabackup, mariabackup etc). This can be especially useful when the xtrabackup and MySQL versions differ. Generally, MySQL is always ahead of xtrabackup and in some situations xtrabackup will fail if the underlying MySQL patch release does not match the xtrabackup patch release. In this scenario, you could utilise the new option to force xtrabackup to not check the versions, for example:

backup-options=--no-server-version-check

Multiple options can be passed as a space separated list, for example.

backup-options=--option1 --option2 --option3

Issues: CT-1595

Support for XA transactions has been added.

Issues: CT-925

A new tpm option has been added replica-tx-commit-level

Valid values of either 1 (default) or 2. Setting this value to 2 will change the underlying database property innodb_flush_log_at_trx_commit when the node becomes a replica, resetting it back to 1 if the node is promoted to a primary.

This can improve performance on the apply rates for replicas.

Issues: CT-1241

tpm was failing to check application password when single quoted.

Issues: CT-1257

The --host argument to tpm connector now works properly. Support for the legacy --hosts is also included.

Only one hostname may be specified to tpm connector.

Issues: CT-1543

The tpm update command now properly parses the --ini argument.

Issues: CT-1551

The tungsten_generate_haproxy_for_api and tpm generate-haproxy-for-api command now properly skips sub-services in Composite Active/Active clusters.

Issues: CT-1552

The tpm connector --samples command no longer displays the password in any example.

Issues: CT-1593

The tpm connector command now works when the application user's password contains double quotes.

Issues: CT-1596

The check_tungsten_services -r command no longer takes an extended time to complete when the manager is not running.

Issues: CT-1599

With smartscale enabled, when no writes have been issued to the connection, the user session will remain uninitialized. In that situation, we now send reads to replicas.

Issues: CT-1560

When an RO_RELAXED connection is requested and no replica is available to serve the request, we will now properly reuse the current connection to the primary for executing the request.

Issues: CT-1564

A new tpm option delete-service is now available to simplify the removal of clusters and/or replicator services.

Issues: CT-210, CT-327, CT-1275

A new utility, tungsten_skip_seqno has been added which allows events to be skipped based on filters, allowing the Tungsten Replicator to come back online with less manual intervention.

Issues: CT-42, CT-1087, CT-1453, CT-1511

Full support for proxy-protocol v1 has been added to Tungsten Connector.

Proxy protocol specifications can found here: http://www.haproxy.org/download/1.8/doc/proxy-protocol.txt

Enabling proxy-protocol:

You should now see the origin IP of the connecting application in place of the connector IP in show processlist and tungsten show processlist commands.

Furher reading:

Issues: CT-1533

This release includes a wide range of improvements and bug fixes for tpm and other command-line tools.

Issues: CT-1509

Fixes an issue where the undeployall command would create a root owned wrapper.log in the ./tools directory.

Issues: CT-1516

Fixed a tpm call to systemd services listing which was making tpm wrongly fall back to init.d. This issue was found when testing MariaDB 10.0.

Issues: CT-1524

tpm now supports new MariaDB my_print_defaults flag --defaults-file

Issues: CT-1528

Fixes an issue where null values in JSON structures were incorrectly quoted once replicated.

For example :

[{"example":null}]

would be replicated as

[{"example":"null"}]

Issues: CT-1523

Now properly supporting negative write affinity in Composite Active/Active clusters. For example you can now specify affinity=localSite,-remoteSite:localSite This allows access only to local sites for writes but still allows reads from remote sites if the local one goes down.

Issues: CT-1507

Fixes an issue in Composite Active/Active environments that would occasionally cause the primary node to be set to read only.

Issues: CT-1424

A new --no-routers cli option has been added to the cctrl command.

Using cctrl --no-routers will suppress the collection and display of connectors when issuing the ls command. This is especially useful when there are a lot of connectors installed and gathering the information from them could take a lot of time, causing unexpected timeouts.

Issues: CT-1437

A new affinity feature has been added in this release that will allow you to define a dedicated datasource for writes within a Composite Active/Active deployment.

This feature can be particularly useful to direct all writes for a specific user/application to a dedicated Write endpoint, where you need to remove any chance of writes in multiple clusters conflicting.

This feature can be enabled in a number of ways :- Via edits to the user.map file for Proxy configurations, via calls during connection, or via the --connector-affinity tpm option if using Bridge mode.

For more details on configuring this, and examples, consult the connector specific documentation here:

Issues: CT-1369, CT-1457

Minor improvements to rule handling in the managers.

Issues: CT-1464

Support has now been added to allow monitoring via the New Relic Insights API, using a new script tungsten_newrelic_event

Issues: CT-892

If using mysqldump as your preferred choice of backup method, Stored Procedures and Functions would not be included in the generated dump file when using tungsten_provision_slave.

Issues: CT-1339

In a Composite Active/Active environment, the tungsten_post_process command now intelligently handles cross-site filter properties in the INI so that any defined filters are appended to the default values for any option that resolves to property replicator.stage.q-to-dbms.filters

Issues: CT-1372, CT-1449

Replaced md5 hashing to sha256 within tpm to fix issues where tpm install would fail on FIPS enabled environments.

Issues: CT-1451

When provisioning from a Primary node, in a Composite Active/Active environment, using tungsten_provision_slave, the replicator would fail to reset due to the incorrect service name being used.

Issues: CT-1458

Fixed an issue where, in Composite Active/Active environments, tpm would ignore the affinity passed with the tpm flag --connector-affinity.

Issues: CT-1465

If using tungsten_provision_slave to restore a previously failed Primary node, the provision could fail to recover the node.

Issues: CT-1468

A new mapcharset filter is included, which fixes collation incompatibilities when replicating from MySQL 8 to an older release (5.7 or 5.6)

This new filter should be enabled within any MySQL applier where the source is generated from a MySQL 8 database and the target is MySQL 5.7 or lower.

Issues: CT-1404

If SSL is enabled between replicators for THL Transfer, and a source node fails and becomes unresponsive, or is slow/overloaded, the replicator on the replica can fail to go offline due to a timeout, remaining in a GOING-ONLINE:SYNCHRONIZING state.

In a clustering envronment, this can prevent failovers occuring due to the manager waiting for the replicator which is stuck in a timeout loop.

Issues: CT-1459

Fixed a bug in the Hedera library that prevented Managers to pass through the init state.

Issues: CT-1415

tpm diag now collects routing table information via route -n , and has two new command-line arguments: --include and --groups.

--include specifies a comma-separated list of subroutines to include. Any gather subroutine not listed will be skipped.

--groups specifies a comma-separated list of subroutine groups to include. Any group not listed will be skipped.

Issues: CT-1399

A number of sql_modes have been removed in MySQL 8.0, meaning that when replicating from an earlier version (e.g. 5.7) the applier could error when applying session variables.

A new dropsqlmode filter is now available that should be configured against appliers where the source is version 5.7 or earlier and the target is version 8.0 or later.

Issues: CT-1361

If parallel apply has been enabled, automatic failovers could hang indefinetly if one of more channels are inactive.

Issues: CT-1407