11.5. Functional Testing

The following chapter includes a number of suggested functional tests that can be performed following installation in a Testing/POC environment

For a test to be successful, normal operations should be restored after every test. The default Primary server should regain Primary role with all replicators and data sources ONLINE

The tests are split into the following categories:

A PDF version of these tests can be downloaded by clicking here

11.5.1. Manual and Automatic Failover

Failover Test 1 - Administrator uses Tungsten to promote new Primary

Scenario

shell> cctrl
cctrl> use usa
[LOGICAL] /usa> switch

Expectation

  • The Primary role will move a different server. Remaining servers will be reconfigured accordingly.

  • Examine how the application performs during the process

  • Data should be consistent

Failover Test 2 – Manually kill the Primary database process

Scenario

shell> cctrl
cctrl> use usa
[LOGICAL] /usa> service Primary/mysql stop

Expectation

  • The database server stops and the Primary role is moved to another server.

  • Examine how the application performs during the process

  • Data should be consistent

Failover Test 3 – Remove power from the Primary database server

Scenario

Pull the power plug on the Primary server or run a restart command if that is not an option.

Expectation

  • The database server stops and the Primary role is moved to another server.

  • Examine how the application performs during the process

  • Data should be consistent

Failover Test 4 – Manually kill a Replica database process

Scenario

shell> cctrl
cctrl> use usa
[LOGICAL] /usa> service Replica1/mysql stop

Expectation

  • The Replica data source is marked as FAILED.

  • Examine how the application performs during the process

11.5.2. Backup and Restore

Backup Test 1 – Take a backup of a Replica and restore it to the same server

Scenario

shell> cctrl
cctrl> use usa
[LOGICAL] /usa> datasource Replica1 backup
[LOGICAL] /usa> datasource Replica1 restore

Expectation

  • The commands should complete successfully.

Backup Test 2 – Restore the backup to another server

Scenario

shell> rsync -avze ssh /opt/continuent/backups/ Replica2:/opt/continuent/backups/
shell> cctrl
cctrl> use usa
[LOGICAL] /usa> datasource Replica2 restore

Expectation

  • The commands should complete successfully.

Backup Test 3 – Take a backup of the Primary and restore to a Replica

Scenario

shell> cctrl
cctrl> use usa
[LOGICAL] /usa> datasource Primary backup
Primary> rsync -avze ssh /opt/continuent/backups/ Replica2:/opt/continuent/backups/
[LOGICAL] /usa> datasource Replica2 restore

Expectation

  • The restore command should complete successfully, or.

  • Run `trepctl online -from-event ######:########` to bring the Replica ONLINE

11.5.3. Connectivity

Connectivity Test 1 – Connect to the connector and verify Primary host

Scenario

shell> mysql -h`hostname` -P9999 -uapp -p -e"select @@hostname for update"

Expectation

  • The Primary hostname is returned.

Connectivity Test 2 – Verify access to Replicas

This test is only required if read/write splitting has been enabled. It should be run from a connector running on a server other than the Primary.

Scenario

shell> mysql -h`hostname` -P9999 -uapp -p -e"select @@hostname"

Expectation

  • A hostname is returned.

  • Repeat the process on other hosts until a Replica hostname is returned

Connectivity Test 3 – Verify access to the Primary before and after a switch

Scenario

shell> cctrl
cctrl> use usa
[LOGICAL] /usa> switch

Expectation

  • Run Test 1 again and verify the new Primary hostname is returned.

11.5.4. Performance and Other Tests

Performance Test 1 – Run a load test against the cluster

Scenario

Run load tests of some variety against the cluster to ensure the Connector and Replicator properly handle the load. Recommended load test tools are:

  • HammerDB.

  • sysbench

Expectation

Solution can handle 1TB of data with a minimum of 24k reads per minute and 1k writes per minute, over three tests.

  • Insert only from multiple locations

  • Read/write test

  • Heavy read test with few writes

Replicator Testing

Scenario

Evaluate system responsiveness in conjunction with performance tests (above). This is not a test of network speed

Expectation

Quick replication latency betweens regions (<100ms greater than the current ping latency)

Network Partition Testing

Scenario

Simulate a partitioned network (for example, by modifying security group rules in AWS), and continue to do reads and writes on multiple clusters for 30 minutes.

Expectation

After resolving the partition, clusters should resync.

Scenario

Create a similar network partition, and write the same record on both sides of the partition

Expectation

After resolving the partition, the replicators on both sides should report errors. Demonstrate possible resolutions:

  • Remove record on side, skip record on the other side, and bring replication back online. The record not removed will be replicated

  • Remove both records and skip both transactions

  • Modify records and bring replicators online