13.5. Functional Testing

13.5. Functional Testing
Prev	^Up	Chapter 13. Performance, Tuning and Testing	Next

13.5. Functional Testing

13.5.1. Manual and Automatic Failover
13.5.2. Backup and Restore
13.5.3. Connectivity
13.5.4. Performance and Other Tests

The following chapter includes a number of suggested functional tests that can be performed following installation in a Testing/POC environment

For a test to be successful, normal operations should be restored after every test. The default Primary server should regain Primary role with all replicators and data sources ONLINE

The tests are split into the following categories:

A PDF version of these tests can be downloaded by clicking here

13.5.1. Manual and Automatic Failover

Failover Test 1 - Administrator uses Tungsten to promote new Primary

Scenario	shell> `cctrl` cctrl> `use usa` [LOGICAL] /usa> `switch`
Expectation	The Primary role will move a different server. Remaining servers will be reconfigured accordingly. Examine how the application performs during the process Data should be consistent

Failover Test 2 – Manually kill the Primary database process

Scenario	shell> `cctrl` cctrl> `use usa` [LOGICAL] /usa> `service Primary/mysql stop`
Expectation	The database server stops and the Primary role is moved to another server. Examine how the application performs during the process Data should be consistent

Failover Test 3 – Remove power from the Primary database server

Scenario	Pull the power plug on the Primary server or run a restart command if that is not an option.
Expectation	The database server stops and the Primary role is moved to another server. Examine how the application performs during the process Data should be consistent

Failover Test 4 – Manually kill a Replica database process

Scenario	shell> `cctrl` cctrl> `use usa` [LOGICAL] /usa> `service Replica1/mysql stop`
Expectation	The Replica data source is marked as FAILED. Examine how the application performs during the process

13.5.2. Backup and Restore

Backup Test 1 – Take a backup of a Replica and restore it to the same server

Scenario	shell> `cctrl` cctrl> `use usa` [LOGICAL] /usa> `datasource Replica1 backup` [LOGICAL] /usa> `datasource Replica1 restore`
Expectation	The commands should complete successfully.

Backup Test 2 – Restore the backup to another server

Scenario	shell> `rsync -avze ssh /opt/continuent/backups/ Replica2:/opt/continuent/backups/` shell> `cctrl` cctrl> `use usa` [LOGICAL] /usa> `datasource Replica2 restore`
Expectation	The commands should complete successfully.

Backup Test 3 – Take a backup of the Primary and restore to a Replica

Scenario

shell> cctrl
cctrl> use usa
[LOGICAL] /usa> datasource Primary backup
Primary> rsync -avze ssh /opt/continuent/backups/ Replica2:/opt/continuent/backups/
[LOGICAL] /usa> datasource Replica2 restore

Expectation

The restore command should complete successfully, or.
Run `trepctl online -from-event ######:########` to bring the Replica ONLINE

13.5.3. Connectivity

Connectivity Test 1 – Connect to the connector and verify Primary host

Scenario	shell> mysql -h`hostname` -P9999 -uapp -p -e"select @@hostname for update"
Expectation	The Primary hostname is returned.

Connectivity Test 2 – Verify access to Replicas

This test is only required if read/write splitting has been enabled. It should be run from a connector running on a server other than the Primary.

Scenario	shell> mysql -h`hostname` -P9999 -uapp -p -e"select @@hostname"
Expectation	A hostname is returned. Repeat the process on other hosts until a Replica hostname is returned

Connectivity Test 3 – Verify access to the Primary before and after a switch

Scenario	shell> `cctrl` cctrl> `use usa` [LOGICAL] /usa> `switch`
Expectation	Run Test 1 again and verify the new Primary hostname is returned.

13.5.4. Performance and Other Tests

Performance Test 1 – Run a load test against the cluster

Scenario	Run load tests of some variety against the cluster to ensure the Connector and Replicator properly handle the load. Recommended load test tools are: HammerDB. sysbench
Expectation	Solution can handle 1TB of data with a minimum of 24k reads per minute and 1k writes per minute, over three tests. Insert only from multiple locations Read/write test Heavy read test with few writes

Replicator Testing

Scenario	Evaluate system responsiveness in conjunction with performance tests (above). This is not a test of network speed
Expectation	Quick replication latency betweens regions (<100ms greater than the current ping latency)

Network Partition Testing

Scenario	Simulate a partitioned network (for example, by modifying security group rules in AWS), and continue to do reads and writes on multiple clusters for 30 minutes.
Expectation	After resolving the partition, clusters should resync.

Scenario	Create a similar network partition, and write the same record on both sides of the partition
Expectation	After resolving the partition, the replicators on both sides should report errors. Demonstrate possible resolutions: Remove record on side, skip record on the other side, and bring replication back online. The record not removed will be replicated Remove both records and skip both transactions Modify records and bring replicators online

Prev	Up	Next
13.4. Connector Memory Management	^Level	Appendix A. Release Notes

Continuent Documentation