5.7.2. Composite Cluster Site Failover (Forced Switch)

In the event the Primary site goes down, and a graceful manual switch is not possible, the composite master role can be failed over to the Disaster Recovery cluster using cctrl. The failover command performs the forced switch operation. It will try to update the configuration of the east data service but will not fail if not successful.

In this example, hosts db1 (the composite master), db2 and db3 in cluster east have been shut down. To force dataservice west to become the primary, login to a node in that cluster and get into cctrl:

shell> cctrl -multi
Tungsten Clustering 5.1.1 build 202
west: session established
[LOGICAL] / > use global
[LOGICAL] /global > ls

COORDINATOR[db4:AUTOMATIC:ONLINE]

DATASOURCES:
+----------------------------------------------------------------------------+
|east(composite master:SHUNNED(FAILSAFE_SHUN))                               |
|STATUS [SHUNNED] [2015/04/14 01:46:59 AM UTC]                               |
+----------------------------------------------------------------------------+

+----------------------------------------------------------------------------+
|west(composite slave:ONLINE)                                                |
|STATUS [OK] [2015/04/14 01:46:59 AM UTC]                                    |
+----------------------------------------------------------------------------+

Mark the east data service as failed to prevent further actions:

[LOGICAL] /global > datasource east fail

WARNING: This is an expert-level command:
Incorrect use may cause data corruption
or make the cluster unavailable.

Do you want to continue? (y/n)> y
WARNING: UNABLE TO REACH PHYSICAL DATA SERVICE 'east' AT THIS TIME.
EXCEPTION: Unable to unable to continue with command because no manager is available in service 'east'.

CONTINUING WITH COMMAND
COMPOSITE DATA SOURCE 'east' IS NOW IN THE FAILED STATE

[LOGICAL] /global > ls

COORDINATOR[db4:AUTOMATIC:ONLINE]

DATASOURCES:
+----------------------------------------------------------------------------+
|east(composite master:FAILED(MANUALLY-FAILED))                              |
|STATUS [CRITICAL] [2015/04/14 03:11:27 PM UTC]                              |
|REASON[MANUALLY-FAILED]                                                     |
+----------------------------------------------------------------------------+

+----------------------------------------------------------------------------+
|west(composite slave:ONLINE)                                                |
|STATUS [OK] [2015/04/14 02:37:32 PM UTC]                                    |
+----------------------------------------------------------------------------+

Issue the failover command to force the west dataservice to become the composite master:

[LOGICAL] /global > failover
WARNING: DATA SERVICE 'east' IS NOT AVAILABLE. CANNOT GET STATE
WARNING: CAN'T GET POLICY MODE FOR SERVICE 'east'. CONTINUING.
WARNING: CAN'T SET POLICY MODE 'maintenance' FOR SERVICE 'east'. CONTINUING.
SELECTED SLAVE: 'west@global'
WARNING: UNABLE TO REACH PHYSICAL DATA SERVICE 'east' AT THIS TIME.
EXCEPTION: Unable to unable to continue with command because no manager is available in service 'east'.

CONTINUING WITH COMMAND
ENSURING THAT WE CATCH UP WITH THE MOST ADVANCED RELAY
composite data source 'west@global' is now OFFLINE
WARNING: UNABLE TO REACH PHYSICAL DATA SERVICE 'west' AT THIS TIME.
EXCEPTION: Unable to unable to continue with command because no manager is available in service 'east'.

CONTINUING WITH COMMAND
PUT THE NEW MASTER 'west@global' ONLINE
WARNING: CAN'T SET POLICY MODE 'AUTOMATIC' FOR SERVICE 'east'. CONTINUING.
REVERT POLICY: MAINTENANCE => AUTOMATIC
FAILOVER TO 'west@global' WAS SUCCESSFUL
[LOGICAL] /global > ls

COORDINATOR[db4:AUTOMATIC:ONLINE]

DATASOURCES:
+----------------------------------------------------------------------------+
|east(composite master:SHUNNED(MANUAL-FAILOVER))                             |
|STATUS [SHUNNED] [2015/04/14 02:13:18 AM UTC]                               |
+----------------------------------------------------------------------------+

+----------------------------------------------------------------------------+
|west(composite master:ONLINE)                                               |
|STATUS [OK] [2015/04/14 02:13:23 AM UTC]                                    |
+----------------------------------------------------------------------------+

Composite Master Dataservice (Primary) - west

[LOGICAL] /global > use west 
[LOGICAL] /west > ls

COORDINATOR[db4:AUTOMATIC:ONLINE]

ROUTERS:
+----------------------------------------------------------------------------+
|connector@db4[9757](ONLINE, created=0, active=0)              |
|connector@db5[9781](ONLINE, created=0, active=0)              |
|connector@db6[9944](ONLINE, created=0, active=0)              |
+----------------------------------------------------------------------------+

DATASOURCES:
+----------------------------------------------------------------------------+
|db4(master:ONLINE, progress=7, THL latency=0.110)                           |
|STATUS [OK] [2015/04/14 02:13:23 AM UTC]                                    |
+----------------------------------------------------------------------------+
|  MANAGER(state=ONLINE)                                                     |
|  REPLICATOR(role=master, state=ONLINE)                                     |
|  DATASERVER(state=ONLINE)                                                  |
|  CONNECTIONS(created=0, active=0)                                          |
+----------------------------------------------------------------------------+

+----------------------------------------------------------------------------+
|db5(slave:ONLINE, progress=7, latency=0.172)                                |
|STATUS [OK] [2015/04/14 02:13:23 AM UTC]                                    |
+----------------------------------------------------------------------------+
|  MANAGER(state=ONLINE)                                                     |
|  REPLICATOR(role=slave, master=db4, state=ONLINE)                          |
|  DATASERVER(state=ONLINE)                                                  |
|  CONNECTIONS(created=0, active=0)                                          |
+----------------------------------------------------------------------------+

+----------------------------------------------------------------------------+
|db6(slave:ONLINE, progress=7, latency=0.173)                                |
|STATUS [OK] [2015/04/14 02:13:23 AM UTC]                                    |
+----------------------------------------------------------------------------+
|  MANAGER(state=ONLINE)                                                     |
|  REPLICATOR(role=slave, master=db4, state=ONLINE)                          |
|  DATASERVER(state=ONLINE)                                                  |
|  CONNECTIONS(created=0, active=0)                                          |
+----------------------------------------------------------------------------+