The first step in recovering the SHUNNED dataservice is to re-provision the nodes if the data has gotten out of sync. See Section 6.6.1.1, “Provision or Reprovision a Replica” for more information.
Once the failed site has been restored, the shunned/superseded dataservice can be brought back online using cctrl . The recover command performs this operation, annotating the progress.
shell> cctrl -multi
Tungsten Cluster 7.1.4 build 10
west: session established
[LOGICAL] / > use global
[LOGICAL] /global > ls
COORDINATOR[db4:AUTOMATIC:ONLINE]
DATASOURCES:
+----------------------------------------------------------------------------+
|east(composite master:SHUNNED(SUPERSEDED)) |
|STATUS [SHUNNED] [2015/04/14 02:28:53 AM UTC] |
+----------------------------------------------------------------------------+
+----------------------------------------------------------------------------+
|west(composite master:ONLINE) |
|STATUS [OK] [2015/04/14 02:13:23 AM UTC] |
+----------------------------------------------------------------------------+
SHUNNED(SUPERSEDED) Composite Active Dataservice -
east
[LOGICAL] / > use east
[LOGICAL] /east > ls
COORDINATOR[db2:AUTOMATIC:ONLINE]
ROUTERS:
+----------------------------------------------------------------------------+
|connector@db1[10051](ONLINE, created=0, active=0) |
|connector@db2[16111](ONLINE, created=0, active=0) |
|connector@db3[16036](ONLINE, created=0, active=0) |
|connector@db4[9757](ONLINE, created=1, active=0) |
|connector@db5[9781](ONLINE, created=1, active=0) |
|connector@db6[9944](ONLINE, created=1, active=0) |
+----------------------------------------------------------------------------+
DATASOURCES:
+----------------------------------------------------------------------------+
|db1(master:SHUNNED(SUPERSEDED), progress=7, THL latency=0.934) |
|STATUS [SHUNNED] [2015/04/14 02:28:53 AM UTC] |
+----------------------------------------------------------------------------+
| MANAGER(state=ONLINE) |
| REPLICATOR(role=master, state=ONLINE) |
| DATASERVER(state=ONLINE) |
| CONNECTIONS(created=3, active=0) |
+----------------------------------------------------------------------------+
+----------------------------------------------------------------------------+
|db2(slave:SHUNNED(SUPERSEDED), progress=7, latency=6.488) |
|STATUS [SHUNNED] [2015/04/14 02:28:53 AM UTC] |
+----------------------------------------------------------------------------+
| MANAGER(state=ONLINE) |
| REPLICATOR(role=slave, master=db1, state=ONLINE) |
| DATASERVER(state=ONLINE) |
| CONNECTIONS(created=0, active=0) |
+----------------------------------------------------------------------------+
+----------------------------------------------------------------------------+
|db3(slave:SHUNNED(SUPERSEDED), progress=7, latency=11.164) |
|STATUS [SHUNNED] [2015/04/14 02:28:53 AM UTC] |
+----------------------------------------------------------------------------+
| MANAGER(state=ONLINE) |
| REPLICATOR(role=slave, master=db1, state=ONLINE) |
| DATASERVER(state=ONLINE) |
| CONNECTIONS(created=0, active=0) |
+----------------------------------------------------------------------------+
Use the recover to bring the SHUNNED dataservice back online as a composite Replica:
[LOGICAL] /global > recover
IDENTIFIED DATASOURCE 'east@global' FOR RECOVERY
COULD NOT IDENTIFY ACTIVE PRIMARY FOR SERVICE 'east'
ATTEMPTING TO IDENTIFY A FAILED PRIMARY FOR 'east'
PHYSICAL DATA SERVICE 'east' DOES NOT HAVE AN ACTIVE RELAY
FORCING THE PHYSICAL RELAY TO BE 'db1'
DATASOURCE 'db1@east' IS NOW A RELAY
RECOVERED 2 DATA SOURCES IN SERVICE 'east'
composite data source 'east@global' role is now SLAVE
composite data source 'east' is now OFFLINE
REVERT SET POLICY AUTOMATIC
RECOVERY OF COMPOSITE SERVICE 'global' IS COMPLETE
[LOGICAL] /global > ls
COORDINATOR[db2:AUTOMATIC:ONLINE]
DATASOURCES:
+----------------------------------------------------------------------------+
|east(composite slave:ONLINE) |
|STATUS [OK] [2015/04/14 04:12:01 AM UTC] |
+----------------------------------------------------------------------------+
+----------------------------------------------------------------------------+
|west(composite master:ONLINE) |
|STATUS [OK] [2015/04/14 02:28:53 AM UTC] |
+----------------------------------------------------------------------------+
Recovered Composite Passive Dataservice (DR) -
east
[LOGICAL] /global > use east
[LOGICAL] /east > ls
COORDINATOR[db2:AUTOMATIC:ONLINE]
ROUTERS:
+----------------------------------------------------------------------------+
|connector@db1.vagrant-erics[10051](ONLINE, created=0, active=0) |
|connector@db2.vagrant-erics[16111](ONLINE, created=0, active=0) |
|connector@db3.vagrant-erics[16036](ONLINE, created=0, active=0) |
|connector@db4.vagrant-erics[9757](ONLINE, created=1, active=0) |
|connector@db5.vagrant-erics[9781](ONLINE, created=1, active=0) |
|connector@db6.vagrant-erics[9944](ONLINE, created=1, active=0) |
+----------------------------------------------------------------------------+
DATASOURCES:
+----------------------------------------------------------------------------+
|db1(relay:ONLINE, progress=7, latency=0.000) |
|STATUS [OK] [2015/04/14 04:11:52 AM UTC] |
+----------------------------------------------------------------------------+
| MANAGER(state=ONLINE) |
| REPLICATOR(role=relay, master=db4, state=ONLINE) |
| DATASERVER(state=ONLINE) |
| CONNECTIONS(created=3, active=0) |
+----------------------------------------------------------------------------+
+----------------------------------------------------------------------------+
|db2(slave:ONLINE, progress=7, latency=6.000) |
|STATUS [OK] [2015/04/14 04:11:56 AM UTC] |
+----------------------------------------------------------------------------+
| MANAGER(state=ONLINE) |
| REPLICATOR(role=slave, master=db1, state=ONLINE) |
| DATASERVER(state=ONLINE) |
| CONNECTIONS(created=0, active=0) |
+----------------------------------------------------------------------------+
+----------------------------------------------------------------------------+
|db3(slave:ONLINE, progress=7, latency=11.000) |
|STATUS [OK] [2015/04/14 04:12:01 AM UTC] |
+----------------------------------------------------------------------------+
| MANAGER(state=ONLINE) |
| REPLICATOR(role=slave, master=db1, state=ONLINE) |
| DATASERVER(state=ONLINE) |
| CONNECTIONS(created=0, active=0) |
+----------------------------------------------------------------------------+