The first step in recovering the SHUNNED dataservice is to re-provision the nodes if the data has gotten out of sync. See Section 6.6.1.1, “Provision or Reprovision a Replica” for more information.
Once the failed site has been restored, the shunned/superseded dataservice can be brought back online using cctrl . The recover command performs this operation, annotating the progress.
shell>cctrl -multiTungsten Cluster 7.1.4 build 10 west: session established [LOGICAL] / >use global[LOGICAL] /global >lsCOORDINATOR[db4:AUTOMATIC:ONLINE] DATASOURCES: +----------------------------------------------------------------------------+ |east(composite master:SHUNNED(SUPERSEDED)) | |STATUS [SHUNNED] [2015/04/14 02:28:53 AM UTC] | +----------------------------------------------------------------------------+ +----------------------------------------------------------------------------+ |west(composite master:ONLINE) | |STATUS [OK] [2015/04/14 02:13:23 AM UTC] | +----------------------------------------------------------------------------+
SHUNNED(SUPERSEDED) Composite Active Dataservice -
east
[LOGICAL] / >use east[LOGICAL] /east >lsCOORDINATOR[db2:AUTOMATIC:ONLINE] ROUTERS: +----------------------------------------------------------------------------+ |connector@db1[10051](ONLINE, created=0, active=0) | |connector@db2[16111](ONLINE, created=0, active=0) | |connector@db3[16036](ONLINE, created=0, active=0) | |connector@db4[9757](ONLINE, created=1, active=0) | |connector@db5[9781](ONLINE, created=1, active=0) | |connector@db6[9944](ONLINE, created=1, active=0) | +----------------------------------------------------------------------------+ DATASOURCES: +----------------------------------------------------------------------------+ |db1(master:SHUNNED(SUPERSEDED), progress=7, THL latency=0.934) | |STATUS [SHUNNED] [2015/04/14 02:28:53 AM UTC] | +----------------------------------------------------------------------------+ | MANAGER(state=ONLINE) | | REPLICATOR(role=master, state=ONLINE) | | DATASERVER(state=ONLINE) | | CONNECTIONS(created=3, active=0) | +----------------------------------------------------------------------------+ +----------------------------------------------------------------------------+ |db2(slave:SHUNNED(SUPERSEDED), progress=7, latency=6.488) | |STATUS [SHUNNED] [2015/04/14 02:28:53 AM UTC] | +----------------------------------------------------------------------------+ | MANAGER(state=ONLINE) | | REPLICATOR(role=slave, master=db1, state=ONLINE) | | DATASERVER(state=ONLINE) | | CONNECTIONS(created=0, active=0) | +----------------------------------------------------------------------------+ +----------------------------------------------------------------------------+ |db3(slave:SHUNNED(SUPERSEDED), progress=7, latency=11.164) | |STATUS [SHUNNED] [2015/04/14 02:28:53 AM UTC] | +----------------------------------------------------------------------------+ | MANAGER(state=ONLINE) | | REPLICATOR(role=slave, master=db1, state=ONLINE) | | DATASERVER(state=ONLINE) | | CONNECTIONS(created=0, active=0) | +----------------------------------------------------------------------------+
Use the recover to bring the SHUNNED dataservice back online as a composite Replica:
[LOGICAL] /global >recoverIDENTIFIED DATASOURCE 'east@global' FOR RECOVERY COULD NOT IDENTIFY ACTIVE PRIMARY FOR SERVICE 'east' ATTEMPTING TO IDENTIFY A FAILED PRIMARY FOR 'east' PHYSICAL DATA SERVICE 'east' DOES NOT HAVE AN ACTIVE RELAY FORCING THE PHYSICAL RELAY TO BE 'db1' DATASOURCE 'db1@east' IS NOW A RELAY RECOVERED 2 DATA SOURCES IN SERVICE 'east' composite data source 'east@global' role is now SLAVE composite data source 'east' is now OFFLINE REVERT SET POLICY AUTOMATIC RECOVERY OF COMPOSITE SERVICE 'global' IS COMPLETE [LOGICAL] /global >lsCOORDINATOR[db2:AUTOMATIC:ONLINE] DATASOURCES: +----------------------------------------------------------------------------+ |east(composite slave:ONLINE) | |STATUS [OK] [2015/04/14 04:12:01 AM UTC] | +----------------------------------------------------------------------------+ +----------------------------------------------------------------------------+ |west(composite master:ONLINE) | |STATUS [OK] [2015/04/14 02:28:53 AM UTC] | +----------------------------------------------------------------------------+
Recovered Composite Passive Dataservice (DR) -
east
[LOGICAL] /global >use east[LOGICAL] /east >lsCOORDINATOR[db2:AUTOMATIC:ONLINE] ROUTERS: +----------------------------------------------------------------------------+ |connector@db1.vagrant-erics[10051](ONLINE, created=0, active=0) | |connector@db2.vagrant-erics[16111](ONLINE, created=0, active=0) | |connector@db3.vagrant-erics[16036](ONLINE, created=0, active=0) | |connector@db4.vagrant-erics[9757](ONLINE, created=1, active=0) | |connector@db5.vagrant-erics[9781](ONLINE, created=1, active=0) | |connector@db6.vagrant-erics[9944](ONLINE, created=1, active=0) | +----------------------------------------------------------------------------+ DATASOURCES: +----------------------------------------------------------------------------+ |db1(relay:ONLINE, progress=7, latency=0.000) | |STATUS [OK] [2015/04/14 04:11:52 AM UTC] | +----------------------------------------------------------------------------+ | MANAGER(state=ONLINE) | | REPLICATOR(role=relay, master=db4, state=ONLINE) | | DATASERVER(state=ONLINE) | | CONNECTIONS(created=3, active=0) | +----------------------------------------------------------------------------+ +----------------------------------------------------------------------------+ |db2(slave:ONLINE, progress=7, latency=6.000) | |STATUS [OK] [2015/04/14 04:11:56 AM UTC] | +----------------------------------------------------------------------------+ | MANAGER(state=ONLINE) | | REPLICATOR(role=slave, master=db1, state=ONLINE) | | DATASERVER(state=ONLINE) | | CONNECTIONS(created=0, active=0) | +----------------------------------------------------------------------------+ +----------------------------------------------------------------------------+ |db3(slave:ONLINE, progress=7, latency=11.000) | |STATUS [OK] [2015/04/14 04:12:01 AM UTC] | +----------------------------------------------------------------------------+ | MANAGER(state=ONLINE) | | REPLICATOR(role=slave, master=db1, state=ONLINE) | | DATASERVER(state=ONLINE) | | CONNECTIONS(created=0, active=0) | +----------------------------------------------------------------------------+