6.13.1. Reset a Single Site in a Multi-Site/Active-Active Topology


The procedures in this section are designed for the Multi-Site/Active-Active topology ONLY. Do NOT use these procedures with version 6.x Composite Active/Active.

For version 6.x Composite Active/Active, please refer to Deploying Composite Active/Active Clustering.

Under certain conditions, dataservices in a Multi-Site/Active-Active configuration may drift and/or become inconsistent with the data in another dataservice. If this occurs, you may need to re-provision the data on one or more of the dataservices after first determining the definitive source of the information.

In the following example the west service has been determined to be the definitive copy of the data. To fix the issue, all the datasources in the east service will be reprovisioned from one of the datasources in the west service.

The following is a guide to the steps that should be followed. In the example procedure it is the east service that has failed:

  1. Put the dataservice into MAINTENANCE mode. This ensures that Tungsten Cluster will not attempt to automatically recover the service.

    cctrl [east]> set policy maintenance
  2. Put the Tungsten Connector for the dataservice offline:

    cctrl [east]> router * offline
  3. Stop all services running in east:

    shell east> /opt/continuent/tungsten/cluster-home/bin/stopall
    shell east> /opt/replicator/tungsten/cluster-home/bin/stopall
  4. Disable cross-site replication and reset the replication position:

    shell west> /opt/replicator/tungsten/tungsten-replicator/bin/trepctl -service east offline
    shell west> /opt/replicator/tungsten/tungsten-replicator/bin/trepctl -service east reset -all -y
  5. Reprovision the Primary node in east:

    shell east{Primary}> /opt/continuent/tungsten/tungsten-replicator/scripts/tungsten_provision_slave --source west{Replica}
  6. Restart the services in east:

    shell east> /opt/continuent/tungsten/cluster-home/bin/startall
    shell east> /opt/continuent/tungsten/tungsten-replicator/bin/trepctl online
    shell east> /opt/replicator/tungsten/cluster-home/bin/startall
  7. Ensure that the new service is functioning normally:

    shell east> echo ls | /opt/continuent/tungsten/tungsten-manager/bin/cctrl
    shell east> /opt/replicator/tungsten/tungsten-replicator/bin/trepctl status
  8. Bring the remote replicators back ONLINE

    shell west> /opt/replicator/tungsten/tungsten-replicator/bin/trepctl -service east online

    Set the cluster to normal operational state:

    cctrl> router * online
    cctrl> set policy automatic
  9. Reprovision the remaining nodes in the east cluster

    shell east{Replica1}> /opt/continuent/tungsten/tungsten-replicator/scripts/tungsten_provision_slave \
        --source east{Primary}
    shell east{Replica1}> /opt/continuent/tungsten/cluster-home/bin/startall
    shell east{Replica1}> /opt/replicator/tungsten/cluster-home/bin/startall
    shell east{Replica2}> /opt/continuent/tungsten/tungsten-replicator/scripts/tungsten_provision_slave \
        --source east{Replica1}
    shell east{Replica2}> /opt/continuent/tungsten/cluster-home/bin/startall
    shell east{Replica2}> /opt/replicator/tungsten/cluster-home/bin/startall