3.4.5. Resetting a single dataservice

Under certain conditions, dataservices in an active/active configuration may drift and/or become inconsistent with the data in another dataservice. If this occurs, you may need to re-provision the data on one or more of the dataservices after first determining the definitive source of the information.

In the following example the west service has been determined to be the definitive copy of the data. To fix the issue, all the datasources in the east service will be reprovisioned from one of the datasources in the west service.

The following is a guide to the steps that should be followed. In the example procedure it is the east service that has failed:

  1. Put the dataservice into MAINTENANCE mode. This ensures that Tungsten Cluster will not attempt to automatically recover the service.

    cctrl [east]> set policy maintenance
  2. On the east, failed, Tungsten Cluster service, put each Tungsten Connector offline:

    cctrl [east]> router * offline
  3. Reset the local failed service on all servers connected to the remote failed service. For example, on west{1,2,3} reset the west_from_east service:

    shell west> trepctl -service west_from_east offline
    shell west> trepctl -service west_from_east reset -all -y
  4. Reset the local service on each server within the failed region (east{1,2,3}):

    shell east> trepctl -service east offline
    shell east> trepctl -service east reset -all -y
  5. Restore a backup on each host (east{1,2,3}) in the failed east service from a host in the west service:

    shell east> tungsten_provision_slave \
        --direct --source=west1
  6. Place all the services on west{1,2,3} back online:

    shell west> trepctl -service west_from_east online
  7. On the east, failed, Tungsten Cluster service, put each Tungsten Connector online:

    cctrl [east]> router * online
  8. Set the policy back to automatic:

    cctrl> set policy automatic