Under certain conditions, dataservices in an active/active configuration may drift and/or become inconsistent with the data in another dataservice. If this occurs, you may need to re-provision the data on one or more of the dataservices after first determining the definitive source of the information.
In the following example the west
service has been determined to be the definitive copy of the data. To fix
the issue, all the datasources in the
east service will be reprovisioned
from one of the datasources in the
west service.
The following is a guide to the steps that should be followed. In the
example procedure it is the
east service
that has failed:
Put the dataservice into
MAINTENANCE mode. This
ensures that Tungsten Cluster will not attempt to automatically recover
the service.
cctrl [east]> set policy maintenance
On the east, failed,
Tungsten Cluster service, put each Tungsten Connector offline:
cctrl [east]> router * offline
Reset the local failed service on all servers connected to
the remote failed service. For example, on
west{1,2,3} reset the
west_from_east service:
shell west>trepctl -service west_from_east offlineshell west>trepctl -service west_from_east reset -all -y
Reset the local service on each server within the failed
region (east{1,2,3}):
shell east>trepctl -service east offlineshell east>trepctl -service east reset -all -y
Restore a backup on each host
(east{1,2,3}) in the failed
east service from a host in the
west service:
shell east> tungsten_provision_slave \
--direct --source=west1
Place all the services on
west{1,2,3} back online:
shell west> trepctl -service west_from_east online
On the east, failed,
Tungsten Cluster service, put each Tungsten Connector online:
cctrl [east]> router * onlineSet the policy back to automatic:
cctrl> set policy automatic