3.2.8. Provisioning during live operations

In the event of a failure within one host in the service where you need to reprovision the host from another running slave:

  • Identify the servers that are failed. All servers that are not the master for their region can be re-provisioned using a backup/restore of the master (see Section 5.10, “Creating a Backup” or using the tungsten_provision_slave script.

  • To re-provision an entire region, follow the steps below. The east region is used in the example statements below:

    1. To prevent application servers from reading and writing to the failed service, place the Tungsten Connector offline within the failed region:

      cctrl [east]> router * offline
    2. On all servers in other regions (west{1,2,3}):

      shell> /opt/replicator/tungsten/tungsten-replicator/bin/trepctl -service east offline
      shell> /opt/replicator/tungsten/tungsten-replicator/bin/trepctl -service east reset -all -y
    3. On all servers in the failed region (east{1,2,3}):

      shell> /opt/replicator/tungsten/tungsten-replicator/bin/replicator stop
      shell> /opt/replicator/tungsten/tools/tpm reset
      shell> /opt/continuent/tungsten/tungsten-replicator/scripts/tungsten_provision_slave \
          --direct --source=west1
    4. Check that Tungsten Clustering is working correctly and all hosts are up to date:

      cctrl [east]> ls
    5. Restart the Tungsten Replicator service:

      shell> /opt/replicator/tungsten/tungsten-replicator/bin/replicator start
    6. On all servers in other regions (west{1,2,3}):

      shell> /opt/replicator/tungsten/tungsten-replicator/bin/trepctl -service east online