6.11.6. Rebuilding a Lost Datasource

If a datasource has been lost within the dataservice, for example, a complete hardware failure or disk crash, the datasource can be added back to the cluster once the operating system and other configuration have been completed. Essentially, the process is the same as when initially setting up your node, with the node being re-confirmed as part of the running service, installing and configuring only the returning node to the cluster.

In the following steps, the host host3 is being recovered into the cluster:

  1. Setup the host with the prerequisites, as described in Appendix B, Prerequisites .

  2. Restore a snapshot of the data taken from another Replica into the dataserver. If you have existing backups of this Replica or another, they should be used. If not, take a snapshot of an existing Replica and use this to apply the data to the Replica. This will need to be performed outside of the Tungsten Cluster service using the native restore method for the backup method you have chosen. The backup must include the entire schema of your database, including the tungsten schemas for your services.

  3. The next steps depend on the availability of the hostname. If the hostname of the datasource that was lost can be reused, then the host can be reconfigured within the existing service. If the hostname is not available, the service must be reconfigured to remove the old host, and add the new host.

    Reusing an Existing Hostname

    1. Obtain a copy of the /etc/tungsten/tungsten.ini from an existing node and copy this to the same location on the new host

    2. Download the same version of the clustering software used on the other nodes and place this within /opt/continuent/software.

    3. Extract the tar file and change into the new directory that will be created, for example:

      shell> tar zxvf tungsten-clustering-6.1.25-6.tar.gz
      shell> cd tungsten-clustering-6.1.25-6
    4. Install the software:

      shell> ./tools/tpm install

    Removing and Adding a new Host

    1. Remove the existing (lost) datasource from the cluster using cctrl . First switch to administrative mode:

      [LOGICAL] /alpha > admin

      Remove the host from the dataservice:

      [ADMIN] /alpha > rm host3
      
      WARNING: This is an expert-level command:
      Incorrect use may cause data corruption
      or make the cluster unavailable.
      
      Do you want to continue? (y/n)>
    2. Update the /etc/tungsten/tungsten.ini on all hosts and change the hostname to match the new host being replaced.

    3. Update the installation across all the hosts:

      shell> cd /opt/continuent/software/tungsten-clustering-6.1.25-6
      shell> tools/tpm update --replace-release
    4. Now follow the same steps in the section above "Reusing an Existing Hostname" to install the software on the new host

  4. Use cctrl to check and confirm the operation of the restore datasource.

The restored host should be part of the cluster and accepting events from the Primary as configured.