Tungsten Clustering

Reprovision a MySQL Replica using rsync

The steps below will guide you through the process of restoring a MySQL Replica node by using rsync.

The following process has the following caveats:

You can sustain downtime on the node used as a source.
You can either ssh between hosts as root, or have root level access to temporarily change file ownership.

Steps

Establish ssh as root between hosts, or if you can't set up ssh as root, on the target host, chown ownership of mysql target directories to tungsten, then run rsync as command in Step 3 as the tungsten user.
On failed host:
- Shut down mysql if it is still running.
- Determine all mysql data directories (datadir, binary log dir, etc) e.g. /var/lib/mysql
- Clear out database files from the failed host.
```
shell> rm -rf /var/lib/mysql/*
```
On Source host:
- Within cctrl, shun the node that will used as the source.
```
cctrl> datasource sourcenode shun
```
- Shut down mysql.
- Use rsync to copy the datafiles to the target (specify 'z' if CPU available for compression)
```
sourcehost> rsync -avz --progress /source/dir/ targetHost:/target/dir/
```
Wait for the rsync to complete.
On Source host:
- Restart mysql.
- Bring the replicator online.
```
shell> trepctl online
```
- Wait for replication to catch up, then use cctrl to recover the node.
```
cctrl> datasource sourcenode recover
```
On restored, target, host:
- Fix the ownership on ALL data directories, e.g.
```
shell> chown -R mysql: /var/lib/mysql
```
- Start mysql.
- Bring the replicator online.
```
shell> trepctl online
```
- Wait for replication to catch up, then use cctrl to recover the node
```
cctrl> datasource targetnode recover
```