The steps below will guide you through the process of restoring a MySQL Replica node by using rsync.
The following process has the following caveats:
You can sustain downtime on the node used as a source.
You can either ssh between hosts as root, or have root level access to temporarily change file ownership.
Steps
Establish ssh as root between hosts, or if you can't set up ssh as root, on the target host,
chown ownership of mysql target directories to tungsten,
then run rysnc as command in Step 3 as the tungsten user.
On failed host:
Shut down mysql if it is still running.
Determine all mysql data directories (datadir, binary log dir, etc)
e.g. /var/lib/mysql
Clear out database files from the failed host.
shell> rm -rf /var/lib/mysql/*
On Source host:
Within cctrl, shun the node that will used as the source.
cctrl> datasource sourcenode shunShut down mysql.
Use rsync to copy the datafiles to the target (specify 'z' if CPU available for compression)
sourcehost> rsync -avz --progress /source/dir/ targetHost:/target/dir/
Wait for the rsync to complete.
On Source host:
Restart mysql.
Bring the replicator online.
shell> trepctl onlineWait for replication to catch up, then use cctrl to recover the node.
cctrl> datasource sourcenode recover
On restored, target, host:
Fix the ownership on ALL data directories, e.g.
shell> chown -R mysql: /var/lib/mysql
Start mysql.
Bring the replicator online.
shell> trepctl onlineWait for replication to catch up, then use cctrl to recover the node
cctrl> datasource targetnode recover