Tungsten Clustering

Manually Recovering from Another Replica

In the event that a restore operation fails, or due to a significant failure in the dataserver, an alternative option is to seed the failed dataserver directly from an existing running Replica.

For example, on the host host2, the data directory for MySQL has been corrupted, and mysqld will no longer start. This status can be seen from examining the MySQL error log in /var/log/mysql/error.log:

130520 14:37:08 [Note] Recovering after a crash using /var/log/mysql/mysql-bin
130520 14:37:08 [Note] Starting crash recovery...
130520 14:37:08 [Note] Crash recovery finished.
130520 14:37:08 [Note] Server hostname (bind-address): '0.0.0.0'; port: 13306
130520 14:37:08 [Note]   - '0.0.0.0' resolves to '0.0.0.0';
130520 14:37:08 [Note] Server socket created on IP: '0.0.0.0'.
130520 14:37:08 [ERROR] Fatal error: Can't open and lock privilege tables: Table 'mysql.host' doesn't exist
130520 14:37:08 [ERROR] /usr/sbin/mysqld: File '/var/run/mysqld/mysqld.pid' not found (Errcode: 13)
130520 14:37:08 [ERROR] /usr/sbin/mysqld: Error reading file 'UNKNOWN' (Errcode: 9)
130520 14:37:08 [ERROR] /usr/sbin/mysqld: Error on close of 'UNKNOWN' (Errcode: 9)

Performing a restore operation on this Replica may not work. To recover from another running Replica, host3, the MySQL data files can be copied over to host2 directly using the following steps:

Shun the host2 datasource to be restored, and put the replicator service offline using cctrl :
```
[LOGICAL] /alpha > datasource host2 shun
[LOGICAL] /alpha > replicator host2 offline
```
Shun the host3 datasource to be restored, and put the replicator service offline using cctrl :
```
[LOGICAL] /alpha > datasource host3 shun
[LOGICAL] /alpha > replicator host3 offline
```
Stop the mysqld service on host2:
```
shell> sudo /etc/init.d/mysql stop
```
Stop the mysqld service on host3:
```
shell> sudo /etc/init.d/mysql stop
```
Delete the mysqld data directory on host2 :
```
shell> sudo rm -rf /var/lib/mysql/*
```
If necessary, ensure the tungsten user can write to the MySQL directory:
```
shell> sudo chmod 777 /var/lib/mysql
```
Use rsync on host3 to send the data files for MySQL to host2 :
```
shell> rsync -aze ssh /var/lib/mysql/* host2:/var/lib/mysql/
```
You should synchronize all locations that contain data. This includes additional folders such as innodb_data_home_dir or innodb_log_group_home_dir. Check the my.cnf file to ensure you have the correct paths.
Once the files have been copied, the files should be updated to have the correct ownership and permissions so that the Tungsten service can read them.

Recover host3 back to the dataservice:

[LOGICAL:EXPERT] /alpha > datasource host3 recover

Update the ownership and permissions on the data files on host2:

host2 shell> sudo chown -R mysql:mysql /var/lib/mysql
host2 shell> sudo chmod 770 /var/lib/mysql

Clear out the THL files on the target node host2 so the Replica replicator service may start cleanly:
```
host2 shell> thl purge
```
Recover host2 back to the dataservice:
```
[LOGICAL:EXPERT] /alpha > datasource host2 recover
```
The datasource recover command will start MySQL and ensure that the server is accessible before restarting replication. If the MySQL instance does not start, correct any issues and attempt the datasource recover command again.