7.2. Tungsten Manager Failover Behavior

The Tungsten Clustering default position is to protect the integrity of the data. Sometimes this means that a failover may be delayed. It is more important to have all the data than for the write master to be available. This behavior is configurable.

When the Manager quorum invokes a failover automatically, there are various states that each replication stream may be in that must be taken into account by the Manager layer as it coordinates the failover.

This is due to the loosely-coupled, asynchronous nature of Tungsten Replicator.

7.2.1. Failover Replication State Scenarios

The multiple scenarios regarding the state of the data are described below, along with how to locate any associated events:

  • First, the normal situation - all events in the binary logs have been extracted into the master THL, all master THL has been transferred to all slave nodes, and all THL on the slave nodes has been fully applied to all slave databases.

  • Scenario 1 - There are orphaned MySQL binary log events on the old master that were not extracted into THL before a failover occurred

    Figure 7.1. Failover Scenario 1

    Failover Scenario 1

    The tungsten_find_orphaned script can help locate orphaned binary log events that did not make it to the THL on the old master before a failover. For more information, please see Scenario 1 in Section 8.24, “The tungsten_find_orphaned Command”.

  • Scenario 2 - There is orphaned THL left on the old master that did not make it to the new master

    Figure 7.2. Failover Scenario 2

    Failover Scenario 2

    The tungsten_find_orphaned script can help locate orphaned master THL left on the old master that did not make it to the new master before a failover. For more information, please see Scenario 2 in Section 8.24, “The tungsten_find_orphaned Command”.

  • Scenario 3 - There is orphaned slave THL on the new master node that did not get applied into the database before getting promoted to master role

    Figure 7.3. Failover Scenario 3

    Failover Scenario 3

    The tungsten_find_orphaned script can help locate orphaned slave THL on the new master node that did not get applied into the database before getting promoted to master role during a failover. For more information, please see Scenario 3 in Section 8.24, “The tungsten_find_orphaned Command”.

7.2.2. Recovery Behavior After Failover

The Tungsten Manager is able to detect un-extracted, desirable binary log events when recovering the old master via cctrl after a failover.

The cctrl recover command will fail if:

  • any unextracted binlog events exist on the old master that we are trying to recover (Scenario 1)

  • the old master THL contains more events than the slave that will be promoted to a new master (Scenario 2)

In this case, the cctrl recover command will display text similar to the following:

Recovery failed because the failed master has unextracted events in
the binlog. Please run the tungsten_find_orphaned script to inspect
this events. Provided you have a recent backup available, you can
try to restore the data source by issuing the following command:
               datasource {hostname} restore
Please consult the user manual at:
https://docs.continuent.com/tungsten-clustering-6.1/operations-restore.html

The tungsten_find_orphaned script is designed to locate orphaned events. For more information, please see Section 8.24, “The tungsten_find_orphaned Command”.

7.2.3. Failover Response when MySQL Server Fails

The Manager and Replicator behave in concert when MySQL dies on the master node. When this happens, the replicator is unable to update the trep_commit_seqno table any longer, and therefore must either abort extraction or continue extracting without recording the extracted position into the database.

By default, the Manager will delay failover until all remaining events have been extracted from the binary logs on the failing master node as a way to protect data integrity.

This behavior is configurable via the following setting, shown with the default of false:

--property=replicator.store.thl.stopOnDBError=false

Failover will only continue once:

  • all available events are completely read from the binary logs on the master node

  • all events have reached the slaves

When --property=replicator.store.thl.stopOnDBError=true, then the Replicator will stop extracting once it is unable to update the trep_commit_seqno table in MySQL, and the Manager will perform the failover without waiting, at the risk of possible data loss due to leaving binlog events behind. All such situations are logged.

For use cases where failover speed is more important than data accuracy, those NOT willing to wait for long failover can set replicator.store.thl.stopOnDBError=true and still use tungsten_find_orphaned to manually analyze and perform the data recovery. For more information, please see Section 8.24, “The tungsten_find_orphaned Command”.

7.2.4. Failover Response when Slave Applier is Latent

During a failover, the manager will wait until the slave that is the candidate for promotion to master has applied all stored THL events before promoting that node to master.

This wait time can be configured via the manager.failover.thl.apply.wait.timeout=0 property.

The default value is 0, which means "wait indefinitely until all stored THL events are applied".

Any value other than zero invites data loss due to the fact that once the slave is promoted to master, any unapplied stored events in the THL will be ignored, and therefore lost.

Whenever a failover occurs, the slave with most events stored in the local THL is selected so that when the events are eventually applied, the data is as close to the original master as possible with the least number of events missed.

That is usually, but not always, the most up-to-date slave, which is the one with the most events applied.

There should be a good balance between the value for manager.failover.thl.apply.wait.timeout and the value for policy.slave.promotion.latency.threshold=900, which is the "maximum slave latency" - this means the number of seconds to which a slave must be current with the master in order to qualify as a candidate for failover. The default is 15 minutes (900 seconds).