8.12. Switching Master Hosts

In the event of a failure, or during the process of performing maintenance on a running cluster, the roles of the master and slaves within the cluster may need to be swapped.

The basic sequence of operation for switching master and slaves is:

  1. Switch slaves to offline state

  2. Switch master to offline status

  3. Set an existing slave to have the master role

  4. Set each slave with the slave role, updating the master URI (where the THL logs will be loaded) to the new master host

  5. Switch the new master to online state

  6. Switch the new slaves to online state

Depending on the situation when the switch is performed, the switch can be performed either without waiting for the hosts to be synchronized (i.e. in a failure situation), or by explicitly waiting for slave that will be promoted to the master role.

To perform an ordered switch of the master. In the example below, master host host1 will be switched to host3, and the remaining hosts (host1 and host2) will be configured as slaves to the new master:

  1. If you are performing the switch as part of maintenance or other procedures, you should perform a safe switch, ensuring the slaves are up to date with the master:

    1. Synchronize the database and the transaction history log. This will ensure that the two are synchronized, and provide you with a sequence number to ensure the slaves are up to date:

      shell> trepctl -host host1 flush
      Master log is synchronized with database at log sequence number: 1405

      Keep a note of the sequence number.

    2. For each current slave within the cluster, wait until the master sequence number has been reached, and then put the slave into the offline state:

      shell> trepctl -host host2 wait -applied 1405
      shell> trepctl -host host2 offline
      shell> trepctl -host host3 wait -applied 1405
      shell> trepctl -host host3 offline

    If the master has failed, or once the slaves and masters are in sync, you can perform the remainder of the steps to execute the physical switch.

  2. Switch the master to the offline state:

    shell> trepctl -host host1 offline
  3. Configure the new designated master to the master role:

    shell> trepctl -host host3 setrole -role master

    Switch the master to the online state:

    shell> trepctl -host host3 online
  4. For each slave, set the role to slave, supplying the URI of the THL service on the master:

    shell> trepctl -host host1 setrole -role slave -uri thl://host3:2112

    In the above example we are using the default THL port (2112).

    Put the new slave into the online state:

    shell> trepctl -host host1 online

    Repeat for the remaining slaves:

    shell> trepctl -host host2 setrole -role slave -uri thl://host3:2112
    shell> trepctl -host host2 online

Once completed, the state of each host can be checked to confirm that the switchover has completed successfully:

appliedLastEventId     : mysql-bin.000005:0000000000002100;0
appliedLastSeqno       : 1405
appliedLatency         : 0.094
dataServerHost         : host1
masterConnectUri       : thl://host3:2112
role                   : slave
state                  : ONLINE
-----
appliedLastEventId     : mysql-bin.000005:0000000000002100;0
appliedLastSeqno       : 1405
appliedLatency         : 0.149
dataServerHost         : host2
masterConnectUri       : thl://host3:2112
role                   : slave
state                  : ONLINE
-----
appliedLastEventId     : mysql-bin.000005:0000000000002100;0
appliedLastSeqno       : 1405
appliedLatency         : 0.061
dataServerHost         : host3
masterConnectUri       : thl://host1:2112/
role                   : master
state                  : ONLINE

In the above, host1 and host2 are now getting the THL information from host1, with each acting as a slave to the host1 as master.