6.11. Switching Primary Hosts

In the event of a failure, or during the process of performing maintenance on a running cluster, the roles of the Extractor and Appliers within the cluster may need to be swapped.

The basic sequence of operation for switching Primary and Replicas is:

  1. Switch Replicas to offline state

  2. Switch Primary to offline status

  3. Set an existing Replica to have the master role

  4. Set each Replica with the slave role, updating the Extractor URI (where the THL logs will be loaded) to the new Extractor host

  5. Switch the new $ctpri; to online state

  6. Switch the new $ctreps; to online state

Depending on the situation when the switch is performed, the switch can be performed either without waiting for the hosts to be synchronized (i.e. in a failure situation), or by explicitly waiting for the Replica that will be promoted to the Primary role.

To perform an ordered switch of the Primary. In the example below, Primary host host1 will be switched to host3, and the remaining hosts (host1 and host2) will be configured as Replicas to the new Primary:

  1. If you are performing the switch as part of maintenance or other procedures, you should perform a safe switch, ensuring the Replicas are up to date with the Primary:

    1. Synchronize the database and the transaction history log. This will ensure that the two are synchronized, and provide you with a sequence number to ensure the Replicas are up to date:

      shell> trepctl -host host1 flush
      Master log is synchronized with database at log sequence number: 1405

      Keep a note of the sequence number.

    2. For each current Replica within the cluster, wait until the Primary sequence number has been reached, and then put the Replica into the offline state:

      shell> trepctl -host host2 wait -applied 1405
      shell> trepctl -host host2 offline
      shell> trepctl -host host3 wait -applied 1405
      shell> trepctl -host host3 offline

    If the Primary has failed, or once the Replicas and Primaries are in sync, you can perform the remainder of the steps to execute the physical switch.

  2. Switch the Primary to the offline state:

    shell> trepctl -host host1 offline
  3. Configure the new designated Primary to the Primary role:

    shell> trepctl -host host3 setrole -role master

    Switch the Primary to the online state:

    shell> trepctl -host host3 online
  4. For each Replica, set the role to Replica, supplying the URI of the THL service on the Primary:

    shell> trepctl -host host1 setrole -role slave -uri thl://host3:2112

    In the above example we are using the default THL port (2112).

    Put the new Replica into the online state:

    shell> trepctl -host host1 online

    Repeat for the remaining Replicas:

    shell> trepctl -host host2 setrole -role slave -uri thl://host3:2112
    shell> trepctl -host host2 online

Once completed, the state of each host can be checked to confirm that the switchover has completed successfully:

appliedLastEventId     : mysql-bin.000005:0000000000002100;0
appliedLastSeqno       : 1405
appliedLatency         : 0.094
dataServerHost         : host1
masterConnectUri       : thl://host3:2112
role                   : slave
state                  : ONLINE
-----
appliedLastEventId     : mysql-bin.000005:0000000000002100;0
appliedLastSeqno       : 1405
appliedLatency         : 0.149
dataServerHost         : host2
masterConnectUri       : thl://host3:2112
role                   : slave
state                  : ONLINE
-----
appliedLastEventId     : mysql-bin.000005:0000000000002100;0
appliedLastSeqno       : 1405
appliedLatency         : 0.061
dataServerHost         : host3
masterConnectUri       : thl://host1:2112/
role                   : master
state                  : ONLINE

In the above, host1 and host2 are now getting the THL information from host1, with each acting as a Replica to the host1 as Primary.