5.8. Multimaster Recovery

In at Multi-Master topology, a switch or a failover not only promotes a slave to be a new master, but also will require the ability to reconfigure cross-site communications. This process therefore assumes that cross-site communication is online and working. In some situations, it may be possible that cross-site communication is down, or for some reason cross-site replication is in an OFFLINE:ERROR state - for example a DDL or DML statement that worked in the local cluster may have failed to apply in the remote.

If a switch or failover occurs and the process is unable to reconfigure the cross-site replicators, the local switch will still succeed, however the associated cross-site services will be placed into a "SHUNNED(SUBSERVICE-SWITCH-FAILED)" state.

The guide explains how to recover from this situation.

The examples are based on a 2-cluster topology, named NYC and LONDON and the composite dataservice named GLOBAL. The cluster is configured with the following dataservers:

  • NYC : db1 (master), db2 (slave), db3 (slave)

  • LONDON: db4 (master), db5 (slave), db6 (slave)

The cross site replicators in both clusters are in an OFFLINE:ERROR state due to failing DDL.

A switch was then issued, promoting db3 as the new master in NYC and db5 as the new master in LONDON

shell> cctrl -multi -expert
Tungsten Clustering 6.0.3 build 599
nyc: session established
[LOGICAL:EXPERT] / > use london_from_nyc
london_from_nyc: session established, encryption=false, authentication=false
[LOGICAL:EXPERT] /london_from_nyc > ls

COORDINATOR[db6:AUTOMATIC:ONLINE]
 
ROUTERS:
+---------------------------------------------------------------------------------+
|connector@db1[26248](ONLINE, created=0, active=0)                                |
|connector@db2[14906](ONLINE, created=0, active=0)                                |
|connector@db3[15035](ONLINE, created=0, active=0)                                |
|connector@db4[27813](ONLINE, created=0, active=0)                                |
|connector@db5[4379](ONLINE, created=0, active=0)                                 |
|connector@db6[2098](ONLINE, created=0, active=0)                                 |
+---------------------------------------------------------------------------------+
 
DATASOURCES:
+---------------------------------------------------------------------------------+
|db5(relay:SHUNNED(SUBSERVICE-SWITCH-FAILED), progress=6, latency=0.219)          |
|STATUS [SHUNNED] [2018/03/15 10:27:24 AM UTC]                                    |
+---------------------------------------------------------------------------------+
|  MANAGER(state=ONLINE)                                                          |
|  REPLICATOR(role=relay, master=db3, state=ONLINE)                               |
|  DATASERVER(state=ONLINE)                                                       |
|  CONNECTIONS(created=0, active=0)                                               |
+---------------------------------------------------------------------------------+
+---------------------------------------------------------------------------------+
|db4(slave:SHUNNED(SUBSERVICE-SWITCH-FAILED), progress=6, latency=0.252)          |
|STATUS [SHUNNED] [2018/03/15 10:27:25 AM UTC]                                    |
+---------------------------------------------------------------------------------+
|  MANAGER(state=ONLINE)                                                          |
|  REPLICATOR(role=slave, master=db5, state=ONLINE)                               |
|  DATASERVER(state=ONLINE)                                                       |
|  CONNECTIONS(created=0, active=0)                                               |
+---------------------------------------------------------------------------------+
+---------------------------------------------------------------------------------+
|db6(slave:SHUNNED(SUBSERVICE-SWITCH-FAILED), progress=6, latency=0.279)          |
|STATUS [SHUNNED] [2018/03/15 10:27:25 AM UTC]                                    |
+---------------------------------------------------------------------------------+
|  MANAGER(state=ONLINE)                                                          |
|  REPLICATOR(role=slave, master=db4, state=ONLINE)                               |
|  DATASERVER(state=ONLINE)                                                       |
|  CONNECTIONS(created=0, active=0)                                               |
+---------------------------------------------------------------------------------+

In the above example, you can see that all services are in the SHUNNED(SUBSERVICE-SWITCH-FAILED) state, and partial reconfiguration has happened.

The Replicators for db4 and db6 should be slaves of db5, db5 has correctly configured to the new master in nyc, db3. The actual state of the cluster in each scenario maybe different depending upon the cause of the loss of cross-site communication. Using the steps below, apply the necessary actions that relate to your own cluster state, if in any doubt always contact Continuent Support for assistance.

  1. The first step is to ensure the initial replication errors have been resolved and that the replicators are in an online state, the steps to resolve the replicators will depend on the reason for the error.

  2. From one node, connect into cctrl at the expert level

    shell> cctrl -multi -expert
  3. Next, connect to the cross-site subservice, in this example, london_from_nyc

    [LOGICAL:EXPERT] / > use london_from_nyc
    london_from_nyc: session established, encryption=false, authentication=false
  4. Next, place the service into Maintenance Mode

    [LOGICAL:EXPERT] /london_from_nyc > set policy maintenance
  5. Enable override of commands issued

    [LOGICAL:EXPERT] /london_from_nyc > set force true
  6. Bring the relay datasource online

    [LOGICAL:EXPERT] /london_from_nyc > datasource db5 online
  7. If you need to change the source for the relay replicator to the correct, new, master in the remote cluster, take the replicator offline. If the relay source is correct, then move on to step 10

    [LOGICAL:EXPERT] /london_from_nyc > replicator db5 offline
  8. Change the source of the relay replicator

    [LOGICAL:EXPERT] /london_from_nyc > replicator db5 relay nyc/db3
  9. Bring the relay replicator online

    [LOGICAL:EXPERT] /london_from_nyc > replicator db5 online
  10. For each datasource that requires the replicator altering, issue the following commands

    [LOGICAL:EXPERT] /london_from_nyc > replicator {datasource} offline 
    [LOGICAL:EXPERT] /london_from_nyc > replicator {datasource} slave {relay}
    [LOGICAL:EXPERT] /london_from_nyc > replicator {datasource} online

    For example:

    [LOGICAL:EXPERT] /london_from_nyc > replicator db4 offline
    [LOGICAL:EXPERT] /london_from_nyc > replicator db4 slave db5
    [LOGICAL:EXPERT] /london_from_nyc > replicator db4 online
  11. Once all replicators are using the correct source, we can then bring the cluster back

    [LOGICAL:EXPERT] /london_from_nyc > cluster welcome
  12. Some of the datasources may still be in the SHUNNED state, so for each of those, you can then issue the following

    [LOGICAL:EXPERT] /london_from_nyc > datasource {datasource} online

    For example:

    [LOGICAL:EXPERT] /london_from_nyc > datasource db4 online
  13. Once all nodes are online, we can then return the cluster to automatic

    [LOGICAL:EXPERT] /london_from_nyc > set policy automatic
  14. Repeat this process for the other cross-site subservice if required