3.4.3. Best Practices: Composite Multimaster Clusters

Note

In addition to this information, follow the guidelines in Section 2.5, “Best Practices”.

  • Running a Composite Multimaster service uses many different components to keep data updated on all servers. Monitoring the dataservice is divided into monitoring the two different clusters and each cluster sub-service cluster responsible for replication to/from remote clusters.

  • Configure your database servers with distinct auto_increment_increment and auto_increment_offset settings. Each location that may accept writes should have a unique offset value.

Using cctrl gives you the dataservice status. By default, cctrl will connect you to the custer associated with the node that you issue the command from. To start at the top level, issue cctrl -multi instead

At the top level, the composite cluster output shows the composite service, composite cluster members and replication services:

Tungsten Clustering 6.0.5 build 40
east: session established, encryption=false, authentication=false
[LOGICAL] / > ls
usa
  east
    east_from_west
  west

To examine the overall composite cluster status, change to the composite cluster and use ls:

[LOGICAL] / > use usa
[LOGICAL] /usa > ls

COORDINATOR[west3:AUTOMATIC:ONLINE]
   east:COORDINATOR[east3:AUTOMATIC:ONLINE]
   west:COORDINATOR[west3:AUTOMATIC:ONLINE]

ROUTERS:
+---------------------------------------------------------------------------------+
|connector@east1[10583](ONLINE, created=0, active=0)                              |
|connector@east2[10548](ONLINE, created=0, active=0)                              |
|connector@east3[10540](ONLINE, created=0, active=0)                              |
|connector@west1[10589](ONLINE, created=0, active=0)                              |
|connector@west2[10541](ONLINE, created=0, active=0)                              |
|connector@west3[10547](ONLINE, created=0, active=0)                              |
+---------------------------------------------------------------------------------+

DATASOURCES:
+---------------------------------------------------------------------------------+
|east(composite master:ONLINE, global progress=1, max latency=3.489)              |
|STATUS [OK] [2019/12/24 10:21:08 AM UTC]                                         |
+---------------------------------------------------------------------------------+
|  east(master:ONLINE, progress=1, max latency=1.483)                             |
|  east_from_west(relay:ONLINE, progress=1, max latency=3.489)                    |
+---------------------------------------------------------------------------------+
+---------------------------------------------------------------------------------+
|west(composite master:ONLINE, global progress=1, max latency=0.909)              |
|STATUS [OK] [2019/12/24 10:21:08 AM UTC]                                         |
+---------------------------------------------------------------------------------+
|  west(master:ONLINE, progress=1, max latency=0.909)                             |
|  west_from_east(relay:ONLINE, progress=1, max latency=0.903)                    |
+---------------------------------------------------------------------------------+

For each cluster within the composite cluster, four lines of information are provided:

  • |east(composite master:ONLINE, global progress=1, max latency=3.489)              |

    This line indicates:

    • The name and type of the composite cluster, and whether the master in the cluster is online.

    • The global progress. This is a counter that combines the local progress of the cluster, and the replication of data from this cluster to the remote clusters in the composite to this cluster. For example, if data is inserted into west

    • The maximum latency within the cluster.

  • |STATUS [OK] [2019/12/24 10:21:08 AM UTC]                                         |

    The status and date within the master of the cluster.

  • |  east(master:ONLINE, progress=1, max latency=1.483)                             |

    The status and progress of the cluster.

  • |  east_from_west(relay:ONLINE, progress=1, max latency=3.489)                    |

    The status and progress of remote replication from the cluster.

The global progress and the progress work together to provide an indication of the overall replication status within the composite cluster:

  • Inserting data into the master on east will:

    • Increment the progress within the east cluster.

    • Increment the global progress within the east cluster.

  • Inserting data into the master on west will:

    • Increment the progress within the west cluster.

    • Increment the global progress within the west cluster.

Looking at the individual cluster shows only the cluster status, not the cross-cluster status:

[LOGICAL] /east > ls
COORDINATOR[east3:AUTOMATIC:ONLINE]

ROUTERS:
+---------------------------------------------------------------------------------+
|connector@east1[10583](ONLINE, created=0, active=0)                              |
|connector@east2[10548](ONLINE, created=0, active=0)                              |
|connector@east3[10540](ONLINE, created=0, active=0)                              |
|connector@west1[10589](ONLINE, created=0, active=0)                              |
|connector@west2[10541](ONLINE, created=0, active=0)                              |
|connector@west3[10547](ONLINE, created=0, active=0)                              |
+---------------------------------------------------------------------------------+

DATASOURCES:
+---------------------------------------------------------------------------------+
|east1(master:ONLINE, progress=1, THL latency=0.765)                              |
|STATUS [OK] [2019/12/24 10:21:12 AM UTC]                                         |
+---------------------------------------------------------------------------------+
|  MANAGER(state=ONLINE)                                                          |
|  REPLICATOR(role=master, state=ONLINE)                                          |
|  DATASERVER(state=ONLINE)                                                       |
|  CONNECTIONS(created=0, active=0)                                               |
+---------------------------------------------------------------------------------+
+---------------------------------------------------------------------------------+
|east2(slave:ONLINE, progress=1, latency=0.826)                                   |
|STATUS [OK] [2019/12/24 10:21:13 AM UTC]                                         |
+---------------------------------------------------------------------------------+
|  MANAGER(state=ONLINE)                                                          |
|  REPLICATOR(role=slave, master=east1, state=ONLINE)                             |
|  DATASERVER(state=ONLINE)                                                       |
|  CONNECTIONS(created=0, active=0)                                               |
+---------------------------------------------------------------------------------+
+---------------------------------------------------------------------------------+
|east3(slave:ONLINE, progress=1, latency=0.842)                                   |
|STATUS [OK] [2019/12/24 10:21:12 AM UTC]                                         |
+---------------------------------------------------------------------------------+
|  MANAGER(state=ONLINE)                                                          |
|  REPLICATOR(role=slave, master=east1, state=ONLINE)                             |
|  DATASERVER(state=ONLINE)                                                       |
|  CONNECTIONS(created=0, active=0)                                               |
+---------------------------------------------------------------------------------+

Within each cluster, cctrl can be used to monitor the current status. For more information on checking the status and controlling operations, see Section 5.3, “Checking Dataservice Status”.

To monitor all services and the current status, you can also use the multi_trepctl command (part of the Tungsten Replicator installation). This generates a unified status report for all the hosts and services configured:

shell> multi_trepctl --by-service
| host  | servicename    | role   | state  | appliedlastseqno | appliedlatency |
| east1 | east           | master | ONLINE |                5 |          0.440 |
| east2 | east           | slave  | ONLINE |                5 |          0.538 |
| east3 | east           | slave  | ONLINE |                5 |          0.517 |
| east1 | east_from_west | relay  | ONLINE |               23 |          0.074 |
| east2 | east_from_west | slave  | ONLINE |               23 |          0.131 |
| east3 | east_from_west | slave  | ONLINE |               23 |          0.111 |
| west1 | west           | master | ONLINE |               23 |          0.021 |
| west2 | west           | slave  | ONLINE |               23 |          0.059 |
| west3 | west           | slave  | ONLINE |               23 |          0.089 |
| west1 | west_from_east | relay  | ONLINE |                5 |          0.583 |
| west2 | west_from_east | slave  | ONLINE |                5 |          0.562 |
| west3 | west_from_east | slave  | ONLINE |                5 |          0.592 |

In the above example, it can be seen that the west services have a higher applied last sequence number than the east services, this is because all the writes have been applied within the west cluster.

To monitor individual servers and/or services, use trepctl, using the correct servicename. For example, on east1 to check the status of the replicator within the Tungsten Clustering service, use the trepctl services command to get the status of both the local and cross-cluster services:

shell> trepctl status
Processing services command...
NAME              VALUE
----              -----
appliedLastSeqno: 6
appliedLatency  : 0.43
role            : master
serviceName     : east
serviceType     : local
started         : true
state           : ONLINE
NAME              VALUE
----              -----
appliedLastSeqno: 4
appliedLatency  : 1837.999
role            : relay
serviceName     : east_from_west
serviceType     : local
started         : true
state           : ONLINE
Finished services command...

To get a more detailed status, you must explicitly specify the service

shell> trepctl -service east_from_west status