3.2.4. Best Practices: Composite Multimaster Clusters

Note

In addition to this information, follow the guidelines in Section 2.5, “Best Practices”.

  • Running a Multisite/Multimaster service uses many different components to keep data updated on all servers. Monitoring the dataservice is divided into monitoring the two different clusters. Be mindful when using commands that you have the correct path. You should either use the full path to the command under /opt/continuent and /opt/replicator, or use the aliases created by setting the --executable-prefix=mm option. Calling trepctl would become mm_trepctl.

  • Configure your database servers with distinct auto_increment_increment and auto_increment_offset settings. Each location that may accept writes should have a unique offset value.

Using cctrl gives you the dataservice status. Because we are using a composite service, use the cctrl -multi to ensure that cctrl is in multi-cluster mode.

At the top level, the composite cluster output shows the composite service, composite cluster members and replication services:

Tungsten Clustering 6.0.0 build 415
east: session established, encryption=false, authentication=false
[LOGICAL] / > ls
usa
  east
    east_from_west
  west

To examine the overall composite cluster status, change to the composite cluster and use ls:

[LOGICAL] / > use usa
[LOGICAL] /usa > ls

COORDINATOR[msmme1:MAINTENANCE:ONLINE]

ROUTERS:
+---------------------------------------------------------------------------------+
|connector@msmme2[22083](ONLINE, created=0, active=0)                             |
|connector@msmme3[23235](ONLINE, created=0, active=0)                             |
|connector@msmmw1[22095](ONLINE, created=0, active=0)                             |
|connector@msmmw2[11406](ONLINE, created=0, active=0)                             |
|connector@msmmw3[9926](ONLINE, created=0, active=0)                              |
+---------------------------------------------------------------------------------+

DATASOURCES:
DATASOURCES:
+---------------------------------------------------------------------------------+
|east(composite master:ONLINE, global progress=10, max latency=1972.724)          |
|STATUS [OK] [2018/02/04 05:09:49 AM GMT]                                         |
+---------------------------------------------------------------------------------+
|  east(master:ONLINE, progress=6, max latency=0.497)                             |
|  east_from_west(relay:ONLINE, progress=4, max latency=1972.724)                 |
+---------------------------------------------------------------------------------+
+---------------------------------------------------------------------------------+
|west(composite master:ONLINE, global progress=10, max latency=275.619)           |
|STATUS [OK] [2018/02/05 04:59:34 AM GMT]                                         |
+---------------------------------------------------------------------------------+
|  west(master:ONLINE, progress=4, max latency=0.539)                             |
|  west_from_east(relay:ONLINE, progress=6, max latency=275.619)                  |
+---------------------------------------------------------------------------------+

For each cluster within the composite cluster, four lines of information are provided:

  • |east(composite master:ONLINE, global progress=4, max latency=-1.000)             |

    This line indicates:

    • The name and type of the composite cluster, and whether the master in the cluster is online.

    • The global progress. This is a counter that combines the local progress of the cluster, and the replication of data from this cluster to the remote clusters in the composite to this cluster. For example, if data is inserted into west

    • The maximum latency within the cluster.

  • |STATUS [OK] [2018/02/04 05:09:49 AM GMT]                                         |

    The status and date within the master of the cluster.

  • |  east(master:ONLINE, progress=4, max latency=-1.000)                            |

    The status and progress of the cluster.

  • |  east_from_west(relay:ONLINE, progress=-1, max latency=-1.000)                  |

    The status and progress of remote replication from the cluster.

The global progress and the progress work together to provide an indication of the overall replication status within the composite cluster:

  • Inserting data into the master on east will:

    • Increment the progress within the east cluster.

    • Increment the global progress within the east cluster.

  • Inserting data into the master on west will:

    • Increment the progress within the west cluster.

    • Increment the global progress within the west cluster.

Looking at the individual cluster shows only the cluster status, not the cross-cluster status:

[LOGICAL] /east > ls
COORDINATOR[msmme1:MAINTENANCE:ONLINE]

ROUTERS:
+---------------------------------------------------------------------------------+
|connector@msmme2[22083](ONLINE, created=1, active=0)                             |
|connector@msmme3[23235](ONLINE, created=0, active=0)                             |
|connector@msmmw1[22095](ONLINE, created=1, active=0)                             |
|connector@msmmw2[11406](ONLINE, created=1, active=0)                             |
|connector@msmmw3[9926](ONLINE, created=1, active=0)                              |
+---------------------------------------------------------------------------------+

DATASOURCES:
+---------------------------------------------------------------------------------+
|msmme1(master:ONLINE, progress=5, THL latency=0.133)                             |
|STATUS [OK] [2018/02/04 05:09:59 AM GMT]                                         |
+---------------------------------------------------------------------------------+
|  MANAGER(state=ONLINE)                                                          |
|  REPLICATOR(role=master, state=ONLINE)                                          |
|  DATASERVER(state=ONLINE)                                                       |
|  CONNECTIONS(created=1, active=0)                                               |
+---------------------------------------------------------------------------------+
+---------------------------------------------------------------------------------+
|msmme2(slave:ONLINE, progress=5, latency=0.195)                                  |
|STATUS [OK] [2018/02/04 05:21:50 AM GMT]                                         |
+---------------------------------------------------------------------------------+
|  MANAGER(state=ONLINE)                                                          |
|  REPLICATOR(role=slave, master=msmme1, state=ONLINE)                            |
|  DATASERVER(state=ONLINE)                                                       |
|  CONNECTIONS(created=1, active=0)                                               |
+---------------------------------------------------------------------------------+
+---------------------------------------------------------------------------------+
|msmme3(slave:ONLINE, progress=5, latency=0.211)                                  |
|STATUS [OK] [2018/02/04 05:20:18 AM GMT]                                         |
+---------------------------------------------------------------------------------+
|  MANAGER(state=ONLINE)                                                          |
|  REPLICATOR(role=slave, master=msmme1, state=ONLINE)                            |
|  DATASERVER(state=ONLINE)                                                       |
|  CONNECTIONS(created=2, active=0)                                               |
+---------------------------------------------------------------------------------+

Within each cluster, cctrl can be used to monitor the current status. For more information on checking the status and controlling operations, see Section 5.3, “Checking Dataservice Status”.

Note

For convenience, the shell PATH can be updated with the tools and configuration. With two separate services, both environments must be updated. To update the shell with the Tungsten Clustering service and tools:

shell> source /opt/continuent/share/env.sh

To update the shell with the Tungsten Replicator service and tools:

shell> source /opt/replicator/share/env.sh

To monitor all services and the current status, you can also use the multi_trepctl command (part of the Tungsten Replicator installation). This generates a unified status report for all the hosts and services configured:

shell> multi_trepctl --by-service
| host   | servicename    | role   | state  | appliedlastseqno | appliedlatency |
| msmme1 | east           | master | ONLINE |                6 |          0.430 |
| msmme1 | east_from_west | relay  | ONLINE |                4 |       1837.999 |
| msmme2 | east           | slave  | ONLINE |                6 |          0.477 |
| msmme2 | east_from_west | slave  | ONLINE |                4 |       1972.724 |
| msmme3 | east           | slave  | ONLINE |                6 |          0.497 |
| msmme3 | east_from_west | slave  | ONLINE |                4 |       1892.862 |
| msmmw1 | west           | master | ONLINE |                4 |          0.466 |
| msmmw1 | west_from_east | relay  | ONLINE |                6 |        258.646 |
| msmmw2 | west           | slave  | ONLINE |                4 |          0.539 |
| msmmw2 | west_from_east | slave  | ONLINE |                6 |        269.893 |
| msmmw3 | west           | slave  | ONLINE |                4 |          0.503 |
| msmmw3 | west_from_east | slave  | ONLINE |                6 |        275.619 |

In the above example, it can be seen that the west services have a much higher applied last sequence number than the east services, this is because all the writes have been applied within the west cluster.

To monitor individual servers and/or services, use trepctl, using the correct port number and servicename. For example, on east1 to check the status of the replicator within the Tungsten Clustering service, use the trepctl services command to get the status of both the local and cross-cluster services:

shell> trepctl status
Processing services command...
NAME              VALUE
----              -----
appliedLastSeqno: 6
appliedLatency  : 0.43
role            : master
serviceName     : east
serviceType     : local
started         : true
state           : ONLINE
NAME              VALUE
----              -----
appliedLastSeqno: 4
appliedLatency  : 1837.999
role            : relay
serviceName     : east_from_west
serviceType     : local
started         : true
state           : ONLINE
Finished services command...

To get a more detailed status, you must explicitly specify the service

shell> trepctl -service east_from_west status