In addition to this information, follow the guidelines in Section 2.5, “Best Practices”.
Running a Composite Active/Active service uses many different components to keep data updated on all servers. Monitoring the dataservice is divided into monitoring the two different clusters and each cluster sub-service cluster responsible for replication to/from remote clusters.
Configure your database servers with distinct
auto_increment_increment and
auto_increment_offset settings.
Each location that may accept writes should have a unique offset
value.
Using cctrl gives you the dataservice status. By default, cctrl will connect you to the custer associated with the node that you issue the command from. To start at the top level, issue cctrl -multi instead
At the top level, the composite cluster output shows the composite service, composite cluster members and replication services:
Tungsten Clustering 7.0.3 build 141
east: session established, encryption=false, authentication=false
[LOGICAL] / > ls
usa
east
east_from_west
westTo examine the overall composite cluster status, change to the composite cluster and use ls:
[LOGICAL] / >use usa[LOGICAL] /usa >lsCOORDINATOR[west3:AUTOMATIC:ONLINE] east:COORDINATOR[east3:AUTOMATIC:ONLINE] west:COORDINATOR[west3:AUTOMATIC:ONLINE] ROUTERS: +---------------------------------------------------------------------------------+ |connector@east1[10583](ONLINE, created=0, active=0) | |connector@east2[10548](ONLINE, created=0, active=0) | |connector@east3[10540](ONLINE, created=0, active=0) | |connector@west1[10589](ONLINE, created=0, active=0) | |connector@west2[10541](ONLINE, created=0, active=0) | |connector@west3[10547](ONLINE, created=0, active=0) | +---------------------------------------------------------------------------------+ DATASOURCES: +---------------------------------------------------------------------------------+ |east(composite master:ONLINE, global progress=1, max latency=3.489) | |STATUS [OK] [2019/12/24 10:21:08 AM UTC] | +---------------------------------------------------------------------------------+ | east(master:ONLINE, progress=1, max latency=1.483) | | east_from_west(relay:ONLINE, progress=1, max latency=3.489) | +---------------------------------------------------------------------------------+ +---------------------------------------------------------------------------------+ |west(composite master:ONLINE, global progress=1, max latency=0.909) | |STATUS [OK] [2019/12/24 10:21:08 AM UTC] | +---------------------------------------------------------------------------------+ | west(master:ONLINE, progress=1, max latency=0.909) | | west_from_east(relay:ONLINE, progress=1, max latency=0.903) | +---------------------------------------------------------------------------------+
For each cluster within the composite cluster, four lines of information are provided:
|east(composite master:ONLINE, global progress=1, max latency=3.489) |
This line indicates:
The name and type of the composite cluster, and whether the Primary in the cluster is online.
The global progress. This is a counter that combines the local
progress of the cluster, and the replication of data from this
cluster to the remote clusters in the composite to this cluster.
For example, if data is inserted into
west
The maximum latency within the cluster.
|STATUS [OK] [2019/12/24 10:21:08 AM UTC] |
The status and date within the Primary of the cluster.
| east(master:ONLINE, progress=1, max latency=1.483) |
The status and progress of the cluster.
| east_from_west(relay:ONLINE, progress=1, max latency=3.489) |
The status and progress of remote replication from the cluster.
The global progress and the
progress work together to provide an indication of
the overall replication status within the composite cluster:
Inserting data into the Primary on
east will:
Increment the progress within the
east cluster.
Increment the global progress within the
east cluster.
Inserting data into the Primary on
west will:
Increment the progress within the
west cluster.
Increment the global progress within the
west cluster.
Looking at the individual cluster shows only the cluster status, not the cross-cluster status:
[LOGICAL] /east > ls
COORDINATOR[east3:AUTOMATIC:ONLINE]
ROUTERS:
+---------------------------------------------------------------------------------+
|connector@east1[10583](ONLINE, created=0, active=0) |
|connector@east2[10548](ONLINE, created=0, active=0) |
|connector@east3[10540](ONLINE, created=0, active=0) |
|connector@west1[10589](ONLINE, created=0, active=0) |
|connector@west2[10541](ONLINE, created=0, active=0) |
|connector@west3[10547](ONLINE, created=0, active=0) |
+---------------------------------------------------------------------------------+
DATASOURCES:
+---------------------------------------------------------------------------------+
|east1(master:ONLINE, progress=1, THL latency=0.765) |
|STATUS [OK] [2019/12/24 10:21:12 AM UTC] |
+---------------------------------------------------------------------------------+
| MANAGER(state=ONLINE) |
| REPLICATOR(role=master, state=ONLINE) |
| DATASERVER(state=ONLINE) |
| CONNECTIONS(created=0, active=0) |
+---------------------------------------------------------------------------------+
+---------------------------------------------------------------------------------+
|east2(slave:ONLINE, progress=1, latency=0.826) |
|STATUS [OK] [2019/12/24 10:21:13 AM UTC] |
+---------------------------------------------------------------------------------+
| MANAGER(state=ONLINE) |
| REPLICATOR(role=slave, master=east1, state=ONLINE) |
| DATASERVER(state=ONLINE) |
| CONNECTIONS(created=0, active=0) |
+---------------------------------------------------------------------------------+
+---------------------------------------------------------------------------------+
|east3(slave:ONLINE, progress=1, latency=0.842) |
|STATUS [OK] [2019/12/24 10:21:12 AM UTC] |
+---------------------------------------------------------------------------------+
| MANAGER(state=ONLINE) |
| REPLICATOR(role=slave, master=east1, state=ONLINE) |
| DATASERVER(state=ONLINE) |
| CONNECTIONS(created=0, active=0) |
+---------------------------------------------------------------------------------+Within each cluster, cctrl can be used to monitor the current status. For more information on checking the status and controlling operations, see Section 6.3, “Checking Dataservice Status”.
To monitor all services and the current status, you can also use the multi_trepctl command (part of the Tungsten Replicator installation). This generates a unified status report for all the hosts and services configured:
shell> multi_trepctl --by-service
| host | servicename | role | state | appliedlastseqno | appliedlatency |
| east1 | east | master | ONLINE | 5 | 0.440 |
| east2 | east | slave | ONLINE | 5 | 0.538 |
| east3 | east | slave | ONLINE | 5 | 0.517 |
| east1 | east_from_west | relay | ONLINE | 23 | 0.074 |
| east2 | east_from_west | slave | ONLINE | 23 | 0.131 |
| east3 | east_from_west | slave | ONLINE | 23 | 0.111 |
| west1 | west | master | ONLINE | 23 | 0.021 |
| west2 | west | slave | ONLINE | 23 | 0.059 |
| west3 | west | slave | ONLINE | 23 | 0.089 |
| west1 | west_from_east | relay | ONLINE | 5 | 0.583 |
| west2 | west_from_east | slave | ONLINE | 5 | 0.562 |
| west3 | west_from_east | slave | ONLINE | 5 | 0.592 |
In the above example, it can be seen that the
west services have a higher
applied last sequence number than the
east services, this is because all
the writes have been applied within the
west cluster.
To monitor individual servers and/or services, use
trepctl, using the correct servicename.
For example, on east1 to check the
status of the replicator within the Tungsten Cluster service, use the
trepctl services command to get the status of both the
local and cross-cluster services:
shell> trepctl status
Processing services command...
NAME VALUE
---- -----
appliedLastSeqno: 6
appliedLatency : 0.43
role : master
serviceName : east
serviceType : local
started : true
state : ONLINE
NAME VALUE
---- -----
appliedLastSeqno: 4
appliedLatency : 1837.999
role : relay
serviceName : east_from_west
serviceType : local
started : true
state : ONLINE
Finished services command...To get a more detailed status, you must explicitly specify the service
shell> trepctl -service east_from_west status