Checking Replication Status

To check the replication status you can use the trepctl command. This accepts a number of command-specific verbs that provide status and control information for your configured cluster. The basic format of the command is:

shell> trepctl [-host hostname] command

The -host option is not required, and enables you to check the status of a different host than the current node.

To get the basic information about the currently configured services on a node and current status, use the trepctl services command:

shell> trepctl services
Processing services command...
NAME              VALUE
----              -----
appliedLastSeqno: 211
appliedLatency  : 17.66
role            : slave
serviceName     : firstrep
serviceType     : local
started         : true
state           : ONLINE
Finished services command...

In the above example, the output shows the last sequence number and latency of the host, in this case an Applier, compared to the Extractor from which it is processing information. In this example, the last sequence number and the latency between that sequence being processed on the Extractor and applied to the Target is 17.66 seconds. You can compare this information to that provided by the Extractor, either by logging into the Extractor and running the same command, or by using the host command-line option:

shell> trepctl -host host1 services
Processing services command...
NAME              VALUE
----              -----
appliedLastSeqno: 365
appliedLatency  : 0.614
role            : master
serviceName     : firstrep
serviceType     : local
started         : true
state           : ONLINE
Finished services command...

By comparing the appliedLastSeqno for the Extractor against the value on the Applier, it is possible to determine that the Applier and the Extractor are not yet synchronized.

For a more detailed output of the current status, use the trepctl reset command, which provides much more detailed output of the current replication status:

shell> trepctl status
Processing status command...
NAME                     VALUE
----                     -----
appliedLastEventId     : mysql-bin.000064:0000000002757461;0
appliedLastSeqno       : 212
appliedLatency         : 263.43
channels               : 1
clusterName            : default
currentEventId         : NONE
currentTimeMillis      : 1365082088916
dataServerHost         : host2
extensions             : 
latestEpochNumber      : 0
masterConnectUri       : thl://host1:2112/
masterListenUri        : thl://host2:2112/
maximumStoredSeqNo     : 724
minimumStoredSeqNo     : 0
offlineRequests        : NONE
pendingError           : NONE
pendingErrorCode       : NONE
pendingErrorEventId    : NONE
pendingErrorSeqno      : -1
pendingExceptionMessage: NONE
pipelineSource         : thl://host1:2112/
relativeLatency        : 655.915
resourcePrecedence     : 99
rmiPort                : 10000
role                   : slave
seqnoType              : java.lang.Long
serviceName            : firstrep
serviceType            : local
simpleServiceName      : firstrep
siteName               : default
sourceId               : host2
state                  : ONLINE
timeInStateSeconds     : 893.32
uptimeSeconds          : 9370.031
version                : Tungsten Replicator 8.0.4 Build 132
Finished status command...

Similar to the host specification, trepctl provides information for the default service. If you have installed multiple services, you must specify the service explicitly:

shell> trepctl -service servicename status

If the service has been configured to operate on an alternative management port, this can be specified using the -port option. The default is to use port 10000.

The above command was executed on the Target host, host2. Some key parameter values from the generated output:

• appliedLastEventId

  This shows the last event from the source event stream that was applied to the database. In this case, the output shows that source of the data was a MySQL binary log. The portion before the colon, mysql-bin.000064 is the filename of the binary log on the Source. The portion after the colon is the physical location, in bytes, within the binary log file.

• appliedLastSeqno

  The last sequence number for the transaction from the Tungsten stage that has been applied to the database. This indicates the last actual transaction information written into the Target database.

  When using parallel replication, this parameter returns the minimum applied sequence number among all the channels applying data.

• appliedLatency

  The appliedLatency is the latency between the commit time and the time the last committed transaction reached the end of the corresponding pipeline within the replicator.

  In replicators that are operating with parallel apply, appliedLatency indicates the latency of the trailing channel. Because the parallel apply mechanism does not update all channels simultaneously, the figure shown may trail significantly from the actual latency.

• masterConnectUri

  On an Extractor, the value will be empty.

  On an Applier, the URI of the Extractor from which the transaction data is being read from. The value supports multiple URIs (separated by comma) for topologies with multiple Sources.

• maximumStoredSeqNo

  The maximum transaction ID that has been stored locally on the machine in the THL. Because the Replicator operates in stages, it is sometimes important to compare the sequence and latency between information being read from the source into the THL, and then from the THL into the database. You can compare this value to the appliedLastSeqno, which indicates the last sequence committed to the database. The information is provided at a resolution of milliseconds.

• pipelineSource

  Indicates the source of the information that is written into the THL. For an Extractor, pipelineSource is the MySQL binary log. For an Applier, pipelineSource is the THL of the Extractor.

• relativeLatency

  The relativeLatency is the latency between now and timestamp of the last event written into the local THL. An increasing relativeLatency indicates that the replicator may have stalled and stopped applying changes to the dataserver.

• state

  Shows the current status for this node. In the event of a failure, the status will indicate that the node is in a state other than ONLINE. The timeInStateSeconds will indicate how long the node has been in that state, and therefore how long the node may have been down or unavailable.

The easiest method to check the health of your replication is to compare the current sequence numbers and latencies for each Applier compared to the Extractor. For example:

shell> trepctl -host host2 status|grep applied
appliedLastEventId     : mysql-bin.000076:0000000087725114;0
appliedLastSeqno       : 2445
appliedLatency         : 252.0
...
shell> trepctl -host host1 status|grep applied
appliedLastEventId     : mysql-bin.000076:0000000087725114;0
appliedLastSeqno       : 2445
appliedLatency         : 2.515

note

For parallel replication and complex multi-service replication structures, there are additional parameters and information to consider when checking and confirming the health of replication.

The above indicates that the two hosts are up to date, but that there is a significant latency on the Applier for performing updates.

Tungsten Replicator Schema

Tungsten Replicator creates and updates information in a special schema created within the database which contains more specific information about the replication information transferred. The schema is named according to the servicename of the replication configuration, for example if the server is firstrep, the schema will be tungsten_firstrep.

The sequence number of the last transferred and applied transaction is recorded in the trep_commit_seqno table.

Understanding Replicator States

Each node within the cluster will have a specific state that indicates whether the node is up and running and servicing requests, or whether there is a fault or problem. Understanding these states will enable you to clearly identify the current operational status of your nodes and topology as a whole.

A list of the possible states for the replicator includes:

• START

  The replicator service is starting up and reading the replicator properties configuration file.

• OFFLINE:NORMAL

  The node has been deliberately placed into the offline mode by an administrator. No replication events are processed, and reading or writing to the underlying database does not take place.

• OFFLINE:ERROR

  The node has entered the offline state because of an error. No replication events are processed, and reading or writing to the underlying database does not take place.

• SUSPECT

  This replicator state is only seen within cctrl. The underlying replicator status should be checked to see the full error state by reviewing the output of trepctl reset.

• GOING-ONLINE:PROVISIONING

  The replicator is currently reading provisioning information from the Source database before entering the ONLINE state.

• GOING-ONLINE:RESTORING

  The replicator is preparing to go online and is currently restoring data from a backup.

• GOING-ONLINE:SYNCHRONIZING

  The replicator is preparing to go online and is currently preparing to process any outstanding events from the incoming event stream. This mode occurs when an Applier has been switched online after maintenance, or in the event of a temporary network error where the Applier has reconnected to the Extractor.

• ONLINE

  The node is currently online and processing events, reading incoming data and applying those changes to the database as required. In this mode the current status and position within the replication stream is recorded and can be monitored. Replication will continue until an error or administrative condition switches the node into the OFFLINE state.

• GOING-OFFLINE

  The replicator is processing any outstanding events or transactions that were in progress when the node was switched offline. When these transactions are complete, and the resources in use (memory, network connections) have been closed down, the replicator will switch to the OFFLINE:NORMAL state. This state may also be seen in a node where auto-enable is disabled after a start or restart operation.

• ONLINE:DEGRADED

  This status will be seen on an Extractor replicator and is indicative of the replicator losing connectivity to the Source Database that it is extracting from. The replicator will still continue to extract entries from the binary log that have not yet been processed. After extracting all log entries, the replicator will proceed to the ONLINE:DEGRADED-BINLOG-FULLY-READ state.

• ONLINE:DEGRADED-BINLOG-FULLY-READ

  This status will be seen on an Extractor replicator following the ONLINE:DEGRADED state and indicates that the replicator has completed reading all binlog entries. In a clustering environment, it indicates to the cluster that failover can now proceed.

In general, the state of a node during operation will go through a natural progression within certain situations. In normal operation, assuming no failures or problems, and no management requested offline, a node will remain in the ONLINE state indefinitely.

Maintenance on Tungsten Replicator or the dataserver must be performed while in the OFFLINE state. In the OFFLINE state, write locks on the THL and other files are released, and reads or writes from the dataserver are stopped until the replicator is ONLINE again.

Replicator States During Operations

During a maintenance operation, a node will typically go through the following states at different points of the operation:

Operation	State
Node operating normally	ONLINE
Administrator puts node into offline state	GOING-OFFLINE
Node is offline	OFFLINE:NORMAL
Administrator puts node into online state	GOING-ONLINE:SYNCHRONIZING
Node catches up with Extractor	ONLINE

In the event of a failure, the sequence will trigger the node into the error state and then recovery into the online state:

Operation	State
Node operating normally	ONLINE
Failure causes the node to go offline	OFFLINE:ERROR
Administrator fixes error and puts node into online state	GOING-ONLINE:SYNCHRONIZING
Node catches up with Extractor	ONLINE

During an error state where a backup of the data is restored to a node in preparation of bringing the node back into operation:

Operation	State
Node operating normally	ONLINE
Failure causes the node to go offline	OFFLINE:ERROR
Administrator restores node from backup data	GOING-ONLINE:RESTORING
Once restore is complete, node synchronizes with the Extractor	GOING-ONLINE:SYNCHRONIZING
Node catches up with Extractor	ONLINE

Changing Replicator States

You can manually change the replicator states on any node by using the trepctl command.

To switch to the OFFLINE state if you are currently ONLINE:

shell> trepctl offline

Unless there is an error, no information is reported. The current state can be verified using the trepctl reset:

shell> trepctl status
Processing status command...
...
state                  : OFFLINE:NORMAL
timeInStateSeconds     : 21.409
uptimeSeconds          : 935.072

To switch back to the ONLINE state:

shell> trepctl online

When using replicator states in this manner, the replication between hosts is effectively paused. Any outstanding events from the Extractor will be replicated to the Applier with the replication continuing from the point where the node was switched to the OFFLINE state. The sequence number and latency will be reported accordingly, as seen in the example below where the node is significantly behind the Primary:

shell> trepctl status
Processing status command...
NAME                     VALUE
----                     -----
appliedLastEventId     : mysql-bin.000004:0000000005162941;0
appliedLastSeqno       : 21
appliedLatency         : 179.366