6.9.3. Management and Monitoring of Elasticsearch Deployments

Once the extractor and applier have been installed, services can be monitored using the trepctl command.

For example, to monitor the extractor status:

shell> trepctl status
appliedLastEventId     : mysql-bin.000009:0000000000002298;2340
appliedLastSeqno       : 10
appliedLatency         : 0.788
autoRecoveryEnabled    : false
autoRecoveryTotal      : 0
channels               : 1
clusterName            : alpha
currentEventId         : mysql-bin.000009:0000000000002298
currentTimeMillis      : 1498687871560
dataServerHost         : mysqlhost
extensions             :
host                   : mysqlhost
latestEpochNumber      : 0
masterConnectUri       : thl://localhost:/
masterListenUri        : thl://mysqlhost:2112/
maximumStoredSeqNo     : 10
minimumStoredSeqNo     : 0
offlineRequests        : NONE
pendingError           : NONE
pendingErrorCode       : NONE
pendingErrorEventId    : NONE
pendingErrorSeqno      : -1
pendingExceptionMessage: NONE
pipelineSource         : /var/lib/mysql
relativeLatency        : 99185.56
resourcePrecedence     : 99
rmiPort                : 10000
role                   : master
seqnoType              : java.lang.Long
serviceName            : east
serviceType            : local
simpleServiceName      : east
siteName               : default
sourceId               : mysqlhost
state                  : ONLINE
timeInStateSeconds     : 101347.786
timezone               : GMT
transitioningTo        :
uptimeSeconds          : 101358.88
useSSLConnection       : false
version                : Tungsten Replicator 5.2.2 build 275
Finished status command...

The replicator service operates just the same as a standard master service of a typical MySQL replication service.

The Elasticsearch applier service can be accessed either remotely from the master:

shell> trepctl -host elastichost status
...

Or locally on the Elasticsearch host:

shell> trepctl status
Processing status command...
NAME                     VALUE
----                     -----
appliedLastEventId     : mysql-bin.000008:0000000000412301;0
appliedLastSeqno       : 1296
appliedLatency         : 10.253
channels               : 1
clusterName            : alpha
currentEventId         : NONE
currentTimeMillis      : 1377098139212
dataServerHost         : elastichost
extensions             :
latestEpochNumber      : 1286
masterConnectUri       : thl://host1:2112/
masterListenUri        : null
maximumStoredSeqNo     : 1296
minimumStoredSeqNo     : 0
offlineRequests        : NONE
pendingError           : NONE
pendingErrorCode       : NONE
pendingErrorEventId    : NONE
pendingErrorSeqno      : -1
pendingExceptionMessage: NONE
pipelineSource         : thl://mysqlhost:2112/
relativeLatency        : 771.212
resourcePrecedence     : 99
rmiPort                : 10000
role                   : slave
seqnoType              : java.lang.Long
serviceName            : alpha
serviceType            : local
simpleServiceName      : alpha
siteName               : default
sourceId               : elastichost
state                  : ONLINE
timeInStateSeconds     : 177783.343
transitioningTo        :
uptimeSeconds          : 180631.276
useSSLConnection       : false
version                : Tungsten Replicator 5.2.2 build 275
Finished status command...

Monitoring the status of replication between the master and slave is also the same. The appliedLastSeqno still indicates the sequence number that has been applied to Elasticsearch, and the event ID from Elasticsearch can still be identified from appliedLastEventId.

Sequence numbers between the two hosts should match, as in a master/slave deployment, but due to the method used to replicate, the applied latency may be higher.

To check for information within Elasticsearch, use the curl command-line client:

shell> curl http://$HOSTNAME:9200/_cat/indices?v
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
yellow open tungsten_alphna dpERkG_LT_-YAzhUtZGXDg 5 1 2 0 13kb 13kb

To test replication, execute typical commands in MySQL and use the curl command-line client to read them back:

mysql> create database test;
mysql> use test;
mysql> CREATE TABLE messages (
> `id` int(10) NOT NULL AUTO_INCREMENT,
> `msg` varchar(80) COLLATE utf8_unicode_ci DEFAULT NULL,
> PRIMARY KEY (`id`)
> ) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci;
mysql> insert into messages values (99999,"Hello Elasticsearch");

shell> curl http://$HOSTNAME:9200/_cat/indices?v
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
yellow open test          RaxrMCgDSM-1uwpMEeZPVA 5 1 1 0 6.2kb 6.2kb
yellow open tungsten_east 2pvNVxEzRHqUXTZ0Vz1Jvg 5 1 1 1 6.7kb 6.7kb

shell> curl http://$HOSTNAME:9200/test/messages/99999?pretty
{
	"_index" : "test",
	"_type" : "messages",
	"_id" : "99999",
	"_version" : 1,
	"found" : true,
	"_source" : {
		"msg" : "Hello Elasticsearch",
		"committime" : "2017-06-06 19:09:20.0",
		"id" : "99999",
		"source_table" : "messages",
		"source_schema" : "test"
	}
}
mysql> update messages set msg="Welcome Students!" where id=99999;

shell> curl http://$HOSTNAME:9200/test/messages/99999?pretty
{
	"_index" : "test",
	"_type" : "messages",
	"_id" : "99999",
	"_version" : 2,
	"found" : true,
	"_source" : {
		"msg" : "Welcome Students!",
		"committime" : "2017-06-06 19:47:27.0",
		"id" : "99999",
		"source_table" : "messages",
		"source_schema" : "test"
	}
}

mysql> delete from messages where id=99999;

shell> curl http://$HOSTNAME:9200/test/messages/99999?pretty
{
	"_index" : "test",
	"_type" : "messages",
	"_id" : "99999",
	"found" : false
}

The output should be checked to ensure that information is being correctly replicated. If strings are shown as a hex value, for example:

"title" : "[B@7084a5c"

It probably indicates that UTF8 and/or --mysql-use-bytes-for-string=false options were not used during installation. If you are reading from a cluster this is expected behavior, and you should enable the convertstringfrommysql filter as shown in the installation examples. In pure replicator scenarios, ensure that the --mysql-use-bytes-for-string=false setting is enabled, or that you are using --enable-heterogeneous-service.