Last Updated: 2015-06-01
Condition or Error
Latency is high: master:ONLINE, progress=41331580333, THL latency=78849.733
Causes
There are many possible causes for this error, however, if you see the following within the log on the Primary it may indicate a specific issue:
INFO | jvm 1 | 2016/02/09 15:01:54 | at com.continuent.tungsten.replicator.thl.CommitSeqnoTable.updateLastCommitSeqno(CommitSeqnoTable.java:548) INFO | jvm 1 | 2016/02/09 15:01:54 | at com.continuent.tungsten.replicator.thl.CatalogManager.updateCommitSeqnoTable(CatalogManager.java:223) INFO | jvm 1 | 2016/02/09 15:01:54 | at com.continuent.tungsten.replicator.thl.THL.updateCommitSeqno(THL.java:593) INFO | jvm 1 | 2016/02/09 15:01:54 | at com.continuent.tungsten.replicator.thl.THLStoreApplier.apply(THLStoreApplier.java:163) INFO | jvm 1 | 2016/02/09 15:01:54 | at com.continuent.tungsten.replicator.pipeline.SingleThreadStageTask.apply(SingleThreadStageTask.java:768) INFO | jvm 1 | 2016/02/09 15:01:54 | at com.continuent.tungsten.replicator.pipeline.SingleThreadStageTask.runTask(SingleThreadStageTask.java:501) INFO | jvm 1 | 2016/02/09 15:01:54 | at com.continuent.tungsten.replicator.pipeline.SingleThreadStageTask.run(SingleThreadStageTask.java:176) INFO | jvm 1 | 2016/02/09 15:01:54 | at java.lang.Thread.run(Thread.java:745) INFO | jvm 1 | 2016/02/09 15:01:54 |
The stack trace shows that the replicator is updating the trep_commit_seqno table which is normally a very fast operation.
The underlying reason may either be:
It is possible that updates to MySQL are somehow getting delayed, which would slow down the operation of the replicator as it updates each status update.
Check the block commit size, as low values will increase the number of updates to the table, and if the MySQL server updates are slow, this in turn slows down the operation of the replicator.
Rectifications
Focus on making sure the IO system and MySQL commits are not being blocked.
You can try increasing
replicator.stage.q-to-thl.blockCommitRowCount
so the replicator has less commits to MySQL on the Primary.
You can continue to track progress through trepctl
status -name tasks as you may see the
appliedLastSeqno value updating less
often if you increase this by a lot. Beware that increasing
this value too much increases possible data loss since it
creates less sync points with the Replicas.