Last Updated: 2020-10-13
Condition or Error
Replicator (extractor) is unable to stay online and extract an event. Error logs consistently show a stack trace similar to the following:
2020/07/24 15:06:14.637 | Event extraction failed 2020/07/24 15:06:14.637 | com.continuent.tungsten.replicator.extractor.ExtractorException: Unexpected failure » while extracting event myhost-db-04.qa.mydomain.local (1334) 2020/07/24 15:06:14.637 | at com.continuent.tungsten.replicator.extractor.mysql.MySQLExtractor.extractEvent(Unknown Source) 2020/07/24 15:06:14.637 | at com.continuent.tungsten.replicator.extractor.mysql.MySQLExtractor.extract(Unknown Source) 2020/07/24 15:06:14.637 | at com.continuent.tungsten.replicator.extractor.ExtractorWrapper.extract(Unknown Source) 2020/07/24 15:06:14.637 | at com.continuent.tungsten.replicator.extractor.ExtractorWrapper.extract(Unknown Source) 2020/07/24 15:06:14.637 | at com.continuent.tungsten.replicator.pipeline.SingleThreadStageTask.runTask(Unknown Source) 2020/07/24 15:06:14.637 | at com.continuent.tungsten.replicator.pipeline.SingleThreadStageTask.run(Unknown Source) 2020/07/24 15:06:14.637 | at java.lang.Thread.run(Thread.java:748) 2020/07/24 15:06:14.637 | Caused by: java.lang.IndexOutOfBoundsException 2020/07/24 15:06:14.637 | at java.io.DataInputStream.readFully(DataInputStream.java:192) 2020/07/24 15:06:14.637 | at com.continuent.tungsten.common.io.BufferedFileDataInput.readFully(Unknown Source) 2020/07/24 15:06:14.637 | at com.continuent.tungsten.replicator.extractor.mysql.BinlogReader.read(Unknown Source) 2020/07/24 15:06:14.637 | at com.continuent.tungsten.replicator.extractor.mysql.LogEvent.readDataFromBinlog(Unknown Source) 2020/07/24 15:06:14.637 | at com.continuent.tungsten.replicator.extractor.mysql.LogEvent.readLogEvent(Unknown Source) 2020/07/24 15:06:14.637 | at com.continuent.tungsten.replicator.extractor.mysql.MySQLExtractor.processFile(Unknown Source) 2020/07/24 15:06:14.637 | ... 7 more
Causes
You could be hitting a MySQL bug where the binlog is over-writing itself due to periods in the log-bin my.cnf entry. See https://bugs.mysql.com/bug.php?id=75507 for more details.
Example of my.cnf entry that may trigger this bug:
log-bin = /data/mysql/myhost-db-04.qa.mydomain.local.com-bin
Rectifications
Replace the dots with hyphens and restart MySQL
Example of a fixed my.cnf entry:
log-bin = /data/mysql/myhost-db-04-qa-mydomain-local-com-bin
Adjusting the binlog pattern within MySQL may also require a configuration change to the replicator if the pattern is changed after installation.
To do this, add the repl-datasource-log-pattern to your configuration and issue tpm update