Last Updated: 2016-04-20
Condition or Error
You have missing events, or events that have been correctly extraction from a Tungsten Cluster host.
Causes
There are multiple potential causes for this issue, including, but not limited t
Rectifications
Before proceeding, ensure you have access to the failed Primary server and the new Primary server, if one was promoted. Ensure that your environment is properly initialized. For more information, see Initializing shell environment for VMware Continuent (2105429).
The example operations below are for a service called
east.
To identify transactions that have not been extracted from a MySQL Primary running VMware Continuent replication:
Identify the last transaction replicated to the Replica servers.
If a new Primary has already been promoted or created:
Run the following command:
shell> trepctl -service east status
Note the value of
latestEpochNumber. Assume
5385 is the value of
latestEpochNumber.
Confirm this is the point where replication switched from the failed Primary to the new Primary.
Now determine the THL and associated data from that THL sequence number using:
shell> thl -service east list -headers -low 5385 -high 5385Now compare the last value on each line to ensure they are different. This value indicates the Primary host for the sequence number.
If there are no differences, you need to refer to the trepsvc.log file of the new Primary to find the point where it became the Primary. Search for [binlog-to-q-0] Setting extractor position from the end of the log. This shows where the Primary started the extraction process from a new sequence number. Run the above check for each sequence number until you find the point where extraction switched from one Primary to another.
If no new Primary has been promoted but the Primary did fail:
Run the following command to determine the current sequence number on the Replicas:
shell> dsctl -service east get
If these values are different between multiple Replicas,
use the lowest reported sequence to reset the
position. If the dsctl utility is
not available, see the
trep_commit_seqno
table for information on the last replicated
transaction for the Replica. The location of this table
is different depending on your Replica DBMS.
Assume 5384 is the value of seqno returned by the command.
Check the output of the following command on the Primary and Replicas:
shell> thl -service east list -headers -seqno 5384The output should be used to confirm that each host reports the same information. If there are differences, it is indicative of larger problems and you should reprovision from a known good datasource.
Run the following command to read the events from the Primary:
shell> tungsten_read_master_events --service=east --after=5384This will display the mysqlbinlog output for any transactions that were applied after the listed sequence number.
Use this information to determine the next course of action for your application. You may can choose to ignore the transactions if they are not important or add them to the remaining servers manually and then repair replication.
More Information