Last Updated: 2016-04-20
Condition or Error
You have missing events, or events that have been correctly extraction from a Tungsten Cluster host.
There are multiple potential causes for this issue, including, but not limited t
Before proceeding, ensure you have access to the failed master server and the new master server, if one was promoted. Ensure that your environment is properly initialized. For more information, see Initializing shell environment for VMware Continuent (2105429).
The example operations below are for a service called
To identify transactions that have not been extracted from a MySQL master running VMware Continuent replication:
Identify the last transaction replicated to the slave servers.
If a new master has already been promoted or created:
Run the following command:
trepctl -service east status
Confirm this is the point where replication switched from the failed master to the new master.
Now determine the THL and associated data from that THL sequence number using:
thl -service east list -headers -low 5385 -high 5385
Now compare the last value on each line to ensure they are different. This value indicates the master host for the sequence number.
If there are no differences, you need to refer to the trepsvc.log file of the new master to find the point where it became the master. Search for [binlog-to-q-0] Setting extractor position from the end of the log. This shows where the master started the extraction process from a new sequence number. Run the above check for each sequence number until you find the point where extraction switched from one master to another.
If no new master has been promoted but the master did fail:
Run the following command to determine the current sequence number on the slaves:
dsctl -service east get
If these values are different between multiple slaves,
use the lowest reported sequence to reset the
position. If the dsctl utility is
not available, see the
table for information on the last replicated
transaction for the slave. The location of this table
is different depending on your slave DBMS.
Assume 5384 is the value of seqno returned by the command.
Check the output of the following command on the master and slaves:
thl -service east list -headers -seqno 5384
The output should be used to confirm that each host reports the same information. If there are differences, it is indicative of larger problems and you should reprovision from a known good datasource.
Run the following command to read the events from the master:
tungsten_read_master_events --service=east --after=5384
This will display the mysqlbinlog output for any transactions that were applied after the listed sequence number.
Use this information to determine the next course of action for your application. You may can choose to ignore the transactions if they are not important or add them to the remaining servers manually and then repair replication.