OptimizeUpdates.js Filter

The optimizeupdates filter works with row-based events to simplify the update statement and remove columns/values that have not changed. This reduces the workload and row data exchanged between replicators.

Pre-configured filter name	optimizeupdates
Classname	com.continuent.tungsten.replicator.filter.OptimizeUpdatesFilter
Property prefix	replicator.filter.optimizeupdates
Stage compatibility
tpm Option compatibility
Data compatibility	Row events

The filter operates by removing column values for keys in the update statement that do not change. For example, when replicating the row event from the statement:

mysql>  update testopt set msg = 'String1', string = 'String3' where id = 1;

Generates the following THL event data:

- SQL(0) =
- ACTION = UPDATE
- SCHEMA = test
- TABLE = testopt
- ROW# = 0
- COL(1: id) = 1
- COL(2: msg) = String1
- COL(3: string) = String3
- KEY(1: id) = 1

Column 1 (id) in this case is automatically implied by the KEY entry required for the update.

With the optimizeupdates filter enabled, the data in the THL is simplified to:

- SQL(0) =
- ACTION = UPDATE
- SCHEMA = test
- TABLE = testopt
- ROW# = 0
- COL(2: msg) = String1
- COL(3: string) = String4
- KEY(1: id) = 1

In tables where there are multiple keys the stored THL information can be reduced further.

warning

The filter works by comparing the value of each KEY and COL entry in the THL and determining whether the value has changed or not. If the number of keys and columns do not match then the filter will fail with the following error message:

Caused by: java.lang.Exception: Column and key count is different in this event! Cannot filter

This may be due to a filter earlier within the filter configuration that has optimized or simplified the data. For example, the pkey filter removes KEY entries from the THL that are not primary keys, or dropcolumn which drops column data.

The following error message may appear in the logs and in the output from trepctl reset to indicate that this ordering issue may be the problem:

OptimizeUpdatesFilter cannot filter, because column and key count is different.
Make sure that it is defined before filters which remove keys (eg. PrimaryKeyFilter).