10.4. Filter Reference

The different filter types configured and available within the replicate are designed to provide a number of different functionality and operations. Since the information exchanged through the THL system contains a copy of the statement or the row data that is being updated, the filters allow schemas, table and column names, as well as actual data to be converted at the stage in which they are applied.

Filters are identified according to the underlying Java class that defines their operation. For different filters, further configuration and naming is applied according to the templates used when Tungsten Clustering is installed through tpm.

Tungsten Replicator also comes with a number of JavaScript filters that can either be used directly, or that can be modified and adapted to suit individual requirements. These filter scripts are located in tungsten-replicator/support/filters-javascript.

For the purposes of classification, the different filters have been categorised according to their main purpose:

  • Auditing

    These filters provide methods for tracking database updates alongside the original table data. For example, in a financial database, the actual data has to be updated in the corresponding tables, but the individual changes that lead to that update must also be logged individually.

  • Content

    Content filters modify or update the content of the transaction events. These may alter information, for the purposes of interoperability (such as updating enumerated or integer values to their string equivalents), or remove or filter columns, tables, and entire schemas.

  • Logging

    Logging filters record information about the transactions into the standard replicator log, either for auditing or debugging purposes.

  • Optimization

    The optimization filters are designed to simplify and optimize statements and row updates to improve the speed at which those updates can be applied to the destination dataserver.

  • Transformation

    Transformation filters rename or reformat schemas and tables according to a set of rules. For example, multiple schemas can be merged to a single schema, or tables and column names can be updated

  • Validation

    Provide validation or consistency checking of either the data or the replication process.

  • Miscellaneous

    Other filters that cannot be allocated to one of the existing filter classes.

The list of filters and their basic description are provided in the table below.

Filter Type Description
BidiRemoteSlaveFilter Content Suppresses events that originated on the local service (required for correct slave operation)
BuildAuditTable Auditing Builds an audit table of changes for specified schemas and tables
BuildIndexTable Transformation Merges multiple schemas into a single schema
CaseMappingFilter Transformation Transforms schema, table and column names to upper or lower case
CDCMetadataFilter Auditing Records change data capture for transactions to a separate change table (auditing)
ColumnNameFilter Validation Adds column name information to row-based replication events
ConsistencyCheckFilter Validation Adds consistency checking to events
DatabaseTransformFilter Transformation Transforms database or table names using regular expressions
DummyFilter Miscellaneous Allows for confirmation of filter configuration
EnumToStringFilter Content Updates enumerated values to their string-based equivalent
EventMetadataFilter Content Filters events based on metadata; used by default within sharding and multi-master topologies
HeartbeatFilter Validation Detects heartbeat events on masters or slaves
JavaScriptFilter Miscellaneous Enables filtering through custom JavaScripts
LoggingFilter Logging Logs filtered events through the standard replicator logging mechanism
MySQLSessionSupportFilter Content Filters transactions for session specific temporary tables and variables
OptimizeUpdatesFilter Optimization Optimizes update statements where the current and updated value are the same
PrimaryKeyFilter Optimization Used during row-based replication to optimize updates using primary keys
PrintEventFilter Logging Outputs transaction event information to the replication logging system
RenameFilter Transformation Advanced schema, table and column-based renaming
ReplicateColumnsFilter Content Removes selected columns from row-based transaction data
ReplicateFilter Content Selects or ignores specification schemas and/or databases
SetToStringFilter Content Converts integer values in SET datatypes to string values
ShardFilter Content Used to enforce database schema sharding between specific masters
TimeDelayFilter Miscellaneous Delays transactions until a specific point in time has passed

In the following reference sections:

  • Pre-configured filter name is the filter name that can be used against a stage without additional configuration.

  • Property prefix is the prefix string for the filter to be used when assigning property values.

  • Classname is the Java class name of the filter.

  • Parameter is the name of the filter parameter can be set as a property within the configuration.

  • Data compatibility indicates whether the filter is compatible with row-based events, statement-based events, or both.