Database Replication and Clustering
ansiquotes.js Filterbreadcrumbs.js Filterdbrename.js Filterdbselector.js Filterdbupper.js Filterdropcolumn.js Filterdropcomments.js Filterdropmetadata.js Filterdropstatementdata.js Filterforeignkeychecks.js Filterinsertsonly.js Filternocreatedbifnotexists.js Filtershardbyseqno.js Filtershardbytable.js Filtertosingledb.js Filtertruncatetext.js Filterzerodate2null.js FilterThe different filter types configured and available within the replicate are designed to provide a number of different functionality and operations. Since the information exchanged through the THL system contains a copy of the statement or the row data that is being updated, the filters allow schemas, table and column names, as well as actual data to be converted at the stage in which they are applied.
Filters are identified according to the underlying Java class that defines their operation. For different filters, further configuration and naming is applied according to the templates used when Tungsten Clustering is installed through tpm.
Tungsten Replicator also comes with a number of JavaScript filters that can
either be used directly, or that can be modified and adapted to suit
individual requirements. These filter scripts are located in
tungsten-replicator/support/filters-javascript.
For the purposes of classification, the different filters have been categorised according to their main purpose:
Auditing
These filters provide methods for tracking database updates alongside the original table data. For example, in a financial database, the actual data has to be updated in the corresponding tables, but the individual changes that lead to that update must also be logged individually.
Content
Content filters modify or update the content of the transaction events. These may alter information, for the purposes of interoperability (such as updating enumerated or integer values to their string equivalents), or remove or filter columns, tables, and entire schemas.
Logging
Logging filters record information about the transactions into the standard replicator log, either for auditing or debugging purposes.
Optimization
The optimization filters are designed to simplify and optimize statements and row updates to improve the speed at which those updates can be applied to the destination dataserver.
Transformation
Transformation filters rename or reformat schemas and tables according to a set of rules. For example, multiple schemas can be merged to a single schema, or tables and column names can be updated
Validation
Provide validation or consistency checking of either the data or the replication process.
Miscellaneous
Other filters that cannot be allocated to one of the existing filter classes.
The list of filters and their basic description are provided in the table below.
| Filter | Type | Description |
|---|---|---|
| BidiRemoteSlaveFilter | Content | Suppresses events that originated on the local service (required for correct slave operation) |
| BuildAuditTable | Auditing | Builds an audit table of changes for specified schemas and tables |
| BuildIndexTable | Transformation | Merges multiple schemas into a single schema |
| CaseMappingFilter | Transformation | Transforms schema, table and column names to upper or lower case |
| CDCMetadataFilter | Auditing | Records change data capture for transactions to a separate change table (auditing) |
| ColumnNameFilter | Validation | Adds column name information to row-based replication events |
| ConsistencyCheckFilter | Validation | Adds consistency checking to events |
| DatabaseTransformFilter | Transformation | Transforms database or table names using regular expressions |
| DummyFilter | Miscellaneous | Allows for confirmation of filter configuration |
| EnumToStringFilter | Content | Updates enumerated values to their string-based equivalent |
| EventMetadataFilter | Content | Filters events based on metadata; used by default within sharding and multi-master topologies |
| HeartbeatFilter | Validation | Detects heartbeat events on masters or slaves |
| JavaScriptFilter | Miscellaneous | Enables filtering through custom JavaScripts |
| LoggingFilter | Logging | Logs filtered events through the standard replicator logging mechanism |
| MySQLSessionSupportFilter | Content | Filters transactions for session specific temporary tables and variables |
| OptimizeUpdatesFilter | Optimization | Optimizes update statements where the current and updated value are the same |
| PrimaryKeyFilter | Optimization | Used during row-based replication to optimize updates using primary keys |
| PrintEventFilter | Logging | Outputs transaction event information to the replication logging system |
| RenameFilter | Transformation | Advanced schema, table and column-based renaming |
| ReplicateColumnsFilter | Content | Removes selected columns from row-based transaction data |
| ReplicateFilter | Content | Selects or ignores specification schemas and/or databases |
| SetToStringFilter | Content |
Converts integer values in
SET datatypes to string
values
|
| ShardFilter | Content | Used to enforce database schema sharding between specific masters |
| TimeDelayFilter | Miscellaneous | Delays transactions until a specific point in time has passed |
In the following reference sections:
Pre-configured filter name is the filter name that can be used against a stage without additional configuration.
Property prefix is the prefix string for the filter to be used when assigning property values.
Classname is the Java class name of the filter.
Parameter is the name of the filter parameter can be set as a property within the configuration.
Data compatibility indicates whether the filter is compatible with row-based events, statement-based events, or both.