Chapter 12. Replication Filters

Table of Contents

12.1. Enabling/Disabling Filters
12.2. Enabling Additional Filters
12.3. Filter Status
12.4. Filter Reference
12.4.1. ansiquotes.js Filter
12.4.2. BidiRemoteSlave (BidiSlave) Filter
12.4.3. breadcrumbs.js Filter
12.4.4. CaseTransform Filter
12.4.5. ColumnName Filter
12.4.6. ConvertStringFromMySQL Filter
12.4.7. DatabaseTransform (dbtransform) Filter
12.4.8. dbrename.js Filter
12.4.9. dbselector.js Filter
12.4.10. dbupper.js Filter
12.4.11. dropcolumn.js Filter
12.4.12. dropcomments.js Filter
12.4.13. dropddl.js Filter
12.4.14. dropmetadata.js Filter
12.4.15. droprow.js Filter
12.4.16. dropstatementdata.js Filter
12.4.17. dropsqlmode.js Filter
12.4.18. dropxa.js Filter
12.4.19. Dummy Filter
12.4.20. EnumToString Filter
12.4.21. EventMetadata Filter
12.4.22. foreignkeychecks.js Filter
12.4.23. Heartbeat Filter
12.4.24. insertsonly.js Filter
12.4.25. Logging Filter
12.4.26. MySQLSessionSupport (mysqlsessions) Filter
12.4.27. mapcharset Filter
12.4.28. NetworkClient Filter
12.4.29. nocreatedbifnotexists.js Filter
12.4.30. OptimizeUpdates Filter
12.4.31. PrimaryKey Filter
12.4.32. PrintEvent Filter
12.4.33. Rename Filter
12.4.34. Replicate Filter
12.4.35. ReplicateColumns Filter
12.4.36. Row Add Database Name Filter
12.4.37. Row Add Transaction Info Filter
12.4.38. SetToString Filter
12.4.39. Shard Filter
12.4.40. shardbyseqno.js Filter
12.4.41. shardbytable.js Filter
12.4.42. SkipEventByType Filter
12.4.43. TimeDelay (delay) Filter
12.4.44. TimeDelayMsFilter (delayInMS) Filter
12.4.45. tosingledb.js Filter
12.4.46. truncatetext.js Filter
12.4.47. zerodate2null.js Filter
12.5. Standard JSON Filter Configuration
12.5.1. Rule Handling and Processing
12.5.2. Schema, Table, and Column Selection
12.6. JavaScript Filters
12.6.1. Writing JavaScript Filters
12.6.2. Installing Custom JavaScript Filters

Filtering operates by applying the filter within one, or more, of the stages configured within the replicator. Stages are the individual steps that occur within a pipeline, that take information from a source (such as MySQL binary log) and write that information to an internal queue, the transaction history log, or apply it to a database. Where the filters are applied ultimately affect how the information is stored, used, or represented to the next stage or pipeline in the system.

For example, a filter that removed out all the tables from a specific database would have different effects depending on the stage it was applied. If the filter was applied on the Extractor before writing the information into the THL, then no Applier could ever access the table data, because the information would never be stored into the THL to be transferred to the Targets. However, if the filter was applied on the Applier, then some Appliers could replicate the table and database information, while other Appliers could choose to ignore them. The filtering process also has an impact on other elements of the system. For example, filtering on the Extractor may reduce network overhead, albeit at a reduction in the flexibility of the data transferred.

In a standard replicator configuration with MySQL, the following stages are configured in the Extractor, as shown in Figure 12.1, “Filters: Pipeline Stages on Extractors”.

Figure 12.1. Filters: Pipeline Stages on Extractors

Filters: Pipeline Stages on Extractors

Where:

  • binlog-to-q Stage

    The binlog-to-q stage reads information from the MySQL binary log and stores the information within an in-memory queue.

  • q-to-thl Stage

    The in-memory queue is written out to the THL file on disk.

Within the Applier, the stages configured by default are shown in Figure 12.2, “Filters: Pipeline Stages on Appliers”.

Figure 12.2. Filters: Pipeline Stages on Appliers

Filters: Pipeline Stages on Appliers

  • remote-to-thl Stage

    Remote THL information is read from an trext; datasource and written to a local file on disk.

  • thl-to-q Stage

    The THL information is read from the file on disk and stored in an in-memory queue.

  • q-to-dbms Stage

    The data from the in-memory queue is written to the target database.

Filters can be applied during any configured stage, and where the filter is applied, alters the content and availability of the information. The staging and filtering mechanism can also be used to apply multiple filters to the data, altering content when it is read and when it is applied.

Where more than one filter is configured for a pipeline, each filter is executed in the order it appears in the configuration. For example, within the following fragment:

...
replicator.stage.binlog-to-q.filters=settostring,enumtostring,pkey,colnames
...

settostring is executed first, followed by enumtostring, pkey and finally colnames.

For certain filter combinations this order can be significant. Some filters rely on the information provided by earlier filters.