Standard JSON Filter Configuration

A number of the filters that are included as part of Tungsten Replicator use a standardised form of configuration file that is designed to be easy to use and familiar, while being flexible enough to support the needs of each filter. For the majority of filter configurations, the core focus of the configuration is based on a 'default' setting, and settings that are specific to a schema or table.

The JSON configuration follows this basic model. The following filters support the use of this JSON configuration file format:

• convertstringfrommysql
• pkey
• skipeventbytype

The basic format of the configuration is a JSON file that is split into two sections:

• A default section, which determines what will happen in the absence of a schema/table specific rule.
• A collection of schema and table specific entries that determine what happens for a specific schema/table combination.

Depending on the filter and use case, the information within both sections can then either be further divided into column-specific information, or the information may be configured as key/value pairs, or objects, to configure individual parts of the filter configuration.

For example, the following configuration file is from the pkey filter:

{
  "__default": {
      "IGNORE" : "pkey"
  },
  "test" : {
      "msg" : {
          "msg" : "pkey"
      }
  }
}

The above shows the defaults section, and the schema/table specific section.

note

Depending on the filter, the default section may merely be a placeholder to indicate the format of the file. The __default should never be removed.

The sample shows a full schema name, table name, and then column name configuration.

By comparison, the sample below has only schema and table name information, with the configuration within that section being used to define the key/value pairs for specific operations as part of the skipeventbytype filter:

{
  "__default": {
      "INSERT" : "allow",
      "DELETE" : "allow",
      "UPDATE" : "allow"
  },
  "SCHEMA" : {
      "TABLE" : {
          "INSERT" : "allow",
          "DELETE" : "deny",
          "UPDATE" : "deny"
      }
  }
}

The selection and execution of the rules is determined by some specific rules, as detailed in the following sections.

Rule Handling and Processing

The processing of the rules and the selection of the tables and appropriate response and operation is configured through the combination of the default and schema/table settings according to explicit rules:

• If the incoming data matches the schema and table (and optionally column) according to the rules, use the configuration information in that section.
• If the schema/table is not specified or does not have explicit configuration, use the configuration within the __default section instead.

The default rule is always processed and followed if there is no match for an explicit schema, table, or column definition.

Schema, Table, and Column Selection

The format of the JSON configuration and the selection of the schema, table, and column information is in the form of nested structure of JSON objects. The schema first, then the table, then optionally the column within a nested JSON structure. For example:

"test" : {
  "msg" : {
      "id" : "pkey"
  }
}

In the above example:

• test is the schema name
• msg is the table name within the test schema
• id is the column name within the test.msg table

For different tables within the same schema, place another entry at the same level:

"test" : {
  "msg" : {
      "id" : "pkey"
  },
  "orders" : {
      "id" : "pkey"
  },
}

The above now handles the tables msg and orders within the test schema.

Wildcards are also supported, using the * operator. For example:

"orders" : {
  "*" : {
      "INSERT" : "allow",
      "DELETE" : "deny",
      "UPDATE" : "deny"
  }
}

Would match all tables within the orders schema. If multiple definitions exist, then the matching operates on the closest match first. For example:

"orders" : {
  "sales" : {
      "INSERT" : "deny",
      "DELETE" : "deny",
      "UPDATE" : "deny"
  },
  "*" : {
      "INSERT" : "allow",
      "DELETE" : "deny",
      "UPDATE" : "deny"
  }
}

In the above, if the schema/table combination orders.sales is seen, the rule for that is always used first as it is explicitly stated. Only tables that do not match the wildcards will use the wildcard entry. If neither an explicit schema/table or wildcard exist, the default is used.