maskdata.js Filter

Version availability

Available in 7.1.4 and later.

The maskdata filter enables values in the THL to be obfuscated. This can be useful when replicating Personal Identification Information, such as email addresses, phone number, personal identification numbers and others within the THL but need to be masked on the Target.

warning

Use of this filter should be used with care. The strings generated are purely random and cannot be reverse engineered to the original string, additionally, the same random string will not be generated for the same input, for example "Continuent" may mask to "GuJKUgwnTG" on the first pass and "nnD57Gwby" on the second pass, therefore you MUST not use this filter to mask columns that are part of UNIQUE, PRIMARY or FOREIGN key constraints.

warning

If you choose to set a value to NULL, then you must ensure the target column allows NULL values otherwise the replicator will error.

Pre-configured filter name	maskdata
JavaScript Filter File	tungsten-replicator/support/filters-javascript/maskdata.js
Property prefix	replicator.filter.maskdata
Stage compatibility	All
tpm Option compatibility	svc-extractor-filters, svc-applier-filters
Data compatibility	ROW events only

Parameters

Parameter	Type	Default	Description
definitionsFile	Filename	/opt/continuent/share/maskdata.json	Location of the definitions file for masking columns

The filter is available by default as maskdata, and the filter is configured through a JSON file that defines the list of columns to be masked. The filter relies on the colnames filter being enabled and will only work woth ROW-based replication.

To enable the filter, add the following to your /etc/tungsten/tungsten.ini:

svc-extractor-filters=colnames,maskdata
property=replicator.filter.maskdata.definitionsFile=/opt/continuent/share/maskdata.json

A sample configuration file is provided in tungsten-replicator/support/filters-config/maskdata.json. The JSON options vary depending on the masking required, and are outlined below along with an example:

[
{
  "schema": "hr",
  "table": "employees",
  "columns": [
  {
    "column": "email",
    "format": "email",
    "method": 3
  },
  {
    "column": "phone_number",
    "format": "telno"
  },
  {
    "column": "first_name",
    "format": "setnull"
  },
  {
    "column": "last_name",
    "format": "alphastring",
    "special": "true",
    "cast": "upper"
  }
  ]
}
]

Where:

• schema specifies the name of the schema on which to apply the filtering. If * is given, all schemas are matched.
• table specifies the name of the table on which to apply the filtering. If * is given, all tables are matched.
• columns is an array of column names to be matched.

Format can be set to one of the following:

• email : This will generate a random string in an xxxx@xxxxx.xxx format. Both prefix and domain, when masked, will match the original string length. With this format you can specify 3 different methods using the method option as follows:

  • 1 : Will randomize the first part of the email address, but maintain the original domain. For example, "j.doe@mydomain.co.uk" could mask as "HjnjK@mydomain.co.uk"

  • 2 : Will randomize the domain part of the email address, but maintain the original prefix. For example, "j.doe@mydomain.co.uk" could mask as "j.doe@GhRHMieD.com"

  • 3 : Will randomize all parts of the email address. For example, "j.doe@mydomain.co.uk" could mask as "HjnjK@GhRHMieD.com"

  The cast option can also be supplied to this option. See below for examples.

• telno or numericstring : This will generate a random string of numbers. The string length will match the original string.

• alphastring : This will generate a random string of Alpha characters in a mixed case. The masked string length will match the original string length.

  The cast and special options can also be supplied to this option. See below for examples.

• alphanumericstring : This will generate a random string of AlphaNumeric characters in a mixed case. The masked string length will match the original string length.

  The cast and special options can also be supplied to this option. See below for examples.

• setnull : This will set the value of the column to NULL

The use of cast will control how the masked string is returned, this can be set to one of the following:

• upper : Return all UPPERCASE characters
• lower : Return all lowercase characters
• init : Returns all Lowercase with an Initial uppercase character

If not supplied, the generated string will be a random mix of upper and lower case.

The use of special will also control the inclusion of special characters into the generate string. Set to true to enable. Disabled by default.