5.2.3. Optional Configuration Parameters for Elasticsearch

A number of optional, configurable, properties are available that control how Tungsten Replicator applies and populates information when the data is written into Elasticsearch. The following properties can by set during configuration using --property=PROPERTYNAME=value:

Table 5.1. Optional Elasticsearch Applier Properties

Option	Description
replicator.applier.dbms.docIdFormat	Determines the format of the document ID used to store the information with Elasticsearch
replicator.applier.dbms.embedCommitTime	Sets whether the commit time for the source row is embedded into the document
replicator.applier.dbms.embedSchemaTable	Embed the source schema name and table name in the stored document
replicator.applier.dbms.ignoreDeleteErrors	Whether a failure to delete an entry should raise an error
replicator.applier.dbms.ignoreUpdateErrors	Whether a failure to update an existing entry should raise an error
replicator.applier.dbms.indexName	The string to be used as the index name when the incoming schema name is not used
replicator.applier.dbms.selfGeneratedId	Determines whether to use a self generated document id or one derived from the primary key information
replicator.applier.dbms.typeName	The string to use as the index type name when the incoming table is not used
replicator.applier.dbms.useSchemaAsIndex	Specifies whether the incoming schema name for the data should be used as the Index Name
replicator.applier.dbms.useTableAsType	Specifies whether the incoming table name for the data should be used as the Index Type

− replicator.applier.dbms.docIdFormat

Option	`replicator.applier.dbms.docIdFormat`
Description	Determines the format of the document ID used to store the information with Elasticsearch
Value Type	string
Default	pkey
Valid Values	pkey	Combine the primary key column values into a single string
	pkeyus	Combine the primary key column values into a single string joined by an underscore character
	tspkey	Combine the schema name, table name, and primary key column values into a single string joined by an underscore character
	tspkeyus	Combine the schema name, table name, and primary key column values into a single string

Determines the format of the document ID used to store the information into Elasticsearch. For example, when configured to use tspkeyus, then the format of the document ID will consist of the schemaname, table name and primary key column information separated by underscores, SCHEMANAME_TABLENAME_234.

− replicator.applier.dbms.embedCommitTime

Option	`replicator.applier.dbms.embedCommitTime`
Description	Sets whether the commit time for the source row is embedded into the document
Value Type	boolean
Default	true
Valid Values	false	Do not embed the source database commit time
	true	Embed the source database commit time into the stored document

Embeds the commit time of the source database row into the document information:

{
	"_index" : "test",
	"_type" : "messages",
	"_id" : "99999",
	"_version" : 1,
	"found" : true,
	"_source" : {
		"msg" : "Hello Elasticsearch",
		"committime" : "2017-06-06 19:09:20.0",
		"id" : "99999",
		"source_table" : "messages",
		"source_schema" : "test"
	}
}

− replicator.applier.dbms.embedSchemaTable

Option	`replicator.applier.dbms.embedSchemaTable`
Description	Embed the source schema name and table name in the stored document
Value Type	boolean
Default	true
Valid Values	false	Do not embed the schema or database name in the document
	true	Embed the source schema name and database name into the stored document

If enabled, the documented stored into Elasticsearch will include the source schema and database name. This can be used to identify the source of the information if the schema and table name is not being used for the index and type names (see replicator.applier.dbms.useSchemaAsIndex and replicator.applier.dbms.useTableAsType).

{
	"_index" : "test",
	"_type" : "messages",
	"_id" : "99999",
	"_version" : 1,
	"found" : true,
	"_source" : {
		"msg" : "Hello Elasticsearch",
		"committime" : "2017-06-06 19:09:20.0",
		"id" : "99999",
		"source_table" : "messages",
		"source_schema" : "test"
	}
}

− replicator.applier.dbms.ignoreDeleteErrors

Option	`replicator.applier.dbms.ignoreDeleteErrors`
Description	Whether a failure to delete an entry should raise an error
Value Type	boolean
Default	false
Valid Values	false	Raise a failure when deleting records from Elasticsearch fails
	true	Ignore errors when deleting records from Elasticsearch

The replicator.applier.dbms.ignoreDeleteErrors determines whether the replicator will fail when trying to delete an entry in the Elasticsearch index entry. This option can be useful when deleting data in an Elasticsearch target and data does not need to guarantee to be deleted.

− replicator.applier.dbms.ignoreUpdateErrors

Option	`replicator.applier.dbms.ignoreUpdateErrors`
Description	Whether a failure to update an existing entry should raise an error
Value Type	boolean
Default	false
Valid Values	false	Raise a failure when updating records from Elasticsearch fails
	true	Ignore errors when updating records from Elasticsearch

The replicator.applier.dbms.ignoreUpdateErrors determines whether the replicator will fail when trying to update an entry in the Elasticsearch index entry. This option can be useful when updating data in an Elasticsearch target and data does not need to guarantee to be deleted.

− replicator.applier.dbms.indexName

Option	`replicator.applier.dbms.indexName`
Description	The string to be used as the index name when the incoming schema name is not used
Value Type	string

− replicator.applier.dbms.selfGeneratedId

Option	`replicator.applier.dbms.selfGeneratedId`
Description	Determines whether to use a self generated document id or one derived from the primary key information
Value Type	boolean
Default	false
Valid Values	false	Use the primary key column information as the document ID
	true	Get Elasticsearch to self generate an ID

− replicator.applier.dbms.typeName

Option	`replicator.applier.dbms.typeName`
Description	The string to use as the index type name when the incoming table is not used
Value Type	string

Specifies the type name to be used when replicator.applier.dbms.useTableAsType has been set to false. The value will be used as the index type HTTP://ELASTIC/INDEXNAME/INDEXTYPE.

− replicator.applier.dbms.useSchemaAsIndex

Option	`replicator.applier.dbms.useSchemaAsIndex`
Description	Specifies whether the incoming schema name for the data should be used as the Index Name
Value Type	boolean
Default	true
Valid Values	false	Do not use the incoming schema name as the Elasticsearch index name
	true	Use the incoming schema name as the Elasticsearch index name

Changes whether the string used for the index name within the URL (HTTP://ELASTIC/INDEXNAME/INDEXTYPE). If set to false, the replicator.applier.dbms.typeName must be used to specify the index type.

− replicator.applier.dbms.useTableAsType

Option	`replicator.applier.dbms.useTableAsType`
Description	Specifies whether the incoming table name for the data should be used as the Index Type
Value Type	boolean
Default	true
Valid Values	false	Do not use the incoming table name as the Elasticsearch index type
	true	Use the incoming table name as the Elasticsearch index type

Changes whether the string used for the index type within the URL (HTTP://ELASTIC/INDEXNAME/INDEXTYPE). If set to false, the replicator.applier.dbms.typeName must be used to specify the index type.

Prev	Up	Next
5.2.2. Install Elasticsearch Applier	Home	5.2.4. Management and Monitoring of Elasticsearch Deployments