Replication into S3 requires two separate replicator installations, one that extracts information from the source database, and a second that generates the CSV files, loads those files into S3.
The two replication services can operate on the same machine, (See Section 5.3, “Deploying Multiple Replicators on a Single Host”) or they can be installed on two different machines.
Once you have completed the configuration of the Amazon S3 bucket, you can configure and install the applier as described using the steps below.
Before installing the applier, the following additions need adding to the extractor configuration. Apply the following parameter to the extractor configuration before installing the applier
Add the following the /etc/tungsten/tungsten.ini
[alpha] ...Existing Replicator Config... enable-heterogeneous-service=trueshell>tpm update
The above step is only applicable for standalone extractors. If you are configuring replications from an existing Tungsten Cluster (Cluster-Extractor), follow the steps outlined here to ensure the cluster is configured correctly: Section 3.4.1, “Prepare: Replicating Data Out of a Cluster”
The applier can now be configured. Unpack the Tungsten Replicator distribution in staging directory:
shell> tar zxf tungsten-replicator-7.1.4-10.tar.gzChange into the staging directory:
shell> cd tungsten-replicator-7.1.4-10Configure the installation using tpm:
shell>./tools/tpm configure defaults \ --reset \ --user=tungsten \ --install-directory=/opt/continuent \ --profile-script=~/.bash_profile \ --rest-api-admin-user=apiuser \ --rest-api-admin-pass=secret \ --replicator-rest-api-ssl=true \ --replicator-rest-api-port=8097 \ --replicator-rest-api-authentication=true \ --replicator-rest-api-address=0.0.0.0shell>./tools/tpm configure alpha \ --master=sourcehost \ --members=localhost \ --role=slave \ --batch-enabled=true \ --batch-load-template=s3 \ --datasource-type=file \ --enable-heterogeneous-service=true
shell> vi /etc/tungsten/tungsten.ini[defaults] user=tungsten install-directory=/opt/continuent profile-script=~/.bash_profile rest-api-admin-user=apiuser rest-api-admin-pass=secret replicator-rest-api-ssl=true replicator-rest-api-port=8097 replicator-rest-api-authentication=true replicator-rest-api-address=0.0.0.0[alpha] master=sourcehost members=localhost role=slave batch-enabled=true batch-load-template=s3 datasource-type=file enable-heterogeneous-service=true
Configuration group defaults
The description of each of the options is shown below; click the icon to hide this detail:
For staging configurations, deletes all pre-existing configuration information between updating with the new configuration values.
System User
--install-directory=/opt/continuent
install-directory=/opt/continuent
Path to the directory where the active deployment will be installed. The configured directory will contain the software, THL and relay log information unless configured otherwise.
--profile-script=~/.bash_profile
profile-script=~/.bash_profile
Append commands to include env.sh in this profile script
--replicator-rest-api-ssl=true
Enable SSL for the API.
--replicator-rest-api-port=8097
Port for the Replicator API.
--replicator-rest-api-authentication=true
replicator-rest-api-authentication=true
Enforce authentication for the API.
--replicator-rest-api-address=0.0.0.0
replicator-rest-api-address=0.0.0.0
Address for the API to bind too.
Configuration group alpha
The description of each of the options is shown below; click the icon to hide this detail:
The hostname of the primary (extractor) within the current service.
Hostnames for the dataservice members
What is the replication role for this service?
Should the replicator service use a batch applier
Value for the loadBatchTemplate property
Database type
--enable-heterogeneous-service=true
enable-heterogeneous-service=true
On a Primary
--mysql-use-bytes-for-string
is set to false.
colnames filter is
enabled (in the
binlog-to-q stage
to add column names to the THL information.
pkey filter is
enabled (in the
binlog-to-q and
q-to-dbms stage),
with the
addPkeyToInserts
and
addColumnsToDeletes
filter options set to false.
enumtostring
filter is enabled (in the
q-to-thl stage), to
translate ENUM
values to their string equivalents.
settostring filter
is enabled (in the
q-to-thl stage), to
translate SET
values to their string equivalents.
On a Replica
--mysql-use-bytes-for-string
is set to true.
If your MySQL source is a Tungsten Cluster, ensure the additional steps below are also included in your applier configuration
First, prepare the required filter configuration file as follows on the S3 applier host(s) only:
shell>mkdir -p /opt/continuent/share/shell>cp tungsten-replicator/support/filters-config/convertstringfrommysql.json /opt/continuent/share/
Then, include the following parameters in the configuration
property=replicator.stage.remote-to-thl.filters=convertstringfrommysql
property=replicator.filter.convertstringfrommysql.definitionsFile=/opt/continuent/share/convertstringfrommysql.json
If you plan to make full use of the REST API (which is enabled by default) you will need to also configure a username and password for API access. This must be done by specifying the following options in your configuration:
rest-api-admin-user=tungsten rest-api-admin-pass=secret
Once the prerequisites and configuring of the installation has been completed, the software can be installed:
shell> ./tools/tpm install
If the installation process fails, check the output of the
/tmp/tungsten-configure.log file for
more information about the root cause.
On the host that is loading data into S3, create the
s3-config-
file and then copy that file into the servicename.jsonshare directory
within the installed directory on that host. For example:
shell> cp s3-config-servicename.json /opt/continuent/share/Now the services can be started:
shell> replicator start