3.8.3.1. Replicating Data from a Cluster to a Datawarehouse (Staging Use Case)

The following Staging-method procedure will install the Tungsten Replicator software onto target node host6, extracting from a cluster consisting of three (3) nodes (host1, host2 and host3) and applying into the target datawarehouse via host6.

Important

If you are replicating to a MySQL-specific target, please see Section 3.7, “Replicating Data Out of a Cluster” for more information.

  1. On your staging server, go to the software directory.

    shell> cd /opt/continuent/software
  2. Download the latest Tungsten Replicator version.

  3. Unpack the release package

    shell> tar xvzf tungsten-replicator-6.0.3-599.tar.gz
  4. Change to the unpackaged directory:

    shell> cd tungsten-replicator-6.0.3-599
  5. Execute the tpm command to configure defaults for the installation.

    shell> ./tools/tpm configure defaults \
    --install-directory=/opt/replicator \
    '--profile-script=~/.bashrc' \
    --replication-password=secret \
    --replication-port=13306 \
    --replication-user=tungsten \
    --start-and-report=true \
    --mysql-allow-intensive-checks=true \
    --user=tungsten

    The description of each of the options is shown below; click the icon to hide this detail:

    Click the icon to show a detailed description of each argument.

  6. Configure a cluster alias that points to the masters and slaves within the current Tungsten Clustering service that you are replicating from:

    shell> ./tools/tpm configure alpha \
        --master=host1 \
        --slaves=host2,host3 \
        --thl-port=2112 \
        --topology=cluster-alias

    The description of each of the options is shown below; click the icon to hide this detail:

    Click the icon to show a detailed description of each argument.

    Important

    This dataservice cluster-alias name MUST be the same as the cluster dataservice name that you are replicating from.

  7. On the replication slave node, copy the convertstringfrommysql.json filter configuration sample file into the /opt/replicator/share directory then edit it to suit:

    cp /opt/replicator/tungsten/tungsten-replicator/support/filters-config/convertstringfrommysql.json /opt/replicator/share/
    vi /opt/replicator/share/convertstringfrommysql.json

    Once the convertstringfrommysql JSON configuration file has been edited, update the /etc/tungsten/tungsten.ini file to add and configure any addition options needed for the specific datawarehouse you are using.

  8. Create the configuration that will replicate from cluster dataservice alpha into the database on the host specified by --relay=host6:

    shell> ./tools/tpm configure omega \
    --relay=host6 \
    --relay-source=alpha \
    --repl-svc-remote-filters=convertstringfrommysql \
    --property=replicator.filter.convertstringfrommysql.definitionsFile=/opt/replicator/share/convertstringfrommysql.json \
    --topology=cluster-slave

    The description of each of the options is shown below; click the icon to hide this detail:

    Click the icon to show a detailed description of each argument.

  9. Now finish configuring the omega dataservice with the options specific to the datawarehouse target in use.

    • AWS RedShift Target

      shell> ./tools/tpm configure omega \
      --batch-enabled=true \
      --batch-load-template=redshift \
      --enable-heterogeneous-slave=true \
      --datasource-type=redshift \
      --replication-host=REDSHIFT_ENDPOINT_FQDN_HERE \
      --replication-user=REDSHIFT_PASSWORD_HERE \
      --replication-password=REDSHIFT_PASSWORD_HERE \
      --redshift-dbname=REDSHIFT_DB_NAME_HERE \
      --svc-applier-filters=dropstatementdata \
      --svc-applier-block-commit-interval=10s \
      --svc-applier-block-commit-size=5
      

      The description of each of the options is shown below; click the icon to hide this detail:

      Click the icon to show a detailed description of each argument.

      Please see Install Amazon Redshift Applier (in [Tungsten Replicator 6.1 Manual]) for more information.

    • Vertica Target

      shell> ./tools/tpm configure omega \
      --batch-enabled=true  \
      --batch-load-template=vertica6 \
      --batch-load-language=js  \
      --datasource-type=vertica  \
      --disable-relay-logs=true \
      --enable-heterogeneous-service=true \
      --replication-user=dbadmin \
      --replication-password=VERTICA_DB_PASSWORD_HERE \
      --replication-host=VERTICA_HOST_NAME_HERE \
      --replication-port=5433  \
      --svc-applier-block-commit-interval=5s \
      --svc-applier-block-commit-size=500  \
      --vertica-dbname=VERTICA_DB_NAME_HERE
      

      Please see Install Vertica Applier (in [Tungsten Replicator 6.1 Manual]) for more information.

    • Elasticsearch Target - Native Applier

      shell> ./tools/tpm configure omega \
      --datasource-type=elasticsearch \
      --replication-host=ELASTICSEARCH_HOST_NAME_HERE \
      --replication-password=null \
      --replication-port=9200 \
      --replication-user=root \
      

      Please see Install Elasticsearch Applier (in [Tungsten Replicator 6.1 Manual]) for more information.

    • For additional targets, please see the full list at Deploying Appliers (in [Tungsten Replicator 6.1 Manual]), or click on some of the targets below:

  10. Once the configuration has been completed, you can perform the installation to set up the Tungsten Replicator services using the tpm command run from the staging directory:

    shell> ./tools/tpm install

If the installation process fails, check the output of the /tmp/tungsten-configure.log file for more information about the root cause.

The cluster-slave replicator should now be installed and ready to use.