3.8.3.2. Replicating Data from a Cluster to a Datawarehouse (INI Use Case)

The following INI-based procedure will install the Tungsten Replicator software onto target node host6, extracting from a cluster consisting of three (3) nodes (host1, host2 and host3) and applying into the target datawarehouse via host6.

Important

If you are replicating to a MySQL-specific target, please see Deploying the MySQL Applier (in [Tungsten Replicator 6.1 Manual]) for more information.

  1. On the replication slave node, copy the convertstringfrommysql.json filter configuration sample file into the /opt/replicator/share directory then edit it to suit:

    cp /opt/replicator/tungsten/tungsten-replicator/support/filters-config/convertstringfrommysql.json /opt/replicator/share/
    vi /opt/replicator/share/convertstringfrommysql.json

    Once the convertstringfrommysql JSON configuration file has been edited, update the /etc/tungsten/tungsten.ini file to add and configure any addition options needed for the specific datawarehouse you are using.

  2. Create the configuration file /etc/tungsten/tungsten.ini on the destination DBMS host, i.e. host6:

    [defaults]
    user=tungsten
    install-directory=/opt/replicator
    replication-user=tungsten
    replication-password=secret
    replication-port=3306
    profile-script=~/.bashrc
    mysql-allow-intensive-checks=true
    start-and-report=true
    
    [alpha]
    topology=cluster-alias
    master=host1
    members=host1,host2,host3
    thl-port=2112
    
    [omega]
    topology=cluster-slave
    relay=host6
    relay-source=alpha
    repl-svc-remote-filters=convertstringfrommysql
    property=replicator.filter.convertstringfrommysql.definitionsFile=/opt/replicator/share/convertstringfrommysql.json
    

    The description of each of the options is shown below; click the icon to hide this detail:

    Click the icon to show a detailed description of each argument.

    Important

    The cluster-alias name (i.e. alpha) MUST be the same as the cluster dataservice name that you are replicating from.

    Note

    Do not include start-and-report=true if you are taking over for MySQL native replication. See Section 5.12.1, “Migrating from MySQL Native Replication 'In-Place'” for next steps after completing installation.

  3. Now finish configuring the omega dataservice with the options specific to the datawarehouse target in use.

    Append the appropriate code snippet below to the bottom of the existing [omega] stanza:

    • AWS RedShift Target - Offboard Batch Applier

      batch-enabled=true
      batch-load-template=redshift
      datasource-type=redshift
      enable-heterogeneous-slave=true
      replication-host=REDSHIFT_ENDPOINT_FQDN_HERE
      replication-user=REDSHIFT_PASSWORD_HERE
      replication-password=REDSHIFT_PASSWORD_HERE
      redshift-dbname=REDSHIFT_DB_NAME_HERE
      svc-applier-filters=dropstatementdata
      svc-applier-block-commit-interval=10s
      svc-applier-block-commit-size=5
      

      The description of each of the options is shown below; click the icon to hide this detail:

      Click the icon to show a detailed description of each argument.

      Please see Install Amazon Redshift Applier (in [Tungsten Replicator 6.1 Manual]) for more information.

    • Vertica Target - Onboard/Offboard Batch Applier

      batch-enabled=true 
      batch-load-template=vertica6
      batch-load-language=js 
      datasource-type=vertica
      disable-relay-logs=true
      enable-heterogeneous-service=true
      replication-user=dbadmin
      replication-password=VERTICA_DB_PASSWORD_HERE
      replication-host=VERTICA_HOST_NAME_HERE
      replication-port=5433
      svc-applier-block-commit-interval=5s
      svc-applier-block-commit-size=500
      vertica-dbname=VERTICA_DB_NAME_HERE
      

      Please see Install Vertica Applier (in [Tungsten Replicator 6.1 Manual]) for more information.

    • Elasticsearch Target - Onboard/Offboard Native Applier

      datasource-type=elasticsearch
      replication-host=ELASTICSEARCH_HOST_NAME_HERE
      replication-password=null
      replication-port=9200
      replication-user=root
      

      Please see Deploying the Elasticsearch Applier (in [Tungsten Replicator 6.1 Manual]) for more information.

    • For additional targets, please see the full list at Deploying Appliers (in [Tungsten Replicator 6.1 Manual]), or click on some of the targets below:

  4. Download and install the latest Tungsten Replicator package (.rpm), or download the compressed tarball and unpack it on host6:

    shell> cd /opt/continuent/software
    shell> tar xvzf tungsten-replicator-6.0.3-599.tar.gz
  5. Change to the Tungsten Replicator staging directory:

    shell> cd tungsten-replicator-6.0.3-599
  6. Run tpm to install the Tungsten Replicator software with the INI-based configuration:

    shell > ./tools/tpm install

    During the installation and startup, tpm will notify you of any problems that need to be fixed before the service can be correctly installed and started. If the service starts correctly, you should see the configuration and current status of the service.

If the installation process fails, check the output of the /tmp/tungsten-configure.log file for more information about the root cause.

The cluster-slave replicator should now be installed and ready to use.