5.9.3. Install Amazon Redshift Applier

Replication into Redshift requires two separate replicator installations, one that extracts information from the source database, and a second that generates the CSV files, loads those files into S3 and then executes the statements on the Redshift database to import the CSV data and apply the transformations to build the final tables.

The two replication services can operate on the same machine, (See Section 6.3, “Deploying Multiple Replicators on a Single Host”) or they can be installed on two different machines.

Once you have completed the configuration of the Amazon Redshift database, you can configure and install the applier as described using the steps below.

  1. Before installing the applier, the following additions need adding to the extractor configuration. Apply the following parameters, update the extractor and then install the applier

    • For Staging Install:

      shell> cd tungsten-replicator-6.1.0-54
      shell> ./tools/tpm configure alpha \
        --repl-svc-extractor-filters=colnames,pkey \
        --property=replicator.filter.pkey.addColumnsToDeletes=true \
        --property=replicator.filter.pkey.addPkeyToInserts=true
      shell> ./tools/tpm update
    • For INI Installs: Add the following the /etc/tungsten/tungsten.ini

      
      [alpha]
      ...Existing Replicator Config...
      repl-svc-extractor-filters=colnames,pkey
      property=replicator.filter.pkey.addColumnsToDeletes=true
      property=replicator.filter.pkey.addPkeyToInserts=true
      
      
      shell> tpm update
  2. The applier can now be configured. Unpack the Tungsten Replicator distribution in staging directory:

    shell> tar zxf tungsten-replicator-6.1.0-54.tar.gz
  3. Change into the staging directory:

    shell> cd tungsten-replicator-6.1.0-54
  4. Configure the installation using tpm:

    Show Staging

    Show INI

    shell> ./tools/tpm configure defaults \
        --reset \
        --user=tungsten \
        --install-directory=/opt/continuent \
        --profile-script=~/.bash_profile
    
    shell> ./tools/tpm configure alpha \
        --topology=master-slave \
        --master=sourcehost \
        --members=localhost \
        --datasource-type=redshift \
        --replication-host=redshift.us-east-1.redshift.amazonaws.com \
        --replication-user=awsRedshiftUser \
        --replication-password=awsRedshiftPass \
        --redshift-dbname=dev \
        --batch-enabled=true \
        --batch-load-template=redshift \
        --svc-applier-filters=dropstatementdata \
        --svc-applier-block-commit-interval=10s \
        --svc-applier-block-commit-size=5
    
    shell> vi /etc/tungsten/tungsten.ini
    [defaults]
    user=tungsten
    install-directory=/opt/continuent
    profile-script=~/.bash_profile
    
    [alpha]
    topology=master-slave
    master=sourcehost
    members=localhost
    datasource-type=redshift
    replication-host=redshift.us-east-1.redshift.amazonaws.com
    replication-user=awsRedshiftUser
    replication-password=awsRedshiftPass
    redshift-dbname=dev
    batch-enabled=true
    batch-load-template=redshift
    svc-applier-filters=dropstatementdata
    svc-applier-block-commit-interval=10s
    svc-applier-block-commit-size=5
    

    Configuration group defaults

    The description of each of the options is shown below; click the icon to hide this detail:

    Click the icon to show a detailed description of each argument.

    Configuration group alpha

    The description of each of the options is shown below; click the icon to hide this detail:

    Click the icon to show a detailed description of each argument.

  5. Once the prerequisites and configuring of the installation has been completed, the software can be installed:

    shell> ./tools/tpm install

If the installation process fails, check the output of the /tmp/tungsten-configure.log file for more information about the root cause.

On the host that is loading data into Redshift, create the s3-config-servicename.json file and then copy that file into the share directory within the installed directory on that host. For example:

shell> cp s3-config-servicename.json /opt/continuent/share/

Now the services can be started:

shell> tpm start alpha

Once the service is configured and running, the service can be monitored as normal using the trepctl command. See Section 5.9.6, “Management and Monitoring of Amazon Redshift Deployments” for more information.