4.9.2. Preparing for Amazon S3 Replication

4.9.2. Preparing for Amazon S3 Replication
Prev	^Up	4.9. Deploying the Amazon S3 CSV Applier	Next

4.9.2. Preparing for Amazon S3 Replication

Preparing the hosts for the replication process requires setting some key configuration parameters within the MySQL server to ensure that data is stored and written correctly.

Configure the source and target hosts following the prerequisites outlined in Appendix B, Prerequisites then follow the appropriate steps for the required extractor topology outlined in Chapter 3, Deploying MySQL Extractors.

The following are required for replication to Amazon S3:

An existing Amazon Web Services (AWS) account, and either the AWS Access Key and Secret Key, or configured IAM Roles, required to interact with the account through the API. For information on creating IAM Roles, see Section 4.2.2.2, “Configuring Identity Access Management within AWS”
A configured Amazon S3 service. If the S3 service has not already been configured, visit the AWS console and sign up for the Amazon S3 service.
If using the s3cmd, you should then configure the command to automatically connect to the Amazon S3 service without requiring further authentication, the .s3cfg in the tungsten users home directory should be configured as follows:
- Using Access Keys:
```
[default]
access_key = ACCESS_KEY
secret_key = SECRET_KEY
```
- Using IAM Roles: Leave values blank - copy example as is
```
[default]
access_key = 
secret_key = 
security_token =
```
Create an S3 bucket that will be used to hold the CSV files that are generated by the replicator. This can be achieved either through the web interface, or via the command-line, for example:
```
shell> s3cmd mb s3://tungsten-csv
```
Create an s3-config-servicename.json file based on the sample provided within cluster-home/samples/conf/s3-config-servicename.json within the Tungsten Replicator staging directory, or using the example below.
Once created, the file will be copied into the /opt/continuent/share directory to be used by the batch applier script.
If multiple services are being created, one file must be created for each service.
The following example shows the use of Access and Secret Keys:
```
{
  "awsS3Path" : "s3://your-bucket-for-s3/s3-test",
  "awsAccessKey" : "access-key-id",
  "awsSecretKey" : "secret-access-key",
  "cleanUpS3Files" : "true"
}
```
The following example shows the use of IAM Roles:
```
{
  "awsS3Path" : "s3://your-bucket-for-s3/s3-test",
  "awsIAMRole" : "arn:iam-role",
}
```
The allowed options for this file are as follows:
- awsS3Path — the location within your S3 storage where files should be loaded.
- awsAccessKey — the S3 access key to access your S3 storage. Not required if awsIAMRole is used.
- awsSecretKey — the S3 secret key associated with the Access Key. Not required if awsIAMRole is used.
- awsIAMRole — the IAM role configured to allow Redshift to interact with S3. Not required if awsAccessKey and awsSecretKey are in use.
- s3Binary — the binary to use for loading csv file up to S3. (Valid Values: s3cmd, s4cmd, aws) (Default: s3cmd)
- gzipS3Files — setting to true will result in the csv files being gzipped prior to loading into S3 (Default: false)

Prev	Up	Next
4.9.1. S3 Replication Operation	^Level	4.9.3. Install Amazon S3 Applier

Continuent Documentation

4.9.2. Preparing for Amazon S3 Replication