Handle Amazon Redshift provisioned clusters with Terraform

Amazon Redshift is a quick, scalable, safe, and totally managed cloud knowledge warehouse that makes it easy and cost-effective to research all of your knowledge utilizing commonplace SQL and your current extract, rework, and cargo (ETL); enterprise intelligence (BI); and reporting instruments. Tens of hundreds of shoppers use Amazon Redshift to course of exabytes of information per day and energy analytics workloads corresponding to BI, predictive analytics, and real-time streaming analytics.

HashiCorp Terraform is an infrastructure as code (IaC) instrument that permits you to outline cloud sources in human-readable configuration recordsdata that you could model, reuse, and share. You possibly can then use a constant workflow to provision and handle your infrastructure all through its lifecycle.

On this submit, we exhibit the best way to use Terraform to handle frequent Redshift cluster operations, corresponding to:

• Creating a brand new provisioned Redshift cluster utilizing Terraform code and including an AWS Identification and Entry Administration (IAM) position to it
• Scheduling pause, resume, and resize operations for the Redshift cluster

Answer overview

The next diagram illustrates the answer structure for provisioning a Redshift cluster utilizing Terraform.

Along with Amazon Redshift, the answer makes use of the next AWS companies:

• Amazon Elastic Compute Cloud (Amazon EC2) presents the broadest and deepest compute platform, with over 750 cases and selection of the newest processors, storage, networking, working system (OS), and buy mannequin that can assist you finest match the wants of your workload. For this submit, we use an m5.xlarge occasion with the Home windows Server 2022 Datacenter Version. The selection of occasion sort and Home windows OS is versatile; you possibly can select a configuration that fits your use case.
• IAM lets you securely handle identities and entry to AWS companies and sources. We use IAM roles and insurance policies to securely entry companies and carry out related operations. An IAM position is an AWS id that you could assume to realize momentary entry to AWS companies and sources. Every IAM position has a set of permissions outlined by IAM insurance policies. These insurance policies decide the actions and sources the position can entry.
• AWS Secrets and techniques Supervisor lets you securely retailer the person identify and password wanted to log in to Amazon Redshift.

On this submit, we exhibit the best way to arrange an surroundings that connects AWS and Terraform. The next are the high-level duties concerned:

1. Arrange an EC2 occasion with Home windows OS in AWS.
2. Set up Terraform on the occasion.
3. Configure your surroundings variables (Home windows OS).
4. Outline an IAM coverage to have minimal entry to carry out actions on a Redshift cluster, together with pause, resume, and resize.
5. Set up an IAM position utilizing the coverage you created.
6. Create a provisioned Redshift cluster utilizing Terraform code.
7. Connect the IAM position you created to the Redshift cluster.
8. Write the Terraform code to schedule cluster operations like pause, resume, and resize.

Stipulations

To finish the actions described on this submit, you want an AWS account and administrator privileges on the account to make use of the important thing AWS companies and create the required IAM roles.

Create an EC2 occasion

We start with creating an EC2 occasion. Full the next steps to create a Home windows OS EC2 occasion:

1. On the Amazon EC2 console, select Launch Occasion.
2. Select a Home windows Server Amazon Machine Picture (AMI) that fits your necessities.
3. Choose an applicable occasion sort in your use case.
4. Configure the occasion particulars:
   1. Select the VPC and subnet the place you wish to launch the occasion.
   2. Allow Auto-assign Public IP.
   3. For Add storage, configure the specified storage choices in your occasion.
   4. Add any needed tags to the occasion.
5. For Configure safety group, choose or create a safety group that permits the required inbound and outbound visitors to your occasion.
6. Evaluation the occasion configuration and select Launch to begin the occasion creation course of.
7. For Choose an current key pair or create a brand new key pair, select an current key pair or create a brand new one.
8. Select Launch occasion.
9. When the occasion is working, you possibly can connect with it utilizing the Distant Desktop Protocol (RDP) and the administrator password obtained from the Get Home windows password

Set up Terraform on the EC2 occasion

Set up Terraform on the Home windows EC2 occasion utilizing the next steps:

1. RDP into the EC2 occasion you created.
2. Set up Terraform on the EC2 occasion.

You could replace the surroundings variables to level to the listing the place the Terraform executable is out there.

3. Below System Properties, on the Superior tab, select Setting Variables.

4. Select the trail variable.

5. Select New and enter the trail the place Terraform is put in. For this submit, it’s within the C: listing.

6. Affirm Terraform is put in by coming into the next command:

terraform -v

Optionally, you should utilize an editor like Visible Studio Code (VS Code) and add the Terraform extension to it.

Create a person for accessing AWS by means of code (AWS CLI and Terraform)

Subsequent, we create an administrator person in IAM, which performs the operations on AWS by means of Terraform and the AWS Command Line Interface (AWS CLI). Full the next steps:

1. Create a brand new IAM person.
2. On the IAM console, obtain and save the entry key and person key.

3. Set up the AWS CLI.
4. Launch the AWS CLI and run aws configure and cross the entry key ID, secret entry key, and default AWS Area.

This prevents the AWS person identify and password from being seen in plain textual content within the Terraform code and prevents unintentional sharing when the code is dedicated to a code repository.

Create a person for Accessing Redshift by means of code (Terraform)

As a result of we’re making a Redshift cluster and subsequent operations, the administrator person identify and password required for these processes (totally different than the admin position we created earlier for logging in to the AWS Administration Console) must be invoked within the code. To do that securely, we use Secrets and techniques Supervisor to retailer the person identify and password. We write code in Terraform to entry these credentials throughout the cluster create operation. Full the next steps:

1. On the Secrets and techniques Supervisor console, select Secrets and techniques within the navigation pane.
2. Select Retailer a brand new secret.

3. For Secret sort, choose Credentials for Amazon Redshift knowledge warehouse.
4. Enter your credentials.

Arrange Terraform

Full the next steps to arrange Terraform:

1. Create a folder or listing for storing all of your Terraform code.
2. Open the VS Code editor and browse to your folder.
3. Select New File and enter a reputation for the file utilizing the .tf extension

Now we’re prepared to begin writing our code beginning with defining suppliers. The suppliers definition is a manner for Terraform to get the required APIs to work together with AWS.

4. Configure a supplier for Terraform:

terraform {
required_providers {
aws = {
supply  = "hashicorp/aws"
model = "5.53.0"
}
}
}

# Configure the AWS Supplier
supplier "aws" {
area = "us-east-1"
}

5. Entry the admin credentials for the Amazon Redshift admin person:

knowledge "aws_secretsmanager_secret_version" "creds" {
# Fill within the identify you gave to your secret
secret_id = "terraform-creds"
}
/*json decode to parse the key*/
locals {
terraform-creds = jsondecode(
knowledge.aws_secretsmanager_secret_version.creds.secret_string
)
}

Create a Redshift cluster

To create a Redshift cluster, use the aws_redshift_cluster useful resource:

# Create an encrypted Amazon Redshift cluster

useful resource "aws_redshift_cluster" "dw_cluster" {
cluster_identifier = "tf-example-redshift-cluster"
database_name      = "dev"
master_username    = native.terraform-creds.username
master_password    = native.terraform-creds.password
node_type          = "ra3.xlplus"
cluster_type       = "multi-node"
publicly_accessible = "false"
number_of_nodes    = 2
encrypted         = true
kms_key_id        = native.RedshiftClusterEncryptionKeySecret.arn
enhanced_vpc_routing = true
cluster_subnet_group_name="<<your-cluster-subnet-groupname>>"
}

On this instance, we create a Redshift cluster referred to as tf-example-redshift-cluster, utilizing the ra3.xlplus node sort 2 node cluster. We use the credentials from Secrets and techniques Supervisor and jsondecode to entry these values. This makes positive the person identify and password aren’t handed in plain textual content.

Add an IAM position to the cluster

As a result of we didn’t have the choice to affiliate an IAM position throughout cluster creation, we accomplish that now with the next code:

useful resource "aws_redshift_cluster_iam_roles" "cluster_iam_role" {
cluster_identifier = aws_redshift_cluster.dw_cluster.cluster_identifier
iam_role_arns      = ["arn:aws:iam::yourawsaccountId:role/service-role/yourIAMrolename"]
}

Allow Redshift cluster operations

Performing operations on the Redshift cluster corresponding to resize, pause, and resume on a schedule presents a extra sensible use of those operations. Due to this fact, we create two insurance policies: one that permits the Amazon Redshift scheduler service and one that permits the cluster pause, resume, and resize operations. Then we create a task that has each insurance policies hooked up to it.

You possibly can carry out these steps instantly from the console after which referenced in Terraform code. The next instance demonstrates the code snippets to create insurance policies and a task, after which to connect the coverage to the position.

1. Create the Amazon Redshift scheduler coverage doc and create the position that assumes this coverage:

#outline coverage doc to ascertain the Belief Relationship between the position and the entity (Redshift scheduler)

knowledge "aws_iam_policy_document" "assume_role_scheduling" {
assertion {
impact = "Permit"
principals {
sort        = "Service"
identifiers = ["scheduler.redshift.amazonaws.com"]
}
actions = ["sts:AssumeRole"]
}
}

#create a task that has the above belief relationship hooked up to it, in order that it could possibly invoke the redshift scheduling service
useful resource "aws_iam_role" "scheduling_role" {
identify               = "redshift_scheduled_action_role"
assume_role_policy = knowledge.aws_iam_policy_document.assume_role_scheduling.json
}

2. Create a coverage doc and coverage for Amazon Redshift operations:

/*outline the coverage doc for different redshift operations*/

knowledge "aws_iam_policy_document" "redshift_operations_policy_definition" {
assertion {
impact = "Permit"
actions = [
"redshift:PauseCluster",
"redshift:ResumeCluster",
"redshift:ResizeCluster",
]
sources = ["arn:aws:redshift:*:youraccountid:cluster:*"]
}
}

/*create the coverage and add the above knowledge (json) to the coverage*/
useful resource "aws_iam_policy" "scheduling_actions_policy" {
identify   = "redshift_scheduled_action_policy"
coverage = knowledge.aws_iam_policy_document.redshift_operations_policy_definition.json
}

3. Connect the coverage to the IAM position:

/*join the coverage and the position*/
useful resource "aws_iam_role_policy_attachment" "role_policy_attach" {
policy_arn = aws_iam_policy.scheduling_actions_policy.arn
position       = aws_iam_role.scheduling_role.identify
}

4. Pause the Redshift cluster:

#pause a cluster
useful resource "aws_redshift_scheduled_action" "pause_operation" {
identify     = "tf-redshift-scheduled-action-pause"
schedule = "cron(00 22 * * ? *)"
iam_role = aws_iam_role.scheduling_role.arn
target_action {
pause_cluster {
cluster_identifier = aws_redshift_cluster.dw_cluster.cluster_identifier
}
}
}

Within the previous instance, we created a scheduled motion referred to as tf-redshift-scheduled-action-pause that pauses the cluster at 10:00 PM on daily basis as a cost-saving motion.

5. Resume the Redshift cluster:

identify     = "tf-redshift-scheduled-action-resume"
schedule = "cron(15 07 * * ? *)"
iam_role = aws_iam_role.scheduling_role.arn
target_action {
resume_cluster {
cluster_identifier = aws_redshift_cluster.dw_cluster.cluster_identifier
}
}
}

Within the previous instance, we created a scheduled motion referred to as tf-redshift-scheduled-action-resume that resumes the cluster at 7:15 AM on daily basis in time for enterprise operations to begin utilizing the Redshift cluster.

6. Resize the Redshift cluster:

#resize a cluster
useful resource "aws_redshift_scheduled_action" "resize_operation" {
identify     = "tf-redshift-scheduled-action-resize"
schedule = "cron(15 14 * * ? *)"
iam_role = aws_iam_role.scheduling_role.arn
target_action {
resize_cluster {
cluster_identifier = aws_redshift_cluster.dw_cluster.cluster_identifier
cluster_type = "multi-node"
node_type = "ra3.xlplus"
number_of_nodes = 4 /*enhance the variety of nodes utilizing resize operation*/
traditional = true /*default habits is to make use of elastic resizeboolean worth if we wish to use traditional resize*/
}
}
}

Within the previous instance, we created a scheduled motion referred to as tf-redshift-scheduled-action-resize that will increase the nodes from 2 to 4. You are able to do different operations like change the node sort as effectively. By default, elastic resize might be used, however if you wish to use traditional resize, you need to cross the parameter traditional = true as proven within the previous code. This generally is a scheduled motion to anticipate the wants of peak intervals and resize appripriately for that period. You possibly can then downsize utilizing related code throughout non-peak occasions.

Take a look at the answer

We apply the next code to check the answer. Change the useful resource particulars accordingly, corresponding to account ID and Area identify.

terraform {
  required_providers {
    aws = {
      supply  = "hashicorp/aws"
      model = "5.53.0"
    }
  }
}

# Configure the AWS Supplier
supplier "aws" {
  area = "us-east-1"
}

# entry secrets and techniques saved in secret supervisor
knowledge "aws_secretsmanager_secret_version" "creds" {
  # Fill within the identify you gave to your secret
  secret_id = "terraform-creds"
}

/*json decode to parse the key*/
locals {
  terraform-creds = jsondecode(
    knowledge.aws_secretsmanager_secret_version.creds.secret_string
  )
}

#Retailer the arn of the KMS key for use for encrypting the redshift cluster

knowledge "aws_secretsmanager_secret_version" "encryptioncreds" {
  secret_id = "RedshiftClusterEncryptionKeySecret"
}
locals {
  RedshiftClusterEncryptionKeySecret = jsondecode(
    knowledge.aws_secretsmanager_secret_version.encryptioncreds.secret_string
  )
}

# Create an encrypted Amazon Redshift cluster
useful resource "aws_redshift_cluster" "dw_cluster" {
  cluster_identifier = "tf-example-redshift-cluster"
  database_name      = "dev"
  master_username    = native.terraform-creds.username
  master_password    = native.terraform-creds.password
  node_type          = "ra3.xlplus"
  cluster_type       = "multi-node"
  publicly_accessible = "false"
  number_of_nodes    = 2
  encrypted         = true
  kms_key_id        = native.RedshiftClusterEncryptionKeySecret.arn
  enhanced_vpc_routing = true
  cluster_subnet_group_name="redshiftclustersubnetgroup-yuu4sywme0bk"
}

#add IAM Function to the Redshift cluster

useful resource "aws_redshift_cluster_iam_roles" "cluster_iam_role" {
  cluster_identifier = aws_redshift_cluster.dw_cluster.cluster_identifier
  iam_role_arns      = ["arn:aws:iam::youraccountid:role/service-role/yourrolename"]
}

#for audit logging please create an S3 bucket which has learn write privileges for Redshift service, this instance doesn't embrace S3 bucket creation.

useful resource "aws_redshift_logging" "redshiftauditlogging" {
  cluster_identifier   = aws_redshift_cluster.dw_cluster.cluster_identifier
  log_destination_type = "s3"
  bucket_name          = "your-s3-bucket-name"
}

#to do operations like pause, resume, resize on a schedule we have to first create a task that has permissions to carry out these operations on the cluster

#outline coverage doc to ascertain the Belief Relationship between the position and the entity (Redshift scheduler)

knowledge "aws_iam_policy_document" "assume_role_scheduling" {
  assertion {
    impact = "Permit"
    principals {
      sort        = "Service"
      identifiers = ["scheduler.redshift.amazonaws.com"]
    }

    actions = ["sts:AssumeRole"]
  }
}

#create a task that has the above belief relationship hooked up to it, in order that it could possibly invoke the redshift scheduling service
useful resource "aws_iam_role" "scheduling_role" {
  identify               = "redshift_scheduled_action_role"
  assume_role_policy = knowledge.aws_iam_policy_document.assume_role_scheduling.json
}

/*outline the coverage doc for different redshift operations*/

knowledge "aws_iam_policy_document" "redshift_operations_policy_definition" {
  assertion {
    impact = "Permit"
    actions = [
      "redshift:PauseCluster",
      "redshift:ResumeCluster",
      "redshift:ResizeCluster",
    ]

    sources =  ["arn:aws:redshift:*:youraccountid:cluster:*"]
  }
}

/*create the coverage and add the above knowledge (json) to the coverage*/

useful resource "aws_iam_policy" "scheduling_actions_policy" {
  identify   = "redshift_scheduled_action_policy"
  coverage = knowledge.aws_iam_policy_document.redshift_operations_policy_definition.json
}

/*join the coverage and the position*/

useful resource "aws_iam_role_policy_attachment" "role_policy_attach" {
  policy_arn = aws_iam_policy.scheduling_actions_policy.arn
  position       = aws_iam_role.scheduling_role.identify
}

#pause a cluster

useful resource "aws_redshift_scheduled_action" "pause_operation" {
  identify     = "tf-redshift-scheduled-action-pause"
  schedule = "cron(00 14 * * ? *)"
  iam_role = aws_iam_role.scheduling_role.arn
  target_action {
    pause_cluster {
      cluster_identifier = aws_redshift_cluster.dw_cluster.cluster_identifier
    }
  }
}

#resume a cluster

useful resource "aws_redshift_scheduled_action" "resume_operation" {
  identify     = "tf-redshift-scheduled-action-resume"
  schedule = "cron(15 14 * * ? *)"
  iam_role = aws_iam_role.scheduling_role.arn
  target_action {
    resume_cluster {
      cluster_identifier = aws_redshift_cluster.dw_cluster.cluster_identifier
    }
  }
}

#resize a cluster

useful resource "aws_redshift_scheduled_action" "resize_operation" {
  identify     = "tf-redshift-scheduled-action-resize"
  schedule = "cron(15 14 * * ? *)"
  iam_role = aws_iam_role.scheduling_role.arn
  target_action {
    resize_cluster {
      cluster_identifier = aws_redshift_cluster.dw_cluster.cluster_identifier
      cluster_type = "multi-node"
      node_type = "ra3.xlplus"
      number_of_nodes = 4 /*enhance the variety of nodes utilizing resize operation*/
      traditional = true /*default habits is to make use of elastic resizeboolean worth if we wish to use traditional resize*/
    }
  }
}

Run terraform plan to see an inventory of modifications that might be made, as proven within the following screenshot.

After you will have reviewed the modifications, use terraform apply to create the sources you outlined.

You can be requested to enter sure or no earlier than Terraform begins creating the sources.

You possibly can affirm that the cluster is being created on the Amazon Redshift console.

After the cluster is created, the IAM roles and schedules for pause, resume, and resize operations are added, as proven within the following screenshot.

You can too view these scheduled operations on the Amazon Redshift console.

Clear up

When you deployed sources such because the Redshift cluster and IAM roles, or any of the opposite related sources by working terraform apply, to keep away from incurring expenses in your AWS account, run terraform destroy to tear these sources down and clear up your surroundings.

Conclusion

Terraform presents a robust and versatile resolution for managing your infrastructure as code utilizing a declarative strategy, with a cloud-agnostic nature, useful resource orchestration capabilities, and powerful neighborhood help. This submit supplied a complete information to utilizing Terraform to deploy a Redshift cluster and carry out vital operations corresponding to resize, resume, and pause on the cluster. Embracing IaC and utilizing the precise instruments, corresponding to Workflow Studio, VS Code, and Terraform, will allow you to construct scalable and maintainable distributed purposes, and automate processes.

Concerning the Authors

Amit Ghodke is an Analytics Specialist Options Architect based mostly out of Austin. He has labored with databases, knowledge warehouses and analytical purposes for the previous 16 years. He loves to assist clients implement analytical options at scale to derive most enterprise worth.

Ritesh Kumar Sinha is an Analytics Specialist Options Architect based mostly out of San Francisco. He has helped clients construct scalable knowledge warehousing and massive knowledge options for over 16 years. He likes to design and construct environment friendly end-to-end options on AWS. In his spare time, he loves studying, strolling, and doing yoga.