How Swisscom automated Amazon Redshift as a part of their One Information Platform answer utilizing AWS CDK – Half 1

Swisscom is a number one telecommunications supplier in Switzerland. Swisscom’s Information, Analytics, and AI division is constructing a One Information Platform (ODP) answer that may allow each Swisscom worker, course of, and product to profit from the large worth of Swisscom’s information.

In a two-part sequence, we speak about Swisscom’s journey of automating Amazon Redshift provisioning as a part of the Swisscom ODP answer utilizing the AWS Cloud Improvement Equipment (AWS CDK), and we offer code snippets and the opposite helpful references.

On this put up, we deep dive into provisioning a safe and compliant Redshift cluster utilizing the AWS CDK and talk about the most effective practices of secret rotation. We additionally clarify how Swisscom used AWS CDK customized assets in automating the creation of dynamic person teams which can be related for the AWS Identification and Entry administration (IAM) roles matching completely different job capabilities.

In Half 2 of this sequence, we discover utilizing the AWS CDK and a few of the key matters for self-service utilization of the provisioned Redshift cluster by end-users in addition to different managed companies and purposes. These matters embrace federation with the Swisscom id supplier (IdP), JDBC connections, detective controls utilizing AWS Config guidelines and remediation actions, value optimization utilizing the Redshift scheduler, and audit logging.

Amazon Redshift is a quick, scalable, safe, totally managed, and petabyte scale information warehousing service empowering organizations and customers to research large volumes of knowledge utilizing commonplace SQL instruments. Amazon Redshift advantages from seamless integration with many AWS companies, similar to Amazon Easy Storage Service (Amazon S3), AWS Key Administration Service (AWS KMS), IAM, and AWS Lake Formation, to call just a few.

The AWS CDK helps you construct dependable, scalable, and cost-effective purposes within the cloud with the appreciable expressive energy of a programming language. The AWS CDK helps TypeScript, JavaScript, Python, Java, C#/.Web, and Go. Builders can use one in every of these supported programming languages to outline reusable cloud elements often known as constructs. A knowledge product proprietor in Swisscom can use the ODP AWS CDK libraries with a easy config file to provision ready-to-use infrastructure, similar to S3 buckets; AWS Glue ETL (extract, remodel, and cargo) jobs, Information Catalog databases, and crawlers; Redshift clusters; JDBC connections; and extra, with all of the wanted permissions in only a few minutes.

One Information Platform

The ODP structure relies on the AWS Nicely Architected Framework Analytics Lens and follows the sample of getting uncooked, standardized, conformed, and enriched layers as described in Fashionable information structure. Through the use of infrastructure as code (IaC) instruments, ODP permits self-service information entry with unified information administration, metadata administration (information catalog), and commonplace interfaces for analytics instruments with a excessive diploma of automation by offering the infrastructure, integrations, and compliance measures out of the field. On the similar time, the ODP may even be constantly evolving and adapting to the fixed stream of recent extra options being added to the AWS analytics companies. The next high-level structure diagram exhibits ODP with completely different layers of the trendy information structure. On this sequence, we particularly talk about the elements particular to Amazon Redshift (highlighted in crimson).

Harnessing Amazon Redshift for ODP

A pivotal determination within the information warehousing migration course of entails evaluating the extent of a lift-and-shift method vs. re-architecture. Balancing system efficiency, scalability, and value whereas making an allowance for the inflexible system items requires a strategic answer. On this context, Amazon Redshift has stood out as a cloud-centered information warehousing answer, particularly with its simple and seamless integration into the trendy information structure. Its simple integration and fluid compatibility with AWS companies like Amazon QuickSight, Amazon SageMaker, and Lake Formation additional solidifies its alternative for forward-thinking information warehousing methods. As a columnar database, it’s significantly effectively suited to consumer-oriented information merchandise. Consequently, Swisscom selected to offer an answer whereby use case-specific Redshift clusters are provisioned utilizing IaC, particularly utilizing the AWS CDK.

An important facet of Swisscom’s technique is the mixing of those information area and use case-oriented particular person clusters right into a nearly single and unified information atmosphere, ensuring that information ingestion, transformation, and eventual information product sharing stays handy and seamless. That is achieved by customized provisioning of the Redshift clusters based mostly on person or use case wants, in a shared digital non-public cloud (VPC), with information and system governance insurance policies and remediation, IdP federation, and Lake Formation integration already in place.

Though many controls for governance and safety had been put in place within the AWS CDK assemble, Swisscom customers even have the flexibleness to customise their clusters based mostly on what they want. The cluster configurator permits customers to outline the cluster traits based mostly on particular person use case necessities whereas remaining inside the bounds of outlined finest practices. The important thing configurable parameters embrace node sorts, sizing, subnet sorts for routing based mostly on completely different safety insurance policies per person case, enabling scheduler, integration with IdP setup, and any extra post-provisioning setup, just like the creation of particular schemas and group-level entry on it. This flexibility in configuration is achieved for the Amazon Redshift AWS CDK assemble by a Python information class, which serves as a template for customers to specify elements like subnet sorts, scheduler cron expressions, and particular safety teams for the cluster, amongst different configurations. Customers are additionally in a position to choose the kind of subnets (routable-private or non-routable-private) to stick to community safety insurance policies and architectural requirements. See the next information class choices:

class RedShiftOptions:
    node_type: NodeType
    number_of_nodes: int
    vpc_id: str
    security_group_id: Non-obligatory[str]
    subnet_type: SubnetType
    use_redshift_scheduler: bool
    scheduler_pause_cron: str
    scheduler_resume_cron: str
    maintenance_window: str
    # Extra configuration choices ...

The separation of configuration within the RedShiftOptions information class from the cluster provisioning logic within the RedShiftCluster AWS CDK assemble is according to AWS CDK finest practices, whereby each constructs and stacks ought to settle for a property object to permit for full configurability utterly in code. This separates the considerations of configuration and useful resource creation, enhancing the readability and maintainability. The info class construction displays the person configuration from a configuration file, making it simple for customers to specify their necessities. The next code exhibits what the configuration file for the Redshift assemble appears to be like like:

# ===============================
# Amazon Redshift Choices
# ===============================
# The enriched layer relies on Amazon Redshift.
# This part has properties for Amazon Redshift.
#
redshift_options:
  provision_cluster: true                                     # Skip provisioning Amazon Redshift in enriched layer (required)
  number_of_nodes: 2                                          # Variety of nodes for redshift cluster to provision (non-obligatory) (default = 2)
  node_type: "ra3.xlplus"                                     # Sort of the cluster nodes (non-obligatory) (default = "ra3.xlplus")
  use_scheduler: true                                        # Whether or not to make use of the Amazon Redshift scheduler (non-obligatory)
  scheduler_pause_cron: "cron(00 18 ? * MON-FRI *)"           # Cron expression for scheduler pause (non-obligatory)
  scheduler_resume_cron: "cron(00 08 ? * MON-FRI *)"          # Cron expression for scheduler resume (non-obligatory)
  maintenance_window: "solar:23:45-mon:00:15"                   # Upkeep window for Amazon Redshift (non-obligatory)
  subnet_type: "routable-private"                             # 'routable-private' OR 'non-routable-private' (non-obligatory)
  security_group_id: "sg-test-redshift"                       # Safety group ID for Amazon Redshift (non-obligatory) (reference should exist)
  user_groups:                                                # Person teams and their privileges on default DB
    - group_name: dba
      entry: [ 'ALL' ]
    - group_name: data_engineer
      entry: [ 'SELECT' , 'INSERT' , 'UPDATE' , 'DELETE' , 'TRUNCATE' ]
    - group_name: qa_engineer
      entry: [ 'SELECT' ]
  integrate_all_groups_with_idp: false

Admin person secret rotation

As a part of the cluster deployment, an admin person is created with its credentials saved in AWS Secrets and techniques Supervisor for database administration. This admin person is used for automating a number of setup operations, such because the setup of database schemas and integration with Lake Formation. For the admin person, in addition to different customers created for Amazon Redshift, Swisscom used AWS KMS for encryption of the secrets and techniques related to cluster customers. Using Secrets and techniques Supervisor made it easy to stick to IAM safety finest practices by supporting the automated rotation of credentials. Such a setup will be rapidly carried out on the AWS Administration Console or could also be built-in in AWS CDK code with pleasant strategies within the aws_redshift_alpha module. This module offers higher-level constructs (particularly, Layer 2 constructs), together with comfort and helper strategies, in addition to wise default values. This module is experimental and underneath energetic improvement and should have adjustments that aren’t backward suitable. See the next admin person code:

admin_secret_kms_key_options = KmsKeyOptions(
    ...
    key_name="redshift-admin-secret",
    service="secretsmanager"
)
admin_secret_kms_key = aws_kms.Key(
    scope, 'AdminSecretKmsKey,
    # ...
)

# ...

cluster = aws_redshift_alpha.Cluster(
            scope, cluster_identifier,
            # ...
            master_user=aws_redshift_alpha.Login(
                master_username="admin",
                encryption_key=admin_secret_kms_key
                ),
            default_database_name=database_name,
            # ...
        )

See the next code for secret rotation:

self.cluster.add_rotation_single_user(aws_cdk.Period.days(60))

Strategies similar to add_rotation_single_user internally depend on a serverless utility hosted within the AWS Serverless Utility Mannequin repository, which can be in a unique AWS Area exterior of the group’s permission boundary. To successfully use such capabilities, ensure entry to this serverless repository inside the group’s service management insurance policies. If the entry shouldn’t be possible, take into account implementing options similar to customized AWS Lambda capabilities replicating these functionalities (inside your group’s permission boundary).

AWS CDK customized useful resource

A key problem Swisscom confronted was automating the creation of dynamic person teams tied to particular IAM roles at deployment time. As an preliminary and easy answer, Swisscom’s method was creating an AWS CDK customized useful resource utilizing the admin person to submit and run SQL statements. This allowed Swisscom to embed the logic for the database schema, person group assignments, and Lake Formation-specific configurations straight inside AWS CDK code, ensuring that these essential steps are robotically dealt with throughout cluster deployment. See the next code:

sql = get_rendered_stacked_sqls()

custom_resources.AwsCustomResource(scope, 'RedshiftSQLCustomResource',
                                           on_update=custom_resources.AwsSdkCall(
                                               service="RedshiftData",
                                               motion='executeStatement',
                                               parameters={
                                                   'ClusterIdentifier': cluster_identifier,
                                                   'SecretArn': secret_arn,
                                                   'Database': database_name,
                                                   'Sql': f'{sqls}',
                                               },
                                               physical_resource_id=custom_resources.PhysicalResourceId.of(
                                                   f'{account}-{area}-{cluster_identifier}-groups')
                                           ),
                                           coverage=custom_resources.AwsCustomResourcePolicy.from_sdk_calls(
                                               assets=[f'arn:aws:redshift:{region}:{account}:cluster:{cluster_identifier}']
                                           )
                                        )


cluster.secret.grant_read(groups_cr)

This methodology of dynamic SQL, embedded inside the AWS CDK code, offers a unified deployment and post-setup of the Redshift cluster in a handy method. Though this method unifies the deployment and post-provisioning configuration with SQL-based operations, it stays an preliminary technique. It’s tailor-made for comfort and effectivity within the present context. As ODP additional evolves, Swisscom will iterate this answer to streamline SQL operations throughout cluster provisioning. Swisscom stays open to integrating exterior schema administration instruments or comparable approaches the place they add worth.

One other facet of Swisscom’s structure is the dynamic creation of IAM roles tailor-made for the person teams for various job capabilities inside the Amazon Redshift atmosphere. This IAM function era can be pushed by the person configuration, performing as a blueprint for dynamically defining person function to coverage mappings. This allowed them to rapidly adapt to evolving necessities. The next code illustrates the function task:

policy_mappings = {
    "role1": ["Policy1", "Policy2"],
    "role2": ["Policy3", "Policy4"],
    ...
    # Instance:
    # "dba-role": ["AmazonRedshiftFullAccess", "CloudWatchFullAccess"],
    # ...
}

def create_redshift_role(role_name, policy_list):
   # Implementation to create Redshift function with offered insurance policies
   ...

redshift_role_1 = create_redshift_role(
    data_product_name, "role1", policy_names=policy_mappings["role1"])
redshift_role_1 = create_redshift_role(
    data_product_name, "role1", policy_names=policy_mappings["role1"])
# Instance:
# redshift_dba_role = create_redshift_role(
#   data_product_name, "dba-role", policy_names=policy_mappings["dba-role"])
...

Conclusion

Swisscom is constructing its data-as-a-service platform, and Amazon Redshift has a vital function as a part of the answer. On this put up, we mentioned the elements that must be lined in your IaC finest practices to deploy safe and maintainable Redshift clusters utilizing the AWS CDK. Though Amazon Redshift helps industry-leading safety, there are elements organizations want to regulate to their particular necessities. It’s subsequently vital to outline the configurations and finest practices which can be proper in your group and produce it to your IaC to make it accessible in your finish customers.

We additionally mentioned find out how to provision a safe and compliant Redshift cluster utilizing the AWS CDK and take care of the most effective practices of secret rotation. We additionally confirmed find out how to use AWS CDK customized assets in automating the creation of dynamic person teams which can be related for the IAM roles matching completely different job capabilities.

In Half 2 of this sequence, we’ll delve into enhancing self-service capabilities for end-users. We are going to cowl matters like integration with the Swisscom IdP, establishing JDBC connections, and implementing detective controls and remediation actions, amongst others.

The code snippets on this put up are offered as is and can must be tailored to your particular use instances. Earlier than you get began, we extremely advocate talking to an Amazon Redshift specialist.

In regards to the Authors

Asad bin Imtiaz is an Professional Information Engineer at Swisscom, with over 17 years of expertise in architecting and implementing enterprise-level information options.

Jesús Montelongo Hernández is an Professional Cloud Information Engineer at Swisscom. He has over 20 years of expertise in IT methods, information warehousing, and information engineering.

Samuel Bucheli is a Lead Cloud Architect at Zühlke Engineering AG. He has over 20 years of expertise in software program engineering, software program structure, and cloud structure.

Srikanth Potu is a Senior Guide in EMEA, a part of the Skilled Companies group at Amazon Net Companies. He has over 25 years of expertise in Enterprise information structure, databases and information warehousing.