Many recipes in this chapter presume the availability of an S3 bucket, Glue databases and tables, and an IAM role/user, in addition to the one you are utilizing to implement the recipes, to serve as the grantee for data sharing. Furthermore, certain recipes assume the presence of two AWS accounts to streamline the process of sharing data with other AWS accounts.

You can find the code files for this chapter on GitHub at https://github.com/PacktPublishing/Data-Engineering-with-AWS-Cookbook/tree/main/Chapter02.

Amazon Relational Database Service (RDS) provides a functionality to establish read-only replicas, replicating your database for multiple users’ access and efficient data sharing. These replicas can reside in the same region as, or a different one from, the primary one. This proves beneficial when encountering heavy read loads that could potentially impact RDS performance; replicas can accommodate the heavy read loads without impacting the primary instance’s performance. Additionally, you can create a replica in a region that is closer to your users, which can also help in the event of primary region disruption where the replica can be promoted to a standalone instance, thus allowing seamless operation continuation and disaster recovery.

RDS offers two primary replication modes for read replicas:

• In single-primary mode, a single primary RDS instance manages both read and write operations, while one or more read replicas can...

Redshift data sharing is a feature provided by Amazon Redshift. It enables the sharing of data within and across AWS accounts, allowing different teams or users to access the same data sets for their analytics or reporting needs. Datashare reflects updates in the producer cluster in real time with the consumers ensuring that all users have access to the most up-to-date information without the need to create and maintain duplicate copies of the same data.

In this recipe, we will see how to share Redshift data between two Redshift clusters in different AWS accounts.

Getting ready

This recipe assumes that you have two encrypted Redshift clusters using Ra3 node types. Also, the tables to be shared from the producer cluster should not have interleaved sort keys. If you want to follow the recipe with clusters in the same account, you can skip the authorization and association steps. Instead, directly grant usage of the datashare to...

As a centralized repository, a data lake allows you to store all your structured and unstructured data at any scale without needing upfront data schema definitions. AWS Lake Formation simplifies data lake management on AWS by acting as a layer on top of existing data storage services such as S3. It automatically catalogs data, manages access, and integrates with various AWS services such as Glue and Athena. One of its key features is its integration with AWS Glue. Lake formation in conjunction with AWS Resource Access Manager (RAM) offers a secure and simple method for the sharing of the AWS Glue Data Catalog across multiple AWS accounts. This promotes collaboration and data accessibility across different teams or organizations, allowing them to make the most of their data lakes and extract valuable insights from their shared resources. However, you need to carefully plan and test your data-sharing process. Before sharing the catalog...

AWS Lake Formation simplifies the process of setting up and managing a data lake by providing capabilities for data ingestion, organization, and access control. You can arrange your S3 data into tables within the Lake Formation Data Catalog and implement fine-grained access control on them. This means you can enforce precise access permissions at both the table and column levels, ensuring that sensitive data is safeguarded. Fine-grained access control empowers you to exercise precise control over who can view or modify specific tables and columns within the Lake Formation Data Catalog. Consequently, you can provide varying levels of access to different tables or columns, aligning with the specific requirements of different users or roles.

In this recipe, we will use Lake Formation to establish a table for an S3 dataset and provide granular access controls for it.

Getting ready

To follow this recipe...

S3 presigned URLs provide a secure and controlled way to grant time-limited access to S3 objects. You can specify an expiration time when generating the URL, after which the URL becomes invalid. Presigned URLs can have specific permissions associated with them; you can generate URLs that allow only specific actions (such as read, write, or both) on individual objects. This provides granular control over which operations can be performed on the object while minimizing the exposure of your AWS credentials, as you can avoid the need to embed AWS access credentials directly into your application or to share your AWS access keys.

In this recipe, we will learn how to create a presigned URL to download an S3 object.

Getting ready

This recipe assumes that you have an S3 bucket with an object that you will be creating a presigned URL for. Also, you need to perform these steps using a user who has access to download the object as...

Real-time sharing of S3 data ensures immediate responsiveness to changes in data, facilitating seamless communication between various components of a system. By setting up an S3 bucket to invoke a Lambda function upon the occurrence of any new event, performing the required processing, and publishing the data through notifications to an SNS topic, this method provides an effective and scalable means for broadcasting events to multiple subscribers. It offers a dynamic and responsive approach to sharing information in real time across different systems or applications. This approach proves particularly advantageous in scenarios where the timely distribution of information is crucial for maintaining up-to-date and synchronized systems.

In this recipe, we will learn the process of triggering a Lambda function when new files are created in an S3 bucket, subsequently processing the files and broadcasting them as SNS notifications.

Getting ready

For this...

CloudWatch cross-account sharing in AWS is valuable for centralizing monitoring and management. It enables organizations to share monitoring data, including alarms, dashboards, and logs, across AWS accounts while maintaining resource isolation. This facilitates consolidated reporting, collaboration, troubleshooting, and cost management. Cross-account sharing allows for customized access control and fine-grained permissions, and is scalable for growing organizations. It simplifies the sharing of critical monitoring data, enhancing operational efficiency and visibility without compromising security.

In this recipe, we will learn how to share read-only access to CloudWatch data with another AWS account.

Getting ready

To follow this recipe, you need to have two AWS accounts, one of which must have the CloudWatch log group to be shared. The other account will share it and use it for monitoring.

How to...