Creating a Python pipeline with Amazon S3, Lambda, and Step Functions

In this section, we will create a simple ETL pipeline using AWS Lambda and Step Functions. AWS Lambda is a serverless compute service that allows you to run code without provisioning or managing servers, while Step Functions provides a way to orchestrate the serverless lambda functions and other AWS services into workflows.

Setting the stage with the AWS CLI

Click into the chapter_10 directory of this book’s GitHub repository in your local PyCharm environment. Within the PyCharm terminal, run the following command to configure the AWS CLI:

(Project) usr@project % aws configure

You will then be prompted to enter your access key ID, secret access key, default region name, and default output format. Use your internet browser to log in to your AWS management console to get the following credentials:

AWS Access Key ID [None]: <YOUR ACCESS KEY ID HERE>AWS Secret Access Key [None]: <YOUR SECRET...