Setting up event-driven orchestration with Amazon EventBridge

Creating a workflow that reacts to specific events to trigger AWS Glue components on-demand typically requires the combination of AWS Glue with other AWS services, such as AWS Lambda and Amazon EventBridge (formerly known as Amazon CloudWatch Events).

Now, we want our workflow to react to the addition of a new file in an S3 bucket to trigger an AWS Glue job.

Let’s understand when this is required. A media company processes daily uploads of large CSV files to an S3 bucket. Each time a new CSV file is uploaded, the company needs to trigger an ETL pipeline that extracts data from the CSV, processes it, and stores the information in a data lake for analysis. This workflow is fully automated and event-driven, ensuring immediate processing upon file arrival.

Getting ready

Before proceeding with this recipe, ensure you have completed the steps in the Defining a simple workflow using AWS Glue workflows recipe...