Monitoring your pipeline’s health

To create a robust monitoring and alerting system for your AWS Glue pipeline, you can use Amazon CloudWatch to emit success metrics, create alarms, and monitor the health of your pipeline. We’ll use CloudWatch Metrics, CloudWatch alarms, and Amazon SNS for notifications.

How to do it…

1. Modify the AWS Glue job to emit success metrics:
   1. Update the Glue ETL script: Modify your csv_to_parquet.py script to include the following code to emit custom metrics:

      import sys
      import boto3
      # Emit success metric to CloudWatch
      cloudwatch = boto3.client('cloudwatch')
      cloudwatch.put_metric_data(
          Namespace='GluePipelineMetrics',
          MetricData=[
              {            'MetricName': 'JobSuccess',
                 ...