Use case and architecture overview

For this use case, let's assume you have a consumer-facing website where users are interacting with your web pages by clicking different buttons or links, which are specific to different page navigations, signing up, signing in, or buying products. You have started a promotional sale on your website for a limited duration and you would like to track how users are reacting to it in real time.

To track user activity, your frontend application has integrated click events, which will publish a JSON event string with every mouse click to a message bus such as Amazon Kinesis Data Streams or Kafka. From the message bus, a Spark Streaming-based consumer application will read JSON messages as a micro-batch and write to Amazon S3 data lake for real-time analysis.

To replicate the streaming of click events, we will integrate the Kinesis Data Generator web UI tool, where you can configure a sample JSON event and schedule it to publish a fixed number...