Discovering Amazon Kinesis Data Streams

Amazon KDS is a service composed of streams, shards, and records. A data stream is a logical container of shards. A data stream continuously ingests data from many data sources. Each stream has one or more shards where records are grouped and stored.

Sharding allows the stream to handle more records, while record order is preserved within each shard. Records are the unit of data in the Kinesis data stream, composed of a sequence number, a partition key, and a data blob. KDS segregates the data records belonging to a stream into multiple shards. When you have multiple shards, you can use a partition key to group data on specific shards. Kinesis uses the partition key to assign records to an individual shard. Records are accessed from the stream with the partition level sequence number. Data can be ingested and processed from many sources, such as the listed applications in the diagram:

Figure 4.1 – A stream has...