The S3 service is a highly scalable environment but there are some guidelines that need to be followed to achieve maximum performance from the S3 backend. Your data in S3 is distributed according to the key or key name, which is the name the object identified by in the bucket. The key name determines the partition the data is sotred on within S3. The key can be just the filename, or it can have a prefix. As objects in S3 are grouped and stored in the backend according to their keys, we can expect to achieve at least 3,500 PUT/POST/DELETE and 5,500 GET requests per second per prefix in a bucket. So, if we want to achieve more performance from S3, we need to address multiple partitions at the same time, by distributing the keys across the partitions. In this way we are able to get unlimited performance from S3.
So, let's imagine we have a service...
