Chapter 13, Scaling and Federating Prometheus

1. You should consider sharding when you're sure a single instance isn't enough to handle the load, and you can't run it with more resources.
2. Vertical sharding is used to split scrape workload according to responsibility (for example, by function or team), where each Prometheus shard scrapes different jobs. Horizontal sharding splits loads from a single scrape job into multiple Prometheus instances.
3. To reduce the ingestion load on a Prometheus instance, you should consider dropping unnecessary metrics through the use of metric_relabel_configs rules, or by increasing the scrape interval so that fewer samples are ingested in total.
4. Instance-level Prometheus servers should federate job-level aggregate metrics. Job-level Prometheus servers should federate datacenter-level aggregate metrics.
5. You might need to use metrics only...