Understanding remote write and remote read

Prometheus is purposely limited in the functionality it endeavors to implement. Notably, its storage is tied to a local filesystem, and there is no clustering or replication support for Prometheus instances. When running Prometheus at a sufficient scale, you will inevitably encounter the desire to scale Prometheus beyond these limitations. Rather than constrain you to only using Prometheus, Prometheus has built-in functionality to integrate with other metrics storage systems that are more featureful (but also more complex).

Prometheus does this by exposing two remote storage APIs—one for reading data from a storage system and one for writing data to it. Both APIs are exposed via HTTP, use protocol buffers (protobufs) for communication, and use “snappy” compression for both requests and responses.

Remote write is certainly the more popular of the two since it involves sending data to a larger metrics system, which...