Accessibility to data

Even though there are many existing data sources, some of them might be inaccessible, unreliable, or irrelevant to the purpose of your pipeline. Further, measuring the accessibility of data sources is dependent on circumstances that are oftentimes outside of your direct control, such as procuring the right access credentials, assessing the finesse of a company’s cybersecurity system, or simply having a limitation in your system’s short-term storage capacity.

Whatever the unique limitations of data accessibility are, it needs to be accounted for within the design of your architecture since the ability to connect to data sources with consistency has heavy impacts on downstream workflows. Being aware of the frequency of connection for each data source is essential to not only choose the most accessible data source but also to help determine the cadence of certain analytics due to the restriction of how often various sources can be merged together...