Challenges of external data

When we rely on external data to drive our application, we are implicitly dependent on the quality and stability of that data. This is, of course, true for any data, but when the data is generated by an external source over which we do not have control, the risks are most likely higher. Regardless, when building what we expect to be reliable applications on top of such data feeds, and especially when our data volumes grow, we need to think about how to mitigate these risks.

Data validation

We use the general term data validation to refer to the act of ensuring that incoming data complies with our expectations and potentially applying normalization to modify it accordingly or to even delete malformed or corrupt input. What this actually involves will be very application-specific. In some cases, the important thing is ensuring the system only ingests data that conforms to a given definition of accurate or clean. For our tweet data, we don't care about every single...