6.1 Description

When confronted with raw data acquired from a source application, database, or web API, it’s prudent to inspect the data to be sure it really can be used for the desired analysis. It’s common to find that data doesn’t precisely match the given descriptions. It’s also possible to discover that the metadata is out of date or incomplete.

The foundational principle behind this project is the following:

We don’t always know what the actual data looks like.

Data may have errors because source applications have bugs. There could be ”undocumented features,” which are similar to bugs but have better explanations. There may have been actions made by users that have introduced new codes or status flags. For example, an application may have a ”comments” field on an accounts-payable record, and accounting clerks may have invented their own set of coded values, which they put in the last few characters of this field. This...