The importance of EDA
The term EDA comes from the work of John W. Tukey, one of the most prominent exponents of modern statistical methodology. In his 1977 book Exploratory Data Analysis (hence the acronym EDA), Tukey thinks of EDA as a way to explore data, uncover evidence, and develop hypotheses that can later be confirmed by statistical tests.
His idea was that how we define statistical hypotheses could be based more on observation and reasoning than just sequential tests based on mathematical computations. This idea translates well to the world of machine learning because, as we will discuss in the next section, data can be improved and pre-digested so that learning algorithms can work better and more efficiently.
In an EDA for a Kaggle competition, you will be looking for:
- Missing values and, most importantly, missing value patterns correlated with the target.
- Skewed numeric variables and their possible transformations.
- Rare categories in categorical...
