Working with tidy text mining
The tidytext package handles unstructured text by following the tidy data principle, which mandates that data is represented as a structured, rectangular-shaped, and tibble-like object. In the case of text mining, this requires converting a piece of text in a single cell into one token per row in the DataFrame.
Another commonly used representation for a collection of texts (called a corpus) is the document-term matrix, where each row represents one document (this could be a short sentence or a lengthy article) and each column represents one term (a unique word in the whole corpus, for example). Each cell in the matrix usually contains a representative statistic, such as frequency of occurrence, to indicate the number of times the term appears in the document.
We will dive into both representations and look at how to convert between a document-term matrix and a tidy data format for text mining in the following sections.
