Chapter 3 – Text Wrangling and Preprocessing

1. Tokenization is the process of splitting a string into a list of tokens.
2. The technique to extract the root form of words is called stemming.
3. There is a slight difference between lemmatization and stemming. Lemmatization converts a word to a meaningful base word. The base word is still an actual word. Stemming converts a word to its root form and may not be a common formal word (such as ‘populated’ becoming ‘popul’).
4. spaCy does not automatically remove stop words but rather gives users full control of stop-word removal. It simply tags stop words for us to remove them.
5. PoS labels the correct meaning of a word in a sentence according to its context. It is a system where a word is assigned a syntactic function such as noun, pronoun, adjective, verb, and so on.
6. Gensim’s preprocess_string class performs all text preprocessing tasks including stop-word removal, tagging, punctuation...