Text Representation

A computer operates on zeros and ones, and algorithms operate on numerical values. A computer does not understand beautiful texts such as the plays by William Shakespeare or the books by Leo Tolstoy. So, raw texts need to be converted to numerical values for a computer to process. The first step in NLP is converting texts to numerical values.

In this chapter, we will learn about the basic text representation – Bag-of-Words, Bag-of-N-grams, and TF-IDF. This chapter is for absolute NLP beginners. In this chapter, we will learn how to code with Gensim, scikit-learn, and NLTK. We will cover the following topics:

• What text representation is
• The transition from one-hot encoding to Bag-of-Words to Bag-of-N-grams
• What TF-IDF is
• How to perform Bag-of-Words (BoW) and TF-IDF encoding in Gensim
• The real-world applications of BoW and TF-IDF

By the end of this chapter, you will be able to describe the BoW, Bag-of-N-grams, and TF-IDF methods...