Common NLP Python modules included in this book

This book includes a few Python modules for the best learning outcomes. If an NLP task can be performed by other libraries, such as scikit-learn or NLTK, I will show you the code examples for comparison. The libraries included in this book are detailed in the following sections.

spaCy

spaCy is by far the best production-level, open source library for NLP. It makes many processing tasks easy with reliable code and outcomes. If you work with a large volume of texts for text preprocessing, spaCy is an excellent choice. It is designed to be a simple and concise alternative to C.

It can perform a wide range of NLP operations well. These NLP operations include the following tasks:

• Tokenization: This breaks text into individual words or tokens. To a computer, a sentence is just a string of characters. The string has to be separated into words.
• Part-of-speech (PoS) tagging: This assigns grammatical labels to each word in a sentence. For example, the sentence “She loves the beautiful flower” has a pronoun (“she”), a verb (“loves”), an adjective (“beautiful”), and a noun (“flower”). The labeling for the pronoun, verb, adjective, and noun is called PoS tagging.
• Named entity recognition (NER): This identifies named entities such as names, organizations, locations, and so on. For example, in the sentence “I went to New York City on July 4th,” the named entities would be “New York City” (a place), and “July 4th” (a date). It is worth mentioning that spaCy’s built-in NER models are based on the BERT architecture. As we will learn about BERT in this book, it is helpful to be aware of this.
• Lemmatization: This reduces words to their base or dictionary form. We will learn more about lemmatization in Chapter 3, Text Wrangling and Preprocessing.
• Rule-based matching: This can find sequences of words based on user-defined rules.
• Word vectors: These represent words as numerical vectors. When two words become vectors, they can be compared in the vector space. Word embedding and vectorization is an important step in NLP. spaCy provides the functions to do so. We will learn about the concept and practice of word vectorization in Chapter 7, Using Word2Vec.

spaCy can be easily integrated with other libraries such as Gensim and NLTK. That’s why in many code examples you see that spaCy, Gensim, and NLTK are used together.

These are just some of the main capabilities of spaCy, and it offers many more features and functionalities for NLP tasks.

NLTK

NLTK is an open source Python library for natural language processing. It provides a suite of tools for working with text data, including tokenization, PoS tagging, and NER. It provides interfaces to over 50 corpora and lexical resources, such as WordNet. NLTK also includes a number of pre-trained models for tasks such as sentiment analysis and topic modeling. It is widely used in academia and industry for research and development in NLP. NLTK can perform a range of NLP tasks too, including PoS, NER, sentiment analysis, text classification, and text summarization.