Language Modeling

So far, we have reviewed the most basic techniques for pre-processing text data. Now we are going to dive deep into the structure of natural language – language models. We can consider this topic an introduction to machine learning in NLP.

Introduction to Language Models

A statistical Language Model (LM) is the probability distribution of a sequence of words, which means, to assign a probability to a particular sentence. For example, LMs could be used to calculate the probability of an upcoming word in a sentence. This involves making some assumptions about the structure of the LM and how it will be formed. An LM is never totally correct with its output, but using one is often necessary.

LMs are used in many more NLP tasks. For example, in machine translation, it is important to know what sentence precedes the next. LMs are also used for speech recognition, to avoid ambiguity, for spelling corrections, and for summarization.

Let's see how an LM is mathematically represented...