Latent Dirichlet Allocation

Latent Dirichlet Allocation (LDA) is the prototypical method of performing topic modeling. Rather unfortunately, the acronym LDA is also used for another method in machine learning. This latter method is completely different from LDA and is commonly used as a way to perform dimensionality reduction and classification.

Although LDA involves a substantial amount of mathematics, it is worth exploring some of its technical details in order to understand how the model works and the assumptions that it uses. First and foremost, we should learn about the Dirichlet distribution, which lends its name to LDA.

Note

An excellent reference for a fuller treatment of Topic Models with LDA is the Topic Models chapter in Text Mining: Classification, Clustering, and Applications, edited by A. Srivastava and M. Sahami and published by Chapman & Hall, 2009.

The Dirichlet distribution

Suppose we have a classification problem with K classes and the probability of each class is fixed...