Sentence and paragraph embeddings

A simple, yet surprisingly effective solution for generating useful sentence and paragraph embeddings is to average the word vectors of their constituent words. Even though we will describe some popular sentence and paragraph embeddings in this section, it is generally always advisable to try averaging the word vectors as a baseline.

Sentence (and paragraph) embeddings can also be created in a task optimized way by treating them as a sequence of words, and representing each word using some standard word vector. The sequence of word vectors is used as input to train a network for some task. Vectors extracted from one of the later layers of the network just before the classification layer generally tend to produce very good vector representation for the sequence. However, they tend to be very task specific, and are of limited use as a general vector representation.

An idea for generating general vector representations for sentences that could be...