Word vector representations

Distributional semantics represents words as vectors in the space of senses. The vectors corresponding to the words with similar meanings should be close to each other in this space. How to build such vectors is not a simple question, however. The simplest approach to take is to start from one-hot vectors for the words, but then the vectors will be both sparse and giant, each one of the same length as the number of words in the vocabulary. That's why we use dimensionality reduction with autoencoder-like architecture.