LSTM memory block

To be able to answer these questions, our network must have several memory cells, where each can store quasi-dependent bits of information regarding the subject of our enquiry, the French emperor Napoleon Bonaparte. In practice, an LSTM unit can have multiple memory cells, each storing different representations from the input sequence. One may store the gender of the subject, another may store the fact that there are multiple subjects, and so on. For the purpose of having clear illustrations, we have taken the liberty of depicting only one memory cell per diagram in this chapter. We do this because understanding the principle behind the workings of one cell will suffice to extrapolate the functioning of a memory block with multiple memory cells. The part of the LSTM that contains all its memory cells is referred to as a memory block. The adaptive information...