What makes recurrent networks distinctive from others?

You might be curious to know the specialty of RNNs. This section of the chapter will discuss these things, and from the next section onwards, we will talk about the building blocks of this type of network.

From Chapter 3 , Convolutional Neural Network, you have probably got a sense of the harsh limitation of convolutional networks and that their APIs are too constrained; the network can only take an input of a fixed-sized vector, and also generates a fixed-sized output. Moreover, these operations are performed through a predefined number of intermediate layers. The primary reason that makes RNNs distinctive from others is their ability to operate over long sequences of vectors, and produce different sequences of vectors as the output.

	"If training vanilla neural nets is optimization over functions, training recurrent nets is optimization over programs"
	--Alex Lebrun

We show different types of input-output relationships of the neural...