Chapter 5

1. DDPG is an off-policy algorithm, as it uses a replay buffer.
2. In general, the same number of hidden layers and the number of neurons per hidden layer is used for the actor and the critic, but this is not required. Note that the output layer will be different for the actor and the critic, with the actor having the number of outputs equal to the number of actions; the critic will have only one output.
3. DDPG is used for continuous control, that is, when the actions are continuous and real-valued. Atari Breakout has discrete actions, and so DDPG is not suitable for Atari Breakout.
4. We use the relu activation function, and so the biases are initialized to small positive values so that they fire at the beginning of the training and allow gradients to back-propagate.
5. This is an exercise. See https://gym.openai.com/envs/InvertedDoublePendulum-v2/.
6. This is also an exercise. Notice...