Chapter 1

1. A replay buffer is required for off-policy RL algorithms. We sample from the replay buffer a mini-batch of experiences and use it to train the Q(s,a) state-value function in DQN and the actor's policy in a DDPG.
2. We discount rewards, as there is more uncertainty about the long-term performance of the agent. So, immediate rewards have a higher weight, a reward earned in the next time step has a relatively lower weight, a reward earned in the subsequent time step has an even lower weight, and so on.
3. The training of the agent will not be stable if γ > 1. The agent will fail to learn an optimal policy.
4. A model-based RL agent has the potential to perform well, but there is no guarantee that it will perform better than a model-free RL agent, as the model of the environment we are constructing need not always be a good one. It is also very hard to build an accurate...