You're reading from Deep Reinforcement Learning Hands-On Apply modern RL methods, with deep Q-networks, value iteration, policy gradients, TRPO, AlphaGo Zero and more

Product type Paperback

Published in Jun 2018

Publisher Packt

ISBN-13 9781788834247

Length 546 pages

Edition 1st Edition

Languages

Python

Tools

Deep Reinforcement Learning

Concepts

Deep Reinforcement Learning

Author (1):

Maxim Lapan

View More author details

Table of Contents (21) Chapters

Preface

1. What is Reinforcement Learning? FREE CHAPTER

2. OpenAI Gym

3. Deep Learning with PyTorch

4. The Cross-Entropy Method

5. Tabular Learning and the Bellman Equation

6. Deep Q-Networks

7. DQN Extensions

8. Stocks Trading Using RL

9. Policy Gradients – An Alternative

10. The Actor-Critic Method

11. Asynchronous Advantage Actor-Critic

12. Chatbots Training with RL

13. Web Navigation

14. Continuous Action Space

15. Trust Regions – TRPO, PPO, and ACKTR

16. Black-Box Optimization in RL

17. Beyond Model-Free – Imagination

18. AlphaGo Zero

Other Books You May Enjoy

Leave a review - let other readers know what you think

Index

Proximal Policy Optimization

Historically, this method came from the OpenAI team and was proposed long after TRPO (which is from 2015), but PPO is much simpler than TRPO, so we'll start from it. The paper in which it was proposed is by John Schulman et al and called Proximal Policy Optimization Algorithms, published in 2017 (arXiv:1707.06347).

The core improvement over the classical Asynchronous Advantage Actor-Critic (A3C) method is to change the expression used to estimate the PG. Instead of the gradient of logarithm probability of the action taken, the PPO method uses a different objective: the ratio between the new and the old policy scaled by the advantages.

In math form, the old A3C objective could be written as Proximal Policy Optimization . The new objective proposed by the PPO is . The reason behind changing the objective is the same as for the cross-entropy method from Chapter 4, The Cross-Entropy Method: importance sampling. However, if we just start to blindly maximize this value, it may lead to a very...