Active In
Deep Learning
Artificial Intelligence
Featured Contributions
2.
discussion
[Paper Summary] Bridging the Gap Between Value and Policy Based Reinforcement Learning
Acknowledging that γ-discounted entropy regularization is used in reward expectation, they formulate a new notion of softmax temporal consistency for optimal Q-values as:
Q_*(s,a) = r(s,a)+\gamma \tau \:\log{a'} \ e^{Q_*(s',a')/\tau}
They then introduce a new RL algorithm called Path Consistency Learning (PCL) that extends this softmax temporal consistency to arbitrary (multi-step) trajectories. Pseudocode of PCL Algorithm can be seen in Section 3.5
Unlike algorithms using Qπ-values, PCL seamlessly combines on-policy and off-policy traces. Unlike algorithms based on hard-max consistency and Q◦-values, PCL easily generalizes to multi-step backups on arbitrary paths, while maintaining proper regularization and consistent mathematical justification (that they outline in Section 3 and the appendix).
Read more…(354 words)
paper-notes/Bridging the Gap Between Value and Policy Based Reinforcement Learning.md at master · ethancaballero/paper-notes · GitHub
Unlike algorithms using Qπ-values, PCL seamlessly combines on-policy and off-policy traces.
They compare performance of PCL, A3C, & DQN on the algorithmic tasks from OpenAI Gym, and PCL outperforms or equates A3C & DQN on all tasks.
Inserting just 10
https://github.com/ethancaballero/paper-notes/blob/master/Bridging%20the%20Gap%20Between%20Value%20and%20Policy%20Based%20Reinforcement%20Learning.md
[1702.08892] Bridging the Gap Between Value and Policy Based Reinforcement Learning
https://arxiv.org/abs/1702.08892