Here's our paper on adaptive temporal abstraction in reinforcement learning by deciding on not just what actions to take but also how long you want to execute the action. This explores the simple idea of having to decide on changing an action or not only when necessary and hence achieve higher performance for the same sample complexity. We evaluate this with simple experiments on the DQN framework for Atari games.
Keshav,
Check out the generalized version of the dynamic time scales in Deep RL - our ICLR 17 submission at http://openreview.net/pdf?id=B1GOWV5eg where we use the generic Actor Critic framework (with results on TORCS continuos action as well). The improvements on some of the games over A3C is as high as 3000%.
Here's our paper on adaptive temporal abstraction in reinforcement learning by deciding on not just what actions to take but also how long you want to execute the action. This explores the simple idea of having to decide on changing an action or not only when necessary and hence achieve higher performance for the same sample complexity. We evaluate this with simple experiments on the DQN framework for Atari games.
Thanks Keshav.
Did you try experiments by adding more than |A| actions, i.e. adding 2*|A| or 3*|A|actions. If yes, what were the results like?
Yes we did try with more number of repetition extents, upto 3|A|. The results did improve. However, the network needs more exploration as well to ensure all actions are sufficiently explored over the state space for proper value estimates. So, this sort of approach is not scalable.
As mentioned in the paper, we experimented with a generic structured policy, where you predict the extent of the repetition (we consider all integers from 1 to 100 as the possible extents) along with the probability of the action. This is done using an Actor Critic setup similar to the A3C paper of DeepMind. We have very good results on a few games through this architecture.
Did you try games other than the 3 mentioned in the ...
Here's our paper on adaptive temporal abstraction in reinforcement learning by deciding on not just what actions to take but also how long you want to execute the action. This explores the simple idea of having to decide on changing an action or not only when necessary and hence achieve higher performance for the same sample complexity. We evaluate this with simple experiments on the DQN framework for Atari games.
Here's our paper on automated discovery of high level skills (options) in reinforcement learning through MDP abstraction with extension to large state space problems through Deep Unsupervised Action Conditional Video Prediction Network.
Here is our new paper "A2T: Attend, Adapt and Transfer: Attentive Deep Architecture for Adaptive Transfer from Multiple Sources" - an attention mechanism based (Deep) Transfer Learning paradigm that learns to focus on appropriate experts relevant for the target task dynamic to the state space.
