Fitted Q-iteration in continuous action-space MDPs

Andras Antos (1) , Rémi Munos (2) , Csaba Szepesvari (3)

1 MTA SZTAKI - Computer and Automation Research Institute [Budapest]
2 SEQUEL - Sequential Learning
3 Department of Computing Science [Edmonton]

We consider continuous state, continuous action batch reinforcement learning where the goal is to learn a good policy from a sufficiently rich trajectory generated by some policy. We study a variant of fitted Q-iteration, where the greedy action selection is replaced by searching for a policy in a restricted set of candidate policies by maximizing the average action values. We provide a rigorous analysis of this algorithm, proving what we believe is the first finite-time bound for value-function based algorithms for continuous state and action problems.

Domaines: Apprentissage [cs.LG]

Andras Antos, Rémi Munos, Csaba Szepesvari. Fitted Q-iteration in continuous action-space MDPs. [Technical Report] 2007, pp.24. ⟨inria-00185311v2⟩