Full Gradient DQN Reinforcement Learning: A Provably Convergent Scheme

Konstantin Avrachenkov (1) , Vivek S Borkar (2) , Harsh P Dolhare (2) , Kishor Patil (1)

1 NEO - Network Engineering and Operations
2 EE-IIT - Department of Electrical Engineering [IIT-Bombay]

We analyze the DQN reinforcement learning algorithm as a stochastic approximation scheme using the o.d.e. (for 'ordinary differential equation') approach and point out certain theoretical issues. We then propose a modified scheme called Full Gradient DQN (FG-DQN, for short) that has a sound theoretical basis and compare it with the original scheme on sample problems. We observe a better performance for FG-DQN.

Mots clés: Markov Decision Process (MDP); approximate dynamic programming; Deep Reinforcement Learning (DRL); stochastic approximation; Deep Q-Network (DQN); Full Gradient DQN; Bellman error minimization

Domaines: Apprentissage [cs.LG]; Optimisation et contrôle [math.OC]; Probabilités [math.PR]

Konstantin Avrachenkov, Vivek S Borkar, Harsh P Dolhare, Kishor Patil. Full Gradient DQN Reinforcement Learning: A Provably Convergent Scheme. Alexey Piunovskiy; Yi Zhang. Modern Trends in Controlled Stochastic Processes: Theory and Applications, V.III, 41, Springer International Publishing, pp.192-220, 2021, Emergence, Complexity and Computation, 978-3-030-76928-4. ⟨10.1007/978-3-030-76928-4_10⟩. ⟨hal-03462350⟩