Spectral Analysis of Heavy-Ball Q-value Iteration

Abstract

We study the convergence and acceleration of Qvalue iteration (QVI) with heavy-ball momentum. Although acceleration of value iteration has been studied extensively, there has been less work on the acceleration of heavy-ball QVI for control tasks. We analyze heavy-ball QVI from the viewpoint of switched linear system (SLS) theory and the joint spectral radius (JSR). First, we convert heavy-ball QVI into an exact SLS and interpret its convergence through the JSR. We also give JSR-based conditions under which heavy-ball QVI can be faster than standard QVI.