Reinforcement-Learning-Based Continuation Strategy for Autonomous On-Orbit Assembly

Abstract

Autonomous on-orbit assembly operations are usually translated into complex optimal control problems requiring advanced methods to solve. Solved through either direct methods or indirect methods, these optimal control problems are known to be highly sensitive to the initial guess provided to the solver. This paper proposes an algorithm to reduce the problem’s sensitivity to the initial guess by integrating the continuation method and the notion of reinforcement learning. To this end, a continuation space is formed from a trivial problem with a known solution to the desired optimal control problem. A set of continuation parameters, selected from the state, control, or constraint variables, forms the basis for this continuation space. Finding a suitable continuation path is problematic since it requires a comprehensive search algorithm that considers several factors, such as the choice of continuation parameters and continuation steps for each parameter, to find an appropriate path. To search for this path, a reinforcement learning search agent was proposed based on the Sarsa([Formula: see text]) method to traverse the continuation space in real-time for a continuation trajectory satisfying some desired criteria. Numerical experiments were conducted for a series of autonomous on-orbit assembly operations using robotic spacecraft to demonstrate the capability of the proposed technique. Various simulations with different reward functions showed that the shape and size of the objects in the assembly environment affect the solver’s performance.