Multi-Stage Learning for Grasp-Constrained Object Manipulation with a Simulated Panda Robot

摘要

This repository contains code and experiment assets for RAES (Reward-Aligned Expert Sequencing) and RSTB (Reward-Saturated Temporal Branching)—two lightweight reinforcement learning (RL) frameworks for long-horizon robotic manipulation without demonstrations, learned high-level controllers, or heavy task engineering. We study the robosuite [1] Stack benchmark (reach → grasp → lift/align → stack) and show that coupling continuous shaping with discrete endpoints into reward pairs produces a smoother, more learnable landscape. RAES aligns modular experts with these reward pairs (reach–grasp; lift/align–stack) and executes them sequentially. Across 10M timesteps and 5 seeds, RAES achieves the strongest performance under paired rewards, reaching 223.27 ± 28.68 mean return and 19.30% ± 2.55% success—surpassing PPO and curriculum baselines—while remaining fully RL-based (no demonstrations or hand-coded subtasks). Note: RAES and RASE (Reward Aligned Sequence of Experts) are used interchangeably in the codebase 1. Zhu Y, Wong J, Mandlekar A, Martín-Martín R, Joshi A, Lin K, Maddukuri A, Nasiriany S, Zhu Y. robosuite: A Modular Simulation Framework and Benchmark for Robot Learning. arXiv:2009.12293 [cs.RO]; 2025. Available at: https://arxiv.org/abs/2009.12293.