Demonstration of the Dyna Reinforcement Learning Framework for Reactive Close Proximity Operations

Abstract

Lessons from the International Space Station (ISS) emphasize the necessity of exterior inspection for anomaly detection and maintenance, but current methods rely on costly and limited human extravehicular activities and robotic arms. Deployable free-flying small spacecraft offer a flexible, autonomous solution, capable of comprehensive exterior inspections without human involvement. However, the safety of these spacecraft during close proximity operations remains a concern, particularly given uncertain variability in thruster performance. This paper presents SmallSat Steward, a reactive and integrated architecture for online model learning and trajectory planning based on the Dyna reinforcement learning architecture. By combining model-based planning and direct reinforcement learning, Dyna offers a potentially flexible and computationally efficient solution capable of adapting to changes in thruster performance and other system uncertainties. Preliminary results in both simulation and hardware environments demonstrate the potential of this architecture to successfully regulate position under single and double thruster failures. In simulation, the Dyna-based controller outperformed a PD-LQR controller in $\sim$70\% of all cases. On hardware, Dyna was able to eliminate the steady state error caused by thruster failures.