Deep Reinforcement Learning for Multi-Functional RIS-Aided Over-the-Air Federated Learning in Internet of Robotic Things

Abstract

To facilitate edge intelligence in Internet of Robotic Things (IoRT), over-the-air federated learning (AirFL) is a communication-efficient enabler by virtue of its high spectrum efficiency and low transmission latency. Supported by multi-functional reconfigurable intelligent surface (MF-RIS), the model aggregation process of AirFL can be facilitated thanks to full-space signal amplification. However, the learning performance of AirFL may be degraded by the uncertainty of wireless channels originated from inaccurate channel estimation and robot mobility. In this paper, we investigate the model aggregation problem of MF-RIS-aided AirFL in a dynamic IoRT system with imperfect channel state information (CSI). We aim to minimize the long-term mean square error (MSE) of AirFL model aggregation by jointly optimizing MF-RIS coefficients and transceiver beamforming. To enable online decision-making in the dynamic system, we propose a novel deep reinforcement learning (DRL)-based double-agent algorithm, where one agent first decides operating modes of MF-RIS elements, and then the other agent devises transceiver beamforming and other MF-RIS coefficients referring to the mode switching strategy. Numerical results unveil the effectiveness and robustness of the proposed DRL-based algorithm in suppressing long-term MSE under hostile CSI.