Semantic-Aware Monostatic ISAC System with Dynamic Multimodal Fusion

Abstract

Integrated sensing and communication (ISAC) systems promise unified high-speed connectivity and environmental perception, yet current methods focus mainly on geometry reconstruction and overlook semantic characterization, which is crucial for real-world tasks such as smart building management, autonomous navigation, and robotic operation. To address this gap, we present a novel multimodal dynamic fusion framework that combines real-world radio point-cloud data, acquired by a millimeter-wave (mmWave) monostatic ISAC platform, with deep visual semantic information from camera imagery. First, we introduce a reliability model that evaluates each radio point-cloud measurement by its signal energy and beam-to-normal incidence angle to eliminate false detections and denoise the point cloud. Second, a convolutional neural network generates a bird’s-eyeview (BEV) that encapsulates scene information labels derived from RGB images, including roads, vehicles, and trees, yielding per-region confidence scores. Finally, an adaptive weighting mechanism fuses the radio and visual modalities on a perpoint basis, dynamically adjusting contributions based on sensor reliability and scene context to enable accurate semantic labeling. Extensive field experiments demonstrate that our dynamic fusion approach significantly outperforms static-fusion baselines in both denoising quality and semantic matching accuracy.