Human-in-the-loop control of dynamics and robotics using augmented reality

Abstract

• Augmented Reality (AR) interface allows for real-time monitoring and control of vibrations in dynamic experimentation. • Consolidates data visualization, actuator control, and robotic mediation to manage vibratory responses. • Sensor-equipped accelerometers and exciter controls enable precise generation of structural vibrations. • AR-based control reduces time delay and improves user cognition for handling dynamic inputs. • Application results show AR improves accuracy in managing vibrations and controlling robots. Human-machine interaction (HMI) and human-robot interaction (HRI) offer innovative approaches for the monitoring, analysis, and management of structural dynamics both in laboratory settings and real-world scenarios. In vibratory experimentation, an external force is used to evaluate the dynamic responses of structures. One mode of generation is to use electrodynamic exciters to simulate environmental vibrations. Sensors are attached to the structure to measure dynamic responses and can be deployed by repeatable robots with high endurance. In post-disaster inspections robotic deployment ensures the safety of inspectors and requires on-the-fly control. If the interface between operators and the controls were augmented, operators can visualize experiments, exciter levels, and define robot input while maintaining awareness of the area of interest. Robots can provide better aid to humans if intelligent on-the-fly control of the robot is: (1) quantified and presented to the human; and (2) conducted in real-time for human feedback informed by data. Information provided by the new interface can be used to change the control input based on understanding of real-time parameters. This research proposes the utilization of Augmented Reality (AR) to provide operators with sensor feedback and control for the analysis of dynamic structures. This method improves cognition by allowing the operator to maintain awareness of structures while adjusting conditions accordingly with the assistance of the new real-time interface. An interface application has been developed to plot sensor data in addition to voltage, frequency, and duration controls for vibration simulations. Two more applications were developed following the same framework, one to control the position of a mediating robot and one to control the frequency of the robot's movement. The applications serve as a first-step approach investigating potential robotic sensor deployment in disaster assessment of structures. This paper presents the proposed model for the new vibratory control loop and then compares the new approach with a traditional method by measuring time delay in control input. The study demonstrated through experimental tests that AR enhances time efficiency and accuracy in controlling robotic arm positions, matching actuator frequencies, and deploying sensors with minimal time delay, benefiting both novice and expert users.