Closed-Loop: Vision-Guided Experimental Control in Self-Driving Labs

Abstract

In experimental chemistry, actions are adjusted based on what we see—such as dosing until dissolution, heating until melting, or stirring until mixing is complete. However, current self-driving labs (SDLs) do not monitor these visual cues. HeinSight 4.0 fills this gap by integrating computer vision into SDLs to enable real-time experimental adjustments based on visual feedback. The computer vision system detects equipment (e.g., reactor, vial), classifies chemical phases (solid, liquid, air), and analyzes image features such as turbidity and color. By tracking these physical characteristics frame by frame, HeinSight 4.0 infers physical states (e.g., dissolution, separation). This data feeds into a rule-based system that integrates with the SDL to make real-time experimental adjustments (e.g., stir, heat). We demonstrate HeinSight 4.0 adaptability for iterative refinement of two pharmaceutical case studies: purification (solubility screening) and drug formulation (melt spray congeal). To support broader adoption, we integrated HeinSight 4.0 into a hardware-agnostic SDL architecture and deployed it across two institutions with distinct robotic systems. As an open-source tool, HeinSight 4.0 enables SDLs to see, think, and act in real time.