Integration of LLMs and the Physical World: Research and Application

摘要

The emergence of large language models (LLMs) offers a new opportunity to build LLMs-based applications, such as smart home, as these models have demonstrated general-purpose language understanding by generating coherent and contextually relevant text. However, LLMs are trained on massive amounts of text data to predict tokens, so these models have limitations and it is difficult for them performing physical world tasks directly. To further exploit the potential of LLMs to solve the challenge of integrating them with the physical world, LLMs enhanced and augmented techniques should be addressed, especially reinforcement learning based techniques. In this paper, we study the issue of integrating LLMs with physical world. We first describe the large language models and limitations. Then, we revisit LLMs enhanced and augmented techniques. After that, we present methods of interaction LLMs with physical world, such as integration IoT sensing with LLMs, embodied agent post-training with LLMs, and robot task planning with LLMs. Finally, we provide a case study of smart home powered by LLMs to discuss future research directions of next-generation intelligent smart home, personal health assistant, and LLM-based household robot.