Multi-Stage Hybrid-CNN Transformer Model for Human Intent-Prediction

Abstract

Human intention prediction (HIP) is one aspect of Human-Robot Interaction (HRI) that could facilitate understanding and improving how humans interact with robots and computers. However, current gaze-based intent prediction models that perform well often require invasive methods using specialized equipment. In this paper we present a non-invasive, contactless method for predicting human intentions using a multi-stage hybrid CNN-Transformer framework. The model consists of a depth estimator and two key components: a gazed object predictor and a human intent classifier. The gazed object predictor is a modified Detection-Transformer (DETR) and used a ResNet50 backbone for feature extraction and obtained an accuracy of 32.15% in the custom dataset. Meanwhile, the human intent classifier is a transformer-based classifier that achieved a 98% accuracy when predicting human intention based on a series of gazed objects. The resulting cascaded HIP system attained an accuracy of 54%.