RAG-PRISM: A Personalized, Rapid, and Immersive Skill Mastery Framework with Adaptive Retrieval-Augmented Tutoring

Abstract

The rapid digital transformation of Fourth Industrial Revolution (4IR) systems is transforming today's workforce needs, increasing skill set gaps, especially for the older workforce. With an increasing emphasis on STEM skill sets like robotics, automation, artificial intelligence (AI), and security, the workforce will have to be re-skilled and up-skilled to meet future industry needs. While re-skilling/up-skilling a massive workforce, these programs have to be mindful of trainee's diverse backgrounds, pedagogy styles, and motivations to increase student persistence, retention, and success; to ensure rapid and cost-effective workforce development while ensuring skill building through experiential learning. To address these challenges, we present an adaptive tutoring framework that explores the usage of generative artificial intelligence (AI) combined with Retrieval-Augmented Generation (RAG)s to generate personalized training for each students learning needs. Our framework uses a combination of document hit rate and Mean Reciprocal Rank (MMR) to personalize and optimize the training for the trainee. The framework's personalization is evaluated against a human-generated training to evaluate the framework's quality of personalization through source content alignment, and relevance metrics. We apply the proposed framework for 4 IR cybersecurity learning, through the creation of a synthetic question-answer (QA) dataset emulating trainee behavior, while the RAGs are optimized on a curated cybersecurity learning materials corpus. The proposed framework, is evaluated for its new training generation, by comparison with a set of manually curated queries to represent realistic student interactions. The framework's responses are generated through multiple large language models (LLMs) including GPT-3.5 and GPT-4 variants, which are evaluated for content alignment, and relevance (faithfulness), with GPT-4 having the best performance with a relevancy score of 87 %, and 100 % content alignment. Thus, our preliminary evaluation shows that this dual-mode approach allows the adaptive tutor to serve as both a new personalized topic recommender, providing a novel approach to provide rapid, personalized learning for 4IR learning and workforce development needs.