The Second Coder was a Robot: Generative AI Tools in Establishing Intercoder Reliability

Abstract

Content analysis is a common research method in technical and professional communication (TPC) journals. But TPC content analyses often lack intercoder reliability (ICR) statistics, possibly due to the lack of resources required to train human coders. This proof-of-concept study explores the viability of replacing a second coder with a generative AI tool to calculate an ICR. Using three previously published studies, I calculated a Krippendorff's alpha for various data types and various codebooks using non-modified versions of five popular generative AI tools. While the tools could code TPC data, most tools did not produce alphas strong enough to replace human coders. While it is premature in most cases to replace human coders with generative AI tools to calculate an ICR, generative AI may prove to be useful to researchers as a means of codebook and unit of analysis refinement prior to human coding.