Automatic Rater Generation

Automatic Item Generation (AIG) is a set of methods for creating large, unique item banks for use in psychometric assessments. In my prior work helping Dr. Robert Cialdini create his “Cialdini Influence Assessment”, AIG was invaluable in helping make sure we had very large item banks that are resistant to misappropriation, when administered in a Computer-Adaptive Test. Historically, the AIG item banks like the one I created for Cialdini were developed using rules-based methods but newer approaches incorporate machine learning methods.

But the psychometric AI scene has changed dramatically since December, 2022. The advent of large language models such as ChatGPT offers a new class of psychometrics that is less focused on automating the creation of items, and instead focuses on engineering synthetic raters.

There are several papers where psychologists have successfully shown that GPT-3 and Chat-GPT can emulate real human personality, and cognition. I suggest that this posits a new class of psychometric AI I’m calling “Automatic Rater Generation” (ARG) involves creating prompts that set up the context for each latent trait and are anchored in construct maps (theories about what manifest variables are indicative of a particular level of a latent trait). This allows for the creation of synthetic, on-demand ratings of unobtrusive behavior, often through "digital exhaust". When coupled with trustworthy rules-based Psychometric AI that serves as a "quality control" for the synthetic ratings, ARG has the potential to be a powerful tool for individual, team, and organizational assessment, feedback, and risk mitigation.

In conclusion, AIG has traditionally been used to create item banks for psychometric assessments. However, the advent of large language models and the concept of ARG offers a new approach that focuses on engineering synthetic raters and creating synthetic, on-demand ratings of unobtrusive behavior. When coupled with trustworthy rules-based Psychometric AI, ARG has the potential to be a powerful tool for individual, team, and organizational assessment, feedback, and risk mitigation.