What’s Missing in SIOP’s New AI Assessment Standards?
Last month, the Society for Industrial-Organizational Psychology just released new guidelines for creating Artificially Intelligent assessments for selecting employees. Not surprisingly, the guidelines align well with longstanding scientific principles, and professional practices. While they’re an important contribution overall, they’re disappointing in two ways.
First, the guidelines focus solely on pre-employment assessments. While selection tests are absolutely important, there also are other types of assessments, such as high stakes credentialing (e.g. physicians), that also require especially careful decisions. While I understand that it would have been more effort to broaden the scope of the guidelines, the psychometric and workplace psychology issues are fundamentally the same. Expanding the scope of these guidelines to include other forms of assessments could provide a more comprehensive framework for the use of AI in the workplace. Unfortunately, it follows the same narrow path IO Psychology is notorious for following since its origin in World War I. I suspect that this continues to be an artifact of Civil-Rights-era focus on pre-employment fairness and litigation, rather than a more holistic treatment of our field. Modern IO Psychologists work at all levels of an organization, including the macro strategy, culture, and climate, the middle level with teams and workflows, as well as employee development and promotion. I hope SIOP will author something similar, and with broader coverage of other workplace applications of AI psychometrics.
Secondly, while the authors of these guidelines have done a commendable job in outlining important technical requirements for AI assessments, they don’t appear to be aware of some of the modern advances combining rules-based psychometric AI with computer-science methods. Since 2017, I’ve had excellent success in making sure that computer-science AI algorithms are vetted by Rasch measurement methods (the Many Facet Rasch Model) that automatically make sure the very standards SIOP recommends are automatically met, with explainable methods. Instead, the standard seems to be at a lower level, recommending but not requiring automatic analyses that are grounded in construct maps, so they’re always explainable and unbiased. Perhaps I’ve not done a good job communicating my work in this area, and that may be partially my fault.
I will work on rectifying that, and with some excellent colleagues in Web 3 such as @CryptoTrigs, I will soon demonstrate the utility of this higher standard…and then communicate it more broadly than I have in the past.
