Building justice-informed validity arguments in organizational psychology: Where the evidence is (and isn't)

This review examined what types of validity evidence are presented in recent I-O psychology validation articles and the extent to which these studies incorporate justice-oriented features. We reviewed 63 empirical articles published between 2021 and 2025 in five leading journals that presented new work-relevant instruments or provided validation evidence for existing ones. We used a theory-guided coding protocol to map reported evidence onto the Standards for Educational and Psychological Testing, Kane's interpretation/use argument, and Randall's justice-oriented validity framework. Validation evidence mostly consisted of psychometric analyses that supported domain description, scoring, generalization, and extrapolation inferences. Response-process evidence was uncommon and typically limited to evaluations of item clarity. There was limited attention to the consequences arising from score use. Similarly, justice-relevant evidence focused on statistical comparability across groups, with less emphasis to interest holder involvement, construct representation, structural inequality, or potential harm. These patterns suggest that, in the reviewed articles, psychometric adequacy was prioritized over arguments connecting scores to their meaning and use. We discuss the implications of these findings and outline the challenges of AI-enabled assessments. Stronger validity arguments will need clearer specification of the intended uses of score, greater involvement of affected interest holders, increased focus on respondent interpretation and cultural context, consideration of justice and equity throughout the validation process, and evaluation of the consequences of score use.

As a predominantly quantitative social science, industrial-organizational (I-O) psychology depends heavily on the quality of its measures (Bazzoli and Probst, 2022; Pratt and Bonaccio, 2016) to operationalize latent constructs. As the field becomes increasingly data-driven, those same measures also feed into people analytics, organizational interventions, and employment decisions (Suri and Lakhanpal, 2024).

Because validity is a property of scores (not of a scale or a construct), and it concerns score interpretations and uses (American Educational Research Association et al., 2014), insufficient validity evidence can distort conclusions about constructs, produce misleading inferences, and justify interventions that are not supported by the available evidence (Kane, 2013). Contemporary methodological reviews (e.g., Lambert and Newman, 2023) emphasize stronger construct definition, alignment between definitions and operationalizations, and statistical evaluation of items. Relative to earlier times, these practices reflect meaningful progress. However, more psychometric analyses do not automatically result in well-supported validity arguments. For instance, statistical evidence alone cannot replace response process evidence (Padilla and Benítez, 2014) and consequences of testing, especially for marginalized groups (Kane, 2013; Randall et al., 2022; Sireci, 2021).

This article contributes to the organizational literature on validity and measurement in three interrelated ways. First, we provide a systematic review of studies that developed, revised, or validated I-O-relevant scales across five leading journals to take stock of the kinds of evidence authors report and the parts of the validity argument those pieces of evidence are meant to support. Further, we examine whether and how studies reflect the principles of justice-oriented validity (JAV; Randall et al., 2022) and provide an account of what kind of validity evidence tends to be reported more often and what remains less frequently documented. Second, beyond a descriptive summary of disciplinary practices, our review sheds light on what the field appears to recognize as sufficient evidence to claim that a scale is “validated.” We document the kinds of evidence that authors report, and therefore the kinds of evidence that reviewers and editors appear to reward, which in turn influence what kinds of interpretive claims become normalized in the field. Third, by bringing the JAV lens into a conversation with Kane's framework, this review extends the conversation about validation in I-O psychology beyond psychometric adequacy alone and outlines a conceptual basis to create justice-informed validity arguments.

Anchoring validity in a unitary view, argumentation, and justice frameworks

According to the Standards for Educational and Psychological Testing (henceforth, the Standards; American Educational Research Association et al., 2014), validity is a unitary concept defined as the degree to which evidence and theory support the interpretations of test scores for their intended uses. This unified view links observed test performance to the meaning and use of scores, rather than treating validity as a property of the test itself or the underlying construct. Kane's (2013) argument-based approach to validation introduced the interpretation/use argument (IUA) framework, which structures validation as an evaluation of the chain of inferences and assumptions connecting test responses to the intended interpretations and uses of those same scores. Randall et al. (2022) extended this approach and created the Justice-Oriented Antiracist Validity (JAV) framework, which foregrounds equity in validation work by explicitly attending to construct underrepresentation, item bias, inequitable consequences, and contextualized uses of assessments. Grounded in critical race theory, the JAV approach highlights how ignoring fairness and social context in validation can perpetuate systemic biases and disproportionately harm marginalized groups, leading to unjust decisions in high-stakes settings (Randall et al., 2022; Sireci, 2021).

Toward justice-informed validity arguments in organizations

The IUA (Cook et al., 2015; Kane, 2013) delineates five inferences (i.e., Domain Description, Scoring, Generalization, Extrapolation, and Implications) that connect observed responses to proposed score interpretations and uses. Each inference rests on assumptions that require relevant supporting evidence, although the nature of that evidence varies based on the intended use (Bazzoli and French, 2025). Domain Description concerns the definition and representation of the construct domain, Scoring concerns the rules and models used to convert observed responses into scores, Generalization addresses the consistency of scores across replications, testing conditions, or samples of observations, Extrapolation concerns extending score interpretations to the target domain or relevant external criteria, and Implications concerns the decisions and consequences of testing, including the fairness of decisions.

The JAV framework (Randall, 2021; Randall et al., 2022) extends the IUA by asking how race, power, and structural inequality shape the assumptions underlying each inference. It emphasizes, for example, that the Domain Description inference must interrogate the test developer's positionality and power in defining what is being measured as well as whose knowledge and experiences are represented in (or excluded from) the construct articulation and development process. Bringing the JAV framework into the organizational sciences requires careful effort. The JAV was developed for educational assessments and explicitly investigates how racism and institutional power shape testing for students from historically marginalized communities (Randall, 2021; Randall et al., 2022). Compared to education, measurement in organizations occurs within a more heterogeneous context: scores may be used for theory testing (Cornelissen et al., 2026), organizational diagnosis (Burke, 2023), employee selection and promotion (Guion, 2011), or surveillance (Tursunbayeva et al., 2018), and the consequences of test use as well as the power afforded to those being assessed vary considerably. Moreover, I-O psychology has its own equity/fairness traditions, including research on adverse impact and the diversity–validity dilemma (e.g., Kravitz, 2008; Ployhart and Holtz, 2008). In this tradition, “validity” has generally been understood as evidence of statistical association with future job performance rather than to the full argument concerning score meaning and use. Hence, scholars typically prioritize statistical evidence while giving less attention to how the focal construct and criterion were defined in the first place. In this article, we bring these traditions into conversation.

Taken together, these frameworks serve complementary functions. The Standards establish validity as a unitary argument concerning proposed score interpretations and uses, the IUA outlines the assumptions and type of evidence needed to connect scores and uses, and the JAV framework interrogates how race, power, and structural inequality shape those assumptions. Their integration does not provide a context-neutral checklist or guarantee a complete validity argument. Our review provides guidelines as to what organizational researchers should think about when engaging in validation work, but we recognize that applications must remain sensitive to assessment purpose, institutional context, and the specific forms of marginalization at hand.

Literature search and inclusion/exclusion criteria

Our review covered studies published in issues appearing between January 2021 and December 2025 in five journals that routinely publish scale development and validation research in I-O psychology: Applied Psychology: An International Review, Journal of Applied Psychology, Journal of Business and Psychology, Journal of Occupational and Organizational Psychology, and Personnel Psychology.1 One author manually screened the titles and abstracts of all articles published in these journals during that timeframe to identify potentially eligible studies. The study identification and screening process is summarized in the PRISMA flowchart shown in Figure 1 and the PRIMSA checklist is available in our OSF folder. Articles were eligible for inclusion if they reported original empirical research that introduced a new instrument or provided validation evidence for existing instrument(s). Additionally, the focal construct had to be explicitly situated within employment or organizational contexts. Articles whose main goal was to test a theoretical model, meta-analyses, conceptual papers, commentaries, errata, and editorials were therefore excluded. A list of all included articles is available in our OSF folder (https://osf.io/ce7ux). The review protocol was not preregistered.

Coding protocol

We used a theory-guided coding protocol informed by three complementary frameworks we discussed in Section 1.2. First, the Standards (American Educational Research Association et al., 2014), served as the foundational framework for identification and organization of the different components of validity evidence.2 Second, we relied on the heuristics outlined in the JAV framework to evaluate the extent to which the components of validity addressed issues of power, inequality, or the potential for harm arising from test interpretation or use (Randall et al., 2022). Third, we used the IUA framework (Kane, 2013) to evaluate how the reported validity evidence supported specific inferences between observed scores and intended interpretations and uses. Additionally, we relied on the International Test Commission's (2017) Guidelines for Translating and Adapting Tests to define and standardize codes related to scale translation, cultural adaptation, and multilingual measurement practices. We also drew upon prior empirical reviews of validation practices (e.g., Cizek et al., 2010; Cook et al., 2015; Gotch and French, 2020).

Coding variables

Each article was systematically coded across four broad domains. First, we documented study and scale characteristics to capture the contextual and structural features of each validation effort. This included publication year, journal, sample characteristics (e.g., country, respondent type), stated purpose of the scale, conceptual definition of the construct, response format, and dimensionality. Next, we coded the documented presence of validity evidence categories as defined by the Standards (i.e., 0 = No and 1 = Yes) including evidence based on test content (e.g., domain specification, item generation procedures, and subject matter expert reviews), response processes (e.g., cognitive interviews, think-aloud procedures), internal structure (e.g., exploratory or confirmatory factor analyses, internal consistency reliability, and measurement invariance), relations to other variables (e.g., convergent, discriminant, concurrent, predictive validity evidence), and consequences [e.g., (un)intended uses of scores]. In addition to the binary codes, we recorded the page or table where the evidence appeared and, in many cases, the supporting passage. These materials were used to contextualize coding decisions. Our coding sheet and a blank copy for reuse are available in our OSF folder.

After coding the component or type of validity evidence, we documented whether studies reported evidence relevant to justice-oriented aspects of validity. Because the JAV was introduced as an explicitly antiracist extension of validity theory, we did not require studies to describe their validation efforts using JAV terminology. Instead, we used the framework heuristically to identify reported practices and findings that corresponded to its criteria.3 In adapting it to organizational measurement, we distinguished evidence concerning race or ethnicity from evidence concerning other forms of marginalization. The latter included, for example, gender, disability status, age, religion, socioeconomic position, nationality, and language. We considered these broader forms of evidence justice-oriented extensions of JAV rather than interchangeable with its antiracist core. Importantly, analyses were coded when they corresponded to the JAV heuristics, even when the original authors did not explicitly connect those analyses to structural inequality, power, or harm. Consequently, evidence classified as JAV-relevant should not necessarily be interpreted as evidence that the study adopted a justice-oriented approach. Our coding would indicate that the study reported evidence that could inform a justice-oriented validity argument.

Lastly, validity evidence was evaluated in relation to the IUA framework (Kane, 2013) by mapping the reported evidence onto the five validity inferences discussed in Section 1.2. For studies involving machine-learning or automated scoring systems, we treated the model-generated score or rating as part of the scoring process for coding purposes. When an algorithm replaced or supplemented a human rater, we coded evidence concerning model-human agreement, model consistency, or model-generated scores as relevant to the scoring inference. This decision reflected the logic that automated models can function as “scoring agents” within an assessment system, and therefore require evidence concerning the reliability, consistency, and validity of the scores they generate.

Coding procedure

Coding proceeded in multiple stages to ensure consistency, transparency, and theoretical alignment with the guiding frameworks. We began by coding an initial article to establish shared understanding of the coding manual, refine decision rules, and calibrate interpretations of validity evidence, justice-oriented heuristics, and IUA inferences. For this initial calibration article, all six authors independently provided dichotomous judgments for each category included in the coding sheet (see our OSF folder). Interrater agreement for this calibration exercise was estimated using Fleiss' kappa (κ), which is appropriate for nominal data and extends Cohen's kappa to situations involving more than two raters (Fleiss, 1971). The coefficient was calculated across all coding categories included in the coding sheet. In total, the estimate was based on 36 coding categories, each rated independently by six coders, resulting in 216 coding decisions. Fleiss' kappa indicated substantial agreement during this initial calibration exercise, κ = 0.75, 95% CI [0.66, 0.83], p < 0.001 (see Landis and Koch, 1977).

The initial calibration was followed by two consensus meetings in which discrepancies were discussed, and a further article was coded jointly by all coauthors to further standardize the coding process. Following calibration, the remaining articles were distributed equally among us, with each article independently coded by one coauthor. Questions, ambiguities, and difficult classification decisions were brought to subsequent consensus meetings, during which coding patterns were reviewed, disagreements were resolved, and procedures were further standardized. Thus, final codes reflected consensus decisions rather than majority-rule decisions.

Our coding philosophy emphasized individual decisions and breadth in identifying evidence relevant to validity and justice. Instead of requiring authors to use specific terminology (e.g., IUA, JAV, or the Standards), we coded for the documented presence of evidence and processes that aligned with the frameworks, even when they were implicit. We coded whether evidence was present, not whether it was sufficient, persuasive, or ultimately supportive of a validity claim. This distinction allowed us to systematically document patterns in validation practices without imposing evaluative judgements about the quality or conclusiveness of the studies' validity arguments. We also coded supplemental material when it was referenced in the main article and appeared to contain information that fit into our coding scheme.

Descriptive characteristics

The systematic search yielded 63 articles published across five journals (see Table 1 for descriptive characteristics). The Journal of Business and Psychology contributed the largest share (32%), followed by the Journal of Applied Psychology (25%) and Applied Psychology: An International Review (19%). Samples were drawn primarily from the United States (52%) and consisted mainly of employees or working adults (43%), followed by mixed employee-student samples (37%). Each of the 63 articles examined a unique instrument, with no instrument appearing in more than once across the included studies. Most studies focused on a single instrument, though a small number compared multiple measures simultaneously (e.g., Matthews et al., 2022; Min et al., 2021). Scale development and initial validation were the predominant study objective (65%), with examples including the Career Inaction Scale (D'Huyvetter et al., 2025) and Interpersonal Distrust Scale (Min and Zickar, 2023). The remaining studies (35%) focused on the evaluation of existing instruments, encompassing shortening (e.g., Lagios et al., 2024), invariance testing (e.g., Jang et al., 2022), and format/rescaled comparisons (e.g., Bazzoli and Probst, 2023). Instruments were administered predominantly via self-report Likert or rating scales (79%), followed by a smaller subset using technology-mediated formats such as automated video interviews and game-based assessments (10%; e.g., Ohlms et al., 2025).

Instrument domains were coded into eight categories (see Table 2). Individual (traits and personality, such as conscientiousness, see Silver et al., 2025), interpersonal (relational processes, such as benevolent sexism, see Warren et al., 2023), and work behavior (e.g., employee time theft, see Harold et al., 2022), each containing approximately 19% of scales, followed by organizational context and human resources practices (16%), individual attitudes (10%), wellbeing and states (10%), cognitive abilities (5%), and multiple domains (3%). The latter category included studies that assessed multiple measures covering constructs from different domain categories.

Evidence across IUA inferences

The most common type of evidence in the articles supported generalization and extrapolation, followed by scoring, and then domain description. The implications inference was the least discussed inference (see Table 3 for a summary). Although we organized evidence according to the inference it most directly supported, a given form of evidence could inform more than one inference depending on the interpretation or use claim being evaluated (see Footnote 5).

Domain description

Overall, evidence regarding domain or content description was reported in the majority of the articles (89%, N = 56). Most articles provided content evidence using subject matter expert review and domain mapping. These procedures generally evaluated the relevance, representativeness, and wording of items in relation to the specified construct definition. For example, Gorbatov et al. (2021) administered their items to experts and doctoral students to investigate clarity and content validity and adjusted items based on their feedback. Few articles used qualitative interviews for item development. For instance, Currie and Ehrhart (2025) used qualitative interviews to support their definition of workplace nosiness and create an item pool, and Min and Zickar (2023) used a deductive approach to construct their scale. Two articles specified content validity in relation to translation of a measure: LeBreton et al. (2022) discussed content translation from English to Dutch, and Costantini et al. (2021) discussed translation from English to Italian.

Scoring

Within the Scoring inference, we first consider evidence based on response processes, which was documented in 12 of the reviewed articles (19%). This evidence most often consisted of target respondents' evaluations of item clarity, which was used to revise item wording. Such procedures were coded as response process evidence when they provided information about how members of the intended population understood or engaged with the items. Few articles reported evidence from cognitive interviews or think-aloud procedures (e.g., Lopper et al., 2024; Warren et al., 2023). Herde and Lievens (2023), for example, used a think-aloud procedure to examine the cognitive processes underlying evaluator judgments in speeded assessments.4

Turning to the internal structure evidence that primarily supports scoring, most of the reviewed articles provided evidence of factor structure (n = 53), usually through exploratory or confirmatory factor analytic models, including extensions such as multigroup or multilevel models. Evidence concerning measurement invariance or differential item functioning (DIF) was also coded as supporting scoring when it was used to evaluate whether items or scores functioned similarly across groups. Three articles included machine-learning components in their assessments (Hickman et al., 2022, 2025; Liff et al., 2024). These studies provided evidence relevant to the scoring inference by evaluating the consistency or similarity of model-generated scores with human ratings or other relevant indicators. Few articles provided evidence based on Item Response Theory (e.g., item difficulty, item discrimination, and item fit analyses; see Bazzoli and Probst, 2023 and Min et al., 2021).

Generalization

All the articles, with the exception of Landers et al. (2022) and Smith et al. (2023), provided some evidence of reliability. Most of these articles provided evidence of internal consistency (65%, n = 41), and 25% (n = 16) of the articles provided evidence of test-retest reliability. For instance, Matthews et al. (2022) provided evidence of test-retest reliability for single-item measures across three different timepoints (i.e., 1 day, 2 weeks, and 1 month). Additionally, Liff et al. (2024) and Hickman et al. (2022) provided test-retest reliability for their automatic video interview tools.

When data-driven approaches are used for test development, cross-validation with an independent sample is recommended to support the stability of scoring (see American Educational Research Association et al., 2014). The three articles that used machine learning used either nested k-fold (Hickman et al., 2022, 2025) or 10-fold (Liff et al., 2024) cross-validation in their studies. Additionally, it was common practice to include multiple studies and samples within one article. Some of these samples were drawn from the same population of interest (e.g., Bazzoli and French, 2025; Gajendran et al., 2024). Other articles evaluated their measures using samples drawn from different populations, such as students and working adults (e.g., Gorbatov et al., 2021; Hague et al., 2025). However, in many multi-study articles, additional samples were used to address different components of the validity argument rather than independently test the same assumptions.

Extrapolation

Nearly all the articles (97%, n = 61) provided some evidence to support an extrapolation inference. All the articles, except for Jang et al. (2022), investigated relationships between focal scores and other variables. Most articles provided evidence of predictive (67%, n = 42) and concurrent validity (86%, n = 54). According to the Standards (American Educational Research Association et al., 2014), evidence of predictive validity typically involves a time lag between predictors and outcomes, whereas concurrent validity refers to measures being collected at the same point in time. For example, Herde and Lievens (2023) established that participant scores on speeded simulations predicted a domain-relevant criterion seven months after the initial assessment. However, many of the articles used regression-based analyses on cross-sectional data to support claims of predictive validity (e.g., Jawahar and Mohammed, 2022). Although such models may statistically predict scores on a concurrently measured criterion, they do not involve prediction in the temporal sense described in the Standards. Nevertheless, following our commitment to only evaluate presence/absence of evidence instead of making judgment calls on the quality of such evidence, we deferred to the original authors if it was evident that their intent was to use scores from a focal measure to predict future scores.

In addition to the temporal distinction between predictive and concurrent evidence, many articles also examined incremental validity, which was commonly evaluated by testing whether scores from a newly developed measure accounted for variance in a criterion beyond demographic characteristics, established measures, or conceptually related constructs (see, e.g., Jean et al., 2024).

Implications

Only four articles seemed to provide evidence that could potentially be relevant to the implications inference (Hickman et al., 2025; Jang et al., 2022; Muir et al., 2023; Ohlms et al., 2025). However, it was not always clear that these studies addressed implications in the way Kane (2013) articulates them, namely, as evidence concerning the outcomes and decisions that follow from test use. Much of the evidence presented in these studies appears more directly relevant to other inferences, particularly extrapolation, and in some cases scoring, rather than implications.5

Several studies examined fairness-related concerns that could become relevant to implications if the scores were used to make consequential decisions. For example, Ohlms et al. (2025) raised the possibility that prior video game experience may advantage some test takers on a game-based assessment. Hickman et al. (2025) reported differences across racial groups in automated video interview scores (i.e., Black and African American participants received on average lower scores on general mental ability). These findings may identify potential sources of construct-irrelevant variance or indicate that scores do not function comparably across groups. However, evidence of differential item functioning, subgroup score differences, or other forms of measurement noninvariance does not, by itself, demonstrate consequences of test use. Evidence supporting an implications inference would more directly examine what happens when scores are used for their intended purpose. For a selection or promotion assessment, for example, relevant evidence might concern subgroup differences in pass rates, subsequent job performance, or other outcomes. Indeed, under a stricter interpretation of Kane's framework, it is possible that none of the reviewed studies provided direct evidence concerning the consequences of score use, and the scarcity of implications evidence may reflect the difficulty of evaluating this inference empirically.

Justice-oriented findings

Across the reviewed studies, justice-oriented evidence was limited (n = 19; see Table 4). When such considerations were present, they were most often reflected in evidence about comparability across groups. Among these studies, eight examined race or ethnicity. Seven of these studies examined race or ethnicity in conjunction with one or more other characteristics, most commonly gender (n = 6), followed by neurodivergence (n = 2). Age, sexual orientation, religion, and socioeconomic background were each examined in one study alongside race or ethnicity. Only one study examined race or ethnicity by itself. The remaining 11 studies did not examine race or ethnicity. Within this group, the most frequently considered characteristics were gender (n = 5) and linguistic, national, or cultural context (n = 5). Neurodivergence, religion, educational attainment, and employee category or occupational role were each considered in one study. One study also evaluated prior gaming experience as a potential source of construct-irrelevant variance in a game-based assessment.

Sixteen studies addressed issues related to construct definition, meaning, or interpretation, which can provide evidence of justice-oriented concerns related to construct articulation. The most common pattern (n = 12) involved evaluating whether constructs retained similar meaning across groups, usually through measurement invariance or related analyses. Comparisons were estimated across groups based on disability status, gender, race, ethnicity, language, culture, employee type, and national or cultural context (e.g., Handke et al., 2025; Houle et al., 2022; Jang et al., 2022; Jean et al., 2024; LeBreton et al., 2022; Silver et al., 2025).

A smaller set of seven studies moved beyond statistical invariance to consider whose experiences, values, or social realities the construct might reflect or marginalize (Chen and Bedford, 2022; Hickman et al., 2022, 2025; Jang et al., 2022; Liff et al., 2024; Sakr et al., 2023; Silver et al., 2025). For example, Sakr et al. (2023) developed a scale to assess how employees holding marginalized identities are treated in organizations. These authors centered marginalized employees' experiences and investigated whether associations with workplace and personal outcomes differed across marginalized groups. No study explicitly articulated how the construct was intended to disrupt oppressive narratives, however.

Content-related evidence was identified in three studies, all of which examined whether item language privileged particular linguistic, cultural, or gendered norms (Chen and Bedford, 2022; Jang et al., 2022; Tan et al., 2025). No reviewed study explicitly identified whether items reinforced stereotypes or included items designed to actively disrupt those stereotypes. Two studies documented evidence related to response process aligned with justice-oriented principles (Jang et al., 2022; Silver et al., 2025). For example, Silver et al. (2025) suggested that some conscientiousness items may not be interpreted in the same way by people with and without ADHD. More generally, the reviewed studies rarely examined whether assessment processes themselves may privilege dominant norms or overlook different ways of responding. This pattern suggests that the literature was much more concerned about testing for statistical invariance than on investigating how people may experience or interpret the assessment process itself in different ways.

Other justice-relevant evidence concerned internal structure and relations to other variables. Internal structure evidence was observed in three studies, such as LeBreton et al. (2022), who examined DIF and interpreted some flagged items in terms of cultural or religious meaning. Evidence based on relations to other variables was also limited, appearing in four studies, mostly focusing on whether test-criterion relations generalized across groups. Warren et al. (2023), for example, showed that benevolent sexism in the workplace related to relevant work outcomes for women specifically rather than uniformly across respondents.

Considerations related to potential consequences were observed in six studies, primarily through analyses that aimed to identify whether the assessment might disadvantage certain groups if deployed in organizations. For example, Hickman et al. (2025) evaluated bias across race and gender groups in machine-learning-based automated video interviews designed to assess cognitive ability. As noted in Section 3.2.5 and in Footnote 5, these analyses can identify the potential for inequitable outcomes, though they do not necessarily provide direct evidence of such unequal outcomes.

Our review examined what types of validity evidence is most often reported in recent I-O psychology scale development and validation studies, and what those patterns reveal about the kinds of validity arguments currently privileged in the reviewed journals. Across 63 empirical articles published between 2021 and 2025, we mapped reported evidence onto the Standards, Kane's IUA, and Randall et al.'s JAV framework. Our findings show a clear pattern: validation studies routinely documented evidence related to content, internal structure, reliability, and relations to other variables, but much less often examined response processes, empirical consequences of score use, or justice-oriented concerns. Although recent validation work in the reviewed journals reflects considerable psychometric work, our review points to important gaps and opportunities for a more refined approach to validity in I-O psychology.

Implications

Our findings have implications for how validity is conceptualized and evaluated in I-O psychology research. In the following sections, we discuss these implications for researchers and practitioners.

From psychometric evidence to validity arguments

The pattern of evidence we documented suggests that contemporary I-O scale validation practices may emphasize only some components of the overall validity argument. Psychometric adequacy, particularly internal structure, reliability, and relations to other variables, was privileged over other types of evidence. This pattern is consistent with a recent review of job insecurity measurement (Bazzoli et al., 2025), which found that most scales underwent a limited set of psychometric analyses. More broadly, our findings suggest that validation work was carried out more narrowly than it is conceptualized in the Standards and contemporary validity frameworks.

Our results likewise imply that stronger journal practices may be necessary if I-O psychology is to move toward stronger validity arguments. Because publication patterns help define what counts as sufficient validation, the current overrepresentation of evidence related to scoring, generalization, and extrapolation domains may reflect researcher preferences as well as editorial and reviewer norms. Others have emphasized the need for a more rigorous approach to validation (e.g., Lambert and Newman, 2023), and we reiterate that call and suggest that articulating intended score uses, clarifying which inferences evidence supports and which remain weakly supported, would be meaningful advances toward stronger validation practices in I-O psychology.

Construct articulation and respondent meaning

The overrepresentation of statistical evidence generated after items have been administered may lead to undesirable consequences (Sireci, 2007). Our review indicates that inferences connecting construct definition, respondent interpretation, scoring, and use are often only partially documented. Although content-related procedures were common, they generally involved literature reviews, researcher-generated item pools, and expert evaluation. Direct participation by members of the target population in defining constructs or interpreting items and assessment tasks was much less common. When validation relies predominantly on factor analyses and correlations with similar constructs, there is a risk that constructs become defined post hoc by their empirical covariation patterns rather than by a well-articulated theoretical account of what they represent, for whom, and under what conditions (Sireci, 2007).

We suggest that expert review, factor analysis, internal consistency, and correlations with other constructs may be adequate as preliminary evidence, but they are often insufficient for stronger claims about intended score use. In particular, the limited collection of response process evidence in the reviewed literature indicates a missed opportunity to examine whether respondents interpret items as test developers intended. Even among the 12 studies coded as reporting response process evidence, the depth of engagement varied. Much of this evidence consisted of clarity and readability ratings used to refine wording. Fewer studies examined how respondents comprehended the construct, mapped their experiences onto response options, and arrived at a judgment. Scholarship on response processes has shown that cognitive interviewing and related qualitative methods can help in this regard (Padilla and Benítez, 2014). Although intensive qualitative procedures may be time-consuming, they need not be applied to the full sample. Brief cognitive interviews with small, purposively selected subsamples (e.g., Bazzoli et al., 2021) can be combined with web-based probes embedded in pilot administrations, providing a more scalable way to identify potential differences in item interpretation (Lenzner and Neuert, 2017). Sampling strategies may also focus on groups for whom differences in interpretation are theoretically plausible and potentially more consequential.

Justice across the validation process

Translating insights from the JAV framework into I-O psychology research requires more than a limited set of statistical analyses to support score noninvariance across groups. I-O psychology has a well-established tradition around adverse impact analysis and the diversity–validity dilemma (e.g., Kravitz, 2008; Ployhart and Holtz, 2008), perhaps as a function of the legal frameworks governing selection, which require demonstrating job-relatedness primarily through criterion-related validity evidence and subgroup comparisons on outcomes (U.S. Equal Employment Opportunity Commission, 1979). We are not arguing that I-O psychologists should abandon this framework, but we urge them to interrogate what such an approach leaves unexamined. The JAV framework may be helpful in this regard because it provides guidelines for equity before (i.e., construct definition, assessment design, and criterion selection) as well as after statistical models have been estimated (i.e., consequences of score use). For instance, demonstrating that predictor–criterion relationships hold across groups is valuable, but it says little about who participated in defining the predictor and criterion, whose experiences they represent, or how historical and organizational inequalities may shape access to the opportunities captured by those measures. A justice-oriented approach would also ask whether (i) alternative constructs or criteria might better represent performance, (ii) workers had meaningful input into the assessment process, and (iii) the resulting interpretations and decisions reproduce existing inequities.

Clarify intended score uses

A further pattern concerned slippage between the score uses for which evidence was collected and additional uses. Most studies treated scores primarily as variables for operationalizing constructs and testing theoretical relationships, yet their discussions often argued that those scores could potentially be useful for high-stakes decisions in organizations. In these cases, the additional assumptions were neither disclosed nor evaluated. Such extensions are not inherently inappropriate, but each proposed use requires evidence relevant to the interpretations and decisions (if any) it entails. Evidence supporting the use of scores to operationalize latent variables in a theory testing article does not automatically support their use in personnel decisions (Kane, 2013). For example, evidence of internal structure and correlations with known covariates may support the use of scores for construct operationalization and theory testing, but it does not, by itself, support using those scores to make decisions about employees or applicants. Moving from research to applied uses introduces additional assumptions (e.g., score accuracy, noninvariance across relevant populations and contexts, decision rules, and consequences of classification errors). The type and amount of evidence change as a function of the proposed use. Researchers and practitioners should specify the intended use of the scores a priori, identify the assumptions on which that use depends, and collect evidence addressing these assumptions.

Translation and cultural equivalence

The reviewed literature was dominated by U.S.-based samples (perhaps a byproduct of the journals we included in the search) and self-report formats. This could have likely resulted in limited translation/adaptation work and relatively sparse attention to how respondents from different cultural, linguistic, or social backgrounds interpret items. Translation procedures generally targeted semantic correspondence, whereas empirical analyses occasionally revealed that apparently equivalent language carried different historical or cultural meanings. These cases illustrate why back-translation alone (e.g., Brislin, 1970) may be inadequate and why item-level cultural context should be documented when the construct is defined and response processes are investigated, rather than after measurement noninvariance is detected. More broadly, validity arguments in I-O psychology should be explicit about the possibility that constructs interpretations vary across populations or assessment contexts, including the possibility that the categories through which marginalization is identifies carry different meanings from one setting to another (Karakayali, 2025). Consistent with test adaptation guidance (Hernández et al., 2020; International Test Commission, 2017), organizational researchers should avoid treating translated versions of measures as interchangeable with the originals without documenting adaptation procedures, investigating how target populations interpret their content, and evaluating score invariance empirically. Doing so would improve both interpretive rigor and the credibility of cross-cultural findings.

Limitations and future directions

Although this review provided valuable insights, we acknowledge several limitations. Based on these limitations, we outline several future directions. We organize these considerations around the scope and resolution of the review, the translation of JAV insights across organizational and international contexts, the challenges introduced by AI-enabled assessments, and the need to broaden the evidence supporting organizational validity arguments.

Scope, coverage, and coding

The search strategy was limited to five generalist journals and a 5-year publication period. Although, the overrepresentation of U.S.-based studies may be partly attributable to the composition of our journal sampling frame, it is also consistent with a broader predominance of US samples in psychological science (Arnett, 2008). Restricting the review to English-language publications may also have underrepresented scholarship from non-Anglophone contexts, where approaches such as cultural validity and a focus on score consequences may be prevalent (Slomp et al., 2014; Solano-Flores and Nelson-Barber, 2001). Future reviews could examine a broader range of journals (including regional and specialized outlets, such as the European Journal of Work and Organizational Psychology and International Journal of Selection and Assessment), publication periods, and languages to determine whether the patterns documented in this review generalize beyond mainstream I-O psychology outlets included in this review.

The dichotomous coding scheme presents an additional limitation (Sirriyeh et al., 2012). We coded whether evidence was reported instead of whether it was sufficient or ultimately supportive of the proposed validity claim.6 Consequently, we cannot distinguish between minimal and extensive engagement within a given category. For example, a single internal-consistency coefficient and test-retest reliability across multiple occasions were both classified as reliability evidence. Future reviews would benefit from using a more fine-grained rubric that captures variation in the reported validation practices. For example, a 3-point scale (0 = Missing, 1 = Developing, and 2 = Established) could distinguish between studies that do not address a particular consideration, studies that acknowledge it with limited supporting evidence, and studies that substantively incorporate it into their validation process (Vo, 2024). Such a rubric would permit a more nuanced assessment of validity evidence than our approach and would also reflect the broader principle that social justice is an ongoing process (Koljatic et al., 2021; Randall, 2021).

The nature of the JAV rubric itself presents an additional constraint. This framework was designed to assess documented engagement with racial justice-oriented heuristics rather than to prescribe how such practices should be enacted (Randall et al., 2022) and captures what studies report rather than what they intend. Focusing solely on intent risks overlooking how inequities are reproduced in practice (Oluo, 2018), yet the absence of documented practices should not be read as definitive evidence of their absence from the research process. Coding variability across raters compounds this issue. Shared commitments to equitable practice do not eliminate divergence in interpretation, and differences in positionality may lead raters to different readings of the same material (Milner, 2007).7 Although calibration and consensus procedures were used to address these differences, future reviews could make coder positionality and interpretive disagreement more explicit objects of analysis. Such disagreement may represent coding inconsistency, but it may also reveal meaningful variation in how equity and justice are conceptualized.

Translating the JAV across organizational and international contexts

Our application of JAV represents an initial translation rather than a definitive operationalization for organizational research. The international scope of I-O psychology research presents a related challenge. Applying the JAV framework to studies conducted in Europe, Asia, or Latin America required extending the framework beyond the North American racial context in which it was developed (Randall et al., 2022). At the same time, broadening the JAV to other forms of marginalization risks diluting its explicitly antiracist foundation. In our review, we distinguished evidence concerning race, ethnicity, or racialized processes from justice-oriented extensions concerning other forms of marginalization. Nevertheless, our expansive coding strategy captured a continuum of evidence, ranging from group invariance analyses to more explicit engagement with construct representation and structural inequity.

Future efforts should keep the specificity of the JAV framework while clarifying its relationship to broader justice-oriented validity concerns. One possibility, as we did in this review, is to distinguish among evidence directly addressing race, evidence addressing other forms of structural marginalization, and group comparisons that does not explicitly engage with power or structural inequality. These categories need not be treated as mutually exclusive, especially given the importance of intersectionality, but distinguishing them would prevent any demographic comparison from being treated as inherently justice-oriented. Applications of the JAV framework to organizations should also give greater attention to features of employment systems that differ from many educational assessment contexts, such as employer control over assessment purposes and procedures, applicants' and employees' limited influence over construct definition, and the availability of mechanisms through which assessed individuals can contest score interpretations.

Building validity arguments for AI-generated and algorithmic scores

A subset of studies used AI models or other algorithmic methods to generate or interpret scores. Neither the IUA nor the JAV were developed for assessment contexts in which algorithms generate scores or supplement or replace human judgment. Although we treated model-generated scores and ratings as part of the scoring process, this decision may not fully represent the complexity of AI-enabled assessment systems. The emergence of these systems presents a pressing question: can existing validity frameworks be extended to accommodate Generative AI and algorithmic scores, or do such scores require a distinct framework? We suggest that the core insight of argument-based validity (i.e., proposed interpretations and uses depend on a chain of inferences) provides a useful foundation (see Dorsey and Michaels, 2022). However, given the ease with which anyone can use AI models to build assessments, remaining vigilant to quality measurement process will be critical (e.g., Ho, 2024).

In conventional assessment scenarios, the scoring process is often more directly controlled by researchers: they can define the relevant behavioral indicators, train raters to recognize them using shared standards, and assess the reliability of the resulting ratings (Gorman and Rentsch, 2009; Roch et al., 2012). Conventional self-report measures, by contrast, rely on a well-understood response process in which respondents interpret an item, retrieve and evaluate relevant experiences, map that judgment onto the available response anchors, and select an option that represents their answer numerically (Tourangeau et al., 2000). Deep learning models can learn relationships between patterns in applicant text and expert-defined competency ratings and use those relationships to generate scores on applicant and employee responses (Thompson et al., 2023). Generative AI extends this approach by allowing a pretrained large language model to evaluate a response against natural language instructions or a rubric and return a rating (e.g., Rao and Callison-Burch, 2026). Generative models can be prompted to explain their ratings by identifying evidence and relating it to a given scoring criterion; however, such explanations may be post hoc rationalizations (Turpin et al., 2023) that require independent verification and should not be treated as validity evidence. More broadly, the opacity and evolving behavior of algorithmic scoring systems raise questions about the trust placed in assessment scores and the processes that produce them (Aloisi, 2023). These considerations provide another reason for careful oversight of AI-enabled measurement.

Developing or fine-tuning models with in-house data may appear to reduce some of these concerns, yet it does not necessarily produce a stronger validity argument. Insights from the JAV framework would show that when AI models learn from incumbents or prior hiring decisions, they may encode an organization's existing assumptions about what competence is and how success is judged. The resulting system can create an echo chamber in which historically favored profiles are used to select more applicants who resemble those already employed or rewarded (see e.g., Bazzoli et al., 2026, for a discussion of AI-mediated hiring and socioeconomic status).

Validity frameworks that explicitly deal with the additional inferences to account for AI-enabled assessment, while retaining the central insight that validity concerns the interpretations and uses of scores, are needed. Due to space constraints, we have focused primarily on the scoring inference. A fuller framework would need to address generalization across model versions, extrapolation from model outputs to the constructs and criteria they are intended to represent, and the consequences of using those outputs in organizational decisions, including questions of algorithmic fairness and the fairness of the broader interpretive argument (Huggins-Manley et al., 2022).

Developing an AI-related validity framework that integrates these technical and justice-oriented questions represents an important direction for future psychometric and organizational research. This framework must be ready to evolve as AI is used both for assessment development and by the assessment taker, as constructs may fundamentally change if Generative AI models are allowed to be used to co-produce responses. These uses raise questions about construct drift and model transparency, as examples, and force careful thinking of how claims and inferences may be modified to account for these changes. It may be noteworthy that a revised version of the Standards will be released in 2027, which will likely include guidance on the use of AI in the entire assessment and testing process, and will be a starting point for many to understand how validity in different AI contexts needs to evolve. Generative AI may make it easier to produce and score assessments, but perhaps now, more than ever, we need to lean on sound validity principles and expert judgement to ensure high standards for validation.

Broadening the evidence base for validity arguments in I-O psychology

Beyond these challenges, the findings identify several types of validity evidence that warrant greater attention. First, validity begins before data collection through careful definition and alignment between theory and items (Sireci, 2007). Future studies could examine how organizational constructs are defined, whose perspectives contribute to those definitions, which experiences are treated as representative, and how construct underrepresentation enters the measurement process during item generation.

Second, the relative absence of response process evidence suggests an important methodological agenda. Future validation studies would benefit from more routine use of cognitive interviews, think-aloud protocols, verbal probing, and related methods that examine how respondents interpret items and construct their answers. These approaches are especially relevant when instruments are translated, adapted, shortened, or administered to populations whose lived experiences may differ from those represented during the original development process. Related to this point, our findings showed considerable reliance on expert review, but much less participation of people that will take the assessment. Expertise was typically concentrated among researchers, graduate students, or other professionals, while employees and applicants were more often asked to evaluate item clarity after the construct domain had already been defined. Research in other domains indicates that involving affected communities throughout item generation, content validation, and cognitive testing can improve the clarity and relevance of measures (Thompson et al., 2020).

Third, greater emphasis on the implications of score use is needed. The evidence needed to support this inference will depend on the intended use of the scores. For a measure developed primarily for theory testing, relevant evidence may include whether score interpretations treat one population's experiences as normative, reinforce individualizing or deficit-based explanations of organizational issues, or direct subsequent scholarship toward certain experiences while leaving others underexamined. For a pre-employment assessment, by contrast, implications evidence would more directly concern selection rates, the distribution of classification errors, applicant experiences, and whether the resulting decisions produce inequitable outcomes. These arguments build on ideas introduced in other fields to treat social impact and equity as integral components of validity (e.g., Kinnear et al., 2024). Across uses, researchers should specify the anticipated consequences of score interpretation and use, identify the stakeholders likely to experience those consequences, and evaluate whether organizations recognize and respond to inequitable effects once they emerge.

This review showed that I-O psychologists have made meaningful progress in scale development and validation, particularly in documenting evidence related to content, internal structure, reliability, and relations to other variables. At the same time, our findings suggest that many validation efforts continue to emphasize psychometric adequacy more than the full interpretive argument linking scores to meaning and use. Response process evidence, investigation of testing implications, and justice-oriented evidence remain comparatively rare, even though these forms of evidence are essential for evaluating whether organizational measures function as intended across people, contexts, and uses. Building stronger validity arguments in I-O psychology will require expanding validation practice beyond those conventional pieces of evidence. Researchers, reviewers, editors, and practitioners should attend more explicitly to how constructs are defined, how respondents interpret items and tasks, how scores are used, and whether those uses reproduce or mitigate inequity. Advancing organizational measurement requires preserving the field's psychometric rigor while extending that rigor to areas that are currently underexamined.