Janelle Arjoon, Jennifer Lewis (University of South Florida, USA)
Many instruments developed for research use by the chemical education community are relatively new. Because psychometric evidence dictates the validity of interpretations made from test scores, the motivation for this study was to investigate what “counts” as psychometric evidence within our community. Using a methodology based on concepts described and operationalized in the Standards for Educational and Psychological Testing (AERA, APA & NCME, 1999), instruments published in the Journal of Chemical Education between 2002 and 2011 and follow-up publications reporting the use of these instruments were examined. Findings suggest that our peer review practices value some types of evidence while ignoring the need for others. Observed differences between recommendations from the Standards and our published work support a general conclusion that our community can be encouraged to gather additional types of psychometric evidence and to report such evidence at the level of detail used in research studies.