The Statistics Concepts Inventory (SCI) is currently under development at the University of Oklahoma. This paper documents the early stages of assessing the validity, reliability, and discriminatory power of a cognitive assessment instrument for statistics. The evolution of test items on the basis of validity, reliability, and discrimination is included. The instrument has been validated on the basis of content validity through the use of focus groups and faculty surveys. Concurrent validity is measured by correlating SCI scores with course grades. The SCI currently attains concurrent validity for Engineering Statistics courses, but fails to do so for Mathematics Statistics courses. Because the SCI is targeted at Engineering departments, this is a good starting point, but the researchers hope to improve the instrument so that it has applicability across disciplines. The test is shown to be reliable in terms of coefficient alpha for most populations. This paper also describes how specific questions have changed as a result of answer distribution analysis, reliability, discrimination, and focus group comments. Four questions are analyzed in detail: 1) one that was thrown out, 2) one that underwent major revisions, 3) one that required only minor changes, and 4) one that required no changes.
The CAUSE Research Group is supported in part by a member initiative grant from the American Statistical Association’s Section on Statistics and Data Science Education