TOWARD THE DEVELOPMENT AND VALIDATION OF THE REASONING ABOUT P-VALUES AND STATISTICAL SIGNIFICANCE SCALE

This paper describes the development and validation of the Reasoning about P-values and Statistical Significance (RPASS) scale. The RPASS was designed to support future research on students' conceptual understanding and misunderstanding of statistical significance and the effects of instructional approaches on this understanding. After unanimous expert content validation and testing, the 27-item RPASS-4 was administered across five introductory courses at California Polytechnic State University (N = 224). Respondents answered 16 of 27 items correctly, on average. Convergent and discriminant validity evidence were sufficient (N = 56). However, internal consistency reliability was low (alpha = .42, N = 224). An optimal subset of 15 items was identified with expected coefficient alpha of .66. However, some of the poorly discriminating items may be indicative of persistent misconceptions. Removing these items from the scale may not be desirable. Implications for future development and research are discussed.