PnB-Th11 - Teaching About Statistical Power: Activity and Evaluation 

Information

This activity, embedded within a course unit on power analysis, walked graduate intro-statistics students through the process of specifying, conducting, and reviewing results of a power analysis for a correlational study. Primary aims were for students to learn that (a) larger sample size increases the probability of detecting a significant finding (or conversely, too small a sample hinders such detection), (b) power analysis allows researchers to estimate the necessary sample size to achieve a desired probability of a significant result, and (c) power analysis requires specifying the magnitude of result (e.g., correlation, mean difference) one expects. The overall power-analysis unit and specific activity were assessed, respectively, via pre-post knowledge test (Snieckus, 2011) and students’ qualitative reports. The test contained five items including definition of power and ways to increase it. The activity began with a research question: What is the correlation in 7th-12th-grade students between academic achievement and self-esteem? The Longitudinal Study of American Youth, with nearly 6,000 cases, was treated as a "population" from which students could draw random samples. The instructor shared Huang's (2011) meta-analysis of self-concept and academic achievement, whose result (average r = .20) provided students an expected correlation to input into a power-calculation website (https://sample-size.net/correlation-sample-size/), along with the desired power (.80, expressed as .20 beta/false-negative rate) and two-tailed significance level (.05). The calculator yielded N = 194 for .80 power to detect r = .20 at p < .05. Each student received a different achievement measure (math, science, or reading, in a given grade) to correlate with self-esteem in 10 random samples (drawn through SPSS) of roughly 200 participants. Students were expected to find significant correlations of roughly .20 in eight of their 10 random samples (i.e., 80% power). On average, each student found seven significant correlations, closely matching expectations from the power analysis (roughly half the correlations were between r = .15-.25, consistent with the meta-analysis). Possible ways to improve the activity include having all students use the same variables so sample sizes and results are consistent across students; using another statistical technique such as a two-sample t-test, which may be easier to grasp; and linking the activity to other ideas on teaching power (several of which appear in the Journal of Statistics and Data Science Education). The demonstration was implemented and assessed at Texas Tech University, a large state-supported Carnegie Research 1 university. The class typically contains around 10 students (mostly master's-level).