Comparison of Learning Outcomes for Simulation-based and Traditional Inference Curricula in a Designed Educational Experiment

Conducting inference is a cornerstone upon which the practice of statistics is based. As such, a large portion of most introductory statistics courses is focused on teaching the fundamentals of statistical inference. The goal of this study is to make a formal comparison of learning outcomes under the traditional and simulation-based inference curricula. A randomized experiment was conducted to administer the two curricula to students in an introductory statistics course. Students of the simulation-based curriculum were found to have improved learning outcomes on topics in statistical inference; however, a clear violation of between-student independence due to group administration of curriculum treatments casts considerable doubt on the statistical significance of these results. A simulation study is used to demonstrate the volatility of Type I error rates in educational studies where classroom level covariance structures exist by comparisons are made on the student level.