Research Topic: Simulation-Based Inference

What is the historical context for SBI?

Simulation-Based Inference (SBI) is a relatively recent pedagogical approach that aims to improve students’ statistical understanding by deemphasizing manual calculations and allowing more emphasis to be placed on interpretation and communication. The SBI approach only became possible in the 2000s due to widespread access to technology in the statistics classroom. The contemporary SBI movement builds on the work of many statistics educators who were exploring the use of simulations in introductory statistics courses, though George Cobb persuasively argued in 2005 for placing the “Three Rs” of inference at the center of the introductory statistics curriculum: Randomize (data production), Repeat (using simulation), and Reject (see whether data is usual/unusual). Cobb’s description of the traditional introductory statistics curriculum that is structured around the normal distribution as “a Ptolemaic curriculum” would prove prescient: there was a groundswell of interest in SBI over the next decade. Today, there are many textbooks that draw on the tradition of SBI, and research on SBI is commonplace.

What might SBI look like in the classroom?

• Use manipulatives (pieces of paper, index cards) to simulate a null distribution for statistical inference
•  Use manipulatives (coins, spinners) to simulate a null distribution for inference for a single proportion, in particular
• Use applets (Stapplet, Rossman-Chance applet, Statkey) to simulate a null or bootstrap distribution
• Estimate a p-value using a coin tossing simulation
• Find a difference in group means or proportions, use simulation to find resulting p-value
  • Eliminates need for t-statistic (with corresponding degrees of freedom)
• Use bootstrapping methods (and/or inversions of tests of significance and/or estimated standard errors) from a simulated null distribution to create an estimated confidence interval
• Use permutation tests (of the response variable) to simulate a null distribution for inference of two variables
• Compare the strength of evidence for different test statistics using a simulated null distribution

What research on SBI has been done?

Following Cobb’s seminal work, there was a lull in the conversation as statisticians pondered this innovative approach and considered the ramifications of redesigning an established curriculum. In 2011 – 2012, a research team including Nathan Tintle published work and findings on a randomization-based introductory statistics curriculum they had developed and implemented (Tintle et al, 2011; Tintle et al, 2012). Concurrently, a research team in the Educational Psychology department at the University of Minnesota, led by Garfield, delMas and Zieffler, piloted a study using the CATALYST curriculum, which used simulation and chance models to foster deeper understanding of statistical concepts. This curriculum was thought to generalize Cobb’s 3 R’s to instead include: Model, Randomize and Repeat, and Evaluate. In this curriculum, students were taught by making conjectures, testing those conjectures using simulation and modeling technology (Tinkerplots), and discussing those ideas with peers. This study found that simulation-based inference fosters statistical thinking, which was lacking in the traditional curriculum. Furthermore, students appeared to have a more positive attitude towards statistics by the end of the course, and understood fundamental statistical ideas, even those only touched on informally. These studies started to build support for the claim that a curriculum using SBI leads to desirable student outcomes and does not harm students academically (Garfield, delMas & Zieffler, 2012).

By the year 2014, the conversation on SBI had gained full steam. Researchers continued to build support for an SBI curriculum as well as study the impact of SBI on students’ attitudes towards statistics and students’ understanding of specific topics such as p-values and confidence intervals (Chance & McGaughey, 2014; Roy et al, 2014; Swanson & VanderStoep & Tintle, 2014; Tintle et al, 2014). Tintle et al explained how statistical thinking tends to fall into one of two extremist categories: overconfidence or disbelief. SBI can combat these misunderstandings by teaching that statistical and mathematical thinking are different, statistical thinking supports the research process as a whole, and active learning is an effective pedagogical practice (Tintle et al, 2015).

As the years progressed, more research emerged that compared student performance and conceptual understanding in SBI and non-SBI curricula. These studies found promising results for using SBI curriculum to enhance understanding, performance, and retention, as well as level the playing field for students of varying mathematical ability and level of preparation (Maurer & Lock, 2015; Maurer & Lock, 2016; Tintle et al, 2018; Chance & Mendoza & Tintle, 2018). As with any newly developed curriculum, there are still areas for improvement in the SBI curriculum. Case and Jacobbe have started to identify and target topics with which students appear to be struggling in the SBI curriculum so changes can be made accordingly (Case & Jacobbe, 2018). As more research is conducted, and more statistics departments adopt a simulation-based inference approach, the introductory statistics curriculum will continue to be refined to extract the most benefit from this novel approach.

What are some ideas or research questions for starting to explore SBI?

• Study modeling & simulation, model-eliciting activities, and instructional design principles, in isolation
• Determine the student and instructor level variables that explain the most variation in students’ achievable gain
• Does the order of concept and attitude surveys matter
• Does the specific textbook used in an SBI course affect the outcome
• Is it possible to project a learning trajectory of student understanding
• Consider a more diverse demographic for students included in survey and look at the effect of environment and incentives on student performance
• Do students benefit more from learning the theoretical concept immediately following the corresponding topic in the SBI curriculum or collectively at the end of the course
• Do students have a deeper understanding of p-values following an SBI versus non-SBI course

What are some recent SBI textbooks (within the last 5 years)?

Key Articles (as cited throughout)

George Cobb’s description of the traditional introductory statistics curriculum as “Ptolemaic” was originally delivered in a speech at the 2005 United States Conference on Teaching Statistics. A journal article based on this speech was published in 2007 and is the usual citation for Cobb’s “Ptolemaic curriculum” comments.

• Cobb, G. W. (2007). The Introductory Statistics Course: A Ptolemaic Curriculum? Technology Innovations in Statistics Education, 1(1), 16

Early Research on SBI

• Tintle, N., VanderStoep, J., Holmes, V., Quisenberry, B., and Swanson, T. (2011), “Development and Assessment of a Preliminary Randomization-Based Introductory Statistics Curriculum,” Journal of Statistics Education, 19, 1-25
• Tintle, N., Topliff, K., VanderStoep, J., Holmes, V., and Swanson, T. (2012), “Retention of Statistical Concepts in a Preliminary Randomization-Based Introductory Statistics Curriculum,” Statistics Education Research Journal, 11, 21-40.
• Garfield, J., delMas, R., & Zieffler, A. (2012). Developing statistical modelers and thinkers in an introductory, tertiary-level statistics course. ZDM, 44(7), 883–898. https://doi.org/10.1007/s11858-012-0447-5 (Links to an external site.)

Middle Research on SBI

• Chance, B., and McGaughey, K. (2014), ‘‘Impact of a Simulation/Randomization-Based Curriculum on Student Understanding of p-Values and Confidence Intervals,” in Proceedings of the 9th International Conference on Teaching Statistics (Vol. 9). Available at http://iase-web.org/icots/9/proceedings/pdfs/ICOTS9_6B1_CHANCE.pdf.
• Roy, S., Rossman, A., Chance, B., Cobb, G., VanderStoep, J., Tintle, N., and Swanson, T. (2014), “Using Simulation/Randomization to Introduce p-Value in Week 1,” in Proceedings of the 9th International Conference on Teaching Statistics (Vol. 9), pp. 1-6.
• Swanson, T., VanderStoep, J., and Tintle, N. (2014), “Student Attitudes Toward Statistics From a Randomization-Based Curriculum,” in Proceedings of the International Conference on Teaching Statistics (ICOTS 9). Available at http://iase-web.org/icots/9/proceedings/pdfs/ICOTS9_1F1_SWANSON.pdf.
• Tintle, N. L., Rogers, A., Chance, B., Cobb, G., Rossman, A., Roy, S., and Vanderstoep, J. (2014), “Quantitative Evidence for the Use of Simulation and Randomization in the Introductory Statistics Course,” in Proceedings of the 9th International Conference on Teaching Statistics. Available at http://iase-web.org/icots/9/proceedings/pdfs/ICOTS9_8A3_TINTLE.pdf
• Tintle, N., Chance, B., Cobb, G., Roy, S., Swanson, T., & VanderStoep, J. (2015). Combating Anti-Statistical Thinking Using Simulation-Based Methods Throughout the Undergraduate Curriculum. The American Statistician, 69(4), 362–370. https://doi.org/10.1080/00031305.2015.1081619 (Links to an external site.)
• Maurer, K., and Lock, E. (2015). Bootstrapping in the Introductory Statistics Curriculum. Technology Innovations in Statistics Education, 9(1).

Late Research on SBI

• Maurer, K., & Lock, D. (2016). Comparison of Learning Outcomes for Simulation-based and Traditional Inference Curricula in a Designed Educational Experiment. Technology Innovations in Statistics Education, 9(1). https://doi.org/10.5070/T591026161 (Links to an external site.)
• Tintle, N., Clark, J., Fischer, K., Chance, B., Cobb, G., Roy, S., Swanson, T., & VanderStoep, J. (2018). Assessing the Association Between Precourse Metrics of Student Preparation and Student Performance in Introductory Statistics: Results from Early Data on Simulation-Based Inference vs. Nonsimulation-Based Inference. Journal of Statistics Education, 26(2), 103–109. https://doi.org/10.1080/10691898.2018.1473061 (Links to an external site.)
• Chance, B., Mendoza, S., & Tintle, N. (2018). Student gains in conceptual understanding in introductory statistics with and without a curriculum focused on simulation-based inference. In M. A. Sorto, A. White, & L. Guyot (Eds.), Looking back, looking forward. Proceedings of the Tenth International Conference on Teaching Statistics (p. 6). International Statistical Institute.
• Case, C., & Jacobbe, T. (2018). A framework to characterize student difficulties in learning inference from a simulation-based approach. Statistics Education Research Journal, 17(2), 9–29. https://doi.org/10.52041/serj.v17i2.156