Teaching introductory students how to evaluate evidence

Abstract

There is a lot a statistician can (and should) consider when evaluating evidence, and not all of it can be mastered after a single course. What aspects of evaluating evidence are most important for our introductory students to understand? I’ll provide my answer here, drawing from backgrounds in causal inference and statistics education, and will open this up for more general discussion in the breakout to follow. Suppose we want to evaluate evidence that A causes better outcomes than B. If better outcomes are observed in the A group, I want my students to deeply understand that there are (at least) 3 possible explanations for this:

1. random chance
2. the groups differed at baseline (confounding)
3. A causes better outcomes than B.

Therefore, evaluating evidence for (3) requires first evaluating evidence against (1) and (2). While inference and confounding are certainly not new to introductory courses, I will argue that these should remain as cornerstones in an introductory course, but with modern approaches that focus on understanding and unite under the umbrella of evaluating evidence.