Confidence Doesn't Mean Probability: The Case of Brody and Jamie's Colleague

Abstract

Background. Professional associations recommend reporting interval estimates, like confidence intervals, as part of responsible communication of data-based results. The concept of confidence intervals (CIs) comprises a complex set of statistics curricular ideas, and communicating uncertainty associated with CIs requires a robust understanding of interpretations of CIs and confidence levels (C-Lvls). It is hypothesized that difficulties in understanding CIs reported in the literature are due, in part, to the frequentist definition of probability as the long-run behavior of a random process and the fundamental difference in probabilities associated with an estimator and an estimate. One newly uncovered conceptualization of CIs is the Capture/Not Capture conceptualization: students explain the difference in the meanings of the words confident and probability in the interpretation of a CI as being because a calculated CI interval [CI estimate] either captures (with a probability of 1) or does not capture (with a probability of 0) the parameter of interest. Six sub-tasks from a larger study were designed to answer the research question: Do individuals who hold the Capture/Not Capture conceptualization demonstrate knowledge about why the Capture/Not Capture conceptualization is true? Meaning, do participants have knowledge about the difference between a CI estimator and CI estimate. Methods. The talk presents a single case, Brody, from a larger study: how he used six tasks, including the Jamie’s Colleague task, to demonstrate connections needed to understand the Capture/Not Capture conceptualization and its connection to the interpretations of CIs and C-Lvls. Brody, a senior statistics student, engaged in two, one-hour, task-based interviews. The theoretical framework for this case study is a formal concept image for the concept of confidence intervals, derived from statistical definitions and associated curricular concepts. A thematic analysis of the Jamie’s Colleague Task produced several different conceptualizations of CIs and insights on Capture/Not Capture conceptualization. Findings. Brody previously demonstrated knowledge of interpretations of C-Lvls in two ways: the long-run interpretation and the proportion of statistics in a region of the sampling distribution. He interpreted a CI without associating the equivalence of proportion of statistics and probability of selecting a statistic, which continued to his interpretations of C-Lvl, and the capture/not capture analogy. Through the Jamie’s Colleague task, Brody connected his interpretations of C-Lvl and CI by making sense of the coverage probability associated with an CI estimator. These results are foundational for illustrating personal concept images of CIs, which will be the focus of future work. Implications. This case study provides evidence that understanding the difference between an estimator and an estimate and the coverage probability of CI estimators may improve understanding of CI and C-Lvls interpretations. The results are the first to connect the concerns of other researchers about potential problems with confident in the CI interpretation and provide a way to address the confusion. The Jamie’s Colleague task could help students make the distinction in the theoretically complex CI and C-Lvls interpretations. The identification of Brody’s personal concept image of CI provides a foundation for future work in the research of learning trajectories for CIs.