By Robert Padgett (Butler University)
Information
The aptly named confusion matrix is the tool used to summarize the results of classification procedures that are widely used in the health, behavioral, and data sciences. In the health sciences the tool is used to evaluate the accuracy and efficiency of a diagnostic test, in psychology it is used to summarize the results of a signal detection task, and the method is the primary tool for assessing a prediction outcome of a machine learning algorithm. For students new to this technique, many terms like sensitivity, specificity, precision, and response bias, can cause, in a word, confusion. A desire to help students learn such concepts while incorporating important lessons about implicit bias in the science classroom has led to a recent revival (e.g., Brown-Kramer & Hillard, 2021; Padgett & Morris, 2023) of interest in the Police Officer's dilemma task. We updated our version of the task so that it could be completed in 50-minute standard class using the students’ own laptops. In the task, participants quickly classify 80 photos depicting individuals holding a gun or some innocuous object. Students receive immediate feedback on the classification problem in a confusion matrix but not about classification differences by the depicted person's race. In the presentation I will demonstrate how instructors can use the task in their own classroom and how it helps students understand confusion matrix assessment in ways to which students can relate. Most importantly, I will show how the results, when presented by target race, reveal the impact of implicit bias and how recognizing that fact helps students better understand concepts like positive predictive value and response bias. To assess the effectiveness of the activity, we include a Canvas based assessment of student mastery of the calculation and interpretation of key confusion matrix measures. The results from the assessment will be shared in the presentation. A discussion facilitation guide will be provided to help instructors transform the task from just a lesson in classification into an important DEI discussion. We have used this demo in a variety of classes at a liberal arts 4-year degree granting institution. It is useful in any class where the concept of a confusion matrix is discussed including: a general science core course open to all university students, students enrolled in a Beginning to Advanced level Statistics course, as well as a typical Introduction to Data Science course.