Examining the Role of Context in Statistical Literacy Outcomes using an Isomorphic Assessment Instrument

Abstract

Background. The central role of statistical literacy has been discussed extensively in the statistics education literature, emphasizing its importance as a learning outcome and in promoting a citizenry capable of interacting with the world in an informed and critical manner. However, little is known about the association between student learning outcomes, context choices (e.g., application domain) in classroom examples, assessment tasks, etc., and students’ perceptions about context choices. This research considered the assessment of contextualized statistical literacy - statistical literacy vis-a-vis contexts with personal relevance or significance to the respondents which in this instance were college students. Specifically, the context of the ongoing COVID-19 pandemic was considered. Two research questions were addressed: (RQ1) Can an isomorphic instrument measure the same underlying construct as the original if all isomorphic items are dependent on relevant contexts? An isomorphic item is identical to a base item in structure (concept, phrasing, as well as distractors) and differs only in the context, continuing to measure the same underlying construct. (RQ2) Do students perform comparably on both these assessments? Our work contributes to the growing literature on assessing and improving people’s statistical literacy vis-à-vis contexts important in their professional and personal lives. Methods. Towards the stated goals, an isomorphic assessment of an existing research-based instrument was developed and piloted in an undergraduate introductory statistics class at a research university. The Basic Literacy in Statistics (BLIS) assessment (Ziegler, 2014) was modified to develop M-BLIS (Modified BLIS). For this work, all non-anchor items were modified to incorporate a topic pertaining to the COVID-19 pandemic. Data from the pilot were analyzed to compare psychometric properties of BLIS and M-BLIS to discuss reliability evidence and to develop a validity argument. Test-takers' responses were also analyzed in relation to various respondent demographics, survey responses, and item characteristics. Findings. The study resulted in two key findings. First, the two assessment instruments had comparable psychometric properties, suggesting that a carefully designed isomorphic assessment can allow for a reliable measurement of statistical literacy in specific contexts. Second, test takers scored lower on M-BLIS than on BLIS, suggesting that context probably matters. A year into the COVID-19 pandemic (as of April 2021), students who were finishing-up a semester of college-level introductory statistics scored lower on a pandemic-specific assessment of statistical literacy as compared to another version with a variety of non-pandemic contexts. Implications. These findings can direct future research towards two key purposes. First, the development of additional isomorphic instruments to measure statistical literacy in various disciplinary or societal contexts, and the investigation of the transfer and cognitive processes behind statistical problem solving to better understand the role of contexts. The instruments can be useful to assess performance of curricular or pedagogical strategies designed to improve statistical literacy. On the teaching end, educators should consider whether incorporating relevant topics into the curriculum may be more beneficial to expose students to relevant topics from a statistical standpoint as well as to improve literacy outcomes.