Solving problems is not enough: Assessing and diagnosing the ways in which students organize statistical concepts

In many statistics courses homework exercises and examinations focus primarily on solving problems. Marks are assigned to students' responses according to the degree to which a problem solution is correct and/or to which a student's procedure employed in the solution is correct. When a statistics course has an applications or data analysis orientation, instructors often find that even though students can solve textbook and examination problems, they are frequently unable to apply probability and statistics to solve "real world" research problems in which judgments have to be made about the technique(s) to be used and in which substantive interpretations of the results of statistical analyses need to be made. The paper reviews several formats of examination questions and assessment procedures which have been used over the years in noncalculus courses in applied statistical methods which focus on data analyses, parameter estimation, and hypothesis testing. Among the types of assessment techniques reviewed are short-answer questions, essay questions, yes-no questions with student-provided justifications, concept-oriented multiple-choice items, masterlist items, analogical reasoning items, graphic inference items, free association tasks, and concept mapping tasks. The paper also reports on a computer-assisted test of knowledge structure called MicroCAM. This test allows students to create on a computer screen a spatial representation of the way in which they perceive key statistical concepts to be linked one to another. The test also permits students to specify the type of relationship which links two or more concepts together. In this way a student's unique knowledge structure is revealed. Implications of these different assessment methods for diagnosing students' learning difficulties and for teaching statistics to mathematically naive students are discussed.