How does the student untrained in advanced statistics interpret results of research that reports a group difference? In two studies, statistically untrained college students were presented with abstracts or professional associations’ reports and asked for estimates of scores obtained by the original participants in the studies. These estimates were converted to inferred effect sizes and compared with the actual effect sizes. Inferred effect sizes substantially overestimated actual effect sizes for all reports, a phenomenon dubbed the tall-tale effect. The effect was obtained with a variety of reports and statistics. The tall-tale effect could be controlled somewhat with simple changes in wording. This finding suggests a program of research which would better calibrate inferences with those actually obtained in the research.
The CAUSE Research Group is supported in part by a member initiative grant from the American Statistical Association’s Section on Statistics and Data Science Education