Subjects were given an experimental task in which they had to play the role of a quality-control researcher for a company. They had to consider a hypothetical experiment that involves testing a sample of batteries from a truck load, which may or may not be substandard. In the main experiment, subjects were given information about the prior probability of substandard truck loads (base rate), the degree of variability of battery life, and the mean difference between standard and substandard batteries, all of which are formally relevant to the decision, and they were also told the number of batteries in the truck (population size) that is formally irrelevant. The task was to decide (intuitively) how many batteries to test to achieve a specified error rate using a specified decision rule. In a second study, subjects were given a similar scenario, but asked to rate which pieces of information would be relevant to the decision. Subjects showed themselves to be sensitive to the effects of sample variability and base rate when making intuitive design decisions, though an odd effect of the mean difference factor was observed. There is also clear confirmation of a bias-to-weight sample size by population size as reported in earlier research using different kinds of judgment tasks.
The CAUSE Research Group is supported in part by a member initiative grant from the American Statistical Association’s Section on Statistics and Data Science Education