The following three probabilities seem crucial when interpreting data, especially in the behavioral sciences:1) the probability that an effect is present in the population, 2) the probability that a replication is significant; and 3) the probability that the effect for a single individual in the population is in the expected direction. In our study, we asked 51 subjects (university students and lecturers in psychology) to estimate these probabilities after reading a short description of a hypothetical experiment with as outcomes only p-value and sample size given. Large variations in estimated probabilities were found. However estimates of the probabilities tended to increase as a positive function of sample size for a fixed p-value. Simulation studies show that , assuming a uniform prior distribution for the parameter, this turns out to be incorrect for all three probabilities.
The CAUSE Research Group is supported in part by a member initiative grant from the American Statistical Association’s Section on Statistics and Data Science Education