The difference between "significant" and "not significant" is not itself statistically significant.


Authors: 
Gelman, A., & Stern, H.
Volume: 
60(4)
Pages: 
4-Jan
Year: 
2006
Publisher: 
The American Statistician
Abstract: 

It is common to summarize statistical comparisons by declarations of statistical<br>significance or non-significance. Here we discuss one problem with such declarations,<br>namely that changes in statistical significance are often not themselves statistically<br>significant. By this, we are not merely making the commonplace observation that<br>any particular threshold is arbitrary - for example, only a small change is required to<br>move an estimate from a 5.1% significance level to 4.9%, thus moving it into statistical<br>significance. Rather, we are pointing out that even large changes in significance levels<br>can correspond to small, non-significant changes in the underlying variables.<br>The error we describe is conceptually different from other oft-cited problems - that<br>statistical significance is not the same as practical importance, that dichotomization into<br>significant and non-significant results encourages the dismissal of observed differences<br>in favor of the usually less interesting null hypothesis of no difference, and that any<br>particular threshold for declaring significance is arbitrary. We are troubled by all of<br>these concerns and do not intend to minimize their importance. Rather, our goal is to<br>bring attention to what we have found is an important but much less discussed point.<br>We illustrate with a theoretical example and two applied examples.

The CAUSE Research Group is supported in part by a member initiative grant from the American Statistical Association’s Section on Statistics and Data Science Education

register