It is common to summarize statistical comparisons by declarations of statistical<br>significance or non-significance. Here we discuss one problem with such declarations,<br>namely that changes in statistical significance are often not themselves statistically<br>significant. By this, we are not merely making the commonplace observation that<br>any particular threshold is arbitrary - for example, only a small change is required to<br>move an estimate from a 5.1% significance level to 4.9%, thus moving it into statistical<br>significance. Rather, we are pointing out that even large changes in significance levels<br>can correspond to small, non-significant changes in the underlying variables.<br>The error we describe is conceptually different from other oft-cited problems - that<br>statistical significance is not the same as practical importance, that dichotomization into<br>significant and non-significant results encourages the dismissal of observed differences<br>in favor of the usually less interesting null hypothesis of no difference, and that any<br>particular threshold for declaring significance is arbitrary. We are troubled by all of<br>these concerns and do not intend to minimize their importance. Rather, our goal is to<br>bring attention to what we have found is an important but much less discussed point.<br>We illustrate with a theoretical example and two applied examples.
The CAUSE Research Group is supported in part by a member initiative grant from the American Statistical Association’s Section on Statistics and Data Science Education