Significance Testing Principles

• Effect and Sample Size

This handout lists the most commonly used effect sizes, adjustments, and rules of thumb concerning sample size calculation.

• Analysis Tool: The R Project for Statistical Computing

R is a language and environment for statistical computing and graphics. It is a GNU project which is similar to the S language and environment which was developed at Bell Laboratories (formerly AT&T, now Lucent Technologies) by John Chambers and colleagues. R can be considered as a different implementation of S. There are some important differences, but much code written for S runs unaltered under R.

R provides a wide variety of statistical (linear and nonlinear modelling, classical statistical tests, time-series analysis, classification, clustering, …) and graphical techniques, and is highly extensible. The S language is often the vehicle of choice for research in statistical methodology, and R provides an Open Source route to participation in that activity.

• Analysis Tool: Bayes Factor Robustness [Two sample t-test] (Shiny App)

Check how your Bayes factor conclusion depends on the r-scale parameter.

• Analysis Tool: p-Value Analyzer (Shiny App)

This Shiny app implements the p-curve (Simonsohn, Nelson, & Simmons, 2014; see http://www.p-curve.com) in its previous ("app2") and the current version ("app3"), the R-Index and the Test of Insufficient Variance, TIVA (Schimmack, 2014; see http://www.r-index.org/), and tests whether p values are reported correctly.

• When does a significant p-value indicate a true effect? (Shiny App)

When does a significant p-value indicate a true effect?  This app will help with understanding the Positive Predictive Value (PPV) of a p-value.

This app is based on Ioannidis, J. P. A. (2005). Why most published research findings are false. PLoS Medicine, 2(8), e124. http://doi.org/10.1371/journal.pmed.0020124

• Feeling Bayes Factor: Height Difference Between Males and Females (Shiny App)

Can you "see" a group mean difference, just by eyeballing the data? Is your gut feeling aligned to the formal index of evidence, the Bayes factor?

• What does a Bayes factor look like? [The urn model] (Shiny App)

Visualizing the Bayes factor (quantification of evidence supporting a null or altermative hypothesis) using the urn model.

• True Scientific Discoveries: What Research Approach is Most Cost-Effective? (Shiny App)

Use presets or change parameter values manually to explore the cost-effectiveness of different research approaches to unearth true scientific discoveries. For detailed explanation and conceptual background, see LeBel, Campbell, & Loving (in press, JPSP), Table 3. This app is an extension of Zehetleitner and Felix Schönbrodt's (2016) positive predictive value app

• Analysis Tool: Distribution of Cohen's d, p-values, and power curves for an independent two-tailed t-test (Shiny App)

Plot the theoretical p-value distribution and power curve for an independent t-test based on the effect size, sample size, and alpha.

• Analysis Tool: Vovk-Sellke Maximum p-Ratio (Shiny App)

Explore the Vovk-Sellke Maximum p-Ratio, a measure that indicates the maximum diagnosticity of a given p-value. Choose your own p-value to find out how diagnostic it is for your research!