Sandbox

Quotations

We tolerate the pathologies of quantification — a dry, abstract, mechanical type of knowledge — because the results are so powerful. Numbering things allows tests, comparisons, experiments. Numbers make problems less resonant emotionally but more tractable intellectually. In science, in business and in the more reasonable sectors of government, numbers have won fair and square.
--Gary Wolf

Writing in The data-driven life, New York Times, 26 April 2010

Submitted by Bill Peterson

Odds are, it’s wrong--Part II

[http://www.causeweb.org/wiki/chance/index.php/Chance_News_63#Odds_are.2C_it.27s_wrong A previous Chance News wiki] referred to a Science News article by Tom Siegfried. The article, which focuses on statistics used in the medical field, may be found at [1] and is worth some elaboration; be sure to read the the comments reacting to what Siegfried writes. There you will find mention of circumcision, condoms, defense of statistics in medicine, praise for the author, condemnation of the author, and somehow, reference to Scott Reuben who faked data for Pfizer and Merck.

Siegfried’s main contention is that despite its prevalence in the medical sphere (and dominance elsewhere as well), Fisher’s p-value approach is inadequate and misleading at best. Because of this “p-value mania,” Siegfried quotes two researchers who claim “that in modern [medical] research, false findings may be the majority or even the vast majority of published research claims,” and “There are more false claims made in the medical literature than anybody appreciates,” respectively.

Criticism of p-value is hardly new. Put “criticism of p-value” into a browser and you will get 4,520,000 hits, many of which are more informative than Siegfried’s article. Try [2] as an example.

Discussion

1. To see why critics of p-value say it is the wrong-way round, consider Prob ( brown eyes | Costa Rican) and Prob (Costa Rican | brown eyes). Compare with Prob (data | Null Hypothesis is true) and Prob (Null Hypothesis is true | data). For an interesting illustration of the difference between these conditional probabilities regarding the O.J. Simpson murder case see Steven Strogetz’s NYT article.

2. Critics of p-value say that the above #1 is not strong enough of a criticism because p-value deals not with “data” that actually occurred but with “data at least this extreme.” Why is this a potent criticism?

3. Siegfried rightfully refers to “randomized, controlled clinical trials that test drugs for their ability to cure or their power to harm” as the “gold standard” for medical research. “Such trials assign patients at random to receive either the substance being tested or a placebo.” However, see Judson’s NYT article, Enhancing the Placebo which discusses how non-placebo a placebo can be. What does this do to clinical trials and the gold standard?

4. Siegfried suggests that Bayesian inference is preferable to the frequentist p-value approach of Fisher. If this is so, why is it that p-value approach is so dominant, long after Fisher himself died?