Sandbox: Difference between revisions

From ChanceWiki
Jump to navigation Jump to search
Line 1: Line 1:
==Odds are, it’s wrong--Part II==
==Deception and waste of time==
An entry in [http://www.causeweb.org/wiki/chance/index.php/Chance_News_63#Odds_are.2C_it.27s_wrong Chance News 63] presented a Science News article by Tom Siegfried.  The article, which focuses on statistics used in the medical field, may be found [http://www.sciencenews.org/view/feature/id/57091/title/Odds_Are,_Its_Wrong here] and is worth some elaboration; be sure to read the [http://www.sciencenews.org/view/feature/id/57091/title/Odds_Are,_Its_Wrong#comment_editorcomments the comments reacting to what Siegfried writes.]  There you will find mention of circumcision, condoms, defense of statistics in medicine, praise for the author, condemnation of the author--and somehow, reference to Scott Reuben, who faked data for Pfizer and Merck (see [http://www.causeweb.org/wiki/chance/index.php/Chance_News_45#Serious_Medical_Fraud Serious medical fraud] in Chance News 45).


Siegfried’s main contention is that despite its prevalence in the medical sphere (and dominance elsewhere as well), Fisher’s p-value approach is inadequate and misleading at best.  Because of  this “p-value mania,” Siegfried quotes two researchers who claim “that in modern [medical] research, false findings may be the majority or even the vast majority of published research claims,” and “There are more false claims made in the medical literature than anybody appreciates,” respectively.
[http://www.causeweb.org/wiki/chance/index.php/Chance_News_64#If_you_take_away_my_time_for_your_research.2C_you_owe_me_ten_bucks Steven Smith’s wiki] is worth expanding a bit.  Deception in psychology is quite common and in fact, there exists an entire book devoted to the subject, Illusions of Reality: A History of Deception in Social Psychology, by James H. Korn,.  “Stanley Milgram [famous for obedience studies] used the term technical illusions because he thought the word deception had a negative moral bias”--italics in the original.  Most people outside of the psychology realm recognize a convenient euphemism when they see one.


Criticism of p-value is hardly new.  Put “criticism of p-value” into a browser and you will get 4,520,000 hits, many of which are more informative than Siegfried’s article.  Try [http://ije.oxfordjournals.org/cgi/content/full/32/5/699 The P-value, devalued] from the ''International Journal of Epidemiology'' as an example.
However, the researchers who so annoyed Andrew Gelman are not psychologists but business school assistant professors, who, unfortunately like their psychologist colleagues, have to publish and seek research which can be done inexpensively.  The one thing that sets this research apart is that the duped individuals were faculty members rather than that customary captive, victimized class known as convenient undergraduates.
  [http://www.stat.columbia.edu/~cook/movabletype/archives/2010/05/63000_worth_of.html Nevertheless, Gelman’s initial reaction] was to say “$63,000 worth of abusive research…or just a waste of time.” [http://www.stat.columbia.edu/~cook/movabletype/archives/2010/05/another_update.html He later modified his views] but perhaps he should not have.  The lay public is all too familiar with the “Lies, damned lies, and statistics” apocryphally attributed to Twain and Disraeli.  Almost as well known is “If you torture the data enough, it will confess to anything.”  Perhaps an even more serious condemnation of the use of statistics is that many studies which utilize statistics to justify their existence are just not worth undertaking despite their titillation value and low p-values.  Intercessory prayer, lucky charms and extrasensory perception come readily to mind.  Unfortunately, just these kinds of investigations resonate with journalists.  


'''Discussion'''
Discussion


1.  To see why critics of p-value say it is the wrong-way round, consider
1.  Consider the following University of Michigan research as an [http://www.sciencemag.org/cgi/content/abstract/328/5979/709 illustration of a deception study].  It is entitled “Washing Away Postdecisional Dissonance.”  The technical illusion in this paper had to do with preference for a product--first, music CDs (40 undergraduates) and then, jam jars (85 undergraduates)-- when in fact the real interest was in hand washing afterwards to determine its effect on regret.  Why did the Wall Street Journal choose to comment on it?  Estimate the cost of doing the study. If inference to a larger population is desired, what would be the relevant larger population?
Prob ( brown eyes | Costa Rican) and Prob (Costa Rican | brown eyes).  Compare with
Prob (data | Null Hypothesis is true) and Prob (Null Hypothesis is true | data).  For an interesting illustration of the difference between these conditional probabilities  regarding the O.J. Simpson murder case see [http://opinionator.blogs.nytimes.com/2010/04/25/chances-are/?scp=1&sq=strogatz%20oj%20simpson&st=cse Steven Strogatz’s NYT article] (25 April 2010).


2. Critics of p-value say that the above #1 is not strong enough of a criticism because p-value deals not with “data” that actually occurred but with “data at least this extreme.”  Why is this a potent criticism?
2. The hand washing study begins with the statement: “Hand washing removes more than dirt--it also removes the guilt of past misdeeds, weakens the urge to engage in compensatory behavior, and attenuates the impact of disgust on moral judgment.”  Based on the music CDs and the jam jars, it concludes with hand washing “can also cleanse us from traces of past decisions, reducing the need to justify them.”  How would you set up a different deceptive experiment to show whether or not this is true?


3.  Siegfried rightfully refers to “randomized, controlled clinical trials that test drugs for their ability to cure or their power to harm” as the “gold standard” for medical research.  “Such trials assign patients at random to receive either the substance being tested or a placebo.” However, see [http://opinionator.blogs.nytimes.com/2010/05/03/enhancing-the-placebo/ Judson’s NYT article, Enhancing the Placebo] (3 May 2010), which discusses how non-placebo a placebo can be.  What does this do to clinical trials and the gold standard?
3.  Quite apart from dubious statistics, deception can be a dangerous endeavor as [http://www.usatoday.com/news/washington/2010-01-26-senator-office-landrieu_N.htm James O'Keefe] might attest to.  After his initial success with duping ACORN, he overreached when he tried “to tamper with Democratic Sen. Mary Landrieu’s office phones” by posing as a telephone worker.  More germane to this discussion, see [http://findarticles.com/p/articles/mi_m3190/is_6_36/ai_82804057/ the case of Francis Flynn.]  He wrote to 240 New York restaurants “claiming to have contracted food poisoning while dining at their establishments” in order to “to help collect data for a research study he had developed to determine how restaurateurs responded to complaints.”  Eventually, when his deception was found out, 10 of the restaurants “filed a $100 million class-action lawsuit against Flynn and the school [Columbia University], claiming libel and emotional distress.”  That was about ten years ago and in spite of this, he has now been promoted and has moved to another coast.
 
4.  There is a spectrum: explanation, euphemism, deception, fraud.  For each of the following, justify what category is applicable to these oft-seen advertisements.
a. “Absolutely free.  Shipping and handling charges may apply.”
b. “Up to 30% off.”
c. “For your convenience, dinner that night is not included.”
d. “The illustration shown on the cereal box is enlarged to better display the contents.”
e. “Taxes and fees are extra.”
f. “Premium quality.”
g. “No entrance or sign-up fee.”
h. “Limited supply only.”


4.  Siegfried suggests that Bayesian inference is preferable to the frequentist p-value approach of Fisher.  If this is so, why is it that p-value approach is so dominant, long after Fisher himself died?


Submitted by Paul Alper
Submitted by Paul Alper

Revision as of 20:29, 28 May 2010

Deception and waste of time

Steven Smith’s wiki is worth expanding a bit. Deception in psychology is quite common and in fact, there exists an entire book devoted to the subject, Illusions of Reality: A History of Deception in Social Psychology, by James H. Korn,. “Stanley Milgram [famous for obedience studies] used the term technical illusions because he thought the word deception had a negative moral bias”--italics in the original. Most people outside of the psychology realm recognize a convenient euphemism when they see one.

However, the researchers who so annoyed Andrew Gelman are not psychologists but business school assistant professors, who, unfortunately like their psychologist colleagues, have to publish and seek research which can be done inexpensively. The one thing that sets this research apart is that the duped individuals were faculty members rather than that customary captive, victimized class known as convenient undergraduates. 

Nevertheless, Gelman’s initial reaction was to say “$63,000 worth of abusive research…or just a waste of time.”  He later modified his views but perhaps he should not have.  The lay public is all too familiar with the “Lies, damned lies, and statistics” apocryphally attributed to Twain and Disraeli.  Almost as well known is “If you torture the data enough, it will confess to anything.”   Perhaps an even more serious condemnation of the use of statistics is that many studies which utilize statistics to justify their existence are just not worth undertaking despite their titillation value and low p-values.  Intercessory prayer, lucky charms and extrasensory perception come readily to mind.  Unfortunately, just these kinds of investigations resonate with journalists.

Discussion

1. Consider the following University of Michigan research as an illustration of a deception study. It is entitled “Washing Away Postdecisional Dissonance.” The technical illusion in this paper had to do with preference for a product--first, music CDs (40 undergraduates) and then, jam jars (85 undergraduates)-- when in fact the real interest was in hand washing afterwards to determine its effect on regret. Why did the Wall Street Journal choose to comment on it? Estimate the cost of doing the study. If inference to a larger population is desired, what would be the relevant larger population?

2. The hand washing study begins with the statement: “Hand washing removes more than dirt--it also removes the guilt of past misdeeds, weakens the urge to engage in compensatory behavior, and attenuates the impact of disgust on moral judgment.” Based on the music CDs and the jam jars, it concludes with hand washing “can also cleanse us from traces of past decisions, reducing the need to justify them.” How would you set up a different deceptive experiment to show whether or not this is true?

3. Quite apart from dubious statistics, deception can be a dangerous endeavor as James O'Keefe might attest to. After his initial success with duping ACORN, he overreached when he tried “to tamper with Democratic Sen. Mary Landrieu’s office phones” by posing as a telephone worker. More germane to this discussion, see the case of Francis Flynn. He wrote to 240 New York restaurants “claiming to have contracted food poisoning while dining at their establishments” in order to “to help collect data for a research study he had developed to determine how restaurateurs responded to complaints.” Eventually, when his deception was found out, 10 of the restaurants “filed a $100 million class-action lawsuit against Flynn and the school [Columbia University], claiming libel and emotional distress.” That was about ten years ago and in spite of this, he has now been promoted and has moved to another coast.

4. There is a spectrum: explanation, euphemism, deception, fraud. For each of the following, justify what category is applicable to these oft-seen advertisements. a. “Absolutely free. Shipping and handling charges may apply.” b. “Up to 30% off.” c. “For your convenience, dinner that night is not included.” d. “The illustration shown on the cereal box is enlarged to better display the contents.” e. “Taxes and fees are extra.” f. “Premium quality.” g. “No entrance or sign-up fee.” h. “Limited supply only.”


Submitted by Paul Alper

Retrieved from "https://www.causeweb.org/wiki/chance/index.php?title=Sandbox&oldid=10725"