Chance News 75
Quotations
"I suspect the amygdala did not evolve to store odds ratios and heterogeneity P scores, but when an adverse event has prompted me to review the literature, I come away with a clearer understanding. There’s nothing like a baby free-floating in the abdomen to drive home the lessons from a prospective study of risk factors for uterine rupture. And that clarity of understanding will serve the next at-risk patient I encounter."
Submitted by Steve Simon.
Forsooth
Freshman composition, for example, “does not demand that faculty ask existential questions.” Ditto for courses in “Security and Protective Services,” and “Business Statistics.” These are, she says, “fields of study with fairly definitive answers” and it would be hard to argue that they are “essential to civilization.” Those who teach these and similarly vocational subjects “don’t really need the freedom to ask controversial questions in discussing them.”
Naomi Schaefer Riley, The Faculty Lounges: and Other Reasons Why You Won't Get the College Education You Paid For
as quoted in Vocationalism, academic freedom and tenure, by Stanley Fish, New York Times, 11 July 2011
Submitted by Paul Alper
Discussion of Ariely
A post in Chance News 74 described Paul Ariely's 2008 book Predictably Irrational: The Hidden Forces That Shape Our Decisions as a great summer read [1], while pointing out that it was not written as an academic work. Paul Alper wrote to say that he had occasion to review the book in the context of some related work, and had identified some statistical concerns. As Paul writes:
Ariely enjoys concocting experiments to demonstrate the irrationality. For example, he finds that satisfaction with a product depends on the price paid for the product--for example Bayer aspirin vs. the identical generic. Or, the enticing but utterly misleading “Free gift” will alter a decision. Reviewers loved his book. Nonetheless, there are some serious shortcomings.
- He invariably gives the average value of one group (e.g., satisfaction of Bayer aspirin users) compared to the other group (e.g., satisfaction of generic aspirin users) but he almost never indicates the variability. Averages alone are meaningless.
- Almost never does he state how many subjects are involved in each arm of a study.
- Almost all of his samples are convenience ones, rather than random samples.
- Almost all of his samples are MIT students, but his implicit inference is to the world at large.
- His examples of predictable irrationality appear unfailingly successful leading me to suspect a “file-drawer” issue--experiments which showed nothing in particular or the negative of what he theorizes, are put aside and not counted.
The earlier post also noted that Ariely has a new book, The Upside of Irrationality: The Unexpected Benefits of Defying Logic at Work and at Home. This was reviewed by the New York Times; you can find a link to the review and read Ariely's reaction on his blog here. He notes that that he had consciously adopted a more conversational style in the book, and that this had drawn some criticism from the Times. He invited readers to submit their own opinions on this. Readers come down on both sides, and it is interesting to read the comments. One statistically minded reader wrote:
Of course your [sic] irrationally asking for personal thoughts in comments instead of a (slightly) more accurate poll or a (very) accurate scientific survey.
Submitted by Bill Peterson, based on a message from Paul Alper
The perils of genetic testing
How Bright Promise in Cancer Testing Fell Apart by Gina Kolata, The New York Times, July 7, 2011.
We have seen a lot of advances in genetics recently, and there has been the hope that these would translate into better clinical care. But making the bridge from the laboratory to clinical practice has been much more difficult than expected. A program at Duke, for example, was supposed to identify weak spots in a cancer genome so that drugs could be targeted to that weak spot rather than just trying a range of different drugs in sequence.
But the research at Duke turned out to be wrong. Its gene-based tests proved worthless, and the research behind them was discredited. Ms. Jacobs died a few months after treatment, and her husband and other patients’ relatives are suing Duke.
The problems at Duke are not an isolated problem.
The Duke case came right after two other claims that gave medical researchers pause. Like the Duke case, they used complex analyses to detect patterns of genes or cell proteins. But these were tests that were supposed to find ovarian cancer in patients’ blood. One, OvaSure, was developed by a Yale scientist, Dr. Gil G. Mor, licensed by the university and sold to patients before it was found to be useless.
The other, OvaCheck, was developed by a company, Correlogic, with contributions from scientists from the National Cancer Institute and the Food and Drug Administration. Major commercial labs licensed it and were about to start using it before two statisticians from M. D. Anderson discovered and publicized its faults.
The two statisticians, Keith Baggerly and Kevin Coombes, have made a career of debunking medical claims. In 2004, they (along with another M.D. Anderson statistician, Jeffrey Morris, published a paper that demonstrated serious flaws in the use of proteomic mass spectra to identify early ovarian cancer from normal tissue. The complex method proposed by Petrocoin et al in 2002 was apparently an artefact of equipment drift that would have been prevented if the original researchers had taken simple steps like randomizing the order of analysis of cancer and normal tissues.
Baggerly and Coombes had also found problems with the data supporting the Duke test.
Dr. Baggerly and Dr. Coombes found errors almost immediately. Some seemed careless — moving a row or a column over by one in a giant spreadsheet — while others seemed inexplicable. The Duke team shrugged them off as "clerical errors."
Even though Baggerly and Coombes published a critique in a statistics journal, the Duke team continued to promote their genetic test. It was something else entirely that caused the problems with the Duke test to be treated seriously by the broader research community.
The situation finally grabbed the cancer world’s attention last July, not because of the efforts of Dr. Baggerly and Dr. Coombes, but because a trade publication, The Cancer Letter, reported that the lead researcher, Dr. Potti, had falsified parts of his résumé. He claimed, among other things, that he had been a Rhodes scholar.
Researchers in this area have a new-found sense of humility.
With such huge data sets and complicated analyses, researchers can no longer trust their hunches that a result does — or does not — make sense.
Submitted by Steve Simon
In measuring hunger, quality may be more important than quantity
A Taste Test for Hunger Robert Jensen and Nolan Miller, The New York Times, July 9, 2011.
The traditional measure of global hunger is the number of calories consumed. If you consume less calories than you need, then you are classified as hungry. But this had led to some paradoxical results. There is an alternative way of measuring hunger. You need to
start with a baseline, namely the share of calories people get from the cheapest foods available to them: typically staples like rice, wheat or cassava. We call this the “staple calorie share.” We measure how many calories people get from these low-cost foods and how much they get from more expensive foods like meat. The greater the share of calories they receive from the former, the hungrier they are.
The rationale for this is simple.
Imagine you are a poor consumer in a developing country. You have very little money in your pocket, not enough to afford all the calories you need. And suppose you have only two foods to choose from, rice and meat. Rice is cheap and has a lot of calories, but you prefer the taste of meat. If you spent all your money on meat, you would get too few calories. You might do this every so often, but usually you would spend almost all of your money on rice; when faced with true hunger, taste is a luxury you can’t afford.
But suppose you had a bit more money. You would probably add some meat to your diet, because now you can afford to do so while still getting the calories you need. You might even like meat so much that you start to switch away from rice even if you haven’t quite met your complete calorie needs, as long as you aren’t too far below.
The authors argue that this approach removes some of the variations associated with the traditional calorie count method: some people need more calories than others, for example. They illustrate how the new measure of hunger performs better than the traditional measure in explaining trends in hunger in China from 1991 to 2001.
Submitted by Steve Simon
Questions
1. What are some other ways that you might assess hunger on a global scale?
2. What are some possible pitfalls to the use of this new measure of hunger.
More on US visa lottery
"Plaintiffs Lose Fight Over Visa Lottery"
by Miriam Jordan, The Wall Street Journal, July 15, 2010
Applicants who had been notified that they had won a US visa in the May 2011 lottery drawing have lost a federal court case in which they tried to stop a new drawing of the 2011 lottery. (See Chance News 74[2].)
It turns out that the May sampling process had violated a legal requirement of randomness: "a computer error caused 90% of the 50,000 winners to be selected from the first two days of applications instead of from the entire 30-day registration period."
The court hearing earlier this week focused on the meaning of "random." Lawyers for the plaintiffs argued that they had been randomly chosen and that they didn't know that by filling their applications in the first two days that they would gain any advantage. They also contended that the "outcome was indeed not uniform, but nevertheless still random as required by law."
The State Department argued that the results didn't represent a fair, random selection.
This example reminds me of the 1970 Vietnam War draft lottery issues related to sampling in an inadequately stirred-up jar of birthday capsules. For this first draft lottery since 1940, 366 balls were dumped from a box into a glass jar, from which balls were drawn. The result was that higher numbers (fall birthdays) were somewhat more likely to have been chosen first and, consequently, their holders somewhat more likely to have been drafted into military service. See a 10-minute video of the actual Fall 1969 drawing for the 1970 draft, “CBS news draft lottery nov 1969”, and/or Norton Starr's 1997 article, "Nonrandom Risk: The 1970 Draft Lottery", for the raw data and a statistical discussion of it.
Submitted by Margaret Cibes
Discussion
1. The article does not specify the daily distribution of applications over the 30-day application period.
(a) If the daily distribution of applications had been uniform over the 30 days, what percent of the 50,000 winners would you have expected from a random selection process?
(b) If the distribution of applications had been skewed so that an overwhelming majority of them had been received during the first two days, could the resulting 22,000 "winners" have been expected?
2. Would you consider the May result an "error"? Would you attribute the "error" to a "computer"?
3. Might you have agreed with the plaintiffs that the May outcome was "still random"?
Note: Figures cites this article vary among news reports. See Diversity Visa Lottery 2011 Results for more accurate figures and other information from the U.S. State Department.