Chance News 79
Quotations
"...risk, essentially, is measurable whereas uncertainty is not measurable.
"In Mr. Cain’s case, I think we are dealing with an instance where there is considerable uncertainty."
Submitted by Paul Alper
"Experts have a poor understanding of uncertainty. Usually, this manifests itself in the form of overconfidence: experts underestimate the likelihood that their predictions might be wrong. …. [E]xperts who use terms like “never” and “certain” too often are playing Russian roulette with their reputations."
"I used to be annoyed when the margin of error was high in a forecasting model that I might put together. Now I view it as perhaps the single most important piece of information that a forecaster provides. When we publish a forecast on FiveThirtyEight, I go to great lengths to document the uncertainty attached to it, even if the uncertainty is sufficiently large that the forecast won’t make for punchy headlines."
"Another fundamental error: when you have such little data, you should almost never throw any of it out, and you should be especially wary of doing so when it happens to contradict your hypothesis."
Submitted by Margaret Cibes
Forsooth
Fraud may just be the tip of the iceberg
Fraud Case Seen as a Red Flag for Psychology Research by Benedict Carey, The New York Times, November 2, 2011.
A recently revealed case about fraud may point to a much larger problem.
A well-known psychologist in the Netherlands whose work has been published widely in professional journals falsified data and made up entire experiments, an investigating committee has found. Experts say the case exposes deep flaws in the way science is done in a field, psychology, that has only recently earned a fragile respectability.
The psychologist accused of fraud took advantage of some common practices in the field.
Dr. Stapel was able to operate for so long, the committee said, in large measure because he was “lord of the data,” the only person who saw the experimental evidence that had been gathered (or fabricated). This is a widespread problem in psychology, said Jelte M. Wicherts, a psychologist at the University of Amsterdam. In a recent survey, two-thirds of Dutch research psychologists said they did not make their raw data available for other researchers to see. “This is in violation of ethical rules established in the field,” Dr. Wicherts said.
The field also appears to be rather careless about their statistical analyses.
In an analysis published this year, Dr. Wicherts and Marjan Bakker, also at the University of Amsterdam, searched a random sample of 281 psychology papers for statistical errors. They found that about half of the papers in high-end journals contained some statistical error, and that about 15 percent of all papers had at least one error that changed a reported finding — almost always in opposition to the authors’ hypothesis.
This is not a surprise to psychologists.
Researchers in psychology are certainly aware of the issue. In recent years, some have mocked studies showing correlations between activity on brain images and personality measures as “voodoo” science, and a controversy over statistics erupted in January after The Journal of Personality and Social Psychology accepted a paper purporting to show evidence of extrasensory perception. In cases like these, the authors being challenged are often reluctant to share their raw data. But an analysis of 49 studies appearing Wednesday in the journal PLoS One, by Dr. Wicherts, Dr. Bakker and Dylan Molenaar [available here], found that the more reluctant that scientists were to share their data, the more likely that evidence contradicted their reported findings.
Submitted by Steve Simon
Remark
Andrew Gelman's blog has often considered questions of cheating in science. The following quote from E. J. Wagenmakers, a Dutch professor at Amsterdam University, appeared in a post from September 9 of this year :
Diederik Stapel was not just a productive researcher, but he also made appearances on Dutch TV shows. The scandal is all over the Dutch news. Oh, one of the courses he taught was on something like 'Ethical behavior in research', and one of his papers is about how power corrupts. It doesn’t get much more ironic than this. I should stress that the extent of the fraud is still unclear.
This is perhaps doubly ironic, in that the psychologists have been caught making psychological errors.
Submitted by Paul Alper
Another Remark
"Much of Prof. Stapel's work made it into newspapers in no small part because he delivered scientific evidence for contentions journalists wanted to believe …..”
Other stories include “Diederik Stapel; The Lying Dutchman”, in The Washington Post and “Massive Fraud Uncovered in Work by Social Psychologist”, the latter an article reprinted in the Scientific American, with permission from Nature. Both articles are dated November 1, 2011.
Submitted by Margaret Cibes
Marilyn tackles a dice problem
Ask Marilyn, by Marilyn vos Savant, Parade, 23 October 2011
It has been a while since we've reported on an "Ask Marilyn" story. In the Sunday column referenced above, a reader asks:
I’m a math instructor and I think you’re wrong about this question [originally from Marilyn's July 23 column]: “Say you plan to roll a die 20 times. Which result is more likely: (a) 11111111111111111111; or (b) 66234441536125563152?” You said they’re equally likely because both specify the number for each of the 20 tosses. I agree so far. However, you added, “But let’s say you rolled a die out of my view and then said the results were one of those series. Which is more likely? It’s (b) because the roll has already occurred. It was far more likely to have been that mix than a series of ones.” I disagree. Each of the results is equally likely—or unlikely. This is true even if you are not looking at the result.
Marilyn responds: "My answer was correct. To convince doubting readers, I have, in fact, rolled a die 20 times and noted the result, digit by digit. It was either: (a) 11111111111111111111; or (b) 63335643331622221214. Do you still believe that the two series are equally likely to be what I rolled?"
Many people will remember that the infamous Monty Hall problem first gained national attention after appearing in an "Ask Marilyn" column in 1990. (More nostalgia: In the inaugural issue of Chance News, Laurie Snell described a New Yorker cartoon inspired by Monty's game show, Let's Make a Deal). One important lesson from that discussion was that the host's behavior mattered, and that the problem was not well-defined without a model for how he chooses a door to open.
In that spirit, it might be relevant to consider how the other string of "rolls" was chosen. Marilyn's answer suggests that she was already planning to write down a string of twenty 1s and a string of twenty actual rolls. If you know that is going to happen, then even before the roll has occurred, you might be prepared to guess that the real string is not the one consisting of twenty 1s.
Discussion
- What do you think Marilyn had in mind when she wrote "because the roll has already occurred..."?
- I've just rolled a die twenty times. Which of the following do you think it is: (i) 23164351631546631444; or (ii) 61655214235336553132? What if I had told you in advance that one of these is composed from the exchanges of the cell phone numbers for my family members, skipping the 0s, 7s, 8s and 9s? (What if I tell you that (i) was in fact created this way?)
Submitted by Bill Peterson