Chance News 101

Quotations

“It is easy to lie with statistics. It is hard to tell the truth without it.”

Andrejs Dunkels, cited in Naked Statistics, 2013

Submitted by Margaret Cibes

---

“Recently, I tried ‘treeing out’ (as the decision buffs put it) the choice [my patient] faced. …. When all the possibilities and consequences were penciled out, my decision tree looked more like a bush.”

Atul Gawande in Complications: A Surgeon’s Note on an Imperfect Science, 2002

Submitted by Margaret Cibes

---

From Significance, July 2014:

“Access to large databases does not reduce the need to understand why these particular samples are available.” (original emphasis)

“Who’s afraid of the big black box?”

“[In the early 1970s] I would be called to the bedside of a woman in labour; and instead of asking myself what was the best treatment for her I would find myself asking ‘Who is this woman’s consultant?’ Because all the consultants had their own favoured treatments and responses, whatever I suggested had to fit in which each of them. Nor was it evidence-based decision-making: it was eminence-based decision-making….” (emphasis added)

Dr. Archie Chalmers interviewed in
“We need the public to become better BS detectors”

Note: British Clinician Archie Chalmers created the Cochrane Collaboration in early 1993 to provide medical workers with easily accessible/readable summaries of all clinical trials on various treatments.

Submitted by Margaret Cibes

Forsooth

From Naked Statistics, by Charles Whelan, 2013

“Every fall, several Chicago newspapers and magazines publish a ranking of the ‘best’ high schools in the region. …. Several of the high schools consistently at the top of the rankings are selective enrollment schools …. One of the most important admissions criteria is standardized test scores. So let’s summarize: (1) these schools are being recognized as ‘excellent’ for having students with high test scores; (2) to get into such a school, one must have high test scores. This is the logical equivalent of giving an award to the basketball team for doing such an excellent job of producing tall students.”

“[T]he most commonly stolen car is not necessarily the kind of car that is most likely to be stolen. A high number of Honda Civics are reported stolen because there are a lot of them on the road; the chances that any individual Honda Civic is stolen … might be quite low. In contrast, even if 99 percent of all Ferraris are stolen, Ferrari would not make the ‘most commonly stolen’ list, because there are not that many of them to steal.”

“What became known as Meadow’s Law – the idea that one infant death is a tragedy, two are suspicious and three are murder – is based on the notion that if an event is rare, two or more instances of it in the same family are so improbable that they are unlikely to be the result of chance. Sir Roy [Meadow in 1993] told the jury in one of these cases that there was a one in 73m chance that two of the defendant's babies could have died naturally. He got this figure by squaring 8,500—the chance of a single cot death in a non-smoking middle-class family—as one would square six to get the chance of throwing a double six.” [footnote: “The Probability of Injustice”, Economist, January 22, 2004]

“John Ioannidis, a Greek doctor and epidemiologist, examined forty-nine studies published in three prominent medical journals [footnote: “Contradicted and Initially Stronger Effects in Highly Cited Clinical Research”, JAMA, July 13, 2005] …. Yet one-third of the research was subsequently refuted by later work. …. Dr. Ioannidis estimates that roughly half of the scientific papers published will eventually turn out to be wrong. His research was published in the Journal of the American Medical Association, one of the journals in which the articles he studied had appeared. This does create a certain mind-being irony. If Dr. Ioannidis’s research is correct, then there is a good chance that his research is wrong.” (emphasis added)

Submitted by Margaret Cibes

Atlanta cheating scandal

“Are drastic swings in CRCT scores valid?”
by John Perry and Heather Vogell, The Atlanta Journal-Constitution
Updated July 5, 2011 from original post October 19, 2009

The trial of school employees allegedly involved in a widespread cheating scandal is underway in Georgia with respect to the Criterion-Referenced Competency Tests (CRCT) in four Atlanta elementary schools. Charges stem from student test results that showed “extraordinary gains or drops in scores” between spring of 2008 and 2009. There are about a dozen defendants in this trial; a number of others have accepted plea agreements or received postponements.

In West Manor and Peyton Forest elementary schools, for instance, students went from among the bottom performers statewide to among the best over the course of one year. The odds of making such a leap were less than 1 in a billion.

The Atlanta Journal-Constitution conducted a study of average grade-level scores in reading, math and language arts for students in grades 3 to 5. It found that Peyton Forest School had an average math score that rose from among the lowest in the state for 2008 third-graders to fourth statewide for 2009 fourth-graders (a gain of over 6 standard deviations). It also noted that the fourth-graders had scored at the lowest level on math practice tests administered two months before the actual tests in 2009. (The newspaper’s general methodology is described at the end of the article.)

The study also found that West Manor School’s statewide math ranking rose from 830 for fourth-graders in 2008 to the highest statewide for fifth-graders in 2009. The average score had risen by nearly 90 points year to year (a gain of over 6 standard deviations), compared the statewide average rise of about 15 points. Similarly, the gain in average score from practice test to actual test in 2009 was highly unusual.

A third school’s results showed results in the opposite direction with respect to average writing scores – from the top statewide average score for fourth-graders in 2008 to an extremely low average score for fifth-graders in 2009.

Two common aspects of a statistical analyses of average scores are erasure marks - with an unusual number of answers changed from incorrect to correct - and unexpected patterns of responses - with students getting more hard than easy questions correct. State investigators found strong evidence of the former in the Atlanta schools.

A district spokesman has offered some explanations: some schools might teach their curricula differently, the changes might be random, class sizes might be small, or teacher turnover could be high in Atlanta.

See the [http://media.cmgdigital.com/shared/news/documents/2013/03/29/DOC0070fc4ee16.pdf

formal indictment].  

The newspaper also examined test results for 69,000 schools in 49 states and found “high concentrations of suspect scores” in 196 school districts. See these results, along with a more detailed methodology and a list of study consultants with their feedback here.

Submitted by Margaret Cibes