Hello Everyone,
It has been a joy to read about of the great data sets and activities! I did not have time to respond yesterday, but here goes now.
I've got an introductory class with 26 students, and we meet all the time in a computer classroom. Inspired by the Mosaic Project folks, we teach with R in the R Studio Server environment. Our approach to simulation-based inference is about 60% if of the way from traditional inference towards Locke5/Intro to Statistical Investigations.
We do some simulation-based inference very early in the course -- binomial stuff, on the first day of class, in fact -- and we try again to work it in early with the chi-square test for the relationship between two factor variables. We're in that unit now, so students have played a couple of times with a "slow-simulation" app to get an idea of the null distribution of the chi-square statistic; subsequently they have been exposed to the standard chi-square test where the null distribution is approximated by a chi-square density curve. They are told that it was quite a godsend for Mr. Pearson to have stumbled on such a family of approximating curves, because Mr. Pearson had no access whatsoever to computing machinery.
Groundhog Day began with a little come-to-Jesus chat about the first data analysis report, which the students tackled over the weekend. The project involved looking at some data from a Current Population Survey and to investigate the relationship between hourly wages and such factors as sex, union membership status, race, etc. Apparently there is a rule at my College that during Greek Rush Week critical thinking is forbidden, including the act of determining the type of variables involved in your research question prior to choosing your analytical tools. Accordingly about a third of the students had attempted to make bar charts and cross-tables to investigate, for example, whether men or women earn more, even though wage is a numerical variable. So we cleared that up, I hope.
Harrumph. On with the intended show.
Today's plan is to revisit simulation one more time, in a situation where you really need it (rather small number of observations). I bring up the "ledge-jump" data (#59 in the classic Handbook of Small Data Sets). a social psychologist studied 21 incidents in Britain involving a person threatening to leap from the ledge of a building or other high structure. The idea was to see what factors might affect the behavior of the crowd that gathers in the street below.
To forestall morbid thoughts among students I get up on a nearby table and assure them through role-play that nobody really gets hurt: the fire truck comes right away and the firemen teeter back and forth with their big yellow trampoline, your rabbi, psychiatrist and spouse are phoned and within minutes they are leaning out of one nearby windows, soothing words are spoken, sage advice is given, hope is restored. Eventually they talk you back inside.
But in the meantime a crowd has gathered. Sometimes they wait more or less in silence, appropriately mindful of the seriousness of the situation playing out above. Sometimes, though, they begin to bait the would-be jumper (muster up Cockney accent): "Go on, jump will ya?"
In the 21 incidents under study, the following was found:
weather/crowd Baiting Polite
cool 2 7
warm 8 4
Discussion, with students ( me still on table):
Me: Somebody could say that, for one reason or another, a crowd is more likely to bait in warm weather. Others might say that the outcome of these 21 incidents had nothing to do with weather, but is just the result of random variation in other factors -- above and beyond the weather at each particular incident. Can you think of any other things besides the weather that might affect crowd behavior?
Student: How many people show up to watch. The more that show up, the more likely there jerk who will yell "jump' and get the rest of 'em started.
Student: How long they have to wait. Maybe if they stand around a long time they'll get impatient.
Student (looking right at me, still on table): What about the dorkiness of the would-be jumper?
Me: Uh, maybe. Gee, thanks, Zach.
I get down and we work it out on the blackboard: if weather has nothing to do with crowd behavior, then our best guess based on the data is that in each incident, regardless of weather, there is a 10/21 chance for the crowd to bait. The chi-square test function that the class uses has built-in provisions for simulation, of three possible types:
"random" make sense when the observed data are a random ample a larger population, and chance comes into play just in the matter of who gets into the sample. For example, if you randomly sample people and ask their sex and where they prefer to sit a in classroom (front, middle or back), then chance is not in how a fixed person will respond but in whether or not that person gets into the sample.
"double-fixed" (corresponding to the way simulation is done in R's chisq.test(), and probably in many other software systems as well), appears to be ideally suited for randomized experiment in which the Null hypothesis imagines that a subject's response is the same regardless of which treatment group one is placed into. In that case a different random assignment of subjects to treatment groups might result in a different table, but the row and column sums would be the same as for the table we observe in the actual experiment.
"fixed" seems to be the right thing for the ledge-jump situation, if we assume that the 21 incidents weren't sampled randomly out of some larger population, that they were the only 21 incidents that occurred under the period of study in the region under study. In that case the weather at the time of each incident simply was it was, and chance comes into the production of the observed table through random variation in all other factors (conditional upon the weather).
So we do simulation with the "fixed" option. Everybody's P-value comes in around 5%, so we decide that don't have overwhelming evidence that weather and crowd behavior are related.
Now I come round to my questions.
When we teach inference though simulation, we don't want it to become another "black box" for students. We want them to see that the simulation method generates simulated data that reasonably could occur if the study were to be conducted in a hypothetical world where the Null hypothesis is definitely true. Hence we the simulation method has to model, quite transparently, the role that we think chance played -- if the Null is true -- in giving us the data we actually see.
But there appears to be controversy among statisticians as to which simulation method is best to use for contingency tables. (See e.g., Agresti, Categorical Data Analysis Third Edition section 3.5.6). I suppose that sometimes it's possible for a particular
simulation method not to model the role of chance very well, but to possess superior statistical properties nonetheless, maybe even be the state-of-the art method. (This situation seems to occur also in bootstrap hypothesis testing, where the more preferred
re-sampling method is rather more difficult to justify intuitively than is the "naive" re-sampling method.)
Where does that leave us in teaching? Do we stick with simulation methods that model the role of chance intuitively? In that case we may get rather far into the weeds (after all, three options for simulation in the two-factor is alot for students to handle so early in the course), and we also end up, from time to time, using methods that won't be recommended in data analysis applications down the road. On the other hand if we employ simulation methods without regard to how intuitively they model the presumed role of chance variation in the production of the data, then we are back to using statistical procedures as black boxes that don't convey insight to students at the introductory level.
My dilemma may be due in part to a lack of formal statistical training. Has anyone else found themselves puzzled by similar questions?