Table of Contents

1. What is statistics education research?

In general, statistics education research focuses on both the instruction and learning of statistics at all levels, using any appropriate research methodology (qualitative and quantitative) that addresses the question being asked. Statistics education research is designed so that the results will have direct implications for instruction. When stating the research question, the investigator needs to consider how the study will relate to actual classroom instruction and settings. The research methodology should follow from the research question (and not the other way around). Meaningful measurements must be identified that address the stated research question (e.g., focus groups to identify students' interpretation of learning goals in addition to objective measures of learning). Methods should be used to minimize confounding variables or alternative interpretations of results (e.g., inclusion of both quantitative assessments and interviews to determine if the results corroborate similar interpretations). Finally, the research report should address classroom implications, as well as the generation of new research questions generated by the findings.

2. How can I get started doing statistics education research?

Curiosity initiates research. First, identify areas you are curious about. Then, investigate what's been done on that topic. For example, are you interested in investigating students' statistical reasoning about some concept? Are you interested in the impact of some teaching innovation on student learning?

Search the existing statistics education research literature, conferences, and proceedings. Be sure to search across multiple disciplines, such as mathematics education, statistics and statistics education, psychology, science education, etc. Use the online database at CAUSEweb to help locate pertinent articles, journals and conferences. CAUSEweb also includes information on some articles not yet published in mainstream journals.

It may be helpful also to search other online databases such as: www.eric.ed.gov or scholar.google.com. Also, you may find it helpful to read this 2006 Joint Statistical Meetings proceedings paper that addresses some practical issues in conducting statistics education research: "Practical Issues in Conducting Statistics Education Research"

3. How should I identify a good problem to study?

Try to keep a narrow focus. Think about questions that you are interested in or others are interested in answering. Ask questions that could be useful for improving teaching and learning of statistical concepts. For example, if you are interested in finding out the impact of computers, the question, "What is the impact of using computers to teach statistics" is too broad and it may be difficult to determine meaningful measures. The question, "What is the impact of using computer simulations on student learning of the concept of distribution?" is more focused and measurable. Here are a few more examples of researchable questions:

• Does lecturing to students about a topic help them understand the topic better than if they read about it on their own?

• What aspects of learning activities are most effective for student learning?
  What is an effective way to help students understand the concept of p-value?
  Encourage them to compare 2-3 strategies?

• Are hands-on activities an effective way to help students overcoming the difficulty of learning abut the sampling distribution of sample mean?

Also, there are chapters addressing this topic such as Kathryn R. Wentzel's "Developing and Nurturing Interesting and Researchable Ideas" (in the Sage Handbook for Research in Education: Engaging ideas and enriching inquiry, edited by Cliftton F. Conrad and Ronald C. Serlin). Also, at USCOTS 2007, the "I Wonder" session (whose slides are posted on CAUSEWeb) involved several statistics education researchers sharing how they identified their research questions.

4. What kind of outcomes should I measure in my research study?

The research question you are studying should drive the outcomes you measure, and you may want to look at more than one outcome. Often the more relevant outcome of interest is to demonstrate an improvement in student reasoning, but sometimes it may also be of interest to examine other affective factors, such as attitudes about statistics and statistics anxiety.

Here are some issues to consider in measuring learning outcomes: Try to use reliable and validated measures of the specific learning outcomes related to your particular research question (e.g., reasoning about p-value). This leads to more credible and generalizable results. Many good tools are available from the ARTIST website. We caution against using final exams or course grades as outcome measures for several reasons: (1) they are specific to a particular teacher and course, (2) they may be more affected by good study habits than learning outcomes, and (3) they tend to be more global measures of academic performance than precise measures of conceptual understanding. It is also important to consider how the abstract ideas you are interested in studying can be translated into specific, well-defined instruments as well as procedures for administering them. For example, if you were interested in whether some intervention improved student learning, you should use a valid and reliable instrument that measures the particular concept your intervention was designed to address. Moreover, you should be careful to specifically define the treatments in your intervention. If there is no available instrument for the topic you want to measure, developing a new assessment tool is possible, but bear in mind that this is a major undertaking and should be done with someone who has expertise in measurement. To complement these quantitative measures, other methods (e.g., clinical interviews) are often effective. See also: FAQ #9

Here are some issues to consider in measuring attitudes and other affective outcomes: Some instruments for measuring students' attitudes toward statistics (SATS) and statistics anxiety (The Statistics Anxiety Rating Scale) already exist. However, there are problems in focusing on improvement in these outcomes in general: there is often so little change in attitudes over the course of one semester, that it is difficult to measure any improvement, and these measures are hard to distinguish from more general constructs (e.g., attitudes toward classes in general or academic anxiety).

Other outcome measures of potential interest include time that students spend on a particular task, overall course attendance, retention beyond the current statistics course, success in future statistics courses.

5. How do I find potential collaborators?

It is much better to collaborate with others on any research project. It is important to bring different backgrounds and perspectives to a study, to discuss your ideas with others, to have discussions of methods, design and results with others, either as collaborators or as advisors to your project. It also helps if you can work with someone from a different discipline, such as education or psychology, who knows about student learning and educational research methods (if you are a statistician) or with someone from statistics (if you are in education or psychology). If you are not teaching, but your work is focused on teaching, be sure to include a teacher in your research group. To find collaborators, first look at your own institutions, but in different departments. To look for collaborators beyond your institution, you can take advantage of list serves and newsletters.

6. How do I identify and recruit participants for my study?

It's important to determine the population it will be feasible to represent in your study and to think creatively about how to obtain a diverse and/or representative group of participants (e.g., not only students registered in one course or only the first respondents to fliers or the most motivated potential participants). Spend time identifying the stakeholders (those most impacted and interested) in your study and involve them. You could consider different sampling methods and/or offering incentives to target individuals. Clarify from the beginning the purpose and sponsorship of your research, and the expectations and time requirement from their involvement. For example, it might prove beneficial to pilot test and evaluate your data collection instrument to ensure that you have minimized the time requirement of the participants. You should contact potential participants at the beginning of the study to verify feasibility of obtaining the desired sample size.

7. What types of incentives can be used for participants?

In classroom research, you can consider offering incentives to encourage student participation. These incentives can take many different forms and will depend on the time commitment and tasks you are asking of the students (e.g., surveys vs. videotaped interviews) as well as whether they are currently involved in your course, someone else's course or have already completed the course of interest. For example, for a quiz concerning student understanding of a topic in the course, it is best to embed this assessment as part of the course grade, ensuring full and authentic participation. For a supplementary activity with students outside of class, you could offer an incentive to encourage participation. This could include monetary incentives (e.g., a cash award or a bookstore gift certificate), a "treat" (e.g., homemade cookies), or extra credit points in the course. A ballpark monetary amount for larger tasks could be roughly $10/hour of time. This can be increased on follow-up if students need more prodding. It is important to remember that too large of a dollar amount can be coercive and any amount can impact the type of student that agrees to volunteer and should be discussed in your study conclusions.

Similarly, if you plan to work with other instructors, you might also consider an incentive system to encourage and sustain their full participation. This can range from travel and time reimbursements to course release for longer term projects. The longer and more involved the project, the higher the drop out rate. Collaborators should be treated as equals with full participation, ownership, and co-authorship opportunities during the project.

8. What do I have to know about using human subjects in my research?

Your institution has either a Human Subjects Review Board (HSRB) or an Institutional Review Board (IRB) that oversees all research projects involving human subjects. The title of your board may be different, though these are the two most common. The primary role of the HSRB/IRB is to help you evaluate the feasibility of your project as it relates to ethics, protocol, and confidentiality/anonymity. HSRB/IRBs usually have a formal review process with specific institutional forms for you to complete. These forms will ask you for your research question(s), participant description and recruitment, data collection procedures, consent process, and possibly a brief literature review - these requirements vary by institution. If you are a student, you must have a faculty member sign as advisor or as primary investigator.

This process must be completed before you collect data. Data collected prior to HSRB/IRB approval cannot be published or used as part of your final study report. Given this imperative, and the fact that most HSRB/IRBs meet once a month, you must get an early start on the process once you have identified your research question and design. Depending on the complexity and content of your application, the review process may take a while. This may also include revision and resubmission before the board gives final approval.

Prior to applying for HSRB/IRB approval, most if not all institutions in the United States require all researchers associated with the study to have completed some form of federal training on Human Subjects Research. Again, your institution will have information about the opportunities for this training, which may be either face-to-face workshops or online training courses with some evaluation component.

Note there are many special circumstances in research with human subjects r proposal so these issues can be addressed. Ultimately, our advice is to make sure you are aware of and familiar with the HSRB/IRB approval process at your institution by contacting the appropriate office as soon as you are sure you will be doing a study with human subjects.

9. Which research designs are "acceptable" for statistics education research?

The research question drives the design of the study - use whichever research design appropriately addresses your research question(s). The particular way you design your study is open - research in statistics education has included the "traditional" quantitative designs, qualitative designs, as well as the newly emerging mixed-methods designs (designs that mix both quantitative and qualitative data, analysis, and/or interpretation). Each family of designs has its own purposes, language, procedures, advantages and disadvantages - again, always driven by the research question(s) you are investigating. Rather than describe each of the many ways to design a study, we suggest researchers consult appropriate textbooks and websites devoted to the major paradigms. Researchers are also wise to read published reports of statistics education research to see how others have addressed similar research questions.

Many questions related to the teaching and learning of statistics can be answered a variety of ways - those well-versed only in quantitative methods may want to explore options in qualitative or mixed-method paradigms to extend their work in new directions. However, individual researchers each have their own strengths, and it is just as important to remember you are contributing to a larger body of knowledge - one in which other researchers may take up your questions in new ways. You may also choose to collaborate with other researchers whose strengths may be different from your own.

10. Where Can I find the Funding Support?

For a starter, the best source of funding may be your university or college instructors and state educational funding resource such as Eisenhower Grant for K-12 teachers. Most universities provide seed grant for a research initiative. Check out the funding opportunity from your university first.

Try to find research collaborators within or outside your university. Team projects often are considered more favorably by funding agencies. Do your homework to make a solid preparation for the proposed research project such as conducting a pilot study or collecting data that are of useful for making your case, completing the IRB approval at your institution and so on.

In addition to the possible seed grant from your institution, Chances are your institution has a division that can help you locating funding agencies. Make an appointment to consult with individuals in this division. The following is a very short list of funding agencies which have supported educational related research.

NSF (http://www.nsf.gov/)
US Department of Education (http://www.ed.gov/index.jhtml)
No Child Left Behind Initiative (http://www.ed.gov/nclb/landing.jhtml)
Department of Education at State Level (check the web site of your state government.)
Eisenhower Grant for K-12 related research (check the web site of your state government or your school principal)
American Educational Research Association (http://www.aera.net)
ADC Foundation (http://www.adc.com/aboutadc/adcfoundation/)
Pew Foundation (http://www.pewtrusts.com/)
Kellogg Foundation (http://www.wkkf.org/)