Introduction

A Researcher’s Toolkit: Using Different Methods to Shed Light on Critical Questions

A lot of research studies focus on comparing what goes in the box and what comes out of the box after the treatment. Our study focuses on unpacking what’s inside the box. What’s happening, and what makes it happen?
—EDC’s Johannah Nikula

With all of the emphasis today on randomized controlled trials in education research, it’s easy to underplay the role and value of other research methodologies. Education funders and policymakers—including the U.S. Department of Education—have been pushing for more randomized controlled trials out of legitimate concern about the level of rigor in education research. Too many innovations have been warmly embraced and implemented in school districts with very little evidence showing that they improve teaching and learning. Randomized trials are the best methodology available to determine if a given innovation actually caused a positive change in the classroom—i.e., the difference between what goes into “the box” and what comes out of it.

As researcher Johannah Nikula points out, however, that sort of research doesn’t always shed light on what’s happening inside the box. How does the innovation actually work? For example, in addition to knowing that a given curriculum, professional development program, or piece of technology is effective, we also want to know why it’s effective. Those insights help us generalize our findings and inform the development of future innovations.

As an organization with a long history in both health and education research, EDC has wide experience with a full range of methodologies. We have, for example, participated in several national public health studies funded by the National Institutes of Health in which we use randomized controlled trials to assess the impact of education interventions aimed at reducing high-risk behaviors. In recent years, we’ve begun to incorporate some form of randomized assignments into more and more of our school-based projects.

Types of Research

Basic Research: Research
directed toward gaining fundamental knowledge and a full understanding of a problem, or how a process, tool, or phenomenon works.

Applied Research: Research directed toward the practical application of knowledge to solve a specific problem or fill
a particular need.

Experimental Research: A
study in which treatment and control groups are set up to test the effectiveness of a new process, tool, or phenomenon in specific settings.

The research methodologies we use for a particular study depend on the questions we set out to answer. EDC both creates new programs and evaluates programs developed by other organizations. The methods we are likely to employ during the early development of an innovation differ markedly from the methods we would use for a fully developed tool or resource. Typically, the stages of research follow a continuum from basic research to applied research to controlled trials and other experimental methods (see box). In the early stages of development of a tool (e.g., a curriculum or technology application), we use basic and applied research methods to explore both broad and narrow questions— to understand what we are putting into the box and how it is designed to work. After initial testing and refinement of the tool, we move on to studying its effectiveness in different kinds of settings. In the latter stages of development, randomized controlled trials can be a particularly valuable tool for isolating and studying key variables, such as: How effective is the curriculum when delivered as a stand-alone instrument compared to the curriculum coupled with a professional development workshop?

Several articles in this issue of Mosaic illustrate the way we employ basic, applied, and experimental research to understand both the innovation itself and the effects it has on participants. Each of the profiled projects use a combination of research methods. For example:

In a synthesis of studies on science education, EDC analyzed hundreds of research reports, many of which drew conclusions about the success or failure of various programs characterized as “inquiry science.” As EDC researchers examined all the reports, however, they found that the definition of “inquiry science” varied tremendously. The process of synthesizing the research reports will provide the clearest picture to date of what “inquiry science” actually looks like in classroom practice.
The Turning to the Evidence project focuses on the creation of two professional development programs built around the use of “classroom artifacts”—student work, transcripts, and videos of studentteacher dialogue. In addition to designing one of the programs, EDC is carrying out an evaluation in which we are trying to understand both what’s in the box (What do the artifacts contribute to the programs?) and what’s coming out of the box (Are the programs successful at improving teachers’ knowledge and practice?). The evaluation uses a variety of methodologies, including control groups, pre- and post-intervention surveys and mathematics tests, and blind coding of teacher and student work.
EDC’s Center for Online Professional Education (COPE) has carried out a range of studies on online courses for teachers. COPE has spent several years designing and testing various models of online courses and studying a wide range of variables that affect a course’s success, from technology issues (such as choice of software) to facilitation issues to policy considerations (such as the school district’s level of commitment to professional development). In its newest study, COPE is zeroing in on some key variables in facilitation by using random assignment of teachers to courses and other experimental research techniques.

...in addition to knowing that a given curriculum, professional development program, or piece of technology is effective, we also want to know why it’s effective.

In other articles in this issue, staff members from EDC’s Health and Human Development Programs report on research into a variety of promising school health initiatives being carried out in the United States and China. The China study is measuring the effectiveness of a schoolbased nutrition and health program using both quantitative and qualitative measures. The researchers explain that the qualitative measures were added to the study so that we could better understand the complex environment in which the research takes place, including Chinese cultural values that affect health and nutrition practices.

The importance of understanding and attending to community values and needs is as critical in our U.S.-based research as it is in the China study. Researchers from EDC’s Center for Children and Technology describe the process they use when conducting an evaluation of a new educational technology tool as “listening to local relevance.” As they write in their article, we are not simply assessing the quality of a particular tool or resource; we are studying the relationship between that tool and the school or community that is using it:

No matter how educationally sound a practice or model of technology use is, when it is introduced into a new school [it] changes to fit the particular needs, strengths, and structures of an individual school, classroom, or teacher. Often, this intersection of intervention and local adaptation releases, in a process not unlike fusion, new teaching and learning opportunities.

It’s that process of continual intervention, adaptation, and fusion that makes education research especially challenging. In addition to designing new programs and approaches to improve education and health, EDC also focuses on developing new research tools and strategies to understand what’s going into the box, what is coming out of it, and what’s happening inside to produce the changes we see.

For questions or comments, contact mosaic@edc.org.