Background/Context: Since 2002, U.S. federal funding for educational research has favored the development and rigorous testing of interventions designed to improve student outcomes. However, recent reviews suggest that a large fraction of the programs developed and rigorously tested in the past decade have shown null results on student outcomes and, often, intermediate variables. Scholars reporting on null results often explain such results by citing factors they informally identified while they either delivered or observed the program.
Purpose/Objective/Research Question/Focus of Study: In this paper, we argue for a more systematic approach to examining null results, and develop a framework for evaluating null results based on the policy implementation literature. We then illustrate this approach by examining why one professional development program failed to show impacts on measures of teaching and student learning in a recent study.
Setting: The professional development program took place in a mid-sized urban school district on the East Coast. The provider was fully scaled up, capable of providing professional development in most U.S. locations.
Research Design: The main study of this program was conducted as a cluster randomized trial with 105 teachers in 18 schools. Here, we engage in a qualitative case study, using multiple sources of evidence to assess the likelihood that specific reasons for null results are valid.
Data Collection and Analysis: The case study sources of evidence include observations of professional development; teacher surveys and logs; transcribed videos of teachers’ mathematics instruction; and teacher interviews.
Findings/Results: Our analysis suggested that null impacts could result from district priorities and instructional guidance that compete with professional development goals; weaknesses in the intervention as well as its fit to teachers’ needs; and the difficulty of implementing ambitious instructional practice.
Conclusions/Recommendations: Our findings suggest the need for further elaboration of the null-results framework. They also suggest that professional development providers consider both (a) both the organizations in which programs operate and (b) fit of the program to teachers’ needs as potential barriers to success.
