Publication Type

Can Technology Transform Communication Between Schools, Teachers, and Parents? Evidence from a Randomized Field Trial

2022

We study the adoption and implementation of a new mobile communication application (app) among a sample of 132 New York City public schools. The app provides a platform for sharing general announcements and news, as well as engaging in personalized two-way communication with individual parents. We provide participating schools with free access to the app and randomize schools to receive intensive support (training, guidance, monitoring, and encouragement) for maximizing the efficacy of the app. Although user supports led to higher levels of communication within the app in the treatment year, overall usage remained low and declined in the following year when treatment schools no longer received intensive supports. We find few subsequent effects on perceptions of communication quality or student outcomes. We leverage rich internal user data to explore how take-up and usage patterns varied across staff and school characteristics. These analyses help to identify early adopters and reluctant users, revealing both opportunities and obstacles to engaging parents through new communication technology.

Suggested Citation:

Suggested citation

Kraft, M.A., & Bolves, A.J. (2022). Can Technology Transform Communication Between Schools, Teachers, and Parents? Evidence from a Randomized Field Trial. Education Finance and Policy. https://doi.org/10.1162/edfp_a_00344

Share

Publication Type

Navigating Virtual Delivery of Assessments for Head Start Children During the COVID-19 Pandemic

2022

Many preschool agencies nationwide continue to experience closures and/or conversions to virtual or hybrid instruction due to the ongoing coronavirus disease 2019 (COVID-19) pandemic. Despite the importance of understanding young children’s learning and development during the COVID emergency, limited knowledge exists on adaptable practices for assessing young children during the pandemic. We assess the learning of 336 Head Start children across four states and three different time periods during the 2020 to 2021 school year, using adaptations of traditionally in-person assessments of early numeracy, early literacy, and executive functioning. In doing so, we distill early lessons for the field from the application of a novel, virtual assessment method with the early childhood population. This article describes the adaptations of assessment administration for virtual implementation and incorporation of feedback into continued virtual delivery of assessments. Applications and limitations in broader contexts are discussed.

Suggested Citation:

Suggested citation

Leiah J. G. Groom-Thomas, L., Lee, M.G., Todd, C.S., Lynch, K., Loeb, S., McConnell, S., & Lydia, C. (2022). Navigating Virtual Delivery of Assessments for Head Start Children During the COVID-19 Pandemic. Journal of Early Intervention. https://doi.org/10.1177/10538151221085942

Share

Publication Type

Strengthening the Research Base that Informs STEM Instructional Improvement Efforts: A Meta-Analysis

2019

We present results from a meta-analysis of 95 experimental and quasi-experimental preK-12 science, technology, engineering, and mathematics (STEM) professional development and curriculum programs, seeking to understand what content, activities and formats relate to stronger student outcomes. Across rigorously conducted studies, we found an average weighted impact estimate of +0.21 standard deviations. Programs saw stronger outcomes when they helped teachers learn to use curriculum materials; focused on improving teachers' content knowledge, pedagogical content knowledge and/or understanding of how students learn; incorporated summer workshops; and included teacher meetings to troubleshoot and discuss classroom implementation. We discuss implications for policy and practice.

Suggested Citation:

Suggested citation

Lynch, K., Hill, H., Gonzalez, K., & Pollard, C. (2019). Strengthening the Research Base that Informs STEM Instructional Improvement Efforts: A Meta-Analysis. Educational Evaluation and Policy Analysis. https://doi.org/10.3102/0162373719849044

Share

The Longitudinal Effects of School Improvement Grants

2020

School Improvement Grants (SIG) represent one type of governments’ capacity-building investment to spur sustainable changes in America’s persistently under-performing public schools. This study examines both short- and long-run effects of the first two cohorts of SIG schools from two states and two urban districts across the country. Using dynamic event analyses, we observe that SIG showed larger effects in the second and third years of the intervention than the first year on 3-8th grade student test scores—a pattern of gradually increase over the course the intervention. These positive effects are largely sustained three or four years after the funding ended. In high schools, the SIG effects on 4-year graduation rates were steadily increasing throughout the period of six or seven years after the initial start of the intervention. These patterns of SIG effects mostly apply to each of the four locations, but the magnitude of effects varies across locations, suggesting differential implementations. Moreover, SIG effects on students of color or low-socioeconomic students are similar to, and sometimes a bit larger than, the overall SIG effects. We also conduct a variety of sensitivity and robustness checks. Lastly, we discuss the policy implications of our findings on states’ continuing efforts of transforming public organizations and building their long-term capacity for better performance.

Suggested Citation:

Suggested citation

Loeb, S., Sun, M., & Kennedy, A. (2020). The Longitudinal Effects of School Improvement Grants. Educational Evaluation and Policy Analysis.

Share

Publication Type

Early Lessons from Implementing High-Impact Tutoring at Scale

2021
Many education leaders are turning to high-impact tutoring to accelerate student learning and compensate for interrupted instruction due to COVID-19. As federal, state, and local tutoring policies and practices develop, understanding the key barriers and challenges that have the potential to limit program reach and hinder efficacy and then developing targeted approaches to overcoming these barriers can improve the likelihood of success and the ultimate benefits for students. The National Student Support Accelerator, an initiative of the Annenberg Institute at Brown University, has partnered with over a dozen school districts and tutoring providers to implement tutoring programs, to learn from these efforts, and to develop tools and other resources to aid in the provision of high-impact tutoring broadly. This paper draws on interviews with Accelerator partners to provide an up-to-date view of the barriers to implementing quality, school-based tutoring at scale and promising practices to mitigate these challenges.Across multiple studies and reviews of education interventions, researchers have found tutoring to have large, positive impacts on student achievement in both math and reading (Dietrichson, et al., 2017; Fryer, 2017). Tutoring programs can remain effective even as they expand to serve large numbers of students. For example, an analysis of fifteen larger-scale tutoring programs found that initiatives increased student learning by an average of 2-10 additional months (Nickow et al., 2020). However, not all tutoring programs are effective, particularly when implemented at a large scale. The federally-mandated after-school tutoring services under No Child Left Behind’s Supplemental Education Services (SES) program were found to have little to no effect on student outcomes, except in a few more regulated contexts (Hienrich et al., 2014).Given the historical inconsistency of tutoring effectiveness when implemented at scale, how can we set current efforts up for success? The existing research base on challenges and barriers to effective tutoring implementation and how to combat them is slim. Many studies consider small-scale, localized tutoring initiatives, often driven and designed by researchers themselves, and few of these studies comment on program implementation. Those that do cite logistical challenges districts face in establishing regular time and space within the existing school day to conduct tutoring (Allor & McCathren, 2004; Al Otaiba et al., 2005; Bryant et al., 2011).Much of what we know about program implementation comes from research on SES. This program required Title 1 schools to provide parents with the option of enrolling their student in additional learning opportunities to receive “extra academic help, such as tutoring or remedial help” during out-of-school hours (U.S. Department of Education, 2012). Researchers identify the program’s design as a key barrier to success as it depended on parents, who they argue may have limited capacity to discern provider efficacy and to choose from among available tutoring organizations. A qualitative evaluation of SES services found that the majority of students enrolled were not even receiving small-group tutoring but were instead participating in traditional, large-group classroom environments (Heinrich et al., 2014). The program also experienced low enrollment and attendance rates, a challenge which researchers attribute to a combination of after-school schedule conflicts, reliance on parents to identify, enroll, and in some cases, transport students to the programs, and minimal district financial capacity to support program development and administration (Heinrich et al., 2014). Additionally, a study of SES implementation across four large cities found that the average hours of tutoring received per student varied from 15 to 40 depending on the year and location, largely influenced by the availability of program funding (Heinrich et al., 2014).The current push to expand tutoring diverges from SES in that, while many districts are drawing upon federal funding, few federal prescriptions exist for program design requirements except that programs should be “evidence-based” - for example, parents do not need to opt into tutoring for their children nor select the tutoring provider. Moreover, the challenge of ensuring quality SES providers was further exacerbated by the lack of existing, rigorous program evaluations for most organizations and lack of capacity at the district and state levels to fill these research gaps (Heinrich et al., 2010). The knowledge base identifying the elements of high-impact tutoring is much more clear today, though the knowledge of implementation challenges and successes is still sparse.Over the late spring of 2021, we conducted 23 semi-structured interviews with 7 school districts, one charter management organization, and 12 tutoring providers. Three of these organizations participated in two conversations. From these interviews we focused on common challenges and barriers, identifying three overarching themes with 11 subthemes. Knowing these challenges can help focus tutoring implementation, pre-empting common barriers and maximizing the likelihood of program success.

Suggested Citation:

Suggested citation

White, S., Carey, M., O’Donnell, A., & Loeb, S. (2021). Early Lessons from Implementing High-Impact Tutoring at Scale. National Student Support Accelerator.

Share

Sara White

Publication Type

Does Reclassification Change How English Learners Feel About School and Themselves? Evidence From a Regression Discontinuity Design

2022

Reclassification can be an important juncture in the academic experience of English Learners (ELs). Literature has explored the potential for reclassification to influence academic outcomes like achievement, yet its impact on social-emotional learning (SEL) skills, which are as malleable and important to long-term success, remains unclear. Using a regression discontinuity design, we examine the causal effect of reclassification on SEL skills (self-efficacy, growth mindset, self-management, and social awareness) among 4th to 8th graders. In the districts studied, reclassification improved academic self-efficacy by 0.2 standard deviations for students near the threshold. Results are robust to alternative specifications and analyses. Given this evidence, we discuss ways districts might establish practices that instill more positive academic beliefs among ELs.

Suggested Citation:

Suggested citation

Lee, M.G., & Soland, J.G. (2022). Does Reclassification Change How English Learners Feel About School and Themselves? Evidence From a Regression Discontinuity Design. Educational Evaluation and Policy Analysis. https://doi.org/10.3102/01623737221097419

Share

Publication Type

More Than Shortages: The Unequal Distribution of Substitute Teaching

2022

Classroom teachers in the US are absent on average approximately six percent of a school year. Despite the prevalence of teacher absences, surprisingly little research has assessed the key source of replacement instruction: substitute teachers. Using detailed administrative and survey data from a large urban school district, we document the prevalence, predictors, and variation of substitute coverage across schools. Less advantaged schools systematically exhibit lower rates of substitute coverage compared with peer institutions. Observed school, teacher, and absence characteristics account for only part of this school variation. In contrast, substitute teachers’ preferences for specific schools, mainly driven by student behavior and support from teachers and school administrators, explain a sizable share of the unequal distribution of coverage rates above and beyond standard measures in administrative data.

Suggested Citation:

Suggested citation

Liu, J., Loeb, S., & Shi, Y. (2022). More Than Shortages: The Unequal Distribution of Substitute Teaching. Education Finance and Policy. https://doi.org/10.1162/edfp_a_00329

Share

Publication Type

Educational Goods: Values, Evidence, and Decision Making - A Summary

2020

This is a brief summary of the book Educational Goods: Values, Evidence, and Decision‐Making by Harry Brighouse, Helen F. Ladd, Susanna Loeb and Adam Swift. It provides the introduction to the present symposium on this book, which includes the ensuing contributions from Carey Bagelman, Randall Curren, Michael Hand, John Tillson and Winston Thompson, followed by a response from the authors.

Educational decision‐making involves value judgments. As decision makers aim for improvements, they need standards that tell them what counts as an improvement. However, they typically lack a rich and sophisticated language for talking about values and articulating trade‐offs. The book aims to enrich the language available to, and to clarify the thinking of, both educational decision makers and the researchers whose work informs their deliberation, by offering a systematic framework for thinking about the goals of education.

We have coined the term ‘educational goods’ to refer to the knowledge, skills, attitudes and dispositions that children develop both for their benefit and for the benefit of others. We identify, at a general level, six capacities that will tend to support the flourishing of both the agent herself and others in her society: the capacities for economic productivity, personal autonomy, democratic competence, healthy personal relations, regarding others as moral equals, and personal fulfilment. We think of these as the distinctive goods that education aims to produce.

Suggested Citation:

Suggested citation

Brighouse, H., Ladd, H.F., Loeb, S., & Swift, A. (2020). Educational Goods: Values, Evidence, and Decision Making - A Summary. Journal of Philosophy of Education. https://doi.org/10.1111/1467-9752.12508

Share

Publication Type

Relations Among Cultural Learning Beliefs, Self Regulated Learning, and Academic Achievement for Low Income Chinese American Adolescents

2016

This mixed‐methods study of urban low‐income, English‐proficient Chinese American, second‐generation 15‐year‐olds (conducted in 2004; N = 32) examined the relation among the virtue model of learning communicated by parents and adolescents’ learning beliefs, self‐regulated learning (SRL) behaviors, and academic achievement. Analysis of in‐depth individual interviews revealed that for these adolescents, perceptions of family educational socialization predicted students’ endorsement of their culture's virtue‐oriented learning beliefs and that adolescents’ endorsement of these learning beliefs predicted their academic achievement. Importantly, adolescents’ reported that use of SRL strategies mediated the relationship between their endorsement of virtue‐oriented learning beliefs and their academic achievement. Findings are discussed in the context of further research linking cultural learning beliefs, SRL, and children's academic achievement.

Suggested Citation:

Suggested citation

Bempechat, J., Li, J., & Ronfard, S. (2016). Relations Among Cultural Learning Beliefs, Self Regulated Learning, and Academic Achievement for Low Income Chinese American Adolescents. Child Development. https://doi.org/10.1111/cdev.12702

Share