Measuring the Impacts of Teachers: Reply to Rothstein

Using data from North Carolina, Jesse Rothstein (2017) presents a comprehensive replication of Chetty, Friedman, and Rockoff’s [CFR] (2014a,b) results on teachers’ impacts. In addition, Rothstein presents new evidence that he argues raises concerns about three aspects of CFR’s methods and identification assumptions: their treatment of missing data, the validity of their quasi-experimental design, and their method of controlling for observables when estimating teachers’ long-term effects. In this paper, we show that Rothstein’s methodological critiques are not valid by replicating his new empirical findings using simulated data in which none of CFR’s identification assumptions are violated. We also present supplementary empirical evidence from our data supporting the assumptions required for CFR’s original analyses. Together, these results show that: (1) Rothstein’s technique for imputing teacher VA for teachers with missing data generates bias, while subsamples with no missing data yield estimates of forecast bias similar to CFR’s baseline results; (2) his proposed prior score “placebo test” rejects valid quasiexperimental research designs, and the correlation between changes in prior test scores and current teacher value-added he documents is an artifact of estimating teacher value-added using prior test score data; and (3) his method of controlling for covariates yields inconsistent estimates of teachers’ long-term effects, while quasi-experimental designs that do not rely on controls for observables yield estimates of teachers’ long-term impacts similar to CFR’s baseline results. We conclude that Rothstein’s important replication study is entirely consistent with – and in fact reinforces – CFR’s methods and results. Our conclusions match those of Bacher-Hicks, Kane, and Staiger (2014), who replicate both CFR’s results and Rothstein’s findings using data from Los Angeles and also conclude that Rothstein’s results raise no concerns about CFR’s analysis.