The results of the study by Dobbie and Fryer (2010) showed that enrollment in the Promise Academy Charter Middle School run by the Harlem Children’s Zone (HCZ) has the potential to eliminate racial gaps in both math and English Language Arts (ELA) test scores between white and African American middle school students in New York City (NYC), New York.
The academic data showed that in fourth and fifth grade, prior to entering the Promise Academy, lottery winners, lottery losers, and the average black student in NYC had virtually the same math test scores, which were roughly 0.75 standard deviations behind the average white student in the city. The intent-to-treat (ITT) analysis showed that, compared to the control group, lottery winners had a modest but significant increase in math test scores in sixth grade, followed by a more substantial increase in seventh grade and an even larger increase in eighth grade. Lottery winners were 11.9 percent more likely to be performing on grade level in math in the sixth grade, 16.3 percent more likely in the seventh grade, and 27 percent more likely in eighth grade.
The treatment-on-treated (TOT) analysis, which looked at the effects of actually attending the charter middle school, showed a similar pattern. There was a convergence in test scores between students who attended the Promise Academy middle school and the average white student in NYC. In other words, Promise Academy students had nearly caught up to the average white student in New York City public schools in math test scores after 3 years of schooling. The TOT analysis found that lottery winners who enrolled in the charter middle school scored 0.338 standard deviations higher in sixth grade math, 0.371 higher in seventh grade math, and 0.745 higher in eighth grade math than the control group. The results were statistically significant in all 3 years.
As with math scores, lottery winners, losers, and the average African American student in NYC had similar ELA scores, which were roughly 0.65 standard deviations behind the average white student in the city. ITT analysis showed there were no significant differences in ELA scores between the lottery winners and losers until eighth grade. In the eighth grade, Promise Academy students had significantly higher scores than the control group, although the treatment effect was not as strong on ELA scores as it was on math scores (lottery winners scored 0.196 standard deviations higher in eighth grade). The TOT analysis found that scores for students enrolled in Promise Academy had 0.279 standard deviations higher in eighth grade ELA compared to the control group, but there was no effect on sixth- and seventh-grade ELA scores.
Absences and Matriculation
Compared to the control group, lottery winners had fewer absences in the first 180 days of school in every grade. Lottery winners were 2.85 days less absent in sixth grade, 2.31 days less absent in seventh grade, and 3.9 days less absent in eighth grade. There were no significant differences between lottery winners and losers in matriculation.
Elementary School Results
The researchers also analyzed academic results for students in the elementary school portion of the Promise Academy. Overall, Promise Academy students in elementary school significantly improved their math and ELA scores, and reported significantly fewer absences than the control group. The results suggest that the effects of attending the charter elementary school could potentially eliminate any racial gaps between white and black students in NYC. However, the limitations of this analysis make it impossible to draw any definitive conclusions about the treatment effect, and the results should be interpreted with caution. (For more information on the analysis limitations, please see the “Evaluation Methodology” section.)