National Institute of Justice National Institute of Justice. Research. Development. Evaluation. Office of Justice Programs
Crime Solutions.gov
skip navigationHome  |  Help  |  Contact Us  |  Site Map   |  Glossary
Reliable Research. Real Results. skip navigation
skip navigation Additional Resources:

skip navigation

Program Profile: Harlem (NY) Children's Zone – Promise Academy Charter Middle School

Evidence Rating: Effective - One study Effective - One study

Date: This profile was posted on June 13, 2011

Program Summary

A charter middle school that seeks to give students in grades 6–8 a well-rounded, high-quality education. The program is rated Effective. The results of the study showed that enrollment has the potential to eliminate racial gaps in both math and English Language Arts test scores between white and African American middle school students in New York City. There were improvements in math, ELA scores, and fewer absences.

Program Description

Program Goals/Target Population
The Harlem Children’s Zone (HCZ) is a 97-block area in the Harlem neighborhood of New York, NY, that combines "no excuses" charter schools with communitywide initiatives to address some of the main problems that underprivileged children face every day, such as inadequate schools, high-crime neighborhoods, and health issues. One important component of the HCZ Project is the Promise Academy Charter Middle School, which aims to provide students in grades 6–8 with a well-rounded, high-quality education.

The Promise Academy middle school serves predominately low-income, minority students in Harlem who are usually 2 or 3 years behind grade level. While the community programs offered by HCZ are available to anyone living near the area, the charter middle school is not open to all students. Because of the limited amount of space available, admission to the Promise Academy is only offered to students whose numbers are randomly selected during the middle school lottery. The school began in 2004 with 100 sixth graders, and by 2009, there were 700 middle school students in the system.

Program Components
The Promise Academy combines structural reforms with wraparound services to provide students with a comprehensive college preparatory program. The Promise Academy provides an extended school day and year, coordinated afterschool tutoring, and additional classes on Saturdays for anyone who needs extra help with homework. In the classroom, strong focus is placed on mastering basic skills, especially literacy and mathematics skills. Students have more than 2 hours of literacy instruction and more than 90 minutes of mathematics instruction every day. Students also explore coursework in the arts and sciences.

In addition, many students spend 10 or more hours a day in school. An average school day runs from 8 a.m. to 4 p.m., which is a 20 percent increase over a typical school day. Students are also given the opportunity to participate in afterschool programs, such as photography or music activities, that run from 4 p.m. to 6 p.m. The academic year is 210 days of school, which includes a mandatory 25-day summer program. Only 180 days of school are required by law.

The Promise Academy also runs a health clinic in the middle school building that provides students with free medical, dental, and mental-health services. Students are screened upon entering the school, and they receive regular check-ups through the Children’s Health Fund. Students also receive incentives for achievements (such as money or trips), nutritious and freshly made meals, bus fare, and other intangible benefits, such as support from committed staff members.

Additional Information
The Promise Academy Charter Middle School is part of the HCZ Project that provides a continuum of services focusing on the developmental needs of children in Harlem. In addition to the middle school programs, the Project includes elementary school programs that are available to students in grades K–5. However, the evaluation research examining the effectiveness of the elementary school programs has not yet reached the rigor of the research looking at the middle school programs.

Evaluation Outcomes

top border

Study 1

The results of the study by Dobbie and Fryer (2010) showed that enrollment in the Promise Academy Charter Middle School run by the Harlem Children’s Zone (HCZ) has the potential to eliminate racial gaps in both math and English Language Arts (ELA) test scores between white and African American middle school students in New York City (NYC), New York.

 

Math Scores

The academic data showed that in fourth and fifth grade, prior to entering the Promise Academy, lottery winners, lottery losers, and the average black student in NYC had virtually the same math test scores, which were roughly 0.75 standard deviations behind the average white student in the city. The intent-to-treat (ITT) analysis showed that, compared to the control group, lottery winners had a modest but significant increase in math test scores in sixth grade, followed by a more substantial increase in seventh grade and an even larger increase in eighth grade. Lottery winners were 11.9 percent more likely to be performing on grade level in math in the sixth grade, 16.3 percent more likely in the seventh grade, and 27 percent more likely in eighth grade.

 

The treatment-on-treated (TOT) analysis, which looked at the effects of actually attending the charter middle school, showed a similar pattern. There was a convergence in test scores between students who attended the Promise Academy middle school and the average white student in NYC. In other words, Promise Academy students had nearly caught up to the average white student in New York City public schools in math test scores after 3 years of schooling. The TOT analysis found that lottery winners who enrolled in the charter middle school scored 0.338 standard deviations higher in sixth grade math, 0.371 higher in seventh grade math, and 0.745 higher in eighth grade math than the control group. The results were statistically significant in all 3 years.

 

ELA Scores

As with math scores, lottery winners, losers, and the average African American student in NYC had similar ELA scores, which were roughly 0.65 standard deviations behind the average white student in the city. ITT analysis showed there were no significant differences in ELA scores between the lottery winners and losers until eighth grade. In the eighth grade, Promise Academy students had significantly higher scores than the control group, although the treatment effect was not as strong on ELA scores as it was on math scores (lottery winners scored 0.196 standard deviations higher in eighth grade). The TOT analysis found that scores for students enrolled in Promise Academy had 0.279 standard deviations higher in eighth grade ELA compared to the control group, but there was no effect on sixth- and seventh-grade ELA scores.

 

Absences and Matriculation

Compared to the control group, lottery winners had fewer absences in the first 180 days of school in every grade. Lottery winners were 2.85 days less absent in sixth grade, 2.31 days less absent in seventh grade, and 3.9 days less absent in eighth grade. There were no significant differences between lottery winners and losers in matriculation.

 

Elementary School Results

The researchers also analyzed academic results for students in the elementary school portion of the Promise Academy. Overall, Promise Academy students in elementary school significantly improved their math and ELA scores, and reported significantly fewer absences than the control group. The results suggest that the effects of attending the charter elementary school could potentially eliminate any racial gaps between white and black students in NYC. However, the limitations of this analysis make it impossible to draw any definitive conclusions about the treatment effect, and the results should be interpreted with caution. (For more information on the analysis limitations, please see the “Evaluation Methodology” section.)

bottom border

Evaluation Methodology

top border

Study 1

Dobbie and Fryer (2010) examined the effects of being offered admission or attending the Promise Academy charter schools in the Harlem Children’s Zone (HCZ) on academic outcomes. In New York State, oversubscribed charter schools are required by law to allocate enrollment offers through a random lottery system. The study took advantage of this system by comparing the average outcomes of students who “won” the lottery and were offered admission to the Promise Academy to the average outcomes of students who “lost” the lottery and were not offered admission.

 

The sample included students from the middle school lotteries from 2005 through 2006 (data was not available from the 2004 middle school lottery). The treatment group included 169 lottery winners in grades 6–8. The lottery winners were 51 percent male, and 84 percent black and 14.5 percent Hispanic. Seventy-two percent qualified for free lunch. The control group included 289 lottery losers in grade 6–8. The group was 43.3 percent male, and 82.6 percent black and 15.4 percent Hispanic. Sixty-five percent qualified for free lunch. There were no statistically significant differences between the groups.

 

The study used an intent-to-treat (ITT) analysis, which meant the treatment group included all students who were offered admission to Promise Academy, even if they did not attend. The ITT analysis therefore captured the effect of being offered admission, not of actually attending. The study also included treatment-on-treated (TOT) analysis, which looked at the average effect of enrollment in the Promise Academy on those students who won the lottery and chose to attend.

 

Data was collected from files at HCZ as well as administrative data on student demographics and academic outcomes from the New York City Department of Education. Academic outcomes in math and English Language Arts (ELA) were measured by the New York State tests that are conducted in the winters of third through eighth grade. The math portion of the exam includes questions on number sense and operations, algebra, geometry, measurement, and statistics. The section on ELA assesses students on information and understanding, literary response and expression, and critical analysis and evaluation.

 

Note: The researchers also examined the outcomes of the elementary school students. However, it was not possible to estimate the treatment effects using the ITT or TOT method because the elementary school lotteries had never been significantly oversubscribed. As a result, nearly all lottery losers were eventually offered admission into the Promise Academy. The study ultimately used two different statistical approaches to analyze the outcomes of elementary school students, but the results should be viewed with caution due to the methodological limitations.

bottom border

Cost

top border
In the 2008–09 school year, the Harlem Children’s Zone in the Harlem neighborhood of New York, NY, spent approximately $19,272 per pupil at the Promise Academy. This included in-school costs and afterschool programs. This can be compared to the median school district in New York State in 2006 that spent approximately $16,171 per pupil (Dobbie and Fryer 2010).
bottom border

Evidence-Base (Studies Reviewed)

top border
These sources were used in the development of the program profile:

Study 1
Dobbie, Will, and Roland G. Fryer Jr. 2010. Are High-Quality Schools Enough to Increase Achievement Among the Poor? Evidence from the Harlem Children’s Zone. Cambridge, Mass.: The Kennedy School, Harvard University.
http://scholar.harvard.edu/files/fryer/files/hcz_nov_2010.pdf
bottom border

Additional References

top border
These sources were used in the development of the program profile:

Harlem Children’s Zone. 2009. “Home Page.” Accessed April 25, 2011.
http://www.hcz.org/

Hoxby, Caroline M., and Sonali Murarka. 2009. Charter Schools in New York City: Who Enrolls and How They Affect Their Students’ Achievement. Cambridge, Mass.: National Bureau of Economic Research Working Paper.
http://www.nber.org/papers/w14852

Hoxby, Caroline M., Jenny Lee Kang, and Sonali Murarka. 2009. Technical Report: How New York City Charter Schools Affect Achievement. Cambridge, Mass.: National Bureau of Economic Research.
http://www.nber.org/~schools/charterschoolseval/how_nyc_charter_schools_affect_achievement_technical_report_2009.pdf

Top Tier Evidence Initiative. 2010. “Evidence Summary for the Promise Academy Charter Middle School in Harlem Children’s Zone.”

Harlem Children’s Zone. N/d. Whatever It Takes: A White Paper on the Harlem Children’s Zone.
http://wac.adef.edgecastcdn.net/80ADEF/hcz.org/wp-content/uploads/2014/04/HCZ-White-Paper.pdf
bottom border

Related Practices

top border
Following are CrimeSolutions.gov-rated practices that are related to this program:

Targeted Truancy Interventions
These interventions are designed to increase attendance for elementary and secondary school students with chronic attendance problems. The practice is rated Effective for improving attendance.

Evidence Ratings for Outcomes:
Effective - More than one Meta-Analysis Education - Attendance/truancy



Mentoring
This practice provides at-risk youth with positive and consistent adult or older peer contact to promote healthy development and functioning by reducing risk factors. The practice is rated Effective in reducing delinquency outcomes; and Promising in reducing the use of alcohol and drugs; improving school attendance, grades, academic achievement test scores, social skills and peer relationships.

Evidence Ratings for Outcomes:
Effective - One Meta-Analysis Crime & Delinquency - Multiple crime/offense types
Promising - More than one Meta-Analysis Drugs & Substance Abuse - Multiple substances
Promising - One Meta-Analysis Education - Multiple education outcomes
Promising - One Meta-Analysis Mental Health & Behavioral Health - Psychological functioning



Dropout Prevention Programs
School- or community-based programs targeting frequently absent students or students at risk of dropping out of school. These programs are aimed at increasing school engagement, school attachment, and the academic performance of students, with the main objective of increasing graduation rates. The practice is rated Effective for reducing rates of school dropouts, and rated Promising for improving test scores/grades, graduation rates, and attendance.

Evidence Ratings for Outcomes:
Effective - More than one Meta-Analysis Education - Dropout
Promising - One Meta-Analysis Education - Academic achievement/school performance
Promising - One Meta-Analysis Education - Graduation
Promising - One Meta-Analysis Education - Attendance/truancy
bottom border


Program Snapshot

Age: 11 - 14

Gender: Both

Race/Ethnicity: Black, Hispanic

Geography: Urban

Setting (Delivery): School, High Crime Neighborhoods/Hot Spots

Program Type: Academic Skills Enhancement, Afterschool/Recreation, Classroom Curricula, Mentoring, School/Classroom Environment, Truancy Prevention

Current Program Status: Active

Listed by Other Directories: Top Tier Evidence Initiative

Program Director:
Harlem Children’s Zone
35 East 125th Street
New York NY 10035
Phone: 212.360.3255
Fax: 212.289.0661
Website
Email

Researcher:
Will Dobbie
Public Policy Doctoral Program, Harvard Kennedy School
79 John F. Kennedy Street
Cambridge MA 02138
Email