Higher Achievement has been the subject of two randomized controlled trial studies, both of which demonstrate statistically significant effect in academics. This research places Higher Achievement in a select set of nonprofits. Only 2% of nonprofit organizations have undergone these expensive and cumbersome RCTs, and of those only 10% have positive results.
The results of this study were published by MDRC in 2013, with rare, positive findings in three areas: test scores, high school placement, and family engagement. These ground-breaking results led to the largest investment that the U.S. Department of Education has ever made in an after-school program: a $12 million i3 grant over five years.
This grant led to a second RCT, which was published by MDRC in July 2020. Once again, Higher Achievement is among the few with positive, statistically significant results – this time with report card grades. This evaluation also included a robust implementation study, which has led to a refined program model – focused on placement in and readiness for college-preparatory high school programs. The refinements expand on the centerpiece of Higher Achievement’s model – Afterschool Academy and mentoring, dedicates more time for family engagement, focuses on partnerships for summer learning, and reorganizes the staff to deliver high quality, consistent programming. We call this approach Higher Achievement 2.0.
Summary of Findings from the 2020 RCT:
- Report Card Grades – Positive Effects: This study demonstrates that our program demonstrated statistical significance in report card grades (math, reading, science, and social studies) after two years in the program, compared to the equally motivated control group (who also interviewed for Higher Achievement). The p-value for overall GPA is .006, which indicates a high level of confidence. These effects were witnessed in the three new cities to which we expanded – Baltimore, Richmond, and Pittsburgh[1].
- Strongest effects on students who started Higher Achievement on grade level, which is the group for whom the program was designed.
- Outsized positive effect on Black boys’ math grades. Without Higher Achievement, boys’ math performance fell much more than it did for girls, with a p-value of .11.
- Grades linked to life success: researchers note that report card grades are particularly important indicators for longer-term success in life because they combine both content knowledge and social/emotional skills. This link is embodied by the growing movement of test-optional universities, not requiring SAT or ACT, in part because of the evidence that grades are more indicative of preparedness.
- Test scores were positive in Year 2, but not statistically significant. The first RCT study did demonstrate statistical significance in test scores; however, the conditions were different. The first study included an independently administered Stanford 10 exam. The second study looked only at state-administered tests, across four states, which introduced variation to the sample.
- Implementation Study Findings: The report describes the success and challenges that we faced during these pivotal years of growth, as the researchers note that the challenges are common in the nonprofit sector. A summary of implementation study findings are below.
- Mentor Recruitment, Retention, and Training: Higher Achievement successfully recruited an educated group of volunteers to be mentors; however only DC sites met their mentor recruitment goals, leading to larger than the ideal 4:1 scholar: mentor ratios in some sites. Further, Higher Achievement succeeded in ensuring mentors attended the pre-service training, but fewer mentors attended the optional ongoing training on curricula delivery, and thus some of the instructional strategies were underutilized.
- Action plan: The “One Team” staffing model in Higher Achievement 2.0 includes a pivotal new position that is responsible for both mentor recruitment and training in curricula. Further, three staff have been promoted to lead mentor recruitment, with a six-month strategy, supporting by externally facing executive directors who manage the high level corporate and university partnerships.
- Center Director Retention: The interviews suggested that center staff were considered strong overall, but retaining experienced staff for multiple years proved difficult. Over the three year study, more than half of the centers had three or more directors. This turnover leads to challenges with cementing school partnerships, retaining scholars, and instilling a cohesive culture.
- Action plan: Feedback from Center Directors (CDs) informed the HA 2.0 program model. Specific CD-informed elements included: more time for planning in the summer, more time spent on family engagement, and the development of the Center Support Team to offer high quality and consistent resources, trainings, and support in mentoring, curricula, academics, social emotional learning, data analysis, and high school placement.
- Difficulty with Summer Academy delivery: While operating a three to four days/week Afterschool Academy was doable, offered the five days/week summer program proved logistically challenging (access to school buildings, lack of air conditioning). Further, while the youth development elements of the summer academy were engaging, the instruction was inconsistent and difficult to individualize with larger ratios (16 scholars : 1 teacher). Lastly, student attendance tended to be lower (76%) in the summer, compared to 83% after school.
- Action plan: Based on findings from the implementation study and input from staff, Higher Achievement 2.0 reimagines summer. We are expanding on our core competency of Afterschool Academy (adding seven more weeks per year) and offering college trips and individual family advising in the summer, focused on high school placement – starting in 5th grade. Instead of offering a Summer Academy with an entirely new set of teachers, Higher Achievement has developed a partnership with BellXcel, which offers a rigorous, proven five-week summer program in each of our communities.
- Student Attendance and Engagement: Eighty-three percent of students who had access to Higher Achievement report attending regularly, which is notable, especially for middle grade students, who are more independent. However, 40% of staff reported challenges in scholar retention and attendance. Staff and mentors also reported that the curricula at the time contributed to disengagement and limited relationship building.
- Action Plan: A working group of Higher Achievement staff, mentors, alumni, and school partners spent the year of 2019 reviewing and preparing to implement a new mentoring curricula. The new curricula is from open sources in Humanities and STEM, and it is much more engaging and aligned to our social justice pillars. Further, the new, deeper approach to family engagement and high school placement in the summer is intended to increase scholar retention.
- Mentor Recruitment, Retention, and Training: Higher Achievement successfully recruited an educated group of volunteers to be mentors; however only DC sites met their mentor recruitment goals, leading to larger than the ideal 4:1 scholar: mentor ratios in some sites. Further, Higher Achievement succeeded in ensuring mentors attended the pre-service training, but fewer mentors attended the optional ongoing training on curricula delivery, and thus some of the instructional strategies were underutilized.
It is a core value of Higher Achievement to put data into action. With this third party study, the robust data and insights have prompted an updated version of our program model – Higher Achievement 2.0, elements of which are described in the action plan summaries above.
We implement these changes in a vastly changed landscape – of COVID 19, virtual school, and deep racial justice reckoning. We stood up Virtual Higher Achievement on March 30, and have propelled four rapid improvement cycles through September. A summary of our early virtual program is here, our commitment to racial justice is here, and our description of update virtual program for SY20/21 is here.
Now and always, we will listen, analyze data and research, and adapt – to best serve our scholars.
[1] Due to significant data collection challenges with the large number of DC students who attend charter school, data for the DC Metro flagship could not be considered in this second study. The first study, published in 2013, was exclusively focused on DC Metro, and demonstrated statistically significant effects.