The 2020-21 Future Forward Literacy Program: Implementation and Impact during the COVID-19 Pandemic
Future Forward is an early elementary literacy program that integrates one-on-one tutoring with family engagement to support literacy skill-building and the social development of students. In the 2020-21 school year, as part of an Education Innovation and Research (EIR) Mid-Phase grant, the impact of Future Forward on reading achievement was tested with a randomized control study of students in nine schools. Although not statistically significant, the overall impact of Future Forward was estimated at between 0.09 and 0.10 standard deviations, depending on model specifications. A smaller number of schools and implementation challenges, both
resulting from COVID-19, hindered our ability to measure the impact of Future Forward. Even considering the reduced power of the study and implementation challenges, we found strong evidence that Future Forward had a positive impact on underserved students facing more challenges learning to read. Future Forward had roughly three times the impact on Black students (0.34 standard deviations, p =.095) and five times the impact on Black male students (0.54 standard deviation, p =.052) than the overall impact. Black male students with more developed reading skills benefitted even more from their participation in Future Forward (0.74 standard deviations, p < .001).
School perspectives of equity – WEERP Report
In this report from the Wisconsin Educator Effectiveness Research Partnership, school administrators and teachers share their experiences promoting equity in their schools. The results suggest some schools see promoting equity as a critical need. However, most other schools are only somewhat focused on it. Schools engaging in equity has typically involved superficial efforts such as book groups. Few schools have engaged in the systemic changes necessary to promote equity.
Evidence of Discrimination and Bias in the Effectiveness Ratings Assigned to Wisconsin Educators of Color
This study is the first in a series examining the bias and discrimination affecting Wisconsin educators of color. In this study, we examine statewide effectiveness ratings data of over 55,000 educators for evidence of bias and discrimination. The results of this study suggest ratings assigned to educators of color are discriminatory. Administrators view White female educators as the most effective, with Black and Asian male educators viewed as the least effective; 89% and 78% of White female educators are rated as more effective than the average Black and Latinx male educator, respectively. This was true even when comparing the ratings of educators with the same credentials and in the same schools. However, the performance appraisal process is likely just the tip of the iceberg regarding the negative impacts of bias on educators of color. The ratings reflect underlying biases that affect their experiences in ways that go deeper than the scope of this paper.
MPS GEAR UP Evaluation Report: AVID Program
The AVID program is designed to complement and support students who are in the middle tier academically at their schools who would benefit from extra supports to enroll and succeed in college. This is done through enrollment in the AVID elective, which builds students’ study skills, organizational skills, and critical thinking skills.
Interviews suggested benefits to students who take AVID. These related to individual skill-building, relationships built through being a part of an AVID cohort.
MPS GEAR UP Evaluation Report: Gearing Up for a Great Senior Year
Gearing Up for a Great Senior Year (GUGSY), one of several offerings of TEAM GEAR UP, has two core objectives: to better prepare students for their senior year and to improve students’ likelihood of successfully transitioning to college or a career. GUGSY was first offered in August 2016 for students entering their senior year in 2017 at all eight GEAR UP sites. Programming occurred over three consecutive days, with content focused on developing college and career readiness skills across four areas: resume building, college applications, college application essays, and local scholarships.
Pre-program survey results confirmed that while participants had some confidence related to the steps required to go to college, there was a need for education and support related to the four areas.Observations confirmed that GUGSY facilitators provided information and resources across each of the four content areas. Results of the post-program survey showed increased knowledge in each area tested in comparison to pre-program surveys. Participants felt they were significantly more prepared for their senior year of high school. Participants were also more confident they would do well their senior year, transition to college successfully, and to create a good resume.
MPS GEAR UP Postsecondary Impact Study
As part of our evaluation of the MPS Gaining Early Awareness and Readiness for Undergraduate Programs (GEAR UP) grant, we analyzed the impact of GEAR UP participation on a key component of the grant: secondary and postsecondary outcomes. Overall, students in GEAR UP were more likely to graduate high school on-time but not more likely to enroll in a postsecondary institution. However, students who participated in more types of GEAR UP activities were much more likely to graduate high school and attend a postsecondary institution. GEAR UP students, overall, were not more likely to enroll in MATC or UWM. Again however, students who participated in more GEAR UP activities were. Regarding the performance of GEAR UP students attending MATC or UWM, the findings were mixed. The positive of this study at least suggest that GEAR UP likely contributed to the improved postsecondary success of MPS students.
A GEAR UP study for identifying strategies for increasing access and success for MPS students in Advanced Placement courses
The purpose of this study was to collect information from a variety of educators in the Milwaukee Public Schools about aspects of advanced placement (AP) courses that both promote and inhibit student participation and success. This was done with the hope that this information would prove useful to Team GEAR UP in its efforts to both increase participation and improve how well students perform in AP classes. Historical AP data suggest that fewer than 25% of students participate in AP and nearly half of all AP classes taken by students in Team GEAR UP schools result in either a D or U grade. Thus, it is critical that any efforts to increase participation in AP also include efforts for improving how well students perform in these rigorous classes. Interviews with AP teachers and other educators involved in AP were used to inform these efforts.
- Schools should provide more opportunities for teachers to collaborate across grade levels and within departments, as this will allow schools to create more consistent systems of enrollment and support for AP students, as collaboration allows for a more clear understanding of how AP classes fit with other school offerings.
- MPS must address large obstacles that students face in AP classes. The primary issues mentioned by teachers include providing increased academic supports, increasing family engagement, ensuring AP alignment and readiness, enhancing study skills and time management, and building an AP culture within a school.
- Schools should work collaboratively with district leadership, staff, students, and families to examine and eliminate barriers to student engagement and performance in AP classes.
- Schools would benefit from engaging students and families in determining not only the most beneficial type of support, but also when and how to offer extra assistance. This would be helpful in offering more effective targeted supports that take into account the lived realities of students and families in Milwaukee.
MPS Report on the impact of the GEAR UP Credit Recovery Program
Credit Recovery was a targeted outreach and student support effort by Team GEAR UP to assist 9th and 10th grade students to recover failed credits necessary for promotion or graduation requirements. Credit Recovery was conducted in 2013-2014 and expanded in 2014-2015 at eight MPS high schools.
The implementation of Credit Recovery in 2014-2015 was done with attention to how students and teachers were recruited, which classes were offered, how students and school administration were supported, and with thought on how to engage parents. Planning for 2014-2015 Credit Recovery began well in advance, with a thorough review of thousands of students transcripts.
Credit Recovery met the goals of having students pass courses and earn credits. The Credit Recovery program in 2015 met its goals of having students pass courses and recover credits. MPS is a school district in which a high percentage of high school students fail classes. While this trend continues, we recommend Credit Recovery as an effective offering by MPS to help high school students recover credits and avoid falling further behind students in their cohort.
As a result of the success of this GEAR UP pilot, MPS adopted Credit Recovery as an option for all 9th and 10th grade students district wide.
The Sustained Impact of Future Forward
Future Forward is an early primary program that leverages both one-on-one tutoring and parent engagement to develop the literacy skills of struggling readers. In 2010, Future Forward was awarded an i3 grant, under the name of SPARK, to develop and test its impact in seven Milwaukee Public Schools primarily serving low-income Black and Latinx students. The results of a randomized control trial evaluation found Future Forward had positive and statistically significant impacts on foundational literacy, reading achievement, and school attendance (Jones & Christian, 2020). In the current study, we follow study participants five years past the end of the study, from 2016 to 2020, to determine if the benefits gained were sustained.
The results show that former Future Forward participants continued to score higher on the school district’s benchmark Star Reading assessment after the i3-funded study and program ended. The sustained impact of Future Forward was equivalent to approximately 1/3 year of reading development according to national norms, and 1/2 year according to local norms. We also found that former Future Forward students were less likely to be placed into special education. Considering the added cost associated with providing specialized services to ten additional students, Future Forward could ultimately save MPS over one million dollars.
Although the results of this study are positive, it is important to put the impact of Future Forward in the context of a school system that is failing students of color. Although Future Forward had a large sustained impact on reading achievement, former Future Forward participants were still reading several years below grade level by middle school. So while the impact of Future Forward is real, it must be a part of larger efforts in educational resource redistribution to earnestly provide what is owed to students and families (Ladson-Billings, 2006).
Impact evaluation of the 2019-20 Future Forward Program
In 2017, Education Analytics (EA) received an Education Innovation and Research Mid-Phase grant from the U.S. Department of Education to expand FF to 14 schools in seven school districts across three states. The randomized control trial study of the 2019-20 FF program was designed to examine the effectiveness of its scale-up to 14 schools and its impact on reading achievement, social emotional learning (SEL), and school attendance. Regarding scale-up, nearly all participating students received one-on-one tutoring as intended. Fewer students received the full implementation of family engagement (FE) however. In spring of 2020, schools were shut down nationwide because of the COVID-19 pandemic. The disruption on schools limited our study to testing the impact of FF on school attendance only. FF was found to have a statistically significant, positive impact on school attendance. The school attendance of FF participants was 1.3 percentage points higher than students who received Business-as-Usual (BAU) literacy instruction. Further, its impact was greater with Black students, male students, and students with a history of low school attendance. Students at the intersection of these groups (Black male students with low school attendance) were particularly affected, improving their school attendance by six percentage points and 4.2 fewer days absent from
The Results from a Multi-Site Regression Discontinuity Impact Study of the Future Forward Literacy Program
Future Forward (FF) is an early elementary literacy program that pairs one-on-one tutoring, provided by volunteers, with parent engagement. Previously, FF, under the name of SPARK, was awarded an Investing in Innovations (i3) grant to test its impact in seven Milwaukee schools. Two randomized studies of FF/SPARK conducted as part of the i3 grant found FF/SPARK to have positive impacts on literacy, reading achievement, and school attendance. In 2017, Education Analytics received an Education Innovation and Research (EIR) Mid-Phase grant to establish FF in 14 additional schools across three states. During the 2018-19 school year, 12 schools participated in a multi-site regression discontinuity (RD) study of its impact on the literacy development of 222 students. While we did not find a statistically significant impact of FF, the magnitude of the impact estimate was similar to what was reported in the i3 studies. Limitations inherent to RD studies, and implementation challenges, hindered our ability to reliably measure the impact of FF.
Reorganizing the Professional Culture in Wisconsin Schools to Eliminate Racial Achievement Gaps
In this longitudinal study of Wisconsin schools, we examined the impact that schools with a strong professional culture have on the English Language Arts (ELA) and Math achievement of Black, White, and Latinx students. The results suggest much of the large achievement gap between Black, White, and Latinx students is attributable to inequities in the quality of schooling students receive. There are stark differences between the strength of the professional culture in schools serving Black, White, and Latinx students. These differences explain a great deal of why the achievement of Black and Latinx students lags so far behind that of White students. To put it simply, White students attend better schools. Given this, it is clear that the education system in Wisconsin represents a racist institution. However, this study demonstrates the impact that improving school professional culture has on achievement gaps. If more Wisconsin schools serving Black and Latinx students were organized to improve, with a strong professional culture, our school system would be more equitable, more anti-racist, and achievement gaps would be greatly reduced.
Wisconsin families share their experiences with home/remote learning during COVID-19
In March of 2020, the COVID-19 pandemic forced the closing of Wisconsin schools and education to move online. In response to this, as part of the Wisconsin Educator Effectiveness Research Partnership (WEERP), the Office of Socially Responsible Evaluation in Education (SREED) at the University of Wisconsin in Milwaukee worked with the Wisconsin-Minnesota Comprehensive Center for Region 10 (WMCC) and the Wisconsin Department of Public Instruction (DPI) to develop anonymous Family, Teacher, and Student Home/Remote Learning surveys. Districts can use these surveys to inform the development and monitoring of their Home/Remote Learning system. Both the Family and Student Home/Remote Learning surveys are available in English, Spanish, Chinese, and Hmong. The surveys are available to districts free of charge through links on our website at https://uwm.edu/sreed/distance-learning/. As of July 2020, 16 Wisconsin school districts had signed up to use the Family Home/Remote Learning Survey and 3,227 families had completed a survey.
A GEAR UP study about what predicts the college success of former MPS students?
As part of our evaluation of the MPS Gaining Early Awareness and Readiness for Undergraduate Programs (GEAR UP) grant, we analyze the connection between measures of high school performance, such as AP course participation, participation in GEAR UP programming, ACT scores, and GPA with first-year college GPA. Data were provided by the Milwaukee Public Schools (MPS), the University of Wisconsin in Milwaukee (UWM), and Milwaukee Area Technical College (MATC) for students who graduated GEAR UP schools between 2015 and 2018 and then attended either UWM or MATC.
- Although we did not find a connection between GEAR UP participation and college GPA, nearly all students who graduated high school and attended MATC or UWM had participated in several GEAR UP activities. This complicated our ability to measure impact and suggests GEAR UP programming was valued by nearly all students who ultimately went to college.
- Measures of high school GPA were the only unique predictors of of college GPA.
- Although students who participated in AP classes tended to have higher college GPAs, this relationship disappeared after accounting for high school GPA.
- Although students with higher ACT scores tended to have higher college GPAs, this relationship also disappeared after accounting for high school GPA.
View full report here GU Post Secondary Prediction Study
Focus Group of former GEAR UP Students Supported by the Educational Opportunity Program at Marquette University
In June 2019, eight Marquette University undergraduate students of color were convened for a focus group at the Educational Opportunity Program (EOP) in Coughlin Hall on the Marquette University campus. Students were asked to discuss their experiences in both programs and to make suggestions for how access to college and success in college could be supported more effectively for other MPS students like them.
MPS GEAR UP Final Report of Participation and High School Graduation
In this report, we summarize student participation in Gaining Early Awareness and Readiness for Undergraduate Programs (GEAR UP) grant across all seven years of the initiative in the Milwaukee Public Schools. 6,888 students participated in at least one GEAR UP activity. Over 90% of participants were either African American or Latinx and were from low-income families.
The improvement in graduation rates realized by specific GEAR UP schools was associated with how well they implemented the initiative. The two schools that demonstrated the greatest improvement in graduation rates, also demonstrated more intensive GEAR UP participation (students participated across more years and in more activities). This suggests that the impact of GEAR UP on high school graduation is locally determined by the effectiveness of a school’s implementation of the initiative.
Education in the time of COVID-19: Lessons from a school in Hong Kong
This report presents information gathered from one American international school in Hong Kong that began their transition to distance learning in February. As the COVID-19 pandemic continued to stress community and school resources, this school engaged in an internal continuous improvement and monitoring process. While its data collection activities were intended to inform internal decisions, the lessons learned from these efforts may be of use to other schools struggling through their own transition to distance learning while simultaneously dealing with the impact of a pandemic in their community. The hope is that this school’s experiences will provide educators in the United States guidance for how to accomplish and sustain this extremely difficult task.
- The school designed an effective distance learning process, using a variety of tools to facilitate learning.
- As the school remained closed, over time students began to feel more socially isolated.
- Teachers experienced a great deal of stress, both related to the workload required to implement distance learning and concerns about the safety of their families.
- Ongoing and consistent communication throughout the transition has proven critical for its success and the well-being of employees. Parents, students, and employees have also appreciated opportunities to share their experiences with administration.
- Teachers with children face a daunting task of both continuing their work as a teacher AND leading the home schooling of their children.
- Most teachers felt their instructional skills have improved as a result of the switch to distance learning.
Research Brief: The Utility and Cost of Requiring Wisconsin Teachers Pass the FoRT and edTPA assessments
In this study, conducted by The Office of Socially Responsible Evaluation in Education at the University of Wisconsin Milwaukee and the University of Wisconsin System, we explore the utility and impact of the requirement that preservice teachers meet a minimum score on the both the Educative Teacher Performance Assessment (edTPA) and the Wisconsin Foundations of Reading Test (FoRT).
WEERP Brief Nov 2019 – Race, Relational Trust, and Teacher Retention
In this report we explore relational trust as a possible explanation for: 1) why most Wisconsin schools do not have have any teachers of color and 2) the schools with the most students of color have the most difficulty retaining teachers.
We found African American teachers have lower trust with other teachers and are considerably more likely to transfer or leave public education than White teachers. This contributes to the shortage of African American teachers in schools across the state.
We also found White teachers reported less trust with African American or Latinx principals and between teachers, when fewer teachers were White. White teachers were more likely to transfer out of these types of schools, which generally have more students of color. This helps explain the teacher retention challenges experienced in schools comprised almost entirely of students of color.
The Impact of the Wisconsin Educator Effectiveness Process on New Teacher Turnover
In this report, we found the effective implementation of the teacher performance feedback process promotes the retention of new teachers. Providing new teachers accurate performance feedback led to improved trust with their principal, more school commitment and greater retention.
WEERP – Wisconsin Urban Leadership Institute Brief – September 2019
The five largest school districts began working with the Urban League of Greater Madison and the Wisconsin Department of Public Instruction three years ago to develop strategies for promoting equitable education opportunities for students of color. Their efforts resulted in the Wisconsin Urban Leadership Institute. This report presents the results from the first year of this important initiative.
The Impact of the Wisconsin Educator Effectiveness Process on Student Achievement
In this report, we found schools that improved aspects of the teacher performance feedback process also demonstrated improved student achievement results. Schools that increased the opportunities teachers have to use feedback and the amount that teachers used feedback improved their reading and math achievement results.
The Associations between How Schools use the Results of Teacher Evaluations, Principal Preparedness for Completing Teacher Evaluations, and Principal Perceptions of the Process
This report presents the results of a statewide survey of Wisconsin school administrators about their role in the evaluation of teachers.
SREed Brief July 2018 – EE and Job Satisfaction for Novice Teachers
In this brief we examine novice teacher perceptions of their principal and their job and a function of the implementation of their school’s Educator Effectiveness Process.
How are District Differences in the Implementation of Teacher Evaluations associated with Teacher Perceptions of School Leadership and Teacher Job Satisfaction?
We report on the results of a statewide survey of 24,000 teachers—44% of all Wisconsin classroom teachers—about their experiences with teacher evaluations and the feedback process, perceptions of principal effectiveness, and their satisfaction with their job. The results demonstrate a close connection between how school districts implement teacher evaluations and the perceptions of their teachers across a number of school social factors, including the usefulness of performance feedback, principal effectiveness and job satisfaction. Teachers in districts that provide adequate time and support to teachers to complete their evaluation process, along with useful and accurate performance feedback, perceive their principals to be effective leaders and are more satisfied with their job.
Measuring the Effectiveness of Wisconsin Principals: A Study of Wisconsin Framework for Principal Leadership Ratings
This study uses the results of teacher surveys to validate effectiveness ratings assigned to principals as part of the Wisconsin Educator Effectiveness (EE) process. These results suggest ratings reflect important aspects of principal effectiveness and leadership as experienced by teachers and that DPI has developed a process for measuring school leadership that accurately captures the experiences of educators in schools.
Measuring the Principal Effectiveness Equity Gap within Wisconsin School Districts
In this study, we explore the extent that effective principals are equitably distributed across Wisconsin. Within school districts, low-income and diverse schools disproportionally face challenges attracting and retaining effective principals. This results in less effective leadership in these schools. Any efforts to address equity gaps in access to effective teachers must also address equity gaps in access to effective principals. The effectiveness of teachers in diverse and low-income schools is largely determined by the effectiveness of leadership.
Exploring the Perspectives of Milwaukee Opportunity Youth through Photovoice
SREed worked in partnership with the Westlawn Re-engagement Center to conduct a Photovoice evaluation that engaged the voices of Milwaukee Opportunity Youth, young adults ages 16 to 24 who are disconnected from school and work, in developing and refining a pilot program to meet their needs.
Predicting AP Success in the Milwaukee Public Schools
SREed has been working with the Milwaukee Public Schools for a number of years to promote more equitable participation in advanced placement courses. As part of this work, this report summarizes the results of predictive analytics used to identify students with the potential to succeed in AP but who are not participating.
Teacher Job Satisfaction in the Context of the Wisconsin Educator Effectiveness System
This report shows schools that implement EE well have teachers who are more satisfied with their jobs.
Spark i3 Evaluation Study 2
The results of a two-year randomized control trial evaluation of the SPARK literacy model in the Milwaukee Public Schools, funded by an Investing in Innovations (i3) grant, found that it had a positive impact on student literacy, achievement and school attendance.
SPARK i3 Evaluation Study
The results of a two-year randomized control trial evaluation of the SPARK literacy model in the Milwaukee Public Schools, funded by an Investing in Innovations (i3) grant, found that it had a positive impact on reading achievement.