A mixed-methods investigation of how schools in an urban district respond to student benchmark data under shifting accountability incentives

Abstract

This mixed-methods dissertation uses benchmark data to explore how schools in a large urban district responded to shifting accountability policy incentives as the state changed from No Child Left Behind (NCLB) to a waiver accountability system. For five years, the district assessed third through eighth grade students three times per year in math and reading, assigned students labels based on their benchmark scores, and shared that information with schools. In the first year of the waiver, the district provided additional funds to a select group of schools so they could provide targeted interventions to students before the state test. Chapter 1 describes the theoretical and empirical background, the state accountability context, and the district benchmark policy. Chapter 2 explores whether schools used benchmark test-score labels to engage in educational triage during NCLB (i.e., focused on “bubble” students close to proficiency at the expense of low- and high-achievers) and whether that changed during the waiver. Using combined regression discontinuity and difference-in-differences, I find that students labeled “Basic” gain significantly more than those labeled “Below Basic” during NCLB (i.e., schools focused on bubble students) and that lower-performing students benefitted after the waiver was adopted. Chapter 3 investigates three alternative explanations for the Chapter 2 results. This chapter finds that (a) a new educator evaluation system which used value-add metrics and was adopted the year before the waiver contributed to the gains found for low-achievers; (b) schools did not change the way they identified bubble students, which was explored by using district-provided labels which classified students close to proficiency as “priority” students; and (c) the additional funding did not explain the increased focus on low-performers. Chapter 4 gets inside the black box by qualitatively analyzing school proposals detailing how schools planned to spend those additional funds. Those documents indicate that schools were responsive to the waiver accountability incentives, targeting mainly bubble students with interventions. Only 25% of schools included low-achievers in those interventions. Surprisingly, a number of schools used the additional funds to facilitate triage. This dissertation concludes with a discussion of the results and lessons for policymakers, district leaders, and researchers.