Identifying High-Quality Feedback Using Automated Text Classification: An Examination of Principals’ Written Feedback in a Statewide Evaluation System
Gegenheimer, Karin
0000-0002-6884-5558
:
2022-07-15
Abstract
In the past decade, many states and districts have restructured their principal evaluation policies to incorporate formal feedback structures, including observations and post-observation feedback. Research has yet to explore the nature of the feedback that principals receive through formal evaluation. Using statewide administrative data from Tennessee, including micro-level data on the written feedback text that principals receive from evaluators, this dissertation employs a combination of machine learning and regression-based approaches to identify the quality of principals’ written evaluation feedback, explore variation in feedback quality, and examine the extent to which feedback quality is associated with principal attitudes and behaviors. I draw on a large body of literature in organizational psychology and resource management to create a taxonomy of high-quality feedback. With this taxonomy, I use automated text classification, a machine learning technique, to construct measures of feedback quality from the written feedback text, which I then employ in regression-based analyses. Findings indicate that principals’ evaluation feedback largely does not meet quality standards. More effective principals are less likely to receive high-quality refinement (constructive) feedback but are more likely to receive high-quality reinforcement (affirmative feedback). More experienced principals are less likely to receive high-quality reinforcement feedback. Principals in high poverty schools are less likely to receive either type of high-quality feedback, while principals in schools with a larger Latinx student population are more likely. Black evaluators are less likely to provide high-quality refinement or reinforcement feedback, highlighting a need for future work to investigate potential racialized role expectations among principal evaluators. Feedback quality is not significantly associated with principals’ perceptions of evaluation or job satisfaction. Some, but not all, qualities of feedback are associated with principal mobility; different feedback qualities are associated with different mobility patterns. The dissertation concludes with a discussion of implications for policy and practice.