The report is based on analyses of data from site visits, interviews, surveys, and teacher ratings of principal performance. 

• Site visits and interviews. Researchers conducted three rounds of semi-structured interviews with the superintendent and a sample of central office personnel, principal supervisors, and principals in each of the six initiative districts over the course of the initiative. Through the interviews, they aimed to understand districts’, principals’, and supervisors’ ongoing experiences as the district changed the principal supervisor role. 
• Surveys. Researchers administered surveys to all principal supervisors and principals in the participating districts three times during the initiative. The survey data provided the perspectives of a broader range of supervisors and principals regarding each of the initiative’s components. They also administered a survey to principal supervisors in a sample of other urban districts that were not part of the initiative during the final year of the initiative. 
• Teacher ratings of principals’ performance. Researchers measured teachers’ perceptions of principals’ performance using teachers’ ratings of principals from the Vanderbilt Assessment of Leadership in Education (VAL-ED), a principal evaluation tool focused on the types of instructional leadership practices and competencies emphasized by the initiative. They used VAL-ED data from the 2013–2014 through 2017–2018 school years to analyze the initiative’s effects on teachers’ perceptions of principals’ performance. The researchers compared scores between schools in the initiative districts and a set of comparison schools in other districts that also administered the VAL-ED in the relevant years.

Their analyses of data from site visits, interviews, and surveys focused on districts’, supervisors’, and principals’ experiences, successes, and challenges throughout the initiative; lessons learned; and plans for sustaining the accomplishments of the initiative after it ended. The researchers integrated the descriptive survey results with the qualitative data.

To provide rigorous evidence of the effects of the Principal Supervisor Initiative on teachers’ perceptions of principals’ performance, they first used propensity score matching to identify a comparison group of schools that was similar to the schools in the initiative districts before the start of the initiative. They then tracked and compared principals’ VAL-ED scores in the two groups over time to estimate the initiative’s effects on teachers’ perceptions of principals’ performance. In addition, they examined how teachers’ perceptions of principals’ performance and the initiative’s effects varied across the initiative districts and the schools in these districts. They also used survey data to construct measures of initiative implementation factors, including implementation of specific components of the initiative; principal supervisors’ time spent on instructional leadership; and supervisors’ effectiveness. They examined how these factors related to the initiative’s effects on teachers’ perceptions of principals' performance. The full report includes a chart summarizing the data used for each research question.

Limitations 

Although the researchers believe their approach to estimating the Principal Supervisor Initiative’s effects provides the most rigorous possible estimates, it has some limitations.

First, the researchers were only able to examine one outcome measure: teachers’ perceptions of principals’ performance as measured by the VAL-ED. Although the VAL-ED is a valid and reliable measure of teachers’ perceptions of principals’ performance, it may not capture all dimensions of principal’s performance. The initiative could have affected other aspects of performance not captured by VAL-ED. It could also have affected other outcomes of interest to districts, including principal retention or satisfaction. The researchers were not able to measure these outcomes in both initiative and comparison schools, so they could not include them in their analysis of the effort’s effects. 

Second, the researchers cannot draw definitive conclusions about whether the initiative caused any changes they found in teachers’ perceptions of principals’ performance, and they must therefore interpret the estimated effects with caution. Their matching procedure matched schools in the initiative districts to similar schools in non-initiative districts. Although they were able to match schools on a rich set of variables that are likely to be correlated with changes in teachers’ perceptions of principals’ performance and district participation in the PSI, there could be other characteristics they did not observe in their data that were correlated with changes in teachers’ perceptions of principals’ performance. To the extent that such characteristics were excluded from their matching models, the matched comparison groups might not have provided an accurate estimate of how principals in the initiative districts would have performed had they not been in an initiative district.