Researchers interviewed a nationally representative sample of large and medium sized districts. They followed a structured interview protocol and then analyzed the results. 

Researchers interviewed superintendents or other senior district officials knowledgeable about district efforts to prepare, hire, and support school principals in 192 school districts. Interviewees included 55 superintendents; 88 associate, assistant, or area superintendents and 74 leaders with other titles, such as executive director, chief, or director.  In 15 cases, researchers  talked with more than one person in a single district interview. In those cases, they recorded a single response for the district. Any reference to district respondents in this report refers to the collective responses for these 15 cases and not individual interviewees. The overall response rate was 23 percent. This response rate reflects hard refusals as well as nonresponses or inabilities to complete the interview within the short data-collection window.

The 175 10K+ district respondents were somewhat more urban, served fewer white students, and served more free or reduced-price lunch (FRPL)–eligible students than 10K+ districts nationwide. The researchers used survey weights to further improve the extent to which the 10K+ respondent sample matches the population of 10K+ districts on select student enrollment characteristics. Researchers  then used the survey weights to develop the weighted average responses presented in this report. Appendix B of the report has more details about the sample characteristics and weighting.

Questions for Interviewees

Researchers  used a structured interview protocol to gather information about the following topics of interest efficiently and systematically:

•  the extent to which the district views school principals as a key lever for school improvement
•  district satisfaction with the current pool of candidates for principal vacancies
•  whether the district is engaged in specific pipeline activities to prepare, hire, support, and retain principals and the nature of their engagement in these activities
•  the level of interest in specific pipeline activities and perceived barriers to implementing them.

The protocol included closed-ended questions that were limited to a yes, no, or categorical responses and a more-limited number of open-ended questions to gather richer information from district respondents about what these pipeline activities look like on the ground, the level of interest in enhancing key pipeline activities, and what barriers they perceive to implementing these activities. It was designed to be completed in 20–30 minutes.Researchers made a conscious effort to limit the use of jargon from the Principal Pipeline Initiative (PPI) while addressing the core concepts of that effort. The interview protocol and approach were reviewed and approved by the RAND Corporation’s Institutional Review Board.

How Data was Analyzed

Responses to open-ended questions were categorized using predefined codes reflecting expected themes. These codes were refined in the first several weeks of interviews based on topics discussed in the interviews. Interviewers conducted all the coding of open-ended responses, and two researchers deeply familiar with principal pipelines double coded 11 percent of the interviews to assess and ensure coding reliability. Any substantial disagreements in the coding were discussed and resolved at weekly meetings with all interviewers and the research team; these meetings were also a chance to make any necessary updates to the coding plan and the coding itself.

Researchers weighted average responses to close-ended questions and open-ended responses that were coded and quantified for the 10K+ districts. They generated and analyzed tabulated responses for two key subgroups of the weighted 10K+ sample: medium districts (those serving 10,000–49,999 students) and large districts (those serving 50,000 or more students). They also generated tabulations of unweighted responses from small districts to support an exploratory analysis. Finally, they used a regression model that accounted for some district-level factors to examine whether the reported presence or number of specific pipeline activities was associated with the likelihood of reported satisfaction with the principal candidate pool.

Limitations and Strengths

The national estimates reflect findings based on 175 responses from 10K+ districts (i.e., excluding the 17 responses from small districts). These findings rely on self-reports from senior district leaders who are responsible for district efforts to prepare, hire, and support school principals. Because the findings from the PPI evaluation have generated some attention in the press and professional organizations in the education field, it is possible that respondents provided socially desirable responses that emphasize their work in this area, even if their efforts are relatively minor. 

Concurrent with the interview effort, The Wallace Foundation was supporting technical assistance (TA) on principal pipelines for school districts around the country (which is a separate effort from the Wallace-funded PPI that took place from 2011 to 2016). All large districts were invited to participate in the TA, and about one-half chose to do so. The Wallace Foundation also invited some medium districts to participate in the TA effort. The study included all large districts in the country, regardless of whether they chose to participate in the TA. At the request of The Wallace Foundation, researchers excluded from the study medium districts that chose to participate in the TA out of concern about the potential burden that participation might pose for these smaller districts. This means that the sample of medium districts is potentially biased away from districts with the greatest interest in pipeline activities. The response rate was 23 percent overall and varied by district size category and TA participation (see Table A.1 in Appendix A of the report and the related discussion). The response pattern suggests a possibility of unobservable response bias in the sample, with district officials in larger districts and those participating in the TA potentially being more favorably inclined toward or more knowledgeable about principal pipeline activities.