Growth Project Evaluation Strategy

What was the evaluation strategy? What data were examined from which sources? How were these analysed and portrayed?

The present report is a formative evaluation of the Growth Project. It seeks to: (1) make data-based judgements about the extent to which intended growth outcomes (derived from planning documents and goal statements) are being achieved over the defined timeframe, and (2) describe implementation processes and issues in relation to what was planned and what actually occurred within that timeframe.

Thus, the primary focus of the report is a portrayal of what transpired within the 2020/2021 school year because the local leaders (who engaged in the bulk of the work attributable to the project) worked from August 2020 through May 2021. Their work cycle began with their initial training, followed by an extensive recruitment-retention phase with targeted schools, then followed by a range of support and consultation activities, and finally winding out the year with preparation for the upcoming year. This sequence of activities constitutes an evaluable “unit of analysis”.  

Assessing the progress of the Growth Project within a school-year timeframe allows for the establishment of a baseline comparison for this first year of Growth Project operation (post-planning). It was determined that data from the 2019/2020 school year would serve as the baseline for direct comparison. In effect, implementation of the Growth Project began with the staffing and deployment of the initial cadre of local leaders.

Given the above assumptions, the focus of the draft evaluation report is on the activities and formative outcomes of the Growth Project with specific reference to the work of the local leaders within their regions during the past academic year. It should be noted that what transpired within the target regions is not exclusively attributable to the work of the local leaders since YSI has had an ongoing presence (normal operations) in each of the target regions. But the “added value” of the work of the local leaders to facilitate growth can be construed as the independent variable in evaluation terms.  

The report uses a multiple-method approach. It began with a review of numerous documents and work products that appeared in each of the local leaders’ folders. These included the slides and associated documents from their initial training, the documents they produced (unique to their regions and experience), communications tracker and weekly report data, and assorted other documents. A number of planning and programme description documents were also reviewed and discussed with their authors. Additionally, an extensive set of data located in various spreadsheets with YSI archival school and site data were obtained, and ultimately the YSI generic data system was accessed to verify and cross-tabulate outcome data. These were supplemented by contextual geographic and school data available from a variety of governmental and open access sources. All of these data were analysed and condensed to graphical portrayals in chart and map formats contained within the report. 

A highly useful source of qualitative implementation data was gleaned from approximately 14 hours of video interviews with local leaders about their experiences in the first year of programme operation. These were carefully edited into shorter clips to reflect key thematic findings in the last section of the report. Each clip concentrates on some aspect of the Growth Project in the voice of the local leader. Their commentary is both compelling and instructive in terms of what transpired on the “front lines”. 

But it must also be noted that these findings have limitations because there were several types of data that were not available (for various reasons). Site visits were impractical and there were many audiences and participants whose perspectives were not sampled due to resources and time limitation. Similarly, direct observation of transactions between local leaders and participants was not included in the design. Additionally, relevant demographic, geographical and other contextual data that may influence the processes being assessed are presently not available. Hopefully, such sources of data can be incorporated, at least to some extent, into future designs. 

Nonetheless, a clear and illuminating picture of what transpired during the first year of Growth Project operation (implementation) and the extent to which core growth goals were attained (formative outcomes) emerges from the data that was available.