Laura Fox, senior education adviser at the Education Development Trust, explains the different ways that the impact of in-school interventions can be measured in your schools.
Measuring impact: diagnostic improvement or summative evaluation?
It stands to reason that the continuation of any in-school intervention will be determined by an evaluation of impact. With tightly bound budgets, a strategy is more likely to receive prolonged attention and investment if it will lead to improved pupil outcomes, but measuring this can be difficult.
Many schools will follow a systematic evaluation cycle which includes planning, implementation, evaluation of impact, and modification. Yet frequently, little time is left to evaluate, and the opportunity to adjust and improve is limited to a single point at the end of delivery, before the cycle is repeated. By this time, unless pre-planned, there is a limited amount of useful data and any subsequent adaptations will only impact future cycles.
As an example, 1:1 and small group interventions can be highly effective in supporting pupil progress, especially for disadvantaged pupils or those with SEND. In order to measure impact and inform future planning, a school leader is likely to undertake an evaluation which draws on pupil attainment and progress data. While these are useful metrics, overreliance on a summative data point can restrict its usefulness, with both quantitative and qualitative measures proving problematic.
- Firstly, it’s difficult to isolate and accurately measure the many factors that can impact a pupil’s progress, so the validity of quantitative data may be comprised in search of a proof of need
- Secondly, although qualitative data can provide a closer insight into how an intervention was implemented and the perspectives of those involved, sources such as pupil questionnaires and teacher feedback may be captured and/or analysed in hindsight, restricting their reliability
In both cases, inaccurate data analysis may lead to flawed decision-making, and any improvements to delivery only have an impact after their time of need.
A more effective approach is to synergise diagnosis and improvement throughout a programme’s delivery, to have a greater impact during and after implementation. This semantic shift from ‘evaluation’ to ‘diagnosis’ reduces the risk of metric fixation as a broad range of valid data sources can be examined on an ongoing basis, leading to proactive and responsive improvements throughout – increasing the likelihood of improved pupil outcomes.
Data collection methods should be chosen according to the information required to facilitate decision-making and action. For example, school registers can provide an insight into attendance patterns. However, when triangulated with pupil surveys and an informal learning walk, the same data informs a closer analysis of pupil engagement and the quality of the intervention. The data collection doesn't need to occur concurrently but can be carefully mapped to provide insights, without being intrusive to the teaching and learning taking place. Equal attention should be given to any subsequent adjustments, such as parent/carer engagement to address attendance concerns, or additional professional development opportunities to improve the quality of the intervention.
An effective improvement framework might not remove the need for a summative evaluation of overall impact, but capturing and analysing data throughout will speed up the process. Furthermore, when multiple sources of data are intentionally and continuously analysed to inform the evolution of a strategy, this improves the likelihood of achieving its impact-led objectives.
All views expressed in this article are the author's own. Any mention of commercial providers, resources or products is on the author’s recommendation and should not be considered an endorsement by The Key.
This article featured in the April edition of Trust Matters, The Key's newsletter for trust leaders, if you'd like to read future articles sign up to receive the monthly editions here: https://key.sc/trustsignup