What is a pilot impact study?
Pilot impact studies investigate an intervention’s potential for improving its intended child outcomes. In contrast to step 4, which is about determining whether the intervention or service can achieve its outputs, the pilot impact study considers an intervention or service’s potential for achieving its intended outcomes.
Why is it important?
Pilot impact studies are the first time the outcomes identified in your theory of change will have been tested, providing preliminary evidence as to whether your intervention or service has the potential to improve outcomes for children. They are relatively inexpensive, and therefore provide a useful insight into whether it would be appropriate to fund a robust impact evaluation of the intervention or service, as these are much more expensive and often require the commissioning of external evaluators.
Pilot studies are also particularly useful for determining which measures are most appropriate for testing child outcomes for your intervention or service, as well as how to best recruit and retain a sufficiently large and representative study sample.
Key principles of a pilot impact study
Pilot studies should aim to include the following aspects:
- Measure at least one of your the short-term outcomes in your theory of change, both before and after participants have taken part in the intervention or service.
- Use established assessment tools that demonstrate both validity and reliability. These measures should be consistent with the short-term outcomes in your theory of change and appropriate to your target participants.
- Carefully consider recruitment and retention of participants. You want the sample to be large enough that you can be confident it is representative of the target population of your intervention or service.
- Use an appropriate test to identify whether any observed changes in the pre- and post-assessment measure were the result of an actual change rather than chance (this is called testing for statistical significance).
- Report the results in a transparent and thorough way, including details of the methods used and the demographics of those who participated. Ensure you acknowledge the limitations of the approach used.