Robust impact evaluations allow you to confidently state whether your intervention or service has had an impact on children’s outcomes (establishing causality) and how much of an impact it has. They do this by isolating the effect of an intervention from all the other factors that could impact on the outcomes you are evaluating.
Good impact evaluations analyse data collected at two time points – at a minimum, data is collected before and after the intervention or service – and for two different groups, one which received the intervention and one which did not. This approach is only considered robust if the two groups being compared are broadly similar in all important respects. Random assignment, when done properly, helps to ensure equivalence between the two groups being compared, and means that any difference observed in the post-test scores should be attributable to the intervention rather than to differences in the characteristics of the two groups.
In our view, the only acceptable designs for measuring impact robustly and establishing causality are randomised controlled trials (RCTs) or quasi-experimental designs (QEDs). RCTs involve randomly assigning participants into an experimental group (who receive the intervention) or a control group, who do not. QEDs take non-experimental data and transform it using sophisticated statistical techniques to allow comparison between those who received an intervention and those who did not, and to test for causality.
Are you ready for impact evaluation?
Impact evaluations require a significant amount of resource and technical expertise. They are time-consuming and expensive to run. For these reasons, you will need to be sure that your service or intervention is ready for this step. We therefore recommend that you are able to confidently answer ‘yes’ to the following eight questions before you consider yourself ready to move onto robust impact evaluations.
- Does the intervention or service aim to improve outcomes that are important for children’s development (see step 1)?
- Does it have a clearly defined target population that specifies the age of the children and their level of need (see step 1)?
- Is it likely to provide measurable value for this population over what is currently available (see step 1)?
- Can it be feasibly delivered to a high standard (see step 4)?
- Are the intervention’s costs known, and are they proportionate to the intervention’s intended outcomes (see step 4)?
- Is it possible to recruit participants from the target population at sufficient numbers to justify the cost and time of the intervention (see step 4)?
- Is it possible to retain participants from the target population at sufficient numbers to justify the cost and time of the intervention (see step 4)?
- Does the intervention have preliminary evidence from a pilot impact study meeting the minimum step 5 requirements of improving at least one of its intended child outcomes (see step 5)?
So, you are ready for impact evaluation
If you can confidently answer yes to all these questions, it is likely you are ready to robustly measure the impact of your intervention or service. There are numerous ways for doing this. In our guide ‘10 steps for evaluation success’ we talk about randomised controlled trials: step 6 focuses on efficacy trials (used to determine if there is an impact under ideal conditions) and step 7 focuses on effectiveness trials (used to determine if the results from an efficacy trial can be replicated in real-world conditions).
These methods require significant technical expertise to undertake correctly and are highly resource intensive. For these reasons, we strongly recommend that you buy in this expertise by commissioning an external evaluator with experience in impact evaluation, such as a local university, to independently undertake the evaluation.