Academic references to outcomes theory: Duignan (2009d)*
Decision-makers find it really useful because they can use it to select which programs they want to support using a common measure of success (e.g. dollars saved).
However, economic analyses can be 'rubbish in, rubbish out' if there's no clarity about the evidential status of the effect-size estimates being used.
Economic evaluation translates program outcomes into dollar values. This lets you compare programs with each other.
Even when there are no effect-sizes available from impact evaluation (building-block five) it is still possible to do economic analyses, but they should be clearly labelled as 'hypothetical'.
Below are the types of economic analysis that can be done.
1.1. Cost of intervention analysisCost of intervention analysis only looks at the cost of an intervention, not its effectiveness (cost-effectiveness is how much it costs to change an outcome by a certain amount) or the benefits (the result of subtracting the estimated dollar cost of the program from the estimated benefits of the program estimated in dollars). This analysis only allows you to say what the estimated cost of the intervention is (e.g. $1,000 per participant).
1. Analyses where no evidence-based impact/outcome evaluation effect-size information from outcomes theory building-block five is available above the intervention level..
1.2 Hypothetical Cost Benefit Analysis (using hypothetical effect-sizes for high-level outcomes)Where you cannot establish any effect-size from impact evaluation (building-block five ) above the intervention level, but you do have an estimate of the cost of the intervention (from 1.1 on left), you can just use hypothetical (assumed) effect-sizes in an analysis if the analysis is clearly labeled as such. To use this analysis, estimates need to be available for the cost of the intervention (from 1.1. on left) and the costs and benefits of all outcomes need to be reasonably able to be estimated in dollar terms. The effect-size of the main effect of the program on high-level outcomes is then just assumed to be at a specified level. This is an hypothetical cost benefit analysis (e.g. for a hypothetical effect size of 5%, 10% or 20%). It is essential that this type of hypothetical analysis is clearly distinguished from Analysis 3.2 which is based on actual estimates from actual measurements of effect-sizes derived from impact evaluation (building block five ).
This analysis allows you to estimate the overall benefit (or loss) of running the intervention if any of the hypothetical effect-sizes were achieved. For example, there would be a loss of $500 per participant for a 5% effect-size, a gain of $100 for a 10% effect-size and gain of $600 per participant for a 20% effect-size. In traditional cost-benefit analyses this is sometimes explored in a sensitivity analysis . But if there is considerable uncertainty about effect-sizes then the overall cost-benefit analysis should be clearly labled as hypothetical rather than just burying the uncertainty in a sensitivity analysis section.
2.1 Evidence-based Cost Effectiveness Analysis (empirical effect-size estimates available for mid-level steps/outcomes).In this analysis, estimates are available for the attributable effect-size of the intervention on mid-level steps/outcomes. When combined with the estimated cost of the intervention (1.1. above) this allows you to work out the cost of achieving a certain level of effect on mid-level outcomes (e.g. to get a 6% increased in X cost approximately $1,000 per participant). It is important that readers of such analyses understand that they looking at mid-level steps/outcomes and a visual outcomes model should be used to clearly indicate the level at which the effect is located.
2: Analyses when evidence-based impact/outcome evaluation effect-size information from attributing mid-level steps/outcomes is available.
3.1 Evidence-based Cost Effectiveness Analysis (empirical effect-size estimates available for high-level outcomes).Same as 2.1 except the cost of achieving a high-level outcome effect-size of a certain amount for a particular intervention can be worked out.
3: Analyses where evidence-based impact evaluation effect-size information for attributing high-level outcomes is available.
3.2 Evidence-based Cost Benefit Analysis (empirical high-level effect-size estimates are available for high-level outcomes).In this analysis, estimates are available for the cost of the intervention, its attributable effect on high-level outcomes, and the costs and benefits of all outcomes can be reasonably accurately estimated in dollar terms. If this information is not available this type of analysis cannot be done and your only option is to fall back to analysis type 1.2. This analysis lets you compare the overall loss or gain from running the program (e.g. the program cost $1,000 per participant and other negative impacts of the program are estimated at $1,000 while the benefits of the program are estimated at $2,500 per participant. Therefore there is an overall benefit of the program of $500 per participant.)
* This particular reference can be cited to refer to the material on this page.