The Heritage Lottery Fund (HLF) commissioned an independent review of the self-evaluation process and outcomes achieved for 326 completed Heritage Grants projects funded under HLF’s Third Strategic Plan (SP3), which ran between 2008 and 2013.

This review includes a comparative appraisal of the quality, scope and methodology of the self-evaluated reports against six criteria and the type, range and quality of activities and outcomes achieved by completed projects, including a mapping of the impact of their work onto HLF’s current framework of 14 outcomes for heritage, people and communities.

Key findings

Quality of the Self-evaluations

The quality of the submitted self-evaluation reports was assessed on a four-point scale (excellent, good, adequate or poor) using six criteria. These six criteria focused on the extent to which the evaluation:

1. provided a logical framework
2. included appropriate and methodological ways of providing robust evidence
3. demonstrated that data was subject to robust analysis and provided evidence on outcomes
4. was objective and free from bias
5. presented the results clearly
6. included sufficiently clear conclusions and recommendations to enable stakeholders to apply any lessons learned

Overall just over a third, 37%, of reports were graded as good or excellent with just under two thirds, 63%, falling within the adequate or poor categories.

The aggregated findings in this report show that the quality criteria scores have significant dependence on several characteristics, confirmed by a range of statistical tests:

• Evaluation was external: external consultants/organisations tended to write better quality reports.
• Planned share of grant initially allocated to evaluation: where higher amounts of expenditure had been originally allocated for evaluation, the reports tended to be of better quality.
• Report length: longer reports which contained more explanation and more data, tended to be of better quality than the shorter reports.
• The number of HLF outcomes assessed as being met: better quality reports tended to record a greater number of outcomes being met.

Outcomes review

The most commonly mapped outcomes were all outcomes for people and for heritage. There was evidence in over three-quarters of reports for:

• People will have volunteered time (87%)
• Heritage will be better interpreted and explained (79%)
• Heritage will be in better condition (79%)
• People will have learnt about heritage (78%)
• People will have developed skills (77%)
• People will have had an enjoyable experience (76%)

The least commonly reported framework outcome was that a project had helped to boost the local economy, with only 15% of reports containing evidence of this.