POLICY AND PRATICE
Evaluation as a key part of health impact assessment: the English experience
L'évaluation, partie intégrante de l'étude d'impact sur la santé : l'expérience de l'Angleterre
El examen es parte fundamental de la evaluación del impacto sanitario: la experiencia inglesa
Robert J. QuigleyI; Lorraine C. TaylorII,1
IHealth Development Agency, London, England
IIHealth Development Agency, 330 High Holborn, London WC1V 7BA, England (email: lorraine.taylor@hda-online.org.uk)
ABSTRACT
Health impact assessment (HIA) is an approach that has experienced increased activity and wide support within England. This is reflected in the large number of HIAs being undertaken at local and regional levels, by the advocacy of HIA in many national policy documents, and by the setting up of specialized HIA posts and centres. To continue this level of support, and to justify any increase, the approach of HIA must show whether and how the HIA approach informs the decision-making process and, in particular, whether it improves health and reduces health inequalities. The first steps in answering these questions have been taken by the partial evaluation of some completed HIAs within England. A description of these evaluations is presented. Work is also progressing on promising practice guidance for practitioners on the topic of evaluating HIAs. The further development of evaluation within HIA is required, at both a practitioner and research level, and provides an exciting opportunity for people and groups to advance the evidence base for HIA, while also proving its place as an effective public health intervention.
Keywords: Health status indicators; Policy making; Health policy; National health programs; Intersectoral cooperation; Public health administration/trends; Outcome assessment (Health care); Environmental health; Risk assessment/methods; Data collection/methods/standards; Evidence-based medicine; Program evaluation; Socioeconomic factors; Evaluation studies; United Kingdom (source: MeSH, NLM).
RÉSUMÉ
L'étude d'impact sur la santé (EIE) est un processus qui tend à se généraliser en Angleterre où il recueille une large adhésion, d'où le nombre important d'EIE entreprises aux niveaux local et régional, leur promotion dans de nombreux documents d'orientation nationaux et la création de postes et de centres spécialisés. Pour que cette adhésion ne faiblisse pas et pour justifier le recours croissant à ces études, les EIE doivent montrer si et comment elles éclairent la prise de décision et, en particulier, si elles permettent d'améliorer la santé et de réduire les inégalités dans ce domaine. Pour répondre dans un premier temps à ces questions, une évaluation partielle de certaines EIE réalisées en Angleterre a été entreprise. Le présent article décrit ces évaluations. L'élaboration d'un guide pratique prometteur sur l'évaluation des EIE destiné aux spécialistes progresse également. L'évaluation des EIE doit être affinée au niveau tant du spécialiste que du chercheur : ainsi, les individus comme les groupes auront l'occasion exceptionnelle de faire progresser la base factuelle nécessaire à ces études, tout en démontrant par ailleurs l'efficacité du rôle de ces dernières dans l'action de santé publique.
Mots clés: Indicateur état sanitaire; Choix d'une politique; Politique sanitaire; Programme national santé; Coopération intersectorielle; Administration santé publique/orientations; Evaluation résultats (santé); Hygiène environnement; Evaluation risque/ méthodes; Collecte données/méthodes/normes; Médecine factuelle; Evaluation programme; Facteur socio-économique; Etude évaluation; Royaume-Uni (source: MeSH, INSERM).
RESUMEN
La evaluación del impacto sanitario (EIS) es un método cada vez más utilizado en Inglaterra, donde ha recibido amplio apoyo, como demuestra el gran número de EIS que se están llevando a cabo en el ámbito local y regional, la promoción de la EIS en muchos documentos normativos nacionales y la creación de puestos y centros especializados en la EIS. Para mantener este grado de apoyo y justificar el uso creciente de estos estudios, las EIS deben demostrar que aportan información útil al proceso de toma de decisiones y, en particular, que mejoran la salud y reducen las desigualdades sanitarias. Como primer paso para responder a estas preguntas, se han realizado exámenes parciales de algunas de las EIS llevadas a cabo en Inglaterra. En este artículo se describen esos exámenes. También se está elaborando una prometedora guía práctica para los profesionales acerca del examen de las EIS. Es necesario perfeccionar el examen de las EIS, a nivel tanto práctico como de la investigación; así, las personas y los grupos tendrán una excelente oportunidad de hacer avanzar las bases científicas de la EIS, y al mismo tiempo de demostrar su utilidad como intervención eficaz en el campo de la salud pública.
Palabras clave: Indicadores de salud; Formulación de políticas; Política de salud; Programas nacionales de salud; Cooperación intersectorial; Administración en salud pública/tendencias; Evaluación de resultado (Atención de salud); Salud ambiental; Medición de riesgo/métodos; Recolección de datos/métodos/normas; Medicina basada en evidencia; Programa de evaluación; Factores socioeconómicos; Estudios de evaluación; Reino Unido (fuente: DeCS, BIREME).
Introduction
There is currently a window of opportunity for health impact assessment (HIA) in England. At a national level the government has clearly signalled its acknowledgement of the importance of the wider determinants of health and its commitment to promoting HIA (1). The value of using HIA as tool for enhancing healthy public policy has been promoted in several other national policies and programmes (25).
Even though HIA is still a relatively new and developing approach in England there is evidence of variable but steadily increasing activity at both the regional and local level. For example, in a recent mapping exercise undertaken for a retrospective process evaluation, 103 local-level HIA studies were identified (6). To support this growing interest in HIA, specialist centres are emerging. These centres are actively involved in undertaking HIAs and promoting and supporting the approach within their locality and region and across other areas of the country.
In addition to these specialist academic centres there is a growing number of HIA specialist practitioner posts at both regional and local levels. Such post holders are often responsible for commissioning and managing HIA and/or promoting and supporting the development of HIA within their organization and across their locality and region. There are also several independent HIA practitioners, some of whom are attached to academic institutions and other organizations that are supporting or carrying out HIA.
As a consequence of national, regional, and local level work, dedicated HIA resources, toolkits, and websites disseminating this information are being developed (http://www.hiagateway.org.uk). Dedicated training courses, whether short introductory overviews or more intensive courses, are also being delivered.
There is, however, a danger that HIA is seen to be public health's "flavour of the moment". If HIA is to become a durable approach it is important that its value is clearly identified. Questions are being asked at a national, regional, and local level for example, "what does health impact assessment achieve and how?" The only way to answer such a question is by evaluating HIAs. It is widely recognized that the systematic collection of data will permit a judgement about the value of a programme or intervention to be made, allow reflection about what is happening, and provide an assessment of whether goals have been achieved (7, 8).
Within England, the drive to evaluate interventions is gaining pace across all sectors, including health. "Evidence-informed policy" is a catchphrase commonly used in all areas of policy development and service delivery, particularly within central and local government in England. Policy-makers are expected to "make more use of evidence and research" and to "learn from experience, by ensuring that all policies and programmes are clearly specified and evaluated" (9). Evaluation is key to delivering on this agenda.
HIA evaluation in England
In an attempt to maximize this window of opportunity and answer some of the questions about what HIA can achieve and how, four evaluations that we are aware of have been undertaken and reported on to date.
Retrospective process evaluation of five local HIAs
The Health Development Agency (HDA) commissioned a retrospective process evaluation of five completed HIA studies in 2001 (6). The study aimed to identify and explore the processes and mechanisms used in HIA, to summarize the learning achieved to date, to provide recommendations for ways to improve the application of HIA, and to identify priority areas for potential further research. Qualitative in-depth interviews were undertaken with key informants involved in the completed HIA studies so that their experiences and views of the HIA process could be explored.
The main findings showed that participants were aware of the need for (and importance of) monitoring and evaluation and were involved in some limited monitoring and evaluation-related activities for example, internal reflection on the HIA process to inform the development of future work. However, none of the five HIA studies, and no respondents, had been involved in or subjected to an independent/external evaluation. The main reasons for this were the difficulties associated with maintaining the momentum as staff and work priorities moved on. Generally, the HIAs were time-limited projects with finite funding and allocation of staff. Once the bulk of the HIA was completed, staff tended to move on to other projects and further funding (or time) was not available to undertake evaluation activities.
HIA activities have, to date, been focused instead on refining the methodology. The participants considered the HIA process and associated methodology to be developmental, and therefore believed that it was too early to evaluate the impact and outcomes of the HIA process. They also believed that there was a lack of agreed methods and materials to support the evaluation of HIA and that different methods and tools might be needed for the differing levels of HIA. In particular, those involved in rapid case studies felt they had fewer resources available and therefore it would be difficult to undertake extensive monitoring and evaluation activities. Finally, respondents considered that confounding variables made the measurement of outcomes influenced by HIA difficult, if not impossible, to undertake. Determinants of health are complex and based on several interconnected factors. Therefore, it would not be possible to attribute any change in health outcome to the HIA; HIA was considered to be only a small factor in the scheme of planning processes. To adequately track such changes in health outcome, long-term funding and excellent public health data sources would be required both of which are generally considered to be unavailable.
All the respondents were aware of the need for (and importance of) monitoring and evaluation activities, but there was some disagreement about how to progress monitoring and evaluation activities as part of the HIA process and what levels of evaluation were feasible (process, impact, or outcome evaluation). Some, but not all, believed that monitoring and evaluation was crucial and should be built in from the start in order to gather learning across different HIAs, aid development of HIA process, demonstrate the effectiveness and added value of HIA, build confidence in HIA, and help to ensure its long-term sustainability. Other respondents suggested that for each HIA undertaken, the steering group should reconvene after the recommendations have been developed, as only then are practitioners able to identify the variables and indicators that could be used in monitoring and evaluation activities. It was suggested that the less experienced were unable to plan such activities adequately without first going through a HIA.
It was generally agreed, however, that it was timely for HIA practitioners to engage with evaluation and that three levels of evaluation were possible. Further exploration of the feasibility and generation of practical guidance for these was required: first, monitoring, auditing, and evaluating the process of undertaking HIA; second, tracking whether the HIA recommendations influenced the decision-making process; and third, evaluating the effectiveness of the HIA approach in terms of accuracy of predictions and achievement of change in health of the population or determinants of health.
Alconbury audit
In August 1999 the Cambridge Health Authority secured funding from the Department of Health to commission the Anglia Clinical Audit and Effectiveness Team to undertake an external evaluation of the proposed HIA of the Alconbury Airfield Development (10).
The purpose of the audit was to ensure that the HIA was carried out in a structured and systematic way following the Merseyside Guidelines (11) to ensure that the HIA achieved its own objectives, and to facilitate the development of a HIA methodology template for future use. The auditor was given full access to all relevant meetings and documents and a good working relationship was developed. Constructive notes were sent from the auditor to the steering group after each meeting, which meant that ongoing improvements to the process could be made.
The recommendations from the evaluation were very helpful, both during the process and for the follow-up. Key learning points were: first, think about evaluation at the beginning of the HIA process and give someone the responsibility for evaluation; second, after completion of the HIA, follow up on the HIA recommendations to see if they are implemented; and third, continue to evaluate after the proposal has been implemented.
Finningley evaluation
A case study of a comprehensive HIA for a proposed commercial airport in Doncaster was undertaken from the perspective of the project manager, including a process evaluation (12). The process evaluation aimed to identify lessons learnt from the HIA by documenting the steps taken during the HIA, identifying constraints experienced, and identifying useful lessons that may be applicable to other developments. The study design was a descriptive evaluation. The source of data for the evaluation was a document analysis of the minutes of Working Group and Steering Group meetings. The process evaluation showed that the approach taken by the HIA was a systematic process based on the Merseyside Guidelines for HIA (a standard model) (11).
Useful lessons were identified from the process evaluation:
The technical information in the environmental impact assessment was used to predict health impacts.
Using the same data extraction forms for all activities facilitated triangulation of data.
Ensuring that planners and developers recognized the importance of the HIA (and were informed of its progress) helped to take the recommendations forward.
The report was disseminated widely to inform all stakeholders within and outside Doncaster.
The main constraints were the short timescales to fit into planning deadlines, the lack of detail of proposed mitigation measures, limitations of routinely collected data, difficulties of ranking potential health impacts, and fitting a non-statutory assessment into a statutory planning framework.
An impact evaluation is also currently under way. This uses Theory of Change modelling to predict future health outcomes. The impact evaluation aims to show how the HIA has added value to the planning process, and the predicted future health outcomes.
Those involved in the HIA hope to continue working proactively with the planners and the airport developers in drawing up future agreements. It has been agreed that HIA activities will continue throughout the lifetime of the airport if it goes ahead.
London Mayoral Strategy HIA evaluation
The London Health Commission has completed HIAs on the nine draft Mayoral Strategies. The Greater London Authority funded an independent retrospective process evaluation of two of the HIAs and a concurrent evaluation of another two HIAs. A qualitative approach was used, with interviews with workshop attendees, observation of the HIA workshops and preparation meetings, and use of self-completed questionnaires. Following completion of the HIAs, additional work is progressing to track what recommendations the respective strategy development teams will take forward.
Findings from this evaluation are still being analysed and are due to be presented and published shortly. However, preliminary findings of the two retrospective evaluations suggested that, as a consequence of the action-based research approach used, feedback from the retrospective evaluations were useful in informing the preparation and delivery of the later two (concurrent) rapid HIA workshops for example, reducing the number of formal presentations during the workshop, distributing less information before the workshop, and greater planning of the question areas to be asked during the HIA workshop (13).
Taking forward the HIA evaluation agenda
The need for and importance of evaluation within HIA has also been highlighted within a "review of reviews" briefing paper about the effectiveness of the HIA approach (14). This briefing paper aimed to assess review level literature to determine "if and how the HIA approach informs the decision-making process and, in particular, if it improves health and reduces health inequalities". The methodology for undertaking this work was based on a systematic search for relevant systematic reviews, syntheses, or high-quality literature reviews. Following critical appraisal of potential papers, one suitable document (15) was included in the "review of reviews".
Not surprisingly for the new and developing field of HIA, the final conclusion of this review of reviews was "there is currently no review-level evidence available to demonstrate if and how the HIA approach informs the decision-making process and, in particular, if it improves health and reduces health inequalities".
The key recommendation of the review of reviews, in relation to monitoring and evaluation, was that "HIA practitioners need to engage with monitoring and evaluation activities, and disseminate their completed case studies, their evaluation findings, and key lessons learned". In particular, the briefing paper highlighted the need to assess not only how the HIA process is undertaken, but also whether the recommendations generated as a result of the HIA were implemented. Further information on potential process and impact monitoring and evaluation indicators can be found in the briefing paper.
Building on the above findings and examples, it was agreed by the HDA and the Department of Health that there was a need to develop and produce examples and guidance for England in the area of HIA monitoring and evaluation. Research findings highlight the importance of involving people (rather than handing out paper documents) in the successful translation of evidence of promising practice into actual altered practice: "people rather than electronic methods transfer knowledge". Also, getting practitioners who have shown promising practice in their HIAs (in terms of the way it was done and the results they obtained) to describe their own experiences to others is a good method to motivate others to adapt their own practice. In line with this thinking, a one-day "learning from promising practice" workshop was run by the HDA to help generate evaluation examples and guidance (16, 17).
The workshop identified projects that showed aspects of promising practice and identified the particular elements and processes that needed to be in place to make such activities successful. It provided the participants with an opportunity to reflect on their own experience of developing, conducting, and reporting on HIA evaluation, to explore any challenges, problems and potential solutions, and to share learning with the other participants. In an attempt that others may learn from the workshop participants, a short summary bulletin (four sides of A4) outlining promising practice examples and practical recommendations for improving the application of HIA evaluation is currently being developed. It is hoped that this approach of learning from practice will continue in England to further encourage the sharing of experiences and understanding about this important area of HIA. In addition, a detailed description of the workshop activities and an evaluation of its impact will be produced and made available. These will be available on the Heath Impact Assessment GatewayHealth Development Agency web site (www.hiagateway.org.uk).
The briefing paper also highlighted the need to explore the feasibility of assessing the effect of HIA on the health of the local population. Without this, "the effectiveness of HIA and how it contributes to improving health and reducing health inequalities cannot be demonstrated". However, it is difficult to establish a connection between HIA and broader improvements in public health (1820).
Furthermore, there is difficulty demonstrating the health outcomes of the HIA approach because of confounding factors (21). Health outcomes invariably have multiple causes, and each cause can have many health (and non-health) determinants. Attribution of health outcomes to any one intervention/approach is therefore problematic. Paradoxically, implementing HIA recommendations may make it impossible to test whether any detrimental health predictions/concerns highlighted in the HIA were accurate. For these reasons it is necessary for the HIA field to explore whether it is feasible and appropriate to assess the associated health outcomes (both intended and unintended, positive and negative) of the HIA approach. Questions that require exploration include "did the identified predictions materialize?", "were these predictions accurate?", and "did the HIA approach achieve change in the health of the population or health determinants?"
Given the limited evaluation activities within England, and the need to further develop appropriate evaluation methods, it would also be useful to draw on the evaluation experience of other countries for example, the Netherlands and Sweden, which currently have evaluation work under way. To facilitate this and as advocated in England, it is important that the evaluation methods and evaluation reports of other countries are made widely available. The development of the WHO HIA website will provide a dissemination route for such internationally relevant work and may also encourage international collaborative work on this important developing area of HIA. More established impact assessment areas for example, environmental impact assessment will have undertaken evaluation activities and therefore may also have valuable learning to contribute to the HIA field.
Conclusion
To take advantage of the window of opportunity that HIA currently has in England, people working in HIA need to further engage with monitoring and evaluation-related activities, particularly developing, conducting, and reporting HIA process and impact evaluation. This, in turn, represents an opportunity for individuals and groups to make their mark within the field of HIA, while also providing the evidence to justify the continued growth of HIA. The development of HIA evaluation methods and approaches would also benefit from drawing on the experiences of other countries and the evaluation practices of the broader impact assessment field. 
Conflicts of interest: none declared.
References
1. Department of Health. Saving lives: our healthier nation. Cm 4386. London: The Stationery Office; 1999.
2. Department of Health. National service framework for coronary heart disease: main report. London: Department of Health; 2000.
3. Department of Health. The NHS plan. Paper 4818 1. London: The Stationery Office; 2000.
4. Cabinet Office. Bringing Britain together: a national strategy for neighbourhood renewal. London: The Stationery Office; 1998.
5. DETR. A new deal for transport, Government White Paper on the future of transport. London: The Stationery Office; 1998.
6. Jackson N, Taylor L, Quigley R. Findings from a retrospective process evaluation of five HIA studies. London: Health Development Agency (forthcoming).
7. Robson, C. Real world research. Oxford: Blackwell Publishers; 1998.
8. Pawson R, Tilley N. Realistic evaluation. London: Sage; 1997.
9. Cabinet Office. Modernising government. London: The Stationery Office; 1999.
10. Close N. Alconbury airfield development health impact assessment evaluation: report for Cambridgeshire Health Authority. Cambridge: Anglia Clinical Audit and Effectiveness Team; 2001.
11. Scott-Samuel A, Birley M, Ardern K. The Merseyside Guidelines for Health Impact Assessment. University of Liverpool: Merseyside Health Impact Assessment Steering Group; November 1998.
12. Abdel Aziz, M. Conducting a comprehensive health impact assessment: the potential and constraints. Nottingham: School of Community Health Sciences, University of Nottingham; 2001.
13. Bowen, C. Summary of experiences evaluation of the HIAs of Health Development Agency Monitoring and evaluation workshop, 15 November 2002.
14. Taylor L, Quigley R. Health impact assessment: a review of reviews. London: Health Development Agency; 2002.
15. McIntyre L, Petticrew M. (1999). Methods of Health Impact Assessment: a literature review. Glasgow: MRC Social and Public Health Sciences Unit, Occasional paper No. 2. Available from: URL:www.msoc-mrc.gla.ac.uk (accessed 6 January 2003).
16. Cabinet Office. The effectiveness of different mechanisms for spreading good practice. London: The Stationery Office; 2000.
17. NHS Centre for Reviews and Dissemination. Effective Health Care Bulletin: getting evidence into practice. York: University of York, NHS Centre for Reviews and Dissemination; 1999.
18. Department of Health. Health Impact Assessment: report of a methodological seminar. London: Department of Health; 1999.
19. Mahoney M, Durham G. Health impact assessment: a tool for policy development in Australia. Report. Melbourne: Faculty of Health and Behavioural Sciences, Deakin University; 2002.
20. Lock, K. Health impact assessment. BMJ 2000;320:1395-8.
21. Ratner PA, Green LW, Frankish CJ, Chomik T, Larsen C. Setting the stage for health impact assessment. Journal of Public Health Policy 1997;18:67-79.
1 Correspondence should be addressed to this author.