Letters

Water, sanitation, and hygiene evaluation issues

Editor – The article on Programme Saniya in Burkina Faso (1) raises important generic issues for the evaluation of water, sanitation and hygiene education initiatives. According to the UK Department for International Development, "on scores of occasions water and sanitation projects have commissioned epidemiological or demographic evaluations of health benefits ... such studies are time-consuming, expensive, fraught with methodological defects, and frequently produce misleading or ambiguous results" (2). Funding agencies are unlikely to spend their money on such evaluation unless a health impact is explicitly stated as a project objective. Some difficulties arise from how the evidence is defined, such as setting unrealistic targets for demonstrating change, and not adequately differentiating between process, impact and outcome indicators. However, regarding the health promotion materials themselves, there is no debate that their design must be based on local practices and culture; this normally requires initial qualitative research and subsequent use of surveys to assess behaviour change, such as reported from Burkina Faso. We believe that the apparent lack of effectiveness of many such projects, as noted by the authors in their literature review, has less to do with the appropriateness of the educational materials and more to do with evaluation design and measurement issues. Even in the case of the Burkina Faso experience, it is curious that while indicators show relatively static levels (perhaps even a fall-off) for programme coverage, direct observation of hygiene behaviour of mothers and children reveals a positive increase between baseline (1995) and post-intervention (1998) surveys.

The Aga Khan University (AKU) is currently involved in the outcome evaluation of a 5-year (1997–2001) water, sanitation, and health and hygiene education programme targeting 100 villages (>100 000 population) in the Northern Areas and Chitral, Pakistan. The project is being implemented by the Water and Sanitation Extension Programme (WASEP) of the Aga Khan Building and Planning Services, with external donor funding. Villages were selected into the project in phases, based on pre-set criteria as defined by WASEP, incorporating a participatory approach to enhance sustainability after the project. Compared with the Burkina Faso setting, the project area differs greatly in the pre-intervention level of water and sanitation facilities (e.g. <10% of pre-intervention village households had a latrine). In northern Pakistan, villages are served by gravity-flow water supplies using an intricate system of irrigation channels fed by melting snow and ice; in this traditional system, there is intermingling of water for animal and human use. The terrain is mountainous, with villages routinely becoming isolated.

WASEP delivers an integrated package of interventions for three components — water, sanitation, and hygiene education; thus, it addresses not only hygiene behaviour through education, but also provides engineering solutions to improve water quantity and quality. Interventions are targeted at three levels: the village (e.g. improved water supply), the household (e.g. provision of a latrine), and primary schools (health and hygiene education sessions). Interventions are conducted in phases among selected villages, and in stages within villages. As in Burkina Faso, the WASEP project experienced a shortage of donor funds in the fourth and fifth years, slowing implementation of some activities, particularly health and hygiene education; expectations did not diminish regarding the projected health impact.

Throughout implementation, WASEP itself has monitored its engineering and educational interventions, using several methods: periodic household surveys of knowledge, attitude and practice; direct observation of household sanitation; fortnightly diarrhoeal disease surveillance of households; and a water quality monitoring system. Even so, WASEP has opted for an independent evaluation by AKU, with the assistance of the Aga Khan Health Services, Pakistan. The main focus at this time is to assess the health outcome of the project, using a case-control study of diarrhoea as recommended by the World Health Organization (3). We also plan an integrated evaluation exercise, to encompass both engineering and hygiene aspects of the project. As in the case of Programme Saniya in Burkina Faso, this will include data collected at baseline and throughout implementation.

Debra Nanan¹ & Franklin White²

Conflicts of interest: none declared.

1. Curtis V, Kanki B, Cousens S, Diallo I, Kpozehouen A, Sangaré M, et al. Evidence of behaviour change following a hygiene promotion programme in Burkina Faso. Bulletin of the World Health Organization 2001;79:518-27.

2. Department for International Development (DFID). Guidance manual on water supply and sanitation programmes. London: WELL and London School of Hygiene and Tropical Medicine for DFID; 1998.

3. Cousens SN, Mertens TE, Kirkwood BR, Smith PG, Feachem RGA. Case-control studies of common childhood diseases: the example of diarrhoea. London: Macmillan Education Ltd for the World Health Organization; 1995.

¹ Lecturer, Department of Community Health Sciences, The Aga Khan University, Karachi, Pakistan (email: debra.nanan@aku.edu). Correspondence should be addressed to this author.

² Professor & Chair, Department of Community Health Sciences, The Aga Khan University, Karachi, Pakistan.