THEME PAPERS

Criteria for clinical audit of the quality of hospital-based obstetric care in developing countries

Critères de base des audits cliniques de la qualité des soins obstétricaux dans les hôpitaux des pays en développement

Criterios para la auditoría clínica de la calidad de la atención obstétrica hospitalaria en los países en desarrollo

W. Graham^I; P. Wagaarachchi^II; G. Penney^III; A. McCaw-Binns^IV; K. Yeboah Antwi^V; M.H. Hall^VI

^IDirector, Dugald Baird Centre for Research on Women’s Health, Department of Obstetrics and Gynaecology, Aberdeen Maternity Hospital, Cornhill Road, Aberdeen AB25 2ZL, Scotland
^IIClinical Research Fellow, Dugald Baird Centre for Research on Women’s Health, Department of Obstetrics and Gynaecology, University of Aberdeen, Scotland
^IIISenior Lecturer, Department of Obstetrics and Gynaecology, University of Aberdeen and University of Edinburgh, Scotland
^IVLecturer, Department of Community Health and Psychiatry, University of the West Indies, Jamaica
^VDirector, Kintampo Health Research Centre, Ministry of Health, Ghana
VIConsultant Obstetrician/Gynaecologist, Grampian University Hospitals NHS Trust, Scotland

Correspondence

ABSTRACT

Improving the quality of obstetric care is an urgent priority in developing countries, where maternal mortality remains high. The feasibility of criterion-based clinical audit of the assessment and management of five major obstetric complications is being studied in Ghana and Jamaica. In order to establish case definitions and clinical audit criteria, a systematic review of the literature was followed by three expert panel meetings. A modified nominal group technique was used to develop consensus among experts on a final set of case definitions and criteria. Five main obstetric complications were selected and definitions were agreed. The literature review led to the identification of 67 criteria, and the panel meetings resulted in the modification and approval of 37 of these for the next stage of audit. Criterion-based audit, which has been devised and tested primarily in industrialized countries, can be adapted and applied where resources are poorer. The selection of audit criteria for such settings requires local expert opinion to be considered in addition to research evidence, so as to ensure that the criteria are realistic in relation to conditions in the field. Practical methods for achieving this are described in the present paper.

Keywords: labour complications, classification; medical audit, standards; quality assurance, health care, standards; hospitals, district; evidence-based medicine; Ghana; Jamaica

RÉSUMÉ

L’obstétrique est l’une des premières spécialités à avoir adopté l’audit clinique. Une somme d’expérience considérable a été accumulée sur tous les aspects de cet exercice, y compris des évaluations de sa propre efficacité. Si l’on s’exprime désormais couramment en termes de pratique factuelle, d’efficacité clinique et de soins de qualité dans les pays en développement, l’audit clinique n’a guère fait avancer le débat. Défini comme « l’analyse systématique et critique de la qualité des soins », un audit bien mené a deux fonctions. Non seulement l’audit peut servir au suivi des changements à l’appui d’une pratique clinique efficace mais, en tant qu’instrument éducatif, il constitue aussi un mécanisme d’amélioration. La réduction de la mortalité maternelle là où elle continue d’atteindre des niveaux inacceptables passe par l’amélioration de la qualité des soins obstétricaux.

Un audit clinique fondé sur un ensemble de critères suppose l’acceptation préalable par les cliniciens d’une liste de critères concis auxquels correspondent des soins de qualité, compte tenu des ressources disponibles. Des vérificateurs non médicaux peuvent ensuite examiner les dossiers des patientes concernées et noter si les soins dispensés correspondent aux critères convenus. Un audit se déroule habituellement en cinq étapes : établissement des critères, mesure de la pratique actuelle, communication des résultats et fixation des cibles, mise en œuvre des changements et, enfin, réévaluation de la pratique. L’efficacité du cycle se mesure principalement d’après l’évolution de la proportion des complications dont la prise en charge correspond aux critères de la meilleure pratique.

L’hésitation à utiliser les audits fondés sur un ensemble de critères dans les hôpitaux de district des pays en développement vient de ce qu’on ignore si les cas peuvent être choisis sur la base de définitions internationales, s’il est possible de convenir de critères réalistes pour la qualité des soins, s’il existe une documentation adéquate permettant de déterminer si les soins correspondent aux critères, si les informations en retour et l’activité normative seront utiles et si la qualité des soins sera effectivement améliorée.

Le projet décrit dans cet article tente de répondre à ces questions en évaluant la faisabilité et l’efficacité d’un audit effectué dans quatre hôpitaux de district – deux au Ghana et deux à la Jamaïque, par l’analyse « avant et après » propre aux audits. Les soins examinés sont l’évaluation et la prise en charge dans les hôpitaux de district des cinq complications principales responsables des taux élevés de mortalité maternelle : hémorragie, infection des voies génitales, éclampsie, rupture de l’utérus et dystocie.

Cet article a pour but de fournir des informations sur l’achèvement de la première étape : l’établissement des critères auxquels correspondent la meilleure pratique. La réalisation de cette première étape essentielle a demandé six mois à l’équipe du projet. Après un examen systématique de la documentation pertinente, trois réunions d’experts, en Ecosse, au Ghana et à la Jamaïque, ont été organisées pour évaluer les critères qui en émergeaient. Un jeu de 37 critères pour la meilleure pratique a finalement été adopté pour les cinq complications potentiellement mortelles, en même temps que des définitions de travail des cas. Les critères sont des énoncés succincts des meilleures pratiques 1) qui conviennent pour la prise en charge d’urgence d’une complication une fois survenue, 2) qui peuvent se mesurer d’après les dossiers individuels, 3) qui sont des pratiques essentielles, et non facultatives, et 4) qui sont telles que l’examen et le traitement sont disponibles au niveau de l’hôpital de district. Exemples de critères pour les cas d’hémorragie obstétricale potentiellement mortelle : « établir l’hématocrite ou le taux d’hémoglobine de la patiente » ou « administrer des ocytociques en cas d’hémorragie du post-partum ».

Cet article décrit la première étape déterminante d’un audit fondé sur un ensemble de critères. C’est sur la réussite de cette étape que s’appuieront les étapes suivantes, et c’est de cette réussite que dépendra la capacité des audits fondés sur un ensemble de critères à faciliter une pratique obstétricale factuelle dans les hôpitaux de district des pays en développement.

RESUMEN

La obstetricia fue una de las primeras especialidades en adoptar la práctica de las auditorías clínicas. El resultado es que se ha acumulado ya una considerable experiencia en todos los aspectos del proceso, incluidas las evaluaciones de la propia eficacia. Aunque las referencias a la práctica basada en la evidencia, la eficacia clínica y la calidad de la atención son ya frecuentes en los países en desarrollo, la contribución de las auditorías clínicas al debate es aún mínima. Definida como «el análisis sistemático y crítico de la calidad de la asistencia», una auditoría bien realizada tiene una doble función: no sólo permite controlar los cambios en apoyo de una práctica clínicamente eficaz, sino que, como instrumento didáctico, el proceso de auditoría puede servir por sí mismo como mecanismo de mejora. La mejora de la calidad de la atención obstétrica es un requisito clave para reducir la mortalidad materna allí donde ésta sigue siendo inaceptablemente alta.

En la auditoría clínica basada en criterios, los clínicos acuerdan previamente una lista de criterios concisos para definir la asistencia de buena calidad, teniendo en cuenta los recursos disponibles. Ayudantes de auditoría sin título médico pueden entonces revisar las fichas de pacientes pertinentes y registrar si la atención recibida se ajustó a los criterios acordados. Los cinco pasos clásicos de un ciclo de auditoría son los siguientes: establecimiento de criterios, medición de las prácticas seguidas, retroinformación de los resultados y establecimiento de metas, aplicación de los cambios y reevaluación de las prácticas. La eficacia del ciclo puede evaluarse fundamentalmente atendiendo a la variación del porcentaje de complicaciones en que el manejo de los casos satisfizo los criterios definitorios de las prácticas óptimas.

Los interrogantes que plantea la utilización de las auditorías basadas en criterios en los hospitales de distrito en los países en desarrollo estriban en si es factible realizar una selección de casos a partir de las definiciones internacionales, si es posible acordar unos criterios realistas para definir la atención de calidad, si se dispone de la documentación suficiente para evaluar si se han cumplido o no los criterios, si la retroinformación y el establecimiento de normas serán de utilidad, y si la calidad de la asistencia mejorará efectivamente.

El proyecto descrito en este artículo aspira a responder a esas preguntas evaluando la viabilidad y eficacia de un ciclo de auditoría en cuatro hospitales de distrito, dos en Ghana y dos en Jamaica, usando el diseño de «antes y después» propio de las auditorías. El tipo de asistencia sometido a auditoría es la evaluación y el manejo en hospitales de distrito de las cinco complicaciones que más contribuyen a una alta mortalidad materna: hemorragias, infecciones genitales, eclampsia, ruptura uterina y parto obstruido.

El objetivo de este artículo es documentar la feliz conclusión del paso 1, esto es, el establecimiento de criterios definitorios de las prácticas óptimas. Este primer y crucial paso fue llevado a cabo por el equipo del proyecto a lo largo de un periodo de seis meses. Consistió en una revisión estructurada de la literatura, a la que siguieron tres reuniones de expertos dedicadas a evaluar los criterios esbozados. Dichas reuniones se celebraron en Escocia, Ghana y Jamaica. Finalmente se acordó un conjunto de 37 criterios de definición de las prácticas óptimas para las cinco complicaciones potencialmente mortales, junto con definiciones de trabajo de los casos. Los criterios consisten en declaraciones sucintas de las prácticas óptimas, que han de reunir las siguientes condiciones: (1) ser pertinentes para el manejo agudo de una complicación una vez que haya surgido, (2) ser medibles a partir de las fichas de los casos, (3) consistir en prácticas básicas más que optativas, y (4) entrañar pruebas o tratamientos disponibles en los hospitales de distrito. Criterios válidos para los casos de hemorragia obstétrica potencialmente mortal, por ejemplo, son las afirmaciones «debe determinarse el hematócrito o el nivel de hemoglobina de la paciente», y «en el tratamiento de la hemorragia posparto se deben emplear oxitócicos».

En este artículo se ha descrito el primer y crucial paso de un ciclo de auditoría basado en criterios. La feliz conclusión de esta fase sienta las bases para dar los siguientes pasos y para que la auditoría basada en criterios pueda un día facilitar la práctica obstétrica basada en la evidencia en los hospitales de distrito en el mundo en desarrollo.

Introduction

Obstetric care of high quality continues to be a key requirement for reducing maternal mortality (1, 2). However, the provision of effective, appropriate, accessible and affordable obstetric care is difficult to define, measure, resource, sustain and evaluate (3–5). Clearly, the urgency of and the means for improving obstetric care vary enormously between the industrialized countries, where less than 1% of maternal deaths occur each year, and the developing countries, where more than 99% occur (Fig. 1) (6). Nevertheless, the notion of best practice is of relevance in both settings. Optimal management of, for example, a case of puerperal sepsis is essentially the same in a major referral hospital in North America as in a district hospital in sub-Saharan Africa. This common ground provides an opportunity for the sharing of approaches, such as that of clinical audit, to improving the quality of obstetric care.

Of the three dimensions of health care which may be audited (structure, process and outcome), process, i.e. the delivery of care to patients, is the most relevant to the prevention of maternal death, provided that what is involved is known to improve outcome. Criterion-based clinical audit requires approval by clinicians of a list of concise criteria for care of good quality (7), taking into account the resources available (8). Non-medically qualified audit assistants can then screen the case notes of patients and record whether care has met the agreed criteria. The five classic steps of the audit cycle are shown schematically in Fig. 2. The effectiveness of the cycle can be assessed primarily in terms of change in the proportion of complications where management has met the criteria for good care, although structure, e.g. staffing and facilities, must also be monitored.

With regard to the use of criterion-based audit in district hospitals of developing countries, it is not known whether:

– case selection using international definitions is feasible;

– realistic criteria for care of good quality can be agreed;

– documentation is available and adequate to assess whether criteria have been met;

– feedback and standard setting will be helpful;

– the quality of care will improve.

The project described here aims to answer these questions by assessing the feasibility and effectiveness of an audit cycle in two district hospitals in Ghana and two in Jamaica, using the before-and-after design intrinsic in audit (9). The care being audited is the assessment and management of life-threatening complications. Primary and tertiary care are also important but are not considered in the study.

The first step, which took six months, was the establishment of best practice. This involved three main activities:

• a systematic review of the literature in order to find evidence of best practice;

• devising workable definitions for complications;

• expert panel meetings to reach agreement on final definitions and criteria.

Methods

In order to carry out a formal literature search, key words were identified and guidelines for the inclusion or exclusion of studies were drawn up (10). Electronic searches were conducted using Medline and the Cochrane Library, and manual searches were performed on standard obstetric texts, relevant WHO publications and unpublished reports from international safe motherhood projects. A provisional list of criteria of best practice was generated from the 136 references that were reviewed. Precedence was given to evidence from randomized controlled trials, and this was followed by evidence from studies with less robust designs. Finally, expert opinion was considered.

The selection of complications was based on their seriousness and whether they could be concisely and unambiguously defined. The final selection of definitions was made by expert panels after a process that began with standard textbook definitions and proceeded to modifications in the light both of the diagnostic capabilities at the participating hospitals and of the completeness and accuracy of case notes. All apparent cases of the specified complications occurring over a six-month period were identified by consulting labourward books, discharge registers and other sources. They were then reviewed by a member of the study team (PW) before the start of the audit cycle in order to evaluate the suitability of the final working definitions.

There is insufficient research-based evidence to inform many aspects of obstetric care (11). Rather than relying solely on the judgement of an individual to define best practice, a group decision reflecting a consensus development process involving six to ten individuals is increasingly seen as preferable (12). In the present project, three expert panels were constituted with a view to achieving consensus development. They met successively in Scotland (six participants, most of them having had experience in developing countries), Ghana (ten participants), and Jamaica (eight participants). The participants, chosen by the local collaborating project teams, were practising midwives and obstetricians with broadly similar durations of experience. The six members of the first panel were invited initially to complete a questionnaire in which they expressed their opinions on the definitions of complications and on the criteria emerging from the systematic review. They were then asked to discuss these matters at a formal meeting. At all three meetings a similar consensus development process was followed on the basis of a modified nominal group technique (13). Ground rules for accepting or rejecting a definition or criterion were identified. A rapid review was conducted in order to identify any criteria on which there was already obvious agreement. In a second review, opinions were heard on the remaining criteria and votes were taken on their acceptance.

Results

Case definitions

The final case definitions for the selected obstetric complications are presented in Table 1. The panel discussions centred mainly on the necessity to use clinical rather than laboratory features and to ensure that only severe cases were selected. Pre-eclampsia was omitted as a condition since standard definitions would have included many mild cases. Obstructed labour was very strictly defined so as to include only severe cases. A review of a subsample of case notes at the four hospitals showed that only 63% of the cases (193/305) initially indicated on the basis of diagnoses in the summary registers actually met the project’s definitions. Random double-checking of case ascertainment was therefore introduced as part of the main study. The pilot assessment also revealed that 1.6% of women (5/305) had more than one life-threatening complication, and thus there was a need to audit these cases against the criteria for each complication.

Audit criteria

The systematic review led to the identification of 67 criteria. This number was reduced to 37 by the expert panels. Fig. 3 indicates the process whereby the final criteria were selected. The panels identified the following main grounds on which to exclude criteria:

• no relevance to acute management of existing complications, including:

– identification of risk factors for complications;

– preventive or prophylactic management before a complication;

– late follow-up of surviving mothers;

• lack of research evidence and inability to reach agreement among panel members, including:

– the use of blood filters;

– diuretics;

– precise timing and nature of surgical intervention in antepartum and postpartum haemorrhage;

– the use of treatment packs and avoidance of ergometrine and intravenous dextrose in eclampsia;

• impracticality of measurement on the basis of hospital case notes, including:

– the size of intravenous cannulas;

– delay in the availability of blood transfusion;

– postural management;

– aseptic technique in vaginal examination;

– clinical assessment for sequelae of obstructed labour;

• indication of an optional rather than an essential practice in the management of a complication:

– repeated haematological testing during haemorrhage;

– prophylactic cross-matching of blood;

• unavailability of a test or therapy at district hospital level:

– liver function tests in eclampsia;

– transfusion of specific blood components as required.

The elimination process was also guided by the aim of selecting no more than ten criteria for each complication. The final criteria are summarized in Table 2.

Discussion

Considerable experience has now been accumulated on all aspects of clinical audit, including evaluations of its effectiveness (14). In developing countries the language of evidence-based practice, clinical effectiveness and care of high quality is now widely spoken but the contribution of clinical audit is still minimal (15, 16). Defined as the systematic and critical analysis of the quality of care (17), audit can play a dual role: not only can it monitor change in the support of clinically effective practice but as an educational tool it can serve as a mechanism for improvement (18).

In the present project the expert panels in each country made important contributions to the final selection of case definitions and criteria. The consensus development discussions highlighted the important distinction between current practices and those that should be aspired to, taking available resources into consideration. Clinical audit seeks to improve the quality of care by comparing current practice against agreed standards. The potential for stimulating improvements is lost if criteria of best practice are set primarily in accordance with what is known to be current practice. The panel members saw this problem as particularly relevant to applications of audit in developing countries, where limited resources can readily reduce best practice to so-called realistic practice.

This paper has described the first crucial step in a criterion-based audit cycle: identifying criteria of best practice and developing clear case definitions and robust procedures for indicating cases among hospital records. The successful completion of this phase lays the foundation for the remaining steps and for the possibility that criterion-based audit can facilitate evidence-based obstetric practice in district hospitals in the developing world.

Acknowledgements

Particular thanks are given to the members of the expert panels in Ghana, Jamaica, and Scotland. The research project is funded by the Department for International Development of the United Kingdom, which accepts no responsibility for any information provided or views expressed. The present paper has been prepared on behalf of the Criterion-Based Clinical Audit Study Group, comprising: Dr Paul Arthur, Ministry of Health, Kintampo, Ghana; Dr Kofi Asare, Ministry of Health, Sunyani, Ghana; Dr Deanna Ashley, Ministry of Health, Jamaica; Ms Georgiana Gordon, Ministry of Health, Jamaica; Dr Wendy Graham, Aberdeen University, Scotland; Dr Marion Hall, Aberdeen Maternity Hospital, Scotland; Mr Richard Henneh, Ministry of Health, Sunyani, Ghana; Dr Affette McCaw-Binns, University of the West Indies, Jamaica; Dr Gillian Penney, Aberdeen University, Scotland; Dr Mandy Ryan, Aberdeen University, Scotland; Dr Prabhath Wagaarachchi, Aberdeen University, Scotland; and Dr Kojo Yeboah Antwi, Ministry of Health, Kintampo, Ghana.

References

1. Report of the confidential enquiries into maternal deaths in the United Kingdom 1994–96. London, Her Majesty’s Stationery Office, 1998.        

2. Starrs A. Safe motherhood: moving to action. Report of Technical Consultation on Safe Motherhood, 18–23 October 1997. Colombo, Sri Lanka, Family Care International and Inter-Agency Group for Safe Motherhood, 1998.        

3. Adeyi O, Morrow R. Concepts and methods for assessing the quality of essential obstetric care. International Journal of Health Planning and Management, 1996, 11: 119–134.        

4. Mother-baby package: implementing safe motherhood in countries. Geneva, World Health Organization, 1994.        

5. Graham W. Outcomes and effectiveness in reproductive health. Social Science and Medicine, 1998, 47 (12): 1925–1936.        

6. Revised 1990 estimates of maternal mortality: a new approach by WHO and UNICEF. Geneva, World Health Organization, 1996. (WHO/FRH/MSM/96.11; available on the Internet at: http://www.unicef.org/reseval/pdfs/revmm.pdf).        

7. Shaw CD. Criterion-based audit. In: Smith R, ed. Audit in action. London, British Medical Journal Publishing Group, 1992: 122–128.        

8. Penney GC, Glasier A, Templeton A. Agreeing criteria for audit of the management of induced abortion: an approach by national consensus survey. Quality in Health Care, 1993, 2: 167–169.        

9. Walshe K, Spurgeon P. Clinical Audit Assessment Framework. Birmingham, United Kingdom, University of Birmingham, Health Services Management Centre, 1997.        

10. Undertaking Systematic Reviews of Research on Effectiveness. York, United Kingdom, NHS Centre for Review and Dissemination and University of York, 1996 (CRD Report 4).        

11. Chalmers I, Enkin M, Keirse MJNC. Effective practice in pregnancy and childbirth. Volumes 1 and 2. Oxford, Oxford University Press, 1989.        

12. Murphy MK et al. Consensus development methods, and their use in clinical guideline development. Health Technology Assessment, 1998, 2 (3): 1–88.        

13. Bernstein SJ et al. Setting standards for effectiveness: a comparison of expert panels and decision analysis. International Journal of Quality Health Care, 1997, 9: 255–264.        

14. Chapple J. Some examples of successful audit in obstetrics and gynaecology. In: The Yearbook of the Royal College of Obstetricians and Gynaecologists 1995. London, Royal College of Obstetricians and Gynaecologists, 1996: 29–38.        

15. Bulgalho A, Bergstrom S. Value of perinatal audit in obstetric care in the developing world: ten years experience of the Maputo model. Gynaecologic and Obstetric Investigation, 1993, 36: 239–243.        

16. Dyke T. Can medical audit be practised in district hospitals in Papua New Guinea? Papua New Guinea Medical Journal, 1993, 36 (2): 114–119.        

17. Clinical audit in the NHS: a position statement. London, Department of Health, 1996.        

18. Burnett AC, Winyard G. Clinical audit at the heart of clinical effectiveness. Journal of Quality in Clinical Practice, 1998, 18: 3–19.        

Correspondence
W. Graham
Dugald Baird Centre for Research on Women’s Health, Department of Obstetrics and Gynaecology, Aberdeen Maternity Hospital
Cornhill Road, Aberdeen AB25 2ZL, Scotland
E-mail: w.graham@aberdeen.ac.uk