RESEARCH
Methodological trends in studies based on verbal autopsies before and after published guidelines
Tendances méthodologiques des études reposant sur des autopsies verbales avant et après la publication de directives
Tendencias metodológicas de los estudios basados en autopsias verbales antes y después de la publicación de directrices al respecto
Rohina Joshi*; Andre Pascal Kengne; Bruce Neal
The George Institute for International Health, University of Sydney, PO Box M201, Missenden Road, Sydney NSW 2050, Australia
ABSTRACT
OBJECTIVE: To report on the uptake of guidelines published in the early 1990s with specific recommendations about the design of future studies based on verbal autopsy conducted for mortality surveillance.
METHODS: We conducted a systematic literature search of all verbal autopsy studies published before January 2006 and extracted from the studies a standard set of data. We then compared studies designed before and after the recommendations were issued in terms of seven key methodological indicators.
FINDINGS: We found 102 studies conducted in 39 countries; 60 were designed before and 42 after the guidelines were issued. The methods used in these 102 studies varied considerably. While some encouraging trends were noted, there is no evidence that the design recommendations have been systematically implemented. Specifically, there was no clear increase in the proportion of studies with a combined questionnaire (63% before recommendations versus 74% after; P = 0.3), a trained interviewer (70% versus 70%; P = 1.0), a suitable respondent (98% versus 100%; P = 1.0), an optimal recall period (84% versus 97%; P = 0.2), predefined algorithms (28% versus 38%; P = 0.4), an option for assigning multiple causes of death (30% versus 38%; P = 0.3), or a follow-up validation study (83% versus 72%; P = 0.7).
CONCLUSION: Expert recommendations for optimal design of verbal autopsy studies have been incompletely implemented to date. Better uptake of design recommendations through enhanced collaboration between research teams is likely to produce better mortality statistics from an increasing number of verbal autopsy studies.
RÉSUMÉ
OBJECTIF: Dresser un rapport sur l'assimilation des directives publiées au début des années 90 et contenant notamment des recommandations spécifiques sur la conception des futures études de surveillance de la mortalité reposant sur l'autopsie verbale.
MÉTHODES: Nous avons réalisé une revue systématique de l'ensemble des études par autopsie verbale publiées avant janvier 2006 et extrait de ces études un jeu de données standard. Nous avons ensuite comparé les études conçues avant et après la publication des directives en utilisant sept indicateurs méthodologiques clés.
RÉSULTATS: Nous avons recensé 102 études menées dans 39 pays ; 60 d'entre elles avaient été conçues avant et 42 après la publication des directives. Les méthodes utilisées dans ces 102 études étaient très variables. Si certaines tendances encourageantes ont été notées, il n'a pas été relevé de preuve d'une application systématique des recommandations en matière de conception. Plus spécifiquement, il n'a pas été constaté d'augmentation claire de la proportion d'études utilisant un questionnaire combiné (63 % avant la publication des directives contre 74 % après, p = 0,3), un enquêteur formé (70 % contre 70 %, p = 1,0), des sujets appropriés (98 % contre 100 %, p = 1,0), une période de rappel optimale (84 % contre 97 %, p = 0,2), des algorithmes prédéfinis (28 % contre 38 %, p = 0,4), une option pour l'affectation des causes multiples de décès (30 % contre 38 %, p = 0,3) ou une étude de validation dans le cadre du suivi (83 % contre 72 %, p = 0,7).
CONCLUSION: Les directives des experts concernant la meilleure façon de concevoir les études par autopsie verbale sont à ce jour appliquées de manière incomplète. Une meilleure assimilation de ces recommandations en matière de conception grâce à un renforcement de la collaboration entre les équipes de recherche devrait permettre de produire des statistiques de mortalité de plus grande qualité à partir d'un nombre grandissant d'études par autopsie verbale.
RESUMEN
OBJETIVO: Informar sobre el seguimiento de las directrices publicadas a comienzos de los años noventa con recomendaciones específicas para el diseño de los futuros estudios de vigilancia de la mortalidad basados en autopsias verbales.
MÉTODOS: Llevamos a cabo una búsqueda sistemática de la bibliografía para localizar todos los estudios basados en autopsias verbales publicados antes de enero de 2006 y extraer de ellos un conjunto estándar de datos. Para comparar los estudios diseñados antes y después de que se formularan las recomendaciones se emplearon siete indicadores metodológicos clave.
RESULTADOS: Hallamos 102 estudios realizados en 39 países; 60 se habían diseñado antes de que se publicaran las directrices, y 42 después de su publicación. Se observaron diferencias considerables en los métodos utilizados por esos 102 estudios. Aunque se detectaron algunas tendencias alentadoras, no hay indicios de que se hayan aplicado sistemáticamente las recomendaciones en materia de diseño. Más concretamente, no aumentó claramente la proporción de estudios con un cuestionario combinado (63% antes de las recomendaciones frente a 74% después de las mismas; p = 0,3), un encuestador adiestrado (70% frente a 70%; p = 1,0), un encuestado idóneo (98% frente a 100%; p = 1,0), un periodo de rememoración óptimo (84% frente a 97%; p = 0,2), algoritmos predefinidos (28% frente a 38%; p = 0,4), una opción para asignar varias causas de defunción (30% frente a 38%; p = 0,3), o un estudio de validación de seguimiento (83% frente a 72%; p = 0,7).
CONCLUSIÓN: Las recomendaciones formuladas por los expertos para optimizar el diseño de los estudios basados en autopsias verbales se han aplicado solo parcialmente hasta la fecha. Un mayor cumplimiento de ese tipo de recomendaciones gracias a una colaboración más idónea entre los equipos de investigación permitiría probablemente obtener mejores estadísticas de mortalidad a partir del creciente número de estudios basados en autopsias verbales.
Introduction
Information about the causes of death in a population is essential for monitoring health and planning appropriate health services.1 While most higher-income countries have in place robust processes for documenting deaths and their causes, many developing countries lack such systems due to organizational and economic constraints and other factors.2
Verbal autopsy-based cause of death recording systems are widely used in countries where vital registration and death certification systems are weak and most people die at home without medical certification of the cause of death.3 A verbal autopsy is a method used to determine the cause of death from data collected about the symptoms and signs of illness and the events preceding death.4,5 This method is based on the hypothesis that the symptoms and signs surrounding most causes of death can be recognized, recollected and reported by a person present during the period prior to death.6,7 For the information provided by verbal autopsies to be maximally reliable, certain design characteristics need to be incorporated into studies based on verbal autopsy methods. To ensure that the data collected is of high quality, the data collection tool should have structured and unstructured questions, interviewers should be specially trained, interviewees should have remained close to the deceased during illness, and between death and data collection only a short time interval should have elapsed. Similarly, for cause of death to be accurately assigned, algorithms for translating the data into causes of death must be clearly defined and the possibility of assigning multiple causes of death (i.e. immediate, underlying and contributory) must be present. Finally, subsequent validation studies should be carried out.
To optimize the use of these methods, in the early 1990s WHO convened a series of expert meetings that led to the publication of several reports4,5,7,8 in which key design features for studies based on verbal autopsy methods were recommended. Three of the four reports - one of them the outcome of a workshop sponsored jointly by WHO and the United Nations Children's Fund - were published in international journals, and the fourth was published by WHO. However, the extent to which the guidelines were systematically disseminated to relevant parties and their effect on study design are unclear. Thus, the objective of this study was to determine whether the expert recommendations published from 1992 to 1994 influenced the way researchers conduct verbal autopsy studies.
Methods
Search strategy
From June 2005 to May 2006, we conducted computerized searches of PubMed and WHO9 databases using as key phrases "verbal autopsy", "mortality surveillance", "mortality statistics", "post mortem interview" and "cause of death". References quoted in original publications and in the websites of the International Network of Field Sites with Continuous Demographic Evaluation of Populations and their Health (INDEPTH Network)10 and the Adult Morbidity and Mortality Project11 were manually searched for additional information.
Study inclusion and exclusion criteria
To be included in this review, studies had to fulfil certain criteria. They had to: (i) have been published before 2006 in a peer reviewed journal or as a report accessible through a web search; (ii) describe the verbal autopsy method used, and (iii) be written in English or French. A study reporting data in more than one paper was included only once, and data from the paper which described the methods in the greatest detail were used.
Data extraction
Standard information was extracted from all eligible studies by one reviewer for English-language studies (RJ) and another for French-language studies (A-PK). The information included the following: country in which the data were collected; age group studied; number of deaths observed; approach used to classify deaths; questionnaire format; interviewers' education and gender; type of respondent from whom information was obtained; recall period; granting of consent; type and nature of adjudication process; number of causes of death assignable per case; and presence or absence of a validation study. Unless otherwise indicated, all studies were assumed to have been designed three years prior to publication.
Outcomes
For this study, outcomes were based on seven key recommendations extracted from the published guidelines (Table 1), as follows: (i) a combined questionnaire including both a series of structured questions and an open narrative section, (ii) a trained interviewer educated in verbal autopsy methods and interviewing skills, (iii) a suitable respondent who tended to the deceased during illness (usually a family member or caretaker), (iv) a recall period of less than 5 years between the dates of death and data collection, (v) predefined criteria or algorithms to guide the assignment of causes of death, (vi) ability to assign multiple causes of death, such as immediate, underlying, and contributory, and (vii) the performance of a validation study to ascertain the reliability of the causes of death assigned. Immediate, underlying and contributory causes are illustrated by the example of a death from stroke (underlying cause) resulting from a secondary pneumonia (immediate cause) in a patient with a history of hypertension (contributory cause).
Analysis
For the primary analyses, studies were divided into those designed before and after 31 December 1994. The proportion of studies in each of these two groups that contained each recommended methodological feature was compared. Sensitivity analyses were performed using the dates when the guidelines were laid down - 31 December 1992 and 31 December 1996 - with no effect on the study conclusions for any measure. Studies for which the particular indicator measured was not reported were excluded from the relevant analysis. Means were compared with Pearson's χ2 test, and all analyses were performed with SPSS version 12.0 (SPSS Inc., Chicago, IL, United States of America). Fisher's exact test was used for expected counts less than 5.
Results
The search identified 102 studies conducted in 39 developing countries (Appendix A, available at: http://www.thegeorgeinstitute.org/iih/index.cfm?7E6B9894-AED4-1907-C50B-BBFFB82559F8#methodological-trends-in-studies-using-verbal-autopsies). The first study was conducted in Bangladesh in 196812 and the last in the Democratic Republic of the Congo in 2003-2004.13 The 99 studies that reported data comprised a total of 139 258 deaths. The number of deaths in each study ranged from 15 to 80 000, with a mean of 1407 and a median of 271. Sixty studies were designed before and 42 studies after the end of 1994. Half of all the studies were of deaths in infants and children, about one-fourth were of maternal deaths and about one-fourth were about deaths in adulthood or at any age.
Methodology indices
The methodology indices used were reported in 80% of the studies or more, except in the case of the recall period and the performance of a validation study, which were reported in only 58.8% and 27% of the studies, respectively (Table 2). Of the studies that provided methodology data, two-thirds or more followed each of the four main guideline recommendations regarding data collection. Almost all studies used a suitable respondent and had a recall period of less than 5 years. There was, however, no strong evidence that these or other recommended data collection indices were used more often after 1994 (P > 0.22).
Recommended methods for assigning the cause of death were applied much less frequently than recommended data collection methods. Predefined algorithms for assigning the cause of death or the ability to assign multiple causes of death were found in only about one-third of the verbal autopsy studies. Once again, there was no evidence of a post-1994 increase in the proportion of studies in which the recommended ways of assigning the cause of death had been used (P > 0.33).Validation studies had been conducted in about three-fourths of the studies that reported their presence or absence, but only about one-fourth of all studies provided data about this aspect of study design. No statistical difference was found in the proportion of studies carried out before and after the end of 1994 for which a validation study was conducted (P = 0.69).
Discussion
This review provides a comprehensive update on the verbal autopsy studies performed in the world up to the time of this review and of the methods applied in them. It highlights the variability of the methods used and the negligible influence exerted by a series of methodological guidelines and reports developed in the early 1990s.4,5,7
The efforts undertaken in the early 1990s to standardize the design and conduct of verbal autopsy studies lead one to assume that suboptimal methods were felt to have important consequences. Similarly, the content of the recommendations issued at the time suggests that certain methodological features were felt to ensure a yield of maximally reproducible, valid and comparable information from subsequent verbal autopsy studies. Unfortunately, these features, which were identified and specified more than a decade ago, have scarcely been implemented, perhaps because researchers have not known about them, have lacked access to the journals that published them or have been unable to afford to apply them in full. It is also possible that the absence of evidence in support of many of the recommendations lessened their value in the eyes of researchers designing new studies. The explanation is beyond the scope of this study but would be a good topic for future research, since some of the possibilities listed above can be addressed quite easily.
It is difficult to know for certain what effect, if any, not following the guidelines had on study findings. Nearly all guideline recommendations appear to have been based on consensus expert opinion rather than on scientific evidence, and it is impossible to know if they do in fact represent best practice. However, despite the lack of evidence, the rationale behind most of the recommendations is plain. For example, the ability to assign multiple causes of death rather than a single cause leads to better opportunities to explore existing problems, such as national biases in cause of death assignment, and allows a more informed evaluation of study results. Likewise, a data collection strategy that combines structured and open data collection techniques is likely to lead to better data collection than one restricted to either technique alone.
In the present study, the evaluation of verbal autopsy study techniques was restricted to a few key design features that were identified as important in most of the recommendations and that could be easily and objectively extracted from most of the study reports identified. Aspects of study design besides those reported here may have been of equal or greater importance to study integrity. Of the methodological indices studied, several showed positive trends. However, there is no evidence to indicate that such trends are not the result of chance. While the analyses were limited in their ability to detect progress towards the incorporation of some recommended design features that were already being used fairly widely (mainly data collection indices), the study was well placed to detect the potential effects of the guidelines on less widely used features pertaining to cause of death assignment and validation. In particular, little movement was noted towards the use of pre-specified algorithms to aid the allocation of cause of death or the assignment of multiple causes of death, two methodological features that would probably greatly enhance the quality of information derived from verbal autopsy projects. Furthermore, despite specific recommendations in the guidelines, very few studies have attempted to validate their findings, and those that have reported doing so have used very different methods. Validation studies that compare the causes of death derived from the verbal autopsy against gold standard reference diagnoses are particularly necessary.
This review was limited by an absence of data from several studies, which reduced statistical power and introduced the risk of bias. Given the long time elapsed since many of the reports were published, contacting their authors directly was considered impossible and further bias could have been introduced by the likelihood that it was easier to contact the authors of more recent reports. Although efforts were made to identify all verbal autopsy studies conducted, some may have been missed due to publication in a language other than English or French or in technical reports not widely available. However, it is unlikely that the missing data would have led to a different conclusion regarding the uneven and rather poor quality of many verbal autopsy studies.
In the context of changing patterns of death in many lower-income countries that rely on verbal autopsy systems for mortality surveillance,14,15 high quality, standardized techniques are becoming increasingly important.16 Better uptake of key design recommendations would undoubtedly increase the reproducibility of verbal autopsy studies and the validity and comparability of their results, as well as enhance decision-making processes surrounding health care in lower-income settings. However, how uptake might be further enhanced is not immediately clear, since the recommendations already appear in leading medical journals and/or as downloadable technical reports.17,18 The establishment of an international network of researchers involved in conducting verbal autopsy projects may facilitate the design of future studies, although it would clearly be impossible to see to it that they were all performed within the framework of such a group. More systematic interaction between international collaborations, such as the INDEPTH Network,10 the Adult Morbidity and Mortality Project,11 India's Sample Registration System's prospective study19 and the Sample Vital Registration with Verbal Autopsy14 initiative, might be one practical way of enhancing the way verbal autopsy studies are conducted. A recent publication by WHO, representing one more effort to set international standards for verbal autopsy methods, is a positive step forward but unlikely to suffice in and of itself.20
Competing interests: None declared.
References
1. Lopez AD. Causes of death: an assessment of global patterns of mortality around 1985. World Health Stat Q 1990;43:91-9. PMID:2375132
2. Mathers CD, Fat DM, Inoue M, Rao C, Lopez AD. Counting the dead and what they died from: an assessment of the global status of cause of death data. Bull World Health Organ 2005;83:171-7. PMID:15798840
3. Kahn K, Tollman SM, Garenne M, Gear JSS. Who dies from what? Determining the cause of death in South Africa's rural north east. Trop Med Int Health 1999;4:433-41. PMID:10444319 doi:10.1046/j.1365-3156.1999.00415.x
4. Chandramohan D, Maude GH, Rodrigues LC, Hayes RJ. Verbal autopsies for adult deaths: issues in their development and validation. Int J Epidemiol 1994;23:213-22. PMID:8082945 doi:10.1093/ije/23.2.213
5. World Health Organization, United Nations Children's Fund. Measurement of overall and cause specific mortality in infants and children: memorandum from a WHO/UNICEF meeting. Bull World Health Organ 1994;72:707-13. PMID:7955018
6. Snow RW, Armstrong JRM, Forster D, Winstanley MT, Marsh VM, Newton CR, et al. Childhood deaths in Africa: uses and limitation of verbal autopsies. Lancet 1992;340:351-5. PMID:1353814 doi:10.1016/0140-6736(92)91414-4
7. Bang AT, Reddy MH. SEARCH team. Diagnosis of causes of childhood deaths in developing countries by verbal autopsy: suggested criteria. Bull World Health Organ 1992;70:499-507. PMID:1394784
8. Campbell O, Ronsmans C. Verbal autopsies for maternal deaths: report of a WHO workshop, London, 10-13 January 1994. Geneva: World Health Organization; 1995.
9. World Health Organization [Internet site]. Publications. Available from: http://www.who.int/publications/en/ [accessed on 3 June 2009] .
10. An international network of field sites with continuous demographic evaluation of populations and their health in developing countries [Internet site]. INDEPTH Network; 2005. Available from: http://www.indepth-network.org/ [accessed on 3 June 2009] .
11. Adult Morbidity and Mortality Project [Internet site]. Available from: http://research.ncl.ac.uk/ammp/ [accessed on 3 June 2009] .
12. Chen LC, Gesche MC, Ahmed S, Chowdhury AI, Mosley WH. Maternal mortality in rural Bangladesh. Stud Fam Plann 1974;5:334-41. PMID:4439448 doi:10.2307/1965185
13. Coghlan B, Brennan R, Ngoy P, Dofara D, Otto B, Stewart T. La mortalité en République Démocratique du Congo: résultats d'une enquête nationale réalisée d'avril à juillet 2004. New York, NY: International Rescue Committee; 2004. Available from: http://www.theirc.org/resources/Enquete_de_mortalite-Version_Francaise.pdf [accessed on 3 June 2009] .
14. Setel PW, Sankoh O, Rao C, Velkoff VA, Mathers C, Gonghuan Y, et al. Sample registration of vital events with verbal autopsy: a renewed commitment to measuring and monitoring vital statistics. Bull World Health Organ 2005;83:611-7. PMID:16184280
15. Okosun IS, Deve GE. Verbal autopsy: a necessary solution for the paucity of mortality data in the less-developed countries. Ethn Dis 2001;11:575-7. PMID:11763281
16. Anker M. The effect of misclassification error on reported cause-specific mortality fractions from verbal autopsy. Int J Epidemiol 1997;26:1090-6. PMID:9363532 doi:10.1093/ije/26.5.1090
17. Larson E. Status of practice guidelines in the United States: CDC guidelines as an example. [PMID: 12689796]. Prev Med 2003;36:519-24. PMID:12689796 doi:10.1016/S0091-7435(03)00014-8
18. Woodfield J. Spreading the word: disseminating research findings. Loughborough: The Water, Engineering and Development Centre; 1998.
19. Jha P, Gajalakshmi V, Gupta PC, Kumar R, Mony P, Dhingra N, et al. Prospective study of one million deaths in India: rationale, design and validation results. PLoS Med 2006;3:e18. PMID:16354108 doi:10.1371/journal.pmed.0030018
20. Baiden F, Bawah A, Biai S, Binka R, Boermad T, Byasse P, et al. Setting international standards for verbal autopsy. Bull World Health Organ 2007;85:570-1. PMID:17768508 doi:10.2471/BLT.07.043745
(Submitted: 14 November 2007 - Revised version received: 18 September 2008 - Accepted: 30 November 2008 - Published online: 30 June 2009)
* Correspondence to Rohina Joshi (e-mail: rjoshi@thegeorgeinstitute.org).