RESEARCH

A comparison of cluster and systematic sampling methods for measuring crude mortality

Comparaison de deux méthodes de mesure de la mortalité brute : le sondage en grappes et le sondage systématique

Comparación de los métodos de muestreo sistemático y por conglomerados en la medición de la mortalidad bruta

Angela MC Rose^I,1; Rebecca F Grais^I; Denis Coulombier^II; Helga Ritter^III

^IEpicentre, 8 rue St Sabin, 75011 Paris, France
^IIEuropean Centre for Disease Prevention and Control, Solna, Sweden
^IIIMédecins sans Frontières-Belgium, Brussels, Belgium

ABSTRACT

OBJECTIVE: To compare the results of two different survey sampling techniques (cluster and systematic) used to measure retrospective mortality on the same population at about the same time.
METHODS: Immediately following a cluster survey to assess mortality retrospectively in a town in North Darfur, Sudan in 2005, we conducted a systematic survey on the same population and again measured mortality retrospectively. This was only possible because the geographical layout of the town, and the availability of a good previous estimate of the population size and distribution, were conducive to the systematic survey design.
RESULTS: Both the cluster and the systematic survey methods gave similar results below the emergency threshold for crude mortality (0.80 versus 0.77 per 10 000/day, respectively). The results for mortality in children under 5 years old (U5MR) were different (1.16 versus 0.71 per 10 000/day), although this difference was not statistically significant. The 95% confidence intervals were wider in each case for the cluster survey, especially for the U5MR (0.15–2.18 for the cluster versus 0.09–1.33 for the systematic survey).
CONCLUSION: Both methods gave similar age and sex distributions. The systematic survey, however, allowed for an estimate of the town's population size, and a smaller sample could have been used. This study was conducted in a purely operational, rather than a research context. A research study into alternative methods for measuring retrospective mortality in areas with mortality significantly above the emergency threshold is needed, and is planned for 2006.

Keywords: Mortality; Data collection/methods; Sampling studies; Cluster analysis; Households; Sudan (source: MeSH, NLM).

RÉSUMÉ

OBJECTIF: Comparer les résultats obtenus par deux techniques d'enquête par sondage (en grappes ou systématique) utilisées pour mesurer rétrospectivement la mortalité dans une population identique, sur la même période environ.
MÉTHODES: Immédiatement après avoir réalisé en 2005 une enquête par sondage en grappe, destinée à évaluer rétrospectivement la mortalité dans une ville du nord du Darfour, au Soudan, on a procédé à une enquête par sondage systématique sur la même population et à une nouvelle mesure rétrospective de la mortalité. Cette opération n'a été possible que parce que la configuration géographique de la ville et la disponibilité de bonnes estimations antérieures de la taille et de la distribution de la population s'y prêtaient.
RÉSULTATS: Les méthodes de sondage en grappes et de sondage systématique ont donné toutes deux des résultats similaires inférieurs au seuil d'émergence pour la mortalité brute (0,80 contre 0,77 pour 10 000/jour respectivement). On a relevé un écart entre les chiffres de mortalité obtenus pour les enfants de moins de 5 ans (U5MR, taux de mortalité pour les moins de 5 ans) : 1,16 pour 10 000/j contre 0,71 pour 10 000/j), cet écart n'étant cependant pas statistiquement significatif. Dans le cas de l'enquête par sondage en grappes, les intervalles de confiance à 95 % étaient plus étendus pour chacune des tranches d'âges et spécialement pour les moins de 5 ans (0,15 - 2,18 pour le sondage en grappes contre 0,09 - 1,33 pour le sondage systématique).
CONCLUSION: Les deux méthodes ont abouti à des distributions similaires en fonction de l'âge et du sexe. L'enquête par sondage systématique a cependant permis d’estimer la population de la ville et elle aurait pu s'effectuer sur un échantillon plus petit. Cette étude a été menée dans une optique purement opérationnelle, plutôt que dans un contexte de recherche. Une étude plus théorique des méthodes utilisables pour mesurer rétrospectivement la mortalité dans des zones où elle est significativement supérieure au seuil d'émergence s’impose et devrait être réalisée en 2006.

Mots clés: Mortalité; Collecte données/méthodes; Etude échantillon; Sondage en grappes; Ménages; Soudan (source: MeSH, INSERM).

RESUMEN

OBJETIVO: Comparar los resultados de dos técnicas de muestreo encuestal (por conglomerados y sistemático) utilizadas para medir la mortalidad retrospectiva en la misma población de forma casi simultánea.
MÉTODOS: Inmediatamente después de realizar una encuesta por conglomerados para evaluar la mortalidad retrospectivamente en una localidad de Darfur Septentrional, Sudán, en 2005, llevamos a cabo una encuesta sistemática en la misma población y medimos de nuevo la mortalidad retrospectivamente. Esto sólo fue posible porque el trazado geográfico de la localidad y el hecho de disponer de una buena estimación previa del tamaño y la distribución de la población propiciaban la encuesta sistemática.
RESULTADOS: El método por conglomerados y el sistemático dieron resultados semejantes por debajo del umbral de emergencia para la mortalidad bruta (0,80 frente a 0,77 por 10 000/día, respectivamente). Los resultados para la mortalidad de menores de 5 años (TMM5) fueron distintos (1,16 frente a 0,71 por 10 000/día), aunque esta diferencia no fue estadísticamente significativa. Los intervalos de confianza del 95% fueron mayores en cada caso para la encuesta por conglomerados, especialmente en lo tocante a la TMM5 (0,15-2,18 para el método por conglomerados, frente a 0,09 - 1,33 para la encuesta sistemática).
CONCLUSIÓN: Ambos métodos arrojaron unas distribuciones de edades y sexos semejantes. La encuesta sistemática, sin embargo, permitió hacer una estimación del tamaño de la población, y podría haberse utilizado una muestra más pequeña. Este estudio se llevó a cabo en un contexto puramente operativo, no de investigación. Es necesario realizar un estudio de investigación sobre métodos alternativos para medir la mortalidad retrospectiva en las áreas con mortalidad significativamente superior al umbral de emergencia, y está previsto hacer tal cosa en 2006.

Palabras clave: Mortalidad; Recolección de datos/métodos; Muestreo; Análisis por conglomerados; Hogares; Sudán (fuente: DeCS, BIREME).

Introduction

The traditional two-stage cluster survey technique, although designed for estimation of vaccination coverage, is often used to provide rapid estimates of retrospective crude mortality rates (CMR) during emergency situations, e.g. in refugee camps or towns with a high propportion of internally displaced people (IDP).¹ Retrospective mortality surveys are used because there is often no registration system for deaths in these situattions. In the two-stage cluster sampling technique, people in 30 clusters of 30 households (30 × 30) are surveyed over a recall period that generally ranges from 30 days to 6 months previously.² The method does not require a list of residents, and is flexible enough to be conducted in a variety of situations. This technique has been validated for assessments of vaccine coverage (30 × 7) and nutritional status (30 × 30) in children,³ but never for CMR, although its use for this purpose is widespread.

Assuming there is no cyclic pattern in the distribution of sampling units, the systematic survey, in which households are selected using a fixed sampling interval and teams proceed in a systematic manner throughout the study area, arguably provides a more representative sample than the cluster.

The cluster sample method assumes that selection of a household within a cluster is not independent of the selection of other households; members of a cluster are therefore likely to be similar. This "design effect" may be especially serious when collecting information on certain infectious diseases, owing to their tendency to occur in clusters.

Mortality, especially in situations of conflict, may also be related to geographical area, for example where there are clusters of disease or violence targeted against a specific group. Thus, in the cluster survey design, a single cluster may by chance entirely incorporate an area of very high mortality, as was found by Roberts et al.⁴ This could lead to an underestimate of overall deaths, if such a cluster were by chance ] not included in the survey. On the other hand, deaths might be overestimated with this survey design if there is geographical clustering in mortality and the high-mortality clusters happen by chance to be over-represented in the survey. An attempt is usually made to account for such intracluster variability by applying a design effect of 2 when calculating the sample size for a cluster survey. This increases the required sample size by a factor of 2; thus the sample size needed for a cluster survey will always be twice that needed in a systematic survey.

At the end of a systematic survey, it is possible to estimate the population size, which is impossible in a cluster survey. It is also possible to compare the mortality according to other population characteristics (such as district or zone) after a systematic survey, sample size permiting; again, this is not possible with cluster survey analysis unless pre-sampling strata are introduced.

For measurements of mortality in a non-structured camp, or a town with no clearly defined streets, the systematic survey may not be the best option. Where the geographical layout of the area being surveyed is fairly geometric, however, it may be both easier and faster to conduct a systematic survey than a cluster survey. The cluster technique is more difficult to understand and thus necessitates more time for training the survey teams, to explain the rationale behind a "cluster" and the need for randomization at every step.

Assessing teams who are using the cluster technique for supervision purposes (e.g. by performing random spot checks) may be difficult, as it may be impossible to ascertain where the team has gone once it has left the centre of the cluster. With a systematic survey, however, it is much simpler to locate the teams, as the survey involves a logical progression from one end of the zone being assessed to the other.

From 21–25 May 2005, a survey was conducted by Epicentre in Serif Umra town in Darfur, Sudan, which assessed (among other indicators) retrospective mortality in 900 households using the traditional 30 × 30 two-stage random cluster survey technique. Immediately following this (28–29 May), retrospective mortality was assessed using the systematic sampling technique. Here we report on results comparing these two techniques for measuring CMR and mortality rate in children under 5 years old (U5MR) in the same population.

Methods

Study population

The population under study was that of the town of Serif Umra in North Darfur, Sudan, in May 2005. Sudan consists of a federation of states, each of which comprises several localities, which are further subdivided into administrative units and then into sectors. Serif Umra town lies in the sector of Serif Umra in Kabkabia, a locality home to a quarter of a million people in the south of North Darfur state. Serif Umra comprises 12 zones and is situated to the north-west of the locality, on the border with West Darfur state. The population estimate for Serif Umra town, according to a survey by local health visitors in February 2005, is 45 316. This is more than three times the original population of the town at the start of the Darfur conflict in 2003, and is the result of an influx of IDP from elsewhere in the region, most of whom arrived in Serif Umra in July–August 2003.

Serif Umra lies in a government-controlled area in which there has never been armed conflict. The town is a melting-pot of ethnic groups, and the lack of conflict could be the result of the varied interests of the population groups present, as well as a large weekly market, to which people come from all over Darfur. The town covers an area of about 5.5 km² and is laid out in orderly streets with clearly delimited compounds in most areas. On the periphery of the town, however, where the newest IDPs may be settling, borders between compounds are less clearly defined and many compounds and dwellings are empty. The survey by health visitors in February 2005 estimated an average of two households per compound with six persons per household (personal communication, Médecins sans Frontières-Belgium, May 2005).

Survey 1: two-stage 30 × 30 cluster sample survey

The two-stage 30 × 30 cluster sample survey was conducted for the assessment of retrospective mortality as well as for other indicators (not presented in this paper). Teams were trained over 3 days, including 1 day for piloting and adjustment of the questionnaires. Following the methodology of a previous survey conducted in 2004 (unpublished Epicentre/Médecins sans Frontières internal report, November 2004), the first stage of this survey consisted of the random selection of 30 clusters from a population list of the 12 geographically limited zones comprising the town. Each zone has a different ethnic composition and estimated population ascertained from an exhaustive survey conducted by health visitors earlier in 2005. Clusters were selected with a probability proportional to the size of the zone⁵ (Table 1). The calculation of sample size incorporated an estimated design effect of 2 and the CMR found in a previous cluster survey in the same town 7 months earlier (0.8 per 10 000/day). A sample size of 5464 was needed, with a precision of ± 0.50 per 10 000/day with 95% confidence, for a recall period of 45 days. With a known average household size of six people, we expected a survey of 30 clusters each surveying 30 households to yield approximately 5400 individuals (900 × 6).

Household selection

For the second stage, 30 households within each cluster were selected and questionnaires were completed during visits by six teams (comprising three investigators each) over 5 days, i.e. each team aimed to complete the survey of one cluster per day. This length of time was needed because information was being collected on all of the other survey indicators, which included measuring the weight and height of children aged 6–59 months (data not shown).

The team went firstly to the centre of the area being surveyed. This was the centre of the entire zone if that zone was of a size to warrant only one cluster. Where more than one cluster was needed, the zone was first divided into approximately equal areas (by population size), using a map previously prepared by the health visitor coordinator, in which each area represented one cluster. The team then went to the centre of the area within the zone corresponding to the cluster to be surveyed that day. Following the classic WHO Expanded Programme on Immunization (WHO EPI) methodology for cluster sampling surveys,2 the team then selected a direction at random from the choice of directions they faced at the centre, using a random number table. They then flipped a coin to select on which side of the street they would count compounds (as the town is fairly densely populated and there are usually compounds on both sides of each street).

Subsequently the team walked along the chosen street from the centre to the periphery, counting and marking the compounds with chalk on the chosen side of the street only, from 1 to N. The teams then selected a random number between 1 and N, using a random number table. They then returned to the selected compound, where they selected one or more households using the following algorithm: from compounds containing from one to three households, only one household would be surveyed. From compounds with four to six households, two would be surveyed. From compounds with seven or more households, three would be surveyed. Prior knowledge of the number of households per compound, from the survey by health visitors in February 2005, indicated that the average was two (with a range from one to seven, depending on the zone). First the households within a compound were numbered and then one of them was selected using a random number table.

A household was defined as a family with one head of household, eating and sleeping under the same roof. After surveying the household(s) in a compound, the team left, and, for the first compound in a cluster only, flipped a coin to determine in which direction they would go (left or right). They turned in this direction and continued to sample one or more households per compound, for consecutive compounds, until they had surveyed 30 households. If they came across a secondary street or a crossroads, they followed the same randomization principle by first choosing whether to stay on the street they were on or to turn off (for a single secondary street: by flipping a coin; if faced with more than one "new" street: by using the random number table). If they selected the option of turning off onto a new street, they would then again flip a coin to decide on which side of the street to continue the survey.

If there was no one available from the selected household to answer questions for the survey, the teams asked people in surrounding households or neighbouring compounds when they were expected to return. An appointment was made for the survey if the household members were away only for the day, and the compound was marked with chalk so that the team could find the household later on. For long-term absences (more than a week) where neighbours could not provide the expected date of return, the household was not included in the survey, but a note was made that an entire household was absent. Another household in the same compound was then selected at random.

Survey 2: systematic sampling of households

The systematic sample survey was performed to estimate mortality only. Each of the 12 zones was sampled systematically, and the results were pooled for the whole town. To obtain a sample of similar size for comparison with the cluster survey technique, at least 900 households needed to be sampled. Information on population size (45 316), number of compounds (3306) and average number of household members (six) from the exhaustive survey by health visitors earlier in the year indicated that there were approximately 7552 households in Serif Umra. To select 900 of these, we therefore needed to sample one in every eight households.

For logistic reasons, we decided to sample one in every three compounds, taking one household at random from each, regardless of how many households there were within the compound. This was because the available information suggested that the number of households per compound in Serif Umra ranged from one to seven (with an average of two). It would have been complex both to explain and to conduct a survey in which every eighth household was sampled, bearing in mind the previous constraint. The one-in-three algorithm, however, would give a sample size of at least 1102 compounds (3306/3), and therefore the same number of households, providing more than the 900 necessary for comparison with the cluster technique.

The teams who had previously conducted the cluster survey were trained for 1 day in the systematic sampling technique. The starting point in each of the 12 zones was a compound chosen at random from the first three compounds on the north-west boundary of the zone. This was done by numbering the compounds and then selecting one using a random number table. The teams moved systematically through the zone, street by street, until they had covered the entire zone. Each team leader prepared a rough sketch-map to show how his team had moved through his zone.

Questionnaires and analysis

Both surveys used the same mortality questionnaires, following the current household census approach, to collect individual data (using one questionnaire per household). The recall period went from the mid-point of the survey back to the date of the meningitis vaccination campaign conducted in the town about 6 weeks earlier.

Questionnaires were numbered by each team in the order the households were visited. Data were entered systematically by zone. A further analysis was done on these data, after artificially decreasing the sample size by half, to observe the effect of a smaller sample size on CMR and U5MR. For this purpose, a second database was created, from which all households with even numbers had been removed. In this way, we artificially created a new sampling interval of six.

Both CMR and U5MR were calculated as number of deaths per 10 000 population (at the end of the survey) per day. Data were entered in EpiData Version 3.1 (The EpiData Association, Odense, Denmark) and analysed in Stata Version 8.2 (StataCorp, College Station, TX, USA).

Results

Training and survey completion

Cluster survey

The cluster survey took 5 days to complete, after 3 days of training, using six teams of health visitors, with three members in each team (from a "pool" of 45 health visitors working in the town).

Systematic survey

The systematic survey used 17 teams with two members in each. Each team included at least one health visitor who had also participated in the cluster survey. Each zone also had one main supervisor, who was the regular health visitor coordinator for that zone. Only one training day was needed, and teams generally completed their survey in one day, taking an average of 6 hours (not counting rest breaks). Two teams continued for a few hours into the second day to complete the survey in their zones.

Description of the population

Table 2 shows the values for the demographic variables calculated for the two types of survey method. The age pyramids for both types of survey (Fig. 1) show fewer males aged 15–35 years than females, but more males aged 5–14, and over 54 years.

The systematic survey confirmed the information from the health visitors' survey earlier in the year that there was an average of six people per household. The "new" estimated population size for Serif Umra was therefore 1188 (the number of households surveyed) × 3 (one in every 3 compounds surveyed) × 2.3 (average number of households per compound from previous survey) × 6.1 (average number of people per household) = 50 003. This reflected an increase of 4687 people in the 3 months since the exhaustive survey had been conducted, or about 1500 people per month.

Retrospective mortality

The results for the CMR and U5MR for both surveys are shown in Table 2 and Fig. 2. As the recall period was slightly longer than that estimated in our initial calculations for sample size, and the calculated cluster survey design effect was 1.0, the precision for the CMR estimate for the cluster survey was ± 0.34 per 10 000/day. For the systematic survey this was ± 0.28 per 10 000/day.

The CMR was similar for both survey methods, for both the point estimate and the 95% confidence intervals (CIs). For both methods, the upper limit of the 95% CI bordered on the emergency threshold of 1 death per 10 000 population per day.⁶ For the U5MR, although the systematic survey yielded a lower estimate than the cluster survey, the 95% CIs overlapped. The upper limit of the 95% CI for the cluster survey was above the emergency threshold of 2 deaths per 10 000/day, unlike that for the systematic survey.

The re-analysis, using half of the systematic survey sample, gave the CMR and U5MR shown in Table 2. As expected, the 95% CIs widened for both CMR and U5MR, more closely approximating those obtained for the cluster survey.

Discussion

Our results show that the traditional two-stage cluster survey design can give similar results to the systematic survey design when making rapid estimates of retrospective mortality in a setting where the mortality point estimate is not above the emergency threshold.

There were some important limitations to our study. Firstly, the sample sizes for both surveys were small. As with much emergency work, retrospective mortality surveys are often done rapidly on low budgets under difficult conditions. For this study, a cluster survey was needed for comparison of mortality with that from a previous cluster survey conducted in the same town 7 months earlier. The opportunity to do a systematic survey alongside the cluster survey for comparison purposes, and to provide information for future research, was therefore seized. The larger sample sizes needed for a more precise comparison would have meant higher costs which, in the operational context in which the cluster survey was undertaken, may have been hard to justify.

Calculation of sample size during study preparation should also take into account the size of the population aged under 5 years. If sample size is calculated considering only CMR, the resulting 95% CIs for U5MR can be so wide that interpretation can be difficult (note the much wider 95% CIs for the U5MR in our study, especially for the cluster survey, seen in Fig. 2).

Secondly, the cluster design in this study, in terms of household selection, did not follow exactly the traditional WHO EPI design for household selection (in terms of choosing the "next-nearest household" strictly by proximity). The design was intended to lessen selection bias by providing rules for how to respond to various choices; for example, when faced with more than one possible direction to follow, or having to decide from which side of the street to select households. In addition, it ensured that teams were more likely to have followed the same algorithm for household selection by attempting to provide as much clear guidance regarding choices as possible. Although selection bias may have been reduced, this design may have led both to more clustering (as teams were following streets rather than selecting the next household based on proximity) and to less overall representativeness of the sample (as any households situated in remote locations or off the street, would have been less likely to have been selected). From our results, however, we know that this design did not lead to more clustering, as our overall design effect for CMR was 1.01.

Thirdly, perhaps the most serious limitation was that in neither survey did every household have an equal chance of being selected. For the cluster survey, an algorithm was followed for selecting one, two or three households at random from compounds containing more than one household. As the town average was two households per compound, most households in a compound in the cluster survey would have had a 50% chance of selection. However, some would have had a 100% chance (only one household in a compound) whereas others would have had only a 33% chance (three or six households in a compound). In the systematic survey, both for logistic reasons and to avoid an excessively complex process of household selection, only one household per compound was selected, regardless of compound size. So each compound had a 33% chance of selection (as the sampling interval was three), but the chance of selection for a household varied according to compound size. As information on number of households within each compound was not collected during the survey, we could not subsequently introduce a probability weighting into the analysis to adjust for these differences.

Finally, although this study compared the cluster with the systematic survey, the latter is not the ideal survey method for measuring CMR. Ideally we should have conducted an exhaustive household survey with which to compare the cluster survey; there is a research study currently in preparation with this aim. The present study was conducted under operational conditions and in a very limited time, to provide some information for future research as well as to investigate the feasibility of conducting such studies together.

Conclusion

As shown by our rapid assessment, both methods yielded a similar result. However mortality in this town was not above the emergency threshold, and it is possible that in areas of high mortality, the cluster technique might prove less suitable. In the field, options such as systematic (or even exhaustive) surveys are not often considered, and cluster surveys may be conducted almost automatically. We suggest that other options can be as good as the cluster survey depending on the field conditions, although these options need to be validated. If the geographical setting permits field teams to cover the area being surveyed in a systematic way, the systematic survey method can provide more robust estimates, with time savings in both training and implementation.

Further studies are planned, in which several alternatives to the cluster method will be investigated fully, especially in areas with high mortality rates (above emergency thresholds), as these may highlight differences between methods that could not be revealed by our study.

Acknowledgements

We are very grateful to the people of Serif Umra for their cooperation and hospitality. We particularly wish to thank Dr Magdi Osman, Ministry of Health, Sudan for his support with this research. Our thanks go to Médecins sans Frontières-Belgium staff in Khartoum and North Darfur for logistic and other support, and particularly to the health visitor teams for data collection and to the translators. Thanks also to Philippe Guérin and Bernadette Gergonne for early advice on the study.

Competing interests: none declared.

References

1. Depoortere E, Checchi F, Broillet F, Gerstl S, Minetti A, Gayraud O, et al. Violence and mortality in West Darfur, Sudan (2003-04): epidemiological evidence from four surveys. Lancet 2004;364:1315-20.        

2. Henderson RH, Sundaresan T. Cluster sampling to assess immunization coverage: review of experience with a simplified method. Bull World Health Organ 1982; 60:253-60.        

3. Lemeshow S, Robinson D. Surveys to measure programme coverage and impact: a review of the methodology used by the expanded programme on immunization. World Health Stat Q 1985;38:65-75.        

4. Roberts L, Lafta R, Garfield R, Khudhairi J, Burnham G. Mortality before and after the 2003 invasion of Iraq: cluster sample survey. Lancet 2004;364:1857-64.        

5. Levy PS, Lemeshow S. Sampling of populations: methods and applications. Third Edition. New York, NY: John Wiley & Sons, Inc; 1999.        

6. Spiegel P, Sheik M, Gotway-Crawford C, Salama P. Health programmes and policies associated with decreased mortality in displaced people in postemergency phase camps: a retrospective study. The Lancet online, 10 November 2002. Available at: URL: http://image.thelancet.com/extras/01art11089web.pdf        

(Submitted: 14 December 2005 – Final revised version received: 30 January 2006 – Accepted: 30 January 2006)

1 Correspondence to Ms Rose (email: angela.rose@epicentre.msf.org).