RESEARCH

Population-based simulations of influenza pandemics: validity and significance for public health policy

Simulations en population d'une pandémie de grippe : validité et signification pour les politiques de santé publique

Simulaciones poblacionales de las pandemias de gripe: validez e importancia para las políticas de salud pública

Toomas Timpka^I,*; Henrik Eriksson^II; Elin A Gursky^III; James M Nyce^IV; Magnus Morin^V; Johan Jenvald^V; Magnus Strömgren^VI; Einar Holm^VI; Joakim Ekberg^I

^IDepartment of Medical and Health Sciences, Linköpings universitet, SE-581 83 Linköping, Sweden
^IIDepartment of Computer Science, Linköpings universitet, Linköping, Sweden
^IIIAnalytic Services Inc., Arlington, VA, United States of America (USA)
^IVDepartment of Anthropology, Ball State University, Muncie, IN, USA
^VVSL Systems AB, Linköping, Sweden
^VIDepartment of Social and Economic Geography, Umeå University, Umeå, Sweden

RESUMO

OBJECTIVE: To examine the validity and usefulness of pandemic simulations aimed at informing practical decision-making in public health.
METHODS: We recruited a multidisciplinary group of nine experts to assess a case-study simulation of influenza transmission in a Swedish county. We used a non-statistical nominal group technique to generate evaluations of the plausibility, formal validity (verification) and predictive validity of the simulation. A health-effect assessment structure was used as a framework for data collection.
FINDINGS: The unpredictability of social order during disasters was not adequately addressed by simulation methods; even minor disruptions of the social order may invalidate key infrastructural assumptions underpinning current pandemic simulation models. Further, a direct relationship between model flexibility and computation time was noted. Consequently, simulation methods cannot, in practice, support integrated modifications of microbiological, epidemiological and spatial submodels or handle multiple parallel scenarios.
CONCLUSION: The combination of incomplete surveillance data and simulation methods that neglect social dynamics limits the ability of national public health agencies to provide policy-makers and the general public with the critical and timely information needed during a pandemic.

RÉSUMÉ

OBJECTIF: Examiner la validité et l'utilité de simulations d'une pandémie de grippe visant à informer les décisions pratiques en matière de santé publique.
MÉTHODES: Nous avons recruté un groupe multidisciplinaire de neuf experts pour évaluer une simulation d'étude de cas sur la transmission de la grippe dans un comté suédois. Nous avons utilisé la technique du groupe nominal (méthode non-statistique) pour obtenir des évaluations de la plausibilité, de la validité formelle (vérification) et de la validité prédictive de la simulation. Une structure d'évaluation des effets sanitaires a été utilisée comme cadre pour recueillir des données.
RÉSULTATS: Les méthodes de simulation ne prennent pas correctement en compte l'imprédictibilité de l'ordre social pendant les catastrophes : même des perturbations mineures de l'ordre social peuvent invalider les hypothèses infrastructurelles clés à la base des modèles actuels de simulation d'une pandémie. En outre, on observe une relation directe entre la flexibilité du modèle et le temps de calcul. Par conséquent, les méthodes de simulation ne peuvent, dans la pratique, tolérer l'intégration d'un ensemble cohérent de modifications dans les sous-modèles microbiologiques, épidémiologiques ou spatiaux, ou encore traiter des scénarios parallèles multiples.
CONCLUSION: Le manque de complétude des données de surveillance et l'utilisation de méthodes de simulation qui négligent la dynamique sociale limitent la capacité des agences de santé publique nationales à fournir en temps utile aux décideurs et à la population générale les informations indispensables pendant une pandémie.

RESUMEN

OBJETIVO: Analizar la validez y la utilidad de las simulaciones de pandemias orientadas a fundamentar la adopción de decisiones prácticas en materia de salud pública.
MÉTODOS: Organizamos un grupo multidisciplinario de nueve expertos para que evaluaran una simulación de estudios de casos de transmisión de la gripe en un distrito de Suecia. Mediante una técnica no estadística de grupos nominales se generaron evaluaciones de la plausibilidad, la validez formal (verificación) y la validez predictiva de la simulación. Como marco de recogida de datos se usó una estructura de evaluación de los efectos sanitarios.
RESULTADOS: La impredecibilidad de los cambios del orden social en las situaciones de desastre es un aspecto que los métodos de simulación no abordaron adecuadamente; incluso ligeras perturbaciones del orden social pueden restar toda validez a algunos supuestos básicos sobre las infraestructuras empleados en los actuales modelos de simulación de pandemias. Además, existe una relación directa entre la flexibilidad de los modelos y el tiempo de computación. El resultado es que, en la práctica, los métodos de simulación no admiten cambios integrados de los submodelos microbiológicos, epidemiológicos y espaciales, ni pueden tampoco manejar varios escenarios paralelos.
CONCLUSIÓN: La confluencia de unos datos de vigilancia incompletos y unos métodos de simulación que ignoran la dinámica social limita la capacidad de los organismos nacionales de salud pública para proporcionar a las instancias normativas y el público en general la información crucial y puntual que se necesita durante una pandemia.

Introduction

By 2006, many countries had responded to WHO initiatives to update contingency plans to mitigate the consequences of an influenza pandemic. However, some general concerns arose in connection with these national plans,^1,2 as it became clear that there would be a shortage of antiviral drugs and vaccine³ and that a pandemic would place new demands on public health information systems. At the global level, WHO's Global Influenza Surveillance Network (FluNet) collects and processes influenza data from 83 countries,⁴ but at the national level few public health surveillance systems can either detect pandemic outbreaks or warn relevant agencies and the public. This inadequacy persists despite a 2005 report to the Government of the United States of America (USA) that identified public health information systems as a priority area for restructuring and investment to secure preparedness for pandemics and bioterrorist attacks.⁵ The development of a National Health Information Infrastructure in the USA had, at the time of the 2005 report, been proposed to detect atypical patterns of health-care use and to provide essential health information to citizens.⁶ This recommendation, however, has not translated into widespread practice, and many health information infrastructure projects remain in the planning stages.

Given that surveillance systems for collecting and analysing pandemic data are not sufficiently robust as a resource for policy planning and decision-making, attention has shifted towards computer-based simulation models. Using artificially generated community models as a basis, workers have forecast the effectiveness of different intervention strategies for containing or delaying the influenza pandemic at its expected source (e.g. rural south-east Asia). Longini et al.⁷ found that if the basic reproductive number (R₀) - the average number of secondary cases that a single case is expected to produce while still infectious in a completely susceptible population - was below 1.60, a prepared response with targeted antiviral drugs would have a high probability of containing the disease. When prevaccination was introduced into the model, targeted antiviral prophylaxis was found to contain an outbreak with an R₀ as high as 2.1. Addressing the same research question, but using an individual-based stochastic simulation model, Ferguson et al.⁸ reported that a combination of geographically targeted prophylaxis and social distancing measures is feasible only if the R₀ is below 1.8. Simulation studies have also included international air transportation patterns in the analyses of the early phases of a pandemic. Colizza et al.⁹ reported that the large-scale therapeutic use of antiviral drugs in all affected countries would mitigate a pandemic effect with an R₀ as high as 1.9 during the first year, if one assumes the antiviral drug supply is sufficient to treat approximately 2-6% of the population and that case detection and drug distribution are efficient. More recently, methods for representing specific social-contact networks in analyses of local influenza transmission have been developed. Using artificially generated social networks grounded in typical American community structures in their analyses, Glass & Glass¹⁰ have suggested that high-school students may form the local transmission backbone of the next pandemic. Therefore, closing schools and keeping students at home during a pandemic would remove the transmission potential in these age groups and could effectively thwart subsequent spread of the disease within a community.

In the absence of reliable pandemic detection systems, computer-based simulations have become an important information tool for both policy-makers and the general public. In this study we examine the validity and usefulness of population-based pandemic simulations from a national-level public health perspective. Specifically, we assess a simulated pandemic influenza outbreak in a Scandinavian community using a non-statistical nominal group technique.

Methods

Case-study simulation

The purpose of the case-study simulation was to investigate two intervention strategies - antiviral drugs and public policy interventions - on influenza transmission in a Swedish municipality. Specifically, we aimed to examine the effects on simulated intervention outcomes of variations in local sociodemographic data, such as alternative population distributions and household structures.

We simulated two different supply situations for each drug and quantified their respective effects as coefficients that modify the basic transmission probabilities assigned to mixing groups. Public policy interventions aimed at reducing the number of contacts were represented as the probabilities that individuals would withdraw from particular mixing groups. In our simulation experiment, we closed schools in an attempt to eliminate interaction within such groups. The resultant information was used to plan a public health response based on the Haddon matrix¹¹ (Table 1). Details of the simulator design, the case study and the results are provided in Appendix A and Appendix B (available at: http://www.crisim.org/documents ).

Assessor panel

We formed a multidisciplinary group of experts with skills in the realistic and practical application of simulations in public health policy-making to assess the plausibility, formal validity and predictive validity of the case-study simulation.¹⁴ The panel of nine assessors was made up of an experienced public health manager, a professor of social medicine, a professor of computer science, a professor of social and economic geography, a professor of medical anthropology, a former head of the simulations section at a national department of defence, a software developer with extensive commercial experience, a social forecasting researcher and a cognitive scientist. Two assessors (HE, MM ) designed and implemented the case-study simulation environment, and two (TT, JJ ) contributed to its design.

Data collection

The nominal group technique was used to assess the case-study simulation. A nominal group analysis is the structured use of group processes for systematically soliciting a set of informed judgments on issues described by limited scenarios or case descriptions.

The health-effect assessment structure was used as a framework for data collection.¹⁵ Specifically, we used an adaptation of the scheme suggested by Veerman et al.¹² and focused the assessment on plausibility, formal validity (verification) and predictive validity. Further details of the framework used for assessing the case-study simulation are in Appendix C (available at: http://www.crisim.org/documents). The experts were instructed to "assess the case-study methods and results with reference to the health impact scheme".

Data analysis

The experts provided the first round of individual comments to the study coordinator, who included them in a case-study assessment document. The data analysis proceeded in cycles during which the experts first individually reviewed the assessment document and then participated in telephone group conference discussions (12 sessions lasting 90 minutes each). When new analysis cycles did not yield significant changes to the document, the assessment findings were considered to be established. In the second step of the analysis, the experts were asked to formulate the implications of the assessment results for simulation strategies on the basis of their own expertise and of the published literature. Specifically, the instructions for the second task were to "analyse the assessment results with respect to practical implications for the application of simulations in public health planning". The experts first provided individual comments, which were composed by the case-study coordinator. Thereafter implications continued to be formulated in a process in which the experts independently reviewed a preliminary case-study document describing the case-study implications. The comments thus gathered were subsequently circulated to the entire expert group, and a consensus document was iteratively drawn up.

Results

Verification of social assumptions

The assessment came to focus on the intersection between the simulations of biological events and of societal processes in the case-study, particularly on the model of social order. When assessing the formal validity of the simulation, the assessors noted that central aspects of the baseline situation in the case-study community had not been included in the models. The implementation of pandemic response plans presumes a close collaboration between many groups, including public health organizations, commercial companies, and law enforcement agencies, and the successful operation of these plans is critically dependent on the protection of these coordinated processes. Although important lessons were learned during previous influenza pandemics¹⁶ and the 2003 epidemic of severe acute respiratory syndrome (SARS),¹⁷ the interdependence among the broad range of processes occurring in a society under true stress (such as a pandemic) is not fully understood. In other words, even relatively minor breaches of the social order may significantly affect key infrastructural elements underpinning current pandemic response strategies. This frailty in the response strategies was not reflected in the description of the baseline situation in the case-study community.

For instance, assumptions were made about an efficient and sustained distribution of available antiviral drugs. However, in the USA pharmaceuticals are often distributed through retail and food stores, whose employees are at particular risk of becoming infected because of the large number of customers. Moreover, many professionals involved in the pandemic response may fail to report to work during a pandemic for reasons other than actually falling ill, e.g. because of breakdown of transportation systems, deficient law enforcement or having to take care of children due to school closures. ¹⁸

We do not know how an influenza pandemic might affect logistics and staff in the distribution chains for pharmaceuticals, hygienic supplies and sanitary equipment during a catastrophic event.¹⁹ Unforeseen changes in behavioural patterns may also occur among acute care and hospital workers,²⁰ who in particular may be prone to abandoning their tasks if they have shorter tenure, high work stress and already strained social relationships.²¹

Cumulative increase of predictive validity

The global spread of an influenza pandemic has been estimated to accelerate at an exponential rate from around day 50.²² Thus, national governments have a short window of time to plan and implement appropriate response measures. The expert panel concluded that before the detection of an outbreak threatening to become a pandemic, effects on social order may not be predictable and the nature of the infectious agent, the efficacy of vaccines, and the availability of antiviral drugs can only be estimated. Thus, the predictive validity of simulations at early stages of a pandemic will inevitably be poor. To increase the validity of the predictions, the submodels used in simulations need to be modified as more information becomes available about factors such as the virulence of the strain and the efficacy of intervention strategies. In the case study, the simulation submodels were rendered flexible by separating them from each other and from the execution algorithms in the software. This separation saved both considerable programming and programme testing time. However, a major drawback of the flexible software identified in the assessment was the length of time each estimation took. For instance, a single estimation of R₀ in the case-study community (140 000 population) lasted approximately 5 hours on a standard personal computer. This direct relation between flexibility and calculation time is a disincentive to using simulations with high predictive value for national-level policy-making during a pandemic, when time is critical.

Towards socially contingent pandemic simulations

In the case-study simulation, questions were formulated in a standard public health framework under the assumption of a sustained social order during the pandemic. A more valid strategy, however, would have been to alter key social structures and processes as a function of disease effect on the community. Unfortunately, little is known about the theoretical and practical means for integrating the simulation of biological events, such as virus transmission between human hosts, with dynamic models of changes in population behaviour. Multi-level simulations based on "synergetics" and game theory,²³ simulations of policy strategies²⁴ and simulations based on geographically explicit data²⁵ are established fields of population research. Similarly, the simulation of virus dissemination in stable societies is also an established field of research on its own.²⁶ Nevertheless, few frameworks are available to support integrated modelling in these conceptually discrete but practically interrelated areas. Moreover, before biological and societal factors can be integrated, a valid common theoretical basis must be established.

Time geography^27,28 captures concurrent social processes as they unfold in time and space, based on the interaction between individuals and groups within the constraints of the physical and biological environment. This theory can be applied to integrate biological and societal tiers in pandemic simulations. In this way, questions analysed in standard public health planning frameworks can be made dependent on changes in the social order (Fig. 1). However, execution of such two-tier simulations requires even more complex and time-consuming computations. To save scarce planning time during a pandemic, generic (XML-based) software specifications can be used to distribute data and algorithms for parallel computing. Such computing arrangements can technically be administered in routine public health settings in the form of arrays of standard computers.

Discussion

Population-based simulations are important sources of knowledge when planning national-level public health responses to pandemic influenza. The results of the case-study simulation assessment primarily apply to the "mixing group" approach to population-based simulations but can be extended to all models in which societies are represented as "compartments" of identical individuals mixing randomly. However, we noted that the formal validity of these simulations is challenged by the failure to take into account the behaviour of people during a pandemic and its effect on the social order. Additionally, the assessment led to questions about how simulation models can be adjusted to reflect dynamic changes in preconditions, e.g. disruptions to drug distribution routines.

Formal methods for macro-level societal forecasting were introduced early in the twentieth century.²⁹ In this research area, disruptions of the normal social organization have been expected, mainly when material resources are scarce or misallocated. When societal forecasting methods are applied to pandemic simulations, the lack of empirical grounding for using formal methods representing social processes in societies under severe strain comes to constitute a dangerous source of error. For example, the estimated effects from quarantine measures in Toronto during the SARS outbreak were diminished because disruptions of the social order led to compliance rates of only 57%.¹³

Furthermore, the application of formal methods to analyse societies under pressure has been strongly opposed by several prominent social scientists. Weber³⁰ and more recently Giddens³¹ have argued that using poorly validated models of social order leads to misunderstandings about the social world and how it operates. Although a central belief during the Enlightenment was that the social order could be controlled if science were sufficiently strong, modern social science, despite being more exact, no longer claims that changes in the social order in societies under severe pressure can be predicted. Moreover, researchers have only just begun to understand how disease shapes behavioural norms, and through them, social structures.³² The limitation of societal forecasting to stable societies is valid for time geography as well. In the context of pandemic simulations, the use of models and concepts from time geography may, however, facilitate the identification of social structures and processes that are likely to remain stable during a pandemic, and those structures that are at high risk of disintegration.

Another issue highlighted by the assessment was the direct relationship between computational time and flexibility in simulation models. In the case study, model flexibility was prioritized above computational efficiency. This strategy rests on the supposition that predictive validity can be increased if assumptions made early during a pandemic can progressively be replaced by validated observations and empirical data. In such situations, however, it is essential to "catalogue" the assumptions to permit accurate changes in the simulation models as new data become available. For example, in the case study it was assumed that asymptomatic infected individuals had a 50% lower risk of transmitting the virus than symptomatic ones. This assumption, which was based on estimates from previous simulation studies^33,34 is to be replaced when empirical data about an actual pandemic strain become available. For each such assumption to be included in a pandemic model, a choice must be made between re-examination of available data and acceptance of a previous estimate.

One possible way to systematically improve the predictive value of pandemic simulations is to include explicit scenario management in the methods. A scenario makes explicit how forecasts are concluded from hypothetical reasoning by overtly drawing out the expected consequences of specific facts and assumptions.^35,36 The use of scenarios is well established as a basis for public health responses to outbreaks of infectious disease.³⁷ Explicit scenario management would make it possible to store and display, at each point in time, the data, information and assumptions employed for a specific simulation model, together with their sources. Flexible simulation models combined with the ability to trace the data included and assumptions to their sources would make it possible to maintain evidence-based scenarios as a basis for improving predictive validity.

Our study has several limitations. First, the expertise of the assessors may not have been broad enough to cover all areas of relevance associated with pandemic simulations. For instance, expertise in legislative issues and statistics might have been useful. However, we kept the number of assessors (nine) to the minimum required by the nominal group technique (n > 8)¹⁴ to facilitate telephone conferences, which we found very useful in bringing together the views of experts in very different locations.

Second, procedural factors may have introduced bias during the case-study review. Originally, we chose the nominal group technique to avoid having particular individuals exert undue influence in face-to-face discussions. However, other researchers contend that participants in a consensus process are prone to reach an agreement if they are given time limits.³⁸ This "bandwagon" effect may be less likely when inspection documents and telephone conferences rather than questionnaires and graphic charts are used for feedback. To avoid premature conclusions, we continued the iterations until no individual group member contributed new comments upon inspecting the documents. Despite these measures, the nominal group procedure cannot be regarded as a guarantee against bandwagon effects or the dominance of individual experts.

Third, the case study did not incorporate the entire scope of pandemic simulation methods and techniques. In simulations of social events using multi-agent models, the behaviour of the agents and the attributes of their environment can both be modelled.³⁹ These multi-agent simulations are not intended to forecast or replicate observables.^23,40 Rather, their aim is to explain the systematic effects of behavioural patterns in explicit "artificial worlds". Several of the issues raised in the present study may not apply to multi-agent simulations. However, the latter might be able to contribute important insights into the interplay between biological processes and changes in population behaviour during planning for a pandemic if appropriate interpretation frameworks are developed. Similarly, not all issues raised necessarily apply to the evolving infectious disease simulation paradigm with explicit representations of empirically grounded social networks.^41,42 However, baseline data on such networks are still scarce,^43,44 and methods for progressively modifying the network models to represent observed behavioural changes during pandemics are needed.

Our results demonstrate that the current methods applied for population-based pandemic simulations have important shortcomings. Formal validity is poor because assumptions about how human populations respond to stress are not included in the simulation models. Predictive validity suffers because simulation methods do not support the dynamic representation of the interaction between the microbiological, epidemiological and societal progressions during a pandemic.

Pandemic simulation methods need to be reformed to include representation of social dynamics and support for rapid model changes if national public health agencies are to provide policy-makers and the general public with important, valid and timely information. Such reformed methods will require a change of analysis paradigm from forecasting to "near-to-real-time" or "nowcasting" and the use of surveillance data for continual updating of simulation models and parameters.

Funding: This work was supported by the Swedish Emergency Management Agency (SEMA-KBM) under contract 0700/2004.

Competing interests: None declared.

References

1. Mounier-Jack S, Coker RJ. How prepared is Europe for a pandemic influenza? Analysis of national plans. Lancet 2006;367:1405-11. PMID:16650650 doi:10.1016/S0140-6736(06)68511-5         

2. Uscher-Pines L, Omer SB, Barnett DJ, Burke TA, Balicer RD. Priority setting for pandemic influenza: an analysis of national preparedness plans. PLoS Med 2006;3:e436. PMID:17048982 doi:10.1371/journal.pmed.0030436         

3. Groeneveld K, van der Noordaa J. Use of antiviral agents and other measures in an influenza pandemic. Neth J Med 2005;63:339-43. PMID:16244380         

4. Hampson AW. Surveillance for pandemic influenza. J Infect Dis 1997;176 Suppl 1;S8-13. PMID:9240687 doi:10.1086/514184         

5. Gursky E. Epidemic proportions: building national public health capabilities to meet national security threats. Arlington, VA: ANSER Institute of Homeland Security; 2005.         

6. Yasnoff WA, Humphreys BL, Overhage JM, Detmer DE, Brennan PF, Morris RW, et al. A consensus action agenda for achieving the national health information infrastructure. J Am Med Inform Assoc 2004;11:332-8. PMID:15187075 doi:10.1197/jamia.M1616         

7. Longini IM Jr, Nizam A, Xu S, Ungchusak K, Hanshaoworakul W, Cummings DA, et al. Containing pandemic influenza at the source. Science 2005;309:1083-7. PMID:16079251 doi:10.1126/science.1115717         

8. Ferguson NM, Cummings DA, Cauchemez S, Fraser C, Riley S, Meeyai A, et al. Strategies for containing an emerging influenza pandemic in Southeast Asia. Nature 2005;437:209-14. PMID:16079797 doi:10.1038/nature04017         

9. Colizza V, Barrat A, Barthelemy M, Valleron AJ, Vespignani A. Modeling the worldwide spread of pandemic influenza: baseline case and containment interventions. PLoS Med 2007;4:e13. PMID:17253899 doi:10.1371/journal. pmed.0040013         

10. Glass LM, Glass RJ. Social contact networks for the spread of pandemic influenza in children and teenagers. BMC Public Health 2008;8:61. PMID:18275603 doi:10.1186/1471-2458-8-61         

11. Haddon W Jr. A logical framework for categorizing highway safety phenomena and activity. J Trauma 1972;12:193-207. PMID:5012817 doi:10.1097/00005373-197203000-00002         

12. Veerman JL, Mackenbach JP, Barendregt JJ. Validity of predictions in health impact assessment. J Epidemiol Community Health 2007;61:362-6. PMID:17372299 doi:10.1136/jech.2006.047480         

13. Svoboda T, Henry B, Shulman L, Kennedy E, Rea E, Ng W, et al. Public health measures to control the spread of the severe acute respiratory syndrome during the outbreak in Toronto. N Engl J Med 2004;350:2352-61. PMID:15175437 doi:10.1056/NEJMoa032111         

14. Jones J, Hunter D. Consensus methods for medical and health services research. BMJ 1995;311:376-80. PMID:7640549         

15. Kemm J. Perspectives on health impact assessment. Bull World Health Organ 2003;81:387. PMID:12894317         

16. Knobler SL, Mack A, Mahmoud A, Lemon SM, eds. The threat of pandemic influenza: are we ready? Washington, DC: The National Academies Press; 2005.         

17. Wang TH, Wei KC, Hsiung CA, Maloney SA, Eidex RB, Posey DL, et al. Optimizing severe acute respiratory syndrome response strategies: lessons learned from quarantine. Am J Public Health 2007;97 Suppl 1;S98-100. PMID:17413071 doi:10.2105/AJPH.2005.082115         

18. Mackler N, Wilkerson W, Cinti S. Will first-responders show up for work during a pandemic? Lessons from a smallpox vaccination survey of paramedics. Disaster Manag Response 2007;5:45-8. PMID:17517362 doi:10.1016/j. dmr.2007.02.002         

19. Davis T, ed. A failure of initiative: the final report of the select bipartisan committee to investigate the preparation for and response to hurricane Katrina. Washington, DC: US House of Representatives; 2006.         

20. Qureshi K, Gershon RR, Sherman MF, Straub T, Gebbie E, McCollum M, et al. Health care workers' ability and willingness to report to duty during catastrophic disasters. J Urban Health 2005;82:378-88. PMID:16000654         

21. Shiao JS, Koh D, Lo LH, Lim MK, Guo YL. Factors predicting nurses' consideration of leaving their job during the SARS outbreak. Nurs Ethics 2007;14:5-17. PMID:17334166 doi:10.1177/0969733007071350         

22. Epstein JM, Goedecke DM, Yu F, Morris RJ, Wagener DK, Bobashev GV. Controlling pandemic flu: the value of international air travel restrictions. PLoS One 2007;2:e401.         

23. Gilbert N, Troitzsch K. Simulations for the social scientist, 2nd ed. Maidenhead: Open University Press; 2005.