EDITORIALS
The art of evaluating the impact of medical science
Liz Allen*
Wellcome Trust, 215 Euston Road, NW1 2BE, London, England
The medical research community has long considered research to be vital to the health and wealth of societies, supporting the view attributed to Mary Lasker, American philanthropist and ardent campaigner for medical research: "If you think research is expensive, try disease."However, in recent years this community has come under increasing pressure to demonstrate the value and extent of the impacts of its labour.1
In the United Kingdom, more than 80% of all research and development funds for science and engineering come from public and charitable sources, accounting for more than £5 billion per annum.2 With such big sums of public funding comes a responsibility to account for how these funds are spent and to ensure that they are spent in the most effective way. In times of recession, frugal governments must make critical choices in fund allocation and so their need for evidence of its impact and effectiveness moves higher up the political agenda. In addition, in the United Kingdom for example, there have been calls for the government-funded research councils to increase the economic impact of their investments3 the implication being that if they fail to do this, the government might choose to invest its funds in other areas that demonstrate greater impact on the economy and society.
In recent decades, the research evaluation industry has grown from relatively modest roots to big business, fuelled in part by the demands of exercises such as the United Kingdom Research Assessment Exercise and other national systems designed to help funders reward quality research and allocate future funding. And evaluation does have an important role to play supporting accountability, policy development and funding strategy. However, it is not easy to evaluate the impact of medical and scientific research in any wholly "scientific"way.
Our problem is what we do; there are no formulae for how to evaluate impact, and its definition varies between those within medical research and those outside this system, as Charlton notes.4 Among medical researchers, progress and success is assessed according to professional and objective criteria such as volume of publication output in prestigious journals and associated citations, securing research grants and running a large team. Outside the system, people want to know what you have discovered and its impact on morbidity and mortality. These are not measured in the same way nor within the same timeframe.
The road to discovery can be long and complex, involving many actors and serendipity. Recent analysis estimated that, for a selection of cardiovascular-related medical interventions, the time lag between original, key research and tangible impact on health was at least 17 years.5 The ways in which research is communicated and applied are many and varied and, perhaps most perplexing for those wanting a simple solution to measuring impact, you can't measure what would have happened had you not funded it in the first place. This puts evaluators in somewhat of a quandary; how can you demonstrate the impact of your funding if it is difficult to isolate the role it has played among a range of other influences? Just because it is difficult, however, does not mean we should not try.
Research evaluators are used to drawing on a range of proxy measures that are relatively easy to capture research publication metrics, intellectual property data, number of Doctorates of Philosophy attained and which tell us something about research progression. But, in our quest to discover impact, what we must avoid is the temptation to measure what we can count rather than measure what counts. To counter some of this quantification of impact, funding agencies are increasingly using qualitative approaches such as research narratives and case studies which allow us to recognize the multitude of actors involved in research and the timeframe to impact.
There are several practical things that evaluators and funders can do, starting with understanding the requirements of their stakeholders and informing them of the complexities of evaluating biomedical research and interpreting certain metrics. They can do a lot to track progress and capture key information as it evolves though funders should be wary about establishing complex systems to evaluate the impacts of research if there is no strategic imperative to do so.
They should also consider whether there are core outputs or impacts that could be tracked across a range of funding initiatives. Funders should look across their portfolios and consider relative performance in relation to organizational objectives. This could also be done across organizations to provide much-needed benchmarks and context. In addition, there may be a role for engaging subject experts more systematically in post-award evaluation.
Collaboration with other funders might also be helpful to harmonize reporting requirements and to track impacts at different stages of the pipeline, i.e. a basic research funder partnering with a funder of delivery of a medical intervention.
But, in all of this, we must be pragmatic and proportionate. The art of evaluating medical research and science is to adopt approaches that are true to the thing you are trying to evaluate. Complex systems that provide no insight into overall strategic direction, and seek to quantify what may essentially not be quantifiable, can quickly become expensive distractions. By bringing together a range of quantitative and qualitative evidence, we can better understand our impact and how we are making a difference. 
References
Available at: http://www.who.int/bulletin/volumes/88/1/09-074823/en/index.html
References
1. Medical research: assessing the benefits to society. London: UK Evaluation Forum; 2006. Available from: http://www.wellcome.ac.uk/stellent/groups/corporatesite/@msh_publishing_group/documents/web_document/wtx031817.pdf [accessed on 4 December 2009] .
2. Department of Business, Innovation and Skills. SET Statistics 2008. - Available from: http://www.dius.gov.uk/science/science_funding/set_stats [accessed on 4 December 2009]. London: HM Stationers; 2008.
3. Research Council Economic Impact Group. Increasing the economic impact of Research Councils. London: Department of Trade & Industry; 2006. Available from: http://www.berr.gov.uk/files/file32802.pdf [accessed on 4 December 2009] .
4. Charlton BG. Boom or bubble? Is medical research thriving or about to crash? Med Hypotheses 2006;66:1-2. PMID:16242249 doi:10.1016/j.mehy.2005.10.004
5. Health Economic Research Group, Office of Health Economics & RAND Europe. Medical research: What's it worth? Estimating the economic benefits from medical research in the UK. London: UK Evaluation Forum; 2008.
*Correspondence to Liz Allen (e-mail: l.allen@wellcome.ac.uk).