COMMENTARY
Research into cost and value in medical education: can we make findings more generalisable?
Kieran Walsh
BMJ Learning, BMJ Publishing Group, London, United Kingdom
ABSTRACT
In recent years a growing number of papers have started to appear in the literature on the subject of cost and value in medical education. However many of the articles describe tactical projects within specific areas. As a result the generalisability of such articles is often questionable. Lack of generalisability will mean that progress in research and practice in this field will be slow. Generalisability of cost-value analyses in any discipline is not always straightforward. However reports on cost and value in medical education should ideally ensure absolute clarity with regard to study centres, enrolment of learners, alternative options, perspectives of stakeholders, resources used and their costs, instruments used, variability, and any problems with the analyses (such as incomplete data).
Key words: cost; medical education
Medical education is expensive [1]. It is also important. These two self-evident statements mean that if we could ensure that we achieve maximum value from our spend in medical education, then clear benefits would accrue to payers (be they individuals or institutions) [2]. This might mean cost savings or better value for a given spend. Cost and value in medical education has traditionally been an under-explored area, but in recent years a growing number of research articles and papers have started to appear in the literature [3, 4]. However many of the articles describe tactical projects within specific areas. As a result the generalisability of such articles is often questionable. Lack of generalisability will mean that progress in research and practice in this field will be slow. Generalisability of cost-value analyses in any discipline is not always straightforward. However it may be that medical education research can learn from medical research - certainly cost-value analyses in medicine have been around for longer and are therefore much more mature. So what makes for generalisability in research in cost analyses in general medical research?
Drummond et al. have made a number of recommendations to improve the generalisability of economic evaluations when conducted alongside randomised trials [5]. They recommend that researchers ensure absolute clarity in their reports with regard to study centres, enrolment of patients, alternative treatments, perspectives of stakeholders, resources used and their costs, instruments used, variability, and any problems with the analyses (such as incomplete data) [5]. All these elements "work" in the context of cost analysis in medical research - however the question remains of the practicality of applying them to medical education economic analyses. Let's take them one by one.
With regard to study centres, most medical education research is carried out in medical schools or postgraduate medical education institutions. Such institutions can be easily described and the results should then be more easily understood and generalised. If research is carried out in more than one institution, then all participating institutions should be described. Most medical education research is carried out in single institutions - this makes understanding the context simpler, but misses out on both the complexity and richness of multi-institutional research. That aside, medical education economic analysts could easily be explicit about the institutions where research is being carried out, and this should facilitate generalisability.
Medical researchers concern themselves with enrolment of patients, but medical education researchers think about enrolment of learners. Ideally learners will be representative of their peers outside of the research -this will enable more straightforward generalisation. If however the research inclusion criteria exclude large numbers of learners, this will be a threat to generalisation. Regardless of whether learners were excluded or not, it is most important that the reporting is clear and explicit about who was included and excluded and why.
If the medical education economic analysis compared the cost and value of a new innovation compared to an alternative form of education, then the reporting should make clear exactly what the alternative was. Vague descriptions have no place in rigorous cost and value analyses. All too often the medical education literature refers to traditional medical education, but what is traditional will vary between time and geographical setting. What was innovative thirty years ago may be traditional now. Innovation in one territory may be standard practice in another. Problem-based learning (PBL) is a classic case in point. Can we really say that it is still innovative? Maybe in a diminishing number of territories. This is not to praise or damn or damn with faint praise PBL but rather a statement about its dissemination. David Prideaux has captured this well: "problem based learning, an educational intervention characterised by small group and self directed learning, is one of medical education's more recent success stories, at least in terms of its ubiquity" [6].
Relevant stakeholder perspective is also as important in medical education research as it is in medical research. The cost and value of medical education interventions will appear differently when examined from diff perspectives. For example the perspective of a university department may be different to that of a government which in turn may be different to that of a medical student. For example the government might want to save money, the university department might want to spend it and the student might just want a grant or a low interest loan. It is best to be clear from which or whose perspective a cost analysis is being portrayed this will help the reader gain a deeper understanding and perhaps to contextualise the results in the perspectives of stakeholders that are important in their own territory.
When medical researchers describe resources used and their costs, they are typically describing drugs or procedures and the costs or tariffs with which they are associated. Economists advise separate descriptions of resources and costs. The parallels in medical education research might be tutor time or simulation equipment and their associated costs. In cost-value medical education research it is best to separate these also. The reason is that customary practice may be different in the country of the reader to that of the researcher. Prices may also be different. However if the author is clear about the quantity of resources used and their cost, the reader will be able to extrapolate this data to their own country - with its particular customary practices and prices.
Just as medical researchers use external, validated and international instruments to report health outcomes, so medical educational researchers should use external, validated and international instruments to report educational outcomes. The reader of the research will then be able to quickly evaluate whether the educational outcomes reported are relevant and applicable to their institution's circumstances.
Variability in cost and value analysis in medical research generally means variability by location in the context of multicentre trials. This is less likely to be a problem in medical education cost and value analyses - most of which are single centre.
Problems with the data can be manifold. There may be uneven baseline demographics among learners at the start, over-emphasis on subgroup analyses in the middle and/or incomplete follow up at the end.
Economic evaluations of in medical education may not always be seamlessly transferable from one context to another. However if researchers consider these elements when carrying out their evaluations and reporting on them, readers will be more likely to be able to tell if the findings are generalisable to their own setting. If the findings are not exactly generalisable, they will likely know how the findings might need to be contextualised to make them fit with their requirements and their setting.
REFERENCES
1. Walsh K. (Ed.). Cost effectiveness in medical education. Radcliffe: Abingdon; 2010.
2. Walsh K, Levin H, Jaye P, Gazzard J. Cost analyses approaches in medical education: there are no simple solutions. Med Educ 2013;47(10):962-8. DOI: 10.1111/medu.12214.
3. Schott NJ, Emerick TD, Metro DG, Sakai T. The cost of resident scholarly activity and its effect on resident clinical experience. Anesth Analg 2013;117(5):1211-6. DOI: 10.1213/ANE.0b013e3182a44d5d.
4. Phillips JP. We must make the cost of medical education reasonable for everyone. Acad Med 2013;88(10):1404. DOI: 10.1097/ACM.0b013e3182a2b013
5. Drummond M, Manca A, Sculpher M. Increasing the generalizability of economic evaluations: recommendations for the design, analysis, and reporting of studies. Int J Technol Assess Health Care 2005;21(2):165-71. DOI: 10.1017.S0266462305050221
6. Prideaux D. Researching the outcomes of educational interventions: a matter of design: RCTs have important limitations in evaluating educational interventions. BMJ 2002;324:126. DOI: 10.1136/bmj.324.7330.126
Address for correspondence:
Kieran Walsh
BMA House, Tavistock Square
London WC1H 9JR
United Kingdom
E-mail: kmwalsh@bmjgroup.com
Accepted on 10 January 2014.