ORIGINAL ARTICLES AND REVIEWS
A quality assessment of systematic reviews on telerehabilitation: what does the evidence tell us?
Marco RoganteI; Dahlia KairyII; Claudia GiacomozziI; Mauro GrigioniI
IDipartimento di Tecnologie e Salute, Istituto Superiore di Sanità, Rome, Italy
IIÉcole de Réadaptation, Faculté de Médecine, Université de Montréal, Montréal, Canada
ABSTRACT
Aims. To evaluate the quality of systematic reviews on telerehabilitation.
Methods. The AMSTAR Assessment of Multiple Systematic Reviews checklist was used to appraise the evidence related to the systematic reviews.
Results. Among the 477 records initially identified, 10 systematic reviews matched the inclusion criteria. Fifty percent were of high quality; anyway the majority of them did not report the following aspects: i) analysis of the grey literature; ii) a list of the excluded studies and their characteristics; iii) the identification of possible source of bias and the assessment of its likehood; iv) an appropriate method to combine the findings of the included studies addressing the heterogeneity as well. From the main findings of the high-scored systematic reviews telerehabilitation resulted at least as effective as usual care: 1) in the short term treatment of mental health related to people affected by spinal cord injury; 2) in rural communities for treating patients affected by chronic conditions; 3) in treating common pathologies (mainly asthma) affecting children and adolescents. As for stroke, evidence is currently insufficient to reach conclusions about its effectiveness. As for costs, there is insufficient evidence to confirm that telerehabilitation is a cost-saving or cost-effective solution.
Conclusions. In the authors' knowledge this is the first attempt to evaluate the quality of systematic reviews on telerehabilitation. This work also identified the main findings related to the high-scored systematic reviews; the analysis confirms that there is a mounting evidence concerning the effectiveness of telerehabilitation, at least for some pathologies.
Key words: telerehabilitation, tele-rehabilitation, systematic review, quality assessment, AMSTAR
INTRODUCTION
With the general ageing of the population at least in industrialized countries and the limited resources devoted to public health, the development of new rehabilitation models and practices seems mandatory in order to cope with the change in population needs [1, 2]. Besides in-hospital rehabilitation interventions and management of patients in the acute phase of a disease (still a crucial aspect of the overall care delivery process) one of the emerging challenges of modern health systems is the management of chronic patients outside the healthcare structures [3, 4].
Telerehabilitation, one of the emerging fields of telemedicine, is defined as the set of tools, procedures and protocols to deliver the rehabilitation process remotely [5]. Generally speaking, the scientific community believes that telerehabilitation will play an important role in improving or, at least maintaining, the continuity of rehabilitation care and services as they are reorganized, as it is able to increase the efficiency of the services while containing costs [6]. Since rehabilitation may be defined as an active process by which those disabled by injury or disease realize their optimal physical, mental and social potential [7], theoretically speaking, telerehabilitation technology may be applied in all the fields where a rehabilitation process is in place [8].
Systematic reviews are used to critically synthesize and report the best available evidence related to a specific field to be investigated by means of a clearly formulated research question. They are purposely designed so that the data incorporated are obtained and analyzed in a structured, transparent and reproducible manner [9, 10]. Systematic reviews are useful tools for clinical and policy decision makers to design both clinical and organizational interventions which are supported by the best evidence coming from the scientific community. In addition systematic reviews may guide researchers to identify eventual gaps in the available scientific evidence, thus suggesting future areas of research [11]. For example, several systematic reviews have pointed out that, despite the growing number of telerehabilitation experiences worldwide, evidence of clinical and economic effectiveness is still lacking [1, 12] especially when used in the routine care [13]. In addition, these reviews have commented on the lack of methodological rigor and variability of approaches used in telerehabilitation studies.
Despite the limitations identified in the literature in the field of telerehabilitation, as this remains a relatively new area of research, with the growing number of telerehabilitation programs and with emerging databases providing potentially useful and reliable data on clinical outcomes [13], it is reasonable to expect that the number of scientific publications will rapidly increase [14], along with the number of systematic reviews on the topic. However, findings of secondary studies, such as reviews, strongly depend on the rigor of the methodology the researchers use, such as the study design, the research question, the inclusion and exclusion criteria used and the manner in which the data is extracted and analyzed. With the proliferation of systematic reviews health providers and decision makers are often faced with studies reporting contrasting results; thus concerns may rise when evaluating their validity, especially when evaluating their real applicability into clinical practice [15]. Therefore, as in other scientific sectors [16], there is a need to critically appraise systematic reviews so as to ensure that the main findings reported are related to the highest available levels of evidence [17]. To our knowledge, no formal appraisal of the scientific quality of systematic reviews on telerehabilitation has been conducted yet; hence, the aim of this paper is to fill this gap by systematically evaluating the quality of evidence related to published systematic reviews of telerehabilitation. The goal is to identify current strengths and weaknesses that impact on the general quality of the reviews, thus identifying relevant points to be taken into account for future research. A further expected outcome of the study will likely be the identification of a subset of reliable findings, related to the quality of telerehabilitation, coming from high quality systematic reviews.
METHODS
Search strategy and eligibility criteria
As for most of the recent systematic reviews on the topic [1, 13, 14, 18] the following databases were searched for scientific articles published in English starting at the earliest date available for each database and ending in June 2014: i) Medline; ii) CINAHL; iii) Cochrane Library; iv) Database of Abstracts of Reviews of Effects (DARE); v) PsychInfo.
For this study, the authors adopted the methodology used in a former review, which described the state of the art of telerehabilitation after the first ten years of life [1], using the keywords telerehabilitation and tele-rehabilitation as search words in all fields of the queried article. Among the set of retrieved articles, all those considered to be systematic reviews were retained. A systematic review was included if it met the following criteria [19]: 1) a set of clearly formulated research objectives or questions are defined, as well as clear and defined eligibility criteria for the selection of relevant studies; 2) a well defined and reproducible methodology is described and applied; 3) a systematic search strategy is defined and implemented; 4) a systematic presentation, analysis, and synthesis is presented concerning the main information extrapolated by the analyzed articles.
The main steps related to the search phase are reported in Figure 1 using the PRISMA flow diagram [20]; after the application of the selected keywords, the entire set of records was analyzed to identify eventual duplication of articles retrieved in different sources; titles and abstract were used to exclude articles which were not systematic reviews. The remaining articles were then assessed in full text for eligibility so as to identify all those systematic reviews matching the inclusion criteria.
Data analysis
Analysis focused on the quality assessment of each systematic review and the narrative reporting of each review main findings, described below.
Quality assessment: AMSTAR checklist
As mentioned previously, the proliferation of systematic review in the clinical field renders it challenging for policy decision-makers to use reviews in making clinical and policy plans as it is difficult to distinguish good from poor-quality reviews; the AMSTAR Assessment of Multiple Systematic Reviews checklist is an easy-to-use tool purposely developed to address with this need [21]. Even if it was originally designed to be used with systematic reviews of randomized clinical trials only, there are no specific restrictions and limitations to apply the tool to a wide variety of systematic reviews, i.e. including controlled trials [22, 23]. The validated version of this tool is based on eleven items to be assessed [22] and is considered suitable for the purposes of the present work [24]. Two reviewers (MR, CG) independently rated study quality using the 11-item AMSTAR checklist; where differences were noted, these were resolved by discussion between the two reviewers, and where agreement could not be reached, the third reviewer solved the discrepancies (MG). Finally the quality assessment of each review was computed by globally summing positive rates, with higher scores indicating a higher level of methodological quality.
Even though there is not a unique and standardized value according to which a review is considered to be of good quality, an AMSTAR score equal or higher than eight is considered to be related to a high quality review [16]. The former criteria have been also employed by the Canadian Agency for Drugs and Technologies in Health with an initiative aimed to help inform the choice and use of practical, evidence-based interventions [25].
Finally the score of each item of the checklist has been individually analyzed (AMSTAR matrix analyzed by columns) in order to identify which items future research should focus on in order to improve the quality of reviews.
Analysis of findings
Data extracted from the included reviews were entered in a spreadsheet under the following headings: i) quantification of the quality of each review; ii) pathology addressed in the review; iii) number of references; iv) number of included studies; v) number of randomized controlled trials (when explicitly retrievable from the review); vi) main study findings; vii) period covered by each review; viii) year of publication. The focus was then centered on all those reviews rated as high quality and their main study findings have been narratively summarized and reported.
RESULTS
The research initially identified 477 records, resulting in 338 different articles to be screened after the elimination of duplicate items which were retrieved in more than one database; of this, 284 articles were excluded since, on the basis of analysis of title and abstract, they could not be classified as systematic reviews, and two were excluded because they had been written in German. Among the remaining 54 articles, the full-text analysis allowed the identification of only 10 different systematic reviews [1, 12-14, 18, 26-30], since the remaining 44 did not match the eligibility criteria reported in the Methods Section.
Quality assessment
The ten systematic reviews resulted in a median AMSTAR score of 7 (interquartile range 4.5-8). Table 1 reports on the AMSATR score for each included review, detailed for each item of the tool, and the score for each single item of the AMSTAR checklist. Fifty percent of the analyzed articles were of high quality (AMSTAR score > 8; Table 1).
When individually analyzing each item of the AMSTAR checklist (analysis by columns), it can be noted that the entire set of the retrieved reviews report a clear and well defined a priori design (Q-1, 10/10), while the majority of them (equal or higher than 80%) met AMSTAR criterion relating to the following aspects: i) using independent reviewers for selecting studies and extracting data (Q-2); ii) performing a comprehensive literature search (Q-3); iii) providing a list of the includes studies with detailed information on their characteristics (Q-6); iv) assessing and documenting the study quality (Q-7); v) the scientific quality of the included studies was used appropriately in formulating conclusions (Q-8). However, fewer studies (equal or less than 50%) met the AMSTAR criteria for the following aspects: i) using the publication status as inclusion criteria, investigating the scientific and the grey literature as well (Q-4); ii) using appropriate methods to combine the findings of the included studies addressing the heterogeneity as well (Q-9), iii) assessing the likelihood of the publication bias (Q-10). Worthy of note is that just one among the selected articles reported a list of the excluded studies (Q-5). As for question 11 of the AMSTAR checklist, investigating the presence in the article of a clause concerning the absence of any conflict of interest, 5 out of ten reviews explicitly reported the declaration; for the remaining set of investigated reviews, it should be noted that the journal publication policies require a declaration of conflicting interests thus, even if it has not been formally computed within the AMSTAR checklist, authors are asked to provide such an information during the publication phase.
Main findings of the studies
The main findings of the included studies are reported in Table 2 where the extracted parameters are summarized. Worthy of note, three out of ten reviews deal with specific pathologies, namely stroke [14, 28] and mental health affecting people after spinal cord injury [26]. The remaining selected reviews focused on multiple pathologies.
The following conclusions were drawn in the five highest-quality systematic reviews:
- with regards to mental health related to people affected by spinal cord injury, clinical results were promising, reporting that telecounselling application was effective especially in the short term; thus telecounselling may facilitate routine psychological follow-up of individuals with a newly acquired injury, who often experience increased apprehension and distress during the transition from primary rehabilitation [26]. However the study highlights the lack of evidence concerning both the long-term effects and the related cost analyses;
- videoconferencing used with patients affected by chronic conditions resulted in similar clinical outcomes as compared to when on-site traditional treatment were used; considering the high satisfaction levels for patients, the authors conclude that telerehabilitation is feasible for people with chronic conditions in rural communities [27];
- with regards to the costs associated to telerehabilitation there is insufficient evidence to confirm that telerehabilitation is a cost-saving or cost-effective solution, even if it is recognized that it can lead to similar clinical outcomes compared to traditional rehabilitation programmes with an overall high acceptance from both patients and therapists [12];
- the main pathology for which telerehabilitation technology was used in children and adolescents was asthma. Most of the included studies have demonstrated that telerehabilitation is at least as effective as traditional treatment; thus telerehabilitation is considered a possible, effective and secure strategy for the treatment of common diseases in the considered population [29];
- in the stroke patient population, despite the theoretical advantages of telerehabilitation, evidence is currently insufficient to reach conclusions about its effectiveness [14].
DISCUSSION
To the authors' knowledge, this is the first attempt to quantitatively assess the quality of systematic reviews dealing with telerehabilitation. The AMSTAR checklist was in fact a versatile and easy-to-use tool suitable for the purposes of the present work. According to the proposed methodology, of the ten systematic reviews which were retrieved and matched the inclusion criteria, 50% resulted to be of high quality (AMSTAR score > 8). Results of the present article highlight the gaps in the quality of the systematic reviews on telerehabilitation, as assessed using the AMSTAR tool:
a) lack of analysis of the grey literature on the topic (7 out of 10 studies did not address this). In addition to not including the grey literature, failure to identify unpublished trials is considered to potentially affect the results of a systematic review [19];
b) lack of a list of excluded studies and their characteristics (9/10). Readers, in fact, may be interested in understanding why some studies have been excluded; thus a brief rationale explaining reasons for exclusion may help;
c) lack of identification of possible sources of bias and the assessment of their likelihood (6/10);
d) most of the retrieved reviews did not use appropriate methods to combine the findings of the included studies or did not address the heterogeneity of the studies (8/10).
As regard with the study findings, three out of ten reviews dealt with specific pathologies, namely stroke [14, 28] and mental health affecting people after spinal cord injury [26]; this is not surprising, since telerehabilitation clinical applications are quite wide, i.e. including treatment of mental health conditions, cardiac or neurological rehabilitation.
In addition, the findings from our study highlight patient population in which there remains a lack of evidence supporting the use of telerehabilitation. A Cochrane systematic review focused on stroke, which was scored as the highest-rated systematic review in our study, pointed out that evidence related to telerehabilitation is currently insufficient to draw conclusions about its effectiveness for the considered pathology [14]. According to the remaining highest-rated reviews, telerehabilitation is considered to be clinically promising since it resulted at least as effective as the traditional treatment for, at least, the following scenarios: the short term treatment of spinal cord injury population dealing with mental health [26], common pathologies (mainly asthma) affecting adolescents [29], and the management of chronic patients in rural settings [27]. Furthermore, studies reported evidence on effectiveness for teledermatology, teleradiology, telemental health fields [12], although, as with stroke, there was a lack of evidence for the long-term management patients dealing with mental health disorders after spinal cord injury.
A further note-worthy aspect is that most of the selected reviews, including those of higher and lower quality, reported high satisfaction and acceptability of telerehabilitation technology for both patients and health operators (Table 2).
Finally there is insufficient evidence to state that telerehabilitation is a cost-saving or cost-effective approach [12], despite the scientific community addresses its potentialities to be an alternative service delivery methods given current health care system constraints [6].
Limitations of the study
Only a total of 10 systematic reviews were identified which matched the inclusion criteria. Several factors may have contributed to this, such as the search strategy and the fact that telerehabilitation is quite a new field. Regarding the search strategy, using a combination of terms aimed at retrieving articles dealing with the rehabilitation process at distance may lead to the identification of a wider set of articles; however, in a former review [1] authors have demonstrated the feasibility of the proposed search strategy which is based on using telerehabilitation and tele-rehabilitation keywords only and the capability to reduce the extent of non relevant articles. Regarding the field of telerehabilitation, it should be considered that it is a quite new field of application, such that we anticipate that with the increasing number of well-structured primary studies on the topic [14] there will be an increase in systematic reviews synthesizing their main findings.
The selected databases for searching the literature were the relevant and commonly searched databases identified in previous studies. However, it is possible that other databases, i.e. all those dealing with Health Technology Assessment (HTA), may be a relevant source of information. HTA, in fact, investigates a given technology in terms of clinical effectiveness, efficiency (cost-effectiveness), safety, related main organizational requirements, as well as ethical, societal and quality perception aspects of its use. Besides the clinical aspects, in fact, societal, economical and organizational issues are key factors in the adoption and uptake of a specific technology [32].
In addition, not including the grey literature and excluding studies not in English could lead to a publication bias that should be taken into account [9].
CONCLUSIONS
The analysis of the findings of the high-scored systematic reviews confirm that there is mounting evidence concerning the effectiveness of telerehabilitation, at least for some pathologies. There is also some evidence with respect to users' acceptance and satisfaction, and overall feasibility related to the discipline. Nevertheless, evidence remains insufficient with respect to the cost-savings and cost-effectiveness of telerehabilitation.
By using an AMSTAR-based methodology, the present study helped identify areas which future systematic reviews may take into account in order to improve their quality.
Conflict of interest statement
There are no potential conflicts of interest or any financial or personal relationships with other people or organizations that could inappropriately bias conduct and findings of this study.
REFERENCES
1. Rogante M, Grigioni M, Cordella D, Giacomozzi C. Ten years of telerehabilitation: a literature overview of technologies and clinical applications. NeuroRehabilitation 2010;27(4):287-304.
2. Zampolini M, Todeschini E, Bernabeu Guitart M, Hermens H, Ilsbroukx S, Macellari V, Giacomozzi C. Tele-rehabilitation: present and future. Ann Ist Sup Sanità 2008;44(2):125-34.
3. Wootton R. Twenty years of telemedicine in chronic disease management - an evidence synthesis. J Telemed Telecare 2012;18(4):211-20.
4. Druss BG, Marcus SC, Olfson M, Tanielian T, Elinson L, Pincus HA. Comparing the national economic burden of five chronic conditions. Health Aff (Millwood) 2001;20(6):233-41.
5. Winters JM. Telerehabilitation research: emerging opportunities. Annu Rev Biomed Eng 2002;4:287-320.
6. Theodoros D, Russell T. Telerehabilitation: current perspectives. Stud Health Technol Inform 2008;131:191-209.
7. World Health Organization. World Report on Disability. International Classification of Functioning, Disability and Health. Geneva: WHO; 2001. Available from: http://whqlibdoc.who.int/publications/2011/9789240685215_eng.pdf
8. Russell TG. Physical rehabilitation using telemedicine. J Telemed Telecare 2007;13(5):217-20.
9. Taylor P, Hussain JA, Gadoud A. How to appraise a systematic review. Br J Hosp Med (London) 2013;74(6):331-4.
10. Greenhalgh T. Papers that summarise other papers (systematic reviews and meta-analyses). BMJ 1997;315(7109):672-5.
11. Egger M, Smith D, O'Rourke K. Rationale, potentials, and promise of systematic reviews. In: Egger M, Smith D, Altman D (Ed.). Systematic reviews in healthcare Meta analysis in context. London: BMJ Books; 2001. pp 3-19.
12. Kairy D, Lehoux P, Vincent C, Visintin M. A systematic review of clinical outcomes, clinical process, healthcare utilization and costs associated with telerehabilitation. Disability and rehabilitation 2009;31(6):427-47.
13. Hailey D, Roine R, Ohinmaa A, Dennett L. Evidence of benefit from telerehabilitation in routine care: a systematic review. J Telemed Telecare 2011;17(6):281-7.
14. Laver KE, Schoene D, Crotty M, George S, Lannin NA, Sherrington C. Telerehabilitation services for stroke. Cochrane Database Syst Rev 2013;16;12.
15. Abalos E, Carroli G, Mackey ME, Bergel E. Critical appraisal of systematic reviews. The WHO Reproductive Health Library, No 4. Geneva: WHO; 2001. (WHO/RHR/01.6).
16. Conway A, Inglis SC, Chang AM, Horton-Breshears M, Cleland JG, Clark RA. Not all systematic reviews are systematic: a meta-review of the quality of systematic reviews for non-invasive remote monitoring in heart failure. J Telemed Telecare. 2013;19(6):326-37.
17. Smith V, Devane D, Begley CM, Clarke M. Methodology in conducting a systematic review of systematic reviews of healthcare interventions. BMC Med Res Methodol 2011;11(1):15.
18. Hailey D, Roine R, Ohinmaa A, Dennett L. The status of telerehabilitation in neurological applications. J Telemed Telecare 2013;19(6):307-10.
19. Higgins JPT, Green S (Ed.). Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [updated March 2011]. The Cochrane Collaboration, 2011. Available from: www.cochrane-handbook.org.
20. Moher D, Liberati A, Tetzlaff J, Altman DG. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA Statement. Open Med 2009;3(3):e123-30.
21. Shea BJ, Grimshaw JM, Wells GA, Boers M, Andersson N, Hamel C, Porter AC, Tugwell P, Moher D, Bouter LM. Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews. BMC Med Res Methodol 2007;7:10.
22. Shea BJ, Hamel C, Wells GA, Bouter LM, Kristjansson E, Grimshaw J, Henry DA, Boers M. AMSTAR is a reliable and valid measurement tool to assess the methodological quality of systematic reviews. J Clin Epidemiol 2009;62(10):1013-20.
23. Worswick J, Wayne SC, Bennett R, Fiander M, Mayhew A, Weir MC, Sullivan KJ, Grimshaw JM. Improving quality of care for persons with diabetes: an overview of systematic reviews what does the evidence tell us? Syst Rev 2013;2:26.
24. Popovich I, Windsor B, Jordan V, Showell M, Shea B, Farquhar CM. Methodological quality of systematic reviews in subfertility: a comparison of two different approaches. PLoS One 2012;7(12):e50403.
25. Grimshaw J, Hill S, Lowe D, Kaufman C, Mayhew A, Fiander M, Taylor M, Worswick J, Wu J. Canadian Agency for Drugs and Technologies in Health, Methods for Development. Available from: www.cadth.ca/en/resources/rx-for-change/methods-for-development.
26. Dorstyn D, Mathias J, Denson L. Applications of telecounselling in spinal cord injury rehabilitation: a systematic review with effect sizes. Clin Rehabil 2013;27(12):1072-83.
27. Steel K, Cox D, Garry H. Therapeutic videoconferencing interventions for the treatment of long-term conditions. J Telemed Telecare. 2011;17(3):109-17.
28. Johansson T, Wild C. Telerehabilitation in stroke care a systematic review. J Telemed Telecare 2011;17(1):1-6.
29. dos Santos MT, Moura SC, Gomes LM, Lima AH, Moreira RS, Silva CD, Guimarães EM. Telehealth application on the rehabilitation of children and adolescents. Rev Paul Pediatr 2014;32(1):136-43.
30. Steins D, Dawes H, Esser P, Collett J. Wearable accelerometry-based technology capable of assessing functional activities in neurological populations in community settings: a systematic review. J Neuroeng Rehabil 2014;11:36.
31. Sorenson C, Drummond M, Kanavos P. Ensuring value for money in Health Care. The role of health technology assessment in the European Union. 2008. Available from: www.euro.who.int/__data/assets/pdf_file/0011/98291/E91271.pdf.
Address for correspondence:
Marco Rogante,
Dipartimento di Tecnologie e Salute, Istituto Superiore di Sanità,
Viale Regina Elena 299, 00161 Rome, Italy.
E-mail: marco.rogante@iss.it.
Received on 17 September 2014.
Accepted on 28 January 2015.