DIGITAL TELE-ECHOCARDIOGRAPHY TODAY: SUCCESSES AND FAILURES
A web based health technology assessment in tele-echocardiography: the experience within an Italian project
Health technology assessment in tele-ecocardiografia basata su web: l'esperienza in un progetto italiano
Daniele GiansantiI; Sandra MorelliI; Giovanni MaccioniI; Lorenzo GuerrieroII; Remo BediniIII; Gennaro PepeIII; Cesare ColomboI; Gabriella BorghiI; Velio MacellariI
IDipartimento di Tecnologie e Salute, Istituto Superiore di Sanità, Rome, Italy
IIIstituto di Fisiologia Clinica, CNR, Pisa, Italy
IIICEFRIEL, Politecnico di Milano, Milan, Italy
SUMMARY
Due to major advances in the information technology, telemedicine applications are ready for a widespread use. Nonetheless, to allow their diffusion in National Health care Systems (NHcSs) specific methodologies of health technology assessment (HTA) should be used to assess the standardization, the overall quality, the interoperability, the addressing to legal, economic and cost benefit aspects. One of the limits to the diffusion of the digital tele-echocardiography (T-E) applications in the NHcS lacking of a specific methodology for the HTA. In the present study, a solution offering a structured HTA of T-E products was designed. The methodology assured also the definition of standardized quality levels for the application. The first level represents the minimum level of acceptance; the other levels are accessory levels useful for a more accurate assessment of the product. The methodology showed to be useful to rationalize the process of standardization and has received a high degree of acceptance by the subjects involved in the study.
Key words: tele-echocardiography, image quality, telemedicine.
RIASSUNTO
Grazie ai grandi progressi nell'information technology le applicazioni di telemedicina sono mature per un uso diffuso. Tuttavia per permettere la loro introduzione nel sistema sanitario nazionale devono essere utilizzate specifiche metodologie di health technology assessment (HTA) per valutare il grado di standardizzazione, la qualità totale, l'interoperabilità, il rispetto dei requisiti legali ed economici e il rapporto costo-beneficio. Con riferimento alla tele-ecocardiografia digitale uno dei limiti è la mancanza di una specifica metodologia di HTA. Nel presente studio, è stata proposta una soluzione che offre un HTA strutturato di prodotti di tele-ecocardiografia (T-E) digitale. La metodologia ha assicurato anche la definizione di livelli standardizzati di qualità per l'applicazione. Il primo livello rappresenta il livello minimo di accettazione; gli altri livelli riguardano aspetti accessori e sono utili per una più accurata valutazione del prodotto. La metodologia si è mostrata di utilità per razionalizzare il processo di standardizzazione ed ha ricevuto un elevato grado di accettazione dei soggetti coinvolti.
Parole chiave: tele-ecocardiografia, qualità delle immagini, telemedicina.
INTRODUCTION
The design and construction of a telemedicine system involves complex technical, legal and regulatory, economic, clinical, and sociological considerations [1, 2]. Thus, an appropriate evaluation of telemedicine applications must take into account not only the effectiveness of the tele-health system itself, but also all of the technical, clinical, legal, and regulatory aspects related to its use. Many studies have evaluated individual aspects of telemedicine (TM), such as clinical accuracy [3], clinical effectiveness [4], and patient satisfaction [5]. The lack of a single study integrating all the above-mentioned issues led these authors to the proposal of a health technology assessment (HTA) system for a wide range investigation of TM [6, 7]. In the specifics, the cited study had showed that the design of a HTA in TM is quite difficult; in fact a TM system is complex and heterogeneous, with components from bioengineering, medical physics, and information technology (software, hardware, the Net). A first proposal of the HTA has been tested in the Italian project e-RMETE (2001-2004: "Regioni per la Medicina Telematica" - Italian Regions for Telematics in Medicine) funded by the Italian Ministry of Health. The procedures are shown in details in the technical report published by the ISS [8] and available in the electronic version at the URL (ermete.ifc.cnr. it). In the cited report it has been also enlightened that two are the core aspects that should be considered in the design of the HTA: the first is how to accumulate information about the system, the second is the methodology to be used for the assessment. The HTA we had designed was thus arranged into two main consecutive phases. The first phase was designed to gather structured information and was based on an interactive questionnaire. The second phase was designed for the assessment and was based on an interactive checklist.
Problem definition and aim of the paper
The step successive to the design of the HTA methodologies defined in [6-8] is to embed them in a web based solution and to tune the procedures for tele-echocardiography (T-E). Up to now web based methodologies have been succesfully investigated in e-learning applications also critical for the amount of the imaging data to be exchanged as in health physics [9-11]. However no one has investigated the use of web methodologies for the HTA in TM and thus in T-E. A web-based solution could be useful and effective providing in fact a structured data exchange among all the actors involved in the process.
MATERIALS AND METHODS
Information and technological features of the tele-assessment system
The HTA system was based on web architectures for information exchange. Two different web portals were used:
1. the first one available at the URL address (www.telemed.iss.it) is ftp-like and has been thought for the exchange of the interactive files relevant to the procedures of HTA, described in Word format in [8] maintained by ISS personnel. The WebDav protocol (web-based distributed authoring and versioning, USA, www.webdav.org) was used to create secure connections. This protocol is very similar to the file-transfer-protocol (ftp) however it does not suffer of the limits in information security;
2. the second one available at the URL address (ermete.ifc.cnr.it) is for the introduction of the T-E application in a catalogue after the positive HTA evaluation. This web is maintened by the Istituto di Fisiologia Clinica, CNR, Pisa, (IFC-CNR). A brief description has been reported in [8].
The structured data exchange
The interaction among the participants was structured into two phases:
Phase 1
The first phase was designed to gather information by means of an interactive questionnaire I-Q.
Table 1 details the main sections of the I-Q, with the persons responsible of the on-line compilation. In brief, the first section concerned the system itself, including the identities of the producer and project manager, a short description of the system, and its destination and intended use. Information gathered in the second section pertained to the topology of the entire system, the type of architecture (client-server, peer to peer, etc.), the policy adopted to assure computer security, and the software packages installed. In the third section, detailed information about the biomedical instruments used in the telemedicine process (in this case, the echo-cardiograph) and the I/O peripherals connected to the nodes of the system (biomedical or non-biomedical) was requested. The fourth section dealt with the network requirements of the system (eg., the use of four ISDN channels, ADSL, leased data line, satellite channels, etc.). This section should also discuss the possibility of using an alternative and always ready communication channel in the case of failures (cold and warm machine method). The fifth section examined available user documentation, installation documentation, availability, and features (e.g. , formatting of the documentation in XML or PDF format, as well as access to the demo program as an Internet download, etc.). As part of the I-Q; the documentation about the system must be submitted in two electronic documentation formats, XML and PDF.
Phase 2
The second phase was designed specifically for the HTA of the T-E system that allows for an evaluation score with different levels much more useful in a HTA than a two level score: passed or not passed.
This phase was based on an interactive checklist (I-C). According to the proposal described in details in 20 it has been structured into 3 sections including different requirements:
section 1: product requirements;
section 2: requirements regarding product design, manufacturing and testing;
section 3: evaluation of legal, economic, social, cost benefit aspects.
The general record of the I-C is described by Table 2. The requirements in the checklist reflect three levels of importance the first is the minimum level of acceptance (MLA) the other are accessory levels (ACC-1; ACC-2) to allow an assessment on more layers. Level MLA is associated with basic, mandatory requirements such as those regulated by national laws, such as those about the data integrity, privacy, maintenance of clinical information, backup requirements. Level ACC-1 concerns the most common voluntary standards such as the DICOM (Standard NEMA, 2005) or others such as, as not exhaustive example, the ones fixed by the ANSI, i.e. the not mandatory ones for national/governative regulations but often de-facto required at the public selections for acquirement. Level ACC-2 are further voluntary standards, general best practice or voluntary quality rules which can be considerd a further enrichment and are never required at the public selections for aquirements.
For this investigation as for others described in [8], the evaluation team [6, 7] decided to focus on some specific sections of the checklist. In particular, in consideration to the high specificity of the application (T-E) the quality of images specific procedures have been planned. In fact, one of the main problems in T-E is the need to have at the reception site high quality images with the same diagnostic potentialities of the transmitted ones. It has to be considered here that data compression (such as Mpeg) and consequent information loss may cause erroneous medical diagnosis [3, 4]. In particular it has to be considered that only a single image in T-E is not only affected by the most investigated quantitative information parameters as assessed on static, but also the "dynamics" of the frame flow. For these reasons, after detailed studies in the fields of T-E [12, 13] particular relevance was given to the tests of image quality comprehending the most commonly used methodologies for testing medical images [14]:
1. request of the simulation of the expected image quality;
2. measurement of the degradation of the transmitted images using the most commonly used parameters such as the resolution, the contrast ratio, the wiener spectrum;
3. the use of specific phantoms and the selection of critical cases for testing;
4. the use of tests such as the receiver operator characteristic (ROC) for the assessment of the diagnostic sensitivity and specificity on the transmitted images;
5. the use of automatic tools such as the one based on the virtual quality metrics (VQM) [12-19].
In particular some of the authors have developed two specific protocols for the assessment of the diagnostic accuracy. The first described in [12] is heterogeneous and comprehends the above listed subjective and objective methods [1-4]. The second described in [13] is based on the VQM. Both the two methods have been inserted in the checklist as suggested methods.
RESULTS
The web solution for the HTA was tested on a digital T-E system (EHCOCARDIOWEB); (www.dgsan.lombardia.it/ricerca_progetti/ministeriali/ermete.htm).
Outcome of phase 1
The I-Qs yielded a complete picture of the status of the T-E applications without loss of information about all the basic elemens of the heterogeneus system. The subjects involved in the study found that all sections of the I-Q were useful in their analyses. In particular, the fifth section provided a structure for organizing the documentation in preparation for the HTA by the I-C. This section, with surprise, also revealed that Acrobat PDF and not the standard Extensible Markup Language (XML, www.w3.org/XML) was the principal format used to organize the documentation.
Outcome of phase 2
Tables 3 and 4 show two sections of a compiled I-C, in which an electronic report was also attached to the answers. In particular Table 3 shows a segment of I-C, designed for the assessment of the standards adopted. The compatibility with DICOM was not considered essential, thus of level ACC-1. In fact many TE systems currently work with MPEG (www.mpeg.og) without passages to DICOM (Standard NEMA, 2005), (www.nema.org/stds/ps3set.cfm). Similar considerations should be done for the requirement about HL7 compatibility. Table 4 shows that the committee had assigned level MLA to the tests designed to assess T-E under critical conditions. Figure 1 shows the number (in percent) of met requirements for each one of the three sections. The number of requirements of level MLA (obligatory to allow a successful HTA) is obviously equal to 100%. This elucidates the possibility to use the HTA to compare different T-E applications. The scientific committee reckoned "essential" some aspects related to the backup and thus assigned them a MLA. The committee decided for the test designed to assess T-E under critical conditions, the question level was MLA. The suggested tests have been also successfully proposed to the scientific community towards specific publications [12, 13]. In spite of the tool VQM has not been used in this project, the committee finds that could be useful for future improvements of the procedure itself.
User satisfaction
The user satisfaction was also tested by means of a dedicated questionnaire submitted to five experts of each TM products (one expert in networking, one in medical physics, one in information technology, one physician, and one surgeon). The approach was similar to that used by Hutten et al. [15] but was specifically optimized for this investigation. The evaluation scale ranged from 1 (excellent) to 5 (insufficient), and the outcome was encouraging. Table 5 lists the features covered by the questionnaire and the scores.
DISCUSSION AND CONCLUSION
In the present study, an on-line web-based system for HTA was designed and validated in digital T-E. The system was based on two interactive core tools. The first one is, the I-Q, allowed no loss of information about the software, hardware, or net components of the complex and heterogeneous TM application. The second one is, the interactive checklist, allowed the HTA of a T-E application. In particular the checklist allow to compare two products which have passed a minimum level of acceptance, (i.e. which have in all the sections the 100% of met MLA) by comparing the number of met accessory levels of type Acc-1 and Acc-2 (Figure 1). The application of this structured methodology, is useful in HTA for example to find the product best of the bunch, (by means of a comparison of the type and number of requirements met as it has been enlucidated by figure a core aspect of the HTA.
How the study contributes to the investigation in digital T-E
In [14] we have considered the limits of digital T-E, The HTA system could be a valid answer to the global limit. However it could be also a valid answer for the other ones, in fact, it includes procedures defined by experts permitting to check that a T-E application full-fits to the essential requirements focused to the solutions of these limits. For example, with reference to the second limit indicated in the table, the scientific committee has decided to introduce the specific requirement (2.11) in the checklist (see Table 4).
Future work
The ISS, in light of its institutional role, conducted this study with the goal of promoting the planned and regulated assessment of the T-E systems. The web-based tool could thus represent a reference guide, and, although currently not under consideration, could be used by National Health Authorities to compare telemedicine systems on the market in order to identify the best ones (best of the breed) for a particular application. ISS is also developing plans to improve the web tool, including broadening its use for other TM systems. In this regard, a non-web-based approach has been tested off-line and was shown to be effective in evaluating less critical applications (tele-pathology, tele-oncology, and hospital nursing management [6, 7] ).
Acknowledgments
A dedicated team at the Istituto Superiore di Sanità, ISS, (Italian Health Institute), conducted the study of setting up of the quality system as a part of the project ERMETE 2001-2004: "e.R.ME. TE. Regioni per la Medicina Telematica" (Italian Regions for Telematics in Medicine).
References
1. Bracale M, Cesarelli M, Bifulco P. Telemedicine services for two islands in the bay of Naples. J Telemed Telecare 2002;8(1):5-10.
2. Bracale M, Monteagudo JL, Rossing N, Hakansson S, Reponen J, Bifulco P. Telemedicine in Europe - The Teleplans project. J Telemed Telecare 2002;8(5):308-10.
3. Williams TL, May CR, Esmail A. Limitations of patient satisfaction studies in telehealthcare: a systematic review of the literature. Telemed J E Health 2001;7(4):293-316.
4. Szot A, Jacobson FL, Munn S, Jazayeri D, Nardell E, Harrison D, Drosten R, Ohno-Machado L, Smeaton LM, Fraser HS. Diagnostic accuracy of chest X-rays acquired using a digital camera for low-cost teleradiology. Int J Med Inform 2004;73(1):65-73.
5. Hailey D, Roine R, Ohinmaa A. Systematic review of evidence for the benefits of telemedicine. J Telemed Telecare 2002;8(Suppl.1):1-30.
6. Giansanti D, Morelli S, Macellari V. Telemedicine technology assessment part II: tools for a quality control system. Telemed J E Health 2007;13(2):130-40.
7. Giansanti D, Morelli S, Macellari V. Telemedicine technology assessment part I: setup and validation of a quality control system. Telemed J E Health 2007;13(2):118-29.
8. Giansanti D, Morelli S, Bedini R, Macellari V. Un'esperienza italiana di controllo di qualità in telemedicina: il progetto eRMETE. Roma: Istituto Superiore di Sanità; 2008. (Rapporti ISTISAN, 08/23).
9. Hutten H, Stiegmaier W, Rauchegger G, Kiss G. A new approach to self-controlled e-learning of selected chapters in medical engineering and other fields at bachelor and master course level. Medical Eng and Phys 2005;27(7):605-9.
10. Pallikarakis N. Development and evaluation of an ODL course on medical image processing. Medical Eng and Phys 2005;27(7):605-9.
11. Stoeva M, Cvetkov A. e-Learning system ERM for medical radiation physics education. Medical Eng and Phys 2005; 27(7):605-9.
12. Giansanti D, Morelli S, Macellari V. A protocol for the assessment of diagnostic accuracy in tele-echocardiography imaging. Telemed J E Health 2007;13(4):399-405.
13. Giansanti D, Giordano A, Morelli S. Validation of an automatic tool for the assessment of image quality in digital tele-echocardiography. J Telemed Telecare 2008;14(7):342-4.
14. Giansanti D, et al. Digital telechocardiography: a look inside. In: Giansanti D, Morelli S (Ed.). Digital tele-echocardiography today: successes and failures. Ann Ist Super Sanità 2009;45(4):357-62.
15. Pinson MH, Wolf S. A new standardized method for objectively measuring video quality IEEE Transactions on broadcasting 2004;50(3):312-22.
16. ITU-R Recommendation BT.500-11 (06/02). Methodology for the subjective assessment of the quality of television pictures. Geneva, Switzerland: ITU; 2002.
17. Pinson MH, Wolf S. Video quality measurement techniques. Washington (DC): NTIA; 2002. (Report, 02-392 NTIA).
18. Pinson MH, Wolf S. In-service video quality metric (IVQM) User's manual. NTIA Handbook. Washington (DC): NTIA; 2002.
19. ITU-R Recommendation BT.500-10: Methodology for the subjective assessment of the quality of television pictures. Geneva, Switzerland: ITU; 2000.
Address for correspondence:
Daniele Giansanti
Dipartimento di Tecnologie e Salute
Istituto Superiore di Sanità
Viale Regina Elena 299
00161 Rome, Italy
E-mail: daniele.giansanti@iss.it
Submitted on invitation.
Accepted on 31 July 2009.