ORIGINAL ARTICLES AND REVIEWS
Exploring the usability of EUCERD core indicators for rare diseases
Rita Maria FerrelliI; Marta De SantisII; Amalia Egle GentileII; Domenica TaruscioII
IUfficio Relazioni Esterne, Istituto Superiore di Sanità, Rome, Italy
IICentro Nazionale Malattie Rare, Istituto Superiore di Sanità, Rome, Italy
ABSTRACT
In the context of the Community Programme in the field of Health, the European Commission financed a series of initiatives to support the development and use of indicators for planning health services for Rare Diseases (RDs). The European Project for Rare Disease National Plans Development (EUROPLAN) elaborated a set of 59 process and outcome indicators, for monitoring the implementation and for evaluating the impact of the National Plans on RDs. Due to the high number and difficulty in handling the indicators, the subsequent Joint Action "Working for RDs" planned to derive a selection of 21 core indicators that were adopted by the European Union Committee of Experts on RDs in June 2013. The descriptive study carried out in the framework of the Joint Action to select the key indicators to orient policies for RDs shows that core indicators represent an excellent opportunity to share knowledge and comparability among Member States.
Key words: rare diseases, health services, health planning, indicators
BACKGROUND
The Council of the European Union, in its 8 June 2009 Recommendation on an action in the field of Rare Diseases (RDs), recommends that "Member States elaborate and adopt a plan or strategy as soon as possible, preferably by the end of 2013 at the latest, aimed at guiding and structuring relevant actions in the field of Rare Diseases within the framework of their health and social systems" [1]. The development and use of indicators is an integral part of planning and designing Health Services, as they help to analyze and compare performance across population groups or geographic areas, and they can be useful in determining policy priorities. In the specific field of RDs, building a set of harmonized indicators on strategies/plans is a complex task because there are thousands of diseases that show different and specific issues and necessities. Due to the heterogeneity of RDs, the low number of patients/disease and the geographical spread, many indicators used for most common diseases are not applicable. Moreover Countries' Health Care Systems do not always follow the same organizational scheme across Europe and this is another difficulty. Nonetheless, in this context, the pooling of resources and the general consensus are the necessary basis to generate indicators. In order, to support this process, the programme of Community action in the field of Public Health co-funded the European Project for Rare Disease National Plans Development (EUROPLAN), a three-year project (2008-2011) coordinated by the Italian National Centre for Rare Diseases at National Institute of Health (Istituto Superiore di Sanità - ISS).
EUROPLAN aimed at identifying indicators to assess RDs initiatives with a view on monitoring the implementation and evaluating the impact of the national/regional plans on RDs, as well as the involved cost in maintaining this public health information system. EUROPLAN was structured along the areas of interest listed in the European Union (EU) proposal for a Council Recommendation, the EU Communication on RDs, namely:
Area 1. Plans and strategies in the field of RDs;
Area 2. Adequate definition, codification and inventorying of RDs;
Area 3. Research on RDs;
Area 4. Centres of Expertise and European Reference Networks for RDs;
Area 5. Gathering the expertise on RDs at European level;
Area 6. Empowerment of patient organisations;
Area 7. Sustainability.
For each area the following points were discussed:
background;
key Message;
rationale;
health context.
Standard definition used to describe the features of indicators took into account the main literature on intrinsic properties (understandability, reliability, validity, consistency, sensitivity, specificity, feasibility), resource demand (availability, sustainability, implementation, workload demand, timeliness) and decision making (applicability, coherence, comprehensiveness, policy relevance) [2-8].
EUROPLAN developed a set of 59 indicators, that were organised in process and outcome indicators, to monitor the implementation and to evaluate the impact of National Plans or Strategies for RDs [9]. While all indicators were considered relevant, from an operational perspective, they were quite numerous and difficult to handle. Therefore, the subsequent Joint Action "Working for Rare Diseases" (EJA) of the European Union Committee of Experts on Rare Diseases (EUCERD) planned to derive a selection of core indicators. In order to reduce the complexity of the list, EUROPLAN 2012-15 (as WP4 of the EJA) identified a series of 21 core indicators, according to criteria of usefulness and feasibility. The selection of core indicators has been the result of the work carried out in several procedural steps by both teams of EUROPLAN and the European Organisation for Rare Diseases (EURORDIS), a non-governmental patient-driven alliance of patient organisations and individuals active in the field of RDs, dedicated to improving the quality of life of all people living with RDs in Europe. The process foresaw two independent methodologies for selecting the "core indicators":
Delphi process (carried out by the ISS) with the collaboration of Ministry of Health representatives of 27 Member States (MS), 10 EURORDIS advisors and 4 experts. Participants were asked to select indicators according to two criteria: usefulness and feasibility of data collection. The process took place from 1 December 2012 to 14 February 2013. A satisfactory level of agreement was reached at the first round of the Delphi;
EURORDIS approach (carried out by EURORDIS) with the participation of 8 EURORDIS advisors in conjunction with their Ministry of Health lead contacts on national plans/strategies.
The drafting group elaborated the first version of the Recommendations, merging the results from the two methodologies and circulated them among Ministry of Health representatives of 27 Member States, 10 EURORDIS Advisors and 4 experts; their comments and amendments were included for the preparation of the second draft, that was discussed and amended during the EUROPLAN Workshop "Key Indicators for National Plans" (ISS, Rome, 25 March 2013). The subsequent drafts circulated among EUCERD members, until adoption. EUCERD adopted the list of 21 core indicators on 6 June 2013 [10].
Following EUCERD adoption of the core indicators in June 2013, a follow-up study about the use of indicators was deemed necessary for their fine-tuning and for improving their potential to orient policies for RDs. Therefore, a study has been planned in order to explore the degree of usability of the core indicators in selected EU Member States and to identify problems in relation to the use of the indicators.
METHOD
The descriptive study was structured in two components:
1. survey on the use of the indicators in selected EU Member States;
2. an exploratory collection of lessons to take into account for strengthening potential indicators to orient policies for RDs.
The sample size was defined after consultation with the EJA team that opted to carry out the study on a limited number of Countries. The selection of the Countries was done with purposive sampling, according to activities carried out in former and successful collaboration with the Italian National Centre for Rare Diseases, namely: Bulgaria, Croatia, Italy, Romania and Spain.
The 21 core indicators were entered in the web-based survey system SurveyMonkey (www.surveymonkey.com). The representatives of the Ministries of Health, directly involved in planning, monitoring and evaluating National Plans for RDs in the five selected Members States, were invited by E-mail to fill-in the questionnaire. Data were analysed by means of descriptive univariate analysis.
The second component of the study included open-answer questions about:
positive aspects highlighted while using the indicators;
problems faced while using the indicators;
opportunity to integrate the indicators.
RESULTS
The main results of the survey for the indicators are summarised in Table 1. Concerning the second part of the study, the following answers were collected in relation to the positive aspects highlighted while using the indicators:
collection of important information about National Plans on RDs and easiness to answer;
excellent opportunity to share knowledge and comparability among countries;
political usefulness;
chance to adapt national RDs policies to best examples available and recommended at EU level;
tool to timely follow up and report on national RDs activities;
possibility to follow-up the progress;
focus on relevant issues regarding development and implementation of National Plans which are common to the 28 EU Member States;
harmonisation of monitoring procedures and criteria and assessment of common RDs policies in the 28 EU Member States;
usefulness to capture and to describe the situation as far as the general measures adopted by countries.
Problems faced while using the indicators relate to the fact that:
quantitative indicators may not reflect qualitative improvements;
comparing Member States' different rules and regulation to make clear the different organization of the countries through short answers can be very difficult;
indicator on participation in European Reference
Networks is difficult to use (since European Reference Networks for RDs are not yet established);
visibility of what is done for RDs patients in programmes to support in their daily life integration (in place in all Countries) is low;
regionalisation may hamper data collection from regions to the national level; e.g. even though Spain has a national policy with public funds for research and for the national plan, there are regional plans on RDs with regional funds and the regional information is difficult to collect. Another example is done by Centers of Expertise: in Spain there are National Centers adhering to the national policy and regional centers adhering to the regional policies. If the indicator mixes up the two terms, the answer cannot be done.
In relation to the opportunity of integrating the indicators, the participants to the survey believe that increasing their number would entail a return to the "starting line". However, they did not exclude a possible future integration with EUCERD criteria on centres of expertise for RDs and Rare Disease Reference Networks, and they also suggest including information on new born screening policies and on the genetic diagnosis and genetic counselling policies.
DISCUSSION AND CONCLUSIONS
The objective of the Core Indicators for Rare Disease National Plans and Strategies is to capture relevant data and information on the process of planning, follow up and implementation of the plans and strategies. They are instrumental for the decision-making process related to the adoption, assessment and further development of RDs public policies. Their use is also for the report to the European Commission and for the annual reporting that Member States are required to do in view of the drafting of the "Report on the State of the Art on Rare Disease Activities in Europe of the EU Committee of Experts on Rare Diseases", published on a yearly basis by the Expert Group on Rare Diseases, former EUCERD.
The indicators are structured around the seven thematic areas of the European Council Recommendations:
plans and strategies in the field of RDs;
adequate definition, codification and inventorying of RDs;
research;
centres of expertise and European Reference Networks for RDs;
gathering the expertise on RDs at European level;
empowerment of patient organisations;
sustainability.
As binary variables, they give at a glance the picture of the Country in relation to the examined issues. As such, they may not be able to give all the related nuances. Qualitative information will be necessary to fill in the gap.
Indicators are supposed to be SMART: Specific, Measurable, Appropriate, Realistic, Time-bound [11]. Therefore, their use will have to be related to the target group. The factors that need to be changed and the objectives under study will have to be achievable and attainable, acceptable for the target group and realistically achievable according to the available resources in terms of time, money and knowledge.
In conclusion, the pilot study on the usability of the core indicators highlighted their usefulness in giving a snapshot of the main areas of concern for national planning for RDs. Moreover, a synthetic representation allows for defining common policies at European level; e.g. the situation about RDs coding, while showing heterogeneity in the system adopted for coding RDs in the Country, allows for identifying a system that is likely to be used in most, hopefully all, of the Member States.
The core indicators represent an excellent opportunity to share knowledge and comparability among Member States. However, it is important to acknowledge the strengths and weaknesses of the single tools and to be aware that quantitative indicators may not reflect qualitative substantial aspects of the issue that they are measuring. This may be studied with appropriate and different tools, according to the objectives that have been set for the matter under study.
Acknowledgements
This work was carried out thanks to the support of the EUCERD Joint Action: Working for Rare Diseases (Grant 20112201), which was funded by the European Commission (Executive Agency for Health and Consumers, now CHAFEA) in the context of the European Union's Second Community Programme in the field of Health.
REFERENCES
1. European Union. Council Recommendation of 8 June 2009 on an action in the field of rare diseases (2009/C 151/02). Official Journal of the European Union C171/7, 3/7/2009. Available from: http://eur-lex.europa.eu/LexUriServ/LexUriServ.do?uri=OJ:C:2009:151:0007:0010:EN:PDF.
2. Cambridge advanced learner's dictionary. Cambridge: Cambridge University Press; 2009.
3. McGlynn EA. Six challenges in measuring the quality of health care. Health Aff 1997;16(3):7-21.
4. Miller GA (Ed). WordNet. Princeton University. The Trustees of Princeton University; 2009. Available from: http://wordnet.princeton.edu/.
5. Murray CJL. Health Statistics 2. Towards good practice for health statistics: lessons from the Millennium Development Goal health indicators. Lancet 2007;369:862-73.
6. Nikolai LA, Bazley JD, Jones JP. Intermediate accounting. 11 ed. Mason OH: South-Western Cengage Learning; 2009.
7. Porta M. A dictionary of epidemiology. 5th ed. New York: Oxford University Press; 2008.
8. WordNet (r) 2.1 (2005). http://onlinedictionary.datasegment.com/word/applicability.
9. EUROPLAN - European Project for Rare Disease National Plans Development. Available from: www.europlanproject.eu/Resources/docs/2008-2011_3.EUROPLANIndicators.pdf.
10. EUCERD - European Union Committee of Experts on Rare Disease. EUCERD Recommendation on core indicators for rare disease national plans/strategies. Available from: http://ec.europa.eu/health/rare_diseases/docs/eucerd_indicators_en.pdf.
11. Doran GT. There's a S.M.A.R.T. way to write management's goals and objectives. Manag Rev (AMA FORUM) 1981;70(11):35-6.
Address for correspondence:
Rita Maria Ferrelli
Uffficio Relazioni Esterne, Istituto Superiore di Sanità
Viale Regina Elena 299
00161 Rome, Italy
E-mail: rita.ferrelli@iss.it
Submitted on 26 March 2015.
Accepted on 3 September 2015.
Conflict of interest statement: There are no potential conflicts of interest or any financial or personal relationships with other people or organizations that could inappropriately bias conduct and findings of this study.