Abstract

This is an integrative review of the literature, which sought to locate evidence regarding the evaluation of the performance of tuberculosis control programs in the Brazilian and Spanish contexts. Articles indexed in databases, search directories and virtual libraries produced in Brazil and Spain from 2002 to 2013 were selected. A total of 17 articles (9 Brazilian and 8 Spanish) were included for analysis. The intention was to identify program evaluation concepts, methodologies and indicators for TB control program evaluation in both countries. In the Spanish context, a concern with results-oriented management and the attempt to develop broader indicators for the assessment of a series of health programs, including those related to tuberculosis control, was identified. The Brazilian references showed greater proximity to the classic methods of evaluation research, and a greater variety of objects of study regarding the activities expected in a control program. The findings highlight the complexity of program evaluation activities, which involve multiple and diverse stakeholders, and point to the need for integration of epidemiological and operational indicators in tuberculosis.

Tuberculosis; Program evaluation; Health services research; Quality indicators in health care

Introduction

For those involved in public health, scrutinizing the performance of ongoing measures undertaken through programs and public policies is an act that is both continuous and extremely necessary. Taking into consideration social, political and economic interests, this type of examination is done in order to attain an efficient and effective model for the provision of health services. It should be one that meets the needs of individuals and communities¹1. Viacava F, Ugá MAD, Porto S, Laguardia J, Moreira RS. Avaliação de Desempenho de Sistemas de Saúde: um modelo de análise. Cien Saude Colet 2012;17(4):921-934..

In Brazil, over the 26 years of the National Health System (SUS), the need for evidence that can feed into the process for making the system better²2. Figueiredo ANA, Tanaka O. A avaliação no SUS como estratégia na reordenação da saúde. Cad Fundap 1996; 19:98-105.,³3. Paim JS. A Constituição Cidadã e os 25 anos do Sistema Único de Saúde (SUS). Cad Saude Publica 2013; 29(10):1927-1936. has resulted in many discussions on the use of evaluation programs in health care. These discussions have been taking place in governmental institutions and in academic and scientific circles⁴4. Santos TMMG, Silva NS, Nogueira LT, Vilarinho LM, Nunes GBL. Tendências da produção científica sobre avaliação em saúde no Brasil.Rev. Rene 2010; 11(3):171-179.,⁵5. Fernandes FMB, Ribeiro JM, Moreira MG. Reflexões sobre avaliação de políticas de saúde no Brasil. Cad Saude Publica 2011; 27(9):1667-1677.. This has been the case for other countries that opted for a national health system such as in Spain that provides universal health coverage as part of their national health system. Their system is similar to the Brazilian health system in that reforms where changes in health care were linked to the process of redemocratization after a military dictatorship⁶6. Campos PM. Estructuras Político-Administrativas y Salud Pública en España. Rev San Hig Púb 1994; 68:57-64..

It is necessary to remember that health care sector reforms in both of these countries came about through concerted mobilized actions which even today fuels tensions between government bodies and society⁷7. Somoza EJL. La evaluación de la reforma sanitaria, una base necesaria para reformas futuras. Rev Esp Salud Pública 2008; 65(4):287-297.,⁸8. Paim JS, Teixeira, CF. Política, planejamento e gestão em saúde: balanço do estado da arte. Rev Saude Publica 2006, 40(Esp):73-78.. This in turn puts a spotlight on the need for mediation between the two parties which then feeds into the public policy formulation process in health care⁸8. Paim JS, Teixeira, CF. Política, planejamento e gestão em saúde: balanço do estado da arte. Rev Saude Publica 2006, 40(Esp):73-78..

The recent successive cuts to Spanish social programs due to the economic crisis⁹9. Ortún V, Callejón M. Crisis en España: ¿cómo renovar los servicios sanitarios. In: Casajuana J, Gérvas J, organizadores. La renovación de la Atención Primaria desde la consulta. Madrid: Springer Healthcare; 2012. p. 157-170. and the barriers people face when using SUS (in the pursuit of complying with certain principles and directives)²2. Figueiredo ANA, Tanaka O. A avaliação no SUS como estratégia na reordenação da saúde. Cad Fundap 1996; 19:98-105.,¹⁰10. Fleury S. Reforma sanitária brasileira: dilemas entre o instituinte e o instituído. Cien Saude Colet 2009; 14(3):743-752. has left many assessors in a difficult position in relation to their roles. It also reinforces the permanent existence of social control measures that dictate end user outcomes. The road to deal with the challenges identified by evaluations that have been done, is long. In order for information from these evaluations to aid in service management and programs, it is necessary to tackle: the different concepts arising from the area, the lack of evaluative processes that have become institutionalized and the multiple dimensions within the challenges which are comparative to those in other social spheres¹¹11. Tanaka OY, Melo C. Avaliação de serviços e programas de saúde para a tomada de decisão. In: Rocha AA, César CLG, organizadores. Saúde pública: bases conceituais. São Paulo: Atheneu; 2008. p. 119-131..

Health care has produced organized responses to deal with problems. With this in mind, we have put a spot on Tuberculosis (TB) which is a disease that has well established methods for diagnosis and treatment. However the management of TB from a health system point of view requires coordinated actions amongst different parties which include prevention and control¹²12. Villa TCS, Ruffino-Netto A, Scatena LM, Andrade RLP, Brunello MEF, Nogueira JA, Palha PF, Sá LD, Assis MMA, Vendramini SHF, Monroe AA, Arcêncio RA, Arakawa T. Health services performance for TB treatment in Brazil: a cross-sectional study. BMC Health Services Research [serial on the internet] 2011; 11(241) [cited 2013 Oct 10]. Available from: http://www.biomedcentral.com/1472-6963/11/241.
http://www.biomedcentral.com/1472-6963/1... .

The epidemiological indicators for TB and treatment outcomes from the two countries show the need to analyze the response capability of the Programs for the Control of Tuberculosis (PCT) in the context of their respective health systems. In Brazil TB continues to be the main cause of death for those infected with HIV/AIDs. It is also the fourth cause of death by an infectious disease among the general population¹³13. Brasil. Ministério da Saúde. Programa Nacional de Controle de Tuberculose. Tuberculose em números. [acessado 2014 jan 23]. Disponível em: http://www.saude.gov.br
http://www.saude.gov.br... with an incident rate of 35.8 cases per 100,000 inhabitants in 2012. The percentage of those cured among newly diagnosed cases was 67.7% in 2011¹³13. Brasil. Ministério da Saúde. Programa Nacional de Controle de Tuberculose. Tuberculose em números. [acessado 2014 jan 23]. Disponível em: http://www.saude.gov.br
http://www.saude.gov.br... . In Spain the incident rate for TB was 13.0 cases per 100,00 inhabitants in 2012. The percentage of those cured among newly diagnosed cases was only 73,3% in 2011¹⁴14. Centro Nacional de Epidemiología, Instituto de Salud Carlos III.Informe epidemiológico sobre la situación de la tuberculosis en España. Año 2012. Madrid: Centro Nacional de Epidemiología, Instituto de Salud Carlos III; 2013.. In Spain 46.3% of registered cases of TB in the same period were cases of foreigners the majority of whom came from: South America, Eastern Europe and the northern Africa. The immigrant’s living conditions and their general precarious circumstances are elements which present difficulties for the public authorities to control the disease in their country¹⁴14. Centro Nacional de Epidemiología, Instituto de Salud Carlos III.Informe epidemiológico sobre la situación de la tuberculosis en España. Año 2012. Madrid: Centro Nacional de Epidemiología, Instituto de Salud Carlos III; 2013..

Both in Brazil and Spain the decentralization and strengthening of the Primary Attention to Health Coordinator role, considered as part of the strategy to effect reform in the health system, established a context that influenced the specific issue of controlling TB. That coupled with health care monitoring are being developed.

The operational complexity in the control of a diseases such as TB and the evidence concerning the monitoring of the performance of health programs are potentially very interesting for researchers and managers in both countries. This article seeks to carry out a bibliographical study on the evaluations of the TB control program based on the literature produced in Brazil and Spain. The choice of these two countries was based on: the similarities of their respective public national health systems, the common challenges encountered in service management over the years, and the development of programs to control diseases.

Material and Methods

This work is an integrated review of the literature whose main purpose is to deepen our understanding of a specific phenomenon looking at both its strengths and weaknesses. The is a rational and critical analysis of the given purpose¹⁵15. Whittemore R, Knafl K. The integrative review: update methodology.J Adv Nurs 2005; 52(5):546-553..

The integrated review consists of an ordered and systematic analysis of the results obtain from literature. It also suggests the development of six steps¹⁵15. Whittemore R, Knafl K. The integrative review: update methodology.J Adv Nurs 2005; 52(5):546-553.: 1) identifying the key investigative question that needs to be answered and which has a structured theme to be both researched and documented. Also we needed to identify descriptors and key-words that could be searched for in the literature; 2) establishing search strategies and defining criteria for what material would be included and excluded; 3) categorization of the selected information and defining information of interest to be reviewed; 4) critical analysis of the final body of work; 5) interpretation of the findings; 6) presentation of the reviewed work.

For this study the question that needed to be addressed was: “How has the issue of performance in relation to the control programs for TB been approached in the technical/scientific literature in Brazil and Spain?”

In this study the following was reviewed: original scientific research, topical and critical articles written in this area, literature on TB, studies of cases and accounts of TB, monographies, dissertations and thesis. We also reviewed: institutional publications (reports, and plans), protocols and practical clinical guides (practical guidelines) produced in Brazil and Spain, literature published in Portuguese, Spanish and English and that was published in the last decade (2002-2013). Information from the following databases was used: CINAHL (Cumulative Index for Nursing and Allied Health Literature), Medline (Medical Literature Analysis and Retrieval System Online), Lilacs (Literatura Latino-Americana e do Caribe em Ciências de Saúde) e SCOPUS (Web of Science). The virtual library SciELO (Scientific Electronic Library Online) also was used as a relevant source for searches. The search engine Google Scholar (Google Acadêmico) complemented searches as it allowed for the identification of material of interest not available in the sources mentioned above.

The choice of search descriptors was done before the start of this study through looking through terminology libraries on health - Descriptors on Health Science (DeCS) from the Regional Medical Library at the Pan-American Organization for Health (OPAS/Bireme) and Medical Subject Headings (MSH) from the National Library of Medicine. Key terms were defined based on their concepts, which would make possible searches easy where a search using a descriptor did not provide satisfactory results.

A bibliographic study was done on the aforementioned databases and sources through cross-referencing the descriptors and the terms mentioned in Chart 1 between the months of October and December 2013. We also applied, as a search strategy, the use of the descriptor tuberculosis and the boolean operation “AND” with the other selected issues to be covered by the study. The aforementioned was combined in itself with the use of the operation “OR”. The recuperation of the material produced in the Brazilian and Spanish contexts was done through the use of the words “Brasil” and “Espanha” in the search fields relative to the country that the research was affiliated to. This was also the case for the title and/or a summary of the material.

The inclusion criteria of identified articles through searches through the presence of theoretical concepts and/or evidence in relation to the evaluation of PCTs in the context of the health system. This was also the case for indicators that were duplications and irrelevant material was excluded from this study.

Once we had all of the documents we critically read through all of the material in order to produce an excellent final product. Data of interest was described and grouped in thematic categories to facilitate the presentation and discussion of the results. The identification and categorization of the utilized indicators in these materials followed the ideas of Waldman¹⁶16. Waldman EA. Vigilância em Saúde Pública. São Paulo: Universidade de São Paulo; 1998. and the Interagency Network for Information in Health (RIPSA)¹⁷17. Rede Interagencial de Informações para a Saúde (RIPSA).Indicadores e dados básicos para a saúde (IDB); 2011. [acessado 2014 jun 11]. Disponível em: http://www.ripsa.org.br/php/level.php?lang=pt&component=68&item=23.
http://www.ripsa.org.br/php/level.php?la... , that defines indicators as a type of quantitative or qualitative measurement. It organizes and captures relevant information from elements that make up an objective that is evaluated. The indicators were categorized as epidemiological (which expresses the magnitude of the disease) and operational (which measures activities carried out and the quantity/quality of actions taken based on a proposal)¹⁷17. Rede Interagencial de Informações para a Saúde (RIPSA).Indicadores e dados básicos para a saúde (IDB); 2011. [acessado 2014 jun 11]. Disponível em: http://www.ripsa.org.br/php/level.php?lang=pt&component=68&item=23.
http://www.ripsa.org.br/php/level.php?la... .

Figure 1 shows the quantity of information obtained and selected in accordance with the specified criteria. The result was 17 documents which included articles, monographies, theses and protocols.

Results

Charts 2 and 3 show the principal data related to the material obtained, according to the year, source, document typology, and other considerations related to the study and main objective (when pertinent). It is based in a Spanish and Brazilian context respectively.

In a general sense the Spanish production is concentrated on a period before the production with the majority of the selected articles published before 2010. The search for Brazilian studies resulted in obtaining recently published articles - which meant that the final document was made up of many articles published in the last five years. In relation to the type of material identified, half of the documents selected came from different types of publications as well as incorporating original research results. For example there were review/critical articles and narrative reviews (n = 5), theses and monographies (n = 3) and protocols (n = 1). All of the above was an important source of knowledge for this present study.

In relation to choosing which original articles to be a part of this study we used a number of strategies. The most relevant was to use those that had the information on epidemiological investigations. This was the case in the majority of the original articles chosen (6 out of 9 publications from both countries). The majority of the articles with these elements were Brazilian (4 Brazilian articles in comparison to 1 Spanish article).

Throughout the research we used indicators and parameters to measure the performance. We used a number of different evaluative indicators that related to the following areas: structure, processes and results¹⁸18. Villalbi JR, Tresseras R. Evaluación de políticas y planes de salud. Gac Sanit 2011; 25(Supl. 1):17-24.

19. Gonzales RIC, Monroe AA, Assis EG, Palha PF, Villa TCS, Ruffino-Netto A. Desempenho de serviços de saúde no tratamento diretamente observado no domicílio para controle da tuberculose. Rev esc enferm USP 2008; 42(4):628-634.

20. Dos Santos DMLR. Implementação do programa de controle da tuberculose do município de Abreu e Lima [monografia]. Recife: Fundação Oswaldo Cruz; 2010.-²¹21. Gonçalves MJF. Avaliação de Programa de Saúde: O Programa Nacional de Controle de Tuberculose no Brasil. Saúde Transform Soc 2012; 3(1):13-17.. We also included categories such as: cover, quality, satisfaction and accessibility¹⁸18. Villalbi JR, Tresseras R. Evaluación de políticas y planes de salud. Gac Sanit 2011; 25(Supl. 1):17-24. all of which forms part of the process. We sought to define efficiency and effectiveness in the relevant contexts¹⁸18. Villalbi JR, Tresseras R. Evaluación de políticas y planes de salud. Gac Sanit 2011; 25(Supl. 1):17-24.,²²22. Chaulk PC, Kazandin VA, Vellejo Gutierrez P. Evaluación en salud pública: lecciones aprendidas de la gestión de la tuberculosis pulmonar.Gac Sanit 2008; 22(4):362-370..

The structural evaluation was defined based on the material and resources that were analyzed which were both material and human¹⁸18. Villalbi JR, Tresseras R. Evaluación de políticas y planes de salud. Gac Sanit 2011; 25(Supl. 1):17-24.,²¹21. Gonçalves MJF. Avaliação de Programa de Saúde: O Programa Nacional de Controle de Tuberculose no Brasil. Saúde Transform Soc 2012; 3(1):13-17.. Also the services and activities that came from the above were also evaluated. The evaluation of the process involved investigations into the activities of the programs, their coverage and their quality¹⁸18. Villalbi JR, Tresseras R. Evaluación de políticas y planes de salud. Gac Sanit 2011; 25(Supl. 1):17-24.,²¹21. Gonçalves MJF. Avaliação de Programa de Saúde: O Programa Nacional de Controle de Tuberculose no Brasil. Saúde Transform Soc 2012; 3(1):13-17.. We also took into account the satisfaction ratings of the patients and accessibility for the services providers to the patients. Finally an evaluation of the results focused on the set targets for the services and activities¹⁸18. Villalbi JR, Tresseras R. Evaluación de políticas y planes de salud. Gac Sanit 2011; 25(Supl. 1):17-24.,²¹21. Gonçalves MJF. Avaliação de Programa de Saúde: O Programa Nacional de Controle de Tuberculose no Brasil. Saúde Transform Soc 2012; 3(1):13-17..

Efficiency was measured looking at the relationship between the resources used and whether the objectives were met based on the acts that were done¹⁸18. Villalbi JR, Tresseras R. Evaluación de políticas y planes de salud. Gac Sanit 2011; 25(Supl. 1):17-24.,²²22. Chaulk PC, Kazandin VA, Vellejo Gutierrez P. Evaluación en salud pública: lecciones aprendidas de la gestión de la tuberculosis pulmonar.Gac Sanit 2008; 22(4):362-370.,²³23. Rodrigo TS. Evaluación de programas de prevención y control de tuberculosis [tese]. Barcelona: Universidad Autonóma de Barcelona; 2003.. The economic evaluation was done analyzing the costs and benefits based on the use of the given resources¹⁸18. Villalbi JR, Tresseras R. Evaluación de políticas y planes de salud. Gac Sanit 2011; 25(Supl. 1):17-24.,²²22. Chaulk PC, Kazandin VA, Vellejo Gutierrez P. Evaluación en salud pública: lecciones aprendidas de la gestión de la tuberculosis pulmonar.Gac Sanit 2008; 22(4):362-370.. Effectiveness was measured based on looking at the results for the target population for a particular program/intervention and the effect and impact of that program on the community¹⁸18. Villalbi JR, Tresseras R. Evaluación de políticas y planes de salud. Gac Sanit 2011; 25(Supl. 1):17-24.,²²22. Chaulk PC, Kazandin VA, Vellejo Gutierrez P. Evaluación en salud pública: lecciones aprendidas de la gestión de la tuberculosis pulmonar.Gac Sanit 2008; 22(4):362-370.,²³23. Rodrigo TS. Evaluación de programas de prevención y control de tuberculosis [tese]. Barcelona: Universidad Autonóma de Barcelona; 2003..

The dimensions and indicators from evaluative research references was also taken into account²⁴24. Oliveira LGD, Natal S. Avaliação de implantação do Programa de Controle da Tuberculose no município de Niterói/RJ. Rev Bras Pneum Sanitaria 2007; 15(1):29-38.

25. Oliveira LGD, Natal S, Felisberto E, Alves CKA, Santos AM. Modelo de avaliação do programa de controle da tuberculose. Cien Saude Colet 2010; 15(1):997-1008.-²⁶26. Heufemann NEC, Gonçalves MJF, Garnelo ML. Avaliação do programa de controle da tuberculose em cenário Amazônico: desafios em Coari. Acta Amaz 2013; 43(1):33-42.. The evaluation on what had been implemented used a logical model. It was based on creating a description of the components, inputs, activities, products, results and impacts expected for the PCT. The theoretical evaluative model looked at whether the resources and services used allowed for expected results to be achieved. We also wanted to see whether the actual results obtained were different to or close to what had be proposed. We also took into account the organizational and political context of the study. This included: political and financial autonomy, inter sectoral actions, social vulnerabilities and management. In relation to implemented programs we took into account: service coverage, access, laboratorial diagnosis and support, pharmaceutical treatment and assistance, integrality and epidemiological vigilance. Related to the aforementioned we also looked at their effects (clinical end results and user satisfaction rating). Each dimension is subdivided into categories. The categories have their own criteria/indicators that make up the matrix used for the evaluations. They assess the degree to which the PCT was implemented into the health system.

The continuation model for attention and the indicators used to measure them were identified in one of the Spanish documents. What was considered continuity of action was “a rational association between proposed clinical interventions, changes that were successfully achieved in the health of individuals and subsequent modifications in the utilization of health services”²²22. Chaulk PC, Kazandin VA, Vellejo Gutierrez P. Evaluación en salud pública: lecciones aprendidas de la gestión de la tuberculosis pulmonar.Gac Sanit 2008; 22(4):362-370.. The definition still includes continuity of attention as an indicator for the treatment of tuberculosis which should also take into account the participation of patients in the period during and after treatment. It should also take into account the integration between different public health bodies and the communities involved in these processes²²22. Chaulk PC, Kazandin VA, Vellejo Gutierrez P. Evaluación en salud pública: lecciones aprendidas de la gestión de la tuberculosis pulmonar.Gac Sanit 2008; 22(4):362-370..

The main identified indicators were categorized as epidemiological or operational and are listed Chart 4.

Discussion

In the area of TB there are difficulties in outlining the scope of studies about the performance of specific program actions or about the services that form part of the program. This is also the case for PCT which practices their own activities. The evidence showed that in situations where evaluations need to be carried out on services which have involved many different organizations and health professional tasked at dealing with TB, it is extremely difficult to establish measurements that take into account the responsibilities of all those involved in the process¹⁸18. Villalbi JR, Tresseras R. Evaluación de políticas y planes de salud. Gac Sanit 2011; 25(Supl. 1):17-24.,²²22. Chaulk PC, Kazandin VA, Vellejo Gutierrez P. Evaluación en salud pública: lecciones aprendidas de la gestión de la tuberculosis pulmonar.Gac Sanit 2008; 22(4):362-370.,³¹31. Villalbi JR, Casasa C, Bartoll X, Artazcoz L, Ballestín M, Borrell C, Camprubí E, Durán J, García R, Rodríguez P, Salamero, M. Indicadores para la gestión de los servicios de salud pública. Gac Sanit 2010; 24(5):378-384.. Reductions made in the amount of material obtained for this study (which was necessary) did not result in chances in wider discussions and dealing with critical issues. However this act made it difficult to systematize the evidence built amidst the variety of publications that were available.

The scarcity of published works in the area of PCT evaluation was noted in the papers that we used for both counties²²22. Chaulk PC, Kazandin VA, Vellejo Gutierrez P. Evaluación en salud pública: lecciones aprendidas de la gestión de la tuberculosis pulmonar.Gac Sanit 2008; 22(4):362-370.,²³23. Rodrigo TS. Evaluación de programas de prevención y control de tuberculosis [tese]. Barcelona: Universidad Autonóma de Barcelona; 2003.,²⁷27. Alonso JLP, García-Martos P, Casanova PM, Saldarreaga MA, Vega J, Gutiérrez JM. Valoración de la aplicación de un programa de prevención y control de tuberculosis. Enferm Infec Microb Clínica 2002; 20(4).,²⁹29. 29 Rodrigo TS, Cayla JA. Efectividad de los programas de control de la tuberculosis en España. Med Clínica 2003; 121(10):375-377.,³³33. Gonçalves MJF, Penna MLF. Morbidade por tuberculose e desempenho do programa de controle em municípios brasileiros, 2001-2003. Rev Saude Publica 2007; 41(Supl. 1):95-102.. The theme of evaluation, in spite of being based on current established theories, is still an area that has been frequently studied in-depth in relation to TB. Kritski et al.³⁵35. Kritski AL, Villa TCS, Trajman A, Lapa e Silva JR, Medronho RA, Ruffino-Netto A. Duas décadas de pesquisa em tuberculose no Brasil: estado da arte das publicações científicas. Rev Saude Publica 2007; 41(Supl. 1):9-14. showed that, in Brazil, there was an increase in the number of study results on TB in indexed periodicals which have used scientific rigor from the year 2000. However they were based on studies related to basic sciences. In Spain, Ramos et al.³⁶36. Ramos JM, Padilla S, Masiá M, Gutiérrez F. A bibliometric analysis of tuberculosis research indexed in PubMed, 1997–2006. Int J Tuberc Lung Dis 2008; 12(12): 1461-1468.showed that studies on TB that exist are mainly clinical studies or commissioned by politicians. Doctors were the the main authors of these medical papers between 1997 and 2006 and hospitals were the main institutions that were responsible for their publications. The findings from Brazilian papers in this area which are recent and which were used in this paper can be related to the prioritization for the implementation of PCT actions in primary care in the country. Research publishers also support work carried out in this area.

The task of evaluating the performance of control programs requires the use of a well-defined terms of reference which allows for the selection and analysis of pertinent indicators. This in turn allows for the study to be done and for the results to be clearly understood³⁷37. Novaes HMD. Avaliação de programas, serviços e tecnologias em saúde. Rev Saude Publica 2000; 34(5):155-159.. However in the area of evaluations, new definitions are constantly created in spite of the fact that many ideas and existing methods are used in specific ways. The upshot is a plethora of terminologies³⁸38. Silva LMV, Formigli VLA. Avaliação em saúde: limites e perspectivas. Cad Saude Publica 1994; 10(1):80-91.,³⁹39. Furtado JP. Um método construtivista para a avaliação em saúde.Cien Saude Colet 2001; 6(1):165-182. and methodologies⁴⁰40. Uchimura KY, Bosi MLM. Avaliação da qualidade ou avaliação qualitativa do cuidado em saúde? Rev Saude Publica 2007; 41(1):150-153. that cover epidemiology to classical theories for evaluative research. Their evaluation includes quantitative and qualitative research. Fernandes et al.⁵5. Fernandes FMB, Ribeiro JM, Moreira MG. Reflexões sobre avaliação de políticas de saúde no Brasil. Cad Saude Publica 2011; 27(9):1667-1677., noted that in spite of the non-consensual nature that prevails in the field of evaluation, this very fact allows for researchers to be creative. It also allows for the identification of risks to be resolved and the publication of results and findings.

The methods and indicators identified in this review confirmed the existence of two strands. One limits the normative evaluation base with specific indicators and are limited to components in the control program. The other is close to the evaluative research with indicators that seek to relate context differences in which the components in the program relate to each other, including subjective aspects²⁰20. Dos Santos DMLR. Implementação do programa de controle da tuberculose do município de Abreu e Lima [monografia]. Recife: Fundação Oswaldo Cruz; 2010.,²⁴24. Oliveira LGD, Natal S. Avaliação de implantação do Programa de Controle da Tuberculose no município de Niterói/RJ. Rev Bras Pneum Sanitaria 2007; 15(1):29-38.,²⁶26. Heufemann NEC, Gonçalves MJF, Garnelo ML. Avaliação do programa de controle da tuberculose em cenário Amazônico: desafios em Coari. Acta Amaz 2013; 43(1):33-42..

The relevance of the use of epidemiological indicators is discussed in the identified material³²32. Braga JU. Vigilância epidemiológica e o sistema de informação da tuberculose no Brasil, 2001-2003. Rev Saude Publica 2007; 41(Supl. 1):77-87.,³⁴34. Santos MSLG. A estratégia DOTS no estado de São Paulo: desafios políticos, técnicos e operacionais no controle da Tuberculose [tese]. Ribeirão Preto: Universidade de São Paulo; 2009., since the magnitude of the disease can influence the recognition of the problem by the health service. Indicators that made references to AIDs and the related infection HIV were also identified for this paper³²32. Braga JU. Vigilância epidemiológica e o sistema de informação da tuberculose no Brasil, 2001-2003. Rev Saude Publica 2007; 41(Supl. 1):77-87.,³³33. Gonçalves MJF, Penna MLF. Morbidade por tuberculose e desempenho do programa de controle em municípios brasileiros, 2001-2003. Rev Saude Publica 2007; 41(Supl. 1):95-102.. Its use was justified based on the known links between the two chronic illnesses, which makes care given more complex and can influence the performance of control programs for both diseases.

Limitations in the use of secondary data and the random nature of the choices for cuts in the points used in the classification criteria for the districts was mentioned³²32. Braga JU. Vigilância epidemiológica e o sistema de informação da tuberculose no Brasil, 2001-2003. Rev Saude Publica 2007; 41(Supl. 1):77-87.. What was also mentioned was the lack of information on the performance of the National Program for the Control of Tuberculosis in Brazil and its districts³³33. Gonçalves MJF, Penna MLF. Morbidade por tuberculose e desempenho do programa de controle em municípios brasileiros, 2001-2003. Rev Saude Publica 2007; 41(Supl. 1):95-102.. Considering the above, the role of the epidemiological regulator and the evaluation of various health information systems that bring together information of interest for a better understanding of the PCT, were issues that were frequently tackled in identified studies¹⁸18. Villalbi JR, Tresseras R. Evaluación de políticas y planes de salud. Gac Sanit 2011; 25(Supl. 1):17-24.,²²22. Chaulk PC, Kazandin VA, Vellejo Gutierrez P. Evaluación en salud pública: lecciones aprendidas de la gestión de la tuberculosis pulmonar.Gac Sanit 2008; 22(4):362-370.,²³23. Rodrigo TS. Evaluación de programas de prevención y control de tuberculosis [tese]. Barcelona: Universidad Autonóma de Barcelona; 2003.,²⁵25. Oliveira LGD, Natal S, Felisberto E, Alves CKA, Santos AM. Modelo de avaliação do programa de controle da tuberculose. Cien Saude Colet 2010; 15(1):997-1008.,²⁹29. 29 Rodrigo TS, Cayla JA. Efectividad de los programas de control de la tuberculosis en España. Med Clínica 2003; 121(10):375-377.,³¹31. Villalbi JR, Casasa C, Bartoll X, Artazcoz L, Ballestín M, Borrell C, Camprubí E, Durán J, García R, Rodríguez P, Salamero, M. Indicadores para la gestión de los servicios de salud pública. Gac Sanit 2010; 24(5):378-384..

The indicators used for evaluation must be clearly defined and objectively measurable. The criteria should be based on scientific evidence so that the consent mechanisms are able to provide relevant and pertinent parameters. This is the prerogative of evaluations done in the area of service management²⁸28. Grupo de trabajo TIR (TB e infecciones respiratorias). Sociedad Española de Neumología y Cirugía Torácica (SEPAR). Normativa sobre la prevención de la tuberculosis, Arch Bronconeumol 2002; 38(9):441-451.,⁴¹41. Tanaka OY, Tamaki EM. O papel da avaliação para a tomada de decisão na gestão de serviços de saúde. Cien Saude Colet 2012; 17(4):821-828.. Aside from this, the indicators ought to be flexible and should consider the context in which that are to be used. This is because sometimes it is necessary to assume that, in certain situations, the objectives are not realistic and one must reconsider intentions in order to obtain certain goals²⁸28. Grupo de trabajo TIR (TB e infecciones respiratorias). Sociedad Española de Neumología y Cirugía Torácica (SEPAR). Normativa sobre la prevención de la tuberculosis, Arch Bronconeumol 2002; 38(9):441-451.. In a general way a reading of the Brazilian literature show major variations in the scope and propositions of the methodologies for evaluating the PCT. The identified operational indicators are aimed at detection quality, diagnosis, treatment and compatibility with a country with an epidemiological situation. In Spain, contact exams and multi-drug resistance emerged as relevant themes and were highlighted in the utilized indicators in publication that covered the state of the art.

In spite of this in both the Brazilian and Spanish productions there was an identification of a necessity for resources and activity integration. Intervention and coordination of the above in the network of health care systems is necessary for the attainment of proposed goals. In order to monitor and evaluate the program it is necessary to determine responsibilities, assign resources to be used and establish criteria.

Villalbi e Tresseras¹⁸18. Villalbi JR, Tresseras R. Evaluación de políticas y planes de salud. Gac Sanit 2011; 25(Supl. 1):17-24.highlight the complicated and confused present situation in evaluating health programs. They noted that the real components of these complex programs are not always explicit particularly throughout the time that they are operational. Gonçalves²¹21. Gonçalves MJF. Avaliação de Programa de Saúde: O Programa Nacional de Controle de Tuberculose no Brasil. Saúde Transform Soc 2012; 3(1):13-17., in the same vein noted that a program that represents all of programs cannot be evaluated based on just one indicator. More than one needs to be analyzed together. In order for an evaluation and its results for a program to be effective, it is often necessary to obtain indicators from other sources of information which are not always integrated in the management of intervention in themselves.

It is worth remembering that Brazil, organized as a republic with 27 federal states and a president, has a health system the responsibility of which is divided amongst three strands of government: union, states and municipalities. Oversight and control lies with the Federal Government. The PCTs are formally organized as part of the above. Local bodies delimit their set objectives and identify key resources and processes that are necessary in order to meet their goals in their particular situation⁴²42. Brasil. Ministério da Saúde (MS). Secretaria de Vigilância em Saúde. Departamento de Vigilância de Doenças Transmissíveis.Experiências de Monitoramento e Avaliação no Controle da Tuberculose no Brasil. Brasília: MS; 2013..

For SUS it is incumbent upon municipalities and the local health care network to take actions to control TB. This includes mapping the spread of the disease and equipping the health centers with appropriate resources⁴²42. Brasil. Ministério da Saúde (MS). Secretaria de Vigilância em Saúde. Departamento de Vigilância de Doenças Transmissíveis.Experiências de Monitoramento e Avaliação no Controle da Tuberculose no Brasil. Brasília: MS; 2013.. The municipalities are organized based on their specific administrative, political and geographic characteristics. They are part of a wider national network each with their own style of governance and it is for this reason it is necessary to have an evaluation methodology that recognizes the above²¹21. Gonçalves MJF. Avaliação de Programa de Saúde: O Programa Nacional de Controle de Tuberculose no Brasil. Saúde Transform Soc 2012; 3(1):13-17.,³²32. Braga JU. Vigilância epidemiológica e o sistema de informação da tuberculose no Brasil, 2001-2003. Rev Saude Publica 2007; 41(Supl. 1):77-87.,³³33. Gonçalves MJF, Penna MLF. Morbidade por tuberculose e desempenho do programa de controle em municípios brasileiros, 2001-2003. Rev Saude Publica 2007; 41(Supl. 1):95-102..

Spain has a parliamentary government under a constitutional monarchy and is organized as a unitary and decentralized state divided up into autonomous communities (CCAA). The structure of the health system lies with the Unitary State administration and is shared with the autonomous communities. These communities have their own independence to manage their own local health system¹⁴14. Centro Nacional de Epidemiología, Instituto de Salud Carlos III.Informe epidemiológico sobre la situación de la tuberculosis en España. Año 2012. Madrid: Centro Nacional de Epidemiología, Instituto de Salud Carlos III; 2013.. TB in Spain is dealt with at Unitary State level by a working group (GT) made up of specialists from each CCAA. Each Autonomous Community develops their own plan of activities. They organize Regional Programs or Committees to control TB in the best way possible based on the particular circumstances of an area¹⁴14. Centro Nacional de Epidemiología, Instituto de Salud Carlos III.Informe epidemiológico sobre la situación de la tuberculosis en España. Año 2012. Madrid: Centro Nacional de Epidemiología, Instituto de Salud Carlos III; 2013..

In Brazilian and Spanish health systems there is a notion of embedding monitoring and evaluation tools for routine service delivery which infers a management based on results approach. In Spain where the public health service covers a whole array of services, an evaluation through the use of appropriate indicators allows for the performance of health care interventions to gain visibility and legitimacy and this benefits the health care professionals. The professional can assess his/her own professional practice in an agile way and the health care managers will be able to assess the coverage and effectiveness of their activities. Government officials will be able to know what has been done, for what reason and at what cost³¹31. Villalbi JR, Casasa C, Bartoll X, Artazcoz L, Ballestín M, Borrell C, Camprubí E, Durán J, García R, Rodríguez P, Salamero, M. Indicadores para la gestión de los servicios de salud pública. Gac Sanit 2010; 24(5):378-384.. The identification of two protocols with directives for the evaluation of PCT among the identified material, shows the efforts that have been made to institutionalize evaluation processes and routines in the management of regional programs. This is also the case for article developed¹⁸18. Villalbi JR, Tresseras R. Evaluación de políticas y planes de salud. Gac Sanit 2011; 25(Supl. 1):17-24.,³¹31. Villalbi JR, Casasa C, Bartoll X, Artazcoz L, Ballestín M, Borrell C, Camprubí E, Durán J, García R, Rodríguez P, Salamero, M. Indicadores para la gestión de los servicios de salud pública. Gac Sanit 2010; 24(5):378-384. with the purpose of debating the use of indicators in the practice of program and service management. This emphasizes the need for information systems which can validate data and this data can be compared with the CCAAs.

In Brazil, in spite of the Health Ministry’s efforts there still exists a fragmentation of evaluative processes which stops their being utilized in coordinated actions. There also is a form of institutionalization focused on the creation of new bodies which makes the system even more bureaucratic than necessary⁴³43. Carvalho ALB, Souza MF, Shimizu HE, Senra IMVB, Oliveira KC. A gestão do SUS e as práticas de monitoramento e avaliação: possibilidades e desafios para a construção de uma agenda estratégica. Cien Saude Colet 2012; 17(4):901-911.. However what is set to be introduced is an agenda for monitoring and evaluation (M&A) in the policies on Brazilian health care. Advances were made with a discussion panel on indicators and a process to reach agreements on goals and propositions⁴⁴44. Lima LD, Queiroz LFN, Machado CV, Viana ALA. Descentralização e regionalização: dinâmica e condicionantes da implantação do Pacto pela Saúde no Brasil. Cien Saude Colet 2012; 17(7):1903-1914.

45. Tamaki EM, Tanaka OY, Felisberto E, Almeida CKA, Junior MD, Albuquerque LCB, Miranda ASD. Metodologia de construção de um painel de indicadores para o monitoramento e a avaliação da gestão do SUS. Cien Saude Colet 2012; 17(4):839-849.-⁴⁶46. Fontenelle LF. Mudanças recentes na Política Nacional de Atenção Básica: uma análise crítica. Rev Bras Med Fam Comunidade 2012; 7(22):5-9. that started in 2006 with the launch of the Health Agreement (Regulation 399/2006). As a result new measures have seen the introduction of innovative models for contracting services such as the Organizational Contract for Public Action in Health Care (COAP) that came about through a decree (7508/11). It proposes management based on results such as the National Program for Better Access, Quality and Attention in Health Care (PMAQAB). This was established under regulations (Regulations 1.654/2011). Another similar program called The Qualification Program for Actions on Monitoring Health (PQAVS) was set up under regulations (Regulation 1708/2013).

The National Program for the Control of Tuberculosis legislated for monitoring and evaluation activities since 2000, which is currently done by a team of professionals from the Health Ministry and by a group of external monitors made up of TB expert from all around the country. Their objective is to contribute in the form of policy and technical development to improve the state’s and districts’ capacity to control TB⁴²42. Brasil. Ministério da Saúde (MS). Secretaria de Vigilância em Saúde. Departamento de Vigilância de Doenças Transmissíveis.Experiências de Monitoramento e Avaliação no Controle da Tuberculose no Brasil. Brasília: MS; 2013..

In this study some relevant publications were not included due to: difficulties in obtaining some material where searches using different descriptors did not prove to be fruitful. The issue of knowing the correct descriptor to use in searches for bibliographic material is a major problem for complex themes. It also has the tendency to reveal sub-themes closely related in the area of health evaluation⁴4. Santos TMMG, Silva NS, Nogueira LT, Vilarinho LM, Nunes GBL. Tendências da produção científica sobre avaliação em saúde no Brasil.Rev. Rene 2010; 11(3):171-179.,⁵5. Fernandes FMB, Ribeiro JM, Moreira MG. Reflexões sobre avaliação de políticas de saúde no Brasil. Cad Saude Publica 2011; 27(9):1667-1677.. In order to improve the selection of relevant documents, the authors systemized search terms. Some articles were excluded from this body of work such as specific sub-themes concerning the control of TB (accessibility to treatment, late diagnosis) as they fell out of the scope of this study.

The area of program evaluation often does not follow conventions and therefore it is sometimes difficult to access information in this area. A large amount of evaluations for control programs is covered in technical reports whose publication and dissemination only occur in the specific health body. This is the same body that executes and coordinates control actions. There is not a tradition of systemizing the content of scientific articles.

Final Considerations

The results reflect the complexities involved in evaluating programs that consist of multiple activities and various parties. This highlights the need for a universal set of epidemiological and operational performance indicators for tuberculosis. There also needs to be the use of different concepts for interpreting data related to the performance of the PCTs. The study based on the given theme was made up by a number of different publications covering: program actions and service or activity attributes in specific populations. Yet there was found less information in this area compared with TB related themes such as clinical or scientific studies. We did, however, identify important debates around the management of services based on the use of evaluative indicators in both countries. Emphasis was placed on evaluative research in the Brazilian context and the debates concerning the use of indicators for the public health system in Spain.

The possibility of the use of the findings from this study in daily practice for health management would constitute an important contribution to the available scientific literature. For example, this could be through the creation of indicators and goals and a monitoring/evaluating system for TB. Finally we suggest the development of comparative studies on similar universal access health systems as in Brazil and Spain in order to deepen our understanding in this area.

References

Publication Dates

History