Abstract

This bibliographic study aims to analyze evaluation studies of primary health care (PHC) in Brazil, focusing on the methodological design adopted and some key evaluation features. We searched the Scientific Electronic Library Online, the Scientific Journals Portal of the Coordination for the Improvement of Higher Education Personnel (CAPES), the Latin American and Caribbean Center on Health Sciences Information (LILACS) database and the Medical Literature Analysis and Retrieval System Online (MEDLINE), using a structured vocabulary search and selecting articles published between 2007 and 2017 that addressed the evaluation of Brazilian PHC services. Forty-one articles were selected by analyzing the following characteristics, among others: year of publication, article modality, PHC service investigated, methodological design, evaluation characteristics and evaluation outcome and potential. Of these 41 studies, the majority (86.8%) originated in field research conducted exclusively in Family Health Units (48.9%). Methodologically, most studies were quantitative; and the Primary Care Assessment Tool was the most used instrument. We also found that Brazilian studies on PHC evaluation reflected the national historical-political structuring of PHC, and for the most part, they reported quality evaluation research. Our review presents the national panorama on PHC evaluation, highlighting the field’s conceptual and practical pluralism, but also its limitations and challenges.

Keywords:
Primary Health Care; Health Care Evaluation; Assessment of Health Research

Introduction

Primary health care (PHC) is the entry level of the health system for addressing the problems and needs of individuals, providing access to a set of actions that go beyond the limits of the clinic, including activities of coordination and integration of the care provided by other services and units of the health system (Starfield, 2002STARFIELD, B. Atenção primária: equilíbrio entre necessidades de saúde, serviços e tecnologia. Brasília, DF: Unesco, 2002.). PHC is organized in a variety of formats throughout the world, according to the political, economic, social and cultural differences of each context (Giovanella, 2006GIOVANELLA, L. A atenção primária à saúde nos países da União Europeia: configurações e reformas organizacionais na década de 1990. Cadernos de Saúde Pública , Rio de Janeiro, v. 22, n. 5, p. 951-963, 2006.; Giovanella; Mendonça, 2012GIOVANELLA, L.; MENDONÇA, M. H. M. Atenção primária à saúde. In: GIOVANELLA, L. et al. (Org.). Políticas e sistema de saúde no Brasil. 2. ed. Rio de Janeiro: Ed. Fiocruz , 2012. p. 493-546.).

The first PHC initiatives under the Brazilian National Health System (SUS) occurred in 1991, with the implementation of the Community Health Agents Program (Fausto and Matta, 2007FAUSTO, M. C. R.; MATTA, G. C. Atenção primária à saúde: histórico e perspectivas. In: MOROSINI, M. V. G. C.; CORBO, A. D. (Org.). Modelos de atenção e a saúde da família. Rio de Janeiro: Ed. Fiocruz , 2007. p. 47-67.). Although this program was launched as a supplementary, non-mandatory modality of PHC, the positive impact of its actions on health conditions of the assisted population, coupled with the need to organize a healthcare model with an emphasis on primary care, motivated the expansion of this experience, which materialized with the implementation of the Family Health Program (FHP) in 1994. The FHP soon gained relevance in the national scenario, taking since 1997 the proportions of a strategy for overhauling the care model then in force and receiving the name of Family Health Strategy (FHS) (Giovanella and Mendonça, 2012GIOVANELLA, L.; MENDONÇA, M. H. M. Atenção primária à saúde. In: GIOVANELLA, L. et al. (Org.). Políticas e sistema de saúde no Brasil. 2. ed. Rio de Janeiro: Ed. Fiocruz , 2012. p. 493-546.).

In this context, PHC functions as a strategy for enforcing citizens’ right to healthcare and for strengthening SUS principles, aiming to expand the access to health services, consolidate the process of decentralization of health care provision, facilitate the process of regionalization agreed between neighboring municipalities and coordinate the integrality of the care (Andrade; Bueno; Bezerra, 2012ANDRADE, L. O. M.; BUENO, I. C. H. C.; BEZERRA, R. C. Atenção primária à saúde e Estratégia Saúde da Família. In: CAMPOS, G. W. S. et al. (Org.). Tratado de saúde coletiva. São Paulo: Hucitec, 2012. p. 783-835.). Its central position in the structure and implementation of the health care network, among other aspects, helps the SUS to go beyond mere service provision and advance towards intersectoral cooperation, understood as an attempt to overcome the fragmentation of public policies by articulating different sectors (linked or not to health) for the benefit of human development (Opas, 2007OPAS - ORGANIZAÇÃO PAN-AMERICANA DE SAÚDE. Renovação da atenção primária em saúde nas Américas: documento de posicionamento da Organização Pan-Americana da Saúde/Organização Mundial da Saúde (OPAS/OMS). Washington, DC, 2007.).

PHC’s recognized potential for reformulating the health system, coupled with its marked expansion since 1994, has placed at the center of the discussions issues related to the efficiency, resilience and capacity to integrate its services with other levels of care (Campos, 2005CAMPOS, C. E. A. Estratégias de avaliação e melhoria contínua da qualidade no contexto da atenção primária à saúde. Revista Brasileira de Saúde Materno Infantil, Recife, v. 5, p. s63-s69, 2005. Suplemento 1.). In response to these demands, the Ministry of Health (MH), in partnership with other institutions, has been promoting a set of strategies for the monitoring and evaluation of PHC services, aiming to expand access and improve quality, as well as developing in these spaces an institutional culture of monitoring and management (Brazil, 2015BRASIL. Ministério da Saúde. Secretaria de Atenção à Saúde. Departamento de Atenção Básica. Programa Nacional de Melhoria do Acesso e da Qualidade da Atenção Básica (PMAQ): manual instrutivo 3º Ciclo (2015-2016). Brasília, DF, 2015.).

Quality evaluation within the framework of the PHC is not restricted to a ministerial prerogative; it has been a relevant topic in the last decades, promoting in academic institutions and health services the development and implementation of a diversity of principles, techniques and instruments (Pinto Júnior et al. al., 2015PINTO JÚNIOR, E. P. et al. Análise da produção científica sobre avaliação, no contexto da saúde da família, em periódicos brasileiros. Saúde em Debate , Rio de Janeiro, v. 39, n. 104, p. 268-278, 2015.).

PHC evaluation is a task that requires the efforts and participation of various institutions and professionals, given its magnitude, complexity and heterogeneity. Thus, evaluating the various aspects of these services and their orientation allows a rigorous production of knowledge concerning their effectiveness and diversity (Brazil, 2010BRASIL. Ministério da Saúde. Secretaria de Atenção à Saúde. Departamento de Atenção Básica. Manual do instrumento de avaliação da atenção primária à saúde: Primary Care Assessment Tool - PCATool. Brasília, DF, 2010. Disponível em: <Disponível em: https://bit.ly/2qB9szE >. Acesso em: 5 fev. 2018.
https://bit.ly/2qB9szE... ).

The scientific field of quality assessment has been expanding and becoming more sophisticated, developing a multiplicity of conceptual models and approaches with different methodologies and occupying a central position in institutions and public policies (Figueiró; Frias; Navarro, 2010FIGUEIRÓ, A. C.; FRIAS, P. G.; NAVARRO, L. M. Avaliação em saúde: conceitos básicos para a prática nas instituições. In: SAMICO, I. et al. Avaliação em saúde: bases conceituais e operacionais. Rio de Janeiro: MedBook , 2010. p. 1-13.).

In the health sector, quality evaluation stands out as one of the best tools for responding to the demands of management, which must justify its decisions for an increasingly demanding clientele and comply with the precepts of achieving universal coverage, expanding access and improving quality amid epidemiological changes and a public finance crisis, which increasingly call for an adequate resource management (Champagne et al., 2016CHAMPAGNE, F. et al. A avaliação no campo da saúde: conceitos e métodos. In: BROUSSELLE, A. et al. Avaliação: conceitos e métodos. Rio de Janeiro: Ed. Fiocruz , 2016. p. 41-60.). Although evaluation has a recognized potential to promote improvements in health systems and has been ever more adopted by them, its practice in Brazil is still considered incipient. This is due, among other factors, to the lack of a specific professional field for evaluators, as well as to the challenges of implementing evaluation methods in contexts other than those for which they were developed (Nemes, 2001NEMES, M. I. B. Avaliação em saúde: questões para o programa DST/aids no Brasil. Rio de Janeiro: Associação Brasileira Interdisciplinar de Aids, 2001.; Novaes, 2000NOVAES, H. M. D. Avaliação de programas, serviços e tecnologias em saúde. Revista de Saúde Pública, São Paulo, v. 34, n. 5, p. 547-549, 2000.; Tanaka, 2017TANAKA, O. Y. Avaliação em saúde: novos tempos, novas construções. In: TANAKA, O. Y.; RIBEIRO, E. L.; ALMEIDA, C. A. L. Avaliação em saúde: contribuições para incorporação no cotidiano. Rio de Janeiro: Atheneu , 2017. p. 1-9.). If on the one hand this situation points to a path full of uncertainties and insecurities, since it may require the adaptation and re-signification of knowledge in the face of diverse realities, on the other hand, it opens a field full of possibilities, in which there are no universal rules (Cazarin, 1998).

Evaluation can be interdisciplinary and democratic and, in order to be carried out satisfactorily, these principles should be clearly apprehended. The success of such assessments should also be based on targeting collective interest interventions with well-defined objectives related to the evaluation’s purpose, gathering the greatest amount of information on the phenomenon, listening to the different stakeholders and developing an objective work plan, ensuring methodological rigor and a consistent theoretical-practical apparatus (Contandriopoulos, 2016CONTANDRIOPOULOS, A. P. Avaliar a avaliação. In: BROUSSELLE, A. et al. Avaliação: conceitos e métodos. Rio de Janeiro: Ed. Fiocruz , 2016. p. 264-272.; Tanaka, 2017TANAKA, O. Y. Avaliação em saúde: novos tempos, novas construções. In: TANAKA, O. Y.; RIBEIRO, E. L.; ALMEIDA, C. A. L. Avaliação em saúde: contribuições para incorporação no cotidiano. Rio de Janeiro: Atheneu , 2017. p. 1-9.).

Considering these principles, PHC’s relevance to the reorientation of health care, the fact that it has been the focus of various evaluations and the fact that scientific production represents the social practices of a given epoch, this review aims to analyze how PHC evaluation is done in Brazil, emphasizing mainly the methodological design adopted and the key evaluation parameters.

Method

This study is an integrative literature review because it allows for the synthesis of existing knowledge and the incorporation of results of significant studies into practices (Souza; Silva; Carvalho, 2010SOUZA, M. T.; SILVA, M. D.; CARVALHO, R. Revisão integrativa: o que é e como fazer? Einstein, São Paulo, v. 8, n. 1, p. 102-106, 2010.).

The survey of the publications to build the empirical basis of analysis occurred during the months of January and February of 2018, covering the period from 2007 to 2017. This period was chosen to enable the analysis of the scientific production in different stages of PHC’s implementation process, especially since the establishment of the National Primary Health Care Policy by Ministerial Ordinance (MO) No. 646, dated March 28, 2006, revised by MO No. 2,488, of October 21, 2011, and the launching of the National Program for Access and Quality Improvement in Primary Care (PMAQ-AB) in 2011.

The following databases were surveyed: the Scientific Electronic Library Online, searching the abstracts; the Scientific Journals Portal of the Coordination for the Improvement of Higher Education Personnel (CAPES), searching the subjects; the Latin American and Caribbean Center on Health Sciences Information (LILACS) database, searching the subjects; and the Medical Literature Analysis and Retrieval System Online (MEDLINE), searching MeSH terms. For the survey, we used a combination of structured vocabularies for indexing - descriptors in health sciences and medical subject headings. The expression “primary health care” was cross-referenced with the descriptors “health services evaluation,” “health programs and projects evaluation,” “health evaluation” and “health care quality, access and evaluation,” using the AND operator. This procedure identified 1143 papers.

We conducted a preliminary selection of the studies, which eliminated duplicates, and excluded results based on criteria that did not require reading the abstracts - i.e. searched for full-text and indexed articles in Portuguese, English or Spanish. We opted for selecting only articles because they present content originating from research reports, theses and dissertations but usually have a wider dissemination; besides consisting of works submitted to rigorous peer review. This preliminary selection led to the exclusion of 1,007 studies.

The second selection step involved reading the abstracts selected in the previous stage (74) and, next, an assessment of the full texts. Two researchers assessed the full texts independently, being selected only original studies, regardless of their nature (field or secondary research). The papers should present empirical evaluation results related to primary care services in Brazil. In this stage were excluded articles that: (1) addressed the development and validation of evaluation instruments and methodologies in primary health care; (2) focused on evaluating a single-purpose PHC program, such as those for tuberculosis, leprosy, prenatal care, etc; (3) dealt exclusively with subjective/experiential aspects; or (4) addressed clinical and epidemiological indicators of specific populations.

Although not included in the list of eligible studies, we also surveyed the bibliography cited by the review articles, applying to them the same selection criteria adopted for the databases.

At the end, 41 articles were selected for this review, of which 35 were obtained through database search and six through the additional search (manual) in the bibliographic references. The entire selection process is shown in the following flowchart (Figure 1).

The two researchers did a thorough reading of the 41 papers selected and transcribed relevant content into a separate form. Data analysis included the publication year, type of the article, region where the research was carried out, source of data or type of PHC services investigated, methodological design and characteristics evaluated.

The variables of the methodological design component were the type of study and approach, the instruments for data collection, the characteristics of the study’s sample/subjects and the evaluated aspects.

The characteristics evaluated were the evaluative question; the type of evaluation - according to subject, characteristics and applicability; the outcome and potential of the studies and the evaluation’s theoretical basis.

The evaluative questions were categorized as “clear” - when identified in the text; “implicit” - when, although not presented in an interrogative format, they were expressed in the objectives and/or the justification; and “non-existent” - when, even indirectly, they could not be clearly identified.

Regarding the nature of the evaluations, the specialized literature on the theme is diversified conceptually and methodologically; however, we chose to use the typology proposed by Novaes (2000NOVAES, H. M. D. Avaliação de programas, serviços e tecnologias em saúde. Revista de Saúde Pública, São Paulo, v. 34, n. 5, p. 547-549, 2000.) for health assessments, also because we found no other recent work that dealt with the subject with the approach and the scope adopted by this author. In this perspective, with regard to the characteristics and applicability of the evaluations, we classified the studies in the following categories: evaluation for decision making, evaluation for management and evaluation research. Regarding the unit of analysis (subject), we classified the articles as relating to: technological evaluation; program evaluation; and evaluation for quality assurance in health care.

The studies’ outcome and potential for application were classified in three categories: “diagnosis,” when the study presents a detailed description of the results of the evaluation process; “proposal for improvement,” when, based on the findings, the authors suggest improvement or maintenance of aspects of the evaluated phenomenon; and “theoretical discussion and advancement,” when the articles extrapolated the mere comparison of characteristics, contributing with theoretical-conceptual discussions.

Results and discussion

Of the 41 studies included in this review (Chart 1), 36 (87.8%) were based on field research and four (9.8%) on secondary data analysis, mostly from public domain databases related to PHC evaluation, such as the Project of Expansion and Consolidation of the Family Health Program (Proesf), the PMAQ-AB and the Evaluation for the Family Health Strategy’s Quality Improvement (EQI). Only one study (2.4%) reported using both field and secondary research as a means of obtaining data.

Regarding the type of PHC services surveyed: 18 (48.6%) of the 37 field studies adopted FHS units as the locus of research; 12 (32.4%) investigated FHS units and Basic Health Units (BHUs); two (5.4%) were conducted exclusively in UBSs; one (2.7%) researched ESF units, BHUs and other PHC units of mixed nature; three (8.1%) did not mention the type of service; and only one article (2.7%) studied emergency units (EUs). In the latter, PHC quality was assessed based on repressed demand, defined by the author as frequent users of PHC services who, on that occasion, sought an EU for non-urgent care.

By counting the number of services investigated (and specified), we verified that approximately 1,324 Family Health Units (FHUs), 255 BHUs, 19 mixed-service PHC units and five EUs were evaluated.

FHUs were the study site for most evaluations, reflecting the numerical, political and social relevance of these services in the context of Brazilian PHC.

FHS is the structuring foundation of the health system reorientation towards a more comprehensive primary care provision (Fertonani et al., 2015FERTONANI, H. P. et al. Modelo assistencial em saúde: conceitos e desafios para a atenção básica brasileira. Ciência & Saúde Coletiva, Rio de Janeiro, v. 20, n. 6, p. 1869-1878, 2015.). Created in the mid-1990s, this strategy was not only responsible for triggering philosophical and structural changes in health care, but also for significantly expanding PHC services, starting with 8,503 Family Health teams and a population coverage of 9.2% in 2000 to more than 42,000 teams and a 66.3% coverage in 2017 (Brazil, 1997BRASIL. Ministério da Saúde. Secretaria de Assistência à Saúde. Coordenação de Saúde da Comunidade. Saúde da Família: uma estratégia para a reorientação do modelo assistencial. Brasília, DF, 1997.).

It is in this scenario of new challenges imposed by the expressive quantitative expansion of the FHS that the Department of Primary Care of the MH have launched strategic processes, aimed at promoting PHC monitoring and evaluation, such as conducting evaluation studies and research (Brazil, 2005BRASIL. Ministério da Saúde. Secretaria de Atenção à Saúde. Departamento de Atenção Básica. Coordenação de Acompanhamento e Avaliação. Avaliação na atenção básica em saúde: caminhos da institucionalização. Brasília, DF, 2005.). According to Almeida and Giovanella (2008ALMEIDA, P. F.; GIOVANELLA, L. Avaliação em atenção básica à saúde no Brasil: mapeamento e análise das pesquisas realizadas e/ou financiadas pelo Ministério da Saúde entre os anos de 2000 e 2006. Cadernos de Saúde Pública, Rio de Janeiro, v. 24, n. 8, p. 1727-1742, 2008.), the Federal Government promoted many researches in this field, starting between 2000 and 2002 and focusing on the process of implementation and monitoring of the program. As of 2004, these researches focused more on evaluating services by assessing the fundamental and qualifying attributes of PHC. These studies were conducted mainly by academic institutions in the Southeast region, a circumstance that may help to understand some of the findings of this review, which showed that most of the evaluations occurred in this region.

Eighteen (48.6%) of the field studies were conducted in the Southeast region (mainly in the states of Minas Gerais and São Paulo, with 16 studies). Nine studies (24.3%) were carried out in the Northeast region; four in the South (10.8%); and three in each of the North and Central West regions (8.1%).

Besides the involvement of Southeastern universities in the development of evaluation researches promoted by the Federal Government, that it concentrates the largest number of master’s and doctoral programs may also have contributed to this region prominence, since they usually produce most of the science in Brazil (Cirani; Campanario; Silva, 2015CIRANI, C. B. S.; CAMPANARIO, M. A.; SILVA, H. H. M. A evolução do ensino da pós-graduação senso estrito no Brasil: análise exploratória e proposições para pesquisa. Avaliação, Sorocaba, v. 20, n. 1, p. 163-187, 2015.). The second largest concentration, in the Northeast, also pointed out by Lentsck, Kluthcovsky and Kluthcovsky (2010LENTSCK, M. H.; KLUTHCOVSKY, A. C. G. C.; KLUTHCOVSKY, F. A. Avaliação do Programa Saúde da Família: uma revisão. Ciência & Saúde Coletiva , Rio de Janeiro, v. 15, p. 3455-3466, 2010. Suplemento 3.), can be explained both because this region pioneered the implementation of Family Health teams and because, as in the Southeast, it produces research of great prestige in the area of evaluation. The small amount of research of this nature in the North and Central-West regions is noteworthy, which shows that the evaluation in primary health care has occurred unequally in the country.

In view of the relevance of PHC to the reorientation of health practices and the heterogeneity of the Brazilian healthcare context, studies involving PHC in different regions and municipalities, of varying sizes, are valuable, especially in the field of evaluation - since, although not producing immediate decisions, they can positively influence decision making and assist in better strategic planning (Bousquat et al., 2017BOUSQUAT, A. E. M. et al. Avaliação da atenção primária à saúde. In: TANAKA, O. Y.; RIBEIRO, E. L.; ALMEIDA, C. A. L. Avaliação em saúde: contribuições para incorporação no cotidiano. Rio de Janeiro: Atheneu, 2017. p. 101-113.).

Most of the publications (58.5%) occurred in the last four years under investigation (2014 to 2017). Regarding the methodology of field studies, reported by the authors, 86.5% of the studies adopted a quantitative approach, 8.1% were both quantitative and qualitative and 5.4% were qualitative research (Table 1). As to the methodological design, 32.4% of the articles were classified as transversal, involving various research approaches, such as exploratory, descriptive, analytical, evaluative and observational. In addition, 29.7% were exclusively transversal, 16.2% were evaluative, 13.5% were descriptive and 2.7% documentary. Two studies (5.4%) did not describe the methodological designs that guided them.

The observed predominance of exclusively quantitative approaches corroborates the results found in a study conducted by Pinto Júnior et al. (2015PINTO JÚNIOR, E. P. et al. Análise da produção científica sobre avaliação, no contexto da saúde da família, em periódicos brasileiros. Saúde em Debate , Rio de Janeiro, v. 39, n. 104, p. 268-278, 2015.). These approaches seek to investigate the phenomena and their elements using mathematics. The selection of subjects in this modality uses random sampling, and the analysis applies statistical methods. Although relevant due to their objectivity, when applied in isolation to health evaluation their scope may be insufficient for the desired understanding, generating only partial answers to evaluative questions (Tanaka, 2017TANAKA, O. Y. Avaliação em saúde: novos tempos, novas construções. In: TANAKA, O. Y.; RIBEIRO, E. L.; ALMEIDA, C. A. L. Avaliação em saúde: contribuições para incorporação no cotidiano. Rio de Janeiro: Atheneu , 2017. p. 1-9.).

The diversity of concepts and methods characteristic of health evaluation is above all an attempt to respond to the heterogeneity and complexity of actions in this area (Samico; Figueiró; Frias, 2010SAMICO, I.; FIGUEIRÓ, A. C.; FRIAS, P. G. Abordagens metodológicas na avaliação em saúde. In: SAMICO, I. et al. Avaliação em saúde: bases conceituais e operacionais. Rio de Janeiro: MedBook , 2010. p. 15-28.). Thus, it is essential that the evaluator aim for transversality, plurality and flexibility also when choosing a methodological approach. In order to do so, evaluation approaches should overcome the dichotomy between quantitative and qualitative methods, applying them in an articulated way, going beyond the mere counting or identification of related factors and advancing towards the understanding of behaviors, perceptions and attitudes of the subjects in a given social context (Pinto Júnior et al., 2015PINTO JÚNIOR, E. P. et al. Análise da produção científica sobre avaliação, no contexto da saúde da família, em periódicos brasileiros. Saúde em Debate , Rio de Janeiro, v. 39, n. 104, p. 268-278, 2015.; Samico; Figueiró; Frias, 2010SAMICO, I.; FIGUEIRÓ, A. C.; FRIAS, P. G. Abordagens metodológicas na avaliação em saúde. In: SAMICO, I. et al. Avaliação em saúde: bases conceituais e operacionais. Rio de Janeiro: MedBook , 2010. p. 15-28.).

The quantitative-qualitative approaches are capable of providing more extensive analyzes, involving aspects of structure, process and result, as well as enabling a better understanding of the relationships and the actors involved (Minayo; Assis; 2005MINAYO, M. C. S.; ASSIS, S. G.; SOUZA, E. R. (Org.). Avaliação por triangulação de métodos: abordagem de programas sociais. Rio de Janeiro: Fiocruz, 2005.). However, this research dynamic should rest on the concept of complementarity; and the researcher should have the maturity to explore their potentials without disregarding their peculiarities, in order to avoid a mere juxtaposition of methods and techniques (Hartz, 1999HARTZ, Z. M. A. Avaliação dos programas de saúde: perspectivas teórico-metodológicas e políticas institucionais. Ciência & Saúde Coletiva , Rio de Janeiro, v. 4, n. 2, p. 341-353, 1999.).

Regarding the instrument for data collection, most of the studies (64.9%) used the Primary Care Assessment Tool (PCATool), of which 17 (70.8%) applied only one of the three existing versions (adult users, children users and professional) and seven (29.2%) applied two or more versions. We also identified other validated instruments among the studies: an article used the EQI’s self-assessment questionnaire and another one used the European Task Force on Patient Evaluation of General Practice Care (Europep) questionnaire. Studies that aimed to conduct the evaluation through the application of instruments built by the authors themselves represented 29.7% of the studies.

PCATool has been described in the literature as a valid and reliable instrument to assess the quality of PHC in different services and perspectives, and is suggested as the preferred tool to verify the degree of compliance with PHC principles (Fracolli et al., 2014FRACOLLI, L. A. et al. Instrumentos de avaliação da atenção primária à saúde: revisão de literatura e metassíntese. Ciência & Saúde Coletiva , Rio de Janeiro, v. 19, n. 12, p. 4851-4860, 2014.; Hauser et al., 2013HAUSER, L. et al. Tradução, adaptação, validade e medidas de fidedignidade do Instrumento de Avaliação da Atenção Primária à Saúde (PCATool) no Brasil: versão profissionais de saúde. Revista Brasileira de Medicina de Família e Comunidade, Rio de Janeiro, v. 8, n. 29, p. 244-255, 2013. Disponível em: <Disponível em: https://bit.ly/2CFtqhd >. Acesso em: 29 jun. 2018.
https://bit.ly/2CFtqhd... ; Landsberg; Souza Neto; Souza, 2010LANDSBERG, G. A. P.; SOUZA NETO, A. I.; SOUZA, R. A. Avaliando os atributos da atenção primária: uma revisão. Avaliação de Políticas Públicas, Fortaleza, v. 2, n. 6, p. 85-93, 2010.). Besides being doubly validated (Harzheim et al., 2006HARZHEIM, E. et al. Quality and effectiveness of different approaches to primary care delivery in Brazil. BMC Health Services Research, London, v. 6, n. 1, p. 156, 2006. Disponível em: <Disponível em: https://bit.ly/2CJ1Kb2 >. Acesso em: 10 maio 2018.
https://bit.ly/2CJ1Kb2... ; Macinko; Almeida; Oliveira, 2003MACINKO, J.; ALMEIDA, C.; OLIVEIRA, E. Avaliação das características organizacionais dos serviços de atenção básica em Petrópolis: teste de uma metodologia. Saúde em Debate, Rio de Janeiro, v. 27, n. 65, p. 243-256, 2003.) and adopted by the Federal Government, the wide use of PCATool in Brazil may be related to the agile format of the questions; it has versions for different health care users and professionals and, due to its international relevance, it has been translated and adapted to countries with different health systems (Bousquat et al., 2017BOUSQUAT, A. E. M. et al. Avaliação da atenção primária à saúde. In: TANAKA, O. Y.; RIBEIRO, E. L.; ALMEIDA, C. A. L. Avaliação em saúde: contribuições para incorporação no cotidiano. Rio de Janeiro: Atheneu, 2017. p. 101-113.).

It is noteworthy that almost a third of the field studies used non-validated instruments. This may constitute a risk for health care evaluation, since the lack of information on the psychometric properties of an instrument makes it hard to assess the accuracy and precision of the measurement of the phenomenon evaluated (Hauser et al., 2013HAUSER, L. et al. Tradução, adaptação, validade e medidas de fidedignidade do Instrumento de Avaliação da Atenção Primária à Saúde (PCATool) no Brasil: versão profissionais de saúde. Revista Brasileira de Medicina de Família e Comunidade, Rio de Janeiro, v. 8, n. 29, p. 244-255, 2013. Disponível em: <Disponível em: https://bit.ly/2CFtqhd >. Acesso em: 29 jun. 2018.
https://bit.ly/2CFtqhd... ). Furthermore, besides preventing biases, the adoption of validated instruments allows greater scientific rigor in the evaluation and the comparison of results among different national and/or international realities.

Regarding the universe of study, approximately 19,847 individuals participated in the investigations, with samples varying between 20 and 1,297 individuals. In almost 73% of the studies, the sample comprised only one category of persons related to the health services evaluated: adult users, caregivers of children, professionals or managers. Among the others, 21.6% involved users, professionals and/or managers together and 5.4% involved adult users and caregivers of children.

The strong academic pressure to publish, together with the restricted availability of spaces for the dissemination of results in journals, contributes to the increasingly narrow focus of academic research, explaining in part the large number of articles discussing the same category of people related to the evaluated phenomena. However, a consistent evaluation process requires a thorough examination of indicators and approaches in order to address the various aspects of the phenomenon investigated. Thus, evaluations that take into account only a single point of view (as observed in the studies analyzed), without discussing those of other individuals involved, limit the transforming potential of their results insofar as they generate unilateral information that may lead to a superficial or misguided comprehension of the reality (Champagne; Contandriopoulos; Tanon, 2016CHAMPAGNE, F.; CONTANDRIOPOULOS, A. P.; TANON, A. Utilizar a avaliação. In: BROUSSELLE, A. et al. Avaliação: conceitos e métodos. Rio de Janeiro: Ed. Fiocruz, 2016. p. 242-261.).

We also examined the studies’ proposed aims, which we present here according to their three types of information sources: secondary data, primary data using PCATool and primary data using other instruments.

The four studies using secondary data aimed to assess the physical structure, primary care outcomes and quality of care by applying PHC ministerial evaluation programs.

The works that applied PCATool (24) aimed to evaluate organizational characteristics, services outcomes and the existence and expansion of PHC attributes. Among them, 70.8% analyzed all the essential and derivative attributes, 8.4% only derived attributes and 20.8% analyzed some PHC attributes but not the entire list of essential or derivative attributes.

Among the field studies that used other assessment instruments other than PCATool (13), the majority aimed to evaluate aspects related to the structure and process components of PHC services, such as access, geographic accessibility, physical structure, materials, human resources, professional qualification, receptivity and satisfaction. Other studies analyzed the implementation of new modalities of PHC services, such as the FHS, by assessing PHC promotion and consolidation initiatives in the country linked to the MH.

The evaluation characteristics identified in the works were also used in the analysis and are summarized in Table 2.

The evaluation question posed to respondents is an essential element in an evaluation, but of the 41 studies examined, only 7.3% clearly reproduced it - 68.3% expressed it implicitly and in 24.4% of the studies it could not be identified even indirectly, therefore being considered non-existent.

The evaluation question gives clarity and precision to the assessment process insofar as it helps to determine the aspects of the intervention to be observed in depth.

The formulation of good evaluation questions contributes to the development of more assertive and relevant research. However, lack of rigor or failure to observe this important step can jeopardize the consistency and purpose of the evaluation, leading to simplistic, difficult to justify conclusions, as well as generating products with credibility compromised by not representing all stakeholders (Cazarin; Mendes; Albuquerque, 2010CAZARIN, G.; MENDES, M. F. M.; ALBUQUERQUE, K. M. Perguntas avaliativas. In: SAMICO, I. et al. Avaliação em saúde: bases conceituais e operacionais. Rio de Janeiro: MedBook, 2010. p. 79-87.).

Regarding the evaluation characteristics and applicability, according to the categorization proposed by Novaes (2000NOVAES, H. M. D. Avaliação de programas, serviços e tecnologias em saúde. Revista de Saúde Pública, São Paulo, v. 34, n. 5, p. 547-549, 2000.), we verified that most of the studies (82.9%) were characterized as evaluation research, 12.2% as evaluation for management and 2.4% as evaluation for decision making.

“Evaluation research” has as its main goal the production of knowledge with scientific validity and potential to support or oppose the implementation or reorientation of policies and practices. Such evaluations seek to establish links between an action and certain modifications of the reality observed, and are often developed under the coordination of academic institutions, either on their own initiative or at the request of public entities (Novaes, 2000NOVAES, H. M. D. Avaliação de programas, serviços e tecnologias em saúde. Revista de Saúde Pública, São Paulo, v. 34, n. 5, p. 547-549, 2000.). The evaluation for management purposes focus on information production, usually generating measures that allow quantification and replication. The intention is to improve the investigated attribute, without, however, investigating causes or proposing a reorientation, aiming instead to ameliorate conditions in a given conjuncture. On the other hand, the evaluation for decision making has as its central goal to improve decision making processes, helping in the solution of problems raised by those directly linked to the evaluated phenomenon. The focus of this modality is to gain in-depth knowledge of an intervention to identify and understand the challenges involved, as well as proposing solutions to them (Novaes, 2000NOVAES, H. M. D. Avaliação de programas, serviços e tecnologias em saúde. Revista de Saúde Pública, São Paulo, v. 34, n. 5, p. 547-549, 2000.).

Knowledge about evaluation typologies is essential for developing adequate evaluation plans, since it helps the professional to select or coordinate procedures, reducing the risk of employing contradictory combinations. However, these modalities do not have an intrinsic value - that is, there are no better or worse ones, but rather those more or less adequate to reach the goals of a given evaluation (Samico; Figueiró; Frias, 2010SAMICO, I.; FIGUEIRÓ, A. C.; FRIAS, P. G. Abordagens metodológicas na avaliação em saúde. In: SAMICO, I. et al. Avaliação em saúde: bases conceituais e operacionais. Rio de Janeiro: MedBook , 2010. p. 15-28.).

Regarding the categorization of the evaluations according to the unit of analysis (subject), we classified 82.9% of the studies as evaluations aimed at ensuring quality in health care. These evaluations are distributed throughout the period defined for this bibliographic survey, but they were more common among studies published after 2013 and in those using PCATool and secondary data. Studies classified as program evaluation accounted for 12.2% of the total. These were more frequent among the texts published until 2012 and those using data collection instruments developed by the authors themselves. Only 2.4% of the works were classified as technological/economic evaluation; and for 2.4% it was not possible to identify the typology used.

The studies seem to reflect the different moments of PHC structuring in Brazil, one of which was the expressive quantitative growth of the FHS (starting in 2000). This scenario of great expansion of services initially led to studies aiming to evaluate the implementation, feasibility and impact (some of the constituent stages of program evaluation) of this new modality in different Brazilian contexts (Almeida; Macinko, 2006ALMEIDA, C.; MACINKO, J. Validação de uma metodologia de avaliação rápida das características organizacionais e do desempenho dos serviços de atenção básica do Sistema Único de Saúde (SUS) em nível local. Brasília, DF: Ministério da Saúde, 2006.; Novaes, 2000NOVAES, H. M. D. Avaliação de programas, serviços e tecnologias em saúde. Revista de Saúde Pública, São Paulo, v. 34, n. 5, p. 547-549, 2000.). Starting in 2011, a period marked by a review of the National Primary Care Policy (PNAB) and the launch of the PMAQ-AB, efforts focused on strengthening and expanding access to and improving the quality of PHC services, seemingly contributing to the increase in researches aimed at measuring the quality of this level of attention (Giovanella; Mendonça, 2012GIOVANELLA, L.; MENDONÇA, M. H. M. Atenção primária à saúde. In: GIOVANELLA, L. et al. (Org.). Políticas e sistema de saúde no Brasil. 2. ed. Rio de Janeiro: Ed. Fiocruz , 2012. p. 493-546.).

Program evaluations are a systematic collection of information about actions, specificities, services, procedures or results of the programs, in order to increase their effectiveness and enable a better decision making regarding new proposals (Champagne et al. 2016CHAMPAGNE, F. et al. A avaliação no campo da saúde: conceitos e métodos. In: BROUSSELLE, A. et al. Avaliação: conceitos e métodos. Rio de Janeiro: Ed. Fiocruz , 2016. p. 41-60.). These programs are conceived as complex processes of organization of actions, created to achieve more comprehensive goals, such as the implementation of health care modalities involving various institutions, services and professionals (Novaes, 2000NOVAES, H. M. D. Avaliação de programas, serviços e tecnologias em saúde. Revista de Saúde Pública, São Paulo, v. 34, n. 5, p. 547-549, 2000.).

On the other hand, service evaluations assess aspects related to technical performance, directly linked to effective care, in order to maximize benefits, reduce risks and improve users’ experiences with the service through an interpersonal relationship based on empathy, sensitivity, reliability and meeting users’ needs (Donabedian, 1978DONABEDIAN, A. The quality of medical care. Science, Washington, DC, v. 200, n. 4344, p. 856-864, 1978.; Harzheim et al., 2018HARZHEIM, E. et al. Estudos de PCATool no Brasil. In: MENDONÇA, M. H. M. et al. Atenção primária à saúde no Brasil: conceitos, práticas e pesquisa. Rio de Janeiro: Ed. Fiocruz , 2018. p. 493-525.). In services, especially in primary care, quality evaluation has played an important role. In this context, evaluations aim to raise awareness about the particularities of the population’s health problems, as well as improving not only the sector’s response to diagnostic needs and its management of diseases, but also preventive actions and general health conditions. Quality, in its broadest sense, also involves users’ satisfaction with the service, costs, professional qualification, health facilities’ safety and pleasant appearance and the adequacy of equipment used in service provision (Starfield, 2002STARFIELD, B. Atenção primária: equilíbrio entre necessidades de saúde, serviços e tecnologia. Brasília, DF: Unesco, 2002.).

In order to select, apply and exploit the full potential that each evaluative modality can offer, the evaluator should rely on a consistent theoretical support. Considering this, we sought to identify the studies’ theoretical references. We found that 16 (39%) original works mentioned the theoretical basis of the evaluation conducted; 22 (53.7%) of them based their research on evaluative methodologies applied to primary health care, without, however, adhering to any specific theory; finally, in three (7.3%) of the studies, we could not identify, even indirectly, the theoretical foundations that supported the evaluation.

The evaluation is a formidable tool for change and innovation, whose success, among other aspects, depends on employing concepts and knowledge of different natures and on the alignment between the evaluated phenomenon and the evaluation’s epistemological, methodological and, above all, political basis. The posture of the evaluator is also crucial for the evaluation’s success and should transcend the theoretical-practical apparatus. A good evaluator should have the courage to show the way; the maturity and methodological flexibility to rethink issues when necessary; and the sensitivity to develop and maintain constant interrelationships with potential stakeholders to ensure the usefulness and adequate use of the evaluation product (Champagne; Contandriopoulos; Tanon, 2016CHAMPAGNE, F.; CONTANDRIOPOULOS, A. P.; TANON, A. Utilizar a avaliação. In: BROUSSELLE, A. et al. Avaliação: conceitos e métodos. Rio de Janeiro: Ed. Fiocruz, 2016. p. 242-261.; Contandriopoulos, 2016CONTANDRIOPOULOS, A. P. Avaliar a avaliação. In: BROUSSELLE, A. et al. Avaliação: conceitos e métodos. Rio de Janeiro: Ed. Fiocruz , 2016. p. 264-272.; Tanaka, 2017TANAKA, O. Y. Avaliação em saúde: novos tempos, novas construções. In: TANAKA, O. Y.; RIBEIRO, E. L.; ALMEIDA, C. A. L. Avaliação em saúde: contribuições para incorporação no cotidiano. Rio de Janeiro: Atheneu , 2017. p. 1-9.).

We also assessed the outcomes and/or potential of the studies. We found that 87.8% of the studies presented a detailed description of the evaluation’s results; 41.5% went beyond a mere comparison of attributes, addressing theoretical-conceptual issues; and 26.8% proposed forms of improving and supporting aspects of the phenomenon evaluated.

Communicating the results should be an important task in an evaluation process, and its content (the evaluation product) should express the same conceptual properties. Thus, the results should respond to the multiple evaluation dimensions, especially the cognitive - generating valid, socially accepted scientific information to assist the theoretical and practical development of the intervention; the normative - enabling the decision making; and instrumental - contributing to the improvement of a given situation (Champagne et al., 2016CHAMPAGNE, F. et al. A avaliação no campo da saúde: conceitos e métodos. In: BROUSSELLE, A. et al. Avaliação: conceitos e métodos. Rio de Janeiro: Ed. Fiocruz , 2016. p. 41-60.).

This survey showed that a significant percentage of the works contributed to this end, stimulating theoretical-practical reflections that favored the sophistication of central aspects of PHC services and/or put forward proposals, aiming to solve problems raised or improve one or more components of the quality of care in these spaces. However, we also identified studies whose results were restricted to the description of the findings, plus a rapid discussion, often based on the comparison with similar studies, in the manner of a research report.

Studies that seek to systematize situations contribute significantly to the advancement of knowledge, because they fill existing gaps, produce answers to hypotheses and problems proposed and help in the understanding of processes, a necessary condition for any evaluation. However, it is imperative that the evaluator be aware that scientific research and evaluation are two partially overlapping areas, and that lack of clarity about this may compromise the achievement of the proposed goals, since an evaluation should not be restricted to information production but rather consists essentially in making judgments (Contandriopoulos, 2016CONTANDRIOPOULOS, A. P. Avaliar a avaliação. In: BROUSSELLE, A. et al. Avaliação: conceitos e métodos. Rio de Janeiro: Ed. Fiocruz , 2016. p. 264-272.; Nemes, 2001NEMES, M. I. B. Avaliação em saúde: questões para o programa DST/aids no Brasil. Rio de Janeiro: Associação Brasileira Interdisciplinar de Aids, 2001.). It is worth mentioning that these works’ results may be only a part of larger studies’ results, given the impossibility of addressing in a single article the totality of the final product of an evaluation. However, we stress the importance of disseminating and making use of these results, even when succinctly communicated, since an evaluation is socially useful insofar as its conclusions are applied by the social agents to whom they are intended (Champagne et al., 2016CHAMPAGNE, F. et al. A avaliação no campo da saúde: conceitos e métodos. In: BROUSSELLE, A. et al. Avaliação: conceitos e métodos. Rio de Janeiro: Ed. Fiocruz , 2016. p. 41-60.).

Health care evaluation should always be action oriented (even indirectly) and, through its critical judgment, should be a lever for transformation, promoting the training and learning of all actors involved, so that they can acquire “new theoretical models that allow them to grasp the complexity of the evaluation and that of the interventions within their contexts” (Contandriopoulos, 2016CONTANDRIOPOULOS, A. P. Avaliar a avaliação. In: BROUSSELLE, A. et al. Avaliação: conceitos e métodos. Rio de Janeiro: Ed. Fiocruz , 2016. p. 264-272., 271).

Final considerations

PHC evaluation is a growing field in several parts of the world, including Brazil, a situation reinforced by the increasing number of studies addressing this subject in recent years.

Our analysis of the evaluation studies revealed a great diversity of concepts and methods, however, we observed a predominance of evaluations conducted in FHUs, which used quantitative methods and were based on the principles of health quality assessment - mostly related to the systemic model proposed by Donabedian and applied to PHC by Starfield. Regarding the strategies to obtain the necessary information for the evaluation, the majority of the studies used validated instruments, mainly the PCATool.

These findings suggest, among other things, that the improvement in Brazilian public policies aimed at institutionalizing service evaluation and promoting the qualification of PHC services, especially in FHS, has surpassed these spaces, influencing even the production of scientific knowledge. Regarding this last point, the main challenge for researchers/evaluators was the articulation of different approaches (quantitative and qualitative), in order to promote a broader apprehension of the phenomenon evaluated and its determinants. As for the product generated by this process, it favored the presentation of information (sometimes essentially technical and descriptive) to the detriment of proposing actions for implementing and improving the service evaluated.

This study allowed us to glimpse the Brazilian scientific panorama of primary health care evaluation; it also stimulated reflections on strategies for applying the products generated by evaluation processes to learning and social transformation. However, it also has limitations: only articles were examined, excluding other relevant sources of scientific information, such as theses, dissertations, books, etc .; it was not possible to assess the totality of the articles published on the subject, due to the search criteria used and the period covered; and the unfeasibility of exploring in depth all the aspects presented by the works researched.

References

Publication Dates

History