ABSTRACT

To encourage improvements in the quality of Primary Health Care in Brasília, the Federal District Health Department established the Qualis-APS Program in 2019, a co-production between healthcare professionals, managers and researchers from the University of Brasília and the Fundação Oswaldo Cruz. By involving managers and workers in the design, coordination, planning and execution, the aim was to promote engagement and a sense of ownership, prerequisites for evidence-based decision-making. The aim is to describe the conception and implementation of the evaluative process and the stages of the first evaluation cycle, concluded in 2022. I. Diagnosis of the structure of all 165 Basic Health Units; II. Elaboration of quality standards to compose self-assessment instruments, based on subsidies obtained in workshops with 544 health professionals and users; III. Development of the Qualis-APS Platform; IV. Self-assessment of the 603 Family Health Teams/Oral Health Teams, 100 Primary Health Care Management Teams and 59 Teams from the Expanded Family Health and Primary Care Centers on the Platform; V. Planning and elaboration of the Action Plan for Quality, made by the teams; VI. On-site evaluation of the Basic Units, by external researchers; VII. Certification of teams by the Health Department. The experience of the first cycle was successful and the second cycle is ongoing now.

KEYWORDS
Intersectoral collaboration; Health services research; Quality assurance; health care; Primary Health Care

Introduction

In Brasília, Federal District (DF), the Family Health Strategy (ESF) became a model in Primary Health Care (PHC) in 2017, when the DF’s Primary Care Policy was established^{11 Secretaria de Estado de Saúde (DF). Portaria 77, de 14 de fevereiro de 2017 que Estabelece a Política de Atenção Primária à Saúde do Distrito Federal. Diário Oficial do Distrito Federal, Brasília, DF. 2017 fev 15; Edição 33; Seção 1:4. 1}. Compared to other entities in the federation, the implementation of the ESF was late. When implementing the new policy, health professionals that were working in the traditional model were allowed to continue, as long as they expressed interest and participated in training and the evaluation process. The conversion of the model led to an increase in ESF coverage from 28% to 69%, in a period of two years, due to the reallocation of professionals and reorganization of teams, associated with the increase in hirings^{22 Corrêa DSRC, Moura AGOM, Quito MV, et al. Movimentos de reforma do sistema de saúde do Distrito Federal: a conversão do modelo assistencial da Atenção Primária à Saúde. Ciênc saúde coletiva. 2019;24(6):2031-41.}.

Six years after the implementation of the new model, the DF State Health Department (SES/DF), with the intention of improving the quality of PHC services, began, in 2018, a partnership with the University of Brasília (UnB) and the Oswaldo Cruz Foundation, aiming to develop and implement the Program for Improvement of Quality in Primary Health Care (Qualis-APS)^{33 Secretaria de Estado de Saúde (DF). Portaria 39, de 23 de janeiro de 2019 que institui, no âmbito do Sistema Único de Saúde do Distrito Federal, o Programa de Qualificação da Atenção Primária à Saúde (QualisAPS). Diário Oficial do Distrito Federal, Brasília, DF. 2019 fev 14; Edição 32; Seção 1:6.}. The program, launched in September 2019, has the following axes: the development and the implementation of an evaluation system, the development of training courses, the production and the dissemination of knowledge about the SUS in Brasília.

Regarding the evaluation of PHC, there is a recent experience, undertaken by the federal government to expand and consolidate the ESF throughout the country, the National Program for Improving Access and Quality of Primary Care (PMAQ-AB). Created in 2011, it was an evaluation strategy to qualify Primary Health Care, based on voluntary adherence^{44 Ministério da Saúde (MS). Portaria MS nº 1.654 de 19 julho de 2011. Institui, no âmbito do Sistema Único de Saúde o Programa Nacional de Melhoria do Acesso e da Qualidade da Atenção Básica. Diário Oficial da União, Brasília, DF. 2011 jul 20; Edição 138; Seção 1:79.}. The PMAQ-AB was developed in three cycles: 1st (November 2011 - March 2013), 2nd (April 2013 - September 2015) and 3rd (October 2015 - December 2019)^{55 Russo LX, Powell-Jackson T, Barreto JOM, et al. Pay for performance in primary care: the contribution of the Programme for Improving Access and Quality of Primary Care (PMAQ) on avoidable hospitalisations in Brazil, 2009-2018. BMJ Glob Health. 2021;6(7):e005429.}. In its third and final cycle (2015), PMAQ achieved very high coverage, with 19 of the 27 Federation Units showing team adherence superior to 90%.

However, it registered a low adherence in Brasília-DF, of around 50.3% of the family health teams (eSF) and oral health teams (eSB)^{66 Ministério da Saúde (BR), Secretaria de Atenção Primária à Saúde. Adesão ao Programa Nacional de Melhoria do Acesso e da Qualidade da Atenção Básica. Adesão ao terceiro ciclo: Relatório Estadual. Brasília, DF: Ministério da Saúde; 2015.} teams. The following problems marked the PMAQ-AD in the DF:

In Brasília, the need to improve the quality of PHC services persisted. To this end, the development of an evaluation system that considered the local reality, guided by the principles of participation and co-construction with different actors, became a requirement. Unlike previous experiences, Qualis-APS innovates by building them based on the singularities of Brasília-DF.

The Basic Project was created in co-production by managers, professionals and researchers. The paradigm of co-production is to serve public interests and become a transformative action promoting the engagement of citizens, social groups and strengthening democracy<sup>88 Ostrom E. Crossing the great divide: coproduction, synergy, and development. World Development. 1996;24(6):1073-87.

9 Oliver K, Kothari A, Mays N. The dark side of coproduction: do the costs outweigh the benefits for health research? Health Res Policy Sys. 2019;17(33):1-10. DOI: https://doi.org/10.1186/s12961-019-0432-3
https://doi.org/10.1186/s12961-019-0432-... -1010 Hansen PM, Synowiec C, Blanchet NJ. Co-production between researchers and policymakers is critical for achieving health systems change. The BMJ Opinion [Internet]. 2021 fev 15 [acesso em 2023 abr 25]. Disponível em: https://blogs.bmj.com/bmj/2021/02/15/co-production-between-researchers-and-policymakers-is-critical-for-achieving-health-systems-change/
https://blogs.bmj.com/bmj/2021/02/15/co-...</sup> , based on the vision and experiences lived in PHC by professionals and users, in the logic of the subject involved in the transformative process^{1111 Freire P. Pedagogia da autonomia: saberes necessários à prática educativa. São Paulo: Paz e Terra; 1996.}.

Based on studies that used co-production of research in health services, the four main reasons presented for co-producing are summarized in table 1.

Figure 1, elaborated from the report of lessons learned from research in the ‘real world’^{1212 Di Giulio A, Defila R. Lessons from “real-world laboratories” about transdisciplinary projects, transformative research and participation. Integration and Implementation Insights [Internet]. 2018 mar 22 [acesso em 2023 abr 25]. Disponível em: https://i2Insights.org/2018/03/22/lessons-from-real-world-laboratories/
https://i2Insights.org/2018/03/22/lesson...} , outlines the mechanisms optimized by the co-production strategy of evaluative research.

The joint work of researchers and managers, from the beginning of a Project or Program, confers scientific legitimacy, characteristic of universities and research institutes, and, simultaneously, the political (institutional) legitimacy of health systems and practical legitimacy, arising from health services (figure 1). Co-production qualifies the knowledge generated and optimizes the development of instruments and data analysis plans. Furthermore, once the results are available, it encourages the appropriation of the knowledge generated and its incorporation to improve health systems and services^{1212 Di Giulio A, Defila R. Lessons from “real-world laboratories” about transdisciplinary projects, transformative research and participation. Integration and Implementation Insights [Internet]. 2018 mar 22 [acesso em 2023 abr 25]. Disponível em: https://i2Insights.org/2018/03/22/lessons-from-real-world-laboratories/
https://i2Insights.org/2018/03/22/lesson...} .

However, interests and priorities may differ between health policy managers and researchers. The temporality is different, as formulators and managers always want quick, viable and pragmatic solutions, since the needs of services are numerous and in part urgent, but science requires careful analysis and considered deliberations^{1313 Bensing JM. Doing the right thing and doing it right: toward a framework for assessing the policy relevance of health services research. Int J Technol Assess Health Care. 2003;19:604-12. DOI: https://doi.org/10.1017/s0266462303000564
https://doi.org/10.1017/s026646230300056...} .

The objective of this manuscript is to report the design and implementation, in co-production, of the PHC evaluation system in Brasília-DF, within the scope of the Qualis-APS Program.

The evaluation system

The evaluation system of PHC in Brasília-DF is developed by researchers from UnB, in partnership with SES/DF, with the aim of stimulating the organization of work processes, according to the health needs of the territories, and also institutionalize evaluation practices, which can encourage an even greater sense of belonging among health workers.

This is an intervention, research and training program, considering that it seeks to bring about changes in work and management processes, produce knowledge about the health system in Brasília-DF and, along the way, carry out training in and through work during the evaluation cycles.

The evaluation system adopted the references of coproduction, participation and autonomy<sup>88 Ostrom E. Crossing the great divide: coproduction, synergy, and development. World Development. 1996;24(6):1073-87.

9 Oliver K, Kothari A, Mays N. The dark side of coproduction: do the costs outweigh the benefits for health research? Health Res Policy Sys. 2019;17(33):1-10. DOI: https://doi.org/10.1186/s12961-019-0432-3
https://doi.org/10.1186/s12961-019-0432-...

10 Hansen PM, Synowiec C, Blanchet NJ. Co-production between researchers and policymakers is critical for achieving health systems change. The BMJ Opinion [Internet]. 2021 fev 15 [acesso em 2023 abr 25]. Disponível em: https://blogs.bmj.com/bmj/2021/02/15/co-production-between-researchers-and-policymakers-is-critical-for-achieving-health-systems-change/
https://blogs.bmj.com/bmj/2021/02/15/co-...

11 Freire P. Pedagogia da autonomia: saberes necessários à prática educativa. São Paulo: Paz e Terra; 1996.-1212 Di Giulio A, Defila R. Lessons from “real-world laboratories” about transdisciplinary projects, transformative research and participation. Integration and Implementation Insights [Internet]. 2018 mar 22 [acesso em 2023 abr 25]. Disponível em: https://i2Insights.org/2018/03/22/lessons-from-real-world-laboratories/
https://i2Insights.org/2018/03/22/lesson...</sup> , participatory evaluation, democratic and emancipatory proposal^{1414 Guba EG, Lincoln YS. Avaliação de quarta geração. Campinas: Editora da Unicamp; 2011.}, analysis of work activity^{1515 Schwartz Y, Durrive L. L’activité en dialogues: entretiens sur l’activité humaine (II). Toulouse: Octarès Éditions; 2009. 296 p.,1616 Schwartz Y. O agir avaliativo entre seus dois polos. Serv Soc e Saúde. 2019;18:e019006.} and continuous quality improvement^{1717 Ministério da Saúde (BR). Secretaria de Atenção à Saúde. A melhoria contínua da qualidade na atenção primária à saúde: conceitos, métodos e diretrizes. Brasília, DF: Ministério da Saúde; 2010.}. The Assessment Methodology, developed in co-production with SES/DF, is detailed in a publication in the Qualis-APS Supplement series^{1818 Scherer MDA, Freitas MABF, organizadoras. Metodologia da avaliação da Atenção Primária à Saúde do Distrito Federal [Internet]. Brasília, DF: Escola de Governo Fiocruz; 2022 [acesso em 2014 mar 8]. Disponível em: https://qualisaps.unb.br/files/CadernosQualisAPS-MetodologiadaavaliacaodaAPSdoDistritoFederal2022.pdf
https://qualisaps.unb.br/files/CadernosQ...} .

The protocol for the implementation and development of the evaluation system was approved by the Research Ethics Committee of the Faculty of Health Sciences of the University of Brasília (CEP/FS-UnB), with opinion No. 3,937,242 (CAAE No. 29640120.6.0000.0030). All ethical precepts were respected.

As described below in each stage, the actions carried out from January 2020 to December 2022 allowed the baseline diagnosis of the Primary Health Care (UBS) structure and the completion of a full evaluation cycle, which included the self-assessment of the eSF/eSB, teams from the Primary Health Care Services Management (GSAP) and teams from the Expanded Centers for Family Health and Primary Care (eNasf-AB). Furthermore, Quality Action Plans (PAQ) were drawn up and executed by the teams, as well as an on-site assessment of the existing UBS and certification of the eSF/eSB by SES/DF was carried out.

The seven steps below were designed, implemented and completed in the aforementioned period:

1. I. Baseline diagnosis of the structure of the 165 UBS and analysis of the UBS’s response capacity to COVID-19;

2. II. Co-production of quality standards and self-assessment instruments;

3. III. Development of the Qualis-APS Platform;

4. IV. Self-assessment of 603 eSF/eSB, 100 GSAPs and 59 Nasf-AB teams;

5. V. Co-construction of the local planning instrument, preparation and execution of the PAQ by eSF/eSB, GSAPs and eNasf-AB;

6. VI. On-site assessment of UBS and user satisfaction, carried out by external researchers;

7. VII. Certification of the eSF/eSB by SES/DF, based on data collected and analyzed by the UnB team of researchers.

I. Initial diagnosis of the structure - baseline

The structure can be defined as the structural conditions of health services (material and human resources and physical environment). An adequate structure enables improvements in professionals’ work processes and better assistance to users^{1919 Ministério da Saúde (BR), Secretaria de Atenção à Saúde. Manual de estrutura física das unidades básicas de saúde: saúde da família. 2. ed. Brasília, DF: Ministério da Saúde; 2008.}.

An initial diagnosis was carried out to evaluate the structure of the 165 UBS in Brasília, using an instrument built for this purpose, after an extensive analysis of available models and existing regulations; there was active participation of SES/DF managers in the co-production of the instruments. Considering the pandemic context and the need for SES/DF to integrate other evaluation mechanisms into Qualis-APS, questions were included in the instrument to evaluate the response capacity of UBSs to the new coronavirus pandemic, as well as the indicators of the Local Management Agreement (AGL), a contractualization tool signed between the Health Regions of the DF Superintendencies and the UBS in their territory. The construction parameters and the instrument are available in an article published by the Qualis-APS^{2020 Furlanetto DLC, Santos W, Scherer MDA, et al. Estrutura e responsividade: a Atenção Primária à Saúde está preparada para o enfrentamento da Covid-19? Saúde debate. 2022;46(134):630-48. DOI: https://doi.org/10.1590/0103-1104202213403P
https://doi.org/10.1590/0103-11042022134...} team.

Data collection took place between August 2020 and January 2021 online and by telephone, given the advent of the COVID-19 pandemic and its measures of social distancing and isolation. The instrument was pre-tested in two prison UBS in the Federal District that do not participate in the program.

The analysis of the results, important for combating COVID-19, was completed and delivered to SES/DF in 2021 and then the Structure Diagnosis was published in the Qualis-APS Supplement series^{2424 Lima AA, Pedrosa CM, Furlanetto DLC, et al., organizadores. Tipologia de estrutura das Unidades Básicas de Saúde do Distrito Federal [Internet]. Brasília, DF: Escola de Governo Fiocruz; 2022 [acesso em 2014 mar 8]. Disponível em: https://qualisaps.unb.br/files/2022_11_29-v8_relatorio-tipologia-estrutura.pdf
https://qualisaps.unb.br/files/2022_11_2...} . The analysis of the capacity to respond to the pandemic indicated that there was readjustment of the physical structure (waiting rooms, internal/external spaces, erection of tents); adequate supply of Personal Protective Equipment (PPE) and COVID-19 tests; active search for COVID-19 suspects by telephone and some home visits; monitoring patient transfer flows and telehealth services. The diagnosis concluded that the UBSs in Brasília-DF managed to reorganize their services to meet the most urgent needs of the pandemic^{2020 Furlanetto DLC, Santos W, Scherer MDA, et al. Estrutura e responsividade: a Atenção Primária à Saúde está preparada para o enfrentamento da Covid-19? Saúde debate. 2022;46(134):630-48. DOI: https://doi.org/10.1590/0103-1104202213403P
https://doi.org/10.1590/0103-11042022134...} . Furthermore, the instrument collected data that allowed the UBSs to be classified according to typology using a method adapted from Giovanella^{2222 Giovanella L, Bousquat A, Fausto MCR, et al. Tipologia das unidades básicas de saúde brasileiras. Novos Caminhos. 2015;5:1-63.} and Bousquat^{2323 Bousquat A, Giovanella L, Fausto MCR. Tipologia da estrutura das unidades básicas de saúde brasileiras: os 5 R. Cad Saúde Pública 2017;33(8):e00037316. DOI: https://doi.org/10.1590/0102-311X00037316
https://doi.org/10.1590/0102-311X0003731...} . The criteria used and the results of the Structure Typology were published in the Qualis-APS Supplement series^{2424 Lima AA, Pedrosa CM, Furlanetto DLC, et al., organizadores. Tipologia de estrutura das Unidades Básicas de Saúde do Distrito Federal [Internet]. Brasília, DF: Escola de Governo Fiocruz; 2022 [acesso em 2014 mar 8]. Disponível em: https://qualisaps.unb.br/files/2022_11_29-v8_relatorio-tipologia-estrutura.pdf
https://qualisaps.unb.br/files/2022_11_2...} .

II. Co-production of quality standards and self-assessment instruments

The participatory construction of quality standards for the development of self-assessment instruments occurred in two stages. In the first, in 2020, the aim was on the GSAPS and eSF/eSB teams. In 2021, the development of standards to form the NASF-AB self-assessment instrument began. The entire process is characterized by innovation as it is based on the knowledge and experience of the participants, in triangulation with the attributes and guiding standards of PHC.

In the months of January and February 2020, 7 face-to-face workshops were held with 425 management and care professionals, from all health regions in Brasília-DF, to support the construction of quality standards that comprised the self-assessment instruments. At the events, 20 focus groups and 21 multi-professional reflective circles were held in which care and management workers - UBS managers and supervisors - contributed with reflections on the quality of PHC in services, the work of the manager and teams and the care provided to users. The workshops took place in neutral locations, not linked to SES/DF facilities, ensuring a welcoming environment that inspired confidence in participants so they could express themselves freely.

With the beginning of restrictions due to the COVID-19 pandemic, the planned activities with users and civil society representatives were adapted to remote mode. With the support of the DF Health Council, four virtual focus groups were held, lasting approximately two hours, in which 31 health advisors representing civil society and professional entities participated.

The perceptions of participants in all workshops, in person and remotely, about ‘what a quality PHC service should be like’ generated the basic raw material for the construction of quality standards that make up the self-assessment instruments of eSF, eSB and teams of GSAP. Qualitative data analysis was carried out using the Idea Association Map technique^{2525 Spink MJ. Linguagem e produção de sentidos no cotidiano. Rio de Janeiro: Centro Edelstein de Pesquisas Sociais; 2010. Cap. III, As múltiplas faces da pesquisa sobre produção de sentidos no cotidiano. p. 38-59.}, in which the narratives were categorized into structure, process or attention to the user, followed by connecting it to the analytical units of the management team (GSAP) and eSF team and eSB. After constructing the map of each region, we moved on to the stage of preparing assertions or declarations of expected quality, that is, the quality standard, always in pairs of researchers, in order to guarantee reliability and adequacy of the information generated.

Still from the point of view of co-production, several discussions were held with the SES/DF technical team to qualify the standards and evaluable elements. These guided the analysis of compliance with each standard, enabling professionals to analyze what they do in relation to what is defined in regulations.

To finalize the standards, the indicators of the AGL were considered, which, associated with the PHC guiding standards and the subsidies provided by workers and users, express the uniqueness of PHC in Brasília-DF. It is noteworthy that all quality standards and evaluable elements present in the evaluation instruments are in accordance with the National Primary Care Policy (PNAB) and the Primary Care Policy of Brasília-DF.

The pre-test of the eSF/eSB and GSAP instruments was carried out in nine eSF/eSB teams and six GSAPs from UBS in different health regions, aiming at the semantic analysis of the corresponding standards and evaluable items, as well as the functionality of the Qualis-APS Platform.

The elaboration of quality standards and the development of instruments for self-assessment by the eSF/eSB and GSAP teams took place in co-production with SES/DF and is described in the Assessment Methodology publication of the Qualis-APS Supplement series^{1818 Scherer MDA, Freitas MABF, organizadoras. Metodologia da avaliação da Atenção Primária à Saúde do Distrito Federal [Internet]. Brasília, DF: Escola de Governo Fiocruz; 2022 [acesso em 2014 mar 8]. Disponível em: https://qualisaps.unb.br/files/CadernosQualisAPS-MetodologiadaavaliacaodaAPSdoDistritoFederal2022.pdf
https://qualisaps.unb.br/files/CadernosQ...} . The instrument for self-assessment of eSF/eSB teams has 45 quality standards, divided into 4 dimensions and 10 subdimensions^{2626 Pedrosa CM, Poças KC, Scherer MDA, et al., organizadoras. Instrumento de autoavaliação das equipes da Estratégia Saúde da família (eSF) e Saúde Bucal (eSB) [Internet]. Brasília, DF: Universidade de Brasília; 2021 [acesso em 2014 mar 8]. Disponível em: https://qualisaps.unb.br/files/CAD_EQUIPE-INSTRUMENTO-COM-PADROES-FINAIS-NOVO-EXPEDIENTE-08072022.pdf
https://qualisaps.unb.br/files/CAD_EQUIP...} . The instrument for self-assessment of GSAP has 36 standards organized into 5 dimensions and 12 subdimensions^{2727 Pedrosa CM, Poças KC, Scherer MDA, et al., organizadoras. Instrumento de autoavaliação da equipe de Gestão da Atenção Primária à Saúde [Internet]. Brasília, DF: Universidade de Brasília; 2021 [acesso em 2014 mar 8]. Disponível em: https://qualisaps.unb.br/files/CAD-GESTAO-INSTRUMENTO-COM-PADROES-FINAIS-NOVO-EXPEDIENTE-08072022.pdf
https://qualisaps.unb.br/files/CAD-GESTA...} .

To establish quality standards for the NASF-AB, 22 activities were carried out in December 2021 in the format of FG, collective interviews and conversation circles, with 119 NASF-AB professionals and 57 users of PHC services. The material collected with the participants’ perceptions about quality care in PHC and the work of the eNASF-AB was systematized and analyzed according to the methodology described above for the eSF/eSB and GSAP, generating, in co-production with the managers of the SES/ DF, 24 quality standards that made up the self-assessment instrument^{2828 Pedrosa CM, organizadora. Manual de avaliação das equipes dos núcleos ampliados de Saúde da Família e Atenção Básica - Nasf-AB [Internet]. Brasília, DF: Universidade de Brasília; 2022 [acesso em 2014 mar 8]. Disponível em: https://qualisaps.unb.br/files/Manual_Avaliacao_Nasf-AB_Final.pdf
https://qualisaps.unb.br/files/Manual_Av...} .

The pre-test of the instrument took place with members of the NASF-AB Technical Chamber, filling the Qualis-APS Platform out in a simulation, to evaluate access and usability of the tool by participants. The information and suggestions presented in the pre-test were considered to make adjustments to the instrument and the Platform.

III. Creation of the Qualis-APS Platform

The Qualis-APS Platform was developed in co-production with the SES/DF technical team to, in addition to housing all Qualis-APS production, allow teams access, both to fill out the assessment instruments and prepare the action plan, and to monitor the filling progress. It has a modular format, facilitating the systematization and analysis of results, with the extraction of personalized reports.

The platform has two interaction environments: a) a public environment, which allows access to information about the Program and published documents about the evaluation process and the results of the Qualis-APS phases; and b) a restricted environment, accessed via login, for SES-DF PHC workers. In this restricted environment, each worker can see the results of their respective team and participate in the current phase, according to the Evaluation Cycle schedule.

Currently, the Platform has 1,247 registered and active users, of which 1,007 are health team professionals, 133 are local managers, 31 are regional managers, 26 are central managers and 12 are from the UnB^{2929 Programa QUALIS APS [Internet]. Plataforma da Avaliação da Atenção Primária à Saúde do Distrito Federal. Rio de Janeiro: 2019 [acesso em 2014 mar 8]. Disponível em: https://qualisaps.unb.br/
https://qualisaps.unb.br/...} .

IV. Self-assessment of eSF/eSB, GSAPs and NASF-AB

In Qualis-APS, the recommended assessment is not limited to checking compliance with norms and standards, but rather a co-learning process. It was conceived from the perspective that workers should have spaces to analyze the work situations they experience, so they can understand what contributes to the quality of services and propose adjustments - if necessary -, according to local uniqueness. Therefore, the teams were instructed to carry out the self-assessment collectively. The teams assessed the degree of compliance in each standard, on a scale of 1 to 5, with 1 corresponding to ‘I do not comply’ and 5 to ‘I fully comply’. There was broad mobilization to complete the respective self-assessment instruments among the eSF/eSB^{2626 Pedrosa CM, Poças KC, Scherer MDA, et al., organizadoras. Instrumento de autoavaliação das equipes da Estratégia Saúde da família (eSF) e Saúde Bucal (eSB) [Internet]. Brasília, DF: Universidade de Brasília; 2021 [acesso em 2014 mar 8]. Disponível em: https://qualisaps.unb.br/files/CAD_EQUIPE-INSTRUMENTO-COM-PADROES-FINAIS-NOVO-EXPEDIENTE-08072022.pdf
https://qualisaps.unb.br/files/CAD_EQUIP...} , GSAPS^{2727 Pedrosa CM, Poças KC, Scherer MDA, et al., organizadoras. Instrumento de autoavaliação da equipe de Gestão da Atenção Primária à Saúde [Internet]. Brasília, DF: Universidade de Brasília; 2021 [acesso em 2014 mar 8]. Disponível em: https://qualisaps.unb.br/files/CAD-GESTAO-INSTRUMENTO-COM-PADROES-FINAIS-NOVO-EXPEDIENTE-08072022.pdf
https://qualisaps.unb.br/files/CAD-GESTA...} and NASF-AB^{2828 Pedrosa CM, organizadora. Manual de avaliação das equipes dos núcleos ampliados de Saúde da Família e Atenção Básica - Nasf-AB [Internet]. Brasília, DF: Universidade de Brasília; 2022 [acesso em 2014 mar 8]. Disponível em: https://qualisaps.unb.br/files/Manual_Avaliacao_Nasf-AB_Final.pdf
https://qualisaps.unb.br/files/Manual_Av...} .

On the Platform, compliance with the standard was indicated by a traffic light, facilitating the visualization of results in graphs/tables and guiding the analysis of results. All 100 GSAPS completed the instrument on the Qualis-APS Platform. Among the 603 eSF/eSB, 599 (99%) completed it within the defined deadline (September 1st to December 17th, 2021).

V. Planning and preparation of a Quality Action Plan

After completing the self-assessment, the teams began preparing the PAQ, guided by an instrument adapted from a model already used by SES/DF in outlining plans related to Regional Management Agreements and AGLs. The objective of this stage is to promote an opportunity for the eSF and eSB teams, as well as local management, to plan, encouraging the participation of professionals in the construction of proposals to improve the services offered and the adaptation of work processes based on local needs and demands.

The PAQ was developed to be completed on the Qualis-APS Platform in specific models for eSF/eSB teams. The eSF/eSB and GSAP teams were encouraged to analyze the results of the self-assessment and define, among the quality standards that performed worst, four to be included in the PAQ, to be the subject of intervention for improvement, two of these standards being related to the AGL indicators.

After selecting the standards, the teams identified problems and developed actions that could be taken to achieve a better result. To this end, they identified the necessary resources, those responsible for the actions, the deadline for each activity, the criteria for checking compliance and were able to indicate the status of the achievement over time. The mobilization and guidance for the preparation of the PAQ was constant throughout the entire process. Of the 603 eSF/eSB, 592 inserted their Action Plans into the Qualis-APS Assessment Platform within the deadline defined by the SES/DF management. After this period, teams could insert new problems and actions, in addition to updating the status of the achievement. In total, 601 Action PAQs were prepared by eSF in partnership with reference eSB and 95 GSAPs prepared the PAQ.

VI. On-site assessment and user satisfaction

This stage was carried out after the execution of the action plan and aimed to evaluate components of the UBS structure, the work process of the care teams and local management, in addition to understanding users’ perceptions of the services offered at PHC. The objective is to provide subsidies to guide the actors involved (care professionals, local, regional and central-level managers, and health council) in making decisions about the continuity and reorientation of processes.

For data collection, a manual and instruments were developed, in co-production with the SES/DF technical team and UnB researchers^{3030 Lima AA, Furlanetto DLC, Santos LMP, et al., organizadores. Manual de avaliação in loco e instrumento [Internet]. Brasília, DF: Universidade de Brasília; 2022 [acesso em 2014 mar 8]. Disponível em: https://qualisaps.unb.br/files/MANUAL_INSTRUTIVO_FINAL_22_08_22_revisto.pdf
https://qualisaps.unb.br/files/MANUAL_IN...} . The manual was structured into five modules: eSF module, with an emphasis on the eSF work process; eSB module, on the eSB work process; GSAP module, related to the GSAP work process; user module, about user satisfaction; on-site observation/visit module with questions relating to the structure of the UBS. All modules underwent semantic review and were pre-tested before data collection began.

Data collection was conducted by evaluators external to SES/DF. The interviewers were previously selected and trained by the team of researchers from the UnB, who also coordinated the entire data collection process in the field. All 165 UBS in Brasília were included in the data collection, which occurred with on-site visits on previously scheduled dates. The tool used was digital, using the REDCap application, installed on tablets. The eSF, eSB and GSAP teams modules were answered through interviews with workers from these teams. For the On-site Observation module, a guided tour was carried out with a member of the management team or someone indicated by them.

The interviewed users were selected on the day of the visit to the UBS, according to the following criteria: user who sought care at any UBS service on the date of the interview (regardless of whether they were successful or not); user aged 18 or over able to answer the questions independently (elderly companions were not included); guardian or caregiver accompanying the child aged zero to 12 years old. Collecting this on-site evaluation data and user satisfaction provided support for the certification stage

VII. Certification of eSF/eSB by SES

Certification is the last phase of an Assessment Cycle of the Qualis-APS Program drawn from the results of all stages of Qualis-APS. In it, each of the eSF/eSB of the PHC in Brasília-DF is recognized for its performance during that Cycle, receiving a rating that varies from five to one Ipê - a native and symbolic tree of the Federal Capital biome, chosen as a symbol of Qualis-APS Program.

Although this phase is the responsibility of SES/DF, its process was the result of several joint construction workshops between UnB and SES/DF, to define the variables and indicators that would make up the Certification. In the 1st Assessment Cycle, the teams were evaluated considering: (a) carrying out the self-assessment within the stipulated period, regardless of the grade given by the teams; (b) preparation and execution of the PAQ within the defined deadline; (c) team performance in the On-site Assessment phase, with selected variables related to the work process; (d) performance in the population registration indicator, in accordance with the SES/DF AGL; (e) user satisfaction, measured in an interview representative of the Health Region. At the conclusion of the 1st Evaluation Cycle of Qualis-APS, SES-DF organized a ceremony to award certification to the 66 health teams with the best evaluation, whose symbol translates as ‘five Ipês’.

Final consideration

The Qualis-APS Program, developed in the form of co-production between academics and local PHC managers in Brasília-DF, established an innovative role for management with the potential to achieve changes in health policies and practices within the scope of PHC. The reasons for developing it in co-production are: (a) improving the quality of the evaluation, by focusing on appropriate topics that need to be elucidated; (b) increase in the probability of using the results effectively, by creating capabilities among non-academics and a sense of empowerment among those potentially interested in the evaluation results; (c) conducting research/evaluation to serve public interests: co-production can be transformative, promoting the engagement of citizens and social groups and fueling democracy; (d) increased sense of ownership, a prerequisite for making decisions using research results, by involving ‘non-researchers’ in coordinating the entire process. Experience has indicated that it is viable to develop proposals for quality improvement, such as the Qualis-APS Program, with quality standards tailored to local specificities, aiming to stimulate institutional evaluation practices that have a continuous nature.

In the co-production process, many changes were made, both to the initially designed project, and to the execution schedule. This allowed adaptation to management needs both in terms of implementation and in strategies prioritized depending on political definitions and the health context, such as the COVID-19 pandemic. Co-production requires dedicated time from everyone involved, which is not always possible in management time, or even considering the project/program financing term, as mentioned by Oliver and collaborators^{99 Oliver K, Kothari A, Mays N. The dark side of coproduction: do the costs outweigh the benefits for health research? Health Res Policy Sys. 2019;17(33):1-10. DOI: https://doi.org/10.1186/s12961-019-0432-3
https://doi.org/10.1186/s12961-019-0432-...} .

The broader and more diverse the group that conducts the co-production work, the richer and more comprehensive the results will be, but the longer it will take, with likely schedule delays. In this context, it will be necessary to have people who can facilitate discussions and mediate the conflicts that inevitably arise in the collaborative process.

References

Publication Dates

History