ABSTRACT

Objective:

This study describes the methodology used in the development of MonitoraSB, an innovation in the field of evaluating and monitoring oral health services in primary health care (PHC) in Brazil, and it discusses its characteristics and possible uses.

Methods:

MonitoraSB includes a matrix of indicators and digital tools, developed in collaboration with oral health team dentists and oral health managers. The indicators evaluate the provision and management of oral health services and were developed on the basis of the model for evaluating the effectiveness of oral health care. The dashboard and calculator were developed to facilitate the operational use of the indicators.

Results:

The 54 indicators in the matrix, with demonstrated content validity and measurability, cover various aspects of oral health care in PHC, from the organization and capacity of services to the profile of care and resolution. The dashboard offers navigation and interactivity, allowing regional, state and municipal analysis and making geographical and temporal comparisons. The calculator allows indicators to be obtained at the local level.

Conclusion:

The implementation of MonitoraSB could promote the strengthening of PHC, supporting informed decision-making and the consolidation of the universal, comprehensive and equitable oral health care model.

Keywords:
Health evaluation; Public health services; Health information systems; Health strategies

INTRODUCTION

Primary health care (PHC), the gateway and structural basis of Unified Health System (SUS), is essential in managing health care¹1 Brasil. Ministério da Saúde. Gabinete do Ministro. Portaria no 2.436, de 21 de setembro de 2017. Aprova a Política Nacional de Atenção Básica (PNAB), estabelecendo a revisão de diretrizes para a organização da Atenção Básica, no âmbito do SUS [Internet]. 2017 [accessed on Mar 30, 2024]. Available at: https://bvsms.saude.gov.br/bvs/saudelegis/gm/2017/prt2436_22_09_2017.html
https://bvsms.saude.gov.br/bvs/saudelegi... . Strengthening PHC is essential for the resilience of health systems²2 Massuda A, Bigoni A, Paschoalotto MA, Tasca R. Rumos para um sistema de saúde resiliente. GV-Executivo 2022; 21(2): 22-9. https://doi.org/10.12660/gvexec.v21n2.2022.85938
https://doi.org/10.12660/gvexec.v21n2.20... , ensuring equitable and quality care for the needs of the population. One of the strategies for this is to institutionalize monitoring and evaluation practices³3 Hone T, Macinko J, Millett C. Revisiting Alma-Ata: what is the role of primary health care in achieving the Sustainable Development Goals? Lancet 2018; 392(10156): 1461-72. https://doi.org/10.1016/S0140-6736(18)31829-4
https://doi.org/10.1016/S0140-6736(18)31... . Monitoring corresponds to the systematic monitoring of services based on information about the reality⁴4 Hartz ZMA, Silva LMV. Avaliação em saúde: dos modelos teóricos à prática da avaliação de programas e sistemas de saúde [Internet]. Salvador: EDUFBA; 2005 [accessed on Jul 29, 2024]. Available at: https://books.scielo.org/id/xzdnf/pdf/hartz-9788575415160.pdf
https://books.scielo.org/id/xzdnf/pdf/ha... .

In Brazil, the institutionalization of PHC assessment processes began with the creation of the Coordination of Monitoring and Evaluation of the Primary Care Department in 2000⁵5 Brasil. Ministério da Saúde. Secretaria de Atenção á Saude. Departamento de Atenção Básica. Coordenação de Acompanhamento e Avaliação. Avaliação na atenção básica em saúde: caminhos da institucionalização. Brasília: Ministério da Saúde; 2005.. Since then, several initiatives have been proposed by the Ministry of Health: Proposal for Evaluation of the Performance of the Health System (PROADESS)⁶6 Viacava F, Ugá MAD, Porto S, Laguardia J, Moreira RS. Avaliação de desempenho de sistemas de saúde: um modelo de análise. Ciênc Saúde Coletiva 2012; 17(4): 921-34. https://doi.org/10.1590/S1413-81232012000400014
https://doi.org/10.1590/S1413-8123201200... , Evaluation for Improvement of the Quality of the Family Health Strategy⁷7 Brasil. Ministério da Saúde. Secretaria de Atenção à Saúde. Departamento de Atenção Básica. Avaliação para melhoria da qualidade da estratégia saúde da família [Internet]. Brasília: Ministério da Saúde; 2005 [accessed on Jul 29, 2024]. Available at: https://bvsms.saude.gov.br/bvs/publicacoes/doc_tec_amq_portugues.pdf
https://bvsms.saude.gov.br/bvs/publicaco... , Evaluation of the Quality of Primary Care Services⁸8 Castanheira ERL, Nemes MIB, Almeida MAS, Puttini RF, Soares ID, Patrício KP, et al. QualiAB: desenvolvimento e validação de uma metodologia de avaliação de serviços de atenção básica. Saude Soc 2011; 20(4): 935-47. https://doi.org/10.1590/S0104-12902011000400011
https://doi.org/10.1590/S0104-1290201100... , Use of the PHC Evaluation Instrument (PCATool)⁹9 Brasil. Ministério da Saúde. Secretaria de Atenção em Saúde. Departamento de Atenção Básica. Manual do instrumento de avaliação da atenção primária à saúde: primary care assessment tool pcatool [Internet]. Brasília: Ministério da Saúde; 2010 [accessed on Jul 29, 2024]. Available at: https://bvsms.saude.gov.br/bvs/publicacoes/manual_avaliacao_atencao_primaria.pdf
https://bvsms.saude.gov.br/bvs/publicaco... , Program for Improvement of Access and Quality of Primary Care (PMAQ-AB)¹⁰10 Brasil. Ministério da Saúde. Secretaria de Atenção à Saúde. Departamento de Atenção Básica. Manual instrutivo para as equipes de atenção básica e NASF. Programa nacional de melhoria do acesso e da qualidade da atenção básica (PMAQ): Terceiro ciclo (2015-2017) [Internet]. Brasília: Ministério da Saúde; 2017 [accessed on Jul 29, 2024]. Available at: http://189.28.128.100/dab/docs/portaldab/documentos/Manual_Instrutivo_3_Ciclo_PMAQ.pdf
http://189.28.128.100/dab/docs/portaldab... , development of the Unified Health System Performance Index (IDSUS)¹¹11 Brasil. Ministério da Saúde. Portal da Saúde. Índice de Desempenho do Sistema Único de Saúde [Internet]. 2012 [accessed on Jul 29, 2024]. Available at: https://idsus.saude.gov.br/
https://idsus.saude.gov.br/... and reformulation of the National Program for Evaluation of Health Services (PNASS)¹²12 Brasil. Ministério da Saúde. Gabinete do Ministro. Portaria no 28, de 8 de janeiro de 2015. Reformula o Programa Nacional de Avaliação de Serviços de Saúde (PNASS) [Internet]. 2015 [accessed on Jul 29, 2024]. Available at: https://bvsms.saude.gov.br/bvs/saudelegis/gm/2015/prt0028_08_01_2015.html
https://bvsms.saude.gov.br/bvs/saudelegi... .

PHC is also the gateway to the National Oral Health Policy (PNSB), which guides the evaluation and monitoring of the results achieved, aiming to guide informed decision-making through indicators¹³13 Brasil. Ministério da Saúde. Secretaria de Atenção à Saúde. Departamento de Atenção Básica. Coordenação Nacional de Saúde Bucal. Diretrizes da política nacional de saúde bucal [Internet]. Brasília: Ministério da Saúde; 2004 [accessed on Mar 30, 2024]. Available at: https://bvsms.saude.gov.br/bvs/publicacoes/politica_nacional_brasil_sorridente.pdf
https://bvsms.saude.gov.br/bvs/publicaco... –¹⁷17 Chaves SCL, Almeida AMFL, Reis CS, Rossi TRA, Barros TRA. Política de Saúde Bucal no Brasil: as transformações no período 2015-2017. Saúde Debate 2018; 42(n esp 2): 76-91. https://doi.org/10.1590/0103-11042018S206
https://doi.org/10.1590/0103-11042018S20... . Oral health indicators were established in the Primary Care Indicators Pact (1998–2006), in the Health Pacts (2007–2011) and in the resolutions of the Tripartite Intermanagerial Commission (2012, 2013 and 2016)¹⁸18 França MASA, Freire MCM, Pereira EM, Marcelo VC. Oral health indicators in the Interfederative Pacts of the Unified Health System: development in the 1998-2016 period. Rev Odontol UNESP 2018; 47(1): 18-24. https://doi.org/10.1590/1807-2577.08417
https://doi.org/10.1590/1807-2577.08417... . The IDSUS included indicators related to access to primary care. In the 3rd Cycle of the PMAQ-AB, oral health monitoring indicators assessed access, continuity of care, resolution and scope of services¹⁰10 Brasil. Ministério da Saúde. Secretaria de Atenção à Saúde. Departamento de Atenção Básica. Manual instrutivo para as equipes de atenção básica e NASF. Programa nacional de melhoria do acesso e da qualidade da atenção básica (PMAQ): Terceiro ciclo (2015-2017) [Internet]. Brasília: Ministério da Saúde; 2017 [accessed on Jul 29, 2024]. Available at: http://189.28.128.100/dab/docs/portaldab/documentos/Manual_Instrutivo_3_Ciclo_PMAQ.pdf
http://189.28.128.100/dab/docs/portaldab... . The Previne Brasil Program defined the indicator dental care for pregnant women to finance oral health teams (eSBs)¹⁹19 Brasil. Ministério da Saúde. Gabinete do Ministro. Portaria no 2.979, de 12 de novembro de 2019. Institui o Programa Previne Brasil, que estabelece novo modelo de financiamento de custeio da Atenção Primária à Saúde no âmbito do SUS, alterando a Portaria de Consolidação no 6/GM/MS, de 28 de setembro de 2017 [Internet]. 2019 [accessed on Jul 29, 2024]. Available at: https://bvsms.saude.gov.br/bvs/saudelegis/gm/2019/prt2979_13_11_2019.html
https://bvsms.saude.gov.br/bvs/saudelegi... .

On April 10, 2024, a new methodology for federal co-financing of the PHC floor was established by the Ministry of Health²⁰20 Brasil. Ministério da Saúde. Gabinete da Ministra. Portaria GM/MS no 3.493, de 10 de abril de 2024. Altera a portaria de consolidação GM/MS no 6, de 28 de setembro de 2017, para instituir nova metodologia de cofinanciamento federal do Piso de Atenção Primária à Saúde no âmbito do Sistema Único de Saúde (SUS) [Internet]. 2024 [accessed on Jul 29, 2024]. Available at: https://www.in.gov.br/en/web/dou/-/portaria-gm/ms-n-3.493-de-10-de-abril-de-2024-553573811
https://www.in.gov.br/en/web/dou/-/porta... . One of the components of co-financing for eSBs aims to stimulate the achievement of agreed indicators, encouraging the improvement of access and quality of services in PHC²⁰20 Brasil. Ministério da Saúde. Gabinete da Ministra. Portaria GM/MS no 3.493, de 10 de abril de 2024. Altera a portaria de consolidação GM/MS no 6, de 28 de setembro de 2017, para instituir nova metodologia de cofinanciamento federal do Piso de Atenção Primária à Saúde no âmbito do Sistema Único de Saúde (SUS) [Internet]. 2024 [accessed on Jul 29, 2024]. Available at: https://www.in.gov.br/en/web/dou/-/portaria-gm/ms-n-3.493-de-10-de-abril-de-2024-553573811
https://www.in.gov.br/en/web/dou/-/porta... .

Despite these monitoring and evaluation initiatives, oral health indicators have only covered some dimensions of service quality and have suffered from discontinuity, which may hinder the provision of services to meet the population's needs¹⁸18 França MASA, Freire MCM, Pereira EM, Marcelo VC. Oral health indicators in the Interfederative Pacts of the Unified Health System: development in the 1998-2016 period. Rev Odontol UNESP 2018; 47(1): 18-24. https://doi.org/10.1590/1807-2577.08417
https://doi.org/10.1590/1807-2577.08417... ,²¹21 França MASA, Freire MCM, Pereira EM, Marcelo VC. Indicadores de saúde bucal propostos pelo Ministério da Saúde para monitoramento e avaliação das ações no Sistema Único de Saúde: pesquisa documental, 2000-2017. Epidemiol Serv Saúde 2020; 29(1): e2018406. https://doi.org/10.5123/S1679-49742020000100002
https://doi.org/10.5123/S1679-4974202000... , constituting a problem that needs to be overcome¹⁴14 Brasil. Presidência da República. Casa Civil. Secretaria Especial para Assuntos Jurídicos. Lei no 14.572, de 8 de maio de 2023. Institui a Política Nacional de Saúde Bucal no âmbito do Sistema Único de Saúde (SUS) e altera a Lei no 8.080, de 19 de setembro de 1990, para incluir a saúde bucal no campo de atuação do SUS [Internet]. 2023 [accessed on Mar 30, 2024]. Available at: https://www.planalto.gov.br/ccivil_03/_ato2023-2026/2023/lei/l14572.htm
https://www.planalto.gov.br/ccivil_03/_a... . Therefore, ensuring the availability, analysis and use of information on the population's health conditions and the health system's response capacity to qualify management and care is a priority, promoting equity and universal access to oral health²²22 Organização Pan-Americana da Saúde. As funções essenciais de saúde pública nas américas - uma renovação para o século 21. Marco conceitual e descrição [Internet]. Washington: Organização Mundial da Saúde; 2022 [accessed on Mar 30, 2024]. Available at: https://iris.paho.org/bitstream/handle/10665.2/55678/9789275722657_por.pdf?sequence=1&isAllowed=y
https://iris.paho.org/bitstream/handle/1... . Additionally, the use of digital information and data management technologies can enhance the use of this information in the formulation of health policies, contributing to the digital transformation of SUS²³23 Brasil. Ministério da Saúde. Gabinete da Ministra. Portaria GM/MS no 3.232, de 1o de março de 2024. Altera a Portaria de Consolidação GM/MS no 5, de 28 de setembro de 2017 para instituir o Programa SUS Digital [Internet]. 2024 [accessed on Mar 30, 2024]. Available at: https://www.in.gov.br/web/dou/-/portaria-gm/ms-n-3.232-de-1-de-marco-de-2024-546278935
https://www.in.gov.br/web/dou/-/portaria... .

However, the fragmentation of monitoring systems and the lack of indicators to guide policies are recognized global challenges²⁴24 Agrasuta V, Thumbuntu T, Karawekpanyawong R, Panichkriangkrai W, Viriyathorn S, Reeponmaha T, et al. Progressive realisation of universal access to oral health services: what evidence is needed? BMJ Global Health 2021; 6(7): e006556. https://doi.org/10.1136/bmjgh-2021-006556
https://doi.org/10.1136/bmjgh-2021-00655... . Recently, the World Health Assembly recommended the implementation of effective surveillance and monitoring systems²⁵25 World Health Organization. Seventy-fourth world health assembly. Oral health [Internet]. 2021 [accessed on Jul 29, 2024]. Available at: https://apps.who.int/gb/ebwha/pdf_files/WHA74/A74_R5-en.pdf
https://apps.who.int/gb/ebwha/pdf_files/... . Accordingly, the World Health Organization (WHO) launched the Global Oral Health Status Report on a digital portal²⁶26 World Health Organization. Global oral health status report. Towards universal health coverage for oral health by 2030 [Internet]. Geneva: WHO; 2022 [accessed on Jul 29, 2024]. Available at: https://iris.who.int/bitstream/handle/10665/364538/9789240061484-eng.pdf?sequence=1
https://iris.who.int/bitstream/handle/10... , providing a comprehensive overview of the global burden of oral diseases based on the most recent data from the Global Burden of Disease (GBD) project, the International Agency for Research on Cancer (IARC), and WHO research. Other digital dashboards, such as that of the Centers for Disease Control and Prevention²⁷27 Centers for Disease Control and Prevention. Explore oral health data by topic [Internet]. 2024 [accessed on Jul 29, 2024]. Available at: https://nccd.cdc.gov/oralhealthdata/rdPage.aspx?rdReport=DOH_DATA.ExploreByTopic&islTopic=&islYear=
https://nccd.cdc.gov/oralhealthdata/rdPa... , were identified, and they present research results with available primary data. No international monitoring initiatives based on routine data from national health information systems were found.

Several initiatives have been developed in Brazil, such as the primary care indicator dashboards²⁸28 Brasil. Ministério da Saúde. Secretaria da Atenção Primária da Saúde. Painéis de indicadores da atenção primária [Internet]. 2024 [accessed on Mar 30, 2024]. Available at: https://sisaps.saude.gov.br/painelsaps/
https://sisaps.saude.gov.br/painelsaps/... , which provide indicators on PHC coverage, service production and family health, among others. These dashboards aim to promote knowledge about PHC, support decision-making and increase the transparency of the Department of Primary Health Care (SAPS), facilitating monitoring and evaluation. The LocalizaSUS panels, created by the Department of Monitoring, Evaluation and Dissemination of Strategic Health Information, of the Secretariat of Information and Digital Health (SEIDIGI) of the Ministry of Health, disseminate strategic data and information to managers, researchers and citizens. Oral health is already being developed in the PHC dashboards, but is not yet available in LocalizaSUS.

In this context, MonitoraSB was developed, a name for a proposal for monitoring oral health services in PHC, consisting of an evaluation matrix of indicators²⁹29 Ferreira RC, Chalub LLFH, Amaral JHL, Pinto RS, Santos JS, Campos FL, et al. Indicadores para monitoramento dos serviços de saúde bucal na Atenção Primária: validação de conteúdo e mensurabilidade. Cien Saude Colet 2023. Ahead of print. and new digital tools³⁰30 Ferreira RC, Houri LCLF, Senna MIB, Amaral JHL, Pinto RS, Ribeiro BA, et al. Painel de indicadores para o monitoramento dos serviços de saúde bucal na APS. [Internet]. 2023 [accessed on Mar 30, 2024]. Available at: https://lookerstudio.google.com/u/0/reporting/86c09403-f4a0-4625-ad1f-239daa77f6a2/page/aKVBD
https://lookerstudio.google.com/u/0/repo... ,³¹31 Houri LCLF, Ferreira RC, Ribeiro BA, Senna MIB, Amaral JHL, Pinto RS, et al. Calculadora de Indicadores de Saúde Bucal. Brasil registro de software [Internet]. 2023 [accessed on Oct 8, 2024]. Available at https://hernanebraga-ppsus-indicator-calculator-st-streamlit-app-xfuk1o.streamlit.app/
https://hernanebraga-ppsus-indicator-cal... . The development of this innovation aimed to contribute to changing the focus of evaluation in PHC, from achieving goals to the regular and continuous use of indicators for monitoring health services and planning at various levels of management, in addition to contributing to the process of monitoring oral health services in PHC in Brazil. This paper describes the methodology for developing MonitoraSB and discusses its characteristics and possible uses.

Process of developing and detailing MonitoraSB

MonitoraSB includes a matrix of indicators and digital tools developed collaboratively by dentists working in eSB in PHC and oral health managers in Minas Gerais, at the municipal and state levels.

Matrix of indicators: the indicators were developed based on the model for assessing the effectiveness of oral health care adapted from Nickel³²32 Nickel DA. Modelo de avaliação da efetividade da Atenção em Saúde Bucal [dissertação de mestrado]. Florianópolis: Universidade Federal de Santa Catarina (UFSC); 2008. and modified by Colussi³³33 Colussi CF. Avaliação da qualidade da atenção em saúde bucal em Santa Catarina [tese de doutorado]. Florianópolis: Universidade Federal de Santa Catarina (UFSC); 2010., in addition to the principles of the PNSB and the National Primary Care Policy. Two dimensions of quality of oral health services were considered, with subdimensions: oral health management (intersectoral action/popular participation, eSB work process) and provision of oral health services (access to oral health services, surveillance without oral health, promotion and prevention and diagnosis, treatment and rehabilitation in oral health)³²32 Nickel DA. Modelo de avaliação da efetividade da Atenção em Saúde Bucal [dissertação de mestrado]. Florianópolis: Universidade Federal de Santa Catarina (UFSC); 2008.,³³33 Colussi CF. Avaliação da qualidade da atenção em saúde bucal em Santa Catarina [tese de doutorado]. Florianópolis: Universidade Federal de Santa Catarina (UFSC); 2010.. The indicators presented content validity and measurability with data from the Health Information System for Primary Care (SISAB)²⁹29 Ferreira RC, Chalub LLFH, Amaral JHL, Pinto RS, Santos JS, Campos FL, et al. Indicadores para monitoramento dos serviços de saúde bucal na Atenção Primária: validação de conteúdo e mensurabilidade. Cien Saude Colet 2023. Ahead of print.. Details on the development of the indicators, including concepts of dimensions/subdimensions, have been presented in previous publications²⁹29 Ferreira RC, Chalub LLFH, Amaral JHL, Pinto RS, Santos JS, Campos FL, et al. Indicadores para monitoramento dos serviços de saúde bucal na Atenção Primária: validação de conteúdo e mensurabilidade. Cien Saude Colet 2023. Ahead of print.,³⁴34 Ferreira RC, Houri LCLF, Senna MIB, Amaral JHL, Pinto RS, Santos JS, et al. Dicionário de indicadores para a avaliação dos serviços de saúde bucal na Atenção Primária à Saúde. 1a ed. Belo Horizonte: FAO UFMG; 2023..

The indicators are calculated using routine data made available by SISAB, which are generated through the registration of user care in the PHC. These data are collected by the two e-SUS PHC software systems: the Electronic Citizen Record or Simplified Data Collection (non-computerized municipality)³⁵35 Brasil. Ministério da Saúde. Gabinete do Ministro. Portaria no 1.412, de 10 de julho de 2013. Institui o Sistema de Informação Em Saúde Para a Atenção Básica (SISAB) [Internet]. 2013 [accessed on Mar 30, 2024]. Available at: https://bvsms.saude.gov.br/bvs/saudelegis/gm/2013/prt1412_10_07_2013.html
https://bvsms.saude.gov.br/bvs/saudelegi... . In SISAB, the databases for the numerator and denominator of the indicators are obtained by consulting the health reports: production and collective activity. Data on the registered population are obtained from the registration panel in this same system. An open-access indicator dictionary presents each indicator's qualification sheets regarding measurement, interpretation, uses, limitations, calculation method, data source and parameter³⁴34 Ferreira RC, Houri LCLF, Senna MIB, Amaral JHL, Pinto RS, Santos JS, et al. Dicionário de indicadores para a avaliação dos serviços de saúde bucal na Atenção Primária à Saúde. 1a ed. Belo Horizonte: FAO UFMG; 2023..

Digital tools: MonitoraSB includes two complementary digital tools to operationalize the use of indicators by services: a dashboard and an indicator calculator. The dashboard presents indicators for Brazil, macro-regions, 27 federative units (FUs) and municipalities, according to the data available in SISAB. The construction involved a data scientist with a degree in Systems Engineering, and followed six steps:

1. Creation of a spreadsheet describing the calculation method, treatment of missing data and filters of the reports to be selected to obtain the numerator and denominator of the indicators;

2. Identification of the numerator and denominator databases to be extracted for the calculation;

3. Double validation of the extraction and calculation methods by the research team and the data scientist. In this stage, the researchers performed manual extraction, linked the databases, and calculated the indicators using Stata 18.0 software. The data scientist carried out the same process to identify inconsistencies and standardize the process;

4. Automation of file extraction and calculation of indicators, creating scripts using Python^® language;

5. Storage of the history of calculated indicators in an analytical database, that is, creation of the data infrastructure, a delta lake composed of the following data layers: Landing zone; Bronze layer; Silver layer; Gold layer;

6. Creation of the dashboard using data contained in the gold layer and making the calculated indicators available on the interactive dashboard for users.

The team defined the characteristics of the dashboard in terms of structure and organization (header, colors, access to the indicator qualification sheet — interactive exploration of the dashboard — drill-through), navigation and interactivity (navigation sidebar by dimension and, separately, for each indicator; presentation of indicators by region, FU and municipality; inclusion of annual and quarterly temporal filters (minimum period of data availability in SISAB at the time of development); presentation of results in graphs or maps for geographic and temporal comparisons; inclusion of socioeconomic (municipal Human Development Index; HDI) and demographic (population size) filters, enabling analysis of disparities in the results. For these last two filters, the population estimate databases made available by IBGE in each year and the HDI database, obtained by accessing the Human Development Atlas, were used. A pre-test of the dashboard's usability was carried out with five PHC dentists, resulting in changes to the design, specifically in the presentation of indicators by region and state, which were separated from the detailed information by municipality.

The calculator allows you to obtain indicators for each eSB using data from e-SUS PHC reports generated locally by managers or professionals. It was developed in Python^® using the open-source tool Streamlit, which allows you to create and host simple applications on the internet for free.

To illustrate the functionality of the dashboard and calculator, the results were shown for the indicator "Coverage of first programmatic dental consultation", from the subdimension "Access to oral health services", which measures the number of first programmatic dental consultations in the PHC in a given location and period, per registered population, in the same location and period (per thousand users).

Evaluation matrix

The evaluation matrix is composed of 54 indicators measurable through SISAB data and 15 validated and non-measurable with the available data, in the format current by the system, in April 2024. The objectives of the indicators were described in Chart 1 and the indicators measurable with SISAB data in Chart 2.

Indicator dashboard

The indicator panel, registered with the National Institute of Industrial Property (BR 51 2023 002193-8), is available for implementation at the link https://lookerstudio.google.com/reporting/86c09403-f4a0-4625-ad1f-239daa77f6a2/page/p_rpxxv2ub8c³⁰30 Ferreira RC, Houri LCLF, Senna MIB, Amaral JHL, Pinto RS, Ribeiro BA, et al. Painel de indicadores para o monitoramento dos serviços de saúde bucal na APS. [Internet]. 2023 [accessed on Mar 30, 2024]. Available at: https://lookerstudio.google.com/u/0/reporting/86c09403-f4a0-4625-ad1f-239daa77f6a2/page/aKVBD
https://lookerstudio.google.com/u/0/repo... . It is worth noting that there are possibilities of integrating MonitoraSB with other monitoring systems in force in Brazil. The panel is in the process of being incorporated by the General Coordination of Oral Health of the Ministry of Health (CGSB) as a tool to be recommended for monitoring oral health services in Brazil and will be hosted in the Strategic Management Room (LocalizaSUS/SEIDIGI). Indicators from the evaluation matrix are also expected to make up the oral health panel, which will integrate with the PHC monitoring system in the panels under construction.

The dashboard has navigation bars on the home page and on the sidebar, allowing quick and interactive navigation from one indicator to another (Figure 1). Navigation by the selected indicator can be performed, obtaining a view by region or state, according to the desired time filter. For example, Figure 2 illustrates the results of the indicator "Coverage of first programmatic dental consultation" for the year 2023, quarter 1 (Figure 2a).

The visualization shows the results for Brazil (12 dental appointments per thousand users) and for each region, with values ranging from 13.5 in the Northeast to 10.4 in the Southeast (per thousand users). The Brazil map shows each state's indicator values, representing a color scale from the lowest to the highest value. The results can also be viewed for each state in bar graphs, allowing comparison and analysis of geographic disparities in the indicators. For example, for this indicator, the lowest results were observed in Roraima and Acre, and the highest in Alagoas (Figure 2a). A detailed view can be seen by selecting a specific region (Figure 2b). When more than one four-month period is selected, the panel also allows temporal comparisons, that is, variations in the indicator over time, according to the selected geographic area. For example, Figure 2b shows the variation in the indicator since the first four-month period of 2020 for each of the states in the Southeast Region (Figure 2b). In this case, lower values of the indicator are observed in the second four months of 2020, coinciding with the interruption of dental services due to the COVID-19 pandemic. Temporal analysis can be useful for assessing trends, effects of health actions, policies and programs, and for resource planning. It is essential for monitoring services and understanding their functioning and response during events such as the pandemic or other situations that require resilience and the ability to adapt and reorganize services.

The analysis by municipality makes it possible to compare indicators according to geographic, demographic or socioeconomic characteristics (Figure 3). For example, Figure 3 illustrates the results of the indicator "Coverage of first programmatic dental consultation" for municipalities in Minas Gerais with up to 5,000 inhabitants and a very low HDI. In this example, the periods between the four-month periods of 2020Q1 and 2023Q1 were selected for temporal evaluation, evidencing a similar pattern of reduction in the indicator in 2020Q2 and many fluctuations, but with a pattern of increase in the number of first programmatic dental consultations for every thousand registered users in many municipalities after 2020Q2.

It was also observed that the indicator was not calculated in some municipalities in 2020Q1 and 2020Q2, evidenced by the timeline's interruption. In these municipalities and four-month periods, there was no record of first programmatic dental consultation. The heat map shows the municipalities with the greatest coverage in colors close to green and those with the lowest indicator values in red. It is worth noting that the map shows the indicator calculated considering the selected period, in the example, for all four-month periods. In addition, the diameter of the circle represents the population size of the municipality. The HDI filter could be used for comparative analysis of municipalities with other HDI values (Figure 3a).

An analysis can be performed by selecting only the municipality. In the example, Alvorada de Minas presented lower indicator values in the periods between 2020Q1 and 2021Q1, with a positive variation since this four-month period (Figure 3b). The dashboard is also a report and database generator, as it allows the download of calculated indicators in spreadsheets for the disaggregation levels of Brazil, region, state and municipality.

Calculator of indicators

The oral health indicator calculator is an open-access online tool (bit.ly/calculadora-indicadores-saude-bucal-ppsus). When accessing the calculator, the user will select the indicator they wish to calculate. Then boxes will appear on the screen to enter each indicator's numerator and denominator data. This data should be obtained from the e-SUS PHC management reports for each eSB. After entering the data and clicking on "Calculate Indicators", the results will be displayed. The user will also have the option to download the results in Excel or *csv format (Figure 4).

MonitoraSB: implications for monitoring and evaluating oral health services

This paper presents an innovative proposal for monitoring and evaluating public oral health services by developing a matrix of indicators and proposing two new, freely accessible digital tools to operationalize it. The matrix, based on national health production data, guarantees the availability of information³⁶36 Organização Pan-Americana da Saúde. Indicadores de saúde: elementos conceituais e práticos [Internet]. Washington: Organização Pan-americana de Saúde; 2019 [accessed on Mar 30, 2024]. Available at: https://www3.paho.org/hq/index.php?option=com_docman&view=download&category_slug=health-analysis-metrics-evidence-9907&alias=45251-indicadores-saude-elementos-conceituais-e-praticos-251&Itemid=270&lang=pt
https://www3.paho.org/hq/index.php?optio... , guiding political decisions and subsidizing social control. The indicators address various aspects of oral health care, from the coverage of services in PHC, both individual and collective, and allow for an understanding of the demand for specialized services, the predominant care model and the rates of care for specific conditions²⁹29 Ferreira RC, Chalub LLFH, Amaral JHL, Pinto RS, Santos JS, Campos FL, et al. Indicadores para monitoramento dos serviços de saúde bucal na Atenção Primária: validação de conteúdo e mensurabilidade. Cien Saude Colet 2023. Ahead of print..

By expanding the scope of indicators related to the provision and management of oral health, the matrix offers new possibilities compared to existing initiatives until now, which focused on data related to access to and coverage of oral health services and which have been reduced over the years¹⁸18 França MASA, Freire MCM, Pereira EM, Marcelo VC. Oral health indicators in the Interfederative Pacts of the Unified Health System: development in the 1998-2016 period. Rev Odontol UNESP 2018; 47(1): 18-24. https://doi.org/10.1590/1807-2577.08417
https://doi.org/10.1590/1807-2577.08417... ,²¹21 França MASA, Freire MCM, Pereira EM, Marcelo VC. Indicadores de saúde bucal propostos pelo Ministério da Saúde para monitoramento e avaliação das ações no Sistema Único de Saúde: pesquisa documental, 2000-2017. Epidemiol Serv Saúde 2020; 29(1): e2018406. https://doi.org/10.5123/S1679-49742020000100002
https://doi.org/10.5123/S1679-4974202000... .

The digital tools operationalize the calculation of indicators and their use by services. By allowing temporal and spatial cuts, the panel facilitates the comparison of results at multiple levels and favors the continuous monitoring of oral health services, allowing for the evaluation, transparency and continuous reorientation of interventions.

As it uses the SUS information systems, it offers a perspective based on reality and enables the qualification of records. This tool also strengthens the use of health information systems, aligning with the implementation of SUS Digital Brasil²³23 Brasil. Ministério da Saúde. Gabinete da Ministra. Portaria GM/MS no 3.232, de 1o de março de 2024. Altera a Portaria de Consolidação GM/MS no 5, de 28 de setembro de 2017 para instituir o Programa SUS Digital [Internet]. 2024 [accessed on Mar 30, 2024]. Available at: https://www.in.gov.br/web/dou/-/portaria-gm/ms-n-3.232-de-1-de-marco-de-2024-546278935
https://www.in.gov.br/web/dou/-/portaria... . By including data fed by all Brazilian municipalities and being an open tool, the MonitoraSB panel democratizes access to information.

As they use Delta Lake in their architecture for data extraction and storage, digital tools favor an innovative ecosystem for storing large volumes and multiple forms of information processing³⁷37 Näher AF, Vorisek CN, Klopfenstein SAI, Lehne M, Thun S, Alsalamah S, et al. Secondary data for global health digitalisation. Lancet Digit Health 2023; 5(2): e93-e101. https://doi.org/10.1016/S2589-7500(22)00195-9
https://doi.org/10.1016/S2589-7500(22)00... ,³⁸38 Brasil. Ministério da Saúde. Secretaria-Executiva. Departamento de Informática do SUS. Estratégia de saúde digital para o Brasil 2020-2028 [Internet]. Brasília: Ministério da Saúde; 2020 [accessed on Mar 30, 2024]. Available at: https://bvsms.saude.gov.br/bvs/publicacoes/estrategia_saude_digital_Brasil.pdf
https://bvsms.saude.gov.br/bvs/publicaco... . This favors the timeliness of information accessed by users through the provision of new data by SISAB and ensures that the right information is available at the right time and in the right place for evidence-based decision-making³⁹39 Mendes EV. As redes de atenção à saúde. Ciên Saúde Coletiva 2010; 15(5): 2297-305. https://doi.org/10.1590/S1413-81232010000500005
https://doi.org/10.1590/S1413-8123201000... .

With a user-friendly interface, the dashboard allows quick visualization of results and multiple comparisons over time by region, population size, and municipal human development index bands. Thus, after selecting the indicator, the tool offers a certain degree of customization and automatic visual resources. Considering the health situation room as a space for systematic analysis of information⁴⁰40 Brasil. Ministério da Saúde. Organização Pan-Americana da Saúde. Salas de situação em saúde: compartilhando as experiências do Brasil [Internet]. Brasília: Ministério da Saúde; 2010 [accessed on Mar 30, 2024]. Available at: https://bvsms.saude.gov.br/bvs/publicacoes/sala_situacao_saude_2010.pdf
https://bvsms.saude.gov.br/bvs/publicaco... , this tool can be understood as such or even as support for local technical teams, providing continuous monitoring of indicators for planning and rapid identification of non-standard occurrences⁴¹41 Grimm SCA, Tanaka OY. Painel de monitoramento municipal: bases para a construção de um instrumento de gestão dos serviços de saúde. Epidemiol Serv Saúde 2016; 25(3): 585-94. https://doi.org/10.5123/S1679-49742016000300014
https://doi.org/10.5123/S1679-4974201600... , assisting in oral health surveillance. In addition to the existing customization, it is possible to calculate indicators individually with the digital calculator, allowing for more specific cuts. These tools help to overcome the fragmentation of oral health data and seek equity in access to information and technologies. These aspects are essential to improve PHC and increase its resilience, especially in a country with great inequalities in the use and access to technologies by professionals and where digital exclusion is still prominent²2 Massuda A, Bigoni A, Paschoalotto MA, Tasca R. Rumos para um sistema de saúde resiliente. GV-Executivo 2022; 21(2): 22-9. https://doi.org/10.12660/gvexec.v21n2.2022.85938
https://doi.org/10.12660/gvexec.v21n2.20... .

However, some challenges must be faced, including the need to resize the indicator matrix, which the use of tools in real contexts should facilitate. On the other hand, the expanded scope of indicators can contribute to changing the monitoring and evaluation models that previously existed in Brazil, which are often linked to meeting targets linked to financial transfers.

From this perspective, MonitoraSB offers possibilities for integration with existing monitoring systems, especially through CGSB and SEIDIGI, as already mentioned. In addition, some of its features, such as the use of data science in its development and the possibility of obtaining information by the municipality in *csv format, also favor interoperability with initiatives such as PROADESS. In this case, one of the possibilities is using health system performance indicators containing geospatial information to understand oral health indicators better⁴²42 Romão AR, Oliveira RAD, Barros H, Carvalho CC. Interoperabilidade do sistema de mapas interativos do PROADESS com Sistemas de Informações Geográficas [Internet]. PROADESS; 2024 [accessed on Mar 30, 2024]. Available at: https://www.proadess.icict.fiocruz.br/NT4interoperabilidadeSIG-PROADESS_junho2024.pdf
https://www.proadess.icict.fiocruz.br/NT... .

Another challenge is the source of the data, which affects the quality of the records in the information systems, which may contain inconsistencies, conflicting information, missing data or even being affected by under-recording. Given its nature, the lowest level of disaggregation of the indicators in the panel is at the municipal level. However, the vast majority of Brazilian municipalities are small, around 70% of which have less than 20 thousand inhabitants, and tools like this can offer significant progress in their management⁴³43 Instituto Brasileiro de Geografia e Estatística. Censo demográfico [Internet]. 2022 [accessed on Mar 30, 2024]. Available at: https://censo2022.ibge.gov.br/panorama/
https://censo2022.ibge.gov.br/panorama/... . The calculator sought to overcome this limitation.

Adopting digital tools in PHC health services is a challenge to overcome. An implementation study in municipal contexts is underway to assess the acceptability, adoption, feasibility, suitability, and sustainability of SUS professionals’ use of MonitoraSB. This study also analyzed its impact on the outcomes of health services, observing the variation of the indicators themselves over time and the quality of data recording in e-SUS PHC. It is also expected that the use of indicators in different contexts will allow for the discussion and definition of acceptable or desirable quality parameters for their measurement.

MonitoraSB provides indicators to monitor the oral health of SUS users, improving health planning and contributing to the advancement of public policies. It provides dynamic and real-time information for managers, academics, social movements and decision-makers. It is expected that the implementation of MonitoraSB will promote the strengthening of PHC and PNSB, contributing to the consolidation of a universal, comprehensive and equitable oral health care model.

REFERENCES

Publication Dates

History