Abstract

This is a methodological study for the development and validation of the Multidimensional Instrument for Evaluating of the Implementation of Psychosocial Care Network (IMAI-RAPS) in Minas Gerais (MG)/Brazil. The study was carried out in three stages: evaluability study, development of the IMAIRAPS, application of the Delphi Technique for content and appearance validation of the questions. The analysis of official documents, literature review and a structured engagement with program members were carried out to clarify its operationalization and focus on the central aspects to be evaluated. A theoreticallogical model of RAPS was built according to the Donabedian triad: structure, process and result and organized into: Minimum Units (Mental Health Care and Psychosocial Rehabilitation), Connectivity (Network Articulation), Integration (Governance and Management of the Care), Normativity (Mental Health Policy and Participation and Social Control), Subjectivity and Structure (Services, Logistics System and Health Education). The IMAI-RAPS was derived from this model, which was validated by 44 experts in the field, indicating the approach of relevant, useful and viable questions for evaluating the structure and process of implementing the program in MG. The use of the Delphi Technique made it possible for the developed products to be marked out by Psychosocial Care Network scholars or professionals from different regions of the country, increasing the analytical power of the tool.

Keywords:
Mental Health; Health Evaluation; National Health Systems; Health Policy

Introduction

Mental health represents one of the highest costs for global public health, with an estimated 13% of the global burden of diseases due to mental and neurological disorders and/or disorders resulting from abuse of alcohol or other drugs (VIGO; THORNICROFT; ATUN, 2016VIGO, D.; THORNICROFT, G.; ATUN, R. Estimating the true global burden of mental illness. Lancet Psychiatry, v. 3, n. 2, p. 171–178, fev. 2016.). This high – and increasing – global burden is associated with several factors, such as the widespread prevalence, early onset, and chronic course of these disorders, as well as the considerable current gap in treatment (KOHN et al., 2018KOHN, R. et al. Mental health in the Americas: an overview of the treatment gap. Revista Panamericana de Salud Pública, v. 42, p. e165, out. 2018.). Thus, there is an urgent need to strengthen the capacity of health systems to respond to these disorders effectively (ZHOU et al., 2018ZHOU, W. et al. Policy development and challenges of global mental health: a systematic review of published studies of national-level mental health policies. BMC Psychiatry, v. 18, n. 1, p. 138, dez. 2018.).

Latin America has made significant progress in improving mental health services in recent decades. These improvements, following changes made earlier in wealthier countries, centered on replacing a focus on hospital care with healthcare structured on community services integrated into primary healthcare (PHC) and emphasizing the protection of human rights and the agency of the users (CALDAS DE ALMEIDA, 2013ALMEIDA, J. M. Mental health services development in Latin America and the Caribbean: achievements, barriers and facilitating factors. International Health, v. 5, n. 1, p. 15-18, mar. 2013.; PATEL et al., 2018). In Brazil, a national mental health policy was legislated in 2001, driven by the Psychiatric Reform movement, which advocated for a community model of care. Since then, there have been several changes in the implementation of territorial services and clinical practices based on psychosocial care, leading the Brazilian experience to occupy a prominent place in the field of global mental health (JACOB et al., 2007JACOB, K. S. et al. Mental health systems in countries: where are we now? The Lancet, v. 370, n. 9592, p. 1061-1077, set. 2007.; ALMEIDA, 2019ALMEIDA, J. M. C. de. Política de saúde mental no Brasil: o que está em jogo nas mudanças em curso. Cadernos de Saúde Pública, v. 35, n. 11, p. e00129519, abr. 2019.).

In 2010, in an effort to further develop decentralization and regionalization strategies to reduce the fragmentation of care in the Brazilian unified health system (Sistema Único de Saúde – SUS), the healthcare networks were restructured, integrating actions and services across several levels of complexity (BRASIL, 2010BRASIL. Portaria no 4.279, de 30 de dezembro de 2010. Estabelece diretrizes para a organização da Rede de Atenção à Saúde no âmbito do Sistema Único de Saúde (SUS). Diário Oficial da União, Brasília, DF, 31 dez. 2010.). In this context, a psychosocial care network (Rede de Atenção Psicossocial - RAPS) was established to improve and coordinate between various healthcare entities, including primary healthcare (Atenção Primária à Saúde – APS), specialized psychosocial care, urgent and emergency care, transient residential care, and hospital care, with an emphasis on deinstitutionalization strategies and psychosocial rehabilitation (BRAZIL. MINISTÉRIO DA SAÚDE, 2011BRASIL. Portaria nº 3.088, de 23 de dezembro de 2011. Institui a Rede de Atenção Psicossocial para pessoas com sofrimento ou transtorno mental e com necessidades decorrentes do uso de crack, álcool e outras drogas, no âmbito do Sistema Único de Saúde. Diário Oficial da União, Brasília, DF, 26 dez. 2011.; MOREIRA; ONOCKO-CAMPOS, 2017ONOCKO-CAMPOS, R. T. et al. Atuação dos Centros de Atenção Psicossocial em quatro centros urbanos no Brasil. Revista Panamericana de Salud Pública, v. 42, p. e113, out. 2018.). There was also a focus on expanding existing services, providing benefits for vulnerable populations such as users of alcohol or other drugs, children and adolescents, homeless individuals, and indigenous populations, as well as increasing access and linking users to healthcare providers (BRAZIL. MINISTÉRIO DA SAÚDE, 2011BRASIL. Portaria nº 3.588, de 21 de dezembro de 2017. Altera as Portarias de Consolidação nº 3 e nº 6, de 28 de setembro de 2017, para dispor sobre a Rede de Atenção Psicossocial, e dá outras providências. Diário Oficial da União: seção 1, Brasília, DF, p. 236, 22 dez. 2017.).

However, it has proven to be a complex task to provide integrated mental healthcare that is universally accessible to the Brazilian population, as well as being anchored in multidisciplinary work and expanded clinical practice. The existence of gaps between the political-assistance guidelines and daily practice has been noted in previous studies (FIOCRUZ. FUNDAÇÃO CALOUSTE GULBENKIAN, 2015FIOCRUZ; FUNDAÇÃO CALOUSTE GULBENKIAN. Inovações e Desafios em Desinstitucionalização e Atenção Comunitária no Brasil. Seminário Internacional de Saúde Mental: Documento Técnico Final. Rio de Janeiro: Fiocruz. Fundação Calouste Gulbenkian. Organização Mundial da Saúde. Ministério da Saúde, 2015.), raising questions about the scope of the changes made and highlighting the need for an evaluation that can find the answers (MOREIRA; BOSI, 2019MOREIRA, D. D. J.; BOSI, M. L. M. Qualidade do cuidado na Rede de Atenção Psicossocial: experiências de usuários no Nordeste do Brasil. Physis: Revista de Saúde Coletiva, v. 29, n. 2, p. 1-22, fev. 2019.).

Evaluations of the Brazilian PCN fall short of the mark in spite of an expansion of studies, as demonstrated in a recent systematic review by Costa et al. (2015). Most research still focuses on isolated services or specific populations. Likewise, few studies use standardized and validated instruments to evaluate the PCN (COSTA; COLUGNATI; RONZANI, 2015COSTA, P. H. A. da; COLUGNATI, F. A. B.; RONZANI, T. M. Avaliação de serviços em saúde mental no Brasil: revisão sistemática da literatura. Ciência & Saúde Coletiva, v. 20, n. 10, p. 3243-3253, out. 2015.). Moreover, most of these studies deal with specific themes and services, such as the assessment scale of patient satisfaction with mental health services (SATIS-BR), the assessment scale of the workplace impact on mental health services (IMPACTO-BR) (BANDEIRA; PITTA; MERCIER, 2000BANDEIRA, M.; PITTA, A. M. F.; MERCIER, C. Escalas brasileiras de avaliação da satisfação (SATIS-BR) e da sobrecarga (IMPACTO-BR) da equipe técnica em serviços de saúde mental. Jornal Brasileiro de Psiquiatria, v. 49, n. 4, p. 105-115, abr. 2000.; BANDEIRA; SILVA, 2012BANDEIRA, M.; SILVA, M. A. da. Escala de Satisfação dos Pacientes com os Serviços de Saúde Mental (SATIS-BR): estudo de validação. Jornal Brasileiro de Psiquiatria, v. 61, n. 3, p. 124-132, jul. 2012.), and an instrument to evaluate the structure and process of mental healthcare in psychosocial care centers (Centros de Atenção Psicossocial – CAPS) (AZEVEDO; SALVETTI; TORRES, 2017AZEVEDO, D. M. de; SALVETTI, M. D. G.; TORRES, G. V. de. Qualidade da assistência em saúde mental: desenvolvimento de um instrumento de avaliação. Revista de Enfermagem do Centro-Oeste Mineiro, v. 7, p. e1685, dez. 2017.). Thus, instruments that can holistically evaluate the Brazilian PCN continue to be a significant gap in the literature (AMARAL; BOSI, 2017AMARAL, C. E. M.; BOSI, M. L. M. O desafio da análise de redes de saúde no campo da saúde coletiva. Saúde e Sociedade, v. 26, n. 2, p. 424-434, jun. 2017.; ONOCKO-CAMPOS et al., 2018).

Therefore, this study aimed to develop and validate a structured and reproducible multidimensional instrument for evaluating the structure and process (DONABEDIAN, 1978DONABEDIAN, A. The quality of medical care. Science, v. 200, n. 4344, p. 856-864, maio. 1978.) of the PCN and its implementation in the Brazilian state of Minas Gerais (MG). This instrument was designed to identify the weaknesses and strengths of the PCN implementation in order to support management and decision-making, as well as fostering a dialogue on the findings from different regions and enabling comparisons over time.

Methodology

This methodological study was conducted using a quantitative approach and a cross-sectional design to validate a multidimensional questionnaire with respect to its content and layout. The questionnaire is an instrument designed to analyze the implementation of the PCN according to the dimensions of structure and process (CHAMPAGNE et al., 2011CHAMPAGNE, F. et al. Análise da implantação. In: BROUSSELLE, A. et al. (Org). Avaliação: Conceitos e Métodos. 1. ed. Rio de Janeiro: Ed. Fiocruz, 2011. p. 292.). Methodological studies use a variety of methods to obtain, organize, and analyze data on the creation and validation of research instruments (POLIT; YANG, 2016POLIT, D.; YANG, F. Measurement and the measurement of change. China: Wolters Kluwer, 2016.). Thus, this study was carried out in three stages: (1) evaluability assessment, (2) development of the measurement instrument, and (3) application of the Delphi technique to validate the content and appearance of the questions.

For the evaluability assessment, a stage consisting of the evaluation plan, a methodology based on the seven-element system of Thurston and Ramaliu (2005), was used as follows: (1) identification and review of available documents; (2) description of the program identifying its goals, objectives, and activities; (3) construction of the theoretical-logical model with a graphical representation of available resources, intended activities, expected impacts, and assumed causal connections; (4) preliminary understanding of how the program operates; (5) development of a theoretical evaluation model; (6) identification of evaluation users and other key stakeholders; and (7) obtaining an agreement on the evaluation procedure.

The first step consisted of identifying people who were involved or affected by the program and who would be able to support and contribute to the evaluation. These stakeholders were chosen to participate in the evaluation plan using the criteria proposed for evaluating public health programs (CENTERS FOR DISEASE CONTROL AND PREVENTION, 1999CENTERS FOR DISEASE CONTROL AND PREVENTION. Framework for Program Evaluation in Public Health. Morbidity and mortality weekly report, v. 48, n. 11, p. 1-58, set. 1999.). The stakeholders selected included agents involved in the planning and coordination of the assessed program, active health professionals, and members of the academic community. The convenience sample consisted of members of the federal, state (Minas Gerais), and local (Belo Horizonte, the state capital) mental health agencies, as well as regional technical offices of the MG Health Department. The bulk of participants was involved in the Minas Gerais PCN since the aim was to advance research into the implementation of the network in the state (CHAMPAGNE et al., 2011).

From January to July 2019, a review was carried out of the scientific literature as well as documents available on government websites relating to the technical and legal frameworks, covering the period from 1991 to 2019. Information on how the PCN operates and the stakeholders involved was of vital importance for describing the program, preparing the theoretical-logical model, and coming to an agreement on the evaluation method.

The PCN theoretical-logical model was based on Amaral and Bosi's (2017) conceptual-analytical model of health networks, which contains five dimensions: minimum units, connectivity, integration, normativity, and subjectivity. This study considered the five dimensions as components, which were further divided into subcomponents. For each subcomponent, the necessary activities (process) to achieve the expected short and medium-term results by applying resources (structure) were listed (CHAMPAGNE et al., 2016CHAMPAGNE, F. et al. Modelizar as intervenções. In: BROUSSELLE, A. et al. (Org). Avaliação: conceitos e métodos. 2. ed. Rio de Janeiro: Editora Fiocruz, 2016. p. 61-74b.). As described later, researchers and stakeholders participated in the model review, re-elaboration, and consensus.

The multidimensional questionnaire was thus formed using 82 evaluative questions on the basis of the theoretical-logical model, 11 of which related to structural dimensions and 71 to process dimensions. The 71 questions on process dimensions were subdivided into five components: 34 on minimum units, 14 on connectivity, six on integration, nine on normativity, and eight on subjectivity. The instrument followed the plan proposed in the design of the PCN and was structured to be answered by individuals involved in mental health programs in the municipalities of MG.

The content and appearance of the questions were validated using the Delphi technique, a method of decision-making used by groups of judges aiming to reach a consensus (JÜNGER et al., 2017JÜNGER, S. et al. Guidance on Conducting and REporting DElphi Studies (CREDES) in palliative care: Recommendations based on a methodological systematic review. Palliative Medicine, v. 31, n. 8, p. 684-706, set. 2017.). The criteria used to select judges in this study followed the recommendations of Scarparo et al. (2012)SCARPARO, A. F. et al. Reflexões sobre o uso da técnica Delphi em pesquisas na enfermagem. Rev Rene, v. 13, n. 1, p. 242-251, jan. 2012., prioritizing working experience, master’s or doctoral studies, or academic positions involving scientific activities in the field over the last five years. The judges chosen included members of the national mental health agency, the MG state mental health agency, and regional technical mental health offices, who also indicated other professionals with suitable experience and knowledge of the Brazilian PCN. In addition, the database of the National Council for Scientific Development (Conselho Nacional de Desenvolvimento Científico e Tecnológico – CNPq) was accessed through the Lattes Platform to search for the most active and influential Brazilian researchers in the field of the study. This search aimed to compose a panel of judges with a variety of experience and professional activities.

The selected judges were all sent the study’s objectives and methodology, the justification of the validation process, and a request for their participation in the study. Subsequently, the form containing the questions of the multidimensional questionnaire was sent using Google Forms®, with each judge asked to consider every question in light of the criteria proposed by PASQUALI (2010) relating to relevance, objectivity, and clarity. The responses were organized on a five-level Likert scale: 1, strongly agree (SA); 2, agree (A); 3, neither agree nor disagree (NN); 4, disagree (D); and 5, strongly disagree (SD). Each question also had a space for judges to suggest changes or comments.

Four attempts were made to send the form between 11/19/19 and 12/20/19. Judges who did not submit their responses to the form during this period were excluded from the sample. Before participating in this study, the experts signed an informed consent form, and their anonymity was guaranteed throughout the entire process.

The content validity index (CVI) was used to identify the degree of agreement among the experts. Although the cutoff points used in the literature to denote consensus vary between 50 and 80% (JÜNGER et al., 2017), this study uses a cutoff point of 75% agreement. The methodology specifies that up to three questionnaire rounds must be conducted among the participants if the 75% consensus agreement is not reached in the first round. The CVI was calculated using the sum of responses classified as SA and A divided by the total number of responses. The content validity ratio (CVR) was also calculated to compare the sum of responses classified as SA and A with the value expected due to random chance. If there is a likelihood of over 5% that the amount of agreement observed is due to random chance, the item must be eliminated or revised. Finally, a Kappa coefficient of agreement was calculated as the ratio between the proportion of times that the judges did agree and the maximum proportion of times that the judges could agree. Kappa values range from -1 (total absence of an agreement) to 1 (total agreement) (ALEXANDRE; COLUCI, 2011ALEXANDRE, N. M. C.; COLUCI, M. Z. O. Validade de conteúdo nos processos de construção e adaptação de instrumentos de medidas. Ciência & Saúde Coletiva, v. 16, n. 7, p. 3061-3068, jul. 2011.).

This study was approved by the human research ethics committee under registry number CAAE: 77798217.1.3001.5091 and is part of the first author’s doctoral research. The study received funding from the Research Support Foundation of MG (Fundação de Amparo à Pesquisa de MG – FAPEMIG).

Results

Program description

The provision of psychosocial care by the territorial services of the Brazilian SUS began in the 1970s, when institutionalization was the primary way of approaching people in mental distress (FIOCRUZ. Fundação Calouste Gulbenkian, 2015). After stalling in the country’s legislature for 12 years, Law No. 10,216 was passed in 2001 and has provided a legislative framework and new direction for the psychosocial care model.

The new care policy was initially based on creating territorial psychosocial care centers (Centro de Atenção Psicossocial – CAPS), as illustrated in Figure 1. These services were established as an intermediate step between outpatient clinics and hospitalization and have progressively increased their level of complexity, adding crisis management, 24-hour operation with overnight admission, and monitoring of critically ill patients after stabilization and their reinsertion into society. Thus, the CAPS subverted the logic of the existing hierarchy, bringing together several levels of healthcare in a single unit (QUINDERÉ; JORGE; FRANCO, 2014QUINDERÉ, P. H. D.; JORGE, M. S. B.; FRANCO, T. B. Rede de Atenção Psicossocial: qual o lugar da saúde mental? Physis: Revista de Saúde Coletiva, v. 24, n. 1, p. 253-271, mar. 2014.; ONOCKO-CAMPOS et al., 2018). CAPS were further diversified through the creation of specific services for people with mental distress from alcohol or other drug use (CAPS AD) and children and adolescents (CAPSi). Meanwhile, several strategies were instituted to expand the new care model, such as the restructuring of hospital psychiatric care to favor small hospitals and short hospitalizations; the development of criteria to protect users’ rights; the creation of therapeutic residential services (TRS) and rehabilitation assistance for long-term hospitalizations (Return Home Program); incentives for the creation of social cooperatives; the integration of mental health care into the primary healthcare system (APS); plans to care for children, adolescents, and homeless individuals in mental distress; integration with emergency services; and the inclusion of backup mental health beds in general hospitals (BRAZIL, 2015BRASIL. MINISTÉRIO DA SAÚDE. Saúde Mental em Dados - 12, ano 10, no 12. Informativo Eletrônico. Brasília, 2015 (acesso em: 21/12/2020)).

According to the report Mental Health in Data by the Brazilian Health Department (HD), periodically released until 2015, the implementation of the PCN accelerated considerably since 2001, with 86% of the population covered by 2014. also compares the number of services implemented in Brazil between 2002 and 2014, reflecting the implementation of a vast network of psychosocial care services with the redistribution of financial resources from the hospital network to community services. In this period, the budget allocated to hospitals decreased from 95% of total spending on mental health to less than 30%, while overall spending on public mental health care increased (BRAZIL, 2015).

The so-called “new” national mental health policy (Política Nacional de Saúde Mental – PNSM) was launched in 2017 after changes in the political direction of the federal government. Professionals and researchers involved in the public provision of mental health services in Brazil saw this policy as a disruption of a policy making tradition that sought consensus with the social organizations historically involved in the process. While the established basis of care provision was not changed, the policy was seen as strengthening a logic of specialization and hospital-centricity through the inclusion of psychiatric hospitals and mental health outpatient clinics (multi-professional specialized mental healthcare team) in the list of components of the PCN, as well as the replacement of beds in general hospitals with specialized wards with up to 30 beds, public funding for the acquisition of electroconvulsive therapy equipment, and the maintenance and financing of therapeutic communities (ALMEIDA, 2019ALMEIDA, J. M. C. de. Política de saúde mental no Brasil: o que está em jogo nas mudanças em curso. Cadernos de Saúde Pública, v. 35, n. 11, p. e00129519, abr. 2019.; BASTOS, 2019BASTOS, F. I. Políticas de drogas no Brasil contemporâneo: aportes da ciência, da clínica e do liberalismo moderno. Cadernos de Saúde Pública, v. 35, n. 11, p. e00125519, set. 2019.; ONOCKO-CAMPOS, 2019ONOCKO-CAMPOS, R. T. Saúde mental no Brasil: avanços, retrocessos e desafios. Cadernos de Saúde Pública, v. 35, n. 11, p. e00156119, jan. 2019.).

Since then, the data on the PCN provided by the Brazilian Health Department has been limited. Although the regulations surrounding the new PNSM have not been revoked, the technical note that was published describing the intended changes was removed from its official website after protests, leaving doubts regarding the magnitude of the changes that will be implemented in the future (ONOCKO-CAMPOS, 2019ONOCKO-CAMPOS, R. T. Saúde mental no Brasil: avanços, retrocessos e desafios. Cadernos de Saúde Pública, v. 35, n. 11, p. e00156119, jan. 2019.).

Understanding the operationalization of the program, identifying those interested in the evaluation, and obtaining an agreement on the evaluation

Throughout the study, several meetings were held between the research group (teachers, technical support scholarships, master’s members, and senior students under scientific initiation) and the stakeholders. Two meetings were conducted with members of the national mental health agency, eight with members of the MG state mental health agency, three with the 28 regional technical mental health offices in MG, and one meeting with the mental health agency of Belo Horizonte. The discussions considered the opinions of all participants involved in the operationalization of the PCN and the main points to be addressed in the research. Participants attended the working meetings and provided feedback as well as information and administrative documents. This involvement made it possible for researchers to approach practical aspects of the operationalization of PCN, helping to address the usefulness, feasibility, property, and accuracy quality standards of evaluative research (CENTERS FOR DISEASE CONTROL AND PREVENTION, 1999CENTERS FOR DISEASE CONTROL AND PREVENTION. Framework for Program Evaluation in Public Health. Morbidity and mortality weekly report, v. 48, n. 11, p. 1-58, set. 1999.; YARBROUGH et al., 2010YARBROUGH, D. B.; SHULA, L. M.; HOPSON, R. K.; CARUTHERS, F. The Program Evaluation Standards: A guide for evaluators and evaluation users. 3a ed. Thousand Oaks, CA: Corwin Press, 2010.; CRAIG; CAMPBELL, 2015CRAIG, P.; CAMPBELL, M. Evaluability Assessment: a systematic approach to deciding whether and how to evaluate programmes and policies: a What Works Scotland Working Paper. Glasgow, 2015.). In addition, stakeholders who showed an interest in the PCN evaluation – even if they did not directly participate in the study – were identified. These interested individuals included 1) national and state technical offices interested in the evaluation as a management tool to monitor the progress of the PCN implementation, diagnose gaps in care, foster regional cooperation in the use of mental health services, and allocate financial investments; 2) regional technical offices seeking to use the results to support municipalities, diagnose gaps in care, and foster regional cooperation in the use of services; 3) municipal technical offices seeking better information on the region’s PCN, as this would help improve and develop new strategies for inter-municipal cooperation in the sharing of services; 4) professionals within the PCN who are interested in better understanding the regional PCN, since this would help improve their local activities; 5) researchers aiming to use the information generated in the evaluation of the PCN to advance research into the public mental healthcare provided by the SUS and expand efforts to evaluate the Brazilian PCN; 6) users and their relatives advocating for improvement in mental healthcare.

Elaboration of the theoretical-logical model for the development of the multidimensional questionnaire

The theoretical-logical model is a visual representation of the functioning of the program evaluated and an objective analysis of the presumed causal relationships between its elements, demonstrating how they work together to solve the problem that gave rise to the intervention (CHAMPAGNE et al., 2016). For the modeling of the PCN, the “Minimum Units” component, including the subcomponents of mental healthcare and psychosocial rehabilitation, was structured as the interventions that were essential for implementing the PCN as originally proposed (). Although Amaral and Bosi (2017) privileged the network’s services in this component, we chose to follow the Donabedian triad of structure/process/result to allocate the services, human resources, logistics system, and health education of professionals in the “Structure” component (DONABEDIAN, 1978DONABEDIAN, A. The quality of medical care. Science, v. 200, n. 4344, p. 856-864, maio. 1978.). The “Connectivity” component (subcomponent: network articulation) detailed the intra-sectorial and inter-sectorial actions, including the collaborative care actions. The “Integration” component included the subcomponents: governance and care management. This arrangement encompasses the relationships between the services and the interventions carried out over time for the continuous monitoring of each user. For the “Normativity” component (sub-components: mental health policy and social participation and control), the PNSM was considered the guideline for PCN orientation. It should be noted that the PCN, like other healthcare networks, presupposes an integrated operation with its components intertwining and generating indirect causal relationships across various components. However, for better visualization, only the presumed direct relationships were displayed, as shown in Figure 2.

Using the Delphi technique to validate the content and appearance of the multidimensional instrument for evaluating the implementation of the PCN (IMAI-RAPS)

Of the 90 invited judges, 44 (48.9%) evaluated the questions extracted from the theoretical-logical model. The age of the judges ranged from 30 to 67 years, with an average of 44.5 years, and 59.1% were female. Postdoctoral researchers made up 9.1% of the judges, with 22.7% reporting themselves as having a doctoral degree, 25% having a master’s degree, 34.1% a postgraduate certification, and 9.1% as having completed the undergraduate level. A majority (63.6%) of judges were psychologists, with nurses making up 13.6%, medical doctors 11.4%, and the remaining 11.4% split between social workers, physiotherapists, occupational therapists, and pharmacists. Concerning the relationship between the judges and the PCN, 47.7% of the specialists are professionals within the network, 34.1% are researchers, and the remaining 18.2% are professionals who work or have experience in management, consulting or coordination related to the PCN. It should be noted that prominent researchers from several regions of the country were among the participants in the group.

All of the questions presented to the judges were validated because all of them received an agreement degree (CVI) above the establiShed threshold of 0.75, with satisfactory values in the other tests. The lowest values for objectivity were 0.77 CVI, 0.55 CVR, and Kappa 0.78. The analysis of the clarity criterion, in turn, revealed that one item was not sufficiently straightforward (CVI: 0.75, CVR: 0.50, and Kappa: 0.76), leading to the exclusion of the question. Comments from several evaluators that certain questions were unnecessary or redundant led to the exclusion of 26 more questions. In addition, most judges suggested excluding the questions related to the subjectivity component, despite being validated, noting that qualitative techniques are needed due to the nature of the subject. Only one question from the subjectivity component was maintained and relocated in the normativity component, as it relates to the sub-component “participation and social control.”

The instrument was submitted to other changes suggested by the judges, such as minor alterations in the text and splitting two questions into four to avoid misinterpretations. The alterations were possible without applying another round of the Delphi technique since there was a consensus above 0.75, and different judges provided similar suggestions on simple changes relating to certain of the terms used. Meanwhile, the instrument was considered intelligible in terms of its appearance. After considering all these changes, the questionnaire was reduced from 82 to 55 questions, of which 21 were on minimum units, 12 on connectivity, eight on integration, eight on normativity, and six on structure (ANNEX 1).

It should be highlighted that the instrument does not include questions related to the quantity of PCN services implemented. Although this information provides important context for the “Structure” component, it represents information that can be accessed through governmental systems such as DATASUS. In addition, including questions on the subject would contradict stakeholders’ emphasis on the need to reduce the questionnaire in order to ensure greater uptake by respondents. Finally, it is noteworthy that the judges considered the instrument to be comprehensive and well-prepared overall. It should also be noted that the comments and suggestions of the judges were detailed instead of general, showing extensive knowledge of the PCN and deep involvement in the validation phase of the study.

Discussion

Although implementing national policies for psychosocial care is an essential tool for improving the population’s mental health (PATEL et al., 2018), and despite the active promotion of mental healthcare by international organizations, only 60% of the member countries of the World Health Organization (WHO) had mental health policies in 2011, while 71% had mental health plans and 59% had legislation in the area. Hence, the evaluability assessment of this study could help to document the progress in implementing the public policy for psychosocial care in Brazil.

However, as demonstrated, these advances were most evident prior to 2016, when a shift in government policies resulted in considerable alterations to the national mental health policy in 2017, jeopardizing the achievements of earlier decades (ALMEIDA, 2019ALMEIDA, J. M. C. de. Política de saúde mental no Brasil: o que está em jogo nas mudanças em curso. Cadernos de Saúde Pública, v. 35, n. 11, p. e00129519, abr. 2019.). In addition, the chronic underfunding of this and other public health policies and the scarcity of large-scale evaluation studies have also been associated with the growing lack of official data since 2015. Thus, Onocko-Campos (2019) warns of the need to evaluate the implementation of services to avoid wasting efforts and financial resources.

Given the subjective aspects that pervade psychosocial care and the recent reformulation of the Brazilian care model, systematizing strategies to evaluate current practices is a challenging but necessary activity to monitor their implementation and functionality (OLIVEIRA et al., 2014OLIVEIRA, M. A. F. de et al. Processos de avaliação de serviços de saúde mental: uma revisão integrativa da literatura. Saúde em Debate, v. 38, n. 101, p. 368-378, jun. 2014.; COSTA; COLUGNATI; RONZANI, 2015COSTA, P. H. A. da; COLUGNATI, F. A. B.; RONZANI, T. M. Avaliação de serviços em saúde mental no Brasil: revisão sistemática da literatura. Ciência & Saúde Coletiva, v. 20, n. 10, p. 3243-3253, out. 2015.; AMARAL; BOSI, 2017AMARAL, C. E. M.; BOSI, M. L. M. O desafio da análise de redes de saúde no campo da saúde coletiva. Saúde e Sociedade, v. 26, n. 2, p. 424-434, jun. 2017.; CORREIA; GOULART; FURTADO, 2017CORREIA, P. C. I.; GOULART, P. M.; FURTADO, J. P. A avaliabilidade dos Núcleos de Apoio à Saúde da Família (Nasf). Saúde em Debate, v. 41, n. spe, p. 345-359, mar. 2017.). Moreover, evaluation studies on mental healthcare networks are also scarce in the international literature (ASHWOOD et al., 2018ASHWOOD, J. S. et al. Evaluation of the Mental Health Services Act in Los Angeles County. Rand health quarterly, v. 8, n. 1, p. 4-7, ago. 2018.; MONTGOMERY et al., 2019MONTGOMERY, L. et al. An evaluation of mental health service provision in Northern Ireland. Health and Social Care in the Community, v. 27, n. 1, p. 105-114, jun. 2019.). Regarding the implementation analysis, a 2015 study highlights the lack of tools for mapping adult mental healthcare services and proposes a protocol to evaluate and monitor the development of integrated systems in eight European countries (SALVADOR-CARULLA et al., 2015SALVADOR-CARULLA, L. et al. Developing a tool for mapping adult mental health care provision in Europe: The REMAST research protocol and its contribution to better integrated care. International Journal of Integrated Care, v. 15, out./dez. 2015.). Other experiences that use instruments similar to those studied in this work are the European service mapping schedule (ESMS), developed to standardize the classification of health services and the care provided to adults in situations of mental distress; the international classification of mental healthcare (ICMHC), which aims to evaluate the different care specialties offered by the health system; and the European sociodemographic schedule (ESDS), aiming at a standardized description of the sociodemographic characteristics of the local of implementation. The ESMS also underwent adaptations for application in other countries and populations, such as children and adolescents (ROMERO-LÓPEZ-ALBERCA et al., 2019ROMERO-LÓPEZ-ALBERCA, C. et al. Standardised description of health and social care: A systematic review of use of the ESMS/DESDE (European Service Mapping Schedule/Description and Evaluation of Services and DirectoriEs). European Psychiatry, v. 61, p. 97-110, ago. 2019.). Another instrument designed to provide data on mental health systems is the World Health Organization's assessment instrument for mental health systems (WHO-AIMS), which aims to collect essential information for the global mental health system (NEISI et al., 2005NEISI, L. et al. A review of the world health organization assessment instrument for mental health system: WHO-AIMS. Quarterly Journal of Nursing Management, v. 6, n. 3, p. 66-77, 2005.). However, the literature reports that the descriptions generated by the application of that tool still provide very general findings (ROMERO-LÓPEZ-ALBERCA et al., 2019). In addition, due to the distinctive characteristics of each national health system, most of these instruments are difficult to replicate and cross-culturally adapt, thus calling for the development of tools designed for the Brazilian model.

Before developing a program evaluation study, it is recommended to conduct a detailed analysis of its policy or concrete situation to determine the feasibility and need for the evaluation (FIGUEIREDO; ANGULO-TUESTA; HARTZ, 2019FIGUEIREDO, T. A.; ANGULO-TUESTA, A.; HARTZ, Z. Avaliabilidade da Política Nacional de Regulação no SUS: uma proposta preliminar. Physis: Revista de Saúde Coletiva, v. 29, n. 2, set. 2019.). Thus, evaluability assessments have gained prominence as they provide a detailed analysis of the intervention, contributing to future evaluative studies by exploring the expectations and needs of the interest groups involved and the degree of organization and implementation of the evaluated initiative (CRAIG; CAMPBELL, 2015CRAIG, P.; CAMPBELL, M. Evaluability Assessment: a systematic approach to deciding whether and how to evaluate programmes and policies: a What Works Scotland Working Paper. Glasgow, 2015.; FIGUEIREDO; ANGULO-TUESTA; HARTZ, 2019FIGUEIREDO, T. A.; ANGULO-TUESTA, A.; HARTZ, Z. Avaliabilidade da Política Nacional de Regulação no SUS: uma proposta preliminar. Physis: Revista de Saúde Coletiva, v. 29, n. 2, set. 2019.). Accordingly, the theoretical-logical model developed in this study was essential to understanding the basic theoretical premises of the PCN, condensing the peculiarities of this complex program in a single image, representing a product that can be used in subsequent evaluative research. Significantly, the disagreements brought by the “new” policy were not neglected by these efforts to track the national mental health policy. The potential induction of changes by the federal government cannot be ignored due to its ability to modify the funding of services and/or procedures (MOREIRA; FERRÉ; ANDRADE, 2017MOREIRA, L. M. de C.; FERRÉ, F.; ANDRADE, E. I. G. Financiamento, descentralização e regionalização: transferências federais e as redes de atenção em Minas Gerais, Brasil. Ciência & Saúde Coletiva, v. 22, n. 4, p. 1245-1256, abr. 2017.; ALMEIDA, 2019ALMEIDA, J. M. C. de. Política de saúde mental no Brasil: o que está em jogo nas mudanças em curso. Cadernos de Saúde Pública, v. 35, n. 11, p. e00129519, abr. 2019.; BASTOS, 2019BASTOS, F. I. Políticas de drogas no Brasil contemporâneo: aportes da ciência, da clínica e do liberalismo moderno. Cadernos de Saúde Pública, v. 35, n. 11, p. e00125519, set. 2019.; ONOCKO-CAMPOS, 2019ONOCKO-CAMPOS, R. T. Saúde mental no Brasil: avanços, retrocessos e desafios. Cadernos de Saúde Pública, v. 35, n. 11, p. e00156119, jan. 2019.).

Identifying the potential use of the results and assessing the sustainability of the proposed intervention are two additional aspects of this assessment. Therefore, this activity cannot be carried out solely by the management nor applied only when urgent or convenient (TANAKA; TAMAKI, 2012TANAKA, O. Y.; TAMAKI, E. M. O papel da avaliação para a tomada de decisão na gestão de serviços de saúde. Ciência & Saúde Coletiva, v. 17, n. 4, p. 821-828, abr. 2012.). Hence, this evaluability assessment engaged members of the national and state mental health agencies, regional technical offices, and workers within the PCN of MG to develop the theoretical-logical model that identifies the central aspects to be evaluated.

Amaral and Bosi (2017) pointed out that the evaluation of the organization of health policies in care networks is not sufficiently studied in the field of public health research. However, understanding the effects of the articulation between services is a fundamental issue since the lack of healthcare coordination is one of the most frequent causes of low problem-solving capacity in healthcare networks (VÁZQUEZ et al., 2015VÁZQUEZ, M.-L. et al. Evaluating the effectiveness of care integration strategies in different healthcare systems in Latin America: the EQUITY-LA II quasi-experimental study protocol. BMJ Open, v. 5, n. 7, p. e007037, jul. 2015.; DE ALMEIDA; DE OLIVEIRA; GIOVANELLA, 2018ALMEIDA, P. F.; OLIVEIRA, S. C.; GIOVANELLA, L. Integração de rede e coordenação do cuidado: o caso do sistema de saúde do Chile. Ciência & Saúde Coletiva, v. 23, n. 7, p. 2213-2228, jul. 2018.). With respect to mental health, these factors are even more essential since the possibility of treatment goes beyond a single type of service and a single therapeutic modality (PATEL et al., 2018PATEL, V. et al. The Lancet Commission on global mental health and sustainable development. The Lancet, v. 392, n. 10157, p. 1553-1598, out. 2018.). Governance, addressed in the theoretical-logical model of this study in the "connectivity" and "integration" components, therefore requires political and technical action to meet the demand for already limited resources in constantly changing circumstances; as a result, it is frequently identified as one of the most critical and complex aspects of any mental health system (NICKELS et al., 2018NICKELS, S. V. et al. Access to mental health care in El Salvador: a case study of progress toward decentralization. Revista Panamericana de Salud Pública, v. 42, p. e172, out. 2018.). In addition, to achieve the integration of care services in the PCN, the adoption of the concept of health regions as administrative units, instead of the municipal model initially adopted by the SUS, is essential for ensuring access to shared facilities that are otherwise financially unfeasible for small municipalities (MACEDO et al., 2017MACEDO, J. P. et al. A regionalização da saúde mental e os novos desafios da Reforma Psiquiátrica Brasileira. Saúde e Sociedade, v. 26, n. 1, p. 155-170, mar. 2017.; MOREIRA; FERRÉ; ANDRADE, 2017MOREIRA, L. M. de C.; FERRÉ, F.; ANDRADE, E. I. G. Financiamento, descentralização e regionalização: transferências federais e as redes de atenção em Minas Gerais, Brasil. Ciência & Saúde Coletiva, v. 22, n. 4, p. 1245-1256, abr. 2017.; TRAPÉ; CAMPOS, 2017TRAPÉ, T. L.; CAMPOS, R. O. The mental health care model in Brazil: analyses of the funding, governance processes, and mechanisms of assessment. Revista de Saúde Pública, v. 51, p. 1-8, mar. 2017.).

Interestingly, although this study was designed to evaluate PCN implementation at the state level, it could be reproducible, with minor adaptations, in other regions of the country since the national regulations were the starting point for the formulation of the study model. Moreover, despite having been designed for response at the municipal level, the instrument was generated based on the structure of the PCN as a hierarchical network of services in each health region. A good indication of this reproducibility was the validation of the evaluative questions in the theoretical-logical model by experts who study mental health in various parts of the country. Content validation was used because it is essential for developing new measurement instruments since it represents the initial stage for associating abstract concepts with observable and measurable indicators (COLUCI; ALEXANDRE; MILANI, 2015COLUCI, M. Z. O.; ALEXANDRE, N. M. C.; MILANI, D. Construção de instrumentos de medida na área da saúde. Ciência & Saúde Coletiva, v. 20, n. 3, p. 925-936, abr. 2015.). However, the application of the Delphi technique by electronic means is a recent procedure with the advantages of reaching diverse and geographically distant actors and the rapid distribution, collection, and data processing; though it also has limitations, such as the increased rates of non-response that leads to considerable sample losses (JÜNGER et al., 2017).

The limitations of this study are related to the complexity of the PCN, encompassing aspects beyond the analytical capacity of a structured questionnaire. Thus, although identifying evaluative questions is fundamental for further evaluation (ARAUJO et al., 2018ARAUJO, R. F. de; et al. Avaliabilidade da compatibilização de indicadores e metas dos instrumentos de planejamento do Ministério da Saúde. Saúde em Debate, v. 42, n. 118, p. 566-578, set. 2018.), more in-depth questions pointed out in the literature and by some judges should be qualitatively analyzed in subsequent studies.

Conclusion

This study included an investigation of the historical processes that culminated in the current Brazilian PCN system, as well as modeling, planning, and verifying the possibility of evaluating its implementation in MG. It therefore aims to enrich the debate on evaluative methodologies that can analyze the relationship between the components of the PCN as well as the relationships between these components and their context. It also served to validate the theoretical-logical model and the multidimensional instrument for evaluating the implementation of a PCN (IMAI-RAPS), which can be disseminated for use in future research.

Considering the current national situation, with marked crises in public health and the recent COVID-19 pandemic that has increased the personal and social costs of mental illness to an unprecedented magnitude (LAKE, 2020LAKE, J. A Mental Health Pandemic: The Second Wave of COVID-19. Psychiatric Times, 15 jun 2020. Disponível em: https://www.psychiatrictimes.com/view/a-mental-health-pandemic-the-second-wave-of-covid-19. Acesso em: 21 dez. 2020.), sharing the information produced in this study is expected to advance studies documenting the progress already achieved by the PNSM in the country and identifying the remaining obstacles, enhancing the strategic management of the SUS for improving psychosocial care services.

References

ANNEX 1

Publication Dates

History