ABSTRACT

Psychosocial Care Centers III (CAPs) are considered strategic in the reorientation of the care model in mental health. However, they still lack systematic evaluation mechanisms. This study presents a set of indicators developed in a participatory process for the Psychosocial Care Centers III of the state of São Paulo. Sixteen indicators, grouped into 8 themes were developed: Attention to crisis situations; Qualification of group meetings; Networking; Management of Psychosocial Care Centers; Continuing education; Individualization of care; Care for people with intellectual disabilities; and Use of medication. The indicators were tested in services and are presented as a potentially useful tool to support the assessment, monitoring and management of Psychosocial Care Centers III.

KEYWORDS:
Health evaluation; Mental health; Mental health services

Introduction

Despite their responsibility in almost all diagnostic and therapeutic procedures, as well as in individual care, health services are insufficiently considered in studies. Clinical and epidemiological research rarely takes into account the importance of services on risk factors for population morbidity and mortality or its effectiveness in patient care. Research on health services aims to overcome this insufficient systematic approach in services evaluation, and achieve a continuous qualification of health care. Evaluation procedures are one of the modalities of this type of research (WALKER, 2014WALKER, D. M. An introduction to health services research. Newbury Park: Sage, 2014.).

The Psychosocial Care Centers (Caps, Centros de Atenção Psicossocial) stand out among the services that resulted from the psychiatric reform in Brazil. Considered as controllers of the mental healthcare network and, more recently, important components of the psychosocial care network, the Caps have shown high effectiveness in the care of people with severe psychiatric disorders (SALLES; BARROS, 2013SALLES, M. M.; BARROS, S. Transformações na atenção em saúde mental e na vida cotidiana de usuários: do hospital psiquiátrico ao Centro de Atenção Psicossocial. Saúde em Debate, Rio de Janeiro, v. 37, n. 97, p. 324-335, jun. 2013.). A study that addresses the current normative model of mental healthcare in Brazil presents the Caps as the main resource for policy orientation of the Psychosocial Attention Networks (RAPs) of the Ministry of Health (TRAPÉ, 2015TRAPÉ, T. L. Redes de Atenção à Saúde Mental: Estudo Comparado Brasil-Catalunha. 120 f. 2015. Tese (Doutorado em Saúde Coletiva) - Faculdade de Ciências Médicas, Universidade Estadual de Campinas. Campinas, 2015.).

The Caps originated in the second half of the 1980s as pioneer services in the city of São Paulo (SP) and, soon afterwards, became an optional network to the psychiatric hospital in the city of Santos (SP). This was an inflection point in care practices to people with severe mental disorders within the Unified Health System (SUS, Sistema Único de Saúde). The work of Pitta (1995)PITTA, A. M. F. et al. Determinantes da qualidade de serviços de saúde mental em municípios brasileiros: estudo da satisfação com os resultados das atividades desenvolvidas por pacientes, familiares e trabalhadores dos serviços. J. Bras. Psiquiatr, Rio de Janeiro, v. 44, n. 9, p. 441-452, 1995. and Libério (1999)LIBÉRIO, M. M. Estudo da satisfação com o CAPS da cidade do Rio de Janeiro: ouvindo seus atores principais. 1999. Dissertação (Mestrado em Psiquiatria) - Instituto de Psiquiatria, Universidade Federal do Rio de Janeiro, Rio de Janeiro, 1999. on quality of care in the Caps, and Bandeira (2000)BANDEIRA, M.; PITTA, A. M. F.; MERCIER, C. Escalas brasileiras de avaliação de satisfação (SATIS-BR) e da sobrecarga (IMPACTO-BR) da equipe técnica em serviços de saúde mental. J. Bras. Psiquiatri, Rio de Janeiro, v. 49, n. 4, p. 105-115, abr. 2000. on work overload of employees in these services are the first initiatives to systematically evaluate the new mental healthcare model in Brazil, since the 1990s.

A review study (DANTAS; ODA, 2014DANTAS, C. R.; ODA, A. M. G. R. Cartografia das pesquisas avaliativas de serviços de saúde mental no Brasil (2004-2013). Physis, Rio de Janeiro, v. 24, n. 4, p. 1127-1179, dez. 2014.) shows a significant increase in scientific production about CSPs from the 2000s. For the authors, the number of what they consider qualitative research and participatory methodologies has hindered the establishment of mental health assessment indicators. However, in the last decades, different instruments than those focused by the authors have been offered by the Pan American Health Organization/World Health Organization (PAHO / WHO) for specific types of Caps and for mental health policies and systems in the country. Although aimed at encouraging the use of indicators in mental health, these tools were unsuccessful in promoting their effective appropriation and use.

With regard to SUS management, the SUS Performance Index (IDSUS) and the National Program for Access and Quality Improvement in Primary Care (PMAQ-AB) stand out as national initiatives for the evaluation of healthcare. In both cases, a series of indicators are applied to a service or a service network, in order to evaluate and induce practices. However, the IDSUS has no indicator related to mental health, and the PMAQ-AB, has four indicators of mental health practices, with exclusive focus on the abusive use of psychoactive substances. It is quite a narrow scope, given the complexity of mental health practices and the organization of services.

The WHO and its regional offices systematically release technical notes, reports and analyzes to support the implementation of common objectives and standards for sectoral policies (WHO, 2013), seeking to establish a consensus and foster the development of mental health indicators. The guidelines for the quality evaluation of mental healthcare and for the health system evaluation tool propose a set of indicators organized in different areas, such as mental health policies and programs, primary health care, human resources, user's rights, psychosocial rehabilitation and intersectoral actions, among several others. These instruments have an explicit intention of inducing good practices and, at the same time, promoting transparency and accountability of mental health initiatives. However, the distance between developers and potential users may lead to the abandonment of proposals or to their decontextualized use, or moreover, to the 'technification' of proposals, which implies ignoring origins, the development process and the intended use of a certain instrument, as stated by Ardila and Stolkiner (2009)ARDILA, S.; STOLKINER, A. Estrategias de evaluación de programas y servicios de atención comunitaria en salud mental: consideraciones metodológicas. In: CONGRESSO INTERNACIONAL DEL INVESTIGACIÓN Y PRÁTICA PROFESIONAL EN PSICOLOGIA DEL MERCOSUR, 1., 2009, Buenos Aires. Anais... Buenos Aires: Acta Academica, 2009. p. 252-257. Disponível em: <http://www.aacademica.org/000-020/210>. Acesso em: 24 out. 2016.
http://www.aacademica.org/000-020/210... .

The purpose of this article is to present and discuss a set of 16 indicators directed to the monitoring, evaluation and potential qualification of the Caps III. The indicators were developed from the collaboration between evaluators from two universities and 58 Caps III employees and managers, in the State of São Paulo. Services were established as the central focus of the evaluation, favoring an effective exchange of knowledge between academics and workers, taking into account the different political and institutional contexts in which the Caps were inserted. The services' crucial issues were defined, seeking to overcome limitations, such as the distance between developers and users, and the risks of the technification of the evaluation instruments.

Methodology

The importance of including non-specialists in evaluation processes has been emphasized and implemented in the literature (BARON; MONNIER, 2003BARON, G.; MONNIER, E. Une approche pluraliste et participative. Informations sociales, Paris, n. 110, p. 120-129, 2003.). The inclusion of people from the evaluated service in parts or during the whole evaluation process occurs either for pragmatic or for ideological reasons, aiming at increasing the usefulness of the results, or to enhance the effects of the evaluation process itself, referred to in the literature as 'process use' (PRESKILL; ZUCKERMAN; MATTHEWS, 2013PRESKILL, H.; ZUCKERMAN, B.; MATTHEWS, B. An exploratory study of process use: findings and implications for future research. American Journal of Evaluation, New Mexico, v. 24, n. 4, p. 423-442, 2013.). In this study, workers and managers of Caps III were included in the process of elaboration and definition of indicators, to effectively consider the issues that permeate workers' daily lives and to subsidize the incorporation and subsequent use of the indicators by the teams involved.

As a strategy to mediate participation and enable the elaboration of the indicators, a 120-hour course on mental health assessment was proposed, taught regularly over 11 months. The themes were selected based on the results of a previous study conducted by the same researchers that formulated and defined a good practices guide for the organization of the Caps III care: 1) Health assessment; 2) Attention to crisis; 3) Working with groups; 4) Territory and mental health; 5) Management in mental health; 6) Individual therapy project; 7) Intellectual deficiency; 8) Psychiatric medication; 9) Therapeutic home services; 10) Psychosocial rehabilitation; and 11) Continuing education. As a starting point, the groups were offered a set of indicators previously developed for the Caps III network of another city of São Paulo (FURTADO; ONOCKO-CAMPOS, 2008FURTADO, J. P.; ONOCKO-CAMPOS, R. Participação, produção de conhecimento e pesquisa avaliativa: a inserção de diferentes atores em uma investigação em saúde mental. Cad. Saúde Pública, Rio de Janeiro, v. 24, n. 11, p. 2671-2680, nov. 2008.).

The definition of indicators based on issues from the Caps III was due to the strategic importance of these services as a consolidation of the psychiatric reform and the implementation of a care network, replacing the asylum model. Open 24 hours a day, 7 days a week, the Caps III serve 150 thousands inhabitants or more, attending to the continuous care of adults with severe mental disorders, including patients in crisis, and it covers issues inherent to the other types of Caps (I and II).

Two openings to attend the course were made available per service, one for a manager and one for an employee. The two persons were selected internally by the units, based on pre-defined criteria: university education (higher level), willingness and disposition to fully participate in the project, and more than six months of employment in the service. Five units requested extra spots, due to the interest of teams in participating, which were approved by the course coordinator, and 3 spots were made available to managers of the Health Department of São Paulo. The 58 managers and employees from 25 Caps III (out of 26, since 1 of the Caps enrolled but abandoned after the first meeting, alleging schedule difficulties) from São Paulo at the time, were divided into 2 groups, according to the proximity of their home town to the 2 participating universities.

After morning conversational discussions conducted by guests of recognized knowledge on one of the themes listed above, Shared Appreciation Groups (GAP) meeting were held, which were supported by masters and doctoral students of the universities' graduate programs. Each GAP included approximately ten attendees of the course, and was accompanied by the same supporter throughout the year. The groups' objective was to deepen the discussion on the topics covered in the morning and to outline indicators related to those same themes.

The GAPs arose from the awareness that a program or service should not be considered as the product of written documents or of what was advocated by those who conceive or manage them; other sources should be considered, including the informal communication. Jalbert (1997)JALBERT, Y. et al. Epsilon: guide d'auto-évaluation des organismes communautaires. Montréal: COCQ-Sida, 1997. proposed and experimented with the GAPs (from the French language: Groupes d'Appréciation Partagée) in order to consolidate the feeling of participation, ensuring the manifestation of the greatest possible number of members around the reflection and evaluation of the action.

The preliminary definition of indicators was based on the convergence between the problems perceived by the agents in their work routine, the reflection induced by the thematic discussion within the GAP and the subsequent confrontation between the proposed indicators and the considerations of the original teams of the same agents, creating a feedback of the process. In addition, two large meetings were held in the universities with the 58 participants in the first and last months of the course, for the establishment of guidelines and, in the final meeting, for the alignment of proposed indicators previously designed and tested within the subgroups.

This research was submitted and approved by the Research Ethics Committee of the Faculty of Medical Sciences of the State University of Campinas (Unicamp), under the file 410/2011, and funded by the Foundation for Research Support of the State of São Paulo (Fapesp), process 2009/53130-3, and the International Development Research Center (IDRC), Canada, file 02 P-18737-2009.

Results

The indicators were grouped into eight themes: Attention to a crisis situation; Qualification of group meetings; Networking; Caps management; Continuing education; Individualization of care; Attention to people with intellectual disabilities; and Use of medication. The classes, the GAP and the proposal of indicators along the course were organized according to these thematic areas, identified as central in previous research (ONOCKO-CAMPOS 2009ONOCKO-CAMPOS, R. et al. Avaliação da rede de centros de atenção psicossocial: entre a saúde coletiva e a saúde mental. Rev. Saúde Pública, São Paulo, v. 43, p. 16-22. 2009.). During the GAP, dozens of indicators were suggested, debated, and analyzed. At the end of the process, each of these themes was considered with one to three indicators. Each indicator had its central components dismantled and detailed, such as: the name of the indicator; its definition (the addressed problem); interpretation (the aspect it evaluates); data source (where to obtain the necessary data); measurement period (time interval between one measurement and the next); calculation method (numerator and denominator of the indicator); and additional observations for specific indicators.

Attention to a crisis situation

The crisis situation was one of the most debated topics by the participants in the GAP and showed different conceptions on the subject. The substitutive role of the Caps in general and of the Caps III in particular invariably navigates through the ability to cope with crisis situations of its users. For this reason, the number 1 indicator (crisis care) evaluates the service's ability to respond locally to acute conditions and their possible extreme variations, avoiding referrals to hospitals or psychiatric emergency rooms. The number 2 indicator (Attention to the family of patients in crisis) evaluates the ability of the Caps to operate on aspects beyond the case itself, considering the demands and needs of the family members, virtually sharing the attention.

Qualification of group meetings

The need to incorporate practices that go beyond individual sessions is a paradigm that supports the psychosocial model and the reorientation of the clinic for substitutive services. However, precisely because they are a consensus, group meetings are implemented without a systematic follow-up that would provide reformulations and their continuous qualification. The indicators defined here demonstrate the workers' concern to subject group sessions to discussion and possible revision by the teams, and also to expand their offer to the users' affective surroundings, i.e. to the family members.

Networking

The interrelation of institutions and professionals is the cornerstone of the mental health care model reorientation, and networking was envisioned as a need by almost all the members of the indicators' elaboration process. The group collectively decided that the relationship between the number of therapeutic projects conducted simultaneously by the Caps and other health services or other sectors with the total number of people receiving care should be use as indicator in order to evaluate the shared work. This indicator seeks to capture interprofessional and intersectoral work capacities, which translate into notion of networking.

Caps management

During the development of the indicators, the inseparability between clinic and management was reaffirmed by the majority of participants. Management organization has a direct impact on teams' orientations, corroborating the need for strategies that integrate management and the clinical practice (ONOCKO CAMPOS 2009ONOCKO-CAMPOS, R. et al. Avaliação da rede de centros de atenção psicossocial: entre a saúde coletiva e a saúde mental. Rev. Saúde Pública, São Paulo, v. 43, p. 16-22. 2009.). One of the indicators for the management of the Caps seeks to capture the manager's participation in the various spaces of collective management and discussion, in which workers and users can directly participate in the decisions about the unit's direction. The second indicator seeks to demonstrate the effectiveness of the manager in providing the necessary structure for the operation of the service. The number of working hours of university professionals per 100 thousand inhabitants in the assigned area was selected as the parameter that would reflect the level of structuring of the Caps.

Continuing education

The clinical-institutional supervision, while enquiring the technical interventions (clinical practices) of each practitioner and team, should also address how the care model is understood and conducted in order to act on the health-disease process. Thus, supervisions have both a clinical and a political dimension, as they question aspects of both the work process and the institutional relations. The convergence of these two analyses derived from the employees' work is the basis for considering clinical-institutional supervision as an integral element of continuing education, together with the investment in institutional training outside the Caps. These were two elements (clinical-institutional supervision and external training) defined as indicators for the continuing training of workers.

Individualization of attention

The Individual Therapeutic Project was considered a decisive strategy for conducting the work according to the course of each user. For the evaluation of this strategy, three complementary indicators were elaborated. First, the number of patients that have a therapeutic plan is obtained; then, the frequency with which this plan is reviewed and adapted, considering eventual changes of the user and its context; and finally, the third indicator defines the number of people in charge of each technician or reference team, a ratio with strong impact in the effective individualization of attention.

Attention to people with intellectual disabilities

The borderline between intellectual disability and mental health is tenuous (SURJUS, ONOCKO-CAMPOS, 2014SURJUS, L. T. S.; ONOCKO-CAMPOS, R. Interface entre Deficiência Intelectual e Saúde Mental: revisão hermenêutica. Revista Saúde Pública, São Paulo, v. 48, n. 3, p. 532-540, jun. 2014.), especially in patients who were submitted to long psychiatric hospitalizations. The way in which the demands and needs related to intellectual disability are considered by the Caps were the focus of the first two indicators of this theme. The indicators investigate the insertion of this clientele in the service and the elaboration of a specific therapeutic plan for them. A third indicator is related to the insertion of people with intellectual disabilities into the Home Therapeutic Services.

Use of medication

The debate about the potentials and limitations of the use of psychiatric medication in mental health is controversial (ONOCKO-CAMPOS , 2013______. Gestão Autônoma da Medicação: uma intervenção analisadora de serviços em saúde mental. Ciênc. Saúde Coletiva, Rio de Janeiro, v. 18, n. 10, p. 2889-2898, out. 2013.). Only one indicator was defined for the topic, addressing the interruption or maintenance of the prescribed medication, within the services. Patient consent, quality of prescriptions and level of user autonomy in medication intake did not reach the necessary consensus for the elaboration of respective indicators, even though they emphatically permeated the debate in many GAPs and in the plenary meetings.

Discussion

The operationalization of mental health indicators is a challenge. First, the tradition of indicators in this area is limited when compared to other health fields, such as basic care and hospital care, which have been in use for a long time and received support by national and international health agencies, and therefore, established the basis for follow-up criteria. In addition, the strong ethical and political character of the psychiatric reform and the consequent difficulty in establishing consensus around some parameters and indicators among agents, also have an impact. Moreover, the object of mental health workers, which is characterized by subjective issues involving the subjects they accompany, requires efforts to relativize and understand the singular matter, generating less permeability for objectification and numerical systematization of their practices around indicators.

During the construction process of these instruments, it was often difficult to objectify and synthesize actions of various natures developed within the Caps III. These difficulties were evidenced, among other characteristics, by the lack of parameters on conceptual aspects fundamental to the services. An illustrative example was the recurrent question regarding a definite conception of a 'crisis'. Extensive debates were required to establish a consensus on conceptions that were initially thought to be well established, given that some were faced daily by the teams. Such inquiries arouse when services' staff were presented with the preliminary version of the indicators. This situation seems to demonstrate the need for the development of a glossary to be used with the set of indicators discussed here.

The course allowed the consideration of the main themes that involve the daily routine, the management and the clinical work of Caps, according to researchers and workers. As part of the process, defined indicators were screened, ensuring 'utility' (meeting the practical information needs), 'feasibility' (requiring readily accessible information) and 'accuracy' (revealing technically appropriate information) characteristics. The collective analyzes of the indicators and their preliminary applications in the services corroborate their satisfactory performance. In addition, the obtained results demonstrate that these instruments are in accordance with the definition of a good evaluation by the American Association of Evaluation (FURTADO; LEPERRIERE, 2012FURTADO J. P.; LEPERRIERE, H. Parâmetros e paradigmas em meta-avaliação: uma revisão exploratória e reflexiva. Ciênci. Saúde Coletiva, Rio de Janeiro, v. 17, n. 3, p. 695-705, mar. 2012.).

The joint input of university researchers and workers from the services, on the one hand, allowed the consideration of issues that would be difficult to perceive from a scientific point of view, and the construction of effective instruments for observation and support of eventual changes. On the other hand, this joined proximity, while broadening the possibility of adoption and use of the indicators by the Caps teams, it simultaneously confers fragility to the indicators caused by the substantial changes in the service context. In addition, it should be emphasized that such an instrument should necessarily be combined with broader policies aimed at continuous qualification and monitoring of services. The fragility of management at various levels, lack of structure and stability of the SUS evaluative bodies and the inefficiency of information systems create a scenario that is unfavorable to the consolidation and expansion of initiatives such as the one presented in this study (TRAPÉ, 2015TRAPÉ, T. L. Redes de Atenção à Saúde Mental: Estudo Comparado Brasil-Catalunha. 120 f. 2015. Tese (Doutorado em Saúde Coletiva) - Faculdade de Ciências Médicas, Universidade Estadual de Campinas. Campinas, 2015.).

The participants from the Caps were essentially professionals with university education in health fields, which may explain the predominance of indicators for processes in relation to those for structure and result, or of 'soft' indicators as reported by Saraceno, Frattura and Betolote (1993)SARACENO, B.; FRATTURA, L.; BERTOLOTE, J. M. Evaluation of psychiatric services: hard and soft indicators. Geneva: WHO, 1993.. Important agents of the process, such as professionals with technical training, users, family members and medical professionals, collaborated only partially and indirectly in periods of deconcentration in the field locally, when of the validation or testing of a certain indicator. Thus, a demand arises for future initiatives that develop evaluative instruments from the perspective of these agents. We identify this as one of the weaknesses of the process registered here.

In spite of this, the approach described provide quality to the participation, as it allowed the appropriation of part of the theoretical reference of evaluation by a group unfamiliar with systematic discussions on the subject. The workers collaborated in the definition of issues, and in the preparation and preliminary application of the indicators, giving depth to their contribution, according to the proposal of Cousins and Chouirnard (2012)COUSINS, J. B.; CHOUIRNARD, J. A. Participatory evaluation up close: an integration of research-based knowledge. Charlotte: Age Publishing, 2012.. For these authors, the extent of participation is defined by the diversity of participants, and the depth by the level of influence in the process of formulating, researching and analyzing the procedures involved in the evaluation. Although we have just taken some steps towards evaluation - defining issues and instruments - and not performed the complete evaluation process, we guarantee effective collaboration in decisive stages of the process.

Conclusion

From an academic point of view, the process undertaken for the elaboration of the indicators combined efforts and effectively linked teaching, research and academic extension. From the point of view of services, the inclusive and formative methodological strategy fostered the critical thinking of workers and managers and the deepening of important and common issues, from which tools for objectifying and monitoring priority problems emerged. This objectification, however, was made within the context in question. After all, indicators are the expression of a concept, and their inseparability must necessarily be understood and considered by agents, overcoming the fetishization or technification of these instruments. The theoretical discussions and debates that preceded the formulation of the indicators proved effective in making the participants aware of the premises behind these indicators.

Effective participation and collaboration require time, appropriate spaces and connections between agents from different fields. The complex task of developing evaluative indicators in a health subarea without tradition in evaluation processes was made possible through the establishment of an institutional space, represented by the course, and a long period of time, represented by the several meetings over a year. This strategy allowed time breaks for the evolution of the debate and the establishment of trust bonds created during the GAP meetings that took place in singular spaces with the support of a single professional at all times.

The effective use of the indicators can contribute to the development of the evaluation culture - evaluation of the Caps and of the instruments themselves - as they can be confronted with diverse realities outside the state of São Paulo and with unforeseen situations. We present these indicators to the scientific community, without the pretension of them being unique or perfect. Instead, we expect to be contributing to the opening of a communication channel and a necessary construction in the current situation of the Brazilian psychiatric reform. Having developed and tested a set of evaluative indicators through an inclusive process of workers and managers represents the strength of our research.

The permeability of municipal managers, regional colleges, states and federation is always critical for the continuous improvement of the services studied herein, and obviously, for the support and expansion of the presented instruments.

References

Publication Dates

History