ABSTRACT

OBJECTIVE

To develop an assessment tool to evaluate the efficiency of federal university general hospitals.

METHODS

Data envelopment analysis, a linear programming technique, creates a best practice frontier by comparing observed production given the amount of resources used. The model is output-oriented and considers variable returns to scale. Network data envelopment analysis considers link variables belonging to more than one dimension (in the model, medical residents, adjusted admissions, and research projects). Dynamic network data envelopment analysis uses carry-over variables (in the model, financing budget) to analyze frontier shift in subsequent years. Data were gathered from the information system of the Brazilian Ministry of Education (MEC), 2010-2013.

RESULTS

The mean scores for health care, teaching and research over the period were 58.0%, 86.0%, and 61.0%, respectively. In 2012, the best performance year, for all units to reach the frontier it would be necessary to have a mean increase of 65.0% in outpatient visits; 34.0% in admissions; 12.0% in undergraduate students; 13.0% in multi-professional residents; 48.0% in graduate students; 7.0% in research projects; besides a decrease of 9.0% in medical residents. In the same year, an increase of 0.9% in financing budget would be necessary to improve the care output frontier. In the dynamic evaluation, there was progress in teaching efficiency, oscillation in medical care and no variation in research.

CONCLUSIONS

The proposed model generates public health planning and programming parameters by estimating efficiency scores and making projections to reach the best practice frontier.

Hospitals, University, organization & administration; Efficiency, Organizational, economics; Programming, Linear, utilization; Health Services Evaluation

INTRODUCTION

Hospitals are complex, multi-professional and interdisciplinary institutions, with specific technological density, according to the Brazilian National Policy for Hospital Care, implemented in the Health Care Networks. They are responsible for providing health care to users on an inpatient basis and via services covering health promotion, disease prevention, diagnosis, therapy, and rehabilitation^aaMinistério da Saúde. Portaria nº 3.390, de 30 de dezembro de 2013. Institui a Política Nacional de Atenção Hospitalar (PNHOSP) no âmbito do Sistema Único de Saúde (SUS), estabelecendo-se as diretrizes para a organização do componente hospitalar da Rede de Atenção à Saúde (RAS). Diario Oficial Uniao. 31 dez 2013; Seção 1:54-6..

Barata et al.¹1. Barata LRB, Mendes JDV, Bittar OJNV. Hospitais de ensino e o Sistema Único de Saúde. Rev Adm Saude. 2010;12(46):7-14. consider health to be a social right. They stress the complementary contribution of teaching hospitals in training health professionals to adequately tackle priority problems of the Brazilian population and developing research to help cope with these problems. Teaching hospitals include all hospital units, general or specialized, public or private, independent or attached to a higher education institution, which provide curricular activities in the field of health^bbMinistério da Saúde. Portaria interministerial nº 2.400, de 2 de outubro de 2007. Estabelece os requisitos para certificação de unidades hospitalares como Hospitais de Ensino. Diario Oficial Uniao. 3 out 2007; Seção 1:102-3.. University Hospitals (UH) belong to a federal or state higher education institution, with coordinated activities in care, teaching, and research.

Since the 1990s, the funding and management crisis of Brazilian hospitals, in particular university hospitals, has been placed on the agenda, with important repercussions in care, teaching, and research activities. Initiatives by the Brazilian Federal Government to face the crisis were based on financial, material and human resources support, and the introduction of improved management practices, among them the National Policy for the Certification of Teaching Hospitals, the National Program for Restructuring Federal University Hospitals, and the creation of the Brazilian Hospital Services Corporation^bbMinistério da Saúde. Portaria interministerial nº 2.400, de 2 de outubro de 2007. Estabelece os requisitos para certificação de unidades hospitalares como Hospitais de Ensino. Diario Oficial Uniao. 3 out 2007; Seção 1:102-3.^,ccBrasil. Decreto nº 7.082, de 27 de janeiro de 2010. Institui o Programa Nacional de Reestruturação dos Hospitais Universitários Federais-REHUF, dispõe sobre o financiamento compartilhado dos hospitais universitários federais entre as áreas da educação e da saúde e disciplina o regime da pactuação global com esses hospitais. Diario Oficial Uniao. 27 jan 2010; Edição extra; Seção 1:1-2.^,ddBrasil. Lei nº 12.550, de 15 de dezembro de 2011. Autoriza o Poder Executivo a criar a empresa pública denominada Empresa Brasileira de Serviços Hospitalares - EBSERH; acrescenta dispositivos ao Decreto-Lei nº 2.848, de 7 de dezembro de 1940 - Código Penal; e dá outras providências. Diario Oficial Uniao. 16 dez 2011; Seção 1:2-3.. Such initiatives aimed to attain excellence in teaching hospitals in care, teaching, and research by establishing management goals that would ensure the best use of resources in carrying out their tasks.

The development of performance studies using efficiency frontiers is appropriate to analyze the use of resources in achieving objectives. Among efficiency studies applied to the field of health, data envelopment analysis (DEA) is the most frequently used technique (48.0% of publications)⁵5. Hollingsworth B. The measurement of efficiency and productivity of health care delivery. Health Econ. 2008;17(10):1107-28. DOI:10.1002/hec.1391.

This study aimed to develop an assessment tool to evaluate efficiency in federal general university hospitals. This tool considers coordination between teaching, care, and research and the monitoring of changes over time.

METHODS

The model is based on dynamic network DEA to help decision-making processes among managers. A longitudinal analysis of interrelations is carried out between the dimensions of care, teaching, and research regarding their efficiency.

DEA is a linear programming technique first described by Charnes et al.²2. Charnes A, Cooper WW, Rhodes E. Measuring the efficiency of decision making units. Eur J Oper Res.1978;12(6):429-44. DOI:10.1016/0377-2217(78)90138-8 (1978). It measures efficiency by comparing similar units (DMU or Decision Making Units) that use the same resources (inputs) and generate the same products (outputs). A DMU is considered efficient when, compared to the others, it produces more outputs with a fixed amount of inputs (output-oriented model), or uses fewer resources to generate a fixed amount of outputs (input-oriented model). Efficient DMUs receive the maximum score of 100% and shape the best practice frontier, which involves the set of compared units. A DMU is only considered efficient when it reaches the Pareto-Koopmans optimality frontier, in which it is not possible to increase an output (or reduce an input) without reducing another output (or increasing another input) at the same time.

In classic models, the frontier may consider constant returns to scale (CRS, when an increase in outputs proportional to the increase in inputs is expected) or variable returns to scale (VRS, usually preferred when comparing units very different in size and outcome scale), a format established a priori by the researcher. Being a deterministic technique, any distance from the frontier (score lower than 100%) results from inefficiency, i.e., from the difference between current (X%) and projected values (100-X%). The projection of inefficient DMUs considers the shortest distance from or the nearest point to the frontier, where similar efficient units are located as benchmarks¹⁴14. Ozcan YA. Health care benchmarking and performance evaluation: an assessment using Data Envelopment Analysis (DEA). 2.ed. San Francisco: Springer; 2014..

In the development of classic models, when the production process generates intermediate outputs that serve as inputs for new processes, network DEA can be used. Network DEA is composed of a family of DEA models that establishes linear constraints for each one of the analyzed dimensions. It is thus possible to consider link variables belonging to more than one dimension and generate efficiency scores for all of them³3. Färe R, Grosskopf S. Network DEA. Socioecon Plann Sci. 2000;34(1):35-49. DOI:10.1016/s0038-0121(99)00012-9.

The models applied to case studies in the literature do not usually pay attention to projections in Pareto-inefficient regions of the frontier, and the projections with slacks in relation to the Pareto-efficient target result in efficiency estimation mistakes⁸8. Lins MPE, Oliveira LB, Silva ACM, Rosa LP, Pereira AO. Performance assessment of alternative energy resources in Brazilian power sector using Data Envelopment Analysis. Renew Sustain Energy Rev. 2011;16(1):898-903. DOI:10.1016/j.rser.2011.09.010. Prominent among the non-radial models that determine efficiencies based on Pareto-efficient targets are the SBM (Slacks-based Model) and Russel models. In this work, we chose to use the DEA model in SBM networks to evaluate efficiency, as proposed by Tone and Tsutsui¹⁵15. Tone K, Tsutsui M. Network DEA: a slacks-based measure approach. Eur J Oper Res. 2010;197(1):243-52. DOI:10.1016/j.ejor.2008.05.027^,1616. Tone K, Tsutsui M. Dynamic DEA with network structure: a slacks-based measure approach. Omega. 2014;42(1):124-31. DOI:10.1016/j.omega.2013.04.002.

For longitudinal monitoring, dynamic DEA enables the generation of efficiency scores for distinct and consecutive times by considering efficiency at each moment and the frontier shift over time. In each studied period, the DMUs use up the same inputs and generate the same outputs. However, some outputs of a period (capital and human resources, for example) can serve as inputs for the following period. These link variables between periods are called carry-over, which can be treated as fixed or mutable according to frontier projection.

The network DEA and dynamic DEA models are integrated (DNSBM – Dynamic Network Slack-based Model) in this study. Vertically, it deals with multiple dimensions linked in a networks structure within each period, and, horizontally, each structure is combined in networks by carry-over variables that link two distinct periods. The model enables the evaluation of: a) overall and single-dimension efficiency, at each evaluated moment; b) changes in overall and single-dimension efficiency, over the studied period. A modified Malmquist Index was developed, which evaluates changes in individuals scores (catch-up component) and the technological frontier shift, or of all hospitals over time (frontier shift)¹⁶16. Tone K, Tsutsui M. Dynamic DEA with network structure: a slacks-based measure approach. Omega. 2014;42(1):124-31. DOI:10.1016/j.omega.2013.04.002. Values higher than 1.0 mean performance progression in the interpretation of the index and its components; lower values mean frontier regression.

The DNSBM model used is output-oriented (aiming at increasing output) and considers variable returns to scale (given the different sizes of the analyzed hospitals). The network structure considers the relation between the dimensions of care, teaching and research, and the dynamic assessment analyzes the 2010-2013 period. The DMUs are the 31 federal general university hospitals, run by the Brazilian Ministry of Education (MEC), identified here by their respective universities. The variables were taken from the Ministry of Education Information System (SISREHUF).

Figure 1 illustrates the studied model schematically and Table 1 describes in detail the variables in the several dimensions: inputs, outputs, link and carry-over. No variables – other than link variables – were repeated in different dimensions.

In treating variables missing in SISREHUF, the study used the highest value (input variable); the lowest value (output variable); and the mean value (link and carry-over variables).

RESULTS

We observed heterogeneity in size and care-related characteristics of these hospitals. Mean care production (8.2% for adjusted outpatient visits and 8.0% for adjusted admissions) and technological structure (16.1%) increased over the period, despite the decrease in the number of active beds (3.7%) and non-medical staff (1.8%). There was an increase in the number of multi-professional residents (193.0%), of medical residents (21.8%) and of undergraduate students in health areas (13.9%) in the teaching dimension. Reported research projects fell by 64.7%, with no proportional fall in graduate students (increase of 1.5%) in the research dimension. We observed no significant numerical change in the number of professors with or without doctoral degrees or of doctors in most institutions (the high mean professor percentage is due to two hospitals with extreme reported values) (Table 2).

The overall mean efficiency score of the university hospitals in 2010, 2011, 2012, and 2013 was, respectively, 43.3%, 55.8%, 63.6%, 55.3%. Mean efficiency in all dimensions increased up to 2012, followed by a slight drop in 2013 (catch-up component). However, the frontier shift varied between the dimensions. In care, the shift fluctuated between regression in the 2010-2011 and 2012-2013 biennia and progression in the 2011-2012 biennium (mean score of 58.0%); teaching progressed in all biennia, especially 2010-2011, with the highest mean score (86.0%); the research frontier remained stationary (mean score of 61.0%) (Figure 2).

Hospitals of different sizes coexisted as benchmarks of best practice (100% score). This indicated the existence of at least two frontier regions of different scales, one with large hospitals of over 200 beds, which were efficient due to high output and complexity, and the second with smaller hospitals, with maximum efficiency measured by low resource consumption. These differences are important for the definition of peers and the choice of benchmarks of inefficient units. Five hospitals were present in the best practice frontiers (Universidade Federal do Maranhao [UFMA], Universidade Federal de Uberlandia [UFU], Universidade Federal de Sao Paulo [UNIFESP], among large hospitals; Universidade Federal de Sergipe [UFS], Universidade Federal da Grande Dourados [UFGD], among small hospitals) and six were efficient in at least two years of the studied period (Hospital de Clinicas de Porto Alegre [HCPA], Universidade Federal do Rio de Janeiro [UFRJ], among large hospitals; Fundacao Universidade do Rio Grande [FURG], Universidade Federal de Alagoas [UFAL], Universidade Federal do Amazonas [UFAM], Universidade Federal de Pelotas [UFPel], among small hospitals) (Figure 2).

The frontier projection suggested possible priorities and goals to be agreed on between managers and UHs. In 2010, at the beginning of the National Program for Restructuring Federal University Hospitals, there was a pressing need to increase output, especially in adjusted outpatient visits (131.0%) and admissions (88.0%), besides graduate students (83.0%). In 2013, the greater need was for an increase in research activities, which additionally require a significant increase in graduate students (80.0%) and care activities for outpatients (69.0%) (Figure 3).

Based on the best performance year (2012), it would be necessary to increase adjusted outpatient visits by 65.0% in care; multi-professional residence students by 13.0% and undergraduate health students by 12.0% in teaching; and graduate students by 48.0% in research for all hospitals to reach the frontier. Hospitals needing to increase output in outpatient visits or add a higher percentage of complexity in care would be Universidade Federal da Paraiba (UFPB), Universidade Federal do Estado do Rio de Janeiro (UNIRIO), Universidade Federal do Para (UFPA), Universidade Federal Fluminense (UFF), and Universidade Federal do Espirito Santo (UFES). Increase in the number of multi-professional residents would be more urgent at Universidade Federal de Campina Grande (UFCG), Universidade Federal de Minas Gerais (UFMG), UFRJ, Universidade Federal do Rio Grande do Norte (UFRN), and Universidade Federal de Santa Catarina (UFSC); increase in the number of undergraduate students at HCPA, UFCG, UFG, and UFSM, in teaching. UFAM, Universidade Federal da Bahia (UFBA), UFPA, UFPB, UFRN, and UNIRIO should have a significant increase in the number of graduate students in teaching.

Considering the number of medical residents as a link variable (care input and teaching output), the necessary values could be negative. For all hospitals to reach the frontier in 2012, there should be a mean reduction of 9.0% in the number of residents. Hospitals such as UFSC and UFCG would need to increase their numbers, but hospitals such as UFBA, UFF, Universidade Federal de Goias (UFG), UFMG, UFRN, Universidade Federal de Santa Maria (UFSM), UNIRIO, and Universidade de Brasilia (UnB) could reduce theirs without harming efficiency. As to the other link variables, adjusted admissions should increase by 34.0% and research projects by 7.0%, on average.

Table 3 presents, besides the overall score and the specific score for care, the financial budget, in millions of reais, of each UH, and the projection for the following year to reach the efficiency frontier. The approach of making projections for the best practice frontier generated financial planning guidelines based on the care output efficiency observed in each hospital.

The mean budget of the hospitals increased from R$42.1 to R$58.0 million over the period (mean increase of 37.8%). The largest gross increase was of UFMA (R$75.5 million) and the largest proportional increase was of UFGD (1,943.0%). Six units reported a lower budget in 2013 compared to 2010: UFAL (-24.2%), UFES (-22.2%), UFPB (-4.4%), UFRN (-33.3%), UnB (-8.5%), and UNIRIO (-92.2%). Given the fluctuation of the values reported by these units, there may be a recording mistake.

In 2013, projections for the following period suggested that seven DMUs would need a budget increase (a total of R$97.2 million), 11 would need a reduction (a total of R$175.6 million) and the others could keep it at the same level. All other parameters being the same, there would be scope for a 4.6% reduction in the total amount of funds apportioned to these hospitals in 2014, without any impact to the system’s efficiency. These guidelines would allow negotiations to increase production or redistribute resources among the units. The best performance year (2012) was the only one in which the projected funds exceeded the actual budget, i.e., the output justified the budget in a year of greater efficiency.

DISCUSSION

The proposed model enables managers to monitor efficiency, plan goals, and establish budget guidelines over time, thus being a tool to support public policy in the field of health.

As shown in Figure 3, the projections for each year generated parameters and goals to be reached in the following period. For example, comparing the 2012 goals with the actual 2013 data, there was an increase in 20.0% in medical residents (against a reduction guideline of 9.0%). For the remaining link variables, adjusted admissions were 19.0% lower than the goal and research projects were 67.0% lower. For the remaining outputs, adjusted outpatient visits were 35.0% lower than expected; undergraduate students, 5.0% lower; graduate students, 15.0% lower. Multi-professional residents, on the other hand, were 7.0% higher than the goal. Such findings partly explain the drop in mean efficiency scores in the three dimensions in 2013.

The comparison between general hospitals, with and without teaching and research activities, not considering these dimensions, minimizes the latter’s efficiency scores. The reason is that various resources of the production process are, in practice, also geared towards non-care activities⁴4. Grosskopf S, Margaritis D, Valdmanis V. Comparing teaching and non-teaching hospitals: a frontier approach (teaching vs. non-teaching hospitals). Health Care Manag Sci. 2001;4(2):83-90. DOI:10.1023/A:1011449425940. The network DEA model, on the other hand, enables comparison of the portion of resources used in each specific dimension, reducing this skew.

Since 2000, university hospitals have shown an effort in academic production, initially with classic DEA models to evaluate the efficiency of MEC units¹²12. Marinho A, Façanha LO. Hospitais universitários: avaliação comparativa da eficiência técnica. Econ Aplicada. 2000;4(2):315-49.. New analysis approaches have been developed for this set of hospitals, with the dimensions either treated separately or grouped in a hierarchical model or by networks⁷7. Lins ME, Lobo MSC, Silva ACM, Fiszman R, Ribeiro VJP. O Uso da Análise Envoltória de Dados (DEA) para avaliação de hospitais universitários brasileiros. Cienc Saude Coletiva. 2007;12(4):985-98. DOI:10.1590/S1413-81232007000400020^,1010. Lobo MSC, Lins MPE, Silva ACM, Fiszman R. Avaliação de desempenho e integração docente-assistencial nos hospitais universitários. Rev Saude Publica. 2010;44(4):581-90. DOI:10.1590/S0034-89102010000400001^,1313. Ozcan YA, Lins ME, Lobo MSC, Silva ACM, Fiszman R, Pereira BB. Evaluating the performance of Brazilian university hospitals. Ann Oper Res. 2010;178(1):247-61. DOI:10.1007/s10479-009-0528-1. Longitudinal evaluation has been used to analyze the impact of the funding reform created by the certification policy for university hospitals⁹9. Lobo MSC, Silva ACM, Lins MPE, Fiszman R. Impacto da reforma de financiamento de hospitais de ensino no Brasil. Rev Saude Publica. 2009;43(3):437-45. DOI:10.1590/S0034-89102009005000023.

The networks model was enhanced in this study by addressing the research dimension, considering only projections in the Pareto-efficient regions of the frontier and monitoring the frontier shift over the years. Such enhancement makes the model a more accurate tool for managers to analyze efficiency.

The tool proposed to monitor teaching hospitals enables managers to: a) generate efficiency scores, overall and for each dimension; b) monitor annual performance changes, overall and for each dimension; c) define and manage goals to improve performance based on the best practice frontier projection; d) program and evaluate dynamically the funding of UHs, based on efficiency. The findings have produced examples for each one of these functions. The data and projections can be analyzed separately, for each unit, or jointly, for a broader assessment of public policy.

The current set of federal university hospitals is heterogeneous in size, output capacity and operation time. Dealing with such differences required adjusting for complexity.

There was an increase in adjusted care output of approximately 8.0% in the last few years. The efficiency analysis shows that use of some resources increased simultaneously, and in a greater proportion, such as technological structure (16.1%) and funding (37.8%). On the other hand, there was a reduction in the number of beds, following a world trend in the past few years, with an increase of those in areas of greater complexity, such as intensive care units. The performance of procedures geared towards outpatients also reduced the need for beds, but there is scope to expand this activity (goal of 69.0% in 2013).

The tripled number of multi-professional residents in teaching output coincides with the policy to implement this modality over the period. In research there was a significant reduction in reported research projects, notwithstanding the quality limitations of the record.

Mean efficiency in UHs increased in all dimensions between 2010 and 2012, with a slight drop in the following year. The frontier shift, on the other hand, was heterogeneous, with higher performance and progress trend in teaching. Due to the recent implementation of correction measures in the UHs, some degree of oscillation is expected in the care frontier until the new management models are stabilized. This finding is consistent with the study of the British National Health System (NHS), which showed a contraction in the first years following its opening to the private market, later followed by an expansion of the efficiency frontier¹¹11. Maniadakis N, Hollingsworth B, Thanassoulis E. The impact of the internal market on hospital efficiency, productivity and service quality. Health Care Manag Sci.1999;2(2):75-85. DOI:10.1023/A:1019079526671.

Unlike with output-oriented goals, the definition of proposed goals considers used resources, peers in scale size, the dynamic variation of peers, and the influence of the whole group being compared. These factors enable a planning better adapted to the reality of each hospital, the definition of priorities and performance monitoring. Thus, they can serve as an additional parameter in physical and budget negotiations between unit managers and health managers, or funding agencies. The latter would be able to monitor the system’s behavior as a whole and publish sharing parameters and funding models that consider criteria related to transparency, equity and needs¹⁷17. Ugá MAD, Porto SM. Financiamento e alocação de recursos em saúde no Brasil. In: Giovanella L, Escorel S, Lobato LVC, Noronha JC, Carvalho AI, organizadores. Políticas e sistema de saúde no Brasil. Rio de Janeiro: Editora Fiocruz; 2008. p.473-506..

The planning tool will only be valid if based on an accurate and reliable information system⁶6. La Forgia GM, Coutollenc BF. Hospital performance in Brazil: the search for excellence. Washington (DC): The World Bank Publications; 2008.. Inadequate records, insufficient use and lack of criticism of the system might be the main limitations of the study’s outcomes. The results and the empirical frontier vary according to the set of variables used. Hence the importance of having various actors involved in evaluating the model’s construction, so that each element of the model is consensually approved for the several hospital profiles to be compared.

REFERENCES

Publication Dates

History