Validity and reliability of a health care service evaluation instrument for tuberculosis

Lucia Marina Scatena Anneliese Domingues Wysocki Aline Ale Beraldo Gabriela Tavares Magnabosco Maria Eugênia Firmino Brunello Antonio Ruffino Netto Jordana de Almeida Nogueira Reinaldo Antonio Silva Sobrinho Ewerton William Gomes Brito Patricia Borges Dias Alexandre ... About the authors

Abstracts

OBJECTIVE

To evaluate the validity and reliability of an instrument that evaluates the structure of primary health care units for the treatment of tuberculosis.

METHODS

This cross-sectional study used simple random sampling and evaluated 1,037 health care professionals from five Brazilian municipalities (Natal, state of Rio Grande do Norte; Cabedelo, state of Paraíba; Foz do Iguaçu, state of Parana; Sao José do Rio Preto, state of Sao Paulo, and Uberaba, state of Minas Gerais) in 2011. Structural indicators were identified and validated, considering different methods of organization of the health care system in the municipalities of different population sizes. Each structure represented the organization of health care services and contained the resources available for the execution of health care services: physical resources (equipment, consumables, and facilities); human resources (number and qualification); and resources for maintenance of the existing infrastructure and technology (deemed as the organization of health care services). The statistical analyses used in the validation process included reliability analysis, exploratory factor analysis, and confirmatory factor analysis.

RESULTS

The validation process indicated the retention of five factors, with 85.9% of the total variance explained, internal consistency between 0.6460 and 0.7802, and quality of fit of the confirmatory factor analysis of 0.995 using the goodness-of-fit index. The retained factors comprised five structural indicators: professionals involved in the care of tuberculosis patients, training, access to recording instruments, availability of supplies, and coordination of health care services with other levels of care. Availability of supplies had the best performance and the lowest coefficient of variation among the services evaluated. The indicators of assessment of human resources and coordination with other levels of care had satisfactory performance, but the latter showed the highest coefficient of variation. The performance of the indicators “training” and “access to recording instruments” was inferior to that of other indicators.

CONCLUSIONS

The instrument showed feasibility of application and potential to assess the structure of primary health care units for the treatment of tuberculosis.

Health Services Evaluation; methods; Tuberculosis; prevention & control; Quality Indicators; Health Care; Indicators of Health Services; Questionnaires; utilization; Validation Studies


INTRODUCTION

Health care services can be evaluated by monitoring the ability to respond to health care needs, determine the effects of interventions, produce evidence, identify and correct problems, provide feedback to health care teams and managers, develop technical material to adequately evaluate the health care provided to patients,4Carvalho ALB, Souza MF, Shimizu HE, Senra IMVB, Oliveira KC. A gestão do SUS e as práticas de monitoramento e avaliação: possibilidades e desafios para a construção de uma agenda estratégica. Cienc Saude Coletiva. 2012;17(4):901-11. DOI:10.1590/S1413-81232012000400012 and support the development of sectoral policies and programs and disseminate their results.8Felisberto E. Da teoria à formulação de uma Política Nacional de Avaliação em Saúde: reabrindo o debate. Cienc Saude Coletiva. 2006;11(3):553-63. DOI:10.1590/S1413-81232006000300002

Despite the presence of conceptual differences, the methodological construction for evaluation of health care services will depend on the clarity of the principles, objectives, and goals of the systems to be evaluated. This will form the basis for selecting variables that will be the object of performance evaluation.2222 Viacava F, Almeida C, Caetano R, Fausto M, Macinko J, Martins M, et al. Uma metodologia de avaliação do desempenho do sistema de saúde brasileiro.Cienc Saude Coletiva. 2004;9(3):711-24. DOI:10.1590/S1413-81232004000300021

Hartz1111 Hartz ZMA. Meta-avaliação da gestão em saúde: desafios para uma “nova saúde pública”. Cienc Saude Coletiva. 2012;17(4):829-37. DOI:10.1590/S1413-81232012000400004 (2012) indicated that it is important for health care systems to achieve their goals according to the established guidelines. Moreover, these systems should have relevant indicators for the evaluation of performance: “the estimation of these indicators may seem simple. However, there is no theoretical or methodological consensus on how these indicators should translate impacts or effects, and these estimations are always a central concern of the evaluators, considering political and institutional priorities, the complexity of each intervention, and the multiple approaches adopted”.

The structure of the health care system was a variable introduced in the model proposed by Brazilian authors among the spectrum of strategies adopted for evaluation of health care systems found in the literature.2222 Viacava F, Almeida C, Caetano R, Fausto M, Macinko J, Martins M, et al. Uma metodologia de avaliação do desempenho do sistema de saúde brasileiro.Cienc Saude Coletiva. 2004;9(3):711-24. DOI:10.1590/S1413-81232004000300021

Development and validation of an instrument for evaluation of the structure of primary health care services for the treatment of tuberculosis (TB) can be the first step toward the development of relevant indicators to evaluate the performance of health care systems.

Previous studies that aimed to validate instruments for evaluation of health care services1212 Harzheim E, Starfield B, Rajmil L, Álvarez-Dardet C, Stein AT. Consistência interna e confiabilidade da versão em português do Instrumento de Avaliação da Atenção Primária (PCATool-Brasil) para serviços de saúde infantil.Cad Saude Publica. 2006;22(8):1649-59. DOI:10.1590/S0102-311X2006000800013 and validation of psychometric indicators used reliability analysis techniques, exploratory factor analysis (EFA),5Carvalho HW, Patrick CJ, Krueger RF, Markon KE, Pinheiro AMV. Validade de construto da versão brasileira do Inventário Espectral de Externalização: evidências a partir de uma amostra de estudantes universitários.Rev Psiquiatr Clin. 2010;37(5):206-11. DOI:10.1590/S0101-60832010000500004,1212 Harzheim E, Starfield B, Rajmil L, Álvarez-Dardet C, Stein AT. Consistência interna e confiabilidade da versão em português do Instrumento de Avaliação da Atenção Primária (PCATool-Brasil) para serviços de saúde infantil.Cad Saude Publica. 2006;22(8):1649-59. DOI:10.1590/S0102-311X2006000800013 and confirmatory factor analysis (CFA).9Fernandes HM, Vasconcelos-Raposo JJB. Análise factorial confirmatória do TEOSQp. Psicol Reflex Crit. 2010;23(1):92-101. DOI:10.1590/S0102-79722010000100012,1313 Herrero J. El Análisis Factorial Confirmatorio en el estudio de la estructura y estabilidad de los instrumentos de evaluación: un ejemplo con el Cuestionario de Autoestima CA-14. Intervención Psicosocial.2010;19(3):289-300.

The objective of this study was to analyze the validity and reliability of an instrument that evaluates the structure of primary health care services for the treatment of TB.

METHODS

This study is part of a series of studiesa a Projeto multicêntrico intitulado “Avaliação da Atenção Básica para o tratamento da tuberculose na perspectiva dos profissionais de saúde e doentes em municípios do Brasil”, FAPESP 2011/09469-6, (GEOTB)/CNPq. conducted in five cities in the Northeast, Southeast and South regions of Brazil for evaluation of health care services for the treatment of TB.

The study was conducted in the following five cities: Sao José do Rio Preto, SP, Southeastern Brazil, with a population of 419,632 inhabitants and 16.4% coverage by the family health team (FHT); Natal, RN, Northeastern Brazil, with 806,203 inhabitants and 30.8% coverage by the FHT; Foz do Iguaçu, PR, Southern Brazil, with 325,132 inhabitants and 39.3% coverage by the FHT; Cabedelo, PB, Northeastern Brazil, with 57,926 inhabitants and 85.0% coverage by the FHT; and Uberaba, MG, Southeastern Brazil, with 296,259 inhabitants and 58.2% coverage by the FHT.b b Ministério da Saúde. Departamento de Atenção Básica. Histórico de cobertura da Saúde da Família. Brasília (DF); 2012 [cited 2011 Jan 1]. Available from:http://dab.saude.gov.br/portaldab/historico_cobertura_sf.php

The study group included health care professionals who constituted the minimum staff involved in the care of TB patients [physicians (general and family practitioners), nurses (nursing assistants/technicians) and community health workers (CHW)] in the primary health care system of the cities evaluated.

A structured instrument was built using 39 closed questions covering three evaluative components: human resources, physical resources, and service organization (see Annex), which were organized dichotomously (yes or no).

The construction of the instrument was based on extensive literature review through searches of quality assessment studies. The structure corresponded to the organization of health care services and included the resources available for the execution of health care services: physical resources (equipment, consumables, and facilities); human resources (number and qualification); and resources for maintenance of the existing infrastructure and technology (deemed as the organization of health care services).7Donabedian A. The effectiveness of quality assurance. Int J Qual Health Care. 1996;8(4):401-7. The Recommendations Manual for Control of Tuberculosis,c c Ministério da Saúde. Secretaria de Vigilância em Saúde. Programa Nacional de Controle da Tuberculose. Manual de recomendações para o controle da Tuberculose no Brasil. Brasília (DF); 2010. Nursing Protocol for Directly Observed Treatment of Tuberculosis,d d Ministério da Saúde. Secretaria de Vigilância em Saúde. Departamento de Vigilância Epidemiológica. Tratamento diretamente observado (TDO) da tuberculose na atenção básica: protocolo de enfermagem. Brasília (DF); 2011. and Best Practice Guidelines for the Care of Patients with TB2424 Williams G, Alarcon E, Jittimanee S, Walusinbi M, Sebek M, Berga E, et al. Guindance for the implementation of best practice for the care of patients with Tuberculosis. Int J Tuberc Lung Dis. 2008;12(3):236-40. were used. These documents provide the most relevant actions for the treatment of TB in health care services at the national and international levels.

The adequacy and consistency of the instrument items were also verified. Each variable that composed the evaluative component of the structure was discussed. The instrument was evaluated by three experts who did not know the study. The following items were qualitatively evaluated: organization, objectivity, clarity, readability, and comprehension of the content. The judges were instructed to write suggestions and comments to improve the items evaluated.1Alexandre NMC, Coluci MZO. Validade de conteúdo nos processos de construção e adaptação de instrumentos de medidas. Cienc Saude Coletiva. 2011;16(7):3061-8. DOI:10.1590/S1413-81232011000800006 The instrument was adjusted after incorporating all suggestions given.

For sample calculation, the number of professionals [including physicians (general and family physicians), nurses (nursing assistants/technicians) and CHW who worked in primary health care services in the municipalities] was calculated using the National Register of Health Care Units and excluding the duplicates. Taking the number of health care professionals working in basic health care in each municipality (489 in Uberaba; 633 in Sao José do Rio Preto; 518 in Foz do Iguaçu, 157 in Cabedelo, and 976 in Natal) as reference, the minimum number of professionals to be interviewed was obtained using the equation .

A sampling error of 0.05, confidence interval of 95%, and population percentage (p) of 50.0% were considered.

The value obtained was corrected for the total number of professionals using the equation . Therefore, a total of 1,037 professionals were interviewed in each municipality participating in the multicenter study (239 in Sao José do Rio Preto, 216 in Uberaba, 225 in Foz do Iguaçu, 240 in Natal, and 117 in Cabedelo). The number of professionals in each category was calculated using stratified proportional sample.

A simple random sampling procedure was used for the selection of health care services. Health care workers identified as key informants in the health care units drawn and who agreed to participate in the study were interviewed. The drawing of health care units was performed until the number of professionals predicted in the sample calculation was reached.

The sample size was calculated to meet the criteria for conducting factor analysis, considering that the sample size should have a ratio of 10 respondents for each questionnaire item, which would allow working with 390 respondents.1010 Hair Jr. JF. Multivariate Data Análysis. 5. ed. Porto Alegre: Bookman; 2005. For variables with dichotomous responses, the recommendation was to work with samples > 1,000 to avoid numerical convergence and singularity.2020 Savalei V, Rhemtulla M. The performance of robust test statistics with categorical data. Brit J Math Stat Psy. 2013;66(2):201-23. DOI:10.1111/j.2044-8317.2012.02049.xTherefore, the calculated sample would allow for a possible loss of 3.5%.

Data were analyzed by descriptive techniques. The reliability of the instrument was measured using Cronbach’s α index. The validity was measured in two ways: using EFA, which aimed to simplify data structure and summarize the information considering the large number of variables involved, and CFA, which tested the preconceived hypothesis about data structure found in EFA.9Fernandes HM, Vasconcelos-Raposo JJB. Análise factorial confirmatória do TEOSQp. Psicol Reflex Crit. 2010;23(1):92-101. DOI:10.1590/S0102-79722010000100012,2020 Savalei V, Rhemtulla M. The performance of robust test statistics with categorical data. Brit J Math Stat Psy. 2013;66(2):201-23. DOI:10.1111/j.2044-8317.2012.02049.x The results of factor analyses allowed clusterization of the questionnaire items into factors.

The average value (μ = P, Percentage of all health care units with the characteristics studied), standard deviation (), and coefficient of variation for the set of items of the questionnaire that composed each factor (structure indicators) were estimated.

Use of EFA assumes that the variables can be grouped according to their correlations to obtain factors containing variables that are highly correlated with each other but are poorly correlated with other variables.1818 Paes NA, Santos CSA. As estatísticas de nascimento e os fatores maternos e da criança nas microrregiões do Nordeste brasileiro: uma investigação usando análise fatorial. Cad Saude Publica. 2010;26(2):311-22. DOI:10.1590/S0102-311X2010000200010,2525 Zanella A, Seidel EJ, Lopes LFD. Validação de questionário de satisfação usando análise fatorial. Rev Ingepro. 2010;2:102-12. However, EFA assumes that the variables are quantitative and continuous and therefore involves calculation of Gaussian correlation matrices to estimate the factors and exploratory factor model. Because the variables were dichotomous, matrices of tetrachoric correlations were used, which is appropriate in variable metrics for estimating the factors and factor model.1919 Reichenheim ME, Moraes CL. Operacionalização de adaptação transcultural de instrumentos de aferição usados em epidemiologia. Rev Saude Publica. 2007;41(4):665-73. DOI:10.1590/S0034-89102006005000035,2020 Savalei V, Rhemtulla M. The performance of robust test statistics with categorical data. Brit J Math Stat Psy. 2013;66(2):201-23. DOI:10.1111/j.2044-8317.2012.02049.x

EFA on the matrix of tetrachoric correlations1616 Maroco J. Análise Estatística com utilização do SPSS. 3. ed. Edições Sílabo; 2007.,1717 Nestler S. A Monte Carlo study comparing PIV, ULS and DWLS in the estimation of dichotomous confirmatory factor analysis. Brit J Math Stat Psy. 2013;66(1):127-43. DOI:10.1111/j.2044-8317.2012.02044.x used principal component analysis to extract the factors, followed by Varimax rotation. The common factors retained were those with eigenvalues of > 1, in line with the screen plot and the percentage of retained variance. It is recommended that the number of factors selected should correspond to a minimum of 60.0% of the total variance and should retain ≥ 3 items in each factor.1212 Harzheim E, Starfield B, Rajmil L, Álvarez-Dardet C, Stein AT. Consistência interna e confiabilidade da versão em português do Instrumento de Avaliação da Atenção Primária (PCATool-Brasil) para serviços de saúde infantil.Cad Saude Publica. 2006;22(8):1649-59. DOI:10.1590/S0102-311X2006000800013 This is because use of a single criterion1616 Maroco J. Análise Estatística com utilização do SPSS. 3. ed. Edições Sílabo; 2007. can lead to retention of more/less factors than those necessary to describe the underlying structure.

The Kaiser-Meyer-Olkin (KMO) test1616 Maroco J. Análise Estatística com utilização do SPSS. 3. ed. Edições Sílabo; 2007.and Bartlett’s test of sphericity1616 Maroco J. Análise Estatística com utilização do SPSS. 3. ed. Edições Sílabo; 2007. were used to assess data adequacy to EFA. The latter test examined whether the correlation matrix of the population was an identity matrix, i.e., no correlation existed between the variables with statistical significance at p < 0.10. The values of the KMO criterion were used to verify adequacy to the method and varied between 0 and 1 (≥ 0.80: outstanding; 0.70-0.79: acceptable; 0.60-0.69: average; 0.50-0.59: poor, and < 0.50: inadequate).1010 Hair Jr. JF. Multivariate Data Análysis. 5. ed. Porto Alegre: Bookman; 2005.

Another measure of data adequacy to EFA includes the values of the principal diagonal of the anti-image correlation matrix. It is known as a measure of sampling adequacy or KMOi and its value should be > 0.5 for each variable in the analysis. KMOi values of < 0.5 indicate that the variable did not fit the structure defined by the other variables and therefore should be excluded from EFA.1616 Maroco J. Análise Estatística com utilização do SPSS. 3. ed. Edições Sílabo; 2007. The statistical significance of the factor loadings that represented the correlation between an original variable and its factor was evaluated for factor interpretation. To achieve a power level of 80.0% and a significance level of 0.05, sample sizes of ≥ 350 and factor loadings with correlation of ≥ 0.30 are suggested.1010 Hair Jr. JF. Multivariate Data Análysis. 5. ed. Porto Alegre: Bookman; 2005.

The commonalities (fraction of the variance that a variable shares with all other variables considered) were evaluated for each variable, observing whether it met the acceptable levels of explanation (> 0.5) and its contribution to the study. A residue matrix (observed correlations minus estimated correlations) with more than 50.0% of the residues with an absolute value of < 0.05 was considered a factorial model indicator with good fit.1616 Maroco J. Análise Estatística com utilização do SPSS. 3. ed. Edições Sílabo; 2007.

The CFA model was adjusted using free asymptotic distribution (FAD) or weighted least squares (WLS) because it did not require the items to have multivariate normality.2020 Savalei V, Rhemtulla M. The performance of robust test statistics with categorical data. Brit J Math Stat Psy. 2013;66(2):201-23. DOI:10.1111/j.2044-8317.2012.02049.x The Statistica software including the Asymptotically Distribution Free Gramian (ADFG) estimator was used in these situations.e e StatSoft Inc. STATISTICA for Windows: computer program manual. Tulsa, OK: StatSoft; 2011, versão 12.

The quality of the CFA model was evaluated using the goodness-of-fit index (GFI),1616 Maroco J. Análise Estatística com utilização do SPSS. 3. ed. Edições Sílabo; 2007. which is the fraction of the correlations between the original variables that are explained by the model. GFI values of < 0.9 indicate poor fit to the data, values of 0.9-0.95 indicate good fit, values of > 0.95 indicate very good fit, and values equal to 1 indicate perfect fit. GFI tends to increase with increased sample size. The adjusted goodness-of-fit index (AGFI) was calculated, which is the value adjusted for the number of degrees of freedom of the sample.1616 Maroco J. Análise Estatística com utilização do SPSS. 3. ed. Edições Sílabo; 2007.

Reliability was calculated using Cronbach’s α index, which measures the internal consistency of questionnaire items.1616 Maroco J. Análise Estatística com utilização do SPSS. 3. ed. Edições Sílabo; 2007. The recommended lower limit for α is 0.7. However, in exploratory studies, values of ≥ 0.6 are acceptable.1010 Hair Jr. JF. Multivariate Data Análysis. 5. ed. Porto Alegre: Bookman; 2005.

This study was approved by the Research Ethics Committee of Escola de Enfermagem de Ribeirão Preto, Universidade de São Paulo, under Protocol 1264/2011 in accordance with Resolution 196/96 of the National Health Council. All participants signed the informed consent form.

RESULTS

A total of 1,037 health care professionals were interviewed: 239 in Sao José do Rio Preto, 216 in Uberaba, 225 in Foz do Iguaçu, 117 in Cabedelo, and 240 in Natal.Table 1 shows the frequency distribution of the questionnaire items for evaluation of the structure of primary health care services for the treatment of TB patients.

Table 1
Frequency distribution of the questionnaire items for evaluation of HCS according to the structure of primary health care units for the treatment of TB. Brazil, 2011.

The validation process of the questionnaire items tested five different EFA models with the preset number of factors between 3 and 7. This is because the scale could conceptually identify ≥ 3 components. Variables with KMOi values of > 0.5 were maintained in the model. The best model considered retention of five factors, with KMO = 0.728, p < 0.0001 for Bartlett’s sphericity test (c2Angelotti LCZ, Alexandre PBD, Miranzi SSC, Scatena LM. Qualidade de dados de notificação e acompanhamento dos casos de tuberculose em Minas Gerais.Rev Enferm Atenc Saude. 2013;2(2):84-98. = 5473.94, gl = 190), 85.9% of the total variance explained, residue matrix with 44.2% of the residues > 0.05, and eigenvalues of > 1.

The five factors retained, their factor loadings, commonalities, eigenvalues, explained variance for each factor, and Cronbach’s α are shown in Table 2.

Table 2
Structure, retained factors, factor loadings, commonalities, eigenvalues, explained variance, and Cronbach’s α. Brazil, 2011.

The first factor, “access to recording instruments”, consisted of the following items: notification forms, medical records, monthly monitoring forms, daily form of directly observed treatment (DOT), and green paper. The second factor, “training of health professionals”, consisted of five items: nurses, nursing assistants/technicians, physicians, CHW trained to care for TB patients in the past three years, and professional training conducted by the epidemiological surveillance department in the last 12 months. The third factor, “coordination of health care services with other levels of care”, consisted of four items: coordination of health care services with other levels of care when the patient required consultations because of other comorbidities, coinfection with HIV, and treatment complications and when difficulties in the execution of DOT occurred. The fourth factor, “professionals involved in the care of TB patients”, consisted of three items: nursing assistants/technicians, physicians, and CHW involved in the care of TB patients in health care services. The fifth factor, “availability of supplies and equipment”, consisted of three items: availability of pots for sputum examination, form for sputum smear microscopy request, and form for microbiological culture request in health care services.

The CFA model adjusted with the ADFG estimator provided a value of 0.996 for the GFI index and 0.995 for the adjusted GFI index.

The mean percentage of questionnaire items that composed each factor (structural indicators) and their standard deviation are shown in Table 3. Coordination of health care services with other levels of care showed higher data variability in health care services.

Table 3
Indicators for evaluation of the structure of health care services for the treatment of tuberculosis. Brazil, 2011.

DISCUSSION

Twenty of the 39 items originally proposed were validated, allowing reduction of the data structure (Table 1). The 20 items were validated in five factors, meeting the conditions for realization of EFA, Bartlett’s sphericity test and KMO with acceptable quality,1616 Maroco J. Análise Estatística com utilização do SPSS. 3. ed. Edições Sílabo; 2007. total variance of > 60.0%, at least three items retained in each factor,1010 Hair Jr. JF. Multivariate Data Análysis. 5. ed. Porto Alegre: Bookman; 2005.,1616 Maroco J. Análise Estatística com utilização do SPSS. 3. ed. Edições Sílabo; 2007. factor loadings of > 0.3, commonalities of > 0.5, and eigenvalues of > 11616 Maroco J. Análise Estatística com utilização do SPSS. 3. ed. Edições Sílabo; 2007. (Table 2).

Reliability analysis indicated Cronbach’s α values that varied between 0.6460 and 0.7802 (Table 2), and these results were within the values considered acceptable in the literature.1010 Hair Jr. JF. Multivariate Data Análysis. 5. ed. Porto Alegre: Bookman; 2005. Cronbach’s α values of factors 4 and 5 (Table 2) were < 0.7 because of the retention of three items for each factor, although more items would be necessary to increase factor reliability. The items proposed in the questionnaire that could be correlated with the items of factor 4 – “nurses” and “other professionals” involved in the care of TB patients – were removed from EFA. The item “nurses” showed variance close to zero because most respondents (99.3%) recognized that this professional category was most closely involved in the care of TB patients. When variance is zero, the measure is constant and the item can be removed from EFA because of the lack of variability.1616 Maroco J. Análise Estatística com utilização do SPSS. 3. ed. Edições Sílabo; 2007. Nurses are recognized as responsible for most of the activities related to the control of TB.3Caliari JS, Figueiredo RM. Tuberculose: perfil de doentes, fluxo de atendimento e opinião de enfermeiros. Acta Paul Enferm.2012;25(1):43-7. The item “other professionals” was removed because the value of commonality was < 0.5, possibly because of little acknowledgment (45.1%) by respondents that these professionals are involved in the care of TB patients. This indicates the difficulty of adding new professional categories in the actions for TB control.2323 Villa TCS, Andrade, RLP, Arakawa T, Magnabosco, GT, Beraldo AA, Monroe AA. Satisfação do usuário com os serviços de atenção à tuberculose em Ribeirão Preto, 2008. Cad Saude Coletiva.2012;20(2):234-43.

The quality of the adjusted CFA model (GFI = 0.995) was assessed as very good1010 Hair Jr. JF. Multivariate Data Análysis. 5. ed. Porto Alegre: Bookman; 2005. in the five municipalities. The five factors retained were adequate to evaluate the structure of health care services for TB treatment in the municipalities evaluated. This result showed that the theoretical framework was adequate for the data under observation. The indicator “training” had the lowest score. Training is among the institutional challenges of the primary health care department of the Brazilian Ministry of Health, which has proposed a revision of the training processes and has emphasized on continued education of health care teams, coordinators, and managers.f f Ministério da Saúde. Departamento de Atenção Básica. Atenção Básica e a Saúde da Família. Brasília (DF); 2013 [cited 2013 Nov 19]. Available from: http://dab.saude.gov.br/atencaobasica.php

Lack of training of health care professionals in the treatment of TB was one of the factors that limited the performance of health care teams. Other studies have indicated the need for continuous training of health care professionals to achieve higher quality of services.6Cotta RMM, Schott M, Azeredo CM, Franceschini SCC, Priore SE, Dias G. Organização do trabalho e perfil dos profissionais do Programa Saúde da Família: um desafio na reestruturação da atenção básica em saúde.Epidemiol Serv Saude. 2006;15(3):7-18.Training of health care professionals should be permanent and should include educational workshops and participation of academics and health care professionals.2121 Torres HC, Amaral MA, Amorim MM, Cyrino AP, Bodstein R. Capacitação de profissionais da atenção primária à saúde para educação em Diabetes Mellitus.Acta Paul Enferm. 2010;23(6):751-6.

The item “access to recording instruments” achieved a low score compared with the other indicators. The quality of data of health surveillance systems is related to the level of completeness and validity of the data recorded. Therefore, access to recording instruments is necessary. In this respect, previous studies have indicated the inadequacy in filling medical reports and data input into the database of the Information System for Notifiable Diseases.2Angelotti LCZ, Alexandre PBD, Miranzi SSC, Scatena LM. Qualidade de dados de notificação e acompanhamento dos casos de tuberculose em Minas Gerais.Rev Enferm Atenc Saude. 2013;2(2):84-98.

Most health care services were coordinated with other levels of care in situations when the patient was coinfected with HIV, required consultations because of other comorbidities, and presented clinical complications and when difficulties in the execution of DOT occurred. However, this indicator had the highest coefficient of variation, i.e., varied more between the services evaluated compared with other indicators. Therefore, managers should pay more attention to services that do not conduct these activities satisfactorily.

The indicator with the best performance was “availability of supplies”. However, the performance was not good enough to ensure that all health care services received the necessary supplies for the treatment of TB.

The questions “does this health care service assist patients outside office hours?”, “are patients advised to seek health care services outside the scheduled date?” and “is there a computerized system in the health care unit?” presented KMOi values of < 0.5 and were therefore removed from EFA. This indicates that these items did not fit the structure established by the other items, i.e., they did not significantly correlated with the other items.1616 Maroco J. Análise Estatística com utilização do SPSS. 3. ed. Edições Sílabo; 2007. Another study reported that municipalities that extended their opening hours could not provide patients with increased access to health care services.1515 Machado CV, Lima LD, Viana LS. Configuração da atenção básica e do Programa Saúde da Família em grandes municípios do Rio de Janeiro, Brasil.Cad Saude Publica. 2008;24(1):545-57. DOI:10.1590/S0102-311X2008001300010

The item related to the guidance provided to patients to spontaneously seek health care services if they experienced clinical complications was present in 95.5% of the health care units. However, this indicator had low variability and was therefore not validated.

A computerized system was not present in most health care units. The system was computerized with online access only in reference centers for TB, which explains the low presence of these systems in basic health care units.1414 Hijjar MA, Gerhardt G, Teixeira GM, Procópio MJ. Retrospect of tuberculosis control in Brazil. Rev Saude Publica. 2007;41Suppl1:50-8. DOI:10.1590/S0034-89102007000800008

Other strategies should be implemented to improve the indicators evaluated. In addition, they should promote the training of health care professionals in the treatment of TB and increase access to recording instruments, considering that these two indicators had the worst scores.

The present instrument showed feasibility of application and potential to evaluate the structure of health care services for the treatment of TB considering the data from different municipalities. The results were reliable considering the fit of the CFA model using FAD methods that were appropriate for the estimation of categorical items (dichotomous) with non-parametric distribution. Accurate estimates can be obtained using estimation methods that are consistent with the characteristics of the instrument items. However, use of dichotomous variables in validation instruments will be possible only with large samples; thus, its implementation may be limited.

Annex   Questionnaire items for evaluation of the structure of health care services for the treatment of TB. Brazil, 2011.

Questionnaire items Human resources 1. Does this service have a nurse responsible for the care of TB patients? 2. Does this service have a nursing assistant/technician responsible for the care of TB patients? 3. Does this service have a physician responsible for the care of TB patients? 4. Does this service have a community health worker responsible for the care of TB patients? 5. Are there other professionals in this service (psychologist, social worker) involved in the care of TB patients? 6. In the past three years, were nursing professionals trained to care for TB patients? 7. In the past three years, were nursing assistants/technicians trained to care for TB patients? 8. In the last three years, were physicians trained to care for TB patients? 9. In the past three years, were CHW trained to care for TB patients? 10. In the past three years, were other professionals trained to care for TB patients? 11. In the last 12 months, did the epidemiological health surveillance agencies conduct training on TB? 12. In the last 12 months, did the health care units train health care workers on TB? 13. Is there a professional responsible for the realization of DOT in this health care unit? Physical resources 14. In this service, are medical clinics available for the care of TB patients? 15. In this service, are nursing care rooms available to TB patients? 16. At present, are basic food staples and food vouchers available to TB patients? 17. At present, are transportation vouchers available to TB patients? 18. Do professionals have access to notification forms? 19. Do professionals have access to medical records? 20. Do professionals have access to monthly treatment monitoring reports? 21. Do professionals have access to daily DOT reports? 22. Do professionals have access to the green paper? 23. Does the health care unit have pots for sputum examination? 24. Has the health care unit requested sputum smear microscopy? 25. Has the health care unit requested microbiological cultures? 26. Has the health care unit requested HIV examination? 27. Does the health care unit have reference and counter-reference cards? 28. Did the availability (delivery) of TB drugs in this health care unit meet the demand in the last 12 months? Service organization 29. Does this health care service assist patients outside business hours (after 5 pm)? 30. Are discussions regarding cases involving TB patients who are treated in this health care unit conducted by health care professionals responsible for the treatment of this condition? 31. Can TB patients under treatment schedule a consultation within 24h in this health care unit in case they experience complications arising from TB or its treatment? 32. Are TB patients under treatment advised to seek health care units outside the scheduled date of return in case they experience any complications arising from TB or its treatment? 33. Does this health care unit have coordination with other levels of care when the patient requires consultations because of other comorbidities (diabetes, hypertension, psychiatric diseases)? 34. Does this health care unit have coordination with other levels of care when the patient is coinfected with HIV? 35. Does this health care unit have coordination with other levels of care when the patient experiences complications arising from treatment? 36. Does this health care unit have coordination with other levels of care when difficulties in the realization of DOT occur? 37. Is the frequency with which the laboratory conducts TB examinations during treatment sufficient to meet the demand for this service? 38. Do the sputum smear microscopy results become available within 7 days? 39. Is there a computerized system in the health care units with information on the patients with TB (scheduled appointments, work absenteeism, issues related with drug therapy)? TB: tuberculosis; CHW: community health worker; DOT: directly observed treatment

REFERENCES

  • 1
    Alexandre NMC, Coluci MZO. Validade de conteúdo nos processos de construção e adaptação de instrumentos de medidas. Cienc Saude Coletiva 2011;16(7):3061-8. DOI:10.1590/S1413-81232011000800006
  • 2
    Angelotti LCZ, Alexandre PBD, Miranzi SSC, Scatena LM. Qualidade de dados de notificação e acompanhamento dos casos de tuberculose em Minas Gerais.Rev Enferm Atenc Saude 2013;2(2):84-98.
  • 3
    Caliari JS, Figueiredo RM. Tuberculose: perfil de doentes, fluxo de atendimento e opinião de enfermeiros. Acta Paul Enferm.2012;25(1):43-7.
  • 4
    Carvalho ALB, Souza MF, Shimizu HE, Senra IMVB, Oliveira KC. A gestão do SUS e as práticas de monitoramento e avaliação: possibilidades e desafios para a construção de uma agenda estratégica. Cienc Saude Coletiva 2012;17(4):901-11. DOI:10.1590/S1413-81232012000400012
  • 5
    Carvalho HW, Patrick CJ, Krueger RF, Markon KE, Pinheiro AMV. Validade de construto da versão brasileira do Inventário Espectral de Externalização: evidências a partir de uma amostra de estudantes universitários.Rev Psiquiatr Clin. 2010;37(5):206-11. DOI:10.1590/S0101-60832010000500004
  • 6
    Cotta RMM, Schott M, Azeredo CM, Franceschini SCC, Priore SE, Dias G. Organização do trabalho e perfil dos profissionais do Programa Saúde da Família: um desafio na reestruturação da atenção básica em saúde.Epidemiol Serv Saude 2006;15(3):7-18.
  • 7
    Donabedian A. The effectiveness of quality assurance. Int J Qual Health Care. 1996;8(4):401-7.
  • 8
    Felisberto E. Da teoria à formulação de uma Política Nacional de Avaliação em Saúde: reabrindo o debate. Cienc Saude Coletiva 2006;11(3):553-63. DOI:10.1590/S1413-81232006000300002
  • 9
    Fernandes HM, Vasconcelos-Raposo JJB. Análise factorial confirmatória do TEOSQp. Psicol Reflex Crit 2010;23(1):92-101. DOI:10.1590/S0102-79722010000100012
  • 10
    Hair Jr. JF. Multivariate Data Análysis. 5. ed. Porto Alegre: Bookman; 2005.
  • 11
    Hartz ZMA. Meta-avaliação da gestão em saúde: desafios para uma “nova saúde pública”. Cienc Saude Coletiva 2012;17(4):829-37. DOI:10.1590/S1413-81232012000400004
  • 12
    Harzheim E, Starfield B, Rajmil L, Álvarez-Dardet C, Stein AT. Consistência interna e confiabilidade da versão em português do Instrumento de Avaliação da Atenção Primária (PCATool-Brasil) para serviços de saúde infantil.Cad Saude Publica 2006;22(8):1649-59. DOI:10.1590/S0102-311X2006000800013
  • 13
    Herrero J. El Análisis Factorial Confirmatorio en el estudio de la estructura y estabilidad de los instrumentos de evaluación: un ejemplo con el Cuestionario de Autoestima CA-14. Intervención Psicosocial.2010;19(3):289-300.
  • 14
    Hijjar MA, Gerhardt G, Teixeira GM, Procópio MJ. Retrospect of tuberculosis control in Brazil. Rev Saude Publica 2007;41Suppl1:50-8. DOI:10.1590/S0034-89102007000800008
  • 15
    Machado CV, Lima LD, Viana LS. Configuração da atenção básica e do Programa Saúde da Família em grandes municípios do Rio de Janeiro, Brasil.Cad Saude Publica 2008;24(1):545-57. DOI:10.1590/S0102-311X2008001300010
  • 16
    Maroco J. Análise Estatística com utilização do SPSS. 3. ed. Edições Sílabo; 2007.
  • 17
    Nestler S. A Monte Carlo study comparing PIV, ULS and DWLS in the estimation of dichotomous confirmatory factor analysis. Brit J Math Stat Psy. 2013;66(1):127-43. DOI:10.1111/j.2044-8317.2012.02044.x
  • 18
    Paes NA, Santos CSA. As estatísticas de nascimento e os fatores maternos e da criança nas microrregiões do Nordeste brasileiro: uma investigação usando análise fatorial. Cad Saude Publica 2010;26(2):311-22. DOI:10.1590/S0102-311X2010000200010
  • 19
    Reichenheim ME, Moraes CL. Operacionalização de adaptação transcultural de instrumentos de aferição usados em epidemiologia. Rev Saude Publica 2007;41(4):665-73. DOI:10.1590/S0034-89102006005000035
  • 20
    Savalei V, Rhemtulla M. The performance of robust test statistics with categorical data. Brit J Math Stat Psy. 2013;66(2):201-23. DOI:10.1111/j.2044-8317.2012.02049.x
  • 21
    Torres HC, Amaral MA, Amorim MM, Cyrino AP, Bodstein R. Capacitação de profissionais da atenção primária à saúde para educação em Diabetes Mellitus.Acta Paul Enferm 2010;23(6):751-6.
  • 22
    Viacava F, Almeida C, Caetano R, Fausto M, Macinko J, Martins M, et al. Uma metodologia de avaliação do desempenho do sistema de saúde brasileiro.Cienc Saude Coletiva 2004;9(3):711-24. DOI:10.1590/S1413-81232004000300021
  • 23
    Villa TCS, Andrade, RLP, Arakawa T, Magnabosco, GT, Beraldo AA, Monroe AA. Satisfação do usuário com os serviços de atenção à tuberculose em Ribeirão Preto, 2008. Cad Saude Coletiva.2012;20(2):234-43.
  • 24
    Williams G, Alarcon E, Jittimanee S, Walusinbi M, Sebek M, Berga E, et al. Guindance for the implementation of best practice for the care of patients with Tuberculosis. Int J Tuberc Lung Dis 2008;12(3):236-40.
  • 25
    Zanella A, Seidel EJ, Lopes LFD. Validação de questionário de satisfação usando análise fatorial. Rev Ingepro 2010;2:102-12.

  • Research supported by the Fundação de Amparo à Pesquisa do Estado de São Paulo (FAPESP – Process 2011/09469-6).
  • a
    Projeto multicêntrico intitulado “Avaliação da Atenção Básica para o tratamento da tuberculose na perspectiva dos profissionais de saúde e doentes em municípios do Brasil”, FAPESP 2011/09469-6, (GEOTB)/CNPq.
  • b
    Ministério da Saúde. Departamento de Atenção Básica. Histórico de cobertura da Saúde da Família. Brasília (DF); 2012 [cited 2011 Jan 1]. Available from:http://dab.saude.gov.br/portaldab/historico_cobertura_sf.php
  • c
    Ministério da Saúde. Secretaria de Vigilância em Saúde. Programa Nacional de Controle da Tuberculose. Manual de recomendações para o controle da Tuberculose no Brasil. Brasília (DF); 2010.
  • d
    Ministério da Saúde. Secretaria de Vigilância em Saúde. Departamento de Vigilância Epidemiológica. Tratamento diretamente observado (TDO) da tuberculose na atenção básica: protocolo de enfermagem. Brasília (DF); 2011.
  • e
    StatSoft Inc. STATISTICA for Windows: computer program manual. Tulsa, OK: StatSoft; 2011, versão 12.
  • f
    Ministério da Saúde. Departamento de Atenção Básica. Atenção Básica e a Saúde da Família. Brasília (DF); 2013 [cited 2013 Nov 19]. Available from: http://dab.saude.gov.br/atencaobasica.php

History

  • Received
    29 Apr 2014
  • Accepted
    7 Aug 2014
  • Online publication
    27 Feb 2015
Faculdade de Saúde Pública da Universidade de São Paulo São Paulo - SP - Brazil
E-mail: revsp@org.usp.br