ABSTRACT:

Objective:

To evaluate the validity and reliability of the Johns Hopkins Fall Risk Assessment Tool (JH-FRAT), which assesses the risk of falls in hospitalized inpatients.

Method:

Study with 297 patients at a hospital in São Paulo, using retrospective data form 2014. Validity was assessed by accuracy (sensitivity, specificity, positive predictive value - PPV and negative predictive value - NPV) and discriminant analysis (comparison of patients with and without falls in relation to the scale items and comparison of previous risk situations in relation to the injury). The χ² test and Fisher’s exact test were used. Reliability was assessed by reproducibility between methods and interobserver test-retest comparison in a subsample of 60 patients. We used the Kappa, quadratic weighted Kappa and PABAK statistics.

Results:

Sensitivity was 97.0%, specificity was 6%, PPV was 36.2% and NPV was 90.6%. Five of the eight items of the scale and the overall classification showed risk discrimination capability (p < 0.050). The risk of previous situations did not discriminate the injury resulting from the falls (p = 0.557). Reproducibility between methods was substantial (PABAK = 0.71). The interobserver reproducibility ranged between items (PABAK 0.25 to 1.00) and was substantial to the overall risk classification (PABAK = 0.71).

Conclusion:

JH-FRAT showed validity and reliability expected of a screening tool for risk of falls, and it can contribute to the implementation of fall management strategies in hospitals.

Keywords:
Data accuracy; Reproducibility of results; Reliability and validity; Nursing methodology research; Accidental falls; Quality of health care

INTRODUCTION

Falls that occur with hospitalized patients are adverse events with negative repercussions for patients and institutions¹1. Schwendimann R, Bühler H, De Geest S, Milisen K. Falls and consequent injuries in hospitalized patients: effects of an interdisciplinary falls prevention program. BMC Health Serv Res 2006; 6: 69. https://dx.doi.org/10.1186%2F1472-6963-6-69
https://dx.doi.org/10.1186%2F1472-6963-6... ^,22. National Health Services. National Patient Safety Agency. Slips, trips and falls in hospital: The third report from the Patient Safety Observatory. London: National Health Services; 2007.^,33. Oliver D, Healy F. Falls risk prediction tools for hospital inpatients: do they work? Nurs Times 2009; 105(7): 18-21.^,44. Cumbler EU, Simpson JR, Rosenthal LD, Likosky DJ. Inpatient falls: defining the problem and identifying possible solutions. Part I: An evidence-based review. Neurohospitalist 2013; 3(3): 135-43. https://doi.org/10.1177/1941874412470665
https://doi.org/10.1177/1941874412470665... ^,55. Martinez MC, Iwamoto VE, Latorre MRDO, Noronha AM, Oliveira APS, Cardoso CEA, et al. Transcultural adaptation of the Johns Hopkins Fall Risk Assessment Tool. Rev Latino-Am Enfermagem 2016; 24: e2783. http://dx.doi.org/10.1590/1518-8345.1158.2783
http://dx.doi.org/10.1590/1518-8345.1158... , whose prevention should adopt evidence-based practices such as having the call bell within the patient’s reach, hospital bed in low position and treatment of the causes of mental confusion²2. National Health Services. National Patient Safety Agency. Slips, trips and falls in hospital: The third report from the Patient Safety Observatory. London: National Health Services; 2007.^,33. Oliver D, Healy F. Falls risk prediction tools for hospital inpatients: do they work? Nurs Times 2009; 105(7): 18-21.^,66. Johns Hopkins Hospital. Fall risk assessment, prevention and management, adult. In: Johns Hopkins Hospital, editor. Nursing practice and organization manual: clinical protocols and procedures. Baltimore: Johns Hopkins Hospital; 2011. v. 2.^,77. Martinez MC, Iwamoto VE, Noronha AM, Oliveira APS, Sant'Ana THS, Cardoso CEA. Avaliação do risco de quedas em pacientes internados: por que realizar e como conduzir? Revista ACRED 2016; 6(11): 136-45.^,88. Agency for Healthcare Research and Quality. Preventing falls in hospitals: a toolkit for improving quality of care. Rockvilee: Agency for Healthcare Research and Quality; 2013.. Among the recommended practices is the use of scales that identify risk conditions for the occurrence of falls, either in the form of specific risks or in risk stratification, directing interventions compatible with the identified needs²2. National Health Services. National Patient Safety Agency. Slips, trips and falls in hospital: The third report from the Patient Safety Observatory. London: National Health Services; 2007.^,33. Oliver D, Healy F. Falls risk prediction tools for hospital inpatients: do they work? Nurs Times 2009; 105(7): 18-21.^,44. Cumbler EU, Simpson JR, Rosenthal LD, Likosky DJ. Inpatient falls: defining the problem and identifying possible solutions. Part I: An evidence-based review. Neurohospitalist 2013; 3(3): 135-43. https://doi.org/10.1177/1941874412470665
https://doi.org/10.1177/1941874412470665... ^,77. Martinez MC, Iwamoto VE, Noronha AM, Oliveira APS, Sant'Ana THS, Cardoso CEA. Avaliação do risco de quedas em pacientes internados: por que realizar e como conduzir? Revista ACRED 2016; 6(11): 136-45.^,88. Agency for Healthcare Research and Quality. Preventing falls in hospitals: a toolkit for improving quality of care. Rockvilee: Agency for Healthcare Research and Quality; 2013..

Among the instruments available in the international literature, two scales went through the cross-cultural adaptation process for use in Brazil, the Morse Fall Scale⁹9. Urbanetto JS, Creutzberg M, Franz F, Ojeda BS, Gustavo AS, Bittencourt HR, et al. Morse fall scale: translation and transcultural adaptation for the Portuguese language. Rev Esc Enferm USP 2013; 47(3): 569-75. http://dx.doi.org/10.1590/S0080-623420130000300007
http://dx.doi.org/10.1590/S0080-62342013... and the Johns Hopkins Fall Risk Assessment Tool (JH-FRAT)⁵5. Martinez MC, Iwamoto VE, Latorre MRDO, Noronha AM, Oliveira APS, Cardoso CEA, et al. Transcultural adaptation of the Johns Hopkins Fall Risk Assessment Tool. Rev Latino-Am Enfermagem 2016; 24: e2783. http://dx.doi.org/10.1590/1518-8345.1158.2783
http://dx.doi.org/10.1590/1518-8345.1158... ; both did not report the measurement properties of the Brazilian version. This study chose to evaluate the measurement properties of the JH-FRAT scale as an instrument whose content’s validity was already demonstrated in the Brazilian version, showing good applicability in systematized protocols and simple operationalization⁵5. Martinez MC, Iwamoto VE, Latorre MRDO, Noronha AM, Oliveira APS, Cardoso CEA, et al. Transcultural adaptation of the Johns Hopkins Fall Risk Assessment Tool. Rev Latino-Am Enfermagem 2016; 24: e2783. http://dx.doi.org/10.1590/1518-8345.1158.2783
http://dx.doi.org/10.1590/1518-8345.1158... .

JH-FRAT was structured by professionals from Johns Hopkins Hospital and the Johns Hopkins University School of Nursing for integration into the institutional fall management program for hospitalized adult patients⁶6. Johns Hopkins Hospital. Fall risk assessment, prevention and management, adult. In: Johns Hopkins Hospital, editor. Nursing practice and organization manual: clinical protocols and procedures. Baltimore: Johns Hopkins Hospital; 2011. v. 2.^,1010. Poe SS, Cvach MM, Gartrell DG, Radzik BR, Joy TL. An Evidence-based Approach to Fall Risk Assessment, Prevention and Management: Lessons Learned. J Nurs Care Qual 2005; 20(2): 107-16.^,1111. Poe SS, Cvach MM, Dawson PB, Straus H, Hill EE. The Johns Hopkins Fall Risk assessment Tool: Postimplementation Evaluation. J Nurs Care Qual 2007; 22(4): 293-8. https://doi.org/10.1097/01.NCQ.0000290408.74027.39
https://doi.org/10.1097/01.NCQ.000029040... . This scale was structured based on risk factors identified in the literature - providing a risk classification for directing preventive measures -, tested in patients and adjusted by a consensus group¹⁰10. Poe SS, Cvach MM, Gartrell DG, Radzik BR, Joy TL. An Evidence-based Approach to Fall Risk Assessment, Prevention and Management: Lessons Learned. J Nurs Care Qual 2005; 20(2): 107-16.. After two years of use in the institution, the acceptability and content of the scale were evaluated, focusing on the relationship between items, the scoring model and the clarity and relevance of the items, resulting in the definition of its final form and guaranteeing its content’s validity¹¹11. Poe SS, Cvach MM, Dawson PB, Straus H, Hill EE. The Johns Hopkins Fall Risk assessment Tool: Postimplementation Evaluation. J Nurs Care Qual 2007; 22(4): 293-8. https://doi.org/10.1097/01.NCQ.0000290408.74027.39
https://doi.org/10.1097/01.NCQ.000029040... .

The scale’s measurement properties were evaluated in patients hospitalized in South Korea¹²12. Kim KS, Kim JA, Choi YK, Kim YJ, Park MH, Kim HY, et al. A Comparative Study on the Validity of Fall Risk assessment Scales in Korean Hospitals. Asian Nurs Res 2011; 5(1): 28-37. https://doi.org/10.1016/S1976-1317(11)60011-X
https://doi.org/10.1016/S1976-1317(11)60... and in the elderly in American nursing homes¹³13. Hnizdo S, Archuleta RA, Taylor B, Kim SC. Validity and reliability of the modified John Hopkins Fall Risk Assessment Tool for elderly patients in home health care. Geriatr Nurs 2013; 34(5): 423-7. https://doi.org/10.1016/j.gerinurse.2013.05.011
https://doi.org/10.1016/j.gerinurse.2013... , showing acceptable sensitivity and specificity¹²12. Kim KS, Kim JA, Choi YK, Kim YJ, Park MH, Kim HY, et al. A Comparative Study on the Validity of Fall Risk assessment Scales in Korean Hospitals. Asian Nurs Res 2011; 5(1): 28-37. https://doi.org/10.1016/S1976-1317(11)60011-X
https://doi.org/10.1016/S1976-1317(11)60... ^,1313. Hnizdo S, Archuleta RA, Taylor B, Kim SC. Validity and reliability of the modified John Hopkins Fall Risk Assessment Tool for elderly patients in home health care. Geriatr Nurs 2013; 34(5): 423-7. https://doi.org/10.1016/j.gerinurse.2013.05.011
https://doi.org/10.1016/j.gerinurse.2013... . These studies, however, present methodological limitations that make it difficult to generalize the results to other populations.

Considering that previous studies have shown that JH-FRAT has acceptable measurement properties¹⁰10. Poe SS, Cvach MM, Gartrell DG, Radzik BR, Joy TL. An Evidence-based Approach to Fall Risk Assessment, Prevention and Management: Lessons Learned. J Nurs Care Qual 2005; 20(2): 107-16.^,1111. Poe SS, Cvach MM, Dawson PB, Straus H, Hill EE. The Johns Hopkins Fall Risk assessment Tool: Postimplementation Evaluation. J Nurs Care Qual 2007; 22(4): 293-8. https://doi.org/10.1097/01.NCQ.0000290408.74027.39
https://doi.org/10.1097/01.NCQ.000029040... ^,1212. Kim KS, Kim JA, Choi YK, Kim YJ, Park MH, Kim HY, et al. A Comparative Study on the Validity of Fall Risk assessment Scales in Korean Hospitals. Asian Nurs Res 2011; 5(1): 28-37. https://doi.org/10.1016/S1976-1317(11)60011-X
https://doi.org/10.1016/S1976-1317(11)60... ^,1313. Hnizdo S, Archuleta RA, Taylor B, Kim SC. Validity and reliability of the modified John Hopkins Fall Risk Assessment Tool for elderly patients in home health care. Geriatr Nurs 2013; 34(5): 423-7. https://doi.org/10.1016/j.gerinurse.2013.05.011
https://doi.org/10.1016/j.gerinurse.2013... and that there are no assessments of risk scales for falls in Brazil, this study aimed to analyze the measurement properties of JH-FRAT for assessing the risk of falls in hospitalized adult patients.

METHOD

The study was approved by the Research Ethics Committee of Hospital Samaritano and followed the principles of the Declaration of Helsinki. Due to the use of secondary data, the Informed Consent was not necessary. Authorization was obtained from the authors of JH-FRAT and of the Brazilian version of the Morse scale for use in this study.

The target population was composed of adult patients hospitalized in a high complexity private hospital in the city of São Paulo, which uses a protocol for the management of falls since 2005, with the implementation of JH-FRAT at the end of 2013. The protocol provides for the daily assessment of individual risks through JH-FRAT, with the adoption of preventive measures according to each patient’s risks and of universal precautionary measures, such as maintaining night lights activated, signaling risks and freeing the environment of obstacles. The evaluation was performed by nurses and filed in an electronic medical record. This study used data from 2014, after a period of training and routine consolidation by the nursing team.

This is a case-control study including patients from the adult hospitalization areas (clinical, surgical, oncology, gynecology and obstetrics and intensive care), with a wide variety of clinical conditions. We selected 100.0% of patients with reported falls in 2014 (99 cases) and 198 patients without reported falls (two controls for each case), resulting in a sample of 297 individuals. The controls selected were patients admitted to the room next to a patient with a reported fall event, in the same hospitalization sector and on the same date of the fall, in order to homogenize the characteristics of the care received. No other pairing criteria were used in order to preserve differences regarding other potential risk factors that are part of JH-FRAT’s operational structure. Some patients had more than one reported fall event, with each being assessed independently. For patients with falls, the last evaluation was considered within the 24 hours before the fall. For patients without falls, the same date for patients with falls was used. Individuals aged less than 18 years and/or whose hospitalization lasted for less than 24 hours were excluded.

Five researcher nurses collected the data retroactively from the electronic medical record, using a form composed of identification fields (chart number, attendance number, age, date of evaluation, hospitalization sector), for items from the JH-FRAT scale and Morse scale. Data were also obtained from the fall notification forms, filled out by nursing assistants.

The JH-FRAT scale is composed of eight aspects of fall risk¹¹11. Poe SS, Cvach MM, Dawson PB, Straus H, Hill EE. The Johns Hopkins Fall Risk assessment Tool: Postimplementation Evaluation. J Nurs Care Qual 2007; 22(4): 293-8. https://doi.org/10.1097/01.NCQ.0000290408.74027.39
https://doi.org/10.1097/01.NCQ.000029040... :

The sum of points from items 2 to 8 provides a score of 0 to 35 points, which is classified as low risk (0-5 points), moderate risk (6-13 points) and high risk (≥ 14 points)¹¹11. Poe SS, Cvach MM, Dawson PB, Straus H, Hill EE. The Johns Hopkins Fall Risk assessment Tool: Postimplementation Evaluation. J Nurs Care Qual 2007; 22(4): 293-8. https://doi.org/10.1097/01.NCQ.0000290408.74027.39
https://doi.org/10.1097/01.NCQ.000029040... . Item 1 of the scale is not included in the computation, only in the risk classification. Further details on the scale are described in the literature⁵5. Martinez MC, Iwamoto VE, Latorre MRDO, Noronha AM, Oliveira APS, Cardoso CEA, et al. Transcultural adaptation of the Johns Hopkins Fall Risk Assessment Tool. Rev Latino-Am Enfermagem 2016; 24: e2783. http://dx.doi.org/10.1590/1518-8345.1158.2783
http://dx.doi.org/10.1590/1518-8345.1158... ^,66. Johns Hopkins Hospital. Fall risk assessment, prevention and management, adult. In: Johns Hopkins Hospital, editor. Nursing practice and organization manual: clinical protocols and procedures. Baltimore: Johns Hopkins Hospital; 2011. v. 2.^,1111. Poe SS, Cvach MM, Dawson PB, Straus H, Hill EE. The Johns Hopkins Fall Risk assessment Tool: Postimplementation Evaluation. J Nurs Care Qual 2007; 22(4): 293-8. https://doi.org/10.1097/01.NCQ.0000290408.74027.39
https://doi.org/10.1097/01.NCQ.000029040... .

The Morse scale¹⁴14. Morse JM. Preventing patient falls. California: Thousand Oaks; 1997. showed satisfactory accuracy in the identification of patients at high risk of falls in international studies¹⁵15. Baek S, Piao J, Jin Y, Lee SM. Validity of the Morse Fall Scale implemented in an electronic medical record system. J Clin Nurs 2014; 23(17-18): 2434-40. https://doi.org/10.1111/jocn.12359
https://doi.org/10.1111/jocn.12359... ^,1616. Costa-Dias MJM, Martins T, Araújo F. Study of the cut-off point of the Morse Fall Scale (MFS). Rev Enf Ref 2014; serIV(1): 65-74. http://dx.doi.org/10.12707/RIII13101
http://dx.doi.org/10.12707/RIII13101... . It is composed of six aspects (history of falls, secondary diagnoses, walking aid, intravenous therapy/devices, gait, and mental state), with scores varying from 0 to 30, providing a score ranging from 0 to 125 points categorized as low risk (0-24 points), medium risk (25-44 points) and high risk (≥ 45 points)⁹9. Urbanetto JS, Creutzberg M, Franz F, Ojeda BS, Gustavo AS, Bittencourt HR, et al. Morse fall scale: translation and transcultural adaptation for the Portuguese language. Rev Esc Enferm USP 2013; 47(3): 569-75. http://dx.doi.org/10.1590/S0080-623420130000300007
http://dx.doi.org/10.1590/S0080-62342013... .

In this study, the validity was evaluated through the analysis of accuracy, criterion (concurrent analysis) and discriminant, and reliability was evaluated through the analysis of reproducibility.

Scales for the assessment of fall risk are considered diagnostic tests for the event risk (fall)¹⁷17. Aranda-Gallardo M, Morales-Asencio JM, Canca-Sanchez JC, Barrero-Sojo S, Perez-Jimenez C, Morales-Fernandez A, et al. Instruments for assessing the risk of falls in acute hospitalized patients: a systematic review and meta-analysis. BMC Health Serv Res 2013; 13: 122. https://dx.doi.org/10.1186%2F1472-6963-13-122
https://dx.doi.org/10.1186%2F1472-6963-1... . The validity of the diagnostic tests is analyzed by means of accuracy studies, in which the results of a test applied to a population are compared to those obtained by a gold standard¹⁷17. Aranda-Gallardo M, Morales-Asencio JM, Canca-Sanchez JC, Barrero-Sojo S, Perez-Jimenez C, Morales-Fernandez A, et al. Instruments for assessing the risk of falls in acute hospitalized patients: a systematic review and meta-analysis. BMC Health Serv Res 2013; 13: 122. https://dx.doi.org/10.1186%2F1472-6963-13-122
https://dx.doi.org/10.1186%2F1472-6963-1... ^,1818. Glaros AG, Kline RB. Understanding the accuracy of tests with cutting scores: the sensitivity, specificity, and predictive value model. J Clin Psychol 1988; 44(6): 1013-23.^,1919. Parikh R, Mathai A, Parikh S, Sekhar GC, Thomas R. Understanding and using sensitivity, specificity and predictive values. Indian J Ophthalmol 2008; 56(1): 45-50.. In this case, the test is the JH-FRAT and the gold standard is the occurrence of fall. Accuracy analysis verified sensitivity, specificity, positive (PPV) and negative (NPV) predictive values and positive (PLR) and negative (NLR) likelihood ratios Sensitivity evaluates the probability of a test being positive when the disease is present and consists of calculating the prevalence of exposure among events; specificity evaluates the probability of a test being negative when there is no disease and consists in calculating the prevalence of non-exposed among non-events; the PPV evaluates the probability of having the disease when the test was positive and consists of calculating the prevalence of events among those exposed; and the NPV evaluates the probability of not having the disease when the test was negative and consists of calculating the prevalence of non-events among non-exposed individuals. PLR informs the probability of a positive result occurring in a person with the disease in relation to the probability of occurrence in a person without the disease. NLR informs the likelihood of a negative result occurring in a person with the disease in relation to the probability of occurrence in a person without the disease³3. Oliver D, Healy F. Falls risk prediction tools for hospital inpatients: do they work? Nurs Times 2009; 105(7): 18-21.^,2020. Streiner DL. Diagnosing Tests: Using and Misusing Diagnostic and Screening Tests. J Personal Assess 2003; 81(3): 209-9. https://doi.org/10.1207/S15327752JPA8103_03
https://doi.org/10.1207/S15327752JPA8103... ^,2121. Brasil. Ministério da Saúde. Secretaria de Atenção à Saúde. Departamento de Atenção Básica. Parte I - Introdução a conceitos teóricos. In: Brasil. Ministério da Saúde. Secretaria de Atenção à Saúde. Departamento de Atenção Básica, editor. Série A. Normas e Manuais Técnicos: Cadernos de Atenção Primária: Rastreamento. Brasília: Ministério da Saúde; 2010. n. 29. p. 17-23.. The fall risk was considered as exposure (JH-FRAT scale), and the analyzes considered two scenarios:

The criterion validity compares the analyzed scale to a valid standard that predicts the same performance²²22. Pasquali L. Validade dos testes psicológicos: será possível reencontrar o caminho? Psic Teor e Pesq 2007; 23(n. esp.):99-107. http://dx.doi.org/10.1590/S0102-37722007000500019
http://dx.doi.org/10.1590/S0102-37722007... ^,2323. Mokkink LB, Terwee CB, Patrick DL, Alonso J, Stratford PW, Knol DL, et al. COSMIN checklist manual. COSMIN. Amsterdam: University Medical Center, Department of Epidemiology and Biostatistics, Institute for Health and Care Research; 2012., being concurrent when the instrument and criterion are applied simultaneously²²22. Pasquali L. Validade dos testes psicológicos: será possível reencontrar o caminho? Psic Teor e Pesq 2007; 23(n. esp.):99-107. http://dx.doi.org/10.1590/S0102-37722007000500019
http://dx.doi.org/10.1590/S0102-37722007... . The concurrent criterion validity investigated the associations between JH-FRAT and Morse risk classification using the χ² test.

The discriminant analysis verifies whether a test is capable of identifying differences in a characteristic of subjects²²22. Pasquali L. Validade dos testes psicológicos: será possível reencontrar o caminho? Psic Teor e Pesq 2007; 23(n. esp.):99-107. http://dx.doi.org/10.1590/S0102-37722007000500019
http://dx.doi.org/10.1590/S0102-37722007... . It was verified whether each of the eight items of the scale was able to discriminate the presence of the specific risk among patients with and without falls using the χ² test.

Also in the discriminant analysis, it was verified whether previous risk-defining situations (item 1 of the scale) were associated with the severity of the injuries between the fall events, through the χ² test. The severity of the injuries was classified, from an adaptation of Schwendimann et al.¹1. Schwendimann R, Bühler H, De Geest S, Milisen K. Falls and consequent injuries in hospitalized patients: effects of an interdisciplinary falls prevention program. BMC Health Serv Res 2006; 6: 69. https://dx.doi.org/10.1186%2F1472-6963-6-69
https://dx.doi.org/10.1186%2F1472-6963-6... , in:

Reproducibility analysis was performed using the interobserver test-retest comparison, with a random sub-sample of 60 patients. Two researchers independently assessed the same patient using the JH-FRAT scale, based on the same reference date, from data recorded in an electronic medical record. Quadratic weighted Kappa (PABAK) and prevalence-adjusted and bias-adjusted Kappa statistics were used. For the interpretation of the results, the criteria of Landis & Koch were used²⁴24. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics 1977; 33(1): 159-74..

A significance level of 5% was considered in all analyzes. The software SPSS (version 23), STATA (version 14) and WinPepi (version 11.65) were used.

RESULTS

Table 1 shows that the study population consisted of 297 patients, 50.8% females and 49.2% males; 55.9% aged 60 years or older; 63.0% married/had a partner; 46.1% had complete superior education; 14.1% in intensive care units; and 85.9% in open units. Patients with and without falls did not present differences as to gender (p = 0.870), age range (p = 0.200), marital status (0,159), educational level (0,189) and reason for hospitalization (p = 0.677). It was observed that 10.8% of the patients had low risk of falls, being 3.0% among those with falls and 14.6% among those without falls. Patients with high risk of falls had a higher percentage of occurrence of events (62.6%) than the others (44.9%) (p = 0.002).

The accuracy analysis of scenario 1 showed sensitivity = 97.0%; specificity = 14.6%; PPV = 36.2%; NPV = 90.6%; PLR = 1.14; and NLR = 4.83. Scenario 2 showed sensitivity = 62.6%; specificity = 55.1%; PPV = 41.1%; NPV = 74.7%; PLR = 1.39; and NLR = 1.47.

Table 2 shows that, in the concurrent criterion validity, there was a statistically significant association between the two scales (p < 0.001). The highest proportion of patients classified as low risk by JH-FRAT was classified as low risk by Morse (46.9%); the highest proportion of patients classified as moderate risk by JH-FRAT was classified as moderate risk by Morse (53.5%); and the highest proportion of patients classified as high risk by JH-FRAT was classified as high risk by Morse (57.6%). When moderate and high risk are grouped, the similarity in results between the two scales rises to 90.2%.

The discriminant analysis showed which items of the scale had a statistically significant difference in the presence of the specific risk (Table 3). In the item “history of falls”, 18.2% of the patients with events had a history of one fall in the six months prior to hospitalization against 2.5% of those without a fall (p < 0.001). In the item “use of drugs with high risk for falls”, the category “2 or more drugs” was observed in 70.7% of patients with events and in 53.5% of those without events, and sedation in the last 24 hours occurred in 3.0% of patients with falls and in 9.1% of those without falls (p = 0.027). The results of the category “sedation in the last 24 hours” are different than expected, but the small number of cases of falls (3 patients) does not have statistical stability. The item “equipment impairing mobility” showed that ≥ 3 devices were present in 5.1% of patients with events and in 11.1% of those without events (p = 0.040). Again, the results may suffer bias from the small number of subjects with falls (five patients). In the item “impaired mobility”, patients with events had higher percentages than those without events in all the categories evaluated (p = 0.003). The same occurred in the item “cognition” (p < 0.001).

Also in the discriminant validity analysis, the previous situations that defined the risk (item 1 of the scale) were not associated with the injury(ies) due to the fall (p = 0.557).

Table 4 shows the results of the reliability analysis through interobserver reproducibility. Quadratic weighted Kappa values ranged from 0.15 to 1.00. There was improvement after adjustment for prevalence and bias, with PABAK ranging from 0.25 to 1.00. The item “prior risk situations” presented reasonable reproducibility (PABAK = 0.24). For the other items, substantial or almost perfect reproducibility (PABAK of 0.60 to 1.00) was observed, and the overall risk classification presented substantial reproducibility (PABAK = 0.71).

DISCUSSION

The study analyzed whether the JH-FRAT scale is valid and reliable for assessing the risk of falls in adult inpatients. The results suggest that JH-FRAT has a better sensitivity at the score’s cutoff point > 6, grouping moderate and high-risk patients in the same category. An explanation for this is the existence of a managed protocol for falls in the study institution, where patients under certain conditions (risk of bleeding, bone fragility, among others) are classified as high risk according to item 1 of the scale (previous defining situations of risk). A modified version of JH-FRAT evaluating 107 outpatients aged 65 years and older in nursing homes in Southern California showed sensitivity of 100.0% at cutoff point > 6 and 86.2% at cutoff point > 13¹³13. Hnizdo S, Archuleta RA, Taylor B, Kim SC. Validity and reliability of the modified John Hopkins Fall Risk Assessment Tool for elderly patients in home health care. Geriatr Nurs 2013; 34(5): 423-7. https://doi.org/10.1016/j.gerinurse.2013.05.011
https://doi.org/10.1016/j.gerinurse.2013... . A JH-FRAT study with 356 patients older than 18 years in five general school hospitals in South Korea, with cutoff point > 13 points, showed a sensitivity of 62.0%¹²12. Kim KS, Kim JA, Choi YK, Kim YJ, Park MH, Kim HY, et al. A Comparative Study on the Validity of Fall Risk assessment Scales in Korean Hospitals. Asian Nurs Res 2011; 5(1): 28-37. https://doi.org/10.1016/S1976-1317(11)60011-X
https://doi.org/10.1016/S1976-1317(11)60... .

JH-FRAT showed better specificity at the cutoff point > 13, separating high-risk patients from the others. Sensitivity and specificity are inversely proportional¹⁹19. Parikh R, Mathai A, Parikh S, Sekhar GC, Thomas R. Understanding and using sensitivity, specificity and predictive values. Indian J Ophthalmol 2008; 56(1): 45-50.^,2121. Brasil. Ministério da Saúde. Secretaria de Atenção à Saúde. Departamento de Atenção Básica. Parte I - Introdução a conceitos teóricos. In: Brasil. Ministério da Saúde. Secretaria de Atenção à Saúde. Departamento de Atenção Básica, editor. Série A. Normas e Manuais Técnicos: Cadernos de Atenção Primária: Rastreamento. Brasília: Ministério da Saúde; 2010. n. 29. p. 17-23. and sensitivity was high. Considering that the systematized actions prevented the occurrence of falls, there was an increase in the number of false positives (patients at risk, but without falls), with consequent decrease in specificity. The diagnosis of a condition and the exclusion of false positives are defined by confirmatory tests that must have high specificity, but are subsequent to screening exams²¹21. Brasil. Ministério da Saúde. Secretaria de Atenção à Saúde. Departamento de Atenção Básica. Parte I - Introdução a conceitos teóricos. In: Brasil. Ministério da Saúde. Secretaria de Atenção à Saúde. Departamento de Atenção Básica, editor. Série A. Normas e Manuais Técnicos: Cadernos de Atenção Primária: Rastreamento. Brasília: Ministério da Saúde; 2010. n. 29. p. 17-23.. The study in southern California showed specificity of 0.0% at the cutoff point > 6 and 49.3% at the cutoff point > 13 points¹³13. Hnizdo S, Archuleta RA, Taylor B, Kim SC. Validity and reliability of the modified John Hopkins Fall Risk Assessment Tool for elderly patients in home health care. Geriatr Nurs 2013; 34(5): 423-7. https://doi.org/10.1016/j.gerinurse.2013.05.011
https://doi.org/10.1016/j.gerinurse.2013... . The study in South Korea, with cutoff point > 13 points, showed specificity of 69.5%¹²12. Kim KS, Kim JA, Choi YK, Kim YJ, Park MH, Kim HY, et al. A Comparative Study on the Validity of Fall Risk assessment Scales in Korean Hospitals. Asian Nurs Res 2011; 5(1): 28-37. https://doi.org/10.1016/S1976-1317(11)60011-X
https://doi.org/10.1016/S1976-1317(11)60... .

The PPVs show that 63.8% (in the cutoff point > 6) and 58.9% (in cutoff point > 13) of patients with a positive risk test received unnecessary preventive care for falls. This interpretation must be cautious. The increased prevalence of the event determines the increase in the PPV and the decrease in the VPN, and both depend on the context in which the test is applied¹⁶16. Costa-Dias MJM, Martins T, Araújo F. Study of the cut-off point of the Morse Fall Scale (MFS). Rev Enf Ref 2014; serIV(1): 65-74. http://dx.doi.org/10.12707/RIII13101
http://dx.doi.org/10.12707/RIII13101... ^,1818. Glaros AG, Kline RB. Understanding the accuracy of tests with cutting scores: the sensitivity, specificity, and predictive value model. J Clin Psychol 1988; 44(6): 1013-23.^,2020. Streiner DL. Diagnosing Tests: Using and Misusing Diagnostic and Screening Tests. J Personal Assess 2003; 81(3): 209-9. https://doi.org/10.1207/S15327752JPA8103_03
https://doi.org/10.1207/S15327752JPA8103... . Considering that the effect of the “treated” population reduces the occurrence of falls among those at risk, bias can occur with decreasing PPV and increasing NPV. The study in southern California showed PPV of 39.7% and NPV of 92.9% at cutoff point > 13 points¹³13. Hnizdo S, Archuleta RA, Taylor B, Kim SC. Validity and reliability of the modified John Hopkins Fall Risk Assessment Tool for elderly patients in home health care. Geriatr Nurs 2013; 34(5): 423-7. https://doi.org/10.1016/j.gerinurse.2013.05.011
https://doi.org/10.1016/j.gerinurse.2013... . The study in South Korea, with cutoff point > 13 points, showed PPV of 33.6% and NPV of 86.0%¹²12. Kim KS, Kim JA, Choi YK, Kim YJ, Park MH, Kim HY, et al. A Comparative Study on the Validity of Fall Risk assessment Scales in Korean Hospitals. Asian Nurs Res 2011; 5(1): 28-37. https://doi.org/10.1016/S1976-1317(11)60011-X
https://doi.org/10.1016/S1976-1317(11)60... .

LR offers an odds ratio measure, combining sensitivity and specificity in the same indicator²⁰20. Streiner DL. Diagnosing Tests: Using and Misusing Diagnostic and Screening Tests. J Personal Assess 2003; 81(3): 209-9. https://doi.org/10.1207/S15327752JPA8103_03
https://doi.org/10.1207/S15327752JPA8103... . The results were greater than 1.00 and thus contribute to the identification of the risk²⁰20. Streiner DL. Diagnosing Tests: Using and Misusing Diagnostic and Screening Tests. J Personal Assess 2003; 81(3): 209-9. https://doi.org/10.1207/S15327752JPA8103_03
https://doi.org/10.1207/S15327752JPA8103... , especially the NLR of scenario 1, in which patients without falls had 4.83 times the chance of obtaining a low risk classification when compared to the others.

For a screening test, the most important feature is sensitivity, with a high rate of true-positive and a low false-negative rate²⁰20. Streiner DL. Diagnosing Tests: Using and Misusing Diagnostic and Screening Tests. J Personal Assess 2003; 81(3): 209-9. https://doi.org/10.1207/S15327752JPA8103_03
https://doi.org/10.1207/S15327752JPA8103... ^,2121. Brasil. Ministério da Saúde. Secretaria de Atenção à Saúde. Departamento de Atenção Básica. Parte I - Introdução a conceitos teóricos. In: Brasil. Ministério da Saúde. Secretaria de Atenção à Saúde. Departamento de Atenção Básica, editor. Série A. Normas e Manuais Técnicos: Cadernos de Atenção Primária: Rastreamento. Brasília: Ministério da Saúde; 2010. n. 29. p. 17-23.. JH-FRAT, which an instrument for risk screening, showed high sensitivity.

The analysis of the validity of concurrent criteria showed equivalences between the risk classification of the two scales (JH-FRAT and Morse), with better performance when the moderate and high risks are grouped, suggesting that JH-FRAT has greater validity when separating the low-risk patients from the others.

Five of the eight JH-FRAT items and the overall risk score discriminated risk, with higher prevalence among patients with falls than among those without falls, as expected.

The previous situations that defined the risk (item 1 of the scale) were not able to discriminate the type of injury resulting from the fall. These results may be biased due to the institutional protocol, reducing both the occurrence and severity of falls.

The prevalence of patients classified as moderate/high risk, by both JH-FRAT and Morse, was 90.2%. A paradox in the Kappa reproducibility analyzes can be induced by a high prevalence of the analyzed condition and can be adjusted by the PABAK statistic²⁵25. Sim J, Wright CC. The Kappa statistic in reliability studies: use, interpretation, and sample size requirements. Phys Ther 2005; 85(3): 257-68.. Reproducibility between tests showed substantial agreement after adjustment for this statistic.

The interobserver test-retest measures showed substantial or near-perfect agreement after the use of the PABAK statistic on all items, except in the “prior risk situations”. The results obtained by quadratic weighted Kappa showed improvement after correction by the PABAK statistic, pointing to the already mentioned paradox. The study in southern California also showed substantial or near-perfect agreement in most items and in the overall classification (cutoff point ≥ 14)¹³13. Hnizdo S, Archuleta RA, Taylor B, Kim SC. Validity and reliability of the modified John Hopkins Fall Risk Assessment Tool for elderly patients in home health care. Geriatr Nurs 2013; 34(5): 423-7. https://doi.org/10.1016/j.gerinurse.2013.05.011
https://doi.org/10.1016/j.gerinurse.2013... .

This study has some limitations. The most important is the fact that it included a “treated” population, since preventive interventions reduce the occurrence and severity of events, and may alter the outcomes and accuracy of a risk assessment scale²⁶26. Oliver D, Daly F, Martin FC, McMurdo MET. Risck factors and risk assessment tools for falls in hospital in-patients: a systematic review. Age Ageing 2004; 33(2): 122-30. https://doi.org/10.1093/ageing/afh017
https://doi.org/10.1093/ageing/afh017... ^,2727. Milisen K, Staelens N, Schwendimann R, De Paepe L, Verhaeghe J, Braes T, et al. Fall Prediction in Inpatients by Bedside Nurses Using the St. Thomas's Risk Assessment Tool in Falling Elderly Inpatients (STRATIFY) Instrument: A Multicenter Study. J Am Geriatr Soc 2007; 55(5): 725-33. https://doi.org/10.1111/j.1532-5415.2007.01151.x
https://doi.org/10.1111/j.1532-5415.2007... . Another limitation refers to the use of data collected retrospectively, a type of collection that is affected by the quality of the available data²⁸28. Dawson-Saunder B, Trapp RG. Reading the medical literature. In: Dawson-Saunder B, Trapp RG, editores. Basic & Clinical Biostatistics. Connecticut: Appleton & Lange; 1994. p. 268-88.. Lastly, the relatively small number of falls assessed may have generated some bias.

Falls are events that are sensitive to care practices and can be prevented through systematized approaches in institutional programs²2. National Health Services. National Patient Safety Agency. Slips, trips and falls in hospital: The third report from the Patient Safety Observatory. London: National Health Services; 2007.^,33. Oliver D, Healy F. Falls risk prediction tools for hospital inpatients: do they work? Nurs Times 2009; 105(7): 18-21.^,44. Cumbler EU, Simpson JR, Rosenthal LD, Likosky DJ. Inpatient falls: defining the problem and identifying possible solutions. Part I: An evidence-based review. Neurohospitalist 2013; 3(3): 135-43. https://doi.org/10.1177/1941874412470665
https://doi.org/10.1177/1941874412470665... ^,55. Martinez MC, Iwamoto VE, Latorre MRDO, Noronha AM, Oliveira APS, Cardoso CEA, et al. Transcultural adaptation of the Johns Hopkins Fall Risk Assessment Tool. Rev Latino-Am Enfermagem 2016; 24: e2783. http://dx.doi.org/10.1590/1518-8345.1158.2783
http://dx.doi.org/10.1590/1518-8345.1158... ^,1515. Baek S, Piao J, Jin Y, Lee SM. Validity of the Morse Fall Scale implemented in an electronic medical record system. J Clin Nurs 2014; 23(17-18): 2434-40. https://doi.org/10.1111/jocn.12359
https://doi.org/10.1111/jocn.12359... ^,2626. Oliver D, Daly F, Martin FC, McMurdo MET. Risck factors and risk assessment tools for falls in hospital in-patients: a systematic review. Age Ageing 2004; 33(2): 122-30. https://doi.org/10.1093/ageing/afh017
https://doi.org/10.1093/ageing/afh017... . The best results are obtained by combining multiple interventions, including individualized risk assessment and universal precautions⁴4. Cumbler EU, Simpson JR, Rosenthal LD, Likosky DJ. Inpatient falls: defining the problem and identifying possible solutions. Part I: An evidence-based review. Neurohospitalist 2013; 3(3): 135-43. https://doi.org/10.1177/1941874412470665
https://doi.org/10.1177/1941874412470665... ^.88. Agency for Healthcare Research and Quality. Preventing falls in hospitals: a toolkit for improving quality of care. Rockvilee: Agency for Healthcare Research and Quality; 2013.. The use of multifaceted interventions can reduce the occurrence of falls by up to 18.0%, but this reduction may vary depending on the quantity and type of actions²2. National Health Services. National Patient Safety Agency. Slips, trips and falls in hospital: The third report from the Patient Safety Observatory. London: National Health Services; 2007.. The results showed JH-FRAT as valid and reliable in terms of accuracy, discriminant capacity and agreement on reproducibility, confirming it as a tool for risk assessment of falls and bringing contributions to professionals and institutions interested in the use of evidence-based strategies for the management of falls in hospitals.

There are few studies on the validity and reliability of JH-FRAT, and the present study is the first in Brazil. It is advisable that new studies be carried out with larger groups of hospitalized patients, preferably with a prospective design. In the absence of ethical conflicts, it is desirable to carry out studies on “untreated” populations and to compare results before and after the implementation of preventive protocols.

CONCLUSIONS

The study suggests that the Brazilian version of the JH-FRAT scale has satisfactory validity and reliability in terms of accuracy, criterion, discriminant capacity and reproducibility. It can be used in the evaluation of the fall risk in hospitalized patients, directing preventive actions. This is the first study in Brazil to evaluate JH-FRAT and has limitations, and new studies are desirable.

ACKNOWLEDGEMENTS

The authors would like to thank Hospital Samaritano de São Paulo for their support in the development of the study through the provision of equipment, materials and human resources, without which research would not be possible.

References

Publication Dates

History