ABSTRACT

Introduction:

A smartphone application named AtestaDO was developed to support physicians with medical certification of the cause of death. The objective of this study is to evaluate the acceptability of the app.

Methods:

Physicians were invited to attend meetings on the proper certification of cause of death, and to evaluate the application in a national workshop in Natal (first stage) and in two large hospitals in Belo Horizonte (second and third stages).

Results:

In Natal, 82% of 38 physicians had more than 20 years of experience and in Belo Horizonte, more than 67% of 58 physicians had less than 5 years of experience. The sections “Application interface”, “How to certify the causes of death”, “Practice with exercises” and “Other information for physicians” were positively evaluated by more than 50% of physicians in Belo Horizonte. In Natal, all sections were positively evaluated by at least 80% of participants. More than 70% of the participants in both Natal and the second stage of Belo Horizonte indicated they would possibly use AtestaDO to guide filling of a death certificate. The probability of using AtestaDO to teach classes on filling death certificates was 83.3% for Natal’s physicians but less than 60% in Belo Horizonte. In the three stages, most physicians would recommend using the application to other colleagues.

Conclusion:

The evaluation of AtestaDO showed good acceptability. We expect that the use of this tool enables improvements in medical certification of causes of death.

Keywords:
Mobile applications; Cause of death; Death Certificate; Mortality; Evaluation study; Smartphone

INTRODUCTION

Mortality data are essential tools for health planning and management. In Brazil, the Sistema de Informação sobre Mortalidade (Mortality Information System - SIM) is universal and well-consolidated, with the certification by physicians of the causes of death in death certificates (DC). However, the quality of this information is not uniform among Brazilian states, with a high proportion of non-informative causes classified as garbage causes (GC) ⁽¹1. Naghavi M, Makela S, Foreman K, O'Brien J, Pourmalek F, Lozano R. Algorithms for enhancing public health utility of national causes-of-death data. Popul Health Metr. 2010;8:9. https://doi.org/10.1186/1478-7954-8-9
https://doi.org/10.1186/1478-7954-8-9... recorded on SIM as the underlying cause of death.

Garbage causes may derive from the improper filling of the DC by physicians, including the non-distinction between the underlying cause of death and the terminal event²2. Degani AT, Patel RM, Smith BE, Grimsley E. The effect of student training on accuracy of completion of death certificates. Med Educ Online. 2009;14:17. https://doi.org/10.3885/meo.2009.Res00315
https://doi.org/10.3885/meo.2009.Res0031... . The lack of training for filling DC has been pointed as one of the factors related to such low precision³3. Rampatige R, Mikkelsen L, Hernandez B, Riley I, Lopez AD. Hospital cause-of-death statistics: what should we make of them? Bull World Health Organ. 2014;92(1):3-3A. http://dx.doi.org/10.2471/BLT.13.134106
http://dx.doi.org/10.2471/BLT.13.134106... ^{), (4}4. Silva PHA, Lima ASD, Medeiros ACM, Bento BM, Silva RJS, Freire FD, et al. Avaliação do conhecimento de médicos professores, residentes e estudantes de medicina acerca da Declaração de Óbito. Rev Bras Educ Méd. 2016;40(2):183-88. http://dx.doi.org/10.1590/1981-52712015v40n2e01532014
http://dx.doi.org/10.1590/1981-52712015v... , which can be improved with educational initiatives⁵5. Myers KA, Farquhar DRE. Improving the accuracy of death Certification. CMAJ. 1998;158(10):1317-23.^{), (6}6. Lakkireddy DR, Basarakodu KR, Vacek JL, Kondur AK, Ramachandruni SK, Esterbrooks DJ, et al. Improving death certificate completion: a trial of two training interventions. J Gen Intern Med. 2007;22(4):544-8. https://doi.org/10.1007/s11606-006-0071-6
https://doi.org/10.1007/s11606-006-0071-... . Several initiatives such as training courses and workshops for physicians in hospitals and the distribution of instruction manuals have been developed in Brazil⁷7. Brasil. Ministério da Saúde. Conselho Federal de Medicina. Centro Brasileiro de Classificação de Doenças. A declaração de óbito: documento necessário e importante. Brasília, DF; 2009.^{), (8}8. Brasil. Ministério da Saúde. Secretaria de Vigilância em Saúde. Departamento de Análise de Situação de Saúde. Manual de instruções para o preenchimento da Declaração de Óbito. Brasília, DF; 2011., seeking to raise awareness among these professionals about the importance of the proper filling of the DC for public health⁴4. Silva PHA, Lima ASD, Medeiros ACM, Bento BM, Silva RJS, Freire FD, et al. Avaliação do conhecimento de médicos professores, residentes e estudantes de medicina acerca da Declaração de Óbito. Rev Bras Educ Méd. 2016;40(2):183-88. http://dx.doi.org/10.1590/1981-52712015v40n2e01532014
http://dx.doi.org/10.1590/1981-52712015v... ^{), (9}9. Lucena L, Cagliari GHB, Tanaka J, Bonamigo EL. Declaração de óbito: preenchimento pelo corpo clínico de um hospital universitário. Rev bioét (Impr.). 2014;22(2):318-24. http://dx.doi.org/10.1590/1983-80422014222013
http://dx.doi.org/10.1590/1983-804220142... ^{), (10}10. Mendonça FM, Drumond E, Cardoso AMP. Problemas no preenchimento da Declaração de Óbito: estudo exploratório. Rev Bras Estud Popul. 2010;27(2):285-95. http://dx.doi.org/10.1590/S0102-30982010000200004
http://dx.doi.org/10.1590/S0102-30982010... . Despite these interventions, one third of the deaths remains with GC as the underlying cause of death¹¹11. França EB, Passos VMA, Malta DC, Duncan BB, Ribeiro ALP, Guimarães MDC, et al. Cause-specific mortality for 249 causes in Brazil and states during 1990-2015: a systematic analysis for the global burden of disease study 2015. Popul Health Metr. 2017;15:39. https://doi.org/10.1186/s12963-017-0156-y
https://doi.org/10.1186/s12963-017-0156-... .

In-person training programs for all physicians represent a major challenge in Brazil due to the number of professionals and the country’s territorial extension. In 2016, a team of experts from the Brazilian Ministry of Health (MS) analyzed cost-effective alternatives to train physicians in cause of death certification. At that time, a proposal for a smartphone application was discussed - the AtestaDO application - to be developed in partnership with the School of Medicine of Universidade Federal de Minas Gerais (UFMG), with support from Vital Strategies and the University of Melbourne, by the Bloomberg Foundation. This proposal is part of a broad intervention initiative, the Data for Health project, which aimed to improve statistics on causes of death. The initiative has proved to be of great relevance by raising awareness physicians, encoders and health managers about the need to adopt routines for the correct filling of DC¹²12. Soares Filho AM, Nóbrega AA, Lobo AP, Silveira AC, Zoca BO, Cunha CC, et al. Melhoria da qualidade de informação em saúde na percepção de técnicos que colaboram na construção coletiva do livro Saúde Brasil. In: Brasil. Ministério da Saúde. Secretaria de Vigilância em Saúde. Departamento de Vigilância de Doenças e Agravos não Transmissíveis e Promoção da Saúde. Saúde Brasil 2017: uma análise da situação de saúde e os desafios para o alcance dos objetivos de desenvolvimento sustentável. Brasília, DF; 2018. p. 387-415..

The proposal of AtestaDO is to have impact on the quality of the causes of death certified by the physician. This study thus sought to evaluate the acceptability of the application as an instrument to assist physicians in the certification of the cause of death.

METHODS

The application was developed for the mobile environment, executable on Android and iOS platforms, and compatible with tablets and smartphones. The application is available for download on both the Play Store and Apple Store; it can be used on offline mode, ensuring the fast and useful access to its informative content. The software and its continuous technical support are provided by the MS.

A preliminary version of AtestaDO was developed from the last manual of the MS and Conselho Federal de Medicina (Brazilian Federal Medicine Council - CFM) on the filling of DC⁷. Teams of physicians from the MS and UFMG performed successive evaluative surveys followed by changes in the content and format of the application, which resulted in a final version Beta 0.3.0, presented at a national meeting about the project in October 2017 and made available for download online. The conduction of the evaluative studies with subsequent elaboration of the various versions of the application followed an action research proposal, a method that assumes that researchers and representative participants of the situation or problem are involved in a cooperative and collaborative way to create intersectoral spaces to broaden the knowledge about the discussed object¹³13. Thiollent M. Metodologia da pesquisa-ação. 16a ed. São Paulo, Cortez. 2008..

The application presents an interactive starting menu structured into six major sections: 1) How to certify the causes of death; 2) Important concepts; 3) Practice with exercises; 4) Other information for physician; 5) Questions and answers; and 6) Legal, normative and bibliographical references. Section 3 presents a tutorial for the correct filling of DC and clinical cases of different medical specialties, including an option to fill causes of death and a system of user response validation.

The evaluation of application acceptability among hospital physicians was performed by UFMG in three stages, in 2016 and 2017. After each stage, the team responsible for the evaluative study suggested adaptations to the MS team responsible for developing the application (Figure 1).

The first stage - identified as Natal - occurred in November 2016 in a national meeting organized by the MS for health services professionals and researchers, focusing on proposals to improve the quality of information on causes of death. All physicians present were invited to participate in a specific meeting coordinated by UFMG, which aimed to evaluate the application’s first version (beta 0.0.7).

In the following stages, physicians from selected hospitals were invited to a training meeting in cause of death certification using AtestaDO. In the second stage, identified as BH_1, the evaluation was performed in two large public hospitals in April and May 2017 (versions 0.1.7 and 0.1.8 - Android and 0.1.6 iOS). Starting from this stage, the MS team started to develop the graphic design, thus improving the application interface. In the third stage (BH_2), a final version (Beta 0.3.0) was tested in a large hospital in Belo Horizonte in December 2017.

Initially, all participating physicians were informed about the study and signed the Term of Free and Informed Consent, ensuring the confidentiality of information. Following, they filled out a brief questionnaire on their professional profile and evaluated clinical cases of hospital deaths (10 in Natal and 5 in Belo Horizonte), with a DC section on filling the causes of death. After finishing the analyzed cases, a training session was performed for downloading and using the AtestaDO application¹1Due to internet connection problems there were issues to download preliminary versions. To mitigate such issue the participants of the final evaluation (BH_2) were instructed to download and install the application after the workshop in an environment with stable internet connection.. A new test that consisted in attesting the cause of death of the same previous cases was thus conducted. Due to the limited time for training, physicians participating in stage BH_2 were invited to respond to a new test in up to one week after the meeting.

A brief questionnaire on acceptability was also applied. Participants were asked to classify the relevance of the application sections on a scale of 1 (not good) to 5 (excellent), and to evaluate its functionality, its contribution to the filling of the DC and the probability of use of the application when filling a DC or teaching classes, indicating whether they would recommend AtestaDO to colleagues. There were also questions that evaluated the means of access to the application and the time spent in reading and using it. Two open questions were inserted, allowing participants to suggest improvements in the application. In stage BH_2 it was verified if the participants had already accessed AtestaDO since the application was already available to download for free at the time.

Given the successive improvements of the application versions, the evaluation questionnaires applied to physicians were adapted for each stage, which is why some questions do not present results for all stages.

The study was approved by the Research Ethics Committees of Universidade Federal de Minas Gerais (UFMG), of Fundação Hospitalar do Estado de Minas Gerais (Hospital Foundation of Minas Gerais - Fhemig) and of Hospital Metropolitano Odilon Behrens (Odilon Behrens Metropolitan Hospital - HMOB).

RESULTS

In total, 96 physicians participated in the three evaluation stages. In Natal (Stage 1), of the 46 physicians attending the meeting, 8 were unable to access the application due to internet connection problems to download it. In stage BH_1, of 44 participants, 34 managed to access the application and respond to the evaluation questionnaire. In the last stage (BH_2), 61 physicians attended the meeting, but only 24 followed the instruction to download the application after the workshop and participated in this study (Figure 1).

In Natal, 82% of the physicians had concluded their undergraduate studies over 20 years ago. In the two workshops in Belo Horizonte, over two thirds of the participants had concluded their undergraduate studies less than 5 years ago, and most were resident physicians. Regarding specialties, most participants in Natal were epidemiologists/public health professionals and pediatricians (26.3% each). On the other hand, in the BH_1 workshop, 65% of the participants were from the medical clinic, and in the BH_2 workshop, 38% were community and family medicine professionals. The main place of professional practice reported in the BH_1 and BH_2 workshops was the hospital, whereas 63% of the participants in Natal reported another location (Table 1).

More than half of the participants in all three stages reported having already filled a DC. However, in the BH_1 and BH_2 workshops, 41% and 25% of physicians, respectively, reported never having filled a DC. More than 90% of Natal’s physicians had attended a lecture or class on filling the DC - most in the institution they work at -, and 97% had already accessed and read the manual by the MS and CFM to fill the DC. In Belo Horizonte (BH_1), 56% of the physicians had already attended a lecture or class about filling the DC (50% during their undergraduate studies), but only 38% of the participants reported having accessed the manual. In stage BH_2, 96% of the physicians reported having attended a lecture or class on the filling of DC (75% during their undergraduate studies) and 58% had access to the manual, of which only 50% reported having read it (Table 2).

About two-thirds of the participants accessed the application by downloading it on smartphones. The contents of AtestaDO were read for 5 to 10 minutes (BH_1, 26.5% of the participants) or 15 to 30 minutes (BH_2, 42% of the participants) (Table 3).

Regarding the acceptability of AtestaDO, the sections were better evaluated in the Natal and BH_2 workshops. The “Application Interface” was well evaluated in the three workshops by more than 82% of physicians. In Natal, all sections were well evaluated by at least 80% of the physicians, except for “Contribution to the filling of the causes of death” (63%). The sections “How to certify the causes of death” and “Other information for physician” were well evaluated, being graded 4 or 5 by more than 50% of participants. The section “Practice with exercises” obtained scores of 4 or 5 among 80.6% of Natal participants. However, the proportions for these scores were lower in Belo Horizonte, 48.5% and 63.6% for BH_1 and BH_2, respectively. The “Important concepts” and “Questions and answers” sections - which were added to the application from 2017 onwards - were well evaluated by more than half of the physicians in Belo Horizonte. The section “Legal, normative and bibliographical references” was well evaluated in the Natal (80.8%) and BH_2 (73.7%) workshops; however, this percentage was 34.8% for BH_1 (Table 4).

The application’s contribution to assigning the cause of death was evaluated by more than half of the physicians with a score of 4 or 5. More than 70% of participants in the Natal and BH_2 stages would possibly use the application to fill the cause of death in the DC. However, only 46% of the users in step BH_1 would possibly use the application for this purpose. The probability of using AtestaDO for teaching was 83% in Natal, and 42% and 59% in Belo Horizonte (BH_1 and BH_2, respectively). In the three stages, most physicians would recommend the application to other colleagues (Table 4).

Regarding the open questions, the suggestion for an application with interactive function in which physicians could include data from a real DC appeared more than once. Moreover, the participants suggested the insertion of images from the fields of the DC, review of cases, presentation of links to technical and normative content, to the Seletor de causa básica (Underlying Cause of Death Selector - SCB) and the inclusion of a dictionary of medical terms.

DISCUSSION

The results of this study indicate good acceptability of the AtestaDO application and the success of the action research proposal, in which the evaluators and the professionals responsible for developing the application were involved in a cooperative and collaborative manner, changing the application’s format and content after each evaluative study.

One of AtestaDO’s advantage is its use in smartphones, which are increasingly used for communication, consultation and follow-up of patients, as well as for researching information on the internet and medical education, including applications for medicine¹⁴14. Boruff JT, Storie D. Mobile devices in medicine: a survey of how medical students, residents, and faculty use smartphones and other mobile devices to find information. J Med Lib Assoc. 2014;102(1):22-30. http://dx.doi.org/10.3163/1536-5050.102.1.006
http://dx.doi.org/10.3163/1536-5050.102.... ^{), (15}15. Ozdalga E, Ozdalga A, Ahuja N. The smartphone in medicine: a review of current and potential use among physicians and students. J Med Internet Res. 2012;14(5):e128. http://dx.doi.org/10.2196/jmir.1994
http://dx.doi.org/10.2196/jmir.1994... . Thus, the wide use of applications is expected from physicians¹⁴14. Boruff JT, Storie D. Mobile devices in medicine: a survey of how medical students, residents, and faculty use smartphones and other mobile devices to find information. J Med Lib Assoc. 2014;102(1):22-30. http://dx.doi.org/10.3163/1536-5050.102.1.006
http://dx.doi.org/10.3163/1536-5050.102.... . Payne et al. ⁽¹⁶16. Payne KFB, Wharrad H, Watts K. Smartphone and medical related app use among medical students and junior doctors in the United Kingdom (UK): a regional survey. BMC Med Inform Decis Mak. 2012;12:121. http://dx.doi.org/10.1186/1472-6947-12-121
http://dx.doi.org/10.1186/1472-6947-12-1... found that 75% of junior physicians had applications related to medicine installed on their smartphone. These applications are more commonly used by undergraduate students, resident physicians¹⁴14. Boruff JT, Storie D. Mobile devices in medicine: a survey of how medical students, residents, and faculty use smartphones and other mobile devices to find information. J Med Lib Assoc. 2014;102(1):22-30. http://dx.doi.org/10.3163/1536-5050.102.1.006
http://dx.doi.org/10.3163/1536-5050.102.... or younger physicians¹⁷17. Patel RK, Sayers AE, Patrick NL, Hughes K, Armitage J, Hunter IA. A UK perspective on smartphone use amongst doctors within the surgical profession. Annals of Medicine and Surgery 2015;4:107-112. https://doi.org/10.1016/j.amsu.2015.03.004
https://doi.org/10.1016/j.amsu.2015.03.0... .

Several studies indicate that educational strategies such as workshops, educational interventions, seminars, and online tutorials increased the accuracy of the cause of death or provided an overall improvement in the certification of causes of death²2. Degani AT, Patel RM, Smith BE, Grimsley E. The effect of student training on accuracy of completion of death certificates. Med Educ Online. 2009;14:17. https://doi.org/10.3885/meo.2009.Res00315
https://doi.org/10.3885/meo.2009.Res0031... ^{), (5}5. Myers KA, Farquhar DRE. Improving the accuracy of death Certification. CMAJ. 1998;158(10):1317-23.^{), (6}6. Lakkireddy DR, Basarakodu KR, Vacek JL, Kondur AK, Ramachandruni SK, Esterbrooks DJ, et al. Improving death certificate completion: a trial of two training interventions. J Gen Intern Med. 2007;22(4):544-8. https://doi.org/10.1007/s11606-006-0071-6
https://doi.org/10.1007/s11606-006-0071-... ^{), (18}18. McAllum C, St George I, White G. Death certification and doctors' dilemmas: a qualitative study of GPs' perspectives. Br J Gen Pract. 2005;55:677-83.^{)- (20}20. Abós R, Pérez G, Rovira E, Canela J, Domènech J, Bardina JR. Programa piloto para la mejora de la certificación de las causas de muerte en atención primaria en Cataluña. Gac Sanit. 2006;20(6):450-6.. Despite several previous initiatives undertaken by the MS to improve the filling of DC - such as training courses for physicians and the manual prepared in partnership with CFM⁷7. Brasil. Ministério da Saúde. Conselho Federal de Medicina. Centro Brasileiro de Classificação de Doenças. A declaração de óbito: documento necessário e importante. Brasília, DF; 2009. -, Brazilian studies show that the quality of the DC filling in the country still needs to improve. França et al. ¹¹11. França EB, Passos VMA, Malta DC, Duncan BB, Ribeiro ALP, Guimarães MDC, et al. Cause-specific mortality for 249 causes in Brazil and states during 1990-2015: a systematic analysis for the global burden of disease study 2015. Popul Health Metr. 2017;15:39. https://doi.org/10.1186/s12963-017-0156-y
https://doi.org/10.1186/s12963-017-0156-... ⁾ found that the proportion of garbage codes declared as the cause of death remained high in 2014 - about 33%. In Belém, 71.5% of medical certifications were found to present some type of error, such as incorrect sequential filling of the causes of death. According to Silva et al. ⁽²¹21. Silva JAC, Yamaki VN, Oliveira JPS, Teixeira RKC, Santos FAF, Hosoume VSN. Commitment in the completion of the medical death certification. Evaluation in Belém, Pará, Brazil in 2010. Rev Assoc Med Bras. 2013;59(4):335-40. http://dx.doi.org/10.1016/j.ramb.2013.03.001
http://dx.doi.org/10.1016/j.ramb.2013.03... , this is related to the lack of specific information in undergraduate curricula and in continuing medical education.

The evaluation of the acceptability of AtestaDO was conducted with three distinct groups in this study. Given that it was national meeting promoted by the MS, the Natal workshop was attended by representatives from different regions of Brazil. These physicians had finished their undergraduate studies for longer and were mostly epidemiologists, public health professionals and pediatricians. Those participating in the Belo Horizonte workshops were mostly young and resident physicians, with under 5 years since finishing their undergraduate studies, and clinicians or family physicians.

These different profiles are reflected in the evaluation results. In Natal, there was a higher percentage of physicians who had already attended a class or lecture on the filling of the DC, having previous experience on this and with previous access to and reading of the manual for filling DC (97.4%). At the Belo Horizonte workshops, at least 25% of the participants reported never having filled a DC, and a low percentage of physicians reported having read the manual, coinciding with results of a study conducted in Joaçaba (SC) ⁽⁹9. Lucena L, Cagliari GHB, Tanaka J, Bonamigo EL. Declaração de óbito: preenchimento pelo corpo clínico de um hospital universitário. Rev bioét (Impr.). 2014;22(2):318-24. http://dx.doi.org/10.1590/1983-80422014222013
http://dx.doi.org/10.1590/1983-804220142... . In another study previously conducted in Belo Horizonte, 75% of 18 physicians interviewed did not know the manual, but 60% reported having received instructions on the completion of DC during their undergraduate studies and 27% reported to receive guidance during their medical residency¹⁰10. Mendonça FM, Drumond E, Cardoso AMP. Problemas no preenchimento da Declaração de Óbito: estudo exploratório. Rev Bras Estud Popul. 2010;27(2):285-95. http://dx.doi.org/10.1590/S0102-30982010000200004
http://dx.doi.org/10.1590/S0102-30982010... .

The best assessments occurred in Natal, BH_2 and BH_1, respectively, coinciding with the best connectivity to download the application. This may also be a result from the different profiles of participating physicians in the evaluation stages. In Natal, physicians already had experience with filling DC and were involved in public health activities, thus being aware of the importance of the quality of information about mortality. In Belo Horizonte, physicians were younger and had less experience in filling DC, and probably greater critical capacity in the use of applications given the previous experience with this type of resource. Given that such young audience is the one that mostly uses applications on their smartphones¹⁴14. Boruff JT, Storie D. Mobile devices in medicine: a survey of how medical students, residents, and faculty use smartphones and other mobile devices to find information. J Med Lib Assoc. 2014;102(1):22-30. http://dx.doi.org/10.3163/1536-5050.102.1.006
http://dx.doi.org/10.3163/1536-5050.102.... ^{), (17}17. Patel RK, Sayers AE, Patrick NL, Hughes K, Armitage J, Hunter IA. A UK perspective on smartphone use amongst doctors within the surgical profession. Annals of Medicine and Surgery 2015;4:107-112. https://doi.org/10.1016/j.amsu.2015.03.004
https://doi.org/10.1016/j.amsu.2015.03.0... and that has the least knowledge about the filling of DC, the availability of AtestaDO would enable the greater access to instructions. Introducing the use of the application in medical residency programs - already at the beginning of the activities via a resolution of the Comissão Nacional de Residência Médica - could be a good incentive in this direction.

The application’s interface was the most well-evaluated item in the three workshops. According to Boruff and Storie¹⁴14. Boruff JT, Storie D. Mobile devices in medicine: a survey of how medical students, residents, and faculty use smartphones and other mobile devices to find information. J Med Lib Assoc. 2014;102(1):22-30. http://dx.doi.org/10.3163/1536-5050.102.1.006
http://dx.doi.org/10.3163/1536-5050.102.... , a usable interface and offline access to information within an application were considered as facilitators to access the resources of a smartphone, functionalities present in AtestaDO. In the most updated version used in the BH_2 workshop, the next items with the highest scores were “Questions and answers” and “Important concepts”, which may be responding to a demand from the public consisting of young physicians. The section “Practice with exercises” was very well evaluated in Natal but not in BH_2, possibly due to the profile of this public - previously described. The worst evaluation of the application - during BH_1 - was probably due to the use of the intermediate version of the application and to the connectivity problem. According to Boruff and Storie¹⁴, access to wi-fi connectivity at the hospital or clinic is recognized as a limiter to the use of mobile devices.

The low adherence to evaluate and answer the questionnaire in the last evaluation (BH_2) may have influenced the results. Such return rate, however, was better than the one found in other studies on the use of mobile devices by physicians, studies and residents¹⁴14. Boruff JT, Storie D. Mobile devices in medicine: a survey of how medical students, residents, and faculty use smartphones and other mobile devices to find information. J Med Lib Assoc. 2014;102(1):22-30. http://dx.doi.org/10.3163/1536-5050.102.1.006
http://dx.doi.org/10.3163/1536-5050.102.... ^{), (16}16. Payne KFB, Wharrad H, Watts K. Smartphone and medical related app use among medical students and junior doctors in the United Kingdom (UK): a regional survey. BMC Med Inform Decis Mak. 2012;12:121. http://dx.doi.org/10.1186/1472-6947-12-121
http://dx.doi.org/10.1186/1472-6947-12-1... . Another limitation raised by the participants was time, considered insufficient to better analyze the application.

The structuring of the coordinating group in the development of the application must be highlighted. This group contributed to improve the performance of AtestaDO in several evaluations conducted with the MS, which included guidelines for the correct filling of DC, guidelines for specifying the underlying cause of death considering the garbage codes, final revision of the content (text), definition of the layout and topic structure, as well as the review of clinical cases and inclusion of the tutorial.

In all workshops the suggestion to make the application more interactive was emphasized, an aspect inserted in the latest version of AtestaDO (0.3.6, available in 2018) by integrating Iris²2. Degani AT, Patel RM, Smith BE, Grimsley E. The effect of student training on accuracy of completion of death certificates. Med Educ Online. 2009;14:17. https://doi.org/10.3885/meo.2009.Res00315
https://doi.org/10.3885/meo.2009.Res0031... ^{), (22}22. Iris Institute. Iris User Reference Manual V5.6.0S1 [Internet]; 2018 [citado em 15 maio 2019]. Disponível em: http://bit.ly/2Lm11jt
http://bit.ly/2Lm11jt... ^{)- (25}25. Harteloh P. The automated coding of causes of death in the Netherlands. Epidemol Int J. 2017;1(1):000102., an automatic system to encode the causes declared by the physician in the DC and select the underlying cause of death. This is the result of a collaborative effort involving several institutions from different countries, including the German Institute of Medical Documentation and Information and the U.S National Center for Health Statistics. This system can be used in multiple languages as long as a dictionary of diagnostic terms is created. In text input mode, the user enters the causes of death in free text, and a dictionary of terms included in Iris translates such information into codes of the International Statistical Classification of Diseases and Related Health Problems - 10^th Revision (ICD-10). Thus, in the last version of the application the practical exercises were made interactive and after filling the causes of death, the physician becomes aware of the quality of the filling.

The good acceptability of the application is also reflected in the number of installations. AtestaDO was installed on approximately 32,000 devices in the period from March 2017 to May 2019, according to the MS. Such initiative is being used by other Latin American countries, which have also developed their applications based on AtestaDO.

In addition to developing this application, the MS has been performing other actions to improve the quality of information such as the proposal to implant Iris in the coding of causes and selection of the underlying cause of death, enabling international comparability. Among other initiatives of the Data for Health project, approximately 70 in-person trainings were conducted for over 2,000 physicians between June 2017 and January 2019, in which AtestaDO was presented and used in practical exercises.

CONCLUSION

The evaluation of the AtestaDO application showed good acceptability. The results indicate that AtestaDO presents a significant potential to impact the quality of the causes of death certified by physician, as it constitutes a complement and alternative for the in-person training of physicians. This is a sustainable long-term initiative, available for free to physicians on multiple application platforms, in addition to being usable offline. Therefore, its improvement deserves continuity with constant updates such as the latest version, which presents greater interactivity with the integration of Iris.

Acknowledgments

The authors would like to thank the physicians who agreed to participate in the evaluation meetings, and Eleonora Gehlen Walcher for the suggestions.

References

Publication Dates

History