Abstract

The world has witnessed a powerful and radical transformation of social, economic and cultural relationships promoted by the Internet. The Internet provides opportunities for access, dissemination and production of information worldwide. Health, for example, stands out as one of the main areas with information of interest to a growing number of users. However, this information is often unsatisfactory, incorrect or incomprehensible. This paper analyzes an experiment of evaluation of information on dengue websites developed in a laboratory of the Oswaldo Cruz Foundation. It counted on the participation of a group of Manguinhos dwellers and Public Health PHC physicians, infectious disease specialists and public health physicians in the development of criteria and the evaluation of websites. This paper shows the main results of this experience, which is innovative because its paper and product differ from those proposed by national and foreign agencies and analysts. This experience supports the establishment of an institutional process that issues a quality seal to websites that comply with the suggested criteria and indicators.

Standards; Evaluation; Internet; Dengue; Community participation

Introduction

In recent years, the world has witnessed an intense and radical transformation in the social, economic and cultural relationships fostered with the advent and expansion of the Internet¹1. Lévy P. Cybercultura. São Paulo: Editora 34; 1999.. This change has allowed anyone with minimal skills and purchasing power to buy electronic devices to produce and access information previously restricted to certain social groups or difficult to reach for most of the population²2. Lemos A, Lévy P. O futuro da internet: Em direção a uma ciberdemocracia planetária. São Paulo: Paulus; 2010.. The Internet is an arena with countless sources of information and possibilities for interactivity between individuals. Search engines facilitate access to these sources and social networks build new online relationship patterns³3. Recuero R. Redes sociais na internet. 2a ed. Porto Alegre: Sulina; 2011.. In addition, the very individual can produce information by organizing his own website, blog or Facebook. Thus, the Internet offers new opportunities for access, dissemination and information production worldwide⁴4. Castiel LR, Vasconcelos-Silva PR. Precariedades do Excesso: Informação e Comunicação em Saúde Coletiva. Rio de Janeiro: Fiocruz; 2006..

Health emerges as one of the main areas with information of interest to a growing number of users, whether they are patients or professionals. This information can help prevent diseases and encourage health promotion⁵5. Garbin HBR, Guilam MCR, Pereira Neto AF. Internet na promoção da saúde: um instrumento para o desenvolvimento de habilidades pessoais e sociais. Physis 2012; 22(1):347-363.. It also builds an informed patient: a special consumer of information about health products and services who feels somehow an expert in a particular subject and can potentially transform the traditional doctor-patient relationship based on authority concentrated in the hands of the professional⁶6. Garbin HB, Pereira Neto A, Guilam MC. A internet, o paciente expert e a prática médica: uma análise bibliográfica. Interface (Botucatu) 2008; 12(26):579-588..

On Facebook, many patients share information and experiences about their problems⁷7. Pereira Neto A, Barbosa L, Silva A, Dantas, M. O paciente informado e os saberes médicos: um estudo de etnografia virtual em comunidades de doentes no Facebook. Hist. cienc. saude-Manguinhos 2015; 22 (Supl.):1653-1671.. The countless sites on themes linked to some extent to health-disease issues are produced by public/private organizations or individuals without any kind of evaluation. This information is often insufficient, unsatisfactory, inaccurate or incomprehensible. As a result, it can put the citizen’s health at risk. In this context, the evaluation of the quality of health information available online has come to play a relevant role in several parts of the world⁸8. Mendonça AP, Pereira Neto A. Critérios de avaliação da qualidade da informação em sites de saúde: uma proposta. Reciis 2015; 9(1):1-15..

Concern over the evaluation of health information available on the Internet is just over fifteen years old. Lopes⁹9. Lopes IL. Iniciativas internacionais para o controle da qualidade da informação em Saúde na Web. Reciis 2012; 6(2):2-12. analyzed thirteen of the main international initiatives proposed by international agencies, private and non-governmental institutions for this purpose. At the end of the evaluation, some institutions issue a quality seal while others provide instructions to information providers. Some other disclose a code of conduct that seeks to guide users to check whether sites comply with pre-established criteria. The author highlights Health on the Net Foundation (HON) and the Discern Questionnaire (DQ). HON is a private institution, based in Switzerland, which has been certifying health web pages since 1995, issuing a quality seal. The British Library and the National Health Services (NHS) established the Discern Questionnaire in 1996. It provides a tool for users to make this assessment. In 1997, the Agency for Health Care Policy and Research and the Health Information Technology Institute (HITI) drew up the document titled Criteria for the Quality of Health Information on the Internet, similar to that of the Discern Questionnaire.

Eysenbach et al.¹⁰10. Eysenbach G, Powell J, Kuss O, Sá E. Empirical studies assessing the quality of health information for consumers on the World Wide Web: a systematic review. JAMA 2002; 287(20):2691-2700. published the results of a comprehensive systematic review, analyzing 79 articles on the subject that used 86 different criteria for the evaluation of health websites. Making a synthesis effort, these authors organized these criteria into five groups: Accuracy, Technical, Design, Scope and Readability. The quality and nature of this research, coupled with the fact that it was published in the American Medical Association journal, made this study one of the main references in the subject at the time of this experiment.

In Brazil, the São Paulo State Regional Council of Medicine (CREMESP) published in 2001 a Resolution accompanied by the “Manual of ethical principles for medical and health websites”. It provides users and website managers with recommendations. In academic terms, the national bibliographic production on the subject is incipient. During the month of September of 2013, we carried out a bibliographic survey about the Brazilian production on Internet and Health available at the Scientific Electronic Library Online (SciELO) database. We used keyword “internet” in the subject search and found 316 titles. Of these, 112 addressed health issues/problems. We read their summaries and found 23 titles dealing specifically with the issue of evaluation of information quality in health websites¹¹11. Barbosa AL, Martins EN. Avaliação da informação dos pacientes sobre miopsias e fotopsias através da Internet. Arq Bras Oftalmol 2007; 70(5):839-843.

12. Carlini BH, Ronzani TM, Martins LF, Gomide HP, Souza ICW. Demand for and availability of online support to stop smoking. Rev Saude Publica 2012; 46(6):1066-1081.

13. Conti MA, Jardim AP, Hearst N, Cordás TA, Tavares H, Abreu CN. Avaliação da equivalência semântica e consistência interna de uma versão em português do Internet Addiction Test (IAT). Rev Psiq Clin 2012; 39(3):106-110.

14. Cubas MR, Felchner PCZ. Análise das fontes de informação sobre os autoexames da mama disponíveis na Internet. Cien Saude Colet 2012; 17(4):965-970.

15. Del Giglio A, Abdala B, Ogawa C, Amado D, Carter D, Gomieiro F, Salama F, Shiroma M, Del Giglio A. Qualidade da informação da internet disponível para pacientes em páginas em português. Rev Assoc Med Bras 2012; 58(6):645-649.

16. Gondim APS, Falcão CB. Avaliação das farmácias virtuais brasileiras. Rev Saude Publica 2007; 41(2):297-300.

17. Gondim APS, Weyne DP, Ferreira BSP. Qualidade das informações de saúde e medicamentos nos sítios brasileiros. Einstein 2012; 10(3):335-341.

18. Guardiola-Wanden-Berghe R, Sanz-Valero J, Wanden-Berghe C. Quality assessment of the Website for Eating Disorders: a systematic review of a pending challenge. Cien Saude Colet 2012; 17(9):2489-2497.

19. Leite F, Correia A. Avaliação da qualidade dos sítios da internet com informação de cárie na criança em língua Portuguesa. Rev Odonto Cienc 2011; 26(2):116-120.

20. Lins TH, Marin HF. Avaliação de website sobre assistência de enfermagem na sala de recuperação pós-anestésica. Acta Paul Enferm 2012; 25(1):109-115.

21. Lopes IL. Novos paradigmas para avaliação da qualidade da informação em saúde recuperada na Web. Ci Inf 2004; 33(1):81-90.

22. Moretti FA, Heymann RE, Marvulle V, Pollak DF, Riera R. Avaliação do nível de conhecimento sobre fibromialgia entre usuários da internet. Rev Bras Reumatol 2011; 51(1):13-19.

23. Moretti FA, Oliveira VE, Silva EMK. Acesso a informações de saúde na internet: uma questão de saúde pública? Rev Assoc Med Bras 2012; 58(6):650-658.

24. Prado OZ, Meyer SB. Avaliação da relação terapêutica na terapia assíncrona via internet. Psicol Estud 2006; 11(2):247-257.

25. Santos V, Portal MM, Ferigolo M, Dantas DCM, Barros HMT, Trindade CS. Sites sobre drogas de abuso: recursos para avaliação. Trab Educ Saúde 2010; 8(3):575-585.

26. Silva LVER, Mello Junior JF, Mion O. Avaliação das informações sobre rinite alérgica em sites brasileiros na rede mundial de computadores (Internet). Rev Bras Otorrinolaringol 2005; 71(5):590-597.

27. Silva RQ, Gubert MB. Qualidade das informações sobre aleitamento materno e alimentação complementar em sites brasileiros de profissionais de saúde disponíveis na internet. Rev Bras Saúde Matern Infant 2010; 10(3):331-340.

28. Silveira PCM, Costa AES, Lima CC. Gagueira na web: qualidade da informação. Rev CEFAC 2012; 14(3):430-437.

29. Souza CLN, Luz ZP, Rabello A. Análise da informação sobre a leishmaniose visceral disponível em portais brasileiros da rede mundial de computadores: internet. Rev Soc Bras Med Trop. 2008; 41(4):352-357.

30. Cronemberger EV, Portocarrero ML, Donato AR, Cunha MS, Barreto TF, Meneses JVL. O uso da internet como fonte de informação sobre cirurgia plástica na Bahia, Brasil. Rev. Bras. Cir. Plást. 2012; 27(4):531-535.

31. Pithon M, Santos E. Information available on the internet about pain after orthognathic surgery: A careful review. Dental Press J. Orthod. 2014; 19(6):86-92.

32. Mori S, Whitaker IY, Marin HF. Avaliação do website educacional em Primeiros Socorros. Rev. esc. enferm. USP 2013; 47(4):950-957.-³³33. Sousa P, Fonseca H, Gaspar P, Gaspar F. Usability of an internet-based platform (Next.Step) for adolescent weight management. J Pediatr 2015; 91(1):68-74.. Each one of them evaluated the quality of information from a specific pathology, behavior or lifestyle associated with health or disease, such as smoking and breastfeeding. In general, these authors take as reference the criteria of the aforementioned international agencies. Eysenbach et al.¹⁰10. Eysenbach G, Powell J, Kuss O, Sá E. Empirical studies assessing the quality of health information for consumers on the World Wide Web: a systematic review. JAMA 2002; 287(20):2691-2700. identified five other criteria in specialized international literature. There is, therefore, no consensus on the criteria and procedures for carrying out this evaluation.

This paper aims to show and analyze the health information quality evaluation experiment developed at the “Internet, Health and Society Laboratory” (LaISS), linked to the Germano Sinval Faria School Health Center, National School of Public Health, Oswaldo Cruz Foundation. On that occasion, 20 dengue websites were evaluated.

Dengue is a reemerging disease in the tropics and a public health problem of global importance³⁴34. World Health Organization (WHO). Dengue: Guidelines for Diagnosis, Treatment, Prevention and Control: New Edition. Geneva: WHO; 2009.. It is one of the major public health concerns in the contemporary world. Some 2.5 billion people in more than 100 countries, approximately 40% of the world’s population, are at risk of being infected by the dengue virus³⁵35. Jaenisch T; IDAMS, Sakuntabhai A; DENFREE, Wilder-Smith A; DengueTools. Dengue Research Funded by the European Commission-Scientific Strategies of Three European Dengue Research Consortia. PLoS Negl Trop Dis 2013; 7(12):e2320.. Dengue fever has been a neglected disease for many years. However, recently, there has been a great deal of investment in research on the subject. One of the signs to that end can be seen in the program called “Comprehensive Control of Dengue Fever under Changing Climatic Conditions” by the European Commission (EC), which, in 2011, offered about € 18 million for research in this area³⁵35. Jaenisch T; IDAMS, Sakuntabhai A; DENFREE, Wilder-Smith A; DengueTools. Dengue Research Funded by the European Commission-Scientific Strategies of Three European Dengue Research Consortia. PLoS Negl Trop Dis 2013; 7(12):e2320.. Dengue has been affecting mainly the populations living in urban areas of developing countries, where high demographic concentration and insufficient basic sanitation infrastructure produce ideal mosquito breeding conditions³⁶36. Guzman A, Istúriz RE. Update on the global spread of dengue. Int J Antimicrob Agents 2010; 36 (Supl. 1):S40-42.. Despite being a low lethality disease, dengue mortality has increased in Brazil due to the expanded geographical disease transmission areas and the introduction of new virus serotypes. Most deaths are avoidable through the timely implementation of therapeutic measures³⁷37. Nogueira RM, Eppinghaus AL. Dengue virus type 4 arrives in the state of Rio de Janeiro: a challenge for epidemiological surveillance and control. Mem Inst Oswaldo Cruz 2011; 106(3):255-256.. Thus, the population’s access to quality information on dengue is essential both to reduce disease transmission and to optimize the use of health services and curb mortality.

Thus, the LaISS chose dengue fever for this pilot research to evaluate websites’ information quality.

Methods

The criteria and indicators used in the experiment developed by the LaISS were built based on the results of the systematic review by Eysenbach et al.¹⁰10. Eysenbach G, Powell J, Kuss O, Sá E. Empirical studies assessing the quality of health information for consumers on the World Wide Web: a systematic review. JAMA 2002; 287(20):2691-2700. and by some reflections introduced by the national literature on the subject¹¹11. Barbosa AL, Martins EN. Avaliação da informação dos pacientes sobre miopsias e fotopsias através da Internet. Arq Bras Oftalmol 2007; 70(5):839-843.

12. Carlini BH, Ronzani TM, Martins LF, Gomide HP, Souza ICW. Demand for and availability of online support to stop smoking. Rev Saude Publica 2012; 46(6):1066-1081.

13. Conti MA, Jardim AP, Hearst N, Cordás TA, Tavares H, Abreu CN. Avaliação da equivalência semântica e consistência interna de uma versão em português do Internet Addiction Test (IAT). Rev Psiq Clin 2012; 39(3):106-110.

14. Cubas MR, Felchner PCZ. Análise das fontes de informação sobre os autoexames da mama disponíveis na Internet. Cien Saude Colet 2012; 17(4):965-970.

15. Del Giglio A, Abdala B, Ogawa C, Amado D, Carter D, Gomieiro F, Salama F, Shiroma M, Del Giglio A. Qualidade da informação da internet disponível para pacientes em páginas em português. Rev Assoc Med Bras 2012; 58(6):645-649.

16. Gondim APS, Falcão CB. Avaliação das farmácias virtuais brasileiras. Rev Saude Publica 2007; 41(2):297-300.

17. Gondim APS, Weyne DP, Ferreira BSP. Qualidade das informações de saúde e medicamentos nos sítios brasileiros. Einstein 2012; 10(3):335-341.

18. Guardiola-Wanden-Berghe R, Sanz-Valero J, Wanden-Berghe C. Quality assessment of the Website for Eating Disorders: a systematic review of a pending challenge. Cien Saude Colet 2012; 17(9):2489-2497.

19. Leite F, Correia A. Avaliação da qualidade dos sítios da internet com informação de cárie na criança em língua Portuguesa. Rev Odonto Cienc 2011; 26(2):116-120.

20. Lins TH, Marin HF. Avaliação de website sobre assistência de enfermagem na sala de recuperação pós-anestésica. Acta Paul Enferm 2012; 25(1):109-115.

21. Lopes IL. Novos paradigmas para avaliação da qualidade da informação em saúde recuperada na Web. Ci Inf 2004; 33(1):81-90.

22. Moretti FA, Heymann RE, Marvulle V, Pollak DF, Riera R. Avaliação do nível de conhecimento sobre fibromialgia entre usuários da internet. Rev Bras Reumatol 2011; 51(1):13-19.

23. Moretti FA, Oliveira VE, Silva EMK. Acesso a informações de saúde na internet: uma questão de saúde pública? Rev Assoc Med Bras 2012; 58(6):650-658.

24. Prado OZ, Meyer SB. Avaliação da relação terapêutica na terapia assíncrona via internet. Psicol Estud 2006; 11(2):247-257.

25. Santos V, Portal MM, Ferigolo M, Dantas DCM, Barros HMT, Trindade CS. Sites sobre drogas de abuso: recursos para avaliação. Trab Educ Saúde 2010; 8(3):575-585.

26. Silva LVER, Mello Junior JF, Mion O. Avaliação das informações sobre rinite alérgica em sites brasileiros na rede mundial de computadores (Internet). Rev Bras Otorrinolaringol 2005; 71(5):590-597.

27. Silva RQ, Gubert MB. Qualidade das informações sobre aleitamento materno e alimentação complementar em sites brasileiros de profissionais de saúde disponíveis na internet. Rev Bras Saúde Matern Infant 2010; 10(3):331-340.

28. Silveira PCM, Costa AES, Lima CC. Gagueira na web: qualidade da informação. Rev CEFAC 2012; 14(3):430-437.

29. Souza CLN, Luz ZP, Rabello A. Análise da informação sobre a leishmaniose visceral disponível em portais brasileiros da rede mundial de computadores: internet. Rev Soc Bras Med Trop. 2008; 41(4):352-357.

30. Cronemberger EV, Portocarrero ML, Donato AR, Cunha MS, Barreto TF, Meneses JVL. O uso da internet como fonte de informação sobre cirurgia plástica na Bahia, Brasil. Rev. Bras. Cir. Plást. 2012; 27(4):531-535.

31. Pithon M, Santos E. Information available on the internet about pain after orthognathic surgery: A careful review. Dental Press J. Orthod. 2014; 19(6):86-92.

32. Mori S, Whitaker IY, Marin HF. Avaliação do website educacional em Primeiros Socorros. Rev. esc. enferm. USP 2013; 47(4):950-957.-³³33. Sousa P, Fonseca H, Gaspar P, Gaspar F. Usability of an internet-based platform (Next.Step) for adolescent weight management. J Pediatr 2015; 91(1):68-74.. Initially, we will show these results to expose those used in the experiment that will be analyzed in this paper. Next, we will describe how these criteria were implemented in the assessment of information in 20 dengue websites developed by the LaISS. Then, we will explain how this evaluation process occurred.

Criteria proposed in literature

A criterion (from the Greek kritérion and Latin criteriu) is a standard that serves as the basis for things and people to be compared and judged. Eysenbach et al.¹⁰10. Eysenbach G, Powell J, Kuss O, Sá E. Empirical studies assessing the quality of health information for consumers on the World Wide Web: a systematic review. JAMA 2002; 287(20):2691-2700. found 86 criteria that were gathered in five groups. The first one, called Technical, covers the attribution of responsibility and reference for the information provided and the presentation of dates of the site’s establishment and update. Del Giglio¹⁵15. Del Giglio A, Abdala B, Ogawa C, Amado D, Carter D, Gomieiro F, Salama F, Shiroma M, Del Giglio A. Qualidade da informação da internet disponível para pacientes em páginas em português. Rev Assoc Med Bras 2012; 58(6):645-649., using the Discern Questionnaire as reference, included in his evaluation a question asking whether the evaluated website clearly provided the sources of information used.

The second criterion found by Eysenbach et al.¹⁰10. Eysenbach G, Powell J, Kuss O, Sá E. Empirical studies assessing the quality of health information for consumers on the World Wide Web: a systematic review. JAMA 2002; 287(20):2691-2700. aims to evaluate the aesthetic aspects of the website such as layout, colors and ease of navigation. It was labeled as “Design” by authors of this systematic review. Among national authors, six studies¹²12. Carlini BH, Ronzani TM, Martins LF, Gomide HP, Souza ICW. Demand for and availability of online support to stop smoking. Rev Saude Publica 2012; 46(6):1066-1081.,¹⁹19. Leite F, Correia A. Avaliação da qualidade dos sítios da internet com informação de cárie na criança em língua Portuguesa. Rev Odonto Cienc 2011; 26(2):116-120.,²²22. Moretti FA, Heymann RE, Marvulle V, Pollak DF, Riera R. Avaliação do nível de conhecimento sobre fibromialgia entre usuários da internet. Rev Bras Reumatol 2011; 51(1):13-19.,³¹31. Pithon M, Santos E. Information available on the internet about pain after orthognathic surgery: A careful review. Dental Press J. Orthod. 2014; 19(6):86-92.

32. Mori S, Whitaker IY, Marin HF. Avaliação do website educacional em Primeiros Socorros. Rev. esc. enferm. USP 2013; 47(4):950-957.-³³33. Sousa P, Fonseca H, Gaspar P, Gaspar F. Usability of an internet-based platform (Next.Step) for adolescent weight management. J Pediatr 2015; 91(1):68-74. have also been concerned with this realm.

According to Eysenbach et al.¹⁰10. Eysenbach G, Powell J, Kuss O, Sá E. Empirical studies assessing the quality of health information for consumers on the World Wide Web: a systematic review. JAMA 2002; 287(20):2691-2700., the third criterion involves the Scope of information provided. This criterion aims to verify whether a website has the subjects relevant to the topic of health-disease addressed. Generally, this criterion is not explicit, but is used in the context of other criteria. This can be seen in the examples of two national studies³⁰30. Cronemberger EV, Portocarrero ML, Donato AR, Cunha MS, Barreto TF, Meneses JVL. O uso da internet como fonte de informação sobre cirurgia plástica na Bahia, Brasil. Rev. Bras. Cir. Plást. 2012; 27(4):531-535.,³²32. Mori S, Whitaker IY, Marin HF. Avaliação do website educacional em Primeiros Socorros. Rev. esc. enferm. USP 2013; 47(4):950-957..

Accuracy is the fourth criterion. It was defined by Eysenbach et al.¹⁰10. Eysenbach G, Powell J, Kuss O, Sá E. Empirical studies assessing the quality of health information for consumers on the World Wide Web: a systematic review. JAMA 2002; 287(20):2691-2700. by level of agreement between the information provided and the best evidence generally accepted by medical practice. In our view, accuracy and actuality are essential attributes for the provision of health information. Therefore, health websites need to show the correct information from the scientific viewpoint. Two national authors³¹31. Pithon M, Santos E. Information available on the internet about pain after orthognathic surgery: A careful review. Dental Press J. Orthod. 2014; 19(6):86-92.,³²32. Mori S, Whitaker IY, Marin HF. Avaliação do website educacional em Primeiros Socorros. Rev. esc. enferm. USP 2013; 47(4):950-957. have also been concerned with this realm of information.

The systematic review by Eysenbach et al.¹⁰10. Eysenbach G, Powell J, Kuss O, Sá E. Empirical studies assessing the quality of health information for consumers on the World Wide Web: a systematic review. JAMA 2002; 287(20):2691-2700. identified a fifth criterion called Readability. It aims to check the level of comprehension of the text available online. Authors who used this criterion verified the length and complexity of sentences and words, using formulas such as Flesh-Kincaid Grade Level Index¹⁰10. Eysenbach G, Powell J, Kuss O, Sá E. Empirical studies assessing the quality of health information for consumers on the World Wide Web: a systematic review. JAMA 2002; 287(20):2691-2700.. Among national authors, two³¹31. Pithon M, Santos E. Information available on the internet about pain after orthognathic surgery: A careful review. Dental Press J. Orthod. 2014; 19(6):86-92.,³²32. Mori S, Whitaker IY, Marin HF. Avaliação do website educacional em Primeiros Socorros. Rev. esc. enferm. USP 2013; 47(4):950-957. have been concerned with this aspect of information.

None of the Brazilian papers¹¹11. Barbosa AL, Martins EN. Avaliação da informação dos pacientes sobre miopsias e fotopsias através da Internet. Arq Bras Oftalmol 2007; 70(5):839-843.

12. Carlini BH, Ronzani TM, Martins LF, Gomide HP, Souza ICW. Demand for and availability of online support to stop smoking. Rev Saude Publica 2012; 46(6):1066-1081.

13. Conti MA, Jardim AP, Hearst N, Cordás TA, Tavares H, Abreu CN. Avaliação da equivalência semântica e consistência interna de uma versão em português do Internet Addiction Test (IAT). Rev Psiq Clin 2012; 39(3):106-110.

14. Cubas MR, Felchner PCZ. Análise das fontes de informação sobre os autoexames da mama disponíveis na Internet. Cien Saude Colet 2012; 17(4):965-970.

15. Del Giglio A, Abdala B, Ogawa C, Amado D, Carter D, Gomieiro F, Salama F, Shiroma M, Del Giglio A. Qualidade da informação da internet disponível para pacientes em páginas em português. Rev Assoc Med Bras 2012; 58(6):645-649.

16. Gondim APS, Falcão CB. Avaliação das farmácias virtuais brasileiras. Rev Saude Publica 2007; 41(2):297-300.

17. Gondim APS, Weyne DP, Ferreira BSP. Qualidade das informações de saúde e medicamentos nos sítios brasileiros. Einstein 2012; 10(3):335-341.

18. Guardiola-Wanden-Berghe R, Sanz-Valero J, Wanden-Berghe C. Quality assessment of the Website for Eating Disorders: a systematic review of a pending challenge. Cien Saude Colet 2012; 17(9):2489-2497.

19. Leite F, Correia A. Avaliação da qualidade dos sítios da internet com informação de cárie na criança em língua Portuguesa. Rev Odonto Cienc 2011; 26(2):116-120.

20. Lins TH, Marin HF. Avaliação de website sobre assistência de enfermagem na sala de recuperação pós-anestésica. Acta Paul Enferm 2012; 25(1):109-115.

21. Lopes IL. Novos paradigmas para avaliação da qualidade da informação em saúde recuperada na Web. Ci Inf 2004; 33(1):81-90.

22. Moretti FA, Heymann RE, Marvulle V, Pollak DF, Riera R. Avaliação do nível de conhecimento sobre fibromialgia entre usuários da internet. Rev Bras Reumatol 2011; 51(1):13-19.

23. Moretti FA, Oliveira VE, Silva EMK. Acesso a informações de saúde na internet: uma questão de saúde pública? Rev Assoc Med Bras 2012; 58(6):650-658.

24. Prado OZ, Meyer SB. Avaliação da relação terapêutica na terapia assíncrona via internet. Psicol Estud 2006; 11(2):247-257.

25. Santos V, Portal MM, Ferigolo M, Dantas DCM, Barros HMT, Trindade CS. Sites sobre drogas de abuso: recursos para avaliação. Trab Educ Saúde 2010; 8(3):575-585.

26. Silva LVER, Mello Junior JF, Mion O. Avaliação das informações sobre rinite alérgica em sites brasileiros na rede mundial de computadores (Internet). Rev Bras Otorrinolaringol 2005; 71(5):590-597.

27. Silva RQ, Gubert MB. Qualidade das informações sobre aleitamento materno e alimentação complementar em sites brasileiros de profissionais de saúde disponíveis na internet. Rev Bras Saúde Matern Infant 2010; 10(3):331-340.

28. Silveira PCM, Costa AES, Lima CC. Gagueira na web: qualidade da informação. Rev CEFAC 2012; 14(3):430-437.

29. Souza CLN, Luz ZP, Rabello A. Análise da informação sobre a leishmaniose visceral disponível em portais brasileiros da rede mundial de computadores: internet. Rev Soc Bras Med Trop. 2008; 41(4):352-357.

30. Cronemberger EV, Portocarrero ML, Donato AR, Cunha MS, Barreto TF, Meneses JVL. O uso da internet como fonte de informação sobre cirurgia plástica na Bahia, Brasil. Rev. Bras. Cir. Plást. 2012; 27(4):531-535.

31. Pithon M, Santos E. Information available on the internet about pain after orthognathic surgery: A careful review. Dental Press J. Orthod. 2014; 19(6):86-92.

32. Mori S, Whitaker IY, Marin HF. Avaliação do website educacional em Primeiros Socorros. Rev. esc. enferm. USP 2013; 47(4):950-957.-³³33. Sousa P, Fonseca H, Gaspar P, Gaspar F. Usability of an internet-based platform (Next.Step) for adolescent weight management. J Pediatr 2015; 91(1):68-74. available in SciELO simultaneously used the five criteria suggested by the systematic review developed by Eysenbach et al.¹⁰10. Eysenbach G, Powell J, Kuss O, Sá E. Empirical studies assessing the quality of health information for consumers on the World Wide Web: a systematic review. JAMA 2002; 287(20):2691-2700.. This intellectual endeavor was undertaken in the work developed by the LaISS.

Criteria used in the research

The evaluation carried out by the LaISS used the five criteria proposed by the systematic review of Eysenbach et al.¹⁰10. Eysenbach G, Powell J, Kuss O, Sá E. Empirical studies assessing the quality of health information for consumers on the World Wide Web: a systematic review. JAMA 2002; 287(20):2691-2700.. Criteria Technical, Scope and Accuracy were used in full, with the same meanings. Design and Readability were implemented differently.

The realm of Design was renamed Interactivity. Thus, the evaluation of websites with this criterion focused on the verification of resources that facilitated navigation and enabled communication between users and websites’ managers.

Regarding Readability, LaISS’ experiment did not use formulas that verify the extent and complexity of sentences and words. These tools have limitations because they do not take into account other subjective aspects that affect the comprehension of texts and words¹⁰10. Eysenbach G, Powell J, Kuss O, Sá E. Empirical studies assessing the quality of health information for consumers on the World Wide Web: a systematic review. JAMA 2002; 287(20):2691-2700.. LaISS’ work incorporated this criterion in a different and innovative way: users evaluated whether or not they understood the information available on the websites. We will discuss how this participation occurred in the presentation of the evaluation process.

It is also worth mentioning how Scope was structured according to the theme of dengue. The website that showed information on prevention, transmission, symptom, diagnosis and treatment of this disease was considered comprehensive.

Each criterion was accompanied by a distinct number of indicators that address specific aspects of each realm of the information. The 63 indicators were designed as a question (Chart 1).

The process

Analyzing the work that integrates the systematic review carried out by Eysenbach et al.¹⁰10. Eysenbach G, Powell J, Kuss O, Sá E. Empirical studies assessing the quality of health information for consumers on the World Wide Web: a systematic review. JAMA 2002; 287(20):2691-2700., we can see that the work developed in the LaISS has introduced some notable contributions in its realization process.

The first and most important was inviting users and professionals to carry out this evaluation. Eysenbach et al.¹⁰10. Eysenbach G, Powell J, Kuss O, Sá E. Empirical studies assessing the quality of health information for consumers on the World Wide Web: a systematic review. JAMA 2002; 287(20):2691-2700. concluded in their systematic review that none of the studies took into account in their evaluation the views of real users or experts on the subject.

In the experience of the LaISS, twenty residents of the Communities of Manguinhos made the assessment on behalf of the Unified Health System (SUS) users. We invited these 20 dwellers since they had previously participated in a digital inclusion activity at the LaISS. This lab is equipped with a room with 10 computers. Thus, it was possible to make two groups with 10 residents who gathered at different days and times: seventeen women and three men, most with incomplete primary education level and age ranging from 30 to 40 years. Many of them had no basic computer skills or knowledge, nor were they familiar with social networks. Thus, a series of activities were carried out to sensitize them about the structures and basic components of WEB pages.

Another challenge was to familiarize these collaborators with the terms commonly used in the Internet universe such as site, homepage, website, link, download, send, delete, which are foreign words difficult to understand for a population with poor schooling. This familiarization lasted about six months.

They received a scholarship from the Vice Presidency of Environment, Care and Health Promotion (VPAAPS) of the Oswaldo Cruz Foundation (Fiocruz). Therefore, we can fairly assume that the socioeconomic profile and level of schooling are broadly similar to that of most Brazilians. The project also had the support of the “Program for Development and Technological Innovation in Public Health in the Territory of Integrated Actions in Health” (PDT-SP-TEIAS).

In addition to dwellers, we invited 10 physicians to make the assessment on behalf of health professionals. The initial idea was to gather 20 physicians, but the voluntary status of this initiative made it difficult to reach this amount. Among the 10 participating physicians are five experienced PHC professionals working at the Germano Sinval Faria School Health Center (CSEGSF), National School of Public Health (ENSP), Fiocruz; one researcher at the Evandro Chagas Clinical Research Institute (IPEC), Fiocruz, a scholar of febrile diseases and four newly graduated physicians who work in the Family Health Strategy in Manguinhos.

Two medical consultants assisted in the elaboration of questions specifically related to this disease. All participated as evaluators of the selected dengue websites. In all, 63 indicators were set by dwellers of the community of Manguinhos and the medical consultants.

To carry out the evaluation, the LaISS built two online tools, one for each group of evaluators. Users’ assessment had a different meaning from that performed by doctors. The first ones verified whether the content was agreeable, clear and if the site was of easy navigation. These aspects were included in the indicators of Readability. The physicians evaluated information compliance with the scientific knowledge about dengue, a criterion known as Accuracy. Both verified the Technical, Interactivity and Scope criteria. The evaluation work took place in the first five months of 2013.

A second contribution introduced by this work developed by the LaISS refers to the selection method of the evaluated sites. In general, authors type some keyword (s) in a search engine, choose the websites and perform the evaluation. In this case, the procedure for selecting the evaluated websites was slightly different.

In the first place, a research was carried out on 40 computers of LAN Houses located in the Communities of Manguinhos. The top 10 sites were listed as a result on the first page of the Google search engine, using the word “dengue”, on each of these computers. In general, users of this browser hardly ever move forward to the second page in their search. Thus, there is a great possibility that these sites are the most accessed by those who carry out the same search to obtain information about this disease in the LAN Houses of these communities. The final result of the implementation of this method enabled the selection of 10 dengue sites. Among them are government sites such as the Ministry of Health, collective construction, such as the Wikipedia and independent private initiatives or linked to major media, such as the G1 of Globo TV Network. The order in which they are shown in Chart 2 follows the sequence found on most of the 40 computers surveyed.

Also included in this list were dengue sites with a quality seal issued by Swiss non-governmental organization HON. The virtual address of this NGO contains HON-certified health sites in different countries and address different diseases or health problems. At the time of this research, five Brazilian dengue sites that received their seal of quality were found. None of them are among the most accessed in Manguinhos Communities. All of them are privately owned and none linked to government agencies, academics, major media or collective production.

This list was completed with the inclusion of three public sites linked to government agencies. Thus, a group of 20 sites was established (Chart 2).

A third innovative aspect refers to the principle of transparency. In the evaluation carried out by the LaISS, mention is made of the evaluated sites and their respective electronic addresses. Of the 23 Brazilian papers that made similar ventures, only five¹²12. Carlini BH, Ronzani TM, Martins LF, Gomide HP, Souza ICW. Demand for and availability of online support to stop smoking. Rev Saude Publica 2012; 46(6):1066-1081.,¹⁹19. Leite F, Correia A. Avaliação da qualidade dos sítios da internet com informação de cárie na criança em língua Portuguesa. Rev Odonto Cienc 2011; 26(2):116-120.,²⁸28. Silveira PCM, Costa AES, Lima CC. Gagueira na web: qualidade da informação. Rev CEFAC 2012; 14(3):430-437.,³¹31. Pithon M, Santos E. Information available on the internet about pain after orthognathic surgery: A careful review. Dental Press J. Orthod. 2014; 19(6):86-92.,³³33. Sousa P, Fonseca H, Gaspar P, Gaspar F. Usability of an internet-based platform (Next.Step) for adolescent weight management. J Pediatr 2015; 91(1):68-74. mentioned the evaluated sites. Transparency was also evidenced in this experiment through the way results were shown. As will be seen below, the final result that each site obtained will be shown. In the national papers, only two²⁸28. Silveira PCM, Costa AES, Lima CC. Gagueira na web: qualidade da informação. Rev CEFAC 2012; 14(3):430-437.,³¹31. Pithon M, Santos E. Information available on the internet about pain after orthognathic surgery: A careful review. Dental Press J. Orthod. 2014; 19(6):86-92. listed electronic addresses of each evaluated site.

A fourth and final contribution deserves mention: the process of constructing the results.

The sixty-three indicators were organized in the form of interrogative sentences. Most have possible answers “Yes” or “No”. They indicate if a certain aspect of the information is available or not. Thus, each indicator had an ideal answer in order to establish whether information was of quality. This strategy was inspired by the concept of compliance. According to the definition of the Brazilian Association of Technical Standards (ABNT), compliance is the “demonstration that the specified requirements related to a product, process, system, person or organism are met” (ABNT NBR ISO/IEC 17000:2005).

Here is an example. The following question appears in the Technical criterion: “Does the site inform who the responsible person is?” This question was elaborated with the intention of verifying whether the information on the individual or legal entity, public or private that is self-declared responsible for the site is explicit. This information is often found on links such as “About Us” or “About”. It may also be next to the copyright symbol (©). In this case, there are two answers: “Yes” and “No”. The site that obtains the “Yes” response from most evaluators complies with this indicator.

Also regarding the process of building results, in the work carried out by the LaISS, all criteria were equally important. This view builds on the following argument: a website that provides quality health information must meet criteria of accountability in relation to the information offered. It must be understandable, correct and current. In the case of dengue, the site should contain information on prevention, transmission, symptoms, diagnosis and treatment and must be interactive. In the LaISS initiative, the number of indicators per criterion varies (Table 1). How did it solve this problem? The solution was to transform results into percentages.

If the question mentioned above about whether or not the person responsible for the site is available, obtaining 15 “Yes” responses among 30 evaluators in that indicator would mean that the evaluated site would have reached 50% compliance.

In the results shown below, the percentage of expected responses within each criterion will be presented. These totals allowed the establishment of a ranking between sites (Table 1) that allows the identification of strengths and weaknesses in each criterion.

Results

Table 1 shows the overall evaluation result. Thus, we have been keen to emphasize that the quality of information in health websites is a problem that deserves to be faced. Thus, we did not highlight the results from the different inputs of dwellers and physicians.

The 20 evaluated sites are in the left column and follow the position that they held in the ranking of the evaluation carried out by the LaISS. This data organization by evaluation criteria used facilitates the work of the website manager, who will be able to identify the strengths and weaknesses of his website or page. Therefore, we would like to underscore this characteristic of the process undertaken. In addition, transparency in the presentation of results by criterion is an important contribution and warrants further evaluation with this method.

The same table reveals that none of the eighteen dengue sites evaluated by dwellers and physicians managed to level at 70% according to the criteria used. If this were the accepted minimum compliance rate, no site or page would receive a seal that recognized its quality.

The fact that none of the 20 sites evaluated has met this level of compliance justifies the need for online information quality assessment initiatives like this.

If the percentage acceptable to a site were 60% of the answers expected by doctors and users, only three sites would have achieved this rate: sites of Combate a Dengue, Cria Saúde and the Rio de Janeiro State Health Secretariat of (SESRJ). Let us look at these cases more closely.

The first ranked, with 65.5% compliance, was the combateadengue.com.br site. This is a dengue-exclusive website. According to its introduction, its information was based on research in several sites on the subject, such as the pages of the Ministry of Health, public institutions and NGOs that carry out research on the disease. This website states that it is keen to transmit information clearly and objectively. Its strength is Readability, evaluated only by Manguinhos dwellers, where it achieved a level of compliance of 70.2%. In the criterion Accuracy (53.5%), evaluated only by physicians, it received lower scores. This is a fundamental aspect in this work of evaluating information quality: the best-ranked site in the final ranking was the one that obtained the highest mean by adding and dividing the scores obtained in each criterion. This site, for example, did not achieve the best score in any particular criterion. The website combateadengue.com.br obtained high means in all criteria, although it did not obtain the highest score in any of them.

Ranked second was the dengue page on the criasaude.com.br website. This site differs from the previous one since it is not dengue-exclusive. Its pages address different diseases, medicines and nutritional guidelines. According to the website, its sources of information are employees, communication agencies, medical books such as the Larousse Médical and medical journals, such as the Tribune Médicale, a medical journal published in Switzerland. Its website informs that it is affiliated with Swisscam Brasil and Pharmanetis Inc³⁸38. Pharmanetis Sàrl [Internet]. Pharmanetis Sàrl (GmbH); c2010. [cited 2014 Feb 3]. Available from: http://www.pharmanetis.com/
http://www.pharmanetis.com/... of Switzerland. This foreign organization’s website specifies its objective: putting itself in the internaut’s shoes, proposing an unforgettable experience to him/her and increasing sales. It states that its clients are pharmaceutical laboratories, clinics and pharmacies. It proposes to edit health-related content on the Internet with strong added value (photo, text, animation, and video) in different languages. It claims that it works with the news as process concept. Thus, it says it posts new information every day on the site. This strategy may explain why this page received the highest score in Scope (72.8%) – the average obtained with the evaluation performed by physicians and dwellers of Manguinhos. When you enter new information every day, the odds of coverage being complete are greater. Research developed by the LaISS has identified online initiatives as these linked to the interests of the international pharmaceutical industry.

Ranked third was the riocontraadengue.com.br site of the State Health Secretariat of Rio de Janeiro. This site is not among those identified in the survey conducted in the LAN Houses of the Communities of Manguinhos. This very fact seems surprising, since this research occurred in a district of a city of this state. One could suppose, therefore, that this site would be one of the most accessed. This shortcoming may be justified by the poor readability rate (46.8%) – a criterion evaluated only by dwellers of Manguinhos. For this reason, it may not be among the preferences of the Manguinhos communities’ residents, despite having obtained the best scores in Accuracy and Interactivity. Therefore, a disparity was identified between the results shown in Table 1 and ranking of Manguinhos’ LAN Houses Computers.

Table 2 was constructed to compare these two results. In the first column to the left is the rank obtained by each site or page in the information quality evaluation. The name of the page or site appears in the next column. In the third column is the rank of each site or page in the search via Google in the computers of the Community of Manguinhos, followed by those who received the HON seal.

Ranked first in the preference of the Community of Manguinhos was 10^th in the evaluation carried out. This is the dengue page of AjaBrasil website. It topped the list of the most visited websites in Manguinhos. This adherence may be explained by the fact that it obtained the best Readability rate of this study (78.4%, Table 1). The fact that the website most accessed by dwellers of Manguinhos is in tenth place justifies the need for online information quality evaluation initiatives like this one.

Another aspect deserves attention. Of the five sites with the HON seal, only one ranked among the top three, namely, the Criasaude website analyzed previously. Let us take a closer look at another site certified by HON: The dengue page of saudeemmovimento.com.br. It ranked 12^th in the evaluation, despite having the HON seal. Its page states that this website has been submitting to the criteria of this organization since July 2001. The last evaluation occurred in July 2012 and its validity expires in July 2014. This seal was conferred because the site complied with the eight “Principles of the HON Code”, namely: responsibility for the site; complementarity of information; privacy; source of information; justification of the benefits of the information provided; transparency; financial interests and advertising policy.

It is clear that criteria used by HON to provide a quality seal are very different from those used in the initiative evaluated in this paper. HON uses criteria that focus on the so-called Technical aspect for the evaluation performed. The other criteria are not considered by the Swiss agency. It also does not clarify how this assessment occurred. This paper reveals the difference between the evaluation carried out and that developed by HON. We suggest that there should be more initiatives such as those developed by the LaISS that do not confine themselves to criteria used by the HON agency.

The Ministry of Health’s website that addresses dengue deserves some comments. In the evaluation performed, this page ranked seventh. This position may be justified by the fact that this page has obtained the lowest Readability compliance rate (28.4%) among all 20 sites or pages evaluated (Table 1). Regarding Accuracy, it obtained the second best compliance rate (62.8%, Table 1). This score does not necessarily indicate that there is technically wrong information but that the information provided is far from desirable in order to disseminate essential concepts such as those related to the identification of the warning signs and the understanding that dengue can be serious and even lead to death, even without the occurrence of bleeding. Thus, the score obtained by the page of the Ministry of Health in this evaluation indicates that it was done in a similar way in all the sites and pages, regardless of their institutional linkage.

Final considerations

LaISS’ work shows some important contributions worth highlighting. Firstly, it should be emphasized that this experience built two products – the criteria and the process of evaluation of information quality in health websites. They are not exactly the same as those proposed by the dominant evaluations, such as the non-governmental organization HON, studies analyzed by Eysenbach et al.¹⁰10. Eysenbach G, Powell J, Kuss O, Sá E. Empirical studies assessing the quality of health information for consumers on the World Wide Web: a systematic review. JAMA 2002; 287(20):2691-2700. and works developed in Brazil. The criteria and the process were shown and analyzed in this paper. Secondly, we highlight the role assigned to users and professionals of the Unified Health System (SUS) as evaluators of information quality. This experience introduced an incremental innovation. In the experiment realized by LaISS, citizens who live in different types of vulnerability conditions, including access to digital media, became subjects of the process.

Assessing the quality of information is not an end in itself. Literature suggests that information plays an important role in health promotion and disease prevention¹³13. Conti MA, Jardim AP, Hearst N, Cordás TA, Tavares H, Abreu CN. Avaliação da equivalência semântica e consistência interna de uma versão em português do Internet Addiction Test (IAT). Rev Psiq Clin 2012; 39(3):106-110.,¹⁴14. Cubas MR, Felchner PCZ. Análise das fontes de informação sobre os autoexames da mama disponíveis na Internet. Cien Saude Colet 2012; 17(4):965-970.,¹⁶16. Gondim APS, Falcão CB. Avaliação das farmácias virtuais brasileiras. Rev Saude Publica 2007; 41(2):297-300.,²²22. Moretti FA, Heymann RE, Marvulle V, Pollak DF, Riera R. Avaliação do nível de conhecimento sobre fibromialgia entre usuários da internet. Rev Bras Reumatol 2011; 51(1):13-19.,²⁶26. Silva LVER, Mello Junior JF, Mion O. Avaliação das informações sobre rinite alérgica em sites brasileiros na rede mundial de computadores (Internet). Rev Bras Otorrinolaringol 2005; 71(5):590-597.,²⁹29. Souza CLN, Luz ZP, Rabello A. Análise da informação sobre a leishmaniose visceral disponível em portais brasileiros da rede mundial de computadores: internet. Rev Soc Bras Med Trop. 2008; 41(4):352-357.,³¹31. Pithon M, Santos E. Information available on the internet about pain after orthognathic surgery: A careful review. Dental Press J. Orthod. 2014; 19(6):86-92.. Garbin et al.⁵5. Garbin HBR, Guilam MCR, Pereira Neto AF. Internet na promoção da saúde: um instrumento para o desenvolvimento de habilidades pessoais e sociais. Physis 2012; 22(1):347-363. affirm that the Internet can be a great ally in the construction of health promotion projects, especially in the development of personal skills. However, if it is not understood and not sorted correctly and updated, its appropriation by the user can be difficult.

With the review of this experiment and the results achieved, we intend to set a debate over the importance of websites certification for the Disease Prevention and Health Promotion⁵5. Garbin HBR, Guilam MCR, Pereira Neto AF. Internet na promoção da saúde: um instrumento para o desenvolvimento de habilidades pessoais e sociais. Physis 2012; 22(1):347-363.. The certification of websites through quality seal seems to be an option to inform both the general public and health professionals about the quality of the information available in a given site¹⁵15. Del Giglio A, Abdala B, Ogawa C, Amado D, Carter D, Gomieiro F, Salama F, Shiroma M, Del Giglio A. Qualidade da informação da internet disponível para pacientes em páginas em português. Rev Assoc Med Bras 2012; 58(6):645-649..

Finally, we emphasize that the results of this project can participate critically and creatively in the international debate on health websites’ evaluation criteria.

References

Publication Dates

History