ABSTRACT

Information and communication technologies have spread worldwide and are increasingly present in almost every aspect of human life, including health. The issue of information quality has assumed great importance amid the widespread dissemination of false knowledge, especially in a pandemic. We evaluated the quality of information on Covid-19 websites of four Health Secretariats in the state of Mato Grosso do Sul, Brazil, from July to August 2020. This participatory evaluation employed five criteria, following the international literature, namely: Technical, Interactivity, Comprehensiveness, Readability, and Accuracy, subdivided into 46 indicators. The results of the evaluated websites point to a low level of compliance with the indicators and criteria adopted and fail to disclose the primary scientific evidence on the topic, available on the Ministry of Health’s website. The high circulation of fake news has marked the Covid-19 pandemic. In this context, Health Secretariats’ pages should display quality and intelligible information about the disease. Only then will they offer reliable informational content based on scientific evidence, contributing to the fight against fake news and its adverse impacts.

KEYWORDS
Internet; Evaluation studies as topic; Coronavirus infections; Online access to information; Quality of health information

Introduction

Since the end of the 20^th century, Information and Communication Technologies (ICT) disseminated through the internet are increasingly blending into almost every aspect of human life. Their global popularization has contributed to changes in practices and actions in different fields¹1 Rice R, Yates S, Blejmar J. Introduction to the Oxford Handbook of digital technology and society: terms, domains, and themes. In: Yates S, Rice R, organizadores. The Oxford Handbook of digital technology and society. Oxford University Press: New York; 2020. p. 3-35.. The internet provides countless sources of information. Furthermore, individuals can produce information by organizing their website, blog, page, or profile on social media platforms. Many people can interact with each other through different online tools and establish social networks, even if they are geographically distant²2 Pereira Neto A, Lima J, Barbosa L, et al. Internet, expert patient e empoderamento: perfis de atuação em comunidades virtuais de renais crônicos. In: Pereira Neto A, Flynn M, organizadores. Internet e saúde no Brasil: desafios e tendências. São Paulo: Cultura Acadêmica; 2021. p. 145-175..

Nowadays, different public and private organizations or individuals produce information on topics linked somehow to health-disease issues. These data are often insufficient, unsatisfactory, obsolete, incorrect, or incomprehensible. Thus, they can put the health of citizens and society at risk. In this context, assessing the quality of health information available online started to assume a relevant role globally³3 Pereira Neto A, Paolucci R. Avaliação da qualidade da informação de saúde na internet: análise das iniciativas brasileiras. In: Pereira Neto A, Flynn M, organizadores. Internet e saúde no Brasil: desafios e tendências. São Paulo: Cultura Acadêmica; 2021. p. 257-291..

In the original term, the current global setting is significantly impacted by the Covid-19 pandemic – or Coronavirus Disease 2019. As of April 2021, more than 132 million confirmed Covid-19 cases were accounted for, with more than 2.8 million deaths from the disease worldwide⁴4 World Health Organization. Coronavirus disease (COVID-19) pandemic. Genebra: WHO; 2020 [acesso em 2021 abr 9]. Disponível em: https://www.who.int/emergencies/diseases/novel-coronavirus-2019.
https://www.who.int/emergencies/diseases... . More than 13 million cases were confirmed in Brazil as of the beginning of that same year, totaling approximately 337 thousand deaths⁵5 Fundação Oswaldo Cruz. Monitora Covid-19. Rio de Janeiro: Fundação Oswaldo Cruz; 2020. [acesso em 2021 abr 9]. Disponível em: https://bigdata-covid19.icict.fiocruz.br/.
https://bigdata-covid19.icict.fiocruz.br... . The emergence and dissemination of Covid-19 produced a series of impacts in different spheres of society, directly or indirectly affecting millions of people around the planet⁶6 Sharifi A, Khavarian-Garmsir A. The COVID-19 pandemic: Impacts on cities and major lessons for urban planning, design, and management. Sci Total Environ. 2020; (749):1-14..

Countless challenges resulting from the pandemic have affected public health, such as adopting several preventive measures, such as the use of masks and alcohol gel and social distancing. The increase in cases sometimes led to the collapse of health systems, particularly in emergency services and hospital intensive care centers⁷7 Santos RC, Carvalho TA, Souza Neto IF, et al. Urgency and emergency in times of covid-19 – an integrative literature review. Res., Soc. Dev. 2021; 10(3):e9110313027.. The health professionals providing care often suffer from anxiety and depression due to exhausting work⁸8 Silva D, Cobucci RN, Soares-Rachetti VP, et al. Prevalência de ansiedade em profissionais da saúde em tempos de covid-19: revisão sistemática com metanálise. Ciênc. Saúde Colet. 2021; 26(2): 693-710.. We should stress that this pandemic mainly affects the socially vulnerable⁹9 Diniz MC, Martins M, Xavier K, et al. Crise global coronavírus: monitoramento e impactos. Cadernos de Prospecção. 2020; 13(2):359-377..

Evidence-based online health information can, for example, reduce the use of health services¹⁰10 Spoelman WA, Bonten TN, Waal MW, et al. Effect of an evidence-based website on healthcare usage: an interrupted time-series study. BMJ Open. 2016; 6(11).. According to Lagan, Sinclair, and Kernohan¹¹11 Lagan BM, Sinclair M, Kernohan WG. What Is the Impact of the Internet on Decision-Making in Pregnancy? A Global Study. Birth. 2011; 38(4):336-345., they can have “a visible impact on women’s decision-making concerning all aspects of their pregnancy”¹¹11 Lagan BM, Sinclair M, Kernohan WG. What Is the Impact of the Internet on Decision-Making in Pregnancy? A Global Study. Birth. 2011; 38(4):336-345.⁽³³⁶⁾. Thus, quality online information from a reliable, up-to-date, and easily understandable source can play a crucial role in self-care practices, service costs, disease prevention, and health promotion¹¹11 Lagan BM, Sinclair M, Kernohan WG. What Is the Impact of the Internet on Decision-Making in Pregnancy? A Global Study. Birth. 2011; 38(4):336-345.,¹²12 Garbin H, Guilam M, Pereira Neto A. Internet na promoção da saúde: um instrumento para o desenvolvimento de habilidades pessoais e sociais. Physis. 2012; 22(1):347-363.. With this information at hand, in the case of Covid-19, citizens will know and possibly adopt the preventive measures recommended by science and where and when to perform diagnostic tests, and receive possible and necessary medical care¹³13 Lima DL, Lopes M, Brito AM. Social media: friend or foe in the COVID-19 pandemic?. Clinics. 2020; (75):e1953..

In the current context, digital media, characterized by a global, instantaneous, and capillary presence, have become crucial and singled out the current pandemic from those that existed until then¹³13 Lima DL, Lopes M, Brito AM. Social media: friend or foe in the COVID-19 pandemic?. Clinics. 2020; (75):e1953.,¹⁴14 Gonzalez-Padilla DA, Tortolero-Blanco L. Social media influence in the COVID-19 Pandemic. Int. Braz. J. Urol. 2020; (46):120-124.. Unlike other historical moments, individuals can have immediate and remote access to updated news about the evolution of the pandemic. In an online survey of the most accessed information sources about Covid-19, Ho, Chen, and Yen¹⁵15 Ho H, Chen Y, Yen C. Different impacts of COVID-19-related information sources on public worry: An online survey through social media. Internet Intervs. 2020; (22):1-6. concluded that:

While it has escalated, the current pandemic has not inaugurated the search for online health information. In recent decades, the number of individuals using ICT to learn more about their health condition has increased¹⁶16 Barbosa L. Comunicação e informação sobre câncer de mama: um estudo comparativo entre um ambiente físico e outro on-line. [tese]. Rio de Janeiro: Fundação Oswaldo Cruz; 2021.. This practice has been driven and is becoming strategic among patients suspected or diagnosed with Covid-19 due to the context of uncertainty produced by the pandemic, characterized by “devastating losses of lives, physical morbidity, mental anguish, and general distress”¹⁷17 Fiedorowicz JG. New challenges in the COVID-19 pandemic. J. Psyc. Research. 2020; (133):110123. (2)2 Pereira Neto A, Lima J, Barbosa L, et al. Internet, expert patient e empoderamento: perfis de atuação em comunidades virtuais de renais crônicos. In: Pereira Neto A, Flynn M, organizadores. Internet e saúde no Brasil: desafios e tendências. São Paulo: Cultura Acadêmica; 2021. p. 145-175..

False news about Covid-19 proliferates before the growing demand for information on the internet. These are fake news in different formats, which usually have an affirmative text. This characteristic leads many people, who typically do not check the information, to believe and share such content¹⁸18 Pereira Neto A, Barbosa L, Flynn M. Prefácio da edição brasileira – Há décadas em que nada acontece. Há semanas em que décadas acontecem. In: Pereira Neto A, Flynn M, organizadores. Internet e saúde no Brasil: desafios e tendências. São Paulo: Cultura Acadêmica; 2021. p. 7-18..

Pereira Neto, Barbosa, and Flynn¹⁸18 Pereira Neto A, Barbosa L, Flynn M. Prefácio da edição brasileira – Há décadas em que nada acontece. Há semanas em que décadas acontecem. In: Pereira Neto A, Flynn M, organizadores. Internet e saúde no Brasil: desafios e tendências. São Paulo: Cultura Acadêmica; 2021. p. 7-18. warn that cases of media misinformation are not new. On the contrary, they have been present since the first writing systems, mainly motivated by financial or ideological reasons, especially after a media concentration in a few large companies. However, ICT facilitated the creation and dissemination of imprecise and misleading news, which could partly explain the dimension of misinformation in recent years.

This trend has increased the possibility of producing fake news about the pandemic. Some studies were recently published evaluating the characteristics and consequences of fake news in the case of Covid-19. Among them, the study by Vasconcelos-Silva and Castiel¹⁹19 Vasconcellos-Silva PR, Castiel LD. COVID-19, as fake news e o sono da razão comunicativa gerando monstros: a narrativa dos riscos e os riscos das narrativas. Cad. Saúde Pública. 2020; 36(7):e00101920. stands out, in which the authors claim that fake news reveals itself as

Galhardi et al.²⁰20 Galhardi CP, Freire N, Minayo M, et al. Fato ou Fake? Uma análise da desinformação frente à pandemia da Covid-19 no Brasil. Ciênc. Saúde Colet. 2020; (25):4201-4210. admit that

None of these papers devote their attention to the reaction and resistance to this trend or present or analyze the possible and available alternatives to address this issue. One of them can be translated as fact-checking²¹21 Fonseca VPS, Freitas F, Gehrke M, et al. Jornalismo guiado por dados como ferramenta de fact-checking: uma experiência laboratorial. Comum. Inov. 2018; (19):67-83.. These are information agencies dedicated to fact-checking. They are currently accredited by the International Fact-Checking Network (IFCN) under the same code of principles. The IFCN is a network that aggregates fact-checking initiatives worldwide. The National Network to Combat Disinformation (RNCD) was recently created in Brazil. It gathers, on a single platform, projects, initiatives, research groups, observatories, and agencies across the country that monitor, identify, denounce and combat fake news²¹21 Fonseca VPS, Freitas F, Gehrke M, et al. Jornalismo guiado por dados como ferramenta de fact-checking: uma experiência laboratorial. Comum. Inov. 2018; (19):67-83..

An alternative reaction is associated with the practice of literacy-based education²²22 Souza N. eHealth literacy entre jovens: estudo exploratório sobre o papel das condições socioeconômicas no uso da informação sobre saúde na Internet. [dissertação]. Rio de Janeiro: Fundação Oswaldo Cruz; 2020.. Sousa et al.²³23 Sousa RA, Moreira K, Farinelli M, et al. Literacia Digital e interfaces com a saúde: uma revisão integrativa. In: Pereira Neto A, Flynn M, organizadores. Internet e saúde no Brasil: desafios e tendências. São Paulo: Cultura Acadêmica; 2021. p. 421-443. admit that digital literacy is a modality of informational competence that allows users to have less difficulty and more skills in searching for and distinguishing information and knowing how to apply them to improve their health and quality of life.

This paper presents and discusses another way to react to fake news. This is the assessment of the quality of information available on health websites³3 Pereira Neto A, Paolucci R. Avaliação da qualidade da informação de saúde na internet: análise das iniciativas brasileiras. In: Pereira Neto A, Flynn M, organizadores. Internet e saúde no Brasil: desafios e tendências. São Paulo: Cultura Acadêmica; 2021. p. 257-291.. It specifically describes and analyzes the result of the quality assessment of four public websites on Covid-19: one linked to the Mato Grosso do Sul State Health Secretariat and three municipal health secretariats from the same federation, namely, Campo Grande, Corumbá, and Dourados.

The research that served as the basis for the preparation of this paper was carried out by a group of professors and students during the online Special Topics Discipline: Evaluation of the Quality of Information on Infectious and Parasitic Diseases Websites, of the Infectious and Parasitic Diseases (PPGDIP) Postgraduate Program of the Federal University of Mato Grosso do Sul (UFMS), from July 8 to August 19, 2020.

The methodology, criteria, and indicators used and the results obtained in evaluating the quality of information on Covid-19 websites of Brazilian government institutions will be presented below.

Material and methods

Criteria built from two systematic reviews were employed to construct the criteria and indicators used to assess the quality of information on Covid-19 of Brazilian government websites. The first identified and analyzed 79 papers published up to 2001²⁴24 Eysenbach G, Powell J, Kuss O, et al. Empirical studies assessing the quality of health information for consumers on the world wide web: a systematic review. Jama. 2002; 287(20):2691-2700.37.. The second was based on 279 papers published from 2001 to 2014, available in international bibliographic databases²⁵25 Paolucci R, Pereira Neto A. Methods for evaluating the quality of information on health websites: Systematic Review (2001-2014). Latin Am J Develop. 2021; 3(3):994-1056..

Both reviews concluded that most studies published in the world used five criteria to assess the quality of information on health websites: technical, interactivity, comprehensiveness, readability, and accuracy. The technical criteria aim to identify whether or not there is a person responsible for the site or the information offered has a source. It also checks, for example, the date of creation and update of this virtual environment. The interactivity criterion is available to verify whether the site provides means of communication between users and content producers, whether it has a ‘Contact Us’ channel, it offers access to social networks, and, for example, search tools and accessibility alternatives for people with disabilities²⁶26 Brasil. Lei nº 13.146, de 6 de julho de 2015. Estatuto da Pessoa com Deficiência. Diário Oficial da União. 7 Jul 2015. are available. The readability criterion assesses whether the information is arranged to ensure the understanding of lay users. The comprehensiveness criterion looks at whether the site offers the set of information relevant to a particular topic or health problem. If it is the case of evaluating the website’s information on communicable disease, the sense of comprehensiveness includes information on prevention, symptoms, diagnosis, and treatment. The accuracy criterion measures the degree of agreement between the information offered on the website and the best evidence available or accepted by science. The study on the assessment of information quality on health websites presented in this work adopted the five criteria mentioned above.

Each criterion consisted of several indicators: specific evaluative questions, mostly with dichotomous structured answers (Yes or No). In the case of technical criteria, a question asked, for example, whether ‘it was updated in the last week’. Regarding comprehensiveness, a question asked whether ‘there is information about symptoms’. Regarding interactivity, a question asked whether the site ‘has any social network’. Concerning readability, one of the questions aimed to know whether the evaluator ‘had difficulty understanding the information on prevention’. We used information about Covid-19 of the official Ministry of Health (MS) website as a reference (https://coronavirus.saude.gov.br/) to build the indicators of the accuracy criterion. Only the accuracy criterion indicators allowed the ‘Yes’, ‘No’, or ‘Partially’ options as correct answers. They were not formulated as a question but as an affirmative sentence. Evaluators should verify whether the information mentioned in the indicator was fully or partially present or absent in the assessed website. One of the indicators used was: ‘A preventive measure is washing hands frequently with soap and water or sanitizing with 70% alcohol gel’. Then, the following question was included: ‘Is this information available on this website?’. The answers could be: ‘Yes’, ‘No’ or ‘Partially’. The technical and readability criteria consisted of six indicators each; comprehensiveness and interactivity for seven indicators each; and accuracy, 20 indicators. The five criteria adopted totaled 46 indicators. The positive response to each of these indicators reveals the level of compliance between the offered and ideal information. These 46 indicators make up the ‘gold standard’; they are part of the information set that an excellent site should have.

Box 1 presents the underlying indicators of each criterion. A large number of accuracy indicators is noteworthy (20), which is justified by the inclusion of indicators related to prevention (4), symptoms (5), transmission (4), diagnosis (3), and treatment (4) in this criterion.

The establishment of the indicators used in the evaluation counted on 14 graduate students (12 masters and two doctoral students) from PPGDIP at UFMS, four men and 12 women. All graduated in biosciences. The two professors responsible for the discipline also participated in this activity.

The assessment was carried out during the Covid-19 pandemic. For this reason, the online completion of the evaluation tool was chosen as an alternative concerning social distancing measures to reduce the transmission of this disease. Thus, a questionnaire was produced using the Google Forms^® tool to measure the existence of information on each evaluated website. The questionnaire included a space for identifying the evaluator and the evaluated website and a set of 46 indicators. The results were stored in a spreadsheet for further analysis.

This same group carried out the evaluation process of the four websites with other external guests. Each student invited a friend or relative to make up the group of evaluators. Regarding the external guests, 13 were graduates in different areas of knowledge, and one was an undergraduate. These external guests did not undergo specific training for handling the tool and assessing. In total, we had 30 evaluators equally distributed gender-wise.

This initiative aimed to carry out a participatory evaluation of the selected websites, including people representing potential users of the information made available by the websites. Thus, we intended to address one of the shortcomings identified by Paolucci and Pereira Neto²⁵25 Paolucci R, Pereira Neto A. Methods for evaluating the quality of information on health websites: Systematic Review (2001-2014). Latin Am J Develop. 2021; 3(3):994-1056.. In their systematic review, the authors found that only 2.5% of studies assessing the quality of information on health websites involved users in the evaluation process²⁵25 Paolucci R, Pereira Neto A. Methods for evaluating the quality of information on health websites: Systematic Review (2001-2014). Latin Am J Develop. 2021; 3(3):994-1056..

The websites of the State Health Secretariat of Mato Grosso do Sul (http://www.coronavirus.ms.gov.br), and of three Municipal Health Secretariats in that federation unit were selected for evaluation: Campo Grande (http://www.campogrande.ms.gov.br/sesau/covid19/), Corumbá (http://sisms.corumba.ms.gov.br/boletim/) and Dourados (http://corona.dourados.ms.gov.br). This choice was intended to present to these bodies a report containing recommendations to improve the quality of the information provided by their online environments.

Results and discussion

Table 1 presents the results obtained in assessing the quality of information on Covid-19 from four Brazilian government websites.

Such results allow us to obtain some findings. The first evidence is related to the low quality of information identified in the four websites evaluated. Only Dourados and Mato Grosso do Sul managed to reach around 60% compliance. The others averaged below 42%.

The positive highlights of the evaluation were the interactivity and readability criteria. In the first case, the Campo Grande, Dourados and Mato Grosso do Sul websites averaged around 70%. On the Dourados and Mato Grosso do Sul websites, 80% of the evaluators identified a search tool and considered it easy to navigate on these websites. The Dourados and Mato Grosso do Sul websites obtained more than 70% compliance concerning the readability criterion. The Dourados website, for example, contains videos that facilitate the understanding of information about this disease. These results indicate that the websites of Campo Grande, Dourados, and Mato Grosso do Sul provide tools to facilitate interaction between the user and the information producer. However, it is essential to emphasize that the advantages of interactive and readable online information are reduced if the content is inaccurate, updated, and aligned with current scientific evidence²⁴24 Eysenbach G, Powell J, Kuss O, et al. Empirical studies assessing the quality of health information for consumers on the world wide web: a systematic review. Jama. 2002; 287(20):2691-2700.37.,²⁵25 Paolucci R, Pereira Neto A. Methods for evaluating the quality of information on health websites: Systematic Review (2001-2014). Latin Am J Develop. 2021; 3(3):994-1056.. In the accuracy criterion, all websites evaluated averaged below 42%. This rate indicates that less than half of the content available on each page evaluated complied with the information available on the official MS website dedicated to the Coronavirus.

The percentage of evaluators who are unable to find the essential information about the disease can be identified in table 2. The high rate of this percentage indicates that this information is unavailable on this site. As can be seen, in none of the cases this percentage reached 100%, which can be explained because some evaluators did not assess carefully. Also, external guests were not submitted to any training.

The results presented in table 2 seem troubling. Regarding prevention, for example, mask use is one of the preventive measures recognized and recommended by the WHO to curb the spread of Covid-19²⁷27 World Health Organization. Mask use in the context of Covid-19. Genebra: WHO; 2020 [acesso em 2021 abr 9]. Disponível em: https://www.who.int/publications/i/item/advice-on-the-use-of-masks--in-the-community-during-home-care-and-in-he-althcare-settings-in-the-context-of-the-novel-co-ronavirus-(2019-ncov)-outbreak.
https://www.who.int/publications/i/item/... . However, this information was not available or was difficult to see on the websites of Corumbá, Campo Grande, and the State Secretariat of Mato Grosso do Sul. In the case of these virtual environments, attention was focused on providing epidemiological information on the evolution of the pandemic in these regions of the state instead of providing information on disease prevention. On the Corumbá website, there is no information related to the necessary frequent hand hygiene or the use of 70% alcohol gel.

We noticed that most of the essential information regarding transmission is missing. Noteworthy is that the Campo Grande and Corumbá websites fail to report that the transmission occurs through contact with surfaces such as cell phones, tables, computers, and doorknobs that may be contaminated. None of the four websites report that transmission can occur through the household contacts of people with suspected or confirmed Covid-19.

We also observed that none of the websites evaluated contained minimum necessary diagnostic information. Almost all evaluators did not identify information regarding the diagnostic test, the need for a medical visit to assess the possibility of having the disease, and the performance of tomography in the case of severe respiratory symptoms.

Disease treatment indicators also showed low compliance in all websites evaluated. Almost all evaluators found that information on this dimension was absent or hardly seen in the four digital environments evaluated.

Some aspects deserve a brief discussion given the results presented above. One of them is related to the methodological procedure adopted in the research that became this paper. As mentioned, the research participants carried out the readability assessment of the four websites.

Eysenbach et al.²⁴24 Eysenbach G, Powell J, Kuss O, et al. Empirical studies assessing the quality of health information for consumers on the world wide web: a systematic review. Jama. 2002; 287(20):2691-2700.37. had already shown that “none of the studies conducted comprehension tests with actual consumers or used judgments of literacy experts”²⁴24 Eysenbach G, Powell J, Kuss O, et al. Empirical studies assessing the quality of health information for consumers on the world wide web: a systematic review. Jama. 2002; 287(20):2691-2700.37.⁽²⁶⁹⁵⁾. Also, they emphasized that the studies used technological resources to assess the readability of information. These authors argue that such formulas

In his systematic review, Paolucci²⁵25 Paolucci R, Pereira Neto A. Methods for evaluating the quality of information on health websites: Systematic Review (2001-2014). Latin Am J Develop. 2021; 3(3):994-1056. concluded that the readability of a website was still performed using formulas such as Flesh-Kincaid Grade Level, Discern tool, and Flesch Reading Ease. Thus, this paper dialogues with the international literature and proposes involving the end-user in the readability assessment.

Another aspect related to the methodological dimension refers to the criteria and indicators used. In this case, the evaluation model adopted in this work differs, in many aspects, from two others present in the international academic scene. One of them is the Health On Net Foundation (HON). Another is the Discern Questionnaire (DQ).

HON (https://www.hon.ch/en/) is a nongovernmental organization created in 1995, based in Switzerland. It evaluates health pages based on eight principles, namely: ‘authorship’, which is related to the presence of the source of the information offered; ‘complementarity’, which is concerned with the fact that the information available does not replace the doctor-patient relationship; ‘confidentiality’, which assesses whether the information provided by patients has not been made public; ‘attribution and update’, which seeks to verify whether the last date of insertion of the information on the website is included; ‘guarantee’, refers to the fact that the digital environment reveals that it has no commercial interest; ‘transparency’ is concerned with users’ access to websites through the several internal communication tools (contact us, e-mail, Facebook); ‘sponsorship transparency’ is dedicated to identifying whether the website indicates its sources of funding; and ‘honesty and transparency about advertising’²⁴24 Eysenbach G, Powell J, Kuss O, et al. Empirical studies assessing the quality of health information for consumers on the world wide web: a systematic review. Jama. 2002; 287(20):2691-2700.37.,²⁵25 Paolucci R, Pereira Neto A. Methods for evaluating the quality of information on health websites: Systematic Review (2001-2014). Latin Am J Develop. 2021; 3(3):994-1056.. While some of the criteria adopted in this paper coincide with the principles adopted by the HON, no reference is made to the accuracy and legibility of information among the principles adopted by the Swiss institution.

The DQ (http://www.discern.org.uk/index.php) is a project founded in 1996-1997 by the British Library and the National Health Services (NHS). This questionnaire provides users with means to assess the quality of written information about treatment options for a health problem. It is divided into three sections. The first is concerned with information reliability. The second leads users to question the quality of information about treatment options. The third guides the user to carry out a general classification of the publication. One of the questions, for example, asks whether the website ‘describes how each treatment works’. In this case, the website may describe this aspect in an incorrect, obsolete, or incomprehensible way. The DQ²⁴24 Eysenbach G, Powell J, Kuss O, et al. Empirical studies assessing the quality of health information for consumers on the world wide web: a systematic review. Jama. 2002; 287(20):2691-2700.37.,²⁵25 Paolucci R, Pereira Neto A. Methods for evaluating the quality of information on health websites: Systematic Review (2001-2014). Latin Am J Develop. 2021; 3(3):994-1056. does not consider this aspect.

For this reason, the criteria and evaluation process adopted in the research that resulted in this paper may be complete. Another point that deserves to be highlighted in the discussion of this paper can be highlighted when comparing the results obtained in this research with others found in previous studies. The problem of low informational quality on institutional websites linked to health secretariats had already been observed in two previous studies: one evaluating the information on dengue websites²⁸28 Pereira Neto AF, Paolucci R, Daumas RP, et al. Avaliação participativa da qualidade da informação de saúde na internet: o caso de sites de dengue. Ciênc. Saúde Colet. 2017; 22(6):1955-1968., and another checking the quality of information on tuberculosis websites²⁹29 Paolucci R, Pereira Neto A, Luzia R. Avaliação da qualidade da informação em sites de tuberculose: análise de uma experiência participativa. Saúde debate. 2017; 41(esp):84-100..

Pereira Neto et al.²⁸28 Pereira Neto AF, Paolucci R, Daumas RP, et al. Avaliação participativa da qualidade da informação de saúde na internet: o caso de sites de dengue. Ciênc. Saúde Colet. 2017; 22(6):1955-1968. assessed the quality of information on 20 websites about dengue, using the same criteria and following a participatory perspective similar to that carried out in the research that resulted in this paper. Seven were linked to Brazilian government agencies, namely: The Ministry of Health (MS); the Oswaldo Cruz Institute (IOC), a unit of the Oswaldo Cruz Foundation; the Dengue Network of the Oswaldo Cruz Foundation (RD); the State Health Secretariats of Paraná (SES-PA), Rio de Janeiro (SES-RJ), and Minas Gerais (SES-MG); and the Recife Municipal Health Secretariat (SMS-RE).

Paolucci, Pereira Neto, and Luzia²⁹29 Paolucci R, Pereira Neto A, Luzia R. Avaliação da qualidade da informação em sites de tuberculose: análise de uma experiência participativa. Saúde debate. 2017; 41(esp):84-100. assessed the quality of information on 12 websites about tuberculosis, using the criteria and perspective of participant research adopted in the study presented here. Six of the websites evaluated²⁹29 Paolucci R, Pereira Neto A, Luzia R. Avaliação da qualidade da informação em sites de tuberculose: análise de uma experiência participativa. Saúde debate. 2017; 41(esp):84-100. are linked to Brazilian government agencies, namely: National Tuberculosis Control Program; MS; Tropical Medicine Foundation; Amazonas (AM) state government; SES-PA and São Paulo (SES-SP); and Municipal Health Secretariats of Porto Alegre (SMS-PA) and Rio de Janeiro (SMS-RJ). The results obtained in the evaluation of information on Brazilian government websites about dengue²⁸28 Pereira Neto AF, Paolucci R, Daumas RP, et al. Avaliação participativa da qualidade da informação de saúde na internet: o caso de sites de dengue. Ciênc. Saúde Colet. 2017; 22(6):1955-1968. were as follows:

The results obtained in the evaluation of information on Brazilian government websites on tuberculosis²⁹29 Paolucci R, Pereira Neto A, Luzia R. Avaliação da qualidade da informação em sites de tuberculose: análise de uma experiência participativa. Saúde debate. 2017; 41(esp):84-100. were as follows:

The compliance rates identified in evaluating Brazilian government websites about dengue are noteworthy. Only one of them had an overall average of over 60% compliance. Regarding accuracy, only two had more than 60% compliance²⁹29 Paolucci R, Pereira Neto A, Luzia R. Avaliação da qualidade da informação em sites de tuberculose: análise de uma experiência participativa. Saúde debate. 2017; 41(esp):84-100.. Regarding tuberculosis, Paolucci, Pereira Neto, and Luzia²⁹29 Paolucci R, Pereira Neto A, Luzia R. Avaliação da qualidade da informação em sites de tuberculose: análise de uma experiência participativa. Saúde debate. 2017; 41(esp):84-100. found that the highest mean of compliance was around 60%. In terms of accuracy, only one achieved more than 70% compliance.

The compliance percentages identified in the study that resulted in this paper reveal that the quality of information about Covid-19 in the Brazilian government websites evaluated is poor. In general terms, this result is close to that obtained by Pereira

Neto et al.²⁸28 Pereira Neto AF, Paolucci R, Daumas RP, et al. Avaliação participativa da qualidade da informação de saúde na internet: o caso de sites de dengue. Ciênc. Saúde Colet. 2017; 22(6):1955-1968. and Paolucci, Pereira Neto, and Luzia²⁹29 Paolucci R, Pereira Neto A, Luzia R. Avaliação da qualidade da informação em sites de tuberculose: análise de uma experiência participativa. Saúde debate. 2017; 41(esp):84-100. in the evaluations of dengue and tuberculosis websites, respectively. The data obtained in the study presented here and those available in the literature²⁸28 Pereira Neto AF, Paolucci R, Daumas RP, et al. Avaliação participativa da qualidade da informação de saúde na internet: o caso de sites de dengue. Ciênc. Saúde Colet. 2017; 22(6):1955-1968.,²⁹29 Paolucci R, Pereira Neto A, Luzia R. Avaliação da qualidade da informação em sites de tuberculose: análise de uma experiência participativa. Saúde debate. 2017; 41(esp):84-100. led to affirm that Brazilian government websites must be evaluated and then modified, considering the minimum conditions of interactivity, readability, and accuracy.

Final considerations

Online health information can be an essential resource for increasing self-care, self-management, empowerment, and adherence to treatment. Therefore, it should have quality and must be interactive, understandable, current, reliable, and scientifically proven²⁴24 Eysenbach G, Powell J, Kuss O, et al. Empirical studies assessing the quality of health information for consumers on the world wide web: a systematic review. Jama. 2002; 287(20):2691-2700.37.. We should stress that it did not refer to its subjective dimension when using the word ‘quality’. The researchers in this study believe that this expression, associated with online health information, is related to the perspective of survival and well-being that it can provide to the population, which can be measured by criteria and indicators³⁰30 Paim I, Nehmy RMQ, Guimarães CG. Problematização do conceito ‘qualidade’ da Informação. Perspec. Ci. Inf. 1996; 1(1):111-119..

The Pan American Health Organization has just confirmed the existence of infodemia, that is, “excess information, some accurate and some not, which hinder finding reputable sources and reliable guidance when needed”³¹31 Organização Pan-Americana da Saúde. Entenda a infodemia e a desinformação na luta contra a Covid-19. Brasil: OPAS; 2020. [acesso em 2021 abr 9]. Disponível em: https://iris.paho.org/bitstream/handle/10665.2/52054/Factsheet-Infodemic_por. pdf?sequence=14.
https://iris.paho.org/bitstream/handle/1... (2)2 Pereira Neto A, Lima J, Barbosa L, et al. Internet, expert patient e empoderamento: perfis de atuação em comunidades virtuais de renais crônicos. In: Pereira Neto A, Flynn M, organizadores. Internet e saúde no Brasil: desafios e tendências. São Paulo: Cultura Acadêmica; 2021. p. 145-175..

What has been done to relieve this problem?

Last year, the consortium formed by institutions linked to the Ministry of Science, Technology, Innovation, and Communications (MCTIC) and the MS launched a call to

One of the themes for the submission of proposals was prevention and control. It is subdivided into five lines. One of them predicted:

We an observe an interest from the leading public funding agencies in supporting research on the topic of fake news during the Covid-19 pandemic. This paper revealed another severe issue. Brazilian government websites have shortcomings and fail to meet the minimum quality criteria, with incorrect, obsolete, incomplete, and illegible data on the subject.

The websites evaluated showed that, in general, the level of compliance with the criteria and indicators used was equal to or less than 61%. In the accuracy criterion, the percentages obtained were below 45%. Thus, we can observe that the online environments of the health secretariats dedicated to Covid-19 do not meet the minimum quality criteria.

In a context in which increasingly more people use the internet to search for health information, the content made available, especially on the websites of Brazilian government agencies, should be technical, comprehensive, interactive, readable, and accurate. We should emphasize that, by achieving the indicated compliance, within the quality of information, the website will be more accessible to citizens, increasing trust in the information offered and, above all, contributing to the circulation of information based on scientific and for coping with fake informational content.

The assessment and certification of Brazilian government websites are an essential alternative to combat fake news on health, especially in a pandemic context. Furthermore, public websites should strive to provide updated, correct, understandable, and quality information. Unfortunately, this was not the reality found due to this assessment. If the public website is of quality, Brazilians will have a place to turn to when they have any doubts regarding information about Covid-19. Brazil has no public or private agency developing systematic activities to assess the quality of health information available on the internet.

Meanwhile, in the United States and Europe, government agencies, professional associations, and researchers have been evaluating the information on health websites for more than two decades. The HON and DQ are the most prominent. Wouldn’t it be the case to build a public agency certifying government health websites?

Limits and potentials

It is noteworthy that the study conducted used a small sample of websites for analysis. However, this evaluation proposal based on criteria and indicators built in a participatory way from systematic reviews can be used to evaluate a more significant number of websites from different Brazilian cities and states. Thus, it will be possible to offer a more comprehensive overview of the quality of information about Covid-19 on websites of public bodies nationwide.

Participatory evaluation is a feature that deserves to be highlighted in the experience carried out and transformed in this paper. Participation occurred at all stages of the research and health website certification processes. Participants contributed to the construction and validation of the indicators and the evaluations of the websites. It is noteworthy that most evaluators reside in Mato Grosso do Sul and have higher education. Thus, while it was not a selection condition, it is reasonable to assume they had health literacy and a similar socioeconomic status. This aspect limits the possibility of generalizing the findings. Despite this, this paper evidences some troubling results.

The high circulation of fake news has marked the Covid-19 pandemic. In this context, the webpages of health secretariats should show high quality and readability information about the disease. Only then will they offer reliable, informative content based on scientific evidence, contributing to coping with fake news and its negative impacts.

This paper aims to encourage further research on the subject. In this sense, Paolucci’s doctoral thesis³³33 Paolucci R. Avaliação da qualidade da informação em sites de saúde: indicadores de acurácia baseada em evidência para tuberculose. [tese]. Rio de Janeiro: Fundação Oswaldo Cruz; 2020. presents an innovative perspective that significantly contributes to the academic debate on the subject. It also aims to include the theme in the research agenda of the MS and the Ministry of Science and Technology and can sensitize health policymakers about the problem of quality of information and the consequences of a quality website for the well-being of citizens. Further studies that identify the short and long-term effects that this information can promote, including health costs¹⁰10 Spoelman WA, Bonten TN, Waal MW, et al. Effect of an evidence-based website on healthcare usage: an interrupted time-series study. BMJ Open. 2016; 6(11)., would be very valuable. Studies evaluating the quality of information on diseases prevalent in developing countries are also noteworthy.

In this paper, we sought to highlight the relevance of the problem of the quality of health information available on the internet, particularly on Brazilian government websites on the Covid-19 pandemic. He suggests other possibilities to reduce this problem, such as health literacy and the creation of news checking agencies.

As presented in this work, the topic has been receiving academia’s attention for some time. In this sense, one of the questions raised by this paper is ‘what can be done so that the issue of evaluating the quality of health websites starts to be discussed by society and becomes a relevant point in the research agenda of development agencies?’.

References

Publication Dates

History