Abstract:

This study assessed the consistency of self-reports of risk behavior (overall and within four specific domains: alcohol use, tobacco use, drug use, and sexual activity) in two editions of the Brazilian National School Based Survey of Adolescent Health (PeNSE): 2009 and 2012. The overall proportion of cases with at least one inconsistent response in the two editions was 11.7% (2.7% on the alcohol items, 2.1% for drug use, 4.3% for cigarette use, 3% for sexual activity) and 22.7% (12.8% on alcohol items, 2.5% for drug use, 4.3% for cigarette use, 4.1% for sexual activity), respectively. Such inconsistency was more prevalent among males, delayed students, those who reported having experimented with drugs, and those who did not have a cellphone. Because inconsistent responses were more prevalent among the students who claimed to have engaged in risky activities, removing inconsistent responders affected the estimated prevalence of all risk behaviors in both editions of the survey. This study supports the importance of performing consistency checks of self-report surveys, following the growing body of literature on this topic.

Keywords:
Risk-Taking; Self Report; Surveys; Methodology; Adolescent

Resumo:

O presente estudo avaliou as inconsistências no autorrelato de comportamentos de risco (geral e em quatro domínios específicos: uso de álcool, tabaco, drogas e atividade sexual) em duas edições da Pesquisa Nacional de Saúde do Adolescente (PeNSE): 2009 e 2012. Nas duas edições, a proporção de casos com ao menos uma resposta inconsistente foi de 11,7% (2,7% nos itens sobre consumo de álcool, 2,1% para uso de drogas, 4,3% para uso de tabaco, 3% para atividade sexual) e 22,7% (12,8% nos itens sobre consumo de álcool, 2,5% para uso de drogas, 4,3% para uso de tabaco, 4,1% para atividade sexual), respectivamente. Tal inconsistência foi mais prevalente entre participantes do sexo masculino, estudantes com atraso escolar, participantes que relataram ter experimentado drogas e participantes que não possuíam telefone celular. Dado que as inconsistências foram mais prevalentes entre os estudantes que declararam ter se engajado nos comportamentos de risco, remover os casos com inconsistência afetou as estimativas de prevalência destes comportamentos em ambas as edições da pesquisa. Este estudo ressalta a importância de testes para a checagem da consistência dos dados autorrelato em pesquisas, acompanhando a crescente literatura na área.

Palavras-chave:
Assunção de Riscos; Autorrelato; Inquéritos; Metodologia; Adolescente

Resumen:

Este estudio evaluó las inconsistencias en las conductas de riesgo de auto-reporte (general y cuatro áreas específicas: el alcohol, el tabaco, las drogas y la actividad sexual) en dos ediciones de la Encuesta Nacional de Salud del Adolescente (PeNSE): 2009 y 2012. En dos ediciones, la proporción de casos con al menos una respuesta inconsistente fue 11,7% (2,7% en alcohol, 2,1% en drogas, 4,3% en tabaco, 3% en actividad sexual) y 22,7% (12,8% en alcohol, 2,5% en drogas, 4,3% en tabaco, 4,1% en actividad sexual), respectivamente. Tal inconsistencia era más frecuente entre los participantes masculinos, los alumnos con retraso escolar, los participantes que reportaron haber consumido drogas y participantes probado que no tenían teléfono celular. Dado que las inconsistencias fueron más prevalentes entre los estudiantes que reportaron haber participado en comportamientos de riesgo, eliminar los casos de inconsistencia afectó a las estimaciones de la prevalencia de estas conductas en las dos ediciones de la encuesta. Este estudio pone de relieve la importancia de las pruebas para comprobar la coherencia de los datos de auto-informe sobre la investigación, a raíz de la creciente literatura en la zona.

Palabras-clave:
Asunción de Riesgos; Autoinforme; Encuestas; Metodología; Adolescente

Introduction

The most widely reported measures of youth risk behavior are items from self-report surveys administered in schools ¹1. Brener ND, Billy JO, Grady WR. Assessment of factors affecting the validity of self-reported health-risk behavior among adolescents: evidence from the scientific literature. J Adolesc Health 2003; 33:436-57.. Results from such surveys are commonly used for developing public policy, identifying problem behaviors within a particular school or district, and planning and evaluating prevention activities.

In Brazil, monitoring adolescents’ health and risk factors has been an ongoing priority of the National School Based Survey of Adolescent Health²2. Instituto Brasileiro de Geografia e Estatística. Pesquisa Nacional de Saúde do Escolar (PeNSE), 2009. Brasília: Instituto Brasileiro de Geografia e Estatística; 2009.^,33. Instituto Brasileiro de Geografia e Estatística. Pesquisa Nacional de Saúde do Escolar (PeNSE), 2012. Brasília: Instituto Brasileiro de Geografia e Estatística; 2012., also known as the PeNSE survey (PeNSE - Pesquisa Nacional de Saúde do Escolar). In the past two decades, as concern about youth health in Brazil has risen, so has the reliance on this survey’s data for identifying risks and temporal trends, and for making policy decisions, especially by epidemiologists interested in identifying the prevalence of health risk behaviors and their associated factors ⁴4. Araújo C, Toral N, Silva ACF, Velásquez-Melendez G, Dias AJR. Estado nutricional dos adolescentes e sua relação com variáveis sociodemográficas: Pesquisa Nacional de Saúde do Escolar (PeNSE), 2009. Ciênc Saúde Coletiva 2010; 15 Suppl 2:3077-84.^,55. Freire MCM, Leles CR, Sardinha LMV, Paludetto Junior M, Malta DC, Peres MA. Dor dentária e fatores associados em adolescentes brasileiros: a Pesquisa Nacional de Saúde do Escolar (PeNSE), Brasil, 2009. Cad Saúde Pública 2012; 28 Suppl:S133-45.^,66. Horta RL, Horta BL, Costa AWN, Prado RR, Oliveira-Campos M, Malta DC. Lifetime use of illicit drugs and associated factors among Brazilian schoolchildren, National Adolescent School-based Health Survey (PeNSE 2012). Rev Bras Epidemiol 2014; 17 Suppl 1:31-45.^,77. Malta DC, Andreazzi MAR, Oliveira-Campos M, Andrade SSCA, Sá NNB, Moura L, et al. Trend of the risk and protective factors of chronic diseases in adolescents, National Adolescent School-based Health Survey (PeNSE 2009 and 2012). Rev Bras Epidemiol 2014; 17 Suppl 1:77-91.^,88. Malta DC, Oliveira-Campos M, Prado RR, Andrade SSC, Mello FCM, Dias AJR, et al. Psychoactive substance use, family context and mental health among Brazilian adolescents, National Adolescent School-based Health Survey (PeNSE 2012). Rev Bras Epidemiol 2014; 17 Suppl 1:46-61.^,99. Malta DC, Sardinha LMV, Mendes I, Barreto SM, Giatti L, Castro IRR, et al. Prevalence of risk health behavior among adolescents: results from the 2009 National Adolescent School-based Health Survey (PeNSE). Ciênc Saúde Coletiva 2010; 15 Suppl 2:3009-19.^,1010. Malta DC, Machado IE, Porto DL, da Silva MMA, de Freitas PC, da Costa AWN, et al. Alcohol consumption among Brazilian Adolescents according to the National Adolescent School-based Health Survey (PeNSE 2012). Rev Bras Epidemiol 2014; 17 Suppl 1:203-14.^,1111. Morais Neto OL, Malta DC, Mascarenhas MDM, Duarte EC, Silva MMA, Oliveira KB, et al. Risk factors for road traffic injury among adolescents in Brazil: National Adolescent School-based Health Survey (PeNSE). Ciênc Saúde Coletiva 2010; 15 Suppl 2:3043-52.^,1212. Oliveira-Campos M, Nunes ML, Madeira FC, Santos MG, Bregmann SR, Malta DC, et al. Sexual behavior among Brazilian adolescents, National Adolescent School-based Health Survey (PeNSE 2012). Rev Bras Epidemiol 2014; 17 Suppl 1:116-30.^,1313. Malta DC, Mascarenhas MDM, Porto DL, Duarte EA, Sardinha LM, Barreto SM, et al. Prevalência do consumo de álcool e drogas entre adolescentes: análise dos dados da Pesquisa Nacional de Saúde Escolar. Rev Bras Epidemiol 2011; 14 Suppl 1:136-46.^,1414. Malta DC, Porto DL, Melo FCM. Família e proteção ao uso de tabaco, álcool e drogas em adolescentes, Pesquisa Nacional. Rev Bras Epidemiol 2011; 14 Suppl 1:166-77.^,1515. Malta DC, Silva MAI, Mello FCM, Monteiro RA, Porto DL, Sardinha LMV, et al. Saúde sexual dos adolescentes segundo a Pesquisa Nacional de Saúde dos Escolares. Rev Bras Epidemiol 2011; 14 Suppl 1:147-56.^,1616. Barreto SM, Giatti L, Oliveira-Campos M, Andreazzi MA, Malta DC. Experimentation and use of cigarette and other tobacco products among adolescents in the Brazilian state capitals (PeNSE 2012). Rev Bras Epidemiol 2014; 17 Suppl 1:62-76.. There is a growing body of evidence about the validity of such self-reported survey data, indicating that (especially for surveys with young samples) estimates of the prevalence of various activities can sometimes be dramatically over, or underestimated ¹⁷17. Cross JE, Newman-Gonchar R. Data quality in student risk behavior surveys and administrator training. J Sch Violence 2004; 3:89-108.^,1818. Cornell DG, Loper AB. Assessment of violence and other high-risk behaviors with a school survey. School Psych Rev 1998; 27:317-30.. The relative validity of dietary ¹⁹19. Tavares LF, Castro IRR, Levy RB, Cardoso LO, Passos MD, Brito FSB. Validade relativa de indicadores de práticas alimentares da Pesquisa Nacional de Saúde do Escolar entre adolescentes do Rio de Janeiro, Brasil. Cad Saúde Pública 2014; 30:1029-41. and physical activity indicators ²⁰20. Tavares LF, Castro IR, Cardoso LO, Levy RB, Claro RM, Oliveira AF. Validade de indicadores de atividade física e comportamento sedentário da Pesquisa Nacional de Saúde do Escolar entre adolescentes do Rio de Janeiro, Brasil. Cad Saúde Pública 2014; 30:1861-74. from the PeNSE questionnaire, as well as its sensitivity, specificity, and correct classification rate have been previously studied. Nevertheless, the data concerning adolescent risk behaviors from the PeNSE survey have apparently never been scrutinized for consistency.

The primary purpose of this study was to examine the consistency of self-reports of risk behavior, overall and within four specific domains: alcohol use, tobacco use, drug use, and sexual activity. A second purpose was to compare the two editions of the PeNSE survey, those of 2009 and 2012, with respect to methodology and sociodemographic aspects of the two years’ sample, to document changes in the inconsistency rates between the two and explore possible causes of those changes. A third purpose was to assess how deleting inconsistent, self-contradictory responses would affect prevalence estimates.

Self-reported measures of risk behavior in survey data: quality issues and concerns

Research on the reliability and validity of self-report has been conducted since the 1940s, but for self-reports of risk behavior, the studies did not begin until the 1980s ²¹21. Midanik L. The validity of self-reported alcohol consumption and alcohol problems: a literature review. Br J Addict 1982; 77:357-82.^,2222. Rodgers JL, Billy JO, Udry JR. The rescission of behaviors: inconsistent responses in adolescent sexuality data. Soc Sci Res 1982; 11:280-96.^,2323. Hilton ME. Inconsistent responses to questions about alcohol consumption in specified settings. Am J Drug Alcohol Abuse 1986; 12:403-13. and for epidemiologists, interest began a bit later ²⁴24. Friedenreich CM. Improving long-term recall in epidemiologic studies. Epidemiology 1994; 5:1-4..

Because risk behaviors are often socially disapproved, one concern is that the young respondents may underreport actual behavior, despite the guaranteed anonymity of their responses. As discussed by Barnea et al. ²⁵25. Barnea Z, Rahav G, Teichman M. The reliability and consistency of self-reports on substance use in a longitudinal study. Br J Addict 1987; 82:891-8., this may happen because of fear of admitting use of illegal substances, or because certain actions are considered shameful. Conversely, some adolescents may “brag” or over-report engaging in disapproved or risky behaviors, perhaps as a result of their desire to conform to the presumed norms of their peers ¹1. Brener ND, Billy JO, Grady WR. Assessment of factors affecting the validity of self-reported health-risk behavior among adolescents: evidence from the scientific literature. J Adolesc Health 2003; 33:436-57.^,2626. Palen L-A, Smith EA, Caldwell LL, Flisher AJ, Wegner L, Vergnani T. Inconsistent reports of sexual intercourse among South African high school students. J Adolesc Health 2008; 42:221-7.^,2727. Dolcini MM, Adler NE, Ginsberg D. Factors influencing agreement between self-reports and biological measures of smoking among adolescents. J Res Adolesc 1996; 6:515-42.^,2828. Matsushima R, Shiomi K. Social self-efficacy and interpersonal stress in adolescence. Soc Behav Pers 2003; 31:323-32.^,2929. van de Mortel TF. Faking it: social desirability response bias in self-report research. Aust J Adv Nurs 2008; 25:40-8..

In addition to verifying these problems of distorted self-presentation, previous research on self-report indicates other sources of error. Studies have shown that the individuals often fail to judge the frequencies accurately ²³23. Hilton ME. Inconsistent responses to questions about alcohol consumption in specified settings. Am J Drug Alcohol Abuse 1986; 12:403-13.^,3030. Blair E, Burton S. Cognitive processes used by survey respondents to answer behavioral frequency questions. J Consum Res 1987; 14:280-8.^,3131. Menon G. The effects of accessibility of information in memory on judgments of behavioral frequencies. J Consum Res 1993; 20: 431-40.^,3232. Stone AA, Bachrach CA, Jobe JB, Kurtzman HS, Cain VS, editors. The science of self-report: Implications for research and practice. New Jersey: Lawrence Erlbaum; 2000.^,3333. Menon G. Judgments of behavioral frequencies: memory search and retrieval strategies. In: Schwarz N, Sudman S, editors. Autobiographical memory and the validity of retrospective reports. New York: Springer; 1994. p. 161-72.^,3434. Menon G, Raghubir P, Schwarz N. Behavioral frequency judgments: an accessibility-diagnosticity framework. J Consum Res 1995; 22:212-28., and that the extent of error associated with the report of frequency of past behaviors varies intensely among studies. Blair & Button ³⁰30. Blair E, Burton S. Cognitive processes used by survey respondents to answer behavioral frequency questions. J Consum Res 1987; 14:280-8. have established three factors that can affect the accuracy of self-reported frequency of past behaviors: the actual frequency of the event (more frequent behaviors are often reported by using estimation methods, rather than on the basis of actual episodic memory), the question wording (the use of “how many times” induces more inaccuracy than the use of “how many”), and the reference time frame (longer time frames increasing the chance of error). More recent research has shown that holding these three factors constant could help improve the accuracy of self-report data ³²32. Stone AA, Bachrach CA, Jobe JB, Kurtzman HS, Cain VS, editors. The science of self-report: Implications for research and practice. New Jersey: Lawrence Erlbaum; 2000.^,3535. Menon G, Yorkston EA. The use of memory and contextual cues in the formation of behavioral frequency judgments. In: Stone AA, Bachrach CA, Jobe JB, Kurtzman HS, Cain VS, editors. The science of self-report: implications for research and practice. New Jersey: Lawrence Erlbaum; 2000. p. 63-79.^,3636. Willis GB. Cognitive interviewing: a tool for improving questionnaire design. Thousand Oaks: Sage Publications; 2004..

Sometimes, the validity of self-reported risk behavior can be assessed by collecting additional, potentially contradictory responses and complementary evidence such as biochemical markers, collateral informant reports and medical interview. Studies of how the validity of self-report can affect the prevalence estimates of risk behaviors (especially for the age at first use of alcohol and drugs) have been intensely developed by epidemiologists and other health sciences researchers, with both community-based and school-based samples ³⁷37. Appel PW, Hoffman JH, Blane HT, Frank B, Oldak R, Burke M. Comparison of self-report and hair analysis in detecting cocaine use in a homeless/transient sample. J Psychoactive Drugs 2001; 33:47-55.^,3838. Colón HM, Robles RR, Sahai H. The validity of drug use responses in a household survey in Puerto Rico: comparison of survey responses of cocaine and heroin use with hair tests. Int J Epidemiol 2001; 30:1042-9.^,3939. Colón HM, Robles RR, Sahai H. The validity of drug use self-reports among hard core drug users in a household survey in Puerto Rico: comparison of survey responses of cocaine and heroin use with hair tests. Drug Alcohol Depend 2002; 67:269-79.^,4040. Fendrich M. The undeniable problem of recanting. Addiction 2005; 100:143-4.^,4141. Fendrich M, Johnson TP, Sudman S, Wislar JS, Spiehler V. Validity of drug use reporting in a high-risk community sample: a comparison of cocaine and heroin survey reports with hair tests. Am J Epidemiol 1999; 149:955-62.^,4242. Fendrich M, Rosenbaum DP. Recanting of substance use reports in a longitudinal prevention study. Drug Alcohol Depend 2003; 70:241-53.^,4343. Johnson TP, Mott JA. The reliability of self-reported age of onset of tobacco, alcohol and illicit drug use. Addiction 2001; 96:1187-98.^,4444. Miller P, Curtis A, Jenkinson R, Droste N, Bowe SJ, Pennay A. Drug use in Australian nightlife settings: estimation of prevalence and validity of self-report. Addiction 2015; 110:1803-10.^,4545. Tassiopoulos K, Bernstein J, Heeren T, Levenson S, Hingson R, Bernstein E. Predictors of disclosure of continued cocaine use. Addict Behav 2006; 31:80-9.. Cross & Newman-Gonchar ¹⁷17. Cross JE, Newman-Gonchar R. Data quality in student risk behavior surveys and administrator training. J Sch Violence 2004; 3:89-108. have shown that the estimated prevalence of risk behaviors, antisocial behaviors, and victimization experiences were substantially reduced when the respondents who gave multiple inconsistent, or extreme responses to other survey items were screened out of the data.

Inaccurate measures of risk behavior not only bias prevalence estimates, but can also act as confounders for the study of morbidity, mortality, and the social and economic outcomes associated with risk behavior, especially for young samples. As the uncritical acceptance of findings from adolescent surveys can lead to erroneous conclusions, our aim is to analyze the consistency of self-reports of risk behavior on the 2009 and 2012 PeNSE surveys, some of the most commonly used data sources about adolescent health in Brazil.

Methods

Data sources and measures

The PeNSE is an ongoing school-based survey, conducted by the Brazilian Ministry of Health, together with the Brazilian Institute of Geography and Statistics (IBGE), to monitor the health of children and adolescents enrolled in the 9^th grade, from public and private schools. In 2009, a sample consisting of 62,910 students from 6,780 schools was drawn in such a way as to be representative of the student population of Brazil’s 26 state capitals plus the Federal District. In 2012, the sample was expanded to be representative of Brazil’s five regions, now including rural and urban areas of the South, Southeast, North, Northeast, Central and also the 26 state capitals and the Federal District, comprising a total of 109,104 students, from 2,842 schools ²2. Instituto Brasileiro de Geografia e Estatística. Pesquisa Nacional de Saúde do Escolar (PeNSE), 2009. Brasília: Instituto Brasileiro de Geografia e Estatística; 2009.^,33. Instituto Brasileiro de Geografia e Estatística. Pesquisa Nacional de Saúde do Escolar (PeNSE), 2012. Brasília: Instituto Brasileiro de Geografia e Estatística; 2012..

In both 2009 and 2012, all students in the selected classrooms who were present on the day of data collection were invited to participate. In 2009, out of the 63,411 students who were present in the classroom, 501 refused to participate (0.8%) and in 2012, from the total of 110,873 students invited, 1,651 refused to participate (1.5%). The 9^th grade was chosen because the students in this grade, mostly aged between 13 and 15 years old, have already acquired the necessary skills to answer self-applicable questionnaires. Other reasons for the choice of 9^th grade were because this group is prone to being exposed to several risk factors, and to permit comparability with various survey systems from other countries ⁴⁶46. Currie C, Nic Gabhainn S, Godeau E; International HBSC Network Coordinating Committee. The health behaviour in school-aged children WHO Collaborative Cross-National (HBSC) Study: origins, concept, history and development 1982-2008. Int J Public Health 2009; 54 Suppl 2:131-9.^,4747. Ministério da Educação. Ensino Fundamental de nove anos: perguntas mais frequentes e respostas da Secretaria de Educação Básica (SEB/MEC). portal.mec.gov.br/seb/arquivos/pdf/Ensfund/ensfund9_perfreq.pdf (accessed on 01/Aug/2015).
portal.mec.gov.br/seb/arquivos/pdf/Ensfu... .

The students answered the questionnaire using a personal digital assistant palmtop, similar to a smartphone. The 2009 questionnaire had 104 questions, involving items about socioeconomic status, social support, bullying, nutritional habits, body image perception, oral health, physical activity, substance use (alcohol, drugs and cigarettes), sexual activity, safety, accidents, and exposure to violence. In 2009, anthropometric measures were collected by the survey team for all the students.

In 2012, questions about asthma, hygiene habits, mental health, work activity, and use of health services were added, summing to a total of 127 questions. Some of the questions from 2009 were altered, and anthropometric measures were not collected in 2012. For further details, see the documentation of the 2009 and 2012 PeNSE survey ²2. Instituto Brasileiro de Geografia e Estatística. Pesquisa Nacional de Saúde do Escolar (PeNSE), 2009. Brasília: Instituto Brasileiro de Geografia e Estatística; 2009.^,33. Instituto Brasileiro de Geografia e Estatística. Pesquisa Nacional de Saúde do Escolar (PeNSE), 2012. Brasília: Instituto Brasileiro de Geografia e Estatística; 2012..

Analytical approach

In order to detect inconsistent responses, we created indicator variables for each behavior (called “flag variables”), which took on the value of 1 if a response was logically inconsistent with a previous statement about having ever engaged in that specific behavior. Using these domain-specific indicators of inconsistency, we then created a variable representing the total number of inconsistent responses given by each participant (2009’s range = 0-19, 2012’s range = 0-23). We also used these indicators to calculate the domain-specific and overall inconsistency rates.

Examples of inconsistencies found in the data included logical forms, such as a report of past use of alcohol at one period, followed by a report of never having drunk alcohol on a subsequent question. The opposite pattern of inconsistency was also considered, where participants who answered “yes” to the “have you ever” question then claimed to have never engaged in that behavior in a subsequent answer. Responses were also flagged as inconsistent when the participant reported a particular age at which he had first engaged in a behavior which was above his current reported age. A list of the questions inspected for inconsistencies in the 2009 and 2012 editions of the PeNSE survey can be found in Table 1.

In order to explore the patterns of inconsistent responding and identify factors that might account for them, we conducted three types of analyses. First, we determined the rates of inconsistent responses among the participants in 2009 and 2012 for each type of behavior (sexual activity, alcohol, cigarettes and drug use), and across all of the questions.

Second, we calculated the prevalence of alcohol, drug and cigarette use, and sexual activity, after cleaning the data of all the cases with inconsistent responses on these domains, and then compared the results with the prevalences reported in previous studies based on the data from the 2009 ⁴4. Araújo C, Toral N, Silva ACF, Velásquez-Melendez G, Dias AJR. Estado nutricional dos adolescentes e sua relação com variáveis sociodemográficas: Pesquisa Nacional de Saúde do Escolar (PeNSE), 2009. Ciênc Saúde Coletiva 2010; 15 Suppl 2:3077-84.^,99. Malta DC, Sardinha LMV, Mendes I, Barreto SM, Giatti L, Castro IRR, et al. Prevalence of risk health behavior among adolescents: results from the 2009 National Adolescent School-based Health Survey (PeNSE). Ciênc Saúde Coletiva 2010; 15 Suppl 2:3009-19.^,1313. Malta DC, Mascarenhas MDM, Porto DL, Duarte EA, Sardinha LM, Barreto SM, et al. Prevalência do consumo de álcool e drogas entre adolescentes: análise dos dados da Pesquisa Nacional de Saúde Escolar. Rev Bras Epidemiol 2011; 14 Suppl 1:136-46.^,1414. Malta DC, Porto DL, Melo FCM. Família e proteção ao uso de tabaco, álcool e drogas em adolescentes, Pesquisa Nacional. Rev Bras Epidemiol 2011; 14 Suppl 1:166-77.^,1515. Malta DC, Silva MAI, Mello FCM, Monteiro RA, Porto DL, Sardinha LMV, et al. Saúde sexual dos adolescentes segundo a Pesquisa Nacional de Saúde dos Escolares. Rev Bras Epidemiol 2011; 14 Suppl 1:147-56. and 2012 ⁶6. Horta RL, Horta BL, Costa AWN, Prado RR, Oliveira-Campos M, Malta DC. Lifetime use of illicit drugs and associated factors among Brazilian schoolchildren, National Adolescent School-based Health Survey (PeNSE 2012). Rev Bras Epidemiol 2014; 17 Suppl 1:31-45.^,88. Malta DC, Oliveira-Campos M, Prado RR, Andrade SSC, Mello FCM, Dias AJR, et al. Psychoactive substance use, family context and mental health among Brazilian adolescents, National Adolescent School-based Health Survey (PeNSE 2012). Rev Bras Epidemiol 2014; 17 Suppl 1:46-61.^,1010. Malta DC, Machado IE, Porto DL, da Silva MMA, de Freitas PC, da Costa AWN, et al. Alcohol consumption among Brazilian Adolescents according to the National Adolescent School-based Health Survey (PeNSE 2012). Rev Bras Epidemiol 2014; 17 Suppl 1:203-14.^,1212. Oliveira-Campos M, Nunes ML, Madeira FC, Santos MG, Bregmann SR, Malta DC, et al. Sexual behavior among Brazilian adolescents, National Adolescent School-based Health Survey (PeNSE 2012). Rev Bras Epidemiol 2014; 17 Suppl 1:116-30.^,1616. Barreto SM, Giatti L, Oliveira-Campos M, Andreazzi MA, Malta DC. Experimentation and use of cigarette and other tobacco products among adolescents in the Brazilian state capitals (PeNSE 2012). Rev Bras Epidemiol 2014; 17 Suppl 1:62-76. editions of the PeNSE survey.

Third, we estimated a logistic regression model, to measure the relationship between giving an inconsistent response, and various participant characteristics: age (categorized according to norms for the 9^th grade in Brazil ⁴⁷47. Ministério da Educação. Ensino Fundamental de nove anos: perguntas mais frequentes e respostas da Secretaria de Educação Básica (SEB/MEC). portal.mec.gov.br/seb/arquivos/pdf/Ensfund/ensfund9_perfreq.pdf (accessed on 01/Aug/2015).
portal.mec.gov.br/seb/arquivos/pdf/Ensfu... as“age appropriate” - ages 13 to 16, “accelerated” - ages 11 or 12, or “delayed” - ages 17 to 19), sex, type of school (private or public), possession of a cellphone, self-report of drug use, and self-reported level of difficulty in answering the questionnaire. As there are strong socioeconomic ⁴⁸48. Reis JGA, Barros R. Desigualdade salarial e distribuição de educação: a evolução das diferenças regionais no Brasil. Pesquisa e Planejamento Econômico 1990; 20:415-78.^,4949. Medeiros M, Oliveira LFB. Desigualdades regionais em educação: potencial de convergência. Soc Estado 2014; 29:561-85. and cultural differences ⁵⁰50. Picanço MRA. A idade da menarca da menina brasileira: os fatores socieconômicos e as diferenças regionais. Análise dos dados da PNSN, 1989 [Doctoral Dissertation]. Rio de Janeiro: Instituto Fernandes Figueira, Fundação Oswaldo Cruz; 1995.^,5151. Heilborn ML. O aprendizado da sexualidade: reprodução e trajetórias sociais de jovens brasileiros. Rio de Janeiro: Editora Garamond/Editora Fiocruz; 2006.^,5252. Pinsky I, Zaleski M, Laranjeira R, Caetano R. First national survey on patterns of alcohol consumption in the Brazilian population. Rev Bras Psiquiatr 2010; 32:214-5. in risk attitudes across Brazil’s five regions, we have also included regional dummies as potential“explanatory” variables.

Results

In 2009, 11.7% of the participants provided inconsistent responses for at least 1 of the 19 questions about risk behavior. In 2012, this percentage increased, with 22.7% of the participants giving an inconsistent response for at least 1 of the 23 questions. Limiting the analysis only to the identical questions in both surveys (19 items), the 2012 edition still showed an inconsistency rate of 22.2%. Table 2 shows the percentage of the participants providing at least one inconsistent response across all items, and for each of the four domains.

Inconsistency rates were higher for the most common behaviors: alcohol (2.7% in 2009 and 12.8% in 2012) and cigarettes (4.3% in 2009 and 2012). It is noteworthy that, in 2012, the question “How old were you when you had your first dose of alcohol?” was alone responsible for 46.6% of all the inconsistencies, but in 2009, inconsistencies on this question constituted only 10.2% of the total.

Only 1.3% of the participants in 2009, and 3.4% of the participants in 2012, were flagged in more than two domains. Inconsistency in one domain was not strongly associated with inconsistency in the others. Most inconsistent responders provided an inconsistent response in only one of the four domains (89.9% in 2009, and 89.1% in 2012).

To analyze how these inconsistency rates may have affected the prevalence estimates of risk behavior which have been published previously on the basis of only the “have you ever” questions ⁶6. Horta RL, Horta BL, Costa AWN, Prado RR, Oliveira-Campos M, Malta DC. Lifetime use of illicit drugs and associated factors among Brazilian schoolchildren, National Adolescent School-based Health Survey (PeNSE 2012). Rev Bras Epidemiol 2014; 17 Suppl 1:31-45.^,77. Malta DC, Andreazzi MAR, Oliveira-Campos M, Andrade SSCA, Sá NNB, Moura L, et al. Trend of the risk and protective factors of chronic diseases in adolescents, National Adolescent School-based Health Survey (PeNSE 2009 and 2012). Rev Bras Epidemiol 2014; 17 Suppl 1:77-91.^,88. Malta DC, Oliveira-Campos M, Prado RR, Andrade SSC, Mello FCM, Dias AJR, et al. Psychoactive substance use, family context and mental health among Brazilian adolescents, National Adolescent School-based Health Survey (PeNSE 2012). Rev Bras Epidemiol 2014; 17 Suppl 1:46-61.^,99. Malta DC, Sardinha LMV, Mendes I, Barreto SM, Giatti L, Castro IRR, et al. Prevalence of risk health behavior among adolescents: results from the 2009 National Adolescent School-based Health Survey (PeNSE). Ciênc Saúde Coletiva 2010; 15 Suppl 2:3009-19.^,1010. Malta DC, Machado IE, Porto DL, da Silva MMA, de Freitas PC, da Costa AWN, et al. Alcohol consumption among Brazilian Adolescents according to the National Adolescent School-based Health Survey (PeNSE 2012). Rev Bras Epidemiol 2014; 17 Suppl 1:203-14.^,1212. Oliveira-Campos M, Nunes ML, Madeira FC, Santos MG, Bregmann SR, Malta DC, et al. Sexual behavior among Brazilian adolescents, National Adolescent School-based Health Survey (PeNSE 2012). Rev Bras Epidemiol 2014; 17 Suppl 1:116-30.^,1313. Malta DC, Mascarenhas MDM, Porto DL, Duarte EA, Sardinha LM, Barreto SM, et al. Prevalência do consumo de álcool e drogas entre adolescentes: análise dos dados da Pesquisa Nacional de Saúde Escolar. Rev Bras Epidemiol 2011; 14 Suppl 1:136-46.^,1414. Malta DC, Porto DL, Melo FCM. Família e proteção ao uso de tabaco, álcool e drogas em adolescentes, Pesquisa Nacional. Rev Bras Epidemiol 2011; 14 Suppl 1:166-77.^,1515. Malta DC, Silva MAI, Mello FCM, Monteiro RA, Porto DL, Sardinha LMV, et al. Saúde sexual dos adolescentes segundo a Pesquisa Nacional de Saúde dos Escolares. Rev Bras Epidemiol 2011; 14 Suppl 1:147-56.^,1616. Barreto SM, Giatti L, Oliveira-Campos M, Andreazzi MA, Malta DC. Experimentation and use of cigarette and other tobacco products among adolescents in the Brazilian state capitals (PeNSE 2012). Rev Bras Epidemiol 2014; 17 Suppl 1:62-76., we calculated the percentage of inconsistent responders in relation to “yes” or “no” responses to those “have you ever” questions. In both 2009 and 2012, there was a higher percentage of self-contradiction among the participants who said yes than among those who said no, except for alcohol experimentation. Table 3 shows the percentage of inconsistent responders in both cases. For example, 8% of the participants in 2009, and 9.6% in 2012, claimed to have experimented with drugs, but 23.1% of the 8% (in 2009) and 31.7% of the 9.6% (in 2012) did not corroborate those positive responses on subsequent questions about the frequency, or age at first use.

As shown in Table 4, the removal of inconsistent responders affects the estimated prevalence of these risk behaviors in different directions, and with different magnitudes.

Next, we examined the participant characteristics, and socio-demographic aspects of the sample, to determine which variables were significantly associated with giving an inconsistent response in at least one item. Table 5 shows the results from the final adjusted logistic regression model, as well as the unadjusted coefficients (univariate regression), and the percentages of inconsistencies within each of the independent variables.

Males were significantly more inconsistent than females in their responses, especially in the domain of sexual activity, where males constituted 64.7% of the inconsistent responders in 2009 and 63.1% in 2012.

Participants who were delayed for the 9^th grade had a higher rate of inconsistency than age-appropriate participants. In 2009, age-delayed participants were two times more inconsistent than age-appropriate respondents (95%CI: 1.96-2.39), and, in 2012, their odds of giving an inconsistent response were 1.13 times higher. Being accelerated (more than two years younger than the expected age for the 9^th grade) was not significantly associated with giving an inconsistent response.

As previous studies have shown that drug users are more prone to being inconsistent responders, we tested this association in the PeNSE data. Our results showed a strong association between self-report of having experimented with drugs and inconsistent responding. In 2009, these participants were almost four times more likely to be inconsistent than the participants who did not report having ever experimented with drugs (OR = 3.97; 95%CI: 3.71-4.24), an association that decreased in 2012, but remained strongly significant (OR = 2.50; 95%CI: 2.48-2.52). Excluding the inconsistent cases from the explanatory variable (self-report of drug use) self-report of drug use was no longer a significant predictor of inconsistency in the 2009 data (OR = 1.11; 95%CI: 0.99-1.25), but remained significant for 2012. In this case, even when limiting the analysis only to the consistent responders, those who reported having experimented with drugs were significantly more inconsistent than those who have not (OR = 2.07; 95%CI: 1.91-2.25) .

Being a student from a public school in 2009 was associated with a 62% increase in the odds of giving an inconsistent response to a least one item (95%CI: 1.52-1.73), but, in 2012, this association curiously changed direction (OR = 0.92; 95%CI: 0.89-0.95).

Self-reported level of difficulty in answering the questionnaire was not significantly associated with being inconsistent, but not having a cellphone was. In 2009, there was a 21% lower likelihood of being inconsistent among cellphone owners than among those who had none. This contrast was in the same direction but much smaller in 2012, when having a cellphone was associated with only a 5% lower likelihood of being an inconsistent responder.

Finally, our multivariate results did not reveal strong regional differences in the odds of an inconsistent response. In both 2009 and 2012, the south and southeast regions were the ones with the highest percentages of inconsistent responders, but there were no statistically significant differences.

Discussion

The results of this study show that the majority of the participants in the PeNSE survey provided consistent reports, but that a sizable minority provided one or more self-contradictory responses. Consistent with previous analyses of other survey data, in both editions of the PeNSE survey, we found that the prevalence of inconsistency was higher for males ⁴⁵45. Tassiopoulos K, Bernstein J, Heeren T, Levenson S, Hingson R, Bernstein E. Predictors of disclosure of continued cocaine use. Addict Behav 2006; 31:80-9.^,5353. Cornell DG, Lovegrove PJ, Baly MW. Invalid survey response patterns among middle school students. Psychol Assess 2014; 26:277-87.^,5454. Maniaci MR, Rogge RD. Caring about carelessness: participant inattention and its effects on research. J Res Pers 2014; 48:61-83.^,5555. Kydd RM, Connor J. Inconsistency in reporting abstention and heavy drinking frequency: associations with sex and socioeconomic status, and potential impacts. Alcohol Alcohol 2015; 50:33-45., delayed students ⁵⁶56. Harris KM, Griffin BA, McCaffrey DF, Morral AR. Inconsistencies in self-reported drug use by adolescents in substance abuse treatment: implications for outcome and performance measurements. J Subst Abuse Treat 2008; 34:347-55., and those who reported having experimented with drugs ²⁶26. Palen L-A, Smith EA, Caldwell LL, Flisher AJ, Wegner L, Vergnani T. Inconsistent reports of sexual intercourse among South African high school students. J Adolesc Health 2008; 42:221-7.^,4444. Miller P, Curtis A, Jenkinson R, Droste N, Bowe SJ, Pennay A. Drug use in Australian nightlife settings: estimation of prevalence and validity of self-report. Addiction 2015; 110:1803-10.^,5656. Harris KM, Griffin BA, McCaffrey DF, Morral AR. Inconsistencies in self-reported drug use by adolescents in substance abuse treatment: implications for outcome and performance measurements. J Subst Abuse Treat 2008; 34:347-55..

Considering that the PeNSE questionnaire is relatively long, and that the use of a smartphone could represent an extra source of difficulty for some of the participants, we tested the effect of possessing a cellphone, and also the self-reported level of difficulty in answering the questionnaire (assessed by the question: “How difficult did you find this questionnaire to answer? 1 - very easy; 2 - easy; 3 - neither difficult nor easy; 4 - difficult; 5 - very difficult). Interestingly, the self-report of difficulty was not significantly associated with being inconsistent, but not having a cellphone was. In 2009, when the percentage of Brazilian children and adolescents who had a cellphone was smaller ⁵⁷57. Cunha P. Adolescentes lideram aumento de uso de celular no Brasil diz Pnad. http://g1.globo.com/tecnologia/noticia/2012/09/adolescentes-lideram-aumento-de-uso-de-celular-no-brasil-diz-pnad.html (accessed on 01/Aug/2015).
http://g1.globo.com/tecnologia/noticia/2... , there was a 21% lower likelihood of being inconsistent for those who had one. On the other hand, in 2012, when cellphones had become more widely available and hence familiar having a cellphone entailed a decrease of only 5% in the likelihood of being an inconsistent responder. This results suggest that familiarity with cellphone positively influence the consistency rates in both 2009 and 2012, a feature that needs to be taken into consideration in the next editions of the survey while recruiting the participants, by offering assistance to those who are not familiar with the technology.

Consistent with the literature ³⁰30. Blair E, Burton S. Cognitive processes used by survey respondents to answer behavioral frequency questions. J Consum Res 1987; 14:280-8.^,3131. Menon G. The effects of accessibility of information in memory on judgments of behavioral frequencies. J Consum Res 1993; 20: 431-40.^,3232. Stone AA, Bachrach CA, Jobe JB, Kurtzman HS, Cain VS, editors. The science of self-report: Implications for research and practice. New Jersey: Lawrence Erlbaum; 2000.^,3333. Menon G. Judgments of behavioral frequencies: memory search and retrieval strategies. In: Schwarz N, Sudman S, editors. Autobiographical memory and the validity of retrospective reports. New York: Springer; 1994. p. 161-72.^,3434. Menon G, Raghubir P, Schwarz N. Behavioral frequency judgments: an accessibility-diagnosticity framework. J Consum Res 1995; 22:212-28.^,3535. Menon G, Yorkston EA. The use of memory and contextual cues in the formation of behavioral frequency judgments. In: Stone AA, Bachrach CA, Jobe JB, Kurtzman HS, Cain VS, editors. The science of self-report: implications for research and practice. New Jersey: Lawrence Erlbaum; 2000. p. 63-79.^,5252. Pinsky I, Zaleski M, Laranjeira R, Caetano R. First national survey on patterns of alcohol consumption in the Brazilian population. Rev Bras Psiquiatr 2010; 32:214-5.^,5858. Burton S, Blair E. Task conditions, response formulation processes, and response accuracy for behavioral frequency questions in surveys. Public Opin Q 1991; 55:50-79., inconsistency rates were higher for the most common behaviors: alcohol and cigarettes, but we found a low percentage of extreme cases of inconsistent responding, with most of the participants being flagged for inconsistency in only one of the four domains. Fewer than 4% of the participants were flagged in more than two domains, consistent with previous estimates of careless responding ⁵⁹59. Meade AW, Craig SB. Identifying careless responses in survey data. Psychol Methods 2012; 17:437-55.^,6060. Johnson JA. Ascertaining the validity of individual protocols from web-based personality inventories. J Res Pers 2005; 39:103-29.. This finding may suggest that, in the case of the PeNSE survey, the inconsistencies are unlikely to derive from “complete indifference or pervasive carelessness” ⁶⁰60. Johnson JA. Ascertaining the validity of individual protocols from web-based personality inventories. J Res Pers 2005; 39:103-29.^,6161. Baer RA, Ballenger J, Berry DT, Wetter MW. Detection of random responding on the MMPI-A. J Pers Assess 1997; 68:139-51..

Such inconsistency was much more prevalent in 2012, especially on the question about age of first use of alcohol. Inconsistent measures of alcohol consumption can bias the estimates of morbidity, mortality, and other associated outcomes. As pointed out by Kydd ⁵⁵55. Kydd RM, Connor J. Inconsistency in reporting abstention and heavy drinking frequency: associations with sex and socioeconomic status, and potential impacts. Alcohol Alcohol 2015; 50:33-45., the validity and possible biases of self-report measures of alcohol consumption have been subjects of considerable research attention, and rates of inconsistency in this domain have been found to vary in relation to gender, ethnicity, and socioeconomic status ⁶²62. Berry DT, Wetter MW, Baer RA, Larsen L, Clark C, Monroe K. MMPI-2 random responding indices: validation using a self-report methodology. Psychol Assess 1992; 4:340-5.^,6363. Midanik LT, Greenfield TK. Defining "current drinkers" in national surveys: results of the 2000 National Alcohol Survey. Addiction 2003; 98:517-22.^,6464. Greenfield TK, Kerr WC. Alcohol measurement methodology in epidemiology: recent advances and opportunities. Addiction 2008; 103:1082-99..

However, because the set of questions about alcohol was not identical in the 2009 and 2012 editions of the PeNSE survey, it is difficult to determine which factors may account for the increase in the inconsistency rate. Nonetheless, it is noteworthy that in the past years several changes were made in the Brazilian regulation for alcohol consumption, especially with the creation of the “Lei Seca” in 2008 - a dry law, different from the old American one - that makes the inspection of drivers stricter in the country, punishing people who are caught having alcohol in their blood when driving. As a consequence of the “Lei Seca”, other changes were made on the alcohol regulation in Brazil, with the Law n. 5,502 from 2013 that made underage alcohol consumption illegal, as well as criminalizing the sale of alcoholic beverages to people under the age of 18. With these changes in the federal regulation, the debate about underage drinking gained the attention of the media and became a great topic of discussion in the popular opinion, which might have indirectly affected the students’ attitudes in reporting their actual drinking habits. Because underage drinking in Brazil is generally socially disapproved, we believe that this may be affecting the youths’ perception of this matter and therefore causing more inconsistencies.

As there was a bigger proportion of inconsistent responses among the participants who claimed that they had “ever” engaged in three of the four types of risk behaviors, it is important to consider how reliance on these “have you ever” questions may have affected estimates of the prevalence of these behaviors among Brazilian middle school students. This prominence of inconsistent responders among the self-proclaimed risk-takers is noteworthy, because excluding those cases changes the estimates of risk behaviors for most of the domains. Most importantly, the results demonstrated that even small percentages of inconsistent responses can change the estimate of reported risk behaviors. Some of our findings replicate prior studies, which have found that excluding the cases that fail consistency checks results in a reduction in estimated rates of risk behaviors ¹⁸18. Cornell DG, Loper AB. Assessment of violence and other high-risk behaviors with a school survey. School Psych Rev 1998; 27:317-30.^,4444. Miller P, Curtis A, Jenkinson R, Droste N, Bowe SJ, Pennay A. Drug use in Australian nightlife settings: estimation of prevalence and validity of self-report. Addiction 2015; 110:1803-10.^,6565. Rosenblatt JA, Furlong MJ. Assessing the reliability and validity of student self-reports of campus violence. J Youth Adolesc 1997; 26:187-202., but our finding that in the specific domains of alcohol use and sexual activity, the removal of inconsistent responders actually elevates estimated rates is apparently unprecedented.

Nevertheless, researchers should be cautious about removing inconsistent cases. The removal of all cases with inconsistency in any domain may increase the researcher’s confidence that the reported behaviors are true behaviors, but it may also inadvertently remove valid data in domains where there are no inconsistencies, and thus create a bias towards more normative behavior. More generally, without further research employing methods of validation that are external to the survey itself, we simply cannot know whether the removal of data from respondents who have contradicted themselves generally tends to improve or damage the accuracy of survey-based prevalence estimates.

Our results suggest the need for more experimental studies about the assessment of risk behavior, via school-based surveys, and could be useful for the study of survey administration routines.

This study supports the importance of performing consistency checks of self-report surveys, following the growing body of literature on this topic. As suggested by Meade & Craig ⁵⁹59. Meade AW, Craig SB. Identifying careless responses in survey data. Psychol Methods 2012; 17:437-55., every survey could benefit from incorporating methods of data screening. Some of these methods entail inserting special items or scales (e.g social desirability, lie scales, bogus items) prior to the administration of survey, while others entail post hoc analysis of response patterns after the data collection. In the PeNSE survey, in particular, inconsistencies could be greatly reduced by the simple expedient of giving access to the risk behavior questions only to those participants who answer “yes” to the relevant “have you ever” question; now that the questionnaire is administered with an electronic device, this skipping over can easily be programmed.

While self-report surveys will likely continue to be an economical method for gathering risk behavior data from the students, their value is fragile, unless data quality issues are systematically addressed in administration, interpretation, reporting, and publishing of results.

Acknowledgments

The authors wish to thank the editorial board and reviewers for valuable contributions to the paper. We also wish to thank Brazilian Graduate Studies Coordinating Board (Capes) and Brazilian National Research Council (CNPq) for help in developing the research, including a CNPq Doctorate Bursary and Sandwich Doctorate, under reference BEX n. 99999.010112/2014-06.

References

Publication Dates

History