ARTIGO ARTICLE
Reproducibility of a food frequency questionnaire for adolescents
Reprodutibilidade de um questionário de freqüência alimentar para adolescentes
Dirce Maria Lobo MarchioniI; Silvia Maria VociI; Flávia Emília Leite de LimaII; Regina Mara FisbergI; Betzabeth SlaterI
IFaculdade de Saúde Pública, Universidade de São Paulo, São Paulo, Brasil
IIFaculdade Evangélica do Paraná, Curitiba, Brasil
ABSTRACT
To assess the reproducibility of a validated 76-item food frequency questionnaire designed to estimate diet in adolescents (Adolescent Food Frequency Questionnaire AFFQ) in the city of Sao Paulo, Brazil, a test-retest study was conducted (n = 49). Intraclass correlation coefficient (ICC), weighted kappa, and percentage of agreement were used in both crude and energy-adjusted nutrient intakes. Bland Altman plots were used to examine the limits of agreement for energy and macronutrients. The ICC ranged from 0.48 (carbohydrates) to 0.65 (vitamin C) in crude values and from 0.25 (total fat) to 0.58 (vitamin C) in adjusted values. Kappa values ranged from 0.28 (protein and fiber) to 0.56 (unsaturated fat). Bland Altman showed a trend towards larger difference in energy according to increased intake values and a bias towards extreme values for fat intake. The percent of individuals classified in the same category on the two occasions was on average 54.2%. By conclusion, the Adolescent Food Frequency Questionnaire showed reasonable reproducibility and can be used in studies that aim to classify groups into intake categories.
Adolescent; Food Habits; Questionnaires; Reproducibility of Results
RESUMO
Para avaliar a reprodutibilidade de um Questionário de Freqüência Alimentar para Adolescentes (QFAA), 76 itens, validado, desenvolvido para estimar a dieta habitual de adolescentes em São Paulo, Brasil, foi conduzido um estudo do tipo teste-reteste (n = 49). Utilizaram-se o coeficiente de correlação intraclasse (CCI), o kappa ponderado e a porcentagem de concordância para os valores de ingestão brutos e ajustados pela energia. Os limites de concordância para energia e macronutrientes foram examinados pelos gráficos de Bland Altman. O CCI variou de 0,48 (carboidratos) a 0,65 (vitamina C) para os nutrientes brutos e de 0,25 (lipídios totais) a 0,58 (vitamina C) para os ajustados. Os valores do kappa variaram de 0,28 (proteínas e fibras) a 0,56 (lipídios insaturados). O gráfico de Bland Altman evidenciou tendência à maior diferença para energia conforme o aumento dos valores de ingestão e para lipídios com os valores extremos. A porcentagem de indivíduos classificados na mesma categoria foi em média 54,2%. Conclui-se que o QFAA apresentou razoável reprodutibilidade e pode ser utilizado em estudos que objetivam classificar os indivíduos em categorias de consumo.
Adolescente; Hábitos Alimentares; Questionários; Reprodutibilidade dos Testes
Introduction
Diet is a relevant factor not only for growth and development, but also for the present and future health of adolescents 1,2,3. There is increasing evidence that diet in childhood and adolescence is related to diseases of adulthood like heart disease, osteoporosis, and cancer 4. Therefore, the evaluation and quantification of adolescents' habitual diet are of great concern. To meet both monitoring and nutritional research aims, a reliable instrument is needed that accurately and precisely assesses this group's dietary intake 5.
The use of food frequency questionnaires has been increasing in epidemiological studies among adolescents, since beginning in adolescence, cognitive processes are considered to be similar to those of adults 3,6,7,8,9. However, inaccurate diet measurements may obscure the relationship between diet and the target outcome 10. It is well known that errors are inherent to any assessment of dietary status, so studies for validating and investigating the reproducibility of measurement instruments need to be considered, since they help understand the relationship between measured and true values 10. Such studies have the aim of increasing accuracy and reducing measurement bias 11. Reproducibility studies assess the extent to which results agree when obtained by the same approach at different points in time 12. A food frequency questionnaire should reproduce similar results when applied under similar circumstances, over a given time interval, and with the same participants 10,13,14.
The objective of this study was to assess the reproducibility of a validated 76-item food frequency questionnaire for assessing habitual diet in adolescents (Adolescent Food Frequency Questionnaire AFFQ) in the city of São Paulo, Brazil.
Material and methods
This was a reproducibility study using a test-retest design.
Forty-nine high school sophomores (second-year students), 25 boys and 24 girls (ages 16-19 years) at a public school located on the West Side of the city of São Paulo were invited and agreed to participate in the study. The AFFQ was administered twice, in May and August of the same year, 2001. On both occasions, students were asked to complete the questionnaire in the classroom, with the presence of a field researcher to help ensure that the questionnaires were filled out under the same conditions.
Adolescent Food Frequency Questionnaire
The instrument was designed to assess the amounts of foods consumed over the preceding six-month period. The development and validation of the AFFQ were described elsewhere 15. In brief, a list of 76 foods or food groups that contributed to at least 90% of energy and macronutrient intake by adolescents was compiled, based on dietary data from 24h food recall 16. Portion sizes were obtained from the same dataset 16. The questionnaire was self-administered, and students were asked to estimate the average frequency of their intake for each food or food group over the previous six months by selecting one of the seven categories: never, less than once a month, one to three times a month, once a week, two to four times a week, once a day, twice or more a day. Nutrient intakes were calculated by multiplying the portion weight by its nutrient content. Specific software Virtual Nutri version 1.0 (Faculdade de Saúde Pública, Universidade de São Paulo, São Paulo, Brasil) was used to analyze the energy and nutrient content of food intake. Here, food nutritional values were compiled either from Brazilian tables, food composition analyses furnished by manufacturers, or for foods unavailable in Brazilian databases, food composition tables from other countries, for example the United States Department of Agriculture (USDA) 17.
In the validation study, the instrument showed good accuracy (crude mean r = 0.57) 16.
Initially, we chose a list of nutrients to examine the reproducibility. It included the macronutrients and nutrients related to frequently reported low intake in this young population, such as iron and calcium, and common diseases in youth (anemia) and adulthood (cardiovascular diseases, cancer, and osteoporosis).
Statistical analysis
Nutrient intakes were log-transformed to improve normality. Most of the nutrients measured by the AFFQ presented high correlations with energy. It was therefore necessary to remove this variation because of its association with measurement error, and this was done by means of the residuals methods proposed by Willet & Stampfer 18.
Intraclass correlation coefficients (ICC) were utilized to assess the reproducibility of the dietary intake reported by the individuals in the AFFQ test-retest. Differences between individual intakes reported by students in the first and second AFFQ were compared using Wilcoxon signed-rank test.
The percentage concordance was determined by identifying the percentages of individuals classified in the same, adjacent, and opposite categories.
Bland Altman plots were used to examine the limits of agreement for energy and macronutrients between AFFQ1 and AFFQ2 19. For each individual the difference in the log-transformed measured intake between the two AFFQs was plotted against their average intake from the same questionnaire 19.
To simultaneously examine the contribution of gender to intake reproducibility, we estimated the partial correlations for total energy intake. Gender was not significantly related to reproducibility (r = 0.08, p = 0.589), so the analysis was conducted with no further gender distinction.
To verify concordance of the ordinal data, i.e. in the categories given by intake in tertiles, weighted kappa was utilized. This was derived from the kappa coefficient, with weights based on the magnitudes of the observed discordance 20.
The study was approved by the Research Ethics Committee of the School of Public Health, University of São Paulo, and the parents and adolescents provided informed consent.
Results
Table 1 presents means and standard deviations of energy and nutrient intake in the AFFQ test and retest. For the majority of the nutrients assessed, the values found in the retest were lower than in the initial test. Statistical differences between AFFQ1 and AFFQ2 were observed for energy, carbohydrates, total fat, unsaturated fat, fiber, and iron.
Intraclass correlation coefficients (ICC) between the two AFFQs ranged from 0.48 (carbohydrates) to 0.65 (vitamin C) in crude values (mean value 0.56) and from 0.25 (total fat) to 0.58 (vitamin C) in adjusted values (mean value 0.43), as shown in Table 2. All ICC values decreased after energy adjustment.
Table 3 presents the results for the capacity of the AFFQ to classify individuals within the same tertile, adjacent tertile, and opposite tertile of nutrient intake, after adjusting for energy, between the test and retest. The proportion of individuals classified in the same tertile averaged 54.2%, ranging from 44.8% (protein) to 63.2% (vitamin C). The percentage of individuals classified in opposite tertiles ranged from 6.1% for unsaturated fat to 14.3% for carbohydrates, total fat, and cholesterol (mean value 11.3%). The values of the weighted kappa coefficient ranged from 0.31 for total fat and cholesterol to 0.56 for unsaturated fat (mean value 0.42).
Bland Altman plots were used to further examine the limits of agreement between the two applications of the AFFQ, by plotting individual differences against the mean of the two questionnaires (Figure 1). The nutrients chosen for the analysis were energy, total fat, carbohydrate, and protein. The means for energy and macronutrients were all positive, indicating a bias towards overestimation in AFFQ1. The Pearson correlation coefficient between the differences and mean of the two questionnaires were all positive, but not statistically significant (energy r = 0.090, p = 0.538; protein r = 0.141, p = 0.333; carbohydrate r = 0.004, p = 0.981; lipid r = 0.231, p = 0.111).
Discussion
This paper reports on the reproducibility of a previously validated food frequency questionnaire, designed to estimate adolescents' habitual food intake. Our results indicate reasonable reproducibility for categorizing individuals according to levels of intake, the usual aim of epidemiological studies.
The mean kappa value was 0.42, thus indicating fair reliability 21. This was similar to the findings by Hansson et al. 22, who reported a median value of 0.40 for the weighted kappa coefficient.
The crude concordance ranged from 44.8% for protein to 63.2% for vitamin C, with a mean of 54.2% of individuals classified in the same tertile. The low percentage of individuals classified in opposite tertiles (11.3%) was noteworthy. Considering the large variability observed in adolescents' food intake 23, this percentage of concordance is considered reasonable for reproducibility studies involving this type of instrument (food frequency questionnaire) and individuals at this stage of life 9. The values obtained in the present study are similar to those found by Vereecken & Maes 9 in a study conducted in Belgium.
The correlations observed in this study are consistent with those from other studies among adolescents. In Rockett et al. 5, the Pearson correlation coefficients ranged from 0.26 for protein and iron to 0.58 for calcium. The correlations observed by Speck et al. 24 ranged from 0.08 to 0.76, with a two-week interval between applications of the questionnaire. Field et al. 25, assessing the performance of a frequency questionnaire applied with a one-year interval, found correlations from 0.18 to 0.47. Robison et al. 26 reported correlation coefficients ranging from 0.44 (protein) to 0.52 (fat).
The time frame is also of concern. The recommendation is that the replication interval should be neither too short (to the point that subjects are able to remember and reproduce the answers by means of a learning process) nor too long (in the sense that it may reflect true changes in diet). In our study, the AFFQ was repeated after a three-month interval, which is considered reasonable 8. However, the time reference can also reflect changes in intake due to seasonality, which may have happened, lowering the true correlations, especially for nutrients in which fruits and vegetables are the main source, like vitamin C and retinol.
Bland Altman showed a trend towards larger difference in energy according to increased intake values and a bias towards extreme values for fat intake. Although these plots are also recommended to graphically investigate potential bias when comparing a method's repeatability, they have been used more extensively in diet validation studies than in reproducibility studies.
To produce reliable estimates of habitual intake by means of a food frequency questionnaire, a crucial ability is to deal with abstract concepts and form a mental image of the diet that is as close as possible to the truth 27. According to Goodwin et al. 28, children 13 to 17 years of age were capable of completing the food frequency questionnaire without their parents' help and to satisfactorily provide dietary information. Likewise, Field et al. 25 found better reproducibility in sixth and seventh-grade as compared to fourth and fifth-grade students (even if the students were the same age). According to Drewnowski 27, the mental image of the habitual diet is also influenced by the foods that individuals like or dislike, thereby becoming a measurement of their attitude towards their own diet. This may introduce errors that increase the variability of the responses and produce lower correlations.
Specific self-reported food intakes are generally higher in the first interview than in the second, as pointed out by McPherson et al. 29. The current study was also consistent with this observation. Cullen & Zakeri 30, in a reproducibility study that considered ethnicity, also found higher values in the first interview, as did Robinson et al. 26 in their study to assess a food frequency questionnaire in adolescent girls in the United Kingdom. Nevertheless, a reproducibility test by Rockett et al. 5 reported similar mean values for energy in the first and second interviews.
Because of the correlation identified between the energy and nutrient intakes in the study population, ICCs were calculated before and after adjustment for energy. After adjustment, all coefficients decreased in relation to the crude value. Yaroch et al. 31 observed the same downward trend. Adjustment for energy would increase the correlation coefficients when the variability in nutrient intake is related to energy intake, but would decrease when the nutrient variability depends on systematic errors of overestimation or underestimation 11. The foods mentioned by adolescents may have low nutritional value (soft drinks, sugar, and fried foods) and high caloric value, thereby favoring a tendency towards exaggerating the diet's nutrient content. Thus, such disproportional results may affect the AFFQ's concordance 4.
Livingstone et al. 32, in a validation study in the early 1990s, called attention to bias caused by lack of compliance (resulting from irritation and boredom) in completing food questionnaires. All survey methods ultimately depend on subjects' motivation, compliance, and ability to accurately report their habitual food intake. Although great effort was made in this study to ensure adolescents' compliance and motivation, we cannot be rule out that some may have gotten bored while filling out the questionnaire.
To our knowledge, this is the first food questionnaire developed and validated in Brazil that accesses this age group. The study complements the assessment of this questionnaire. Although good reproducibility does not ensure validity, reliability studies are often conducted, since they provide information on the answers' consistency and stability 10. The questionnaire's performance differed according to the nutrients, as expected. The worst results were for carbohydrates, total fat, and iron. Carbohydrates and cholesterol also showed the worst classification agreement according to intake tertiles. Even so, the low percentage of individuals classified in opposite tertiles (11.3%) called our attention. It is well known that errors are inherent to any assessment of dietary status. Since epidemiological studies rely on food frequency questionnaires because of the rapid application and low cost, evidence of likely misclassification of individuals based on these dietary measures is essential for our understanding of diet-related disease risk.
As an age group, adolescents have characteristics that need to be taken into account when applying such a research instrument. They need to be highly motivated and have enough time to fill out the questionnaire. In order to enhance their ability to adequately record habitual diet, recall strategies and recall cues can be applied 33,34.
In conclusion, a food frequency questionnaire for adolescents that was developed and previously validated achieved reasonable reproducibility and can be used for studies that aim to classify groups into low, medium, and high consumers.
Contributors
D. M. L. Marchioni was responsible for the statistical analysis, interpretation of the results, and drafting of the manuscript. S. M. Voci, R. M. Fisberg, and F. E. L. Lima collaborated in the interpretation and discussion of the results. B. Slater was responsible for the research design, implementation, and consultancy.
References
1. Centers for Disease Control and Prevention. Update: prevalence of overweight among children, adolescents, and adults United States, 1988-1994. MMWR Morb Mortal Wkly Rep 1997; 46:199-202.
2. Amorim Cruz JA. Dietary habits and nutritional status in adolescents over Europe Southern Europe. Eur J Clin Nutr 2000; 54(1 Suppl):29-35.
3. Lietz G, Barton KL, Longbottom PJ, Anderson AS. Can the EPIC food-frequency questionnaire be used in adolescent populations? Public Health Nutr 2002; 5:783-9.
4. World Health Organization. Expert consultation on diet, nutrition and the prevention of chronic diseases. Geneva: World Health Organization; 2003.
5. Rockett HRH, Wolf AM, Colditz GA. Development and reproducibility of a food frequency questionnaire to assess diets of older children and adolescents. J Am Diet Assoc 1995; 95:336-40.
6. Domel SB, Baranowski T, Leonard SB, Davis H, Riley P, Baranowski J. Accuracy of fourth- and fifth-grade students' food records compared with school-lunch observations. Am J Clin Nutr 1994; 59 Suppl:218-20.
7. Andersen LF, Nes M, Lillegaard IT, Sandstad B, Børneboe G-Eaa, Drevon CA. Evaluation of a quantitative food frequency questionnaire used in a group of Norwegian adolescents. Eur J Clin Nutr 1995; 49:543-54.
8. Livingstone MB, Robson PJ. Measurement of dietary intake in children. Proc Nutr Soc 2000; 59:279-93.
9. Vereecken CA, Maes L. A Belgian study on the reliability and relative validity of Health Behavior in School-Aged Children food-frequency questionnaire. Public Health Nutr 2003; 6:581-8.
10. Nelson M. The validation of dietary assessment. In: Margetts B, Nelson M, editors. Design Concepts in Nutrition Epidemiology. 2nd Ed. Oxford: Oxford University Press; 1997. p. 241-72.
11. Willet WC. Nutritional Epidemiology. 2nd Ed. Oxford: Oxford University Press; 1998.
12. Szklo M, Javier Nieto F. Epidemiology: beyond the basics. Gaithersburg: Aspen Publisher; 2000.
13. Block G, Hartman AM. Issues in reproducibility and validity of dietary studies. Am J Clin Nutr 1989; 50:1133-8.
14. Litwin MS. How to measure survey reliability and validity. London: Sage Publications; 1995.
15. Slater B, Philippi ST, Fisberg RM, Latorre MRDO. Validation of a semi-quantitative adolescent food frequency questionnaire applied at a public school in Sao Paulo, Brazil. Eur J Clin Nutr 2003; 57: 629-35.
16. Nuzzo L. Avaliação do estado nutricional de adolescentes de uma instituição particular de ensino [Master's Thesis]. São Paulo: Faculdade de Saúde Pública, Universidade de São Paulo; 1998.
17. United States Department of Agriculture. USDA national nutrient database for standard reference, release 16 [food composition table]. http://www.nal.usda.gov/fnic/foodcomp (accessed on 28/Jan/2004).
18. Willett WC, Stampfer MJ. Total energy intake: implications for epidemiological analyses. Am J Epidemiol 1986; 124:17-27.
19. Bland JM, Altman DG. Measuring agreement in method comparison studies. Stat Methods Med Res 1999; 8:135-60.
20. Armstrong BK, White E, Saracci R. Principles of Exposure Measurement in Epidemiology. 2nd Ed. Oxford: Oxford University Press; 1995.
21. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics 1977; 33:159-74.
22. Hansson LM, Galanti MR, Bergström R. Factors affecting reproducibility of dietary reports using food frequency questionnaires. Eur J Clin Nutr 2000; 54:658-64.
23. Slater B. Desenvolvimento e validação de um questionário semi-quantitativo de freqüência alimentar para adolescentes [Doctoral Dissertation]. São Paulo: Faculdade de Saúde Pública, Universidade de São Paulo; 2001.
24. Speck BJ, Bradley CB, Harrell JS, Belyea MJ. A food frequency questionnaire for youth: psychometric analysis and summary of eating habits in adolescents. J Adolesc Health 2001; 28:16-25.
25. Field AE, Peterson KE, Gortmaker SL, Cheung L, Rockett H, Fox MK, et al. Reproducibility and validity of a food frequency questionnaire among fourth to seventh grade inner-city school children: implications of age and day-to-day variation in dietary intake. Public Health Nutr 1999; 2:293-300.
26. Robison S, Skelton R, Barker M, Wioman C. Assessing the diet of adolescent girls in the UK. Public Health Nutr 1999; 2:571-7.
27. Drewnowski A. Diet image: a new perspective on the food-frequency questionnaire. Nutr Rev 2001; 59:370-2.
28. Goodwin RA, Brule D, Junkins EA, Dubois S, Beer-Borts S. Development of a food and activity record and a portion-size model booklet for use by 6 to 17 year olds: a review of focus group testing. J Am Diet Assoc 2001; 101:926-8.
29. McPherson R, Hoelscher D, Alexander M, Scanlon K, Serdula M. Dietary assessment methods among school-aged children: validity and reliability. Prev Med 2000; 31 Suppl:11-33.
30. Cullen KW, Zakeri I. The youth/adolescent questionnaire has low validity and modest reliability among low-income African-American and Hispanic seventh- and eighth-grade youth. J Am Diet Assoc 2004; 104:1415-9.
31. Yaroch AL, Resnicow K, Davis M, Davis A, Smith M, Khan LK. Development of a modified picture-sort food frequency questionnaire administered to low-income, overweight, African-American adolescent girls. J Am Diet Assoc 2000; 100:1050-6.
32. Livingstone MBE, Prentice AM, Coward WA, Strain JJ, Black AE, Davies PSW, et al. Validation of estimates of energy intake by weighted dietary record and diet history in children and adolescents. Am J Clin Nutr 1992; 56:29-35.
33. Johnson-Kozlow M, Matt GE, Rock CL. Recall strategies used by respondents to complete a food frequency questionnaire: an exploratory study. J Am Diet Assoc 2006; 106:430-3.
34. Matt GE, Rock CL, Johnson-Kozlow M. Using recall cues to improve measurement of dietary intakes with a food frequency questionnaire in an ethnically diverse population: an exploratory study. J Am Diet Assoc 2006; 106:1209-17.
Correspondence:
B. Slater
Departamento de Nutrição
Faculdade de Saúde Pública
Universidade de São Paulo
Av. Dr. Arnaldo 715, São Paulo, SP
01246-904, Brasil
bslater@usp.br
Submitted on 29/Dec/2005
Final version resubmitted on 11/Dec/2006
Approved on 12/Jan/2007