ARTIGO ARTICLE

Assessing the validity of a food frequency questionnaire among low-income women in São Paulo, southeastern Brazil

Avaliação de validade de um questionário de freqüência alimentar em mulheres de baixa renda residentes em São Paulo, Brasil

Marly Augusto Cardoso; Luciana Yuki Tomita; Elaine Cristina Laguna

Faculdade de Saúde Pública, Universidade de São Paulo, São Paulo, Brasil.

Correspondence

ABSTRACT

RESUMO

Este estudo compara a validade de questionário de freqüência alimentar (QFA) em 93 mulheres participantes de estudo caso-controle em São Paulo, Brasil. Dois QFA (12 meses de intervalo entre QFA1 e QFA2) e 3 recordatórios de 24 horas (R24h) foram obtidos entre 2003 e 2004. Coeficientes de correlação de Pearson (brutos, ajustados pelas calorias totais e com correção intra-indivíduos) foram utilizados na comparação entre os métodos. A concordância entre QFA e R24h foi avaliada pela análise de Bland-Altman. Na avaliação de reprodutibilidade em longo prazo, coeficientes de correlação intraclasse ajustados pelas calorias totais foram próximos de 0,40, com valores maiores para vitamina A e folato (0,50-0,56). Coeficientes de correlação de Pearson ajustados pelas calorias totais e de-atenuados variaram de 0,30-0,54 para macronutrientes a 0,20-0,48 para micronutrientes, com valores maiores para cálcio (0,75). Houve uma pequena proporção de erro de classificação, e a análise de Bland-Altman sugere que o QFA é adequado para a classificação do consumo de nutrientes para o grupo de indivíduos estudado.

Consumo de Alimentos; Estudos de Validação; Questionários; Mulheres

Introduction

In epidemiologic studies, self-reported dietary intake is often used to establish diet-disease associations. The food frequency questionnaire (FFQ) has been the dietary assessment method most frequently used in large-scale studies, primarily because it is easy to administer, is less expensive than other dietary assessment methods, and provides a rapid estimate of usual intake ¹.

Over the years, investigators have recognized that the reported values from FFQ are subject to substantial error, both systematic and random ².

A limited number of validation studies of FFQ have been conducted in developing countries. People from low-income households have less nutritionally adequate diets, living for long periods of time on limited incomes, with lower literacy, numerical and language skills ³. Issues such as questionnaire format, participant motivation, perceived study burden, and repeated administration of the same instrument can each influence our ability to gather reliable and valid self-report dietary intake data ⁴. Results usually show a good agreement in test-retest for foods and nutrients, but with high variability in the dietary intake ⁵. Thus, rather than evaluating the validity of a questionnaire against a "gold standard", it has been discussed whether and by how much the measurement characteristics of dietary assessment instruments differ, especially in research designed to address dietary quality, food inequality or diet as part of health inequalities ³.

We developed a FFQ to assess dietary intake in an urban female population for use in a case control study designed to investigate the relationship between dietary factors, serum vitamin concentrations and cervical cancer in São Paulo city, Brazil. The present study illustrates an assessment of validity of an FFQ with 24hR, using different statistical approaches: Pearson correlation, Bland-Altman method and joint classification analysis.

Subjects and methods

Study population and design

Participants were from the Brazilian Investigation into Nutrition and Cervical Cancer Prevention (BRINCA) Study, a case-control hospital-based study designed to investigate the relationship between dietary factors, serum vitamin concentrations and cervical cancer in São Paulo city. The exclusion criteria were pregnancy, breastfeeding, hysterectomy, positive test for HIV, bleeding, mental disturbance, and radiotherapy or chemotherapy treatments. Between March 2003 and May 2005, 1,676 women completed the study protocol by filling out a questionnaire in three major public hospitals: the Brazilian Institute for Cancer Research, the Leonor Mendes de Barros Hospital and Pérola Byngton Hospital. In the main study reported elsewhere ⁶, we aimed to recruit prospectively 453 controls and 497 newly diagnosed cases of cervical intraepithelial neoplasia. Eligible women were residents of São Paulo aged 21-65 years and had no prior hysterectomy, previous treatment for cervical neoplasia or cancer history. During the same period, control women were selected from among those attending screening in the same clinics where cases were diagnosed. Cases and controls were invited to participate in the study and interviewed before the colposcopy examination to minimize differential recall bias. The Institutional Review Board at the School of Public Health of the University of São Paulo and the Medical Ethical Committees of all participating hospitals approved the study protocol, and a written informed consent was obtained from each participant.

Participants of the BRINCA Study were invited to take part in the validation study of the FFQ. The sample size for the validation study was calculated using a standard formula for correlation coefficients ⁷, and the number of subjects required was about 100. However, a random sub-sample of 145 participants from the main study was invited for this study to cover losses or potential incomplete responses over time. One of the authors (E.C.L.) contacted the participants within 2 weeks after the first FFQ interview at the hospital (FFQ1). Overall, three 24hR and two FFQ interviews (12 months apart) were conducted between 2003 and 2004 to estimate dietary intake during the past year. We collected three dietary recalls, conducted by telephone from each participant. Between September and November 2003, 145 participants completed the first dietary recall. Six months later, 119 (82%) completed the second dietary recall and 93 (64%) participants completed the third dietary recall within an interval of approximately 6 and 12 months of the first dietary recall, respectively. Of the initial study sample, 28 subjects (19% of 145) had moved or changed phone number, 22 (15%) were not contacted after 5 calls including weekend days and 2 (1%) refused to participate. After finishing the last dietary recall, the subjects completed the second FFQ (FFQ2). Participants underwent the standardized telephone interviews by trained nutritionists (E.C.L. and L.Y.T.) blinded to the group assignment in the main study. Women reported in household measures the dietary intake from breakfast of the day before till breakfast the next morning, following the FFQ2 questions on habitual food intake over the past year. Thus, for the present analysis, the validity of the FFQ was assessed among 93 participants (64 controls and 29 cases) from the main study.

Self-administered FFQ

All participants answered a short version of a FFQ previously validated for use with Japanese-Brazilians ^8,9 and adapted to be used at the primary health care level ¹⁰. Women were asked about usual frequency of consumption of 76 foods items and their portion sizes, an open-ended food section, and vitamin and mineral supplements during the previous year. Questions concerning use of sauce, frequency of intake of visible fat and type of fat used in cooking procedures were also asked. Nutrients were analyzed using the Dietsys software version 4.01 (National Cancer Institute, Bethesda, USA) ¹¹. The nutrient database was based primarily on the U.S. Department of Agriculture (USDA) publications supplied by Dietsys and supplemented by the Brazilian Standard Food Composition Table¹².

24-hour diet recalls

The 24hR had been led by telephone, getting information on recipes, portion size and volume. A standardized script was used for the dietary recall by telephone, as developed in a previous study ¹³. The database of the World Food Dietary Assessment System (WFood version 2.0; Office of Technology Licensing, University of California Berkeley, Berkeley, USA) was used to estimate the nutrient composition of diets. The nutrient database was based on the USDA publications supplied by WFood.

Statistical analysis

Means and standard deviations (SD) were calculated for total nutrient intake from each administration of the FFQ and 24hR. Since most nutrient distributions were skewed toward higher values, all variables were natural-log (ln) transformed before statistical analysis. Subsequently, to examine the correlation due to total energy intake we calculate energy-adjusted nutrient intake, which indicate the nutrient composition of the diet by regressing nutrient intake on total energy ⁷. As a measure of long-term reliability, Pearson intra-class correlation coefficients were used to compare the two FFQs for both unadjusted and energy-adjusted nutrients ³.

The validity of the FFQ was evaluated by comparing mean nutrient intake data obtained from FFQ2 with mean nutrient intake data obtained from three dietary recalls. The mean of FFQ2 was used for validation analysis since the reference period for FFQ1 did not correspond to the same time period covered by the dietary recall. The validation approach for the FFQ in this study used three methods: Pearson correlation, Bland-Altman and joint classification analysis.

Pearson correlation coefficients (crude and energy-adjusted) were used to assess the association between nutrient intake estimates from the second FFQ and mean of three dietary recalls, covering the same 12-month period. Attenuation of the Pearson correlation coefficients between 24hR and FFQ, caused by intra-individual day-to-day variation in the nutrient intakes among subjects, was corrected by taking into account the within-to-between person variance ratio calculated from 3 days of 24hR. We deattenuated the crude correlation coefficients by multiplying them with the factor [1 + ( s²_intra/s²_inter )/n]^1/2, where n is the number of repeated 24-hour dietary recall, s²_intra is the intra-individual variance and s²_inter is the inter-individual variance between the 24hR ⁷.

Bland-Altman limits of agreement (LOA) ¹⁴ were used to evaluate the level of agreement between the two dietary methods across the range of intakes. The LOA define the limits within which 95% of the differences are expected to fall (mean ± 2 SD of the differences). Ln transformation was performed since the dietary data were skewed. As described by Flood et al. ¹⁵, after energy adjustment and ln transformation of the data, the difference in nutrient intake between the two methods (lnFFQ-lnDR) was plotted against mean of paired intake values [(lnFFQ + lnDR)/2]. The antilogs of the LOA were then taken, providing a ratio FFQ/dietary recall of the data. The ratios were multiplied by 100 expressing percentages, with 100% representing perfect agreement. The dependency between the two methods was tested by fitting the regression line of differences: if the two methods are equally variable, the correlation between the differences would equal zero.

Misclassification error was assessed by dividing mean intake values from the FFQ and dietary recall into quintiles. Then the percentage of agreement between the two methods (classification in the same quintile) was estimated. Quadratic-weighted kappa statistics were used for comparison across quintiles of nutrient intake from FFQ and dietary recall.

All statistical analyses were performed using the SPSS software version 12.0 (SPSS Inc., Chicago, USA).

Results

The mean (SD) age, educational level, weight, height and body mass index (BMI) of the 93 participants were respectively: 43.3 (10.3) years old, 7.4 (3.3) years of education, 69.4 (14.1) kg, 157.5 (6.0) cm and 28.0 (5.7) kg/m². The median (interquartile ranges) of monthly income per person was USD 70.0 (43.0; 108.0). Half of the participants were non-white (54.8% black and mulato), married (74%) and housewives or unemployed (51%). The regular use of vitamin and mineral supplements was reported by 12% of the participants. Overall, 45% of the study women reported habitual alcohol consumption; the median (interquartile ranges) intake levels among alcohol drinkers was 20.9 (2.0; 51.0) g/day. When comparing participants with non-participants of the validation study (n = 52), significant differences were not observed (mean age 38 years, SD = 10; mean years of education 8.5, SD = 3.6; median income per person: USD 77/month; controls = 31 and potential cases = 21).

Table 1 shows the average and standard deviation of the nutrient intakes estimated by FFQ1 and FFQ2, excluding supplements. The intra-class correlations measuring the reproducibility of the unadjusted nutrients from the FFQ spaced one-year apart ranged from 0.33 for vitamin B₁₂ to 0.59 for cholesterol and 0.58 for energy. Adjustment of the nutrient intake for total energy before testing reproducibility slightly decrease almost all correlations, with the exception of fiber, vitamins A, C and B12, folate and calcium. The higher energy-adjusted intra-class correlation coefficients were observed for folate and vitamin A (0.50 and 0.56, respectively).

Table 2 presents the mean values of nutrient estimates from the three dietary recalls and the second FFQ with validity parameters obtained from three different statistical approaches. After correction for within-person variation, the de-attenuated Pearson correlation coefficients were slightly higher than the energy-adjusted values, ranging from 0.24 to 0.54 for most nutrients. The worse coefficient values were observed for vitamins A (-0.003), E (0.16) and B₁₂ (0.05); the higher values were observed for calcium (0.75), fiber (0.54) and folate (0.48). The mean agreement between the FFQ and the dietary recalls ranged from 87% for fiber to 218% for vitamin E. Although the mean agreement was around 100% for most nutrients, the FFQ could under or overestimate dietary recall. The narrowest LOA was found for carbohydrate and the widest for vitamin E. On the other hand, weighted kappa values mostly showed good agreement for nutrient categories (around 80%); the highest proportion of subjects correctly classified was observed for calcium (88.2%).

Figure 1 and 2 illustrate the Bland-Altman analysis for calcium and vitamin E, respectively, nutrients with the best and worst validity parameters. A fitted regression line indicated a significant linear trend (p < 0.001) for both nutrients, even using ln transformation of the skewed data. That is, a dependency existed between the difference of the two methods and the average of the two methods; across extreme intake levels higher the magnitude of the error between the FFQ and dietary recall can be expected. On the other hand, Figure 3 illustrates the Bland-Altman analysis for total fat with not so good energy-adjusted and de-attenuated Pearson coefficient (0.43) when compared with calcium (0.75). However, a good percentage agreement was observed for total fat (mean = 103, LOA 90-118) without significant linear tendency between the differences of the two methods across intake levels, that is, the two methods were equally variable.

Discussion

In light of the difficulty of women living with limited resources and low literacy to express their habitual food intake, the correlation results of our FFQ were mostly fair to good between repeated long-term administrations, suggesting a reliable method of assessing nutrient intake in our study population. For validity purposes, our results suggest that the FFQ is a valid tool, based on Bland-Altman plots and correlation coefficients, for protein, total fat, carbohydrate, cholesterol, fiber, vitamin C, riboflavin, folate, calcium and magnesium (with energy-adjusted and de-attenuated correlations higher than 0.30 for nutrients comparing the FFQ with three 24hR). The high proportion of nutrient intakes classified within one quintile suggests that the FFQ is adequately ranking the nutrient intake of the study group. Nevertheless, caution should be taken for use of this FFQ among women with similar characteristics since the Bland-Altman limits of agreement illustrate that the FFQ can overestimate or underestimate consumption across intake levels for the following nutrients: vitamins A, E, B6, B₁₂, thiamin and zinc.

The long-term reliability (1-year interval) in this study was lower for all nutrients (ranging from 0.30 to 0.56) when compared with the short-term reliability (ranging from 0.52 to 0.75, within 1-month interval) observed for the original version of this FFQ in Japanese-Brazilian women with higher education levels ⁹. However, it is worth noting that the results for test-retest of the FFQ in the present study are even better than those from previous research on low-income Brazilian workers carried out by Fornés et al. ⁵. In that study, involving 104 women and men aged 18 to 60 years, the energy-adjusted intra-class correlation coefficients ranged from 0.23 to 0.34 in a 5-month interval between the two FFQ, suggesting poor agreement for reproducibility. As expected, the long-term reliability of a FFQ is generally lower than the short-term reliability since real changes in diet could occur besides the memory bias ¹⁶.

In the present study, we found lower correlations after energy adjustment for most nutrients in the validation analysis, except for vitamin C and folate, which habitual dietary sources seem to be less dependent on total energy intake. Similar results were found by Slater et al. ¹⁷ in a validation study of an adolescent food frequency questionnaire applied at a public school in São Paulo. These lower correlations may be explained by an increase in correlated measurement error as a consequence of controlling for total energy intake. Therefore, additional information on Bland-Altman limits of agreement is useful to evaluate the level of agreement between the two dietary methods across the range of intakes and the assessment of misclassification error as well. The differences between the FFQ and dietary recall for estimation of energy intake were previously reported and can be explained by underreporting in the dietary recall ^18,19.

Previous validation studies in Brazilian women have not reported relevant information on Bland-Altman analyses ^20,21. As suggested in a consensus statement on methods for assessing FFQ ¹, Bland-Altman limits of agreement would illustrate whether the FFQ is overestimating or underestimating nutrient consumption across intake levels. In this study, correlation coefficients were very low with Bland-Altman limits of agreement too wide for vitamin A, E and B₁₂, indicating the difficulty in estimating such nutrients because they may be highly concentrated in some foods. In earlier validation studies carried out in Latin-American countries, similar findings have been reported ^22,23. The inclusion of more days of recall would possibly improve these results ²³. Though, for nutrients such as vitamin E, even 12 days of diet recording may be insufficient to measure accurately their habitual intakes ⁹.

The main limitation in the present study is the use of dietary recalls as a reference method since, like FFQ, relies on memory, ensuing in errors such as under- or over-estimation of food consumption. As shown in published data of other validation studies conducted in adult populations in both developed and developing countries ^24,25, the dietary records were used as the gold standard in most of the validation

studies. Dietary records are likely to have the least correlated errors, whereas dietary histories as gold standards are considered the least appropriate ²⁵. However, repeated 24hR are less demanding for participants with low literacy. Thus, multiple 24hR were shown to be the most appropriate method for studies of diet and nutrition including subjects with low-income and/or low-literacy ³.

In conclusion, depending on the use of the nutrient intake estimates, different methods of analysis should be used in assessing the validity of a FFQ in similar populations, providing information about the performance of such method across different levels of nutrient intakes. Although our FFQ was reasonable for ranking individuals according to their nutrient intakes, absolute consumption should be used with caution.

Contributors

M. A. Cardoso participated in the planning of the research; data analysis and interpretation; and writing up of the article. L. Y. Tomita collaborated in data collection and procession; analysis and interpretation of results; and revision of the article. E. C. Laguna participated in the collection and processing of the date; analysis of results and final revision of the article.

Acknowledgments

To São Paulo State Research Foundation (FAPESP) and National Research Council (CNPq) for financial support. To the Brazilian Investigation into Nutrition and Cervical Cancer Prevention Study Team (BRINCA study): L. S. Aguiar, L. C. Almeida, V. D'Almeida, V. A. F. Alves, M. A. A. Andreoli, M. C. Costa, R. L. R. Costa, A. M. Francisco-Bagnariolli, E. L. Franco, M. G. M. C. Ferraz, A. Longatto Filho, J. S. Pereira Sobrinho, J. C. M. Prado, M. D. Ricci, C. Roteli-Martins, M. A. Sperança, M. L. Utagawa, L. L. Villa.

References

1. Cade J, Thompson R, Burley V, Warm D. Development, validation and utilization of food-frequency questionnaires: a review. Public Health Nutr 2002; 5:567-87.         

2. Liu K. Statistical issues related to semiquantitative food-frequency questionnaires. Am J Clin Nutr 1994; 59(1 Suppl):262S-5S.         

3. Vucic V, Glibetic M, Novakovic R, Ngo J, Ristic-Medic D, Tepsic J, et al. Dietary assessment methods used for low-income populations in food consumption surveys: a literature review. Br J Nutr 2009; 101 Suppl 2:S95-101.         

4. Thomson CA, Giuliano A, Rock CL, Ritenbaugh CK, Flatt SW, Faerber S, et al. Measuring dietary change in a diet intervention trial: comparing food frequency questionnaire and dietary recalls. Am J Epidemiol 2003; 157:754-62.         

5. Fornés NS, Stringhini MLF, Elias BM. Reproducibility and validity of a food-frequency questionnaire for use among low-income Brazilian workers. Public Health Nutr 2003; 6:821-7.         

6. Tomita LY, Longatto-Filho A, Costa MC, Andreoli MAA, Villa LL, Franco EL, et al. Diet and serum micronutrients in relation to cervical neoplasia and cancer among low-income Brazilian women. Int J Cancer 2010; 126:703-14.         

7. Willett WC. Nutritional epidemiology. 2^nd Ed. Oxford: Oxford University Press; 1998.         

8. Tomita LY, Almeida LC, Roteli-Martins C, D'Almeida V, Cardoso MA. Dietary predictors of serum total carotene in low-income women living in São Paulo, south-east Brazil. Public Health Nutr 2009; 12:2133-42.         

9. Cardoso MA, Kida AA, Tomita LY, Stocco PR. Reproducibility and validity of a food frequency questionnaire among women of Japanese ancestry living in Brazil. Nutr Res 2001; 21:725-33.         

10. Sartorelli DS, Sciarra EC, Franco LJ, Cardoso MA. Beneficial effects of short-term nutritional counseling at the primary health-care level among Brazilian adults. Public Health Nutr 2005; 8:820-5.         

11. Block G, Coyle LM, Hartman AM, Scoppa SM. Revision of dietary analysis software for the Health Habits and History Questionnaire. Am J Epidemiol 1994; 139:1190-6.         

12. Instituto Brasileiro de Geografia e Estatística. Tabela de composição de alimentos. 4ª Ed. Rio de Janeiro: Instituto Brasileiro de Geografia e Estatística; 1996.         

13. Tomita LY, Cardoso MA. Avaliação da lista de alimentos e porções alimentares de Questionário Quantitativo de Freqüência Alimentar em população adulta. Cad Saúde Pública 2002; 18:1747-56.         

14. Bland JM, Altman DG. Statistical methods for assessing agreement between two methods of clinical measurement. Lancet 1986; 1:307-10.         

15. Flood VM, Smith WT, Webb KL, Mitchell P. Issues in assessing the validity of nutrient data obtained from a food-frequency questionnaire: folate and vitamin B12 examples. Public Health Nutr 2004; 7:751-6.         

16. Goldbohm RA, van't Veer P, van den Brandt PA, van't Hof MA, Brants HA, Sturmans F, et al. Reproducibility of a food frequency questionnaire and stability of dietary habits determined from five annually repeated measurements. Eur J Clin Nutr 1995; 49:420-9.         

17. Slater B, Philippi ST, Fisberg RM, Latorre MR. Validation of a semi-quantitative adolescent food frequency questionnaire applied at a public school in São Paulo, Brazil. Eur J Clin Nutr 2003; 57:629-35.         

18. Verkleij-Haoort AC, Vries JHM, Stegers MPG, Lindemans J, Ursem NTC, Steegers-Theunissen RPM. Validation of the assessment of folate and vitamin B12 intake in women of reproductive age: the method of triads. Eur J Clin Nutr 2007; 61:610-5.         

19. Scagliusi FB, Ferriolli E, Pfrimer K, Laureano C, Cunha CS, Gualano B, et al. Underreporting of energy intake in Brazilian women varies according to dietary assessment: a cross-sectional study using doubly labeled water. J Am Diet Assoc 2008; 108:2031-40.         

20. Lima FEL, Slater B, Latorre MRDO, Fisberg RM. Validade de um questionário quantitativo de frequência alimentar desenvolvido para população feminina no Nordeste do Brasil. Rev Bras Epidemiol 2007; 10:483-90.         

21. Ishihara J, Iwasaki M, Kunieda CM, Hamada GS, Tsugane S. Food frequency questionnaire is a valid tool in the nutritional assessment of Brazilian women of diverse ethnicity. Asia Pac J Clin Nutr 2009; 18:76-80.         

22. Ortega MI, Valencia ME. Measuring the intakes of foods and nutrients of marginal populations in north-west Mexico. Public Health Nutr 2002; 5:907-10.         

23. Bautista LE, Herrán OF, Pryer JA. Development and simulated validation of a food-frequency questionnaire for the Colombian population. Public Health Nutr 2005; 8:181-8.         

24. Henríquez-Sánchez P, Sánchez-Villegas A, Doreste-Alonso J, Ortiz-Andrellucchi A, Pfrimer K, Serra-Majem L. Dietary assessment methods for micronutrient intake: a systematic review on vitamins. Br J Nutr 2009; 102 Suppl 1:S10-37.         

25. Serra-Majem L, Pfrimer K, Doreste-Alonso J, Ribas-Barba L, Sánchez-Villegas A, Ortiz-Andrellucchi A, et al. Dietary assessment methods for intakes of iron, calcium, selenium, zinc and iodine. Br J Nutr 2009; 102 Suppl 1:S38-55.         

Correspondence
M. A. Cardoso
Departamento de Nutrição, Faculdade de Saúde Pública, Universidade de São Paulo
Av. Dr. Arnaldo 715,
São Paulo, SP 01246-904, Brasil
marlyac@usp.br

Submitted on 09/Nov/2009
Final version resubmitted on 11/Aug/2010
Approved on 19/Aug/2010