Abstract

The study aimed to evaluate relative validity and reproducibility of seven WHO indicators of dietary practices in children aged 6-23.9 months. Data from probabilistic sample of children who used primary healthcare services in Rio de Janeiro, Brazil were collected using a 24h dietary recall (24HR) and a closed questionnaire (Q1) on feeding in the day before the study. The last one was reapplied (Q2) around 16 days later. Validity was assessed by comparing the prevalence rates estimated by 24HR and Q1 and calculating the positive (PPV) and negative (NPV) predictive values, sensitivity (Se), specificity (Sp), and accuracy index (AI) for the resulting indicators. For reproducibility, estimated prevalence rates based on Q1 and Q2 were compared and the kappa index and prevalence-adjusted bias-adjusted kappa were estimated. Of the seven estimated indicators, the prevalence of two was overestimated (Continued breastfeeding: 50.0% vs 40.0%; Sweet beverage consumption: 65.1% vs 52.7%) and the prevalence of one was underestimated (Zero vegetable or fruit consumption: 6.5% vs 18.1%). For most indicators, Se and PPV were higher than Sp and NPV. The prevalence rates determined with Q1 and Q2 were similar for 6 indicators. More than half showed good, very good or excellent agreement.

Key words:
Data accuracy; Validity; Reproducibility; Breastfeeding; Complementary feeding

Resumo

O estudo avaliou a validade relativa e a reprodutibilidade de sete indicadores da OMS sobre alimentação de crianças de 6-23,9 meses. Dados de amostra probabilística de usuários de serviços básicos de saúde na cidade do Rio de Janeiro, Brasil, foram coletados por meio de recordatório alimentar de 24 horas (R24h) e questionário fechado (Q1) sobre alimentação no dia anterior ao estudo. Este último foi reaplicado (Q2) em torno de 16 dias depois. A validade foi avaliada comparando-se as prevalências estimadas pelo R24h e Q1 e calculando-se os valores preditivos positivo (VPP) e negativo (VPN), sensibilidade (Se), especificidade (Esp) e índice de acurácia (IA) dos indicadores resultantes. Para reprodutibilidade, as prevalências estimadas com base em Q1 e Q2 foram comparadas e estimados o índice kappa e o kappa ajustado pela prevalência. Dos sete indicadores estimados, houve superestimação da prevalência de dois (aleitamento continuado: 50,0% versus 40,0%; consumo de bebidas adoçadas: 65,1% vs. 52,7%) e subestimação da prevalência de um (não consumo de frutas e hortaliças: 6,5% vs. 18,1%). Para a maioria deles, Se e VPP foram maiores do que Esp e VPN. As prevalências determinadas com Q1 e Q2 foram semelhantes para seis indicadores. Mais da metade dos indicadores apresentaram concordância boa, muito boa ou excelente.

Palavras-chave:
Acurácia; Validade; Reprodutibilidade; Amamentação; Alimentação complementar

Introduction

Feeding plays a fundamental role in adequate growth and development in early childhood¹1 World Health Organization (WHO). Essential nutrition actions: improving maternal, newborn, infant and young child health and nutrition. Geneva: WHO; 2013.. Given the impact of feeding on infant morbidity and mortality²2 Black RE, Allen LH, Bhutta ZA, Caulfield LE, de Onis M, Ezzati M, Mathers C, Rivera J, Maternal and Child Undernutrition Study Group. Maternal and child undernutrition: global and regional exposures and health consequences. Lancet 2008; 371(9608):243-260. and the influence of childhood feeding practices on health in adulthood³3 Miranda EP, Leila M, Santana P, Costa J, Pitangueira D, Assis AM. Age of introduction of complementary feeding and overweight in adolescence and adulthood: a systematic review. Matern Child Nutr 2019; 15(3):e12796.^,44 Victora CG, Adair L, Fall C, Hallal PC, Martorell R, Richter L, Sachdev HS, Maternal and Child Undernutrition Study Group. Maternal and child undernutrition: consequences for adult health and human capital. Lancet 2008; 371(9609):340-357, complementary feeding is an undeniably relevant theme for the public health agenda. This highlights the need for policies to promote healthy feeding in childhood to support and protect breastfeeding and adequate complementary feeding⁵5 Rollins NC, Bhandari N, Hajeebhoy N, Horton S, Lutter CK, Martines JC, Piwoz EG, Richter LM, Victora CG, Lancet Breastfeeding Series Group. Why invest, and what it will take to improve breastfeeding practices? Lancet 2016; 387(10017):491-504.

6 World Health Organization (WHO). Global strategy for infant and young child feeding. Geneva: WHO; 2003.^-77 World Health Organization (WHO). Guiding principles for feeding non-breastfed children 6-24 months of age. Geneva: WHO; 2005..

However, recommended feeding practices for children have not been achieved in various countries⁸8 White JM, Bégin F, Kumapley R, Murray C, Krasevec J. Complementary feeding practices: current global and regional estimates. Matern Child Nutr 2017; 13(Suppl. 2):e12505., including Brazil⁹9 Universidade Federal do Rio de Janeiro. Aleitamento materno: prevalência e práticas de aleitamento materno em crianças brasileiras menores de 2 anos 4: ENANI 2019 [internet]. 2021. [acessado 2022 ago 3]. Disponível em: https://enani.nutricao.ufrj.br/index.php/relatorios/
https://enani.nutricao.ufrj.br/index.php... ^,1010 Universidade Federal do Rio de Janeiro (UFRJ). Alimentação infantil i: prevalência de indicadores de alimentação de crianças menores de 5 anos: ENANI 2019 [Internet]. 2021. [acessado 2022 ago 3]. Disponível em: https://enani.nutricao.ufrj.br/index.php/relatorios/
https://enani.nutricao.ufrj.br/index.php... . The following practices are commonplace: early introduction of foods and beverages¹¹11 Fonseca PCA, Ribeiro SAV, Andreoli CS, de Carvalho CA, Pessoa MC, de Novaes JF, Priore SE, Franceschini SDCC. Association of exclusive breastfeeding duration with consumption of ultra-processed foods , fruit and vegetables in Brazilian children. Eur J Nutr 2018; 58(7):2887-2894.

12 Cordeiro L, Moreira DQ, Brauninger E, Lopes LHK, Bauleo ME, Sarno F. Introdução de alimentos complementares em lactentes. Einstein 2019; 17(3):eAO4412.

13 Zielinska MA, Rust P, Masztalerz-kozubek D, Bichler J, Hamulka J. Factors influencing the age of complementary feeding - a cross-sectional study from two European countries. Int J Environ Res Public Health 2019; 16(20):3799.^-1414 Ortega-Cisneros CM, Basto-Abreu A, Venegas-Andrade A, Rodriguez-Santaolaya P. Complementary feeding practices in Mexican healthy infants: how close are they to the current guidelines? Bol Med Hosp Infant Mex 2019; 76(6):265-272., low dietary diversity⁸8 White JM, Bégin F, Kumapley R, Murray C, Krasevec J. Complementary feeding practices: current global and regional estimates. Matern Child Nutr 2017; 13(Suppl. 2):e12505.^,1515 United Nations Children's Fund (UNICEF). Improving young children's diets during the complementary feeding period - UNICEF Programming Guidence 2020 [Internet]. 2020. [cited 2022 ago 3]. Available from: https://www.unicef.org/documents/improving-young-childrens-diets-during-complementary-feeding-period-unicef-programming
https://www.unicef.org/documents/improvi... ^,1616 Instituto Brasileiro de Geografia e Estatística (IBGE). Pesquisa Nacional de Saúde 2013: ciclos de cida - Brasil e grandes regiões. Rio de Janeiro: IBGE; 2015., low variety (or absence) of fruits and vegetables¹⁶16 Instituto Brasileiro de Geografia e Estatística (IBGE). Pesquisa Nacional de Saúde 2013: ciclos de cida - Brasil e grandes regiões. Rio de Janeiro: IBGE; 2015.^,1717 Gatien SAR, Mann LL, Kirk SFL. Vegetable/fruit intakes of young children at home and in childcare centres. Can J Diet Pract Res 2020; 81(1):15-20., and high consumption of ultra-processed foods¹⁸18 Moubarac J, Batal M, Louzada ML, Steele EM, Monteiro CA. Consumption of ultra-processed foods predicts diet quality in Canada. Appetite 2017; 1(108):512-520.^,1919 Neri D, Monteiro CA, Levy RB. Consumption of ultra-processed foods and its association with added sugar content in the diets of US children, NHANES 2009-2014. Pediatr Obes 2019; 14(12):e12563., even in the first year of life¹⁶16 Instituto Brasileiro de Geografia e Estatística (IBGE). Pesquisa Nacional de Saúde 2013: ciclos de cida - Brasil e grandes regiões. Rio de Janeiro: IBGE; 2015.^,2020 Giesta JM, Zoche E, Corr S, Bosa VL. Associated factors with early introduction of ultra-processed foods in feeding of children under two years old. Cien Saude Colet 2019; 24(7):2387-2398.

21 Batalha MA, França AKTDC, Conceição SIOD, Santos AMD, Silva FS, Padilha LL, Silva AAMD. Processed and ultra-processed food consumption among children aged 13 to 35 months and associated factors. Cad Saude Publica 2017; 33(11):e00152016.^-2222 Dallazen C, Silva SAD, Gonçalves VSS, Nilson EAF, Crispim SP, Lang RMF, Moreira JD, Tietzmann DC, Vítolo MR. Introdução de alimentos não recomendados no primeiro ano de vida e fatores associados em crianças de baixo nível socioeconômico. Cad Saude Publica 2018; 34(2): e00202816..

Dietary assessment in infant and toddlers is a challenge because feeding practices in this phase of life changes quickly and more often than in other phases²³23 Olukotun O, Seal N. A systematic review of dietary assessment tools for children age 11 years and younger. Infant Child Adolesc Nutr 2015; 7(3):139-147.. The most used methods to measure food intake in children under two years of age are the food frequency questionnaire (FFQ) or the measurement of food markers and the 24h dietary recall (24HR), but there are also surveys available which have used direct weighing and food diaries²³23 Olukotun O, Seal N. A systematic review of dietary assessment tools for children age 11 years and younger. Infant Child Adolesc Nutr 2015; 7(3):139-147.^,2424 Mello CS, Barros KV, Morais MB. Alimentação do lactente e do pré-escolar brasileiro: revisão da literatura. J Pediatr (Rio J) 2016; 92(5):451-463..

Both questionnaires and 24HR have pros and cons. While 24HR provides more detailed data, questionnaires require less time to apply. However, for reporting habitual food consumption, the food frequency questionnaire depends more on the respondent’s memory, whereas, when filling out a recall, memory efforts only have to be made for the previous day. The choice of the dietary method will depend on the characteristics and objectives of each study, the target population, as well as the number of resources available. It is noteworthy that important changes in feeding practices occurring in fast succession during the first year of life can limit the use of the food frequency method for measurement of food intake²⁵25 Brasil. Ministério da Saúde (MS). Guia alimentar para crianças brasileiras menores de 2 anos. Brasília: MS; 2019..

Measures are known to be potentially affected by various aspects: evaluators, sample characteristics, type of instrument, and administration method. The collection tool can also be a source of error, thus compromising the quality of the resulting evidence ²⁶26 Streiner DL, Norman GR, Cairney J. Health measurement scales - a practical guide to their development and use. New York: Oxford University Press; 2015.. Tools for collecting dietary data should be validated with reference methods whose reliability is widely-accepted²³23 Olukotun O, Seal N. A systematic review of dietary assessment tools for children age 11 years and younger. Infant Child Adolesc Nutr 2015; 7(3):139-147.^,2727 Frongillo EA, Baranowski T, Subar AF, Tooze JA, Kirkpatrick SI. Establishing validity and cross-context equivalence of measures and indicators. J Acad Nutr Diet 2019; 119(11):1817-1830.. The available research on validation of childhood food intake instruments includes children at least two years old and food frequency questionnaires as the major test method in use²⁸28 Flood VM, Wen LM, Hardy LL, Rissel C, Simpson JM, Baur LA. Reliability and validity of a short FFQ for assessing the dietary habits of 2-5-year-old children, Sydney, Australia. Public Health Nutr 2013; 17(3):498-509.

29 Marcinkevage J, Mayén A, Zuleta C, Digirolamo AM. Relative validity of three food frequency questionnaires for assessing dietary intakes of Guatemalan schoolchildren. PLoS One 2015; 14(10):e0139125.^-3030 Nyström CD, Henriksson H, Alexandrou C, Bergström A, Bonn S, Bälter K, Löf M. Validation of an online food frequency questionnaire against doubly labelled water and 24 h dietary recalls in pre-school children. Nutrients 2017; 9(1):66..

Monitoring children’s dietary practices over time is an essential activity to evaluate public policies and inform on improvements required. Intending to support this process of public policy evaluation, the World Health Organization (WHO) and the United Nations Children’s Fund (UNICEF) have recently updated the set of indicators for monitoring feeding practices based on the food consumed on the day prior to the interview³¹31 World Health Organization (WHO), United Nations Children's Fund (UNICEF). Indicators for assessing infant and young child feeding practices: definitions and measurement methods. Geneva WHO/UNICEF; 2021.. The current scope covers unhealthy foods, dietary diversity (i.e., intake of a wide range of food groups), and non-consumption of fruits and vegetables. However, the small number of studies on psychometric evaluation of instruments that have assessed feeding practices among young children predate this publication³²32 Warkentin S, Mais LA, Latorre MDRDDO, Carnell S, Taddei JADAC. Validation of the comprehensive feeding practices questionnaire in parents of preschool children in Brazil. BMC Public Health 2016; 16:603.

33 Myr RK, Bere E, Øverby NC. Test-retest reliability of a new questionnaire on the diet and eating behavior of one year old children. BMC Res Notes 2015; 8(1):8-16.

34 Judd AL, Beck KL, Mckinlay C, Jackson A, Conlon CA. Validation of a complementary food frequency questionnaire to assess infant nutrient intake. Matern Child Nutr 2020; 16(1):e12879.^-3535 Working Group on Infant and Young Child Feeding Indicators. Developing and validating simple indicators of dietary quality and energy intake of infants and young children in developing countries: summary of findings from analysis of 10 data sets. Connecticut: Food and Nutrition Technical Assistance Project; 2006., and were directed to other age groups and had different objectives.

The current study proposes to help fill the knowledge gap on the performance of instruments used in the evaluation of young children’s food intake in epidemiological surveys. The objective was to assess the relative validity and reproducibility of the set of dietary indicators proposed by WHO and UNICEF in 2021 in Brazilian children aged six to 23 months.

Methods

Study design, population, and sampling

This was a relative validity and reproducibility study of an instrument nested within the survey “Feeding Practices and Nutrition in Preschool Users of the Unified Health System”, conducted in the city of Rio de Janeiro, Brazil, in 2014. The probabilistic survey sample (n = 536) was representative of children six to 59 months of age living in Rio, who received primary health care (PHC) at community health centers from the Unified Health System (SUS), located in neighborhoods in different administrative districts of the city. Sampling was divided into two stages: first, selection of the 33 PHC centers; after that, selection of children from the enrollment lists provided by such centers. Details on the sampling design for the survey are available in Carneiro et al. (2019). The current study analyzed data on children in the stratum under 24 months of age (n = 190) who had complete information for the data analysis. The validity study included individuals that had answered the 24-hour dietary recall (24HR) and the questionnaire applied on the first interview (Q1) (n = 187). The reproducibility study used data on children with questionnaires completed at the first and second interviews (Q1 and Q2) (n = 83), as detailed below.

After the survey data had already been collected for the validity study reported in this manuscript, sampling sufficiency was calculated a posteriori. The calculation was performed with the sskdlg routine (sample size for the kappa statistic of interrater agreement) in Stata v.10, considering the following parameters: the event with the lowest occurrence according to the 24HR, reference method (prevalence of zero vegetable or fruit consumption, 18.1%) and kappa observed for this indicator (0.22); absolute precision of the study of 0.20 and reliability of 95%. These parameters suggested that a sample of 140 children would be enough.

For the reproducibility study, sampling sufficiency was checked, as proposed by Bujang and Baharum³⁶36 Bujang MA, Baharum N. Guidelines of the minimum sample size requirements for Cohen's Kappa. Epidemiol Biostat Public Heal 2017; 14(2):e12267-1-e12267-10., who indicated the need for 72 individuals for an agreement study with 80% test power and 95% confidence, considering kappa values of 0.30 (k1) and 0.60 (k2) for test of hypotheses.

Data collection and variables for characterization of the study group

The parents or guardians of the selected children were invited by telephone contact from the researchers. Those who agreed to participate were scheduled for data collection at the PHC center. Data were collected in paper forms from June to December 2014, in a quiet room at the PHC unit, by nutritionists that had been trained in a 16-hour training session.

All data collection procedures were standardized according to a field manual. Before the beginning of data collection, the instruments were previously tested and a pilot study was carried out in a PHC center in the same city where the study was conducted; such center was not included in the sample.

On the scheduled day, the interview was held with the child’s mother or another guardian (father, grandmother, grandfather, or aunt) for completion of the 24HR and Q1, in this order. Between the application of 24HR and Q1, we carried out an anthropometric assessment.

For completion of the 24HR, the parent or guardian was asked to describe the foods and beverages consumed by the child on the day prior to the interview. The procedure involved recording the types of foods, amounts, preparation, time, place of consumption, and in the case of processed and ultra-processed foods³⁷37 Monteiro CA, Cannon G, Levy R, Moubarac JC, Jaime P, Martins AP, Canella D, Louzada M, Parra D. NOVA. The Star Shines Bright (Food Classification. Public Health). World Nutr 2016; 7(1-3):28-38., the respective brand names and flavors. To help the interviewee recall the portion of food served to the child, utensils and replicas of the foods were used to determine household measures.

In the research in which the present study was nested, the participants had a blood sample drawn and, up to around 16 days later (mean of 16.0 ± 8.4 days), were supposed to return to the PHC center to collect the results of the laboratory analyses. On that occasion, a second interview was conducted; the children’s parents or guardians answered the same closed questionnaire again (Q2), applied under similar conditions to those of Q1. Despite the reminder calls, only 83 children (43.6%) attended the scheduled interview. Nevertheless, the minimum sample size (n = 72) was achieved for the reproducibility study.

To describe the study group and to compare participants that answered only Q1 with those that answered Q1 and Q2, the following demographic and socioeconomic variables were chosen: children’s sex and age group (6-11.9 and 12-23.9 months), maternal age, maternal schooling (some primary education, completed primary education, completed secondary education, or completed higher education), monthly family income (USD values equivalent to minimum wages in BRL - less than 1, 1-2, 2 or more), government benefits (such as the Bolsa Família conditional cash transfer program), and household food security (using the Brazilian Food Insecurity Scale³⁸38 Pérez-Escamilla R, Segall-Corrêa AM, Maranha LK, Sampaio MFA, Marín-León L, Panigassi G. An adapted version of the U.S. Department of Agriculture Food Insecurity Module is a valid tool for assessing household food insecurity in Campinas, Brazil. J Nutr 2004; 134(8):1923-1928.). When mother was not present at the moment of data collection, maternal education was informed by the respondent.

Questionnaire

A structured questionnaire was designed by the authors, based on other models available³⁹39 Oliveira JM, Castro IRR, Silva GB, Venancio SI, Saldiva SRDM. Avaliação da alimentação complementar nos dois primeiros anos de vida: proposta de indicadores e de instrumento. Cad Saude Publica 2015; 31(2):377-394.^,4040 Brasil. Ministério da Saúde (MS). II Pesquisa de Prevalência de Aleitamento Materno nas Capitais Brasileiras e Distrito Federal. Brasília: MS; 2009.. It contained 42 closed questions on food consumption on the previous day, including after-midnight hours on the day of the interview, a particularly important period for infants. The food list included markers of food groups, iron and vitamin A sources and unhealthy foods. Participants were asked whether or not the child had consumed each food or food group and could answer “yes”, “no” or “don’t know”. Questions used to compose the indicators are shown in Chart 1.

The questionnaire does not measure all the elements provided by the WHO in an identical way. However, in general, the items were quite similar.

Dietary indicators

Chart 1 shows the indicators adopted. They were calculated as recommended in the WHO/UNICEF³¹31 World Health Organization (WHO), United Nations Children's Fund (UNICEF). Indicators for assessing infant and young child feeding practices: definitions and measurement methods. Geneva WHO/UNICEF; 2021. reference document. Healthy dietary indicators are based on diet attributes, such as timely introduction of foods (fruit and solid foods for children 6-8m29d of age), dietary diversity (expressed by food groups), energy density (expressed by the frequency of meals with adequate consistency) and presence of food sources of specific micronutrients. Indicators for unrecommended practices assess consumption of ultra-processed foods and sugar or other sweeteners.

Although the question “Did the child eat carrots, squash, or sweet potato?” includes two vegetables, it was not included in the composition of the variable “consumption of vegetables”, because we opted to take a more conservative stance, since the question contains a tuber, which is not part of the vegetable group. Even so, we tested the indicators by both including and excluding this question, and the results were similar for the evaluation of both validity and reproducibility.

The questions considered in the construction of the indicator “Introduction of Solid, Semi-Solid or Soft Foods” were different from those proposed by the WHO. The indicator proposed by the WHO is based on the question “Did the child consume solid, semi-solid or pasty foods?”, which was not well understood by the respondents in the pre-test. Also, this question includes all foods, even ultra-processed ones. In the present study, the indicator was constructed based on questions regarding the consumption of fruits and regular foods.

Importantly, three of the indicators proposed by WHO/UNICEF could not be produced owing to the lack of information on the frequency of meals and number of times children drank milk: “Minimum Meal Frequency”, “Minimum Milk Feeding Frequency for non-breastfed children” and “Minimum Acceptable Diet”. Also, analyses for the indicator “Introduction of Solid, Semi-Solid or Soft Foods” were not consistent because of the small number of sampled children (n = 10) in the age group for this indicator (from 6 months to 8 months and 29 days).

In the food groups, although the legume group includes nuts and seeds in the indicator proposed by WHO/UNICEF, in the present study only beans, peas and lentils were considered for inclusion as the question in the questionnaire focused on these foods. In the indicator “Sweet Beverage Consumption”, we did not consider natural fruit juice, as we did not have information in the questionnaire on whether or not sugar had been added.

Data entry and analysis

Data from Q1, Q2 and sociodemographic characteristics form were entered by two different individuals and consistency of the data entry was assessed using the Epi Info software⁴¹41 Dean AG, Arner TG, Sunki GG, Friedman R, Lantinga M, Sangam S, Zubieta JC, Sullivan KM, Brendel KA, Gao Z, Fontaine N, Shu M, Fuller G, Smith DC, Nitschke DA, Fagan FR. Epi InfoTM, a database and statistics program for public health professionals [Internet]. 2011. [cited 2022 dez 3]. Available from: https://www.cdc.gov/epiinfo/index.html
https://www.cdc.gov/epiinfo/index.html... , version 3.5.2. Prior to 24HR data entry, the information about home-cooked meals was disaggregated for their ingredients. In addition, all the quantities reported in household measures have been converted to grammage. Double entry of data from the 24HR used Microsoft Excel, and consistency of this data entry was assessed using the EpiData 3.1 software⁴²42 Lauritsen JM, Bruus M. EpiData Entry - a comprehensive tool for validated entry and documentation of data [Internet]. [cited 2022 dez 3]. Available from:. https://www.epidata.dk/credit.htm
https://www.epidata.dk/credit.htm... .

Data on the 24HR underwent standardization and coding prior to typing. The homemade culinary preparations mentioned by the participants were broken down into their ingredients and all the homemade measures mentioned by the mothers or guardians were converted into units of mass and volume. For the validity and reproducibility studies, indicators from the 24HR were designed on the basis of the presence or absence of foods/food groups, in the same way as the ones adopted for the indicators from the closed questionnaire.

Validity was assessed by using Q1 as the test method and 24HR as the reference method. Prevalence rates were calculated, based on each of the instruments, and comparison of pairs of proportions used the McNemar chi-square test with significance set at 0.05. We also estimated the test method’s sensitivity, specificity, positive and negative predictive values, and accuracy index, which can be defined as the proportion of correct classifications (true positives plus true negatives among the study individuals)⁴³43 Szklo M, Nieto FJ. Epidemiology beyond the basics. Burlington: Jones and Barlett Publishers; 2007..

The 24HR was chosen as the reference method because it allows us to measure children’s eating practices in more detail than the FFQ. The intention was to ascertain whether the results found for indicators built based on Q1 would be similar to those designed on the basis of the 24HR if they had been applied on the same day. For this reason, a single 24HR was adopted for the validity study.

Reproducibility was assessed by comparison of the designed indicators, based on the answers recorded in Q1 and Q2. Prevalence rates were calculated, and McNemar chi-square test with significance at 0.05 was used to assess potential differences between the proportions. Given that indicators were classified as being present or absent for each participant, the degree of agreement between answers in Q1 and Q2 was determined by calculation of kappa index ²⁶26 Streiner DL, Norman GR, Cairney J. Health measurement scales - a practical guide to their development and use. New York: Oxford University Press; 2015.. Since the use of kappa is limited by the prevalence of the measured attribute, because it shows lower values when the frequency of the target event is far from 50% (a recurrent situation in this study), we also calculated the prevalence-adjusted bias-adjusted kappa (PABAK)⁴⁴44 Byrt T, Bishop J, Carlin JB. Bias, prevalence and kappa. J Clin Epidemiol 1993; 46(5):423-429.. Values greater than 0.92 indicate excellent agreement; from 0.91 to 0.80, very good agreement; from 0.79 to 0.60, good agreement; from 0.59 to 0.40, fair agreement; from 0.39 to 0.20, superficial agreement; and below 0.19, poor agreement⁴⁵45 Byrt T. How good is that agreement? Epidemiol 1996; 7(5):561.. To support the interpretation of PABAK values, the prevalence index and the bias index were calculated⁴⁴44 Byrt T, Bishop J, Carlin JB. Bias, prevalence and kappa. J Clin Epidemiol 1993; 46(5):423-429.^,4646 Sim J, Wright CC. The Kappa statistic in reliability studies: use, interpretation and sample size requirements. Phys Ther 2005; 85(3):257-268..

All the analyses were performed with SPSS Statistics 17.0.

Ethical aspects

The study was approved by the Ethics Committee for Research with Humans, Rio de Janeiro Municipal Health Office (case no. 93/2013).

Participation of the children’s parents/guardians was voluntary, without any financial compensation, and they were offered the possibility of quitting the study at any time. Children were only allowed to be included in the study after their parents or guardians had signed an informed consent form. Children with nutritional disorders (anemia, vitamin A deficiency, low stature, underweight, and excess weight, indicators assessed in the main study) were referred for care in the same PHC centers where the data were collected. The parents also received educational materials on the promotion of healthy feeding on the second day of data collection, after completing the interview, to avoid behavioral changes and socially accepted responses.

Results

Mothers were the main respondents of interviews on both days (88.0% and 80.9%, on the first and second days, respectively), followed by grandmother/grandfather (5.9%; 11.9%) and father (2.3%; 4.4%). In 83% of cases, Q2 was answered by the same respondent who had answered Q1.

Characterization of the study group

Table 1 shows the socioeconomic characteristics of all children (n = 187), of those who had only one interview (n = 104) and of those with two interviews (n = 83). In general, both groups had similar sociodemographic characteristics, except for sex (53.8% female among children who answered only one questionnaire vs. 38.6% among those who answered two questionnaires).

Of the total of 187 children in the sample, 52,9% were boys, 74.3% were aged between one and two years and 92.3% were children of adult mothers (mean age = 26.9 years (SD = 6.7)). Most of the mothers had at least secondary education. As for monthly family income, most of the families earned up to US$ 310.72 monthly, and 41% lived in food insecure households (Table 1).

Validity

In general, there were higher prevalence rates in the indicators produced with data from Q1 when compared to those with data from 24HR (Table 2), with statistically significant difference for three indicators: “Continued Breastfeeding”, “Sweet Beverage Consumption” and “Zero Vegetable or Fruit Consumption”.

Of the seven estimated indicators, there was an overestimation of the prevalence of two indicators (“Continued Breastfeeding” and “Sweet Beverage Consumption”) and an underestimation of the prevalence of one indicator (“Zero Vegetable or Fruit Consumption”), as calculated by Q1 compared to 24HR (statistically significant differences). For six of the seven indicators, the test method’s sensitivity was higher than its specificity. Concerning sensitivity, the questionnaire performed better in the identification of children who consumed unhealthy food, breastmilk, sweetened beverage, and eggs on the day prior to the study’s data collection day. Concerning specificity, the questionnaire performed better in the identification of children who did not consume vegetables or fruits. Specificity was particularly low for unhealthy food consumption (18.1%). Positive predictive values ranged from 78.5% to 95.0% and negative predictive values, from 40.0% to 98.5%. The accuracy index achieved 75.0% or more for all indicators.

Reproducibility

Although higher prevalence rates were found for most of the indicators in the first application of the questionnaire (six out of seven), there was a statistically significant difference between prevalence rates obtained with Q1 and Q2 only for the indicator “Minimum Dietary Diversity” (Table 3). Agreement of the answers according to kappa values between Q1 and Q2 ranged from -0.07 to 0.97, with most being classified as fair or superficial. The analysis of PABAK values shows that more than half presented good, very good, or excellent agreement (four of seven). The following indicators showed worse performance (PABAK less than or equal to 0.59): “Introduction of Solid, Semi-Solid or Soft Foods”, “Minimum Dietary Diversity”, and “Sweet Beverage Consumption”.

Discussion

The findings suggest good reliability of the indicators for the population and context assessed. As for validity, higher prevalence rates for the indicators produced with the data from the questionnaire were an expected result, considering that it may be easier for the respondent to remember what the child consumed when asked directly about a specific food (or food group) than to answer spontaneously, as with the 24HR. As for reproducibility, the prevalence rates produced in the first and second measurements show statistically significant differences for only one indicator; thus, the indicator can be considered as stable when applied at the group level.

The current study included children from six to 23 months of age and compared the test method with the reference method by assessing the presence (yes/no) of food markers. In the relative validity assessment, as the questionnaire had questions that grouped foods together (for example, “fruits” rather than some fruits individually, which could not cover all those ingested by the child), it seemed to be able to cover the items listed in the 24HR.

As commented before, in 17% of cases, Q2 was not answered by the same respondent who had answered Q1. We believe that this is not a problem, since what mattered was for the respondent to know what the child had consumed the previous day.

Of the three indicators with worst performance in the reproducibility study, “Introduction of Solid, Semi-Solid or Soft Foods” was calculated with a small number of participants. The second one, “Minimum Dietary Diversity”, suggests that these food groups are not offered to the children daily or that there is greater variability in daily consumption⁴⁷47 Souza ADM, Pereira RA, Yokoo EM, Levy RB, Sichieri R. Alimentos mais consumidos no Brasil: Inquérito Nacional de Alimentação 2008-2009. Rev Saude Publica 2013; 47(Supl. 1):190-199.^,4848 Kim SA, Moore LV, Galuska D, Wright AP, Harris D, Grummer-Strawn LM, Merlo CL, Nihiser AJ, Rhodes DG, Division of Nutrition, Physical Activity, and Obesity, National Center for Chronic Disease Prevention and Health Promotion, CDC. Vital signs: fruit and vegetable intake among children - United States, 2003-2010. Morb Mortal Wkly Rep 2015; 63(31):671-676.. The last one, “Sweet Beverage Consumption”, suggests that although the prevalence rates were similar, it was not the same children who consumed these products on both days, as the agreement was fair.

Methodological differences between our study and others that examined the performance of dietary indicators directed to young children preclude a comparison of our findings with theirs. The report by the Food and Nutrition Technical Assistance project (“FANTA”)³⁵35 Working Group on Infant and Young Child Feeding Indicators. Developing and validating simple indicators of dietary quality and energy intake of infants and young children in developing countries: summary of findings from analysis of 10 data sets. Connecticut: Food and Nutrition Technical Assistance Project; 2006., which backed previous publications on childhood dietary indicators by the WHO⁴⁹49 World Health Organization (WHO). Indicators for assessing infant and young child feeding practices - Part 1 definitions: conclusions of a consensus meeting held 6-8 November 2007 in Washington D.C., USA [Internet]. 2008. [cited 2022 dez 3]. Available from: https://apps.who.int/iris/handle/10665/43895
https://apps.who.int/iris/handle/10665/4... , showed that food diversity indicators based on food groups were able to discriminate different levels of micronutrient adequacy.

A study that examined the relative validity of telephone interviews on infants’ diet and adherence to a nutritional intervention⁵⁰50 Vähätalo L, Bärlund S, Hannila ML, Uusitalo U, Pigg HM, Salonen M, Nucci A, Krischer JP, Knip M, Akerblom HK, Virtanen SM. Relative validity of a dietary interview for assessing infant diet and compliance in a dietary intervention trial. Matern Child Nutr 2006; 2(3):181-187. used a short questionnaire, and the information was compared to data from two 48-hour recalls (48HR). The kappa index for the evaluation between the questionnaire and the 48HR showed very good agreement between the methods. The questionnaire even captured the unrecommended foods for those consuming the intervention diet. In the present study, the indicators calculated from the questionnaire also showed good validity, but the 24HR captured more unrecommended foods than those listed in the questionnaire.

It is noteworthy that the questionnaire was not designed to capture three types of ultra-processed foods that were consumed by participants, as observed in the 24HR: sugary breakfast cereals, artificial spices or readymade seasonings, and soy-based beverages. The first two were not explicitly mentioned in the questionnaire, so they could not be computed in the indicators on ultra-processed foods. As for the third item, the question on ultra-processed beverages (“industrialized juices in cartons, bottles, or powdered form”) did not mention soy-based beverages.

The fact that Q1 did not capture sugary breakfast cereals, readymade seasonings and soy-based beverages did not compromise the results of the study: all participants that reported consumption of soy-based beverages in the 24-hour dietary recall also answered affirmatively to the question on industrialized beverages; those who reported consumption of sugary breakfast cereals and readymade seasonings on the 24HR also reported at least one of the other ultra-processed foods listed in the questionnaire. In other words, the results of the indicator “Sweet Beverage Consumption” and “Unhealthy Food Consumption” would not have changed if these foods had been included in Q1.

The study has some limitations. The first is the fact that three of the 10 indicators proposed by the WHO for children ≥ 6mo were not designed. To overcome this, the questionnaire would have to contain items about the frequency of meals and the number of times the child consumed milk, but these aspects were not of interest to the survey in which the present study was inserted.

A second weakness was the small number of children aged 6-8 months (n = 10), which compromised analyses related to the indicator “Introduction of Solid, Semi-Solid or Soft Foods”. Studies with larger samples are needed to confirm the performance of this indicator.

Applying 24HR to assess the food consumption of young children is a challenging task. Specific issues of the age group need to be taken into account; for example, checking the consumption of breast milk. A weakness of this study was the failure to include a checklist at the end of the interview to capture foods that are habitually forgotten during reporting on food consumption on the previous day, including breast milk. This led to underreporting of breastfeeding in the 24HR by 16 mothers who later responded positively to the question on breastfeeding in Q1. This suggests that, during 24HR application, some respondents focused only on complementary foods, overlooking breastmilk.

A fourth weakness of the study was the sample loss in the reproducibility study. But this loss does not seem to have been selective, since, except for sex, the children who answered two questionnaires had socio-demographic characteristics similar to those who answered only one questionnaire.

The average number of days between the application of Q1 and Q2 was 16 days. This could have been a problem for children under one year of age, since, in this age group, there can be important changes to their diet in a short period of time. However, it was found that this was not a problem, as there was no statistically significant difference in the analyses for any of the indicators when only children under one year old were selected.

The strengths of the study include: the use of PABAK, which complemented and improved the reproducibility analyses, showing the importance of considering the prevalence of the outcome and observer bias in agreement studies; and the adoption of more updated indicators of WHO, with information about prevalence of consumption of unrecommended foods. This expands the scope of the indicators proposed by the WHO in the past⁴⁹49 World Health Organization (WHO). Indicators for assessing infant and young child feeding practices - Part 1 definitions: conclusions of a consensus meeting held 6-8 November 2007 in Washington D.C., USA [Internet]. 2008. [cited 2022 dez 3]. Available from: https://apps.who.int/iris/handle/10665/43895
https://apps.who.int/iris/handle/10665/4... ^,5151 World Health Organization (WHO), United Nations Children's Fund (UNICEF). Global nutrition monitoring Framework - operational guidance for tracking progress in meeting targets for 2025 [Internet]. 2017. [cited 2022 ago 3]. Available from: https://www.who.int/publications/i/item/9789241513609
https://www.who.int/publications/i/item/... , which had, for years, focused only on recommended complementary feeding practices. The current indicators are in line with epidemiological data, which indicate early consumption of ultra-processed foods.

Conclusion

The findings suggest very good validity and good reproducibility of the indicators, obtained with the test method for the population and the context assessed. However, the questionnaire needs some reformulations to fully capture information for all the indicators proposed by the WHO.

The questionnaire seems to be promising for use in nutritional monitoring systems, surveys aimed at describing the prevalence of markers for healthy and unhealthy feeding in early childhood, or ecological studies on this subject in similar contexts. Its adoption can be advantageous, since its application is simpler and quicker than that of the 24HR (5-7 minutes for Q1 and more than 20 minutes for 24HR). Further studies are needed to assess the performance of the instrument in other contexts and according to age range.

Acknowledgments

The authors wish to thank the field interviewers and the interns who performed the data entry. The authors also wish to thank Municipal Health Department of the city of Rio de Janeiro for the authorization to collect data at health units.

References

Publication Dates

History