Abstract

This study aimed to evaluate the content validity and reliability of an instrument for evaluating the university food environment. A checklist was developed to assess establishments that sell food and beverages in the university environment. The content validation encompassed the development of the instrument, expert evaluation and pretest performance. Reliability was evaluated using a convenience sample (n=64) of establishments distributed across seven campuses of three public universities and was carried out using interobserver (IO) and test-retest (TR) evaluations. Categorical and count variables were analyzed by calculating the percentage agreement (PA), kappa coefficient (k) and prevalence-adjusted, bias-adjusted kappa (ka), and continuous variables were analyzed by the intraclass correlation coefficient (ICC). The checklist consisted of 204 items distributed in seven domains. The instrument’s performance was considered excellent or very good for 91.3% (PA) of the items when evaluated. For IO, 68.3% (k) and 96.5% (ka) had excellent, very good or good agreement, while for TR, 65% (k) and 96.5% (ka) had excellent agreement. The instrument showed satisfactory content validity and reliability for characterizing the food environment at Brazilian universities.

Key words:
Psychometric properties; Reproducibility; Higher education Institutions

Introduction

Efforts have been made to translate concepts and constructs about the food environment into measurement tools that can generate empirical evidence¹1 Glanz K, Sallis JF, Saelens BE, Frank LD. Healthy nutrition environments: Concepts and measures. Am J Heal Promot 2005; 19(5):330-333.

2 Bethlehem JR, Mackenbach JD, Ben-Rebah M, Compernolle S, Glonti K, Bárdos H, Rutter HR, Charreire H, Oppert JM, Brug J, Lakerveld J. The SPOTLIGHT virtual audit tool: A valid and reliable tool to assess obesogenic characteristics of the built environment. Int J Health Geogr 2014; 13:52.

3 Duran AC, Lock K, Latorre MRDO, Jaime PC. Evaluating the use of in-store measures in retail food stores and restaurants in Brazil. Rev Saude Publica 2015; 49:80.

4 Borges CA, Jaime PC. Development and evaluation of food environment audit instrument: AUDITNOVA. Rev Saude Publica 2019; 53:91.

5 Altenburg T, Te Velde S, Chiu KJ, Moschonis G, Manios Y, De Bourdeaudhuij I, Vik FN, Lien N, Brug J, Chinapaw M. Interrater reliability of the ENERGY photo-rating instrument for school environments related to physical activity and eating. J Phys Act Heal 2016; 13(4):433-439.^-66 Giskes K, Van Lenthe FJ, Brug J, Mackenbach JP, Turrell G. Socioeconomic inequalities in food purchasing: The contribution of respondent-perceived and actual (objectively measured) price and availability of foods. Prev Med (Baltim) 2007; 45(1):41-48.. The consumer food environment¹1 Glanz K, Sallis JF, Saelens BE, Frank LD. Healthy nutrition environments: Concepts and measures. Am J Heal Promot 2005; 19(5):330-333., which was the object of interest in the present study, includes factors related to the food itself, i.e., the way in which it is provided or presented and the price, nutritional quality and nutritional labeling (including nutrition claims), among other factors. The consumer’s food environment may be inserted into organizational environments, such as companies and universities¹1 Glanz K, Sallis JF, Saelens BE, Frank LD. Healthy nutrition environments: Concepts and measures. Am J Heal Promot 2005; 19(5):330-333.. In the literature, the most commonly reported method for evaluating the consumer food environment is the auditing of establishments that sell food. Researchers evaluate the variety of products, nutritional information, healthy options, prices, advertisements and other aspects of food²2 Bethlehem JR, Mackenbach JD, Ben-Rebah M, Compernolle S, Glonti K, Bárdos H, Rutter HR, Charreire H, Oppert JM, Brug J, Lakerveld J. The SPOTLIGHT virtual audit tool: A valid and reliable tool to assess obesogenic characteristics of the built environment. Int J Health Geogr 2014; 13:52.^,77 Glanz K, Sallis JF, Saelens BE, Frank LD. Nutrition Environment Measures Survey in Stores (NEMS-S). Development and Evaluation. Am J Prev Med 2007; 32(4):282-289.

8 Gustafson A, Hankins S, Jilcott S. Measures of the consumer food store environment: A systematic review of the evidence 2000-2011. J Community Health 2012; 37(4):897-911.^-99 Caspi CE, Sorensen G, Subramanian SV, Kawachi I. The local food environment and diet: A systematic review. Heal Place 2012; 18(5):1172-1187..

The university food environment, which was the scenario of interest of this study, can be characterized by the availability of foods, preparations and beverages (FPB), physical and financial accessibility, the promotion of FPB, nutritional information and advertising within the campus and may also include the campus surroundings. It is known that this environment influences the eating habits of the individuals who are exposed to it¹⁰10 Roy R, Soo D, Conroy D, Wall CR, Swinburn B. Exploring University Food Environment and On-Campus Food Purchasing Behaviors, Preferences, and Opinions. J Nutr Educ Behav 2019; 51(7):865-875., since it can act as a facilitator of or barrier to healthy choices¹1 Glanz K, Sallis JF, Saelens BE, Frank LD. Healthy nutrition environments: Concepts and measures. Am J Heal Promot 2005; 19(5):330-333.^,1010 Roy R, Soo D, Conroy D, Wall CR, Swinburn B. Exploring University Food Environment and On-Campus Food Purchasing Behaviors, Preferences, and Opinions. J Nutr Educ Behav 2019; 51(7):865-875. given that meals consumed at the university may be an important part of the diet of students, teachers and employees¹¹11 Perez PMP, Castro IRR, Canella DS, Franco ADS. Effect of implementation of a university restaurant on the diet of students in a brazilian public university. Cien Saude Colet 2019; 24(6):2351-2360.^,1212 Perez PMP, Castro IRR, Franco AS, Bandoni DH, Wolkoff DB. Práticas alimentares de estudantes cotistas e não cotistas de uma universidade pública Brasileira. Cien Saude Colet 2016; 21(2):531-542..

As of 2021, Brazil had 302 public higher education institutions (HEIs) and 2,306 private HEIs, which together comprise 8,603,824 students and 386,703 teachers,¹³13 Brasil. Ministério da Educação (MEC). Instituto Nacional de Estudos e Pesquisas Educacionais Anísio Teixeira (INEP). Resumo Técnico do Censo da Educação Superior 2019. Brasília: MEC; 2021. in addition to thousands of technical-administrative employees. Therefore, these locations should be valued as a strategic setting for promoting healthy eating.

The literature on the evaluation of the university food environment is still expanding. Countries such as the United States and Australia have published studies on the characterization of the food environment at universities and their surroundings¹⁴14 Horacek TM, Erdman MB, Reznar MM, Olfert M, Brown-Esters ON, Kattelmann KK, Kidd T, Koenings M, Phillips B, Quick V, Shelnutt KP, White AA. Evaluation of the food store environment on and near the campus of 15 postsecondary institutions. Am J Heal Promot 2013; 27(4):81-91.^,1515 Roy R, Hebden L, Kelly B, De Gois T, Ferrone EM, Samrout M, Vermont S, Allman-Farinelli M. Description, measurement and evaluation of tertiary-education food environments. Br J Nutr 2016; 115(9):1598-1606.. In Brazil, to date, few such studies have been found¹⁶16 Franco ADS, Canella DS, Perez PMP, Bandoni DH, Castro IRRD. Ambiente alimentar universitário: caracterização e mudanças no período de 2011 a 2016 em uma universidade pública brasileira. Rev Nutr 2020; 33:e200058.

17 Rodrigues CB, Monteiro LS, Paula NM, Pereira RA. Ambiente alimentar em um campus universitário: desenvolvimento e análise de instrumento para avaliação de estabelecimentos comerciais. Demetra 2021; 16:e51139.

18 Barbosa R, Henriques P, Guerra H, Ermentino J, Sorares D, Dias P, Ferreira D. Food environment of a Brazilian public university: challenges to promote healthy eating. Rev Chil Nutr 2020; 47(3):443-448.^-1919 Pulz IS, Martins PA, Feldman C, Veiros MB. Are campus food environments healthy? A novel perspective for qualitatively evaluating the nutritional quality of food sold at foodservice facilities at a Brazilian university. Perspect Public Health 2017; 137(2):122-135..

The selection or development of a data collection instrument is an essential step that, if not performed with care and methodological rigor, can be a source of error and misleading conclusions²⁰20 Streiner DL, Norman GR, Cairney J. Reliability. In: Health measurement scales: a practical guide to their development and use. New York: Oxford University Press; 2015. p. 179-199.. Although many studies have been conducted to evaluate the food environment, a small portion of them have assessed the psychometric properties of the instruments used in them²¹21 McKinnon RA, Reedy J, Morrissette MA, Lytle LA, Yaroch AL. Measures of the Food Environment. A Compilation of the Literature, 1990-2007. Am J Prev Med 2009; 36(Supl. 4):124-133.^,2222 Lytle LA, Sokol RL. Measures of the food environment: A systematic review of the field, 2007-2015. Heal Place 2017; 44(2016):18-34.. Among the studies on the university food environment, only two analyzed the reliability of the instrument used¹⁴14 Horacek TM, Erdman MB, Reznar MM, Olfert M, Brown-Esters ON, Kattelmann KK, Kidd T, Koenings M, Phillips B, Quick V, Shelnutt KP, White AA. Evaluation of the food store environment on and near the campus of 15 postsecondary institutions. Am J Heal Promot 2013; 27(4):81-91.^,1616 Franco ADS, Canella DS, Perez PMP, Bandoni DH, Castro IRRD. Ambiente alimentar universitário: caracterização e mudanças no período de 2011 a 2016 em uma universidade pública brasileira. Rev Nutr 2020; 33:e200058., and none evaluated its validity. In addition, of the instruments that have been identified to date, none is able to fully capture the specificities of the Brazilian university food environment or highlight the extensive variety of convenience items and the existence of mixed establishments (that have characteristics of both restaurants and cafeterias). In addition, these instruments do not use the NOVA classification of foods, which has been adopted in the Brazilian guidelines for public policies on food and nutrition²³23 Monteiro CA, Cannon G, Levy RB, Moubarac JC, Louzada MLC, Rauber F, Khandpur N, Cediel G, Neri D, Martinez-Steele E, Baraldi LG, Jaime PC. Ultra-processed foods: What they are and how to identify them. Public Health Nutr 2019; 22(5):936-941. and studies on food, diet quality and health²⁴24 Brasil. Ministério da Saúde (MS). Secretaria de Atenção à Saúde. Departamento de Atenção Básica. Guia alimentar para a população brasileira. Brasília: MS; 2014., to characterize commercial foods and preparations²⁵25 Monteiro CA, Cannon G, Lawrence M, Louzada MLC, Machado PP. Ultra-processed foods, diet quality, and health using the NOVA classification system. Rome: FAO; 2019..

Considering the importance and specificities of the university food environment, the scarcity of instruments for its evaluation and the scarcity of evaluations of the performance of these instruments in different contexts, the present study aimed to evaluate the content validity and reliability of an instrument for evaluating the university food environment.

Methods

The present study is an initiative of the Collaborative Group for Studies on the University Food Environment (Grupo Colaborativo de Estudos sobre o Ambiente Alimentar Universitário - CALU, in Portuguese), which is composed of professors from public universities located in the state of Rio de Janeiro that have a course in nutrition, namely, the Universidade do Estado do Rio de Janeiro (UERJ), Universidade Federal do Rio de Janeiro (UFRJ), Universidade Federal do Estado do Rio de Janeiro (UNIRIO) and Universidade Federal Fluminense (UFF).

Content validation

The content validation of the instrument involved three stages: the development of the instrument, its evaluation by experts²⁶26 Lynn MR. Determination and Quantification of Content Validity. J Experiment Psychol 1986; 136:382-386.^,2727 Haynes B, Haines A. Barriers and bridges to evidence based clinical practice. BMJ 1998; 317(7153):273-276. and the performance of a pretest.

Development of the instrument

The preparation of the instrument was based on two theoretical frameworks: the approach of Caspi et al.⁹9 Caspi CE, Sorensen G, Subramanian SV, Kawachi I. The local food environment and diet: A systematic review. Heal Place 2012; 18(5):1172-1187. regarding access to food and the approach of Glanz et al.¹1 Glanz K, Sallis JF, Saelens BE, Frank LD. Healthy nutrition environments: Concepts and measures. Am J Heal Promot 2005; 19(5):330-333. regarding the consumer food environment. Of the five dimensions proposed by Caspi et al.⁹9 Caspi CE, Sorensen G, Subramanian SV, Kawachi I. The local food environment and diet: A systematic review. Heal Place 2012; 18(5):1172-1187., those that could fully apply to commercial establishments located in universities were considered, namely, availability (including the variety of food available), affordability (including the price of food) and convenience (e.g., open hours, payment methods accepted). These dimensions were complemented by two components proposed by Glanz et al.¹1 Glanz K, Sallis JF, Saelens BE, Frank LD. Healthy nutrition environments: Concepts and measures. Am J Heal Promot 2005; 19(5):330-333.: nutritional information and food promotion.

The first stage of construction of the instrument involved mapping the existing instruments for the evaluation of the consumer’s food environment and the selection of two that closely matched the characteristics of the food environments of Brazilian public universities¹⁴14 Horacek TM, Erdman MB, Reznar MM, Olfert M, Brown-Esters ON, Kattelmann KK, Kidd T, Koenings M, Phillips B, Quick V, Shelnutt KP, White AA. Evaluation of the food store environment on and near the campus of 15 postsecondary institutions. Am J Heal Promot 2013; 27(4):81-91.^,1717 Rodrigues CB, Monteiro LS, Paula NM, Pereira RA. Ambiente alimentar em um campus universitário: desenvolvimento e análise de instrumento para avaliação de estabelecimentos comerciais. Demetra 2021; 16:e51139.. The elaboration of the instrument encompassed three phases, which will be described in detail in the results section: the identification of the domains that should compose the instrument (e.g., characterization of the establishment, available foods, prices, advertisements), the preparation of the items and the structuring of the instrument. During this process, the inclusion of foods based on the recommendations of the Dietary Guidelines for the Brazilian Population²⁴24 Brasil. Ministério da Saúde (MS). Secretaria de Atenção à Saúde. Departamento de Atenção Básica. Guia alimentar para a população brasileira. Brasília: MS; 2014. and international guidelines²⁸28 World Health Organization (WHO). Diet, nutrition and the prevention of chronic diseases. Geneva: WHO Technical Report Series; 2003., the NOVA classification of foods²³23 Monteiro CA, Cannon G, Levy RB, Moubarac JC, Louzada MLC, Rauber F, Khandpur N, Cediel G, Neri D, Martinez-Steele E, Baraldi LG, Jaime PC. Ultra-processed foods: What they are and how to identify them. Public Health Nutr 2019; 22(5):936-941. and foods commonly found in universities¹⁴14 Horacek TM, Erdman MB, Reznar MM, Olfert M, Brown-Esters ON, Kattelmann KK, Kidd T, Koenings M, Phillips B, Quick V, Shelnutt KP, White AA. Evaluation of the food store environment on and near the campus of 15 postsecondary institutions. Am J Heal Promot 2013; 27(4):81-91.^,1717 Rodrigues CB, Monteiro LS, Paula NM, Pereira RA. Ambiente alimentar em um campus universitário: desenvolvimento e análise de instrumento para avaliação de estabelecimentos comerciais. Demetra 2021; 16:e51139. was guaranteed. The items included in the checklist comprised fresh or minimally processed foods, processed foods and culinary preparations based on these foods, as well as ultra-processed foods and culinary preparations containing these foods. Together, these are markers of healthy and unhealthy eating.

Expert evaluation

For the evaluation of the instrument by experts, the literature suggests the participation of five to 20 individuals²⁶26 Lynn MR. Determination and Quantification of Content Validity. J Experiment Psychol 1986; 136:382-386.^,2727 Haynes B, Haines A. Barriers and bridges to evidence based clinical practice. BMJ 1998; 317(7153):273-276.. In the present study, 12 researchers in the field of nutrition from different institutions were invited. Their expertise was in epidemiology, public health, collective food or gastronomy, and they had experience in topics related to the study and/or construction of instruments.

The preliminary version of the instrument was qualitatively evaluated and was performed in two stages: in the first phase, the proposed domains were evaluated, and in the second phase, the items were evaluated, including their relevance and clarity. Regarding relevance, whether each item expressed the expected concept, its relevance and its adequacy for the proposed objectives were evaluated. Regarding clarity, whether the wording of each item expressed the concept in a way that was understandable and consistent with what was expected to be measured was evaluated.

The experts also evaluated the instrument as a whole, examining its scope, i.e., whether each domain was adequately covered by the set of items presented and whether the dimensions proposed by Caspi et al.⁹9 Caspi CE, Sorensen G, Subramanian SV, Kawachi I. The local food environment and diet: A systematic review. Heal Place 2012; 18(5):1172-1187. and the components of the food environment proposed by Glanz et al.¹1 Glanz K, Sallis JF, Saelens BE, Frank LD. Healthy nutrition environments: Concepts and measures. Am J Heal Promot 2005; 19(5):330-333. that were expected to be covered by the instrument had been adequately considered.

Pretest

The final step was a pretest of the instrument. The pretest was conducted by the first author of this article at a commercial establishment located near one of the universities studied that had characteristics similar to those of the establishments of the universities participating in this study.

Assessment of the reliability of the instrument

To evaluate the reliability of the instrument, a convenience sample of establishments with different characteristics distributed over seven campuses of three universities was studied, namely, UFRJ: Cidade Universitária, Praia Vermelha and Professor Aluísio Teixeira (Macaé) campuses; UFF: Gragoatá and Valonguinho campuses; and Unirio: Pasteur and Biomedical Institute campuses. Based on publications on the topic¹⁴14 Horacek TM, Erdman MB, Reznar MM, Olfert M, Brown-Esters ON, Kattelmann KK, Kidd T, Koenings M, Phillips B, Quick V, Shelnutt KP, White AA. Evaluation of the food store environment on and near the campus of 15 postsecondary institutions. Am J Heal Promot 2013; 27(4):81-91.^,1717 Rodrigues CB, Monteiro LS, Paula NM, Pereira RA. Ambiente alimentar em um campus universitário: desenvolvimento e análise de instrumento para avaliação de estabelecimentos comerciais. Demetra 2021; 16:e51139., it was determined at least 50 establishments should be included. All establishments on the campuses of the universities that agreed to participate in the study were included.

Data collection was performed by 15 trained field researchers (nutrition and gastronomy students) using a printed form. The training lasted six hours and was performed in two stages: first, a theoretical stage, in which the purpose of the study, the instrument to be applied and the data collection manual were presented; and second, a practical stage, in which the instrument was applied by the researchers in at least two establishments under the supervision of the study coordinators.

The establishments included in the study were evaluated three times: two researchers (A and B) visited each establishment on the same day at a maximum interval of 30 minutes, and one of the researchers from each pair (A or B) performed a new data collection visit 15 to 30 days after the first evaluation. Data collection occurred between November 2015 and February 2016, which was an academic period at the universities.

Data entry was performed by double entry using Excel software version 2016, which was also used to evaluate the consistency of the database.

The reliability of the instrument was determined by interobserver and the test-retest evaluations.²⁰20 Streiner DL, Norman GR, Cairney J. Reliability. In: Health measurement scales: a practical guide to their development and use. New York: Oxford University Press; 2015. p. 179-199. For the categorical and count variables, the stability of the instrument was evaluated by calculating the percent agreement (PA)²⁹29 Luiz R. Métodos estatísticos em estudos de concordância. In: Medronho RA, Bloch KV, Luiz RR, Werneck GL. Epidemiologia. Atheneu: São Paulo; 2009. p. 343-370. and the kappa index (k), kappa index adjusted for prevalence and interviewer bias (ka) (the prevalence-adjusted, bias-adjusted kappa (PABAK)) for all variables³⁰30 Byrt T, Bishop J, Carlin JB. Bias, prevalence and kappa. J Clin Epidemiol 1993; 46(5):423-429.. For food prices, which were analyzed as continuous variables, the intraclass correlation coefficient (ICC)²⁰20 Streiner DL, Norman GR, Cairney J. Reliability. In: Health measurement scales: a practical guide to their development and use. New York: Oxford University Press; 2015. p. 179-199. was calculated. The PA, k, ka and ICC values were classified according to the criterion of Byrt³¹31 Byrt T. How good ist hat agreement? Epidemiology 1996; 32(5):561-561.: excellent agreement (≥0.92), very good agreement (0.80 to 0.91), good agreement (0.60 to 0.79), fair agreement (0.40 to 0.59), slight agreement (0.20 to 0.39), poor agreement (0.00 to 0.19), and no agreement (<0.00).

All analyses were performed using the Statistical Package for Social Science (SPSS 21) software. To calculate the PABAK and the ICC, it was necessary to create a specific macro.

The study was submitted to the Ethics Committee of the Pedro Ernesto Hospital of the State University of Rio de Janeiro and approved under opinion number 49988015.6.0000.5259.

Results

Content validation

In general, the content validity of the developed instrument was considered satisfactory by the experts, considering that the proposed domains and items included the conceptual dimensions of interest and that these dimensions covered the essential elements of the consumer food environment in the university context.

Based on the experts’ assessment, items were changed or added to or removed from the checklist. The main changes were inclusion of a new domain (“information”); the removal of a domain (“review of the menu”); reorganization of the “environmental observation” domain and the inclusion of 13 items in it; addition of the category “convenience items” to the “beverages, foods and preparations” domain and the inclusion of eight new types of ultra-processed beverages; inclusion of the form of presentation of advertisements in the “advertisements” domain.

The results of the pretest did not indicate a need for changes in the content or form of the instrument. All items were suitable for completion, and no situation of interest was identified that was not covered by the instrument, nor were any unnecessary items noted. The ordering of the modules in the instrument, as well as its formatting, was also shown to be adequate.

The final version of the developed instrument consisted of a checklist that covered the dimensions availability, affordability, convenience⁹9 Caspi CE, Sorensen G, Subramanian SV, Kawachi I. The local food environment and diet: A systematic review. Heal Place 2012; 18(5):1172-1187., nutritional information and food promotion¹1 Glanz K, Sallis JF, Saelens BE, Frank LD. Healthy nutrition environments: Concepts and measures. Am J Heal Promot 2005; 19(5):330-333. and included 204 items distributed across seven different domains, namely, characterization of the establishment; observation of the environment; information made available; beverages, foods and preparations; prices and promotions; availability of healthy substitutions; and advertisements (full text of the instrument is available athttp://www.observatoriodeobesidade.uerj.br/?p=3294), as detailed below:

The types of establishments were classified according to their predominant activity: by-the-kilo restaurants (essentially sells meals and foods by weight); a la carte/portioned/executive-style restaurants (essentially sells meals at a fixed quantity and price); free/rotating restaurant (essentially sells meals using a free (unweighed) or rotating system (variable quantity/weight and a fixed price) system; snack bar (sells mainly snacks/sandwiches and savory snacks and candy items (candy, sweets, ultra-processed beverages); bar (mainly sells alcoholic beverages, cigarettes, candies, snacks and food items); cafeteria (sells coffee and other beverages and, sometimes, foods that can be eaten quickly or light dishes); bomboniere (sells sweets, candies, chewing gum, chocolates, soft drinks, juices, sweets in general and ice cream); and mixed establishment: snack bar/cafeteria + meals by the kilo or snack bar/cafeteria + a la carte meals/portioned items/executive-style (sells different typical cafeteria products with portioned menu items or meals sold by weight)³²32 Tavares L, Perez P, Passos ME, Castro Junior PC, Franco A, Cardoso L, Castro I. Development and application of healthiness indicators for commercial establishments that sell foods for immediate consumption. Foods 2021; 10(6):1434..

To categorize the beverages included in the instrument, the grouping used by Euromonitor International³³33 Euromonitor International. International Marketing Data and Statistics 2013. Londres: Euromonitor International Limited; 2013. for sugary or sweetener-based beverages was used and adapted for the Brazilian market. The adopted categories were flavored (bottled) waters, carbonated or uncarbonated (they contain natural or artificial fruit flavors, usually do not contain artificial colors, may or may not contain sugar); soft drinks: cola and non-cola beverages, regular or low calorie; liquid or powder concentrates for reconstituted beverages; 100% fruit juices; juice-based beverages; nectars; ready-to-drink teas (e.g., iced-tea, green tea and mate with added sugar or other sweeteners); isotonic or replenishing beverages (e.g., sport drinks); energy drinks, generally those containing caffeine and taurine; guarana soft drinks; soy-based beverages, plain or flavored; coconut water; milk-based beverages: whey-based dairy beverages, flavored milks, and mixed milk and fruit preparations, with or without cereals or other items.

Food advertisements were categorized according to their form of presentation. Any word or phrase about the food/product that was presented in a banner/poster produced by the supplier, a banner/poster produced by the establishment, on employees’ clothing, on the product (advertising added to product packaging), on a replica of the product, on display in the establishment or even on the menu was considered.

Reliability of the instrument

A total of 64 establishments were included in the first phase of data collection, and 54 were included in the second round (10 refused to participate). The average duration of data collection at each establishment was 18 minutes.

Regarding the performance of the instrument, excellent or very good agreement was observed for 91.3% of the items evaluated. Regarding the interobserver test, for among the evaluated items (158), 68.3% and 96.5% had excellent, very good or good agreement according to the k (range of -0.51 to 1.00) and ka (range from 0.44 to 1.00), respectively (Table 1). Poor or null agreement was observed for 12% and 0.6% of the items according to the k and ka, respectively; these items were in the domains of characteristics of the establishment, healthy substitutions and nutritional information. For the items in the food prices domain, the ICC ranged from 0.05 to 1.00. Of the evaluated items (n=28), 88% showed excellent, very good or good agreement (Table 2). In this domain, 23% of the convenience items showed poor agreement.

In the test-retest evaluation, of the evaluated items (158), 65% had excellent, very good and good agreement according to k, which ranged from -0.05 to 1.00. Regarding the ka, which ranged from 0.37 to 1.00, 96.5% of the items showed excellent, very good or good agreement (Table 3). Considering k, there was poor or null agreement for 4.8% of the items, which were concentrated in the domains of environmental characteristics; beverages, foods and preparations; and information. For ka, there was no poor or null classification of any item. For the items in the food prices domain (n=28), the ICC ranged from 0.05 to 1.00; 77% showed excellent, very good or good agreement, and none of the items had poor or null agreement (Table 2).

Discussion

The results indicated satisfactory content validity and reliability (interobserver and test-retest) of the instrument for characterizing the consumer food environment at universities in the Brazilian context. Even for the test-retest evaluation, for which some of the differences can be explained by real variations in the environment, particularly with regard to food supply, good performance of the instrument was observed.

Although the performance of the instrument was generally satisfactory, some items showed low performance, especially in the interobserver reliability evaluation. This result suggests the need to improve the training provided for the application of the instrument. A similar situation was observed in a study that evaluated a checklist applied in a hospital food environment³⁴34 Messias GM. Ambiente alimentar hospitalar: desenvolvimento e avaliação de confiabilidade de um instrumento na rede pública de uma metrópole brasileira [tese]. Rio de Janeiro: Universidade do Estado do Rio de Janeiro; 2019.. The food environment is a complex context with significant diversity of establishments, foods, beverages and information, and thorough and standardized evaluation through the appropriate selection and training of evaluators is necessary⁴4 Borges CA, Jaime PC. Development and evaluation of food environment audit instrument: AUDITNOVA. Rev Saude Publica 2019; 53:91.. Although the results of the present study were good, the duration of training was shorter than that reported for other studies³3 Duran AC, Lock K, Latorre MRDO, Jaime PC. Evaluating the use of in-store measures in retail food stores and restaurants in Brazil. Rev Saude Publica 2015; 49:80.^,3535 Saelens BE, Glanz K, Sallis JF, Frank, LD. Nutrition Environment Measures Study in restaurants (NEMS-R): development and evaluation. Am J Prev Med 2007; 32(4):273-281..

Few studies in the literature were identified that determined the content validity of instruments for assessing the food environment, and none involved the university food environment. A review on the assessment of the food environment that included 432 studies published between 2007 and 2015 showed that 28.1% evaluated the validity of the instrument used, and 2.1% assessed the content validity of the instrument²²22 Lytle LA, Sokol RL. Measures of the food environment: A systematic review of the field, 2007-2015. Heal Place 2017; 44(2016):18-34.. Among those studies, that of Pomerleau et al.³⁶36 Pomerleau J, Knai C, Foster C, Rutter H, Darmon N, Derflerova Brazdova Z, Hadziomeragic AF, Pekcan G, Pudule I, Robertson A, Brunner E, Suhrcke M, Gabrijelcic Blenkus M, Lhotska L, Maiani G, Mistura L, Lobstein T, Martin BW, Elinder LS, Logstrup S, Racioppi F, McKee M. Measuring the food and built environments in urban centres: reliability and validity of the EURO-PREVOB Community Questionnaire. Public Health 2013; 127(3):259-267. is noteworthy, as it conducted a content validation procedure similar to that used in the present study; for the development of the instrument it used (the EURO-PREVOB Community Questionnaire), a literature review, expert consultations and a pretest were performed³³33 Euromonitor International. International Marketing Data and Statistics 2013. Londres: Euromonitor International Limited; 2013..

In the international and national literature, there are few studies on the reliability analysis of instruments for assessing the food environment in universities. In addition, because the existing studies use different instruments to evaluate the environment and because reliability is a context-dependent measure³⁷37 Kelly B, Flood VM, Bicego C, Yeatman H. Derailing healthy choices: an audit of vending machines at train stations in NSW. Heal Promot J Aust 2012; 23(1):73-75., the existing studies are not fully comparable with the present evaluation. Nevertheless, it is worth commenting on their results. In Australia, Roy et al.¹⁵15 Roy R, Hebden L, Kelly B, De Gois T, Ferrone EM, Samrout M, Vermont S, Allman-Farinelli M. Description, measurement and evaluation of tertiary-education food environments. Br J Nutr 2016; 115(9):1598-1606. developed a checklist for the evaluation of food availability, physical accessibility (e.g., products near the checkout counter, free access to shelves) and food promotions. The authors performed an interobserver reliability analysis through percentage agreement for two establishments and found 96.0% agreement¹⁵15 Roy R, Hebden L, Kelly B, De Gois T, Ferrone EM, Samrout M, Vermont S, Allman-Farinelli M. Description, measurement and evaluation of tertiary-education food environments. Br J Nutr 2016; 115(9):1598-1606..

In Brazil, Rodrigues et al.¹⁷17 Rodrigues CB, Monteiro LS, Paula NM, Pereira RA. Ambiente alimentar em um campus universitário: desenvolvimento e análise de instrumento para avaliação de estabelecimentos comerciais. Demetra 2021; 16:e51139. adapted a checklist to evaluate the university food environment that included items related to the description of the establishments and the evaluation, the availability of foods/preparations and facilitators and barriers to healthy eating. The interobserver and test-retest reliability was measured by kappa for 64 establishments. For interobserver agreement, the kappa ranged from -0.04 to 1.0. For the test-retest agreement, a variation of 0.06 to 1.0 was observed. The mean kappa index was 0.54 for interobserver agreement and 0.72 for test-retest agreement¹⁷17 Rodrigues CB, Monteiro LS, Paula NM, Pereira RA. Ambiente alimentar em um campus universitário: desenvolvimento e análise de instrumento para avaliação de estabelecimentos comerciais. Demetra 2021; 16:e51139..

It should also be noted that some instruments have been developed with the objective of evaluating the food environment in other scenarios, such as the community or other organizational spaces, such as work and school environments³3 Duran AC, Lock K, Latorre MRDO, Jaime PC. Evaluating the use of in-store measures in retail food stores and restaurants in Brazil. Rev Saude Publica 2015; 49:80.^,3737 Kelly B, Flood VM, Bicego C, Yeatman H. Derailing healthy choices: an audit of vending machines at train stations in NSW. Heal Promot J Aust 2012; 23(1):73-75.^,3838 Martins PA, Cremm EC, Leite FHM, Maron LR, Scagliusi FB, Oliveira MA. Validation of an adapted version of the nutrition environment measurement tool for stores (NEMS-S) in an Urban Area of Brazil. J Nutr Educ Behav 2013; 45(6):785-792.. In Brazil, it is worth mentioning three studies conducted in the state of São Paulo that evaluated the reliability of instruments for evaluating the food environment of restaurants³3 Duran AC, Lock K, Latorre MRDO, Jaime PC. Evaluating the use of in-store measures in retail food stores and restaurants in Brazil. Rev Saude Publica 2015; 49:80.^,44 Borges CA, Jaime PC. Development and evaluation of food environment audit instrument: AUDITNOVA. Rev Saude Publica 2019; 53:91. and food retailers³3 Duran AC, Lock K, Latorre MRDO, Jaime PC. Evaluating the use of in-store measures in retail food stores and restaurants in Brazil. Rev Saude Publica 2015; 49:80.^,3737 Kelly B, Flood VM, Bicego C, Yeatman H. Derailing healthy choices: an audit of vending machines at train stations in NSW. Heal Promot J Aust 2012; 23(1):73-75.^,3838 Martins PA, Cremm EC, Leite FHM, Maron LR, Scagliusi FB, Oliveira MA. Validation of an adapted version of the nutrition environment measurement tool for stores (NEMS-S) in an Urban Area of Brazil. J Nutr Educ Behav 2013; 45(6):785-792.. The authors evaluated the interobserver and test-retest reliability using the kappa and ICC and generally found good reliability for both tools³3 Duran AC, Lock K, Latorre MRDO, Jaime PC. Evaluating the use of in-store measures in retail food stores and restaurants in Brazil. Rev Saude Publica 2015; 49:80.^,44 Borges CA, Jaime PC. Development and evaluation of food environment audit instrument: AUDITNOVA. Rev Saude Publica 2019; 53:91.^,3838 Martins PA, Cremm EC, Leite FHM, Maron LR, Scagliusi FB, Oliveira MA. Validation of an adapted version of the nutrition environment measurement tool for stores (NEMS-S) in an Urban Area of Brazil. J Nutr Educ Behav 2013; 45(6):785-792.. Our study adds to these findings, providing evidence for an instrument that addresses the specificities of the university food environment, including establishments that sell food for immediate consumption (and thus have characteristics similar to those of restaurants and snack bars) but also sell items such as packaged snacks and cookies that are typically sold in supermarkets/markets/grocery stores and are not usually available in restaurants. The instrument evaluated in the present study assesses all types of food available at the university and avoids the need to use two different instruments.

Although most studies on the university food environment have been conducted in public HEIs, given the similarities between the contexts of the food environments of public and private universities, the instrument presented here may also be useful for the latter.

The study has some limitations. Because it used a convenience sample of public universities, its external validity is limited. In addition, although the elaboration of the instrument included consultation with experts in the field, quantitative measures were not used to verify the content validity. Another aspect that warrants mention is the fact that the data collection process lasted three months; therefore, it is possible that seasonal differences in food supply were not captured. Consequently, the possible implications of seasonality, if present, for the performance of the instrument are unknown. A fourth limitation is the fact that, for some of the items that were used to calculate kappa and PABAK, these statistics could not be computed due to the absence of one or more levels in the cross-referenced tables. However, this affected only a small portion of the items (approximately 10%). Thus, the generally good performance of the other items suggests that the performance of these items, if calculated, would not significantly impact the overall good performance of the instrument.

Among the strengths of this study is the evaluation of test-retest reliability, since most evaluations of audit tools only evaluate interobserver reliability at a single moment¹⁵15 Roy R, Hebden L, Kelly B, De Gois T, Ferrone EM, Samrout M, Vermont S, Allman-Farinelli M. Description, measurement and evaluation of tertiary-education food environments. Br J Nutr 2016; 115(9):1598-1606.. Another strength of the study is the broad scope of the adopted instrument. In addition to being a compilation and adaptation of elements from other instruments to better reflect the characteristics of the Brazilian university food environment, the present instrument was conceived and evaluated by researchers with different training profiles and who worked on university campuses with different characteristics, such as being located in the state capital vs. other municipalities and in an urban center vs. in a neighborhood outside the urban center; having two (morning and afternoon) vs. three (morning, afternoon and night) shifts; and having fewer vs. more structured establishments. Another aspect to highlight is the use of ka, which is still infrequently used in the health literature. In addition, the application of the instrument was tested on university campuses that had different characteristics.

Conclusion

The proposed instrument performed well in the context in which it was tested. It can support the characterization of the food environment in both public and private universities (and their surroundings, if the establishments have characteristics similar to those present on the university campuses), monitor their dynamics over time and support interventions for their improvement.

References

Publication Dates

History