Methodology and indicators for NBCAL monitoring in stores and health services: a multicenter study (Multi-NBCAL)

Cristiano Siqueira Boccolini Maria Inês Couto de Oliveira Tereza Setsuko Toma Patricia Lima Pereira Peres Evangelia Kotzias Atherino dos Santos Maria Cristina Passos Muriel Bauermann Gubert Naíza Nayla Bandeira de Sá Rodrigo Pinheiro de Toledo Vianna Daniela Lopes Gomes Talita Maria Alves Lopes da Silva Enilce de Oliveira Fonseca Sally Marina Ferreira Rea About the authors

Abstract

The Brazilian Code of Marketing of Infant and Toddlers Food, Nipples, Pacifiers and Baby Bottles (NBCAL), in force in Brazil since 1988, is still systematically violated, exposing mothers and family members to illegal marketing of products that compete with breastfeeding. This study aimed to describe a multicenter study methodology and propose standardized indicators for NBCAL monitoring. This is a Multicenter Study for NBCAL Compliance Assessment (Multi-NBCAL) conducted in seven Brazilian cities: Rio de Janeiro, São Paulo, Ouro Preto (Minas Gerais State), Florianópolis (Santa Catarina State), Brasília (Federal District), João Pessoa (Paraíba State), and Belém (Pará State). Assessment tools were adapted from NetCode/WHO and IBFAN Brazil (International Baby Food Action Network) to conduct two evaluations: (1) evaluation of NBCAL compliance in stores, and NBCAL knowledge and practices of store managers; (2) evaluation of the interaction between the baby food industry and health professionals and post-partum mothers in maternity hospitals. Five indicators were developed to assess NBCAL compliance in stores; seven indicators to assess the knowledge and practices of store managers; five indicators to assess the provision of incentives to maternity hospitals, health professionals, and mothers by sectors; and five indicators to assess NBCAL knowledge and practices of health professionals. The NBCAL assessment methodology with the proposal of standardized indicators allows comparability of studies about this theme. Using these indicators in periodic national or regional investigation can help monitor the level of NBCAL implementation in Brazil.

Keywords:
Breast Feeding; Marketing; Health Surveillance; Consumer Advocacy


Introduction

Manufacturers and distributors of baby food, nipples, pacifiers and baby bottles are constantly designing marketing strategies that negatively affect the choices and decisions of mothers and family members about how to feed their babies. These practices prevent society from fully exploring the various benefits of breastfeeding 11. United Nations Office of the High Commissioner for Human Rights. Breastfeeding a matter of human rights, say UN experts, urging action on formula milk. http://www.ohchr.org/EN/NewsEvents/Pages/DisplayNews.aspx?NewsID=20904&LangID=E%28 (acessado em 12/Set/2020).
http://www.ohchr.org/EN/NewsEvents/Pages...
,22. Victora CG, Bahl R, Barros AJ, França GV, Horton S, Krasevec J, et al. Breastfeeding in the 21st century: epidemiology, mechanisms, and lifelong effect. Lancet 2016; 387:475-90. and generate increasing sales margins and profits to companies worldwide 33. Rollins NC, Bhandari N, Hajeebhoy N, Horton S, Lutter CK, Martines JC, et al. Why invest, and what it will take to improve breastfeeding practices? Lancet 2016; 387:491-504.,44. Piwoz EG, Huffman S. The impact of marketing of breast-milk substitutes on WHO-recommended breastfeeding practices. Food Nutr Bull 2015; 36:373-86.. In this scenario, effective regulations for the marketing of foods and products that compete with breastfeeding have become a major priority for the World Health Organization (WHO). These regulations aim to provide an environment free from advertising influence to help increase breastfeeding rates and extend breastfeeding duration around the world 55. World Health Assembly Resolution. Guidance on ending inappropriate promotion of foods for infants and young children. http://www.who.int/nutrition/topics/guidance-inappropriate-food-promotion-iyc/en/ (acessado em 10/Set/2020).
http://www.who.int/nutrition/topics/guid...
.

The International Code of Marketing of Breast-Milk Substitutes (also known as the WHO Code), issued by the WHO in 1981 66. World Health Organization. International code of marketing of breast-milk substitutes. Geneva: World Health Organization; 1981., was developed to control the marketing of infant formulas and other products used as breast-milk substitutes. In 1988, the Brazilian Ministry of Health, based on the WHO Code, developed the Code for Commercialization of Food for Infants 77. Ministério da Saúde. Resolução CNS nº 5, de 20 de dezembro de 1988. Normas para comercialização de alimentos para lactentes. Diário Oficial da União 1988; 23 dez., which was later expanded and became the Brazilian Code of Marketing of Infant and Toddlers Food, Nipples, Pacifiers and Baby Bottles (NBCAL), adopted as an ordinance of the Brazilian Ministry of Health (Ordinance n. 2,051 of 200188. Agência Nacional de Vigilância Sanitária. Portaria nº 2.051, de 8 de novembro de 2001. Diário Oficial da União 2001; 9 nov.) and Resolutions RDC/ANVISA n. 22199. Agência Nacional de Vigilância Sanitária. Resolução RDC nº 221, de 5 de agosto de 2002. Diário Oficial da União 2002; 6 ago. and n. 2221010. Agência Nacional de Vigilância Sanitária. Resolução RDC nº 222, de 5 de agosto de 2002. Diário Oficial da União 2002; 6 ago. of 2002, and then as Law n. 11,265 in 20061111. Presidência da República. Lei nº 11.265, de 3 de janeiro de 2006. Regulamenta a comercialização de alimentos para lactentes e crianças de primeira infância e também a de produtos de puericultura correlatos. Diário Oficial da União 2006; 4 jan., regulated in 2015 and 2018 77. Ministério da Saúde. Resolução CNS nº 5, de 20 de dezembro de 1988. Normas para comercialização de alimentos para lactentes. Diário Oficial da União 1988; 23 dez.,1212. Brasil. Decreto nº 8.552, de 3 de novembro de 2015. Regulamenta a Lei nº 11.265, de 3 de janeiro de 2006, que dispõe sobre a comercialização de alimentos para lactentes e crianças de primeira infância e de produtos de puericultura correlatos. Diário Oficial da da União 2015; 4 jan.,1313. Brasil. Decreto nº 9.579, de 22 de novembro de 2018. Consolida atos normativos editados pelo Poder Executivo Federal que dispõem sobre a temática do lactente, da criança e do adolescente e do aprendiz, e sobre o Conselho Nacional dos Direitos da Criança e do Adolescente, o Fundo Nacional para a Criança e o Adolescente e os programas federais da criança e do adolescente, e dá outras providências. Diário Oficial da União 2018; 23 nov.. The acronym NBCAL is not described in official government regulations and ordinances, but it is widely adopted and accepted by civil society, government and academia.

The WHO Code and NBCAL have an interface and direct agreement with the universal human rights 1414. Grummer-Strawn LM, Zehner E, Stahlhofer M, Lutter C, Clark D, Sterken E, et al. New World Health Organization guidance helps protect breastfeeding as a human right. Matern Child Nutr 2017; 13:e12491., the 2030 Agenda and its Sustainable Development Goals (SDGs) 1515. Whitmee S, Haines A, Beyrer C, Boltz F, Capon AG, Dias AGS, et al. Safeguarding human health in the Anthropocene epoch: report of the Rockefeller Foundation-Lancet Commission on Planetary Health. Lancet 2015; 386:1973-2028.. They also observe basic constitutional guarantees, which include food as a social right 1616. Presidência da República. Emenda Constitucional nº 64, de 4 de fevereiro de 2010. Altera o art. 6º da Constituição Federal, para introduzir a alimentação como direito social. Diário Oficial da União 2010; 4 fev. and support other national laws and policies, such as the Brazilian National Food and Nutrition Policy (PNAN) 1717. Departamento de Atenção Básica, Secretaria de Atenção à Saúde, Ministério da Saúde. Política Nacional de Alimentação e Nutrição. Brasília: Ministério da Saúde; 2012. and the Brazilian National Policy for Integral Attention to Children’s Health (PNAISC) 1818. Ministério da Saúde. Portaria nº 1.130, de 5 de agosto de 2015. Institui a Política Nacional de Atenção Integral à Saúde da Criança (PNAISC) no âmbito do Sistema Único de Saúde (SUS). Diário Oficial da União 2015; 6 ago..

However, despite the broad scope of the WHO Code and NBCAL supporting breastfeeding protection 1919. Prado I, Rinaldi AE. Compliance of infant formula promotion on websites of Brazilian manufacturers and drugstores. Rev Saúde Pública 2020; 54:12.,2020. Pérez-Escamilla R, Curry L, Minhas D, Taylor L, Bradley E. Scaling up of breastfeeding promotion programs in low- and middle-income countries: the "breastfeeding gear" model. Adv Nutr 2012; 3:790-800., and the International Baby Food Action Network (IBFAN Brazil) monitoring of NBCAL violations in several Brazilian States for over 40 years 2121. Rede Internacional em Defesa do Direito de Amamentar Brasil. Manual do curso de capacitação em monitoramento. http://www.ibfan.org.br/documentos/ibfan/doc-678.pdf (acessado em 21/Abr/2021).
http://www.ibfan.org.br/documentos/ibfan...
, conflicts of business interest in the country and the lack of systematic NBCAL enforcement have led to frequent violations of the Code 2222. Silva KB, Oliveira MIC, Boccolini CS, Sally EOF. Illegal commercial promotion of products competing with breastfeeding. Rev Saúde Pública 2020; 54:10.. In addition, store managers, health professionals, and mothers have poor knowledge of NBCAL 2323. Silva LIMM, Thé PMP, Medeiros JO, Giacomini SGMO, Rodrigues RCB, Peixoto MMLV. Conhecimento de farmacêuticos sobre aleitamento materno: um estudo nas farmácias comerciais em Fortaleza - CE. Rev Bras Promoç Saúde 2012; 25:482-91., which creates an environment that favors non-compliance with this Law and exposure of families to abusive marketing of these products and baby foods 2222. Silva KB, Oliveira MIC, Boccolini CS, Sally EOF. Illegal commercial promotion of products competing with breastfeeding. Rev Saúde Pública 2020; 54:10..

The Multicenter Study for NBCAL Compliance Assessment (Multi-NBCAL), conducted in seven cities in the five Brazilian macro-regions, used a combination of quantitative and qualitative methods to analyze NBCAL compliance by stores and health services in Brazil and assess the perception of mothers about the influence of marketing on their food choices for their young children, which will be analyzed in another publication.

This study aimed to discuss the quantitative aspect of the method used in the Multi-NBCAL study to assess NBCAL compliance of stores and maternity hospitals, providing research protocols, variables and standardized indicators.

Methods

Multicenter study Multi-NBCAL

This is a Multi-NBCAL conducted in seven Brazilian cities: Rio de Janeiro, São Paulo, Ouro Preto (Minas Gerais State), Florianópolis (Santa Catarina State), Brasília (Federal District), João Pessoa (Paraíba State), and Belém (Pará State), between November 2018 and November 2019, using two components: (1) assessment of stores; (2) assessment of maternity hospitals.

Selection of study centers

Study centers were selected if covering at least one city from each of the five Brazilian macro-regions. The cities were chosen intentionally, according to the availability of institutions and researchers dedicated to the theme.

Fieldwork organization and questionnaire development

Recurrent in-person and virtual meetings were held with the study centers to validate the questionnaire content and the interviewer’s manual, define strategies for fieldwork, data collection and analysis of results, and create a schedule comprising all research project stages. Content was validated by a panel of experts in a process involving collective reading of every question, followed by approval or alteration of questions, field testing, review and reapplication, until all questions (and skip patterns) were approved. Later, in each center, the questionnaires were analyzed by experts not belonging to the research group, and adjustments were made to the instrument.

Two questionnaires were developed for the evaluation of stores: (1) assessment of NBCAL compliance in drugstores and supermarkets; (2) assessment of NBCAL knowledge and practices of store managers. For the evaluation of maternity hospitals, an environment observation instrument and a questionnaire were developed to assess the influence of the infant food industry and products covered by NBCAL on maternity hospitals, health professionals, and postpartum women. In both stages, Law n. 11,265/20061111. Presidência da República. Lei nº 11.265, de 3 de janeiro de 2006. Regulamenta a comercialização de alimentos para lactentes e crianças de primeira infância e também a de produtos de puericultura correlatos. Diário Oficial da União 2006; 4 jan. was considered as the theoretical reference for the questionnaires, as it has legal precedence over decrees 1212. Brasil. Decreto nº 8.552, de 3 de novembro de 2015. Regulamenta a Lei nº 11.265, de 3 de janeiro de 2006, que dispõe sobre a comercialização de alimentos para lactentes e crianças de primeira infância e de produtos de puericultura correlatos. Diário Oficial da da União 2015; 4 jan.,1313. Brasil. Decreto nº 9.579, de 22 de novembro de 2018. Consolida atos normativos editados pelo Poder Executivo Federal que dispõem sobre a temática do lactente, da criança e do adolescente e do aprendiz, e sobre o Conselho Nacional dos Direitos da Criança e do Adolescente, o Fundo Nacional para a Criança e o Adolescente e os programas federais da criança e do adolescente, e dá outras providências. Diário Oficial da União 2018; 23 nov. and technical regulations 88. Agência Nacional de Vigilância Sanitária. Portaria nº 2.051, de 8 de novembro de 2001. Diário Oficial da União 2001; 9 nov.,99. Agência Nacional de Vigilância Sanitária. Resolução RDC nº 221, de 5 de agosto de 2002. Diário Oficial da União 2002; 6 ago.,1010. Agência Nacional de Vigilância Sanitária. Resolução RDC nº 222, de 5 de agosto de 2002. Diário Oficial da União 2002; 6 ago.. Items not included in the Law, such as marketing of nipple protectors, but listed in Resolution RDC/ANVISA 222/20021010. Agência Nacional de Vigilância Sanitária. Resolução RDC nº 222, de 5 de agosto de 2002. Diário Oficial da União 2002; 6 ago., were included in the Multi-NBCAL study.

Multi-NBCAL was designed before the 2018 Presidential Decree n. 9,5971313. Brasil. Decreto nº 9.579, de 22 de novembro de 2018. Consolida atos normativos editados pelo Poder Executivo Federal que dispõem sobre a temática do lactente, da criança e do adolescente e do aprendiz, e sobre o Conselho Nacional dos Direitos da Criança e do Adolescente, o Fundo Nacional para a Criança e o Adolescente e os programas federais da criança e do adolescente, e dá outras providências. Diário Oficial da União 2018; 23 nov. which changes the definition of early childhood. Law n. 11,265/20061111. Presidência da República. Lei nº 11.265, de 3 de janeiro de 2006. Regulamenta a comercialização de alimentos para lactentes e crianças de primeira infância e também a de produtos de puericultura correlatos. Diário Oficial da União 2006; 4 jan. defines early childhood as the period from birth to three years old, while Decree n. 9,579/20181313. Brasil. Decreto nº 9.579, de 22 de novembro de 2018. Consolida atos normativos editados pelo Poder Executivo Federal que dispõem sobre a temática do lactente, da criança e do adolescente e do aprendiz, e sobre o Conselho Nacional dos Direitos da Criança e do Adolescente, o Fundo Nacional para a Criança e o Adolescente e os programas federais da criança e do adolescente, e dá outras providências. Diário Oficial da União 2018; 23 nov. defines early childhood as the period from birth to six years old. Although the law prevails over a decree, future monitoring must consider these discrepancies until a norm defines the valid age.

The questionnaires for these stages were adapted from those used by IBFAN to monitor NBCAL compliance 2121. Rede Internacional em Defesa do Direito de Amamentar Brasil. Manual do curso de capacitação em monitoramento. http://www.ibfan.org.br/documentos/ibfan/doc-678.pdf (acessado em 21/Abr/2021).
http://www.ibfan.org.br/documentos/ibfan...
and the NetCode Toolkit (https://toolkits.knowledgesuccess.org/toolkits/breastfeeding-advocacy-toolkit/netcode) proposed by the WHO/UNICEF to monitor the WHO Code 2424. World Health Organization, United Nations Children's Fund. NetCode toolkit. Monitoring the marketing of breast-milk substitutes: protocol for ongoing monitoring systems. Geneva: World Health Organization; 2017., as well as questions regarding the study objective with a proper structure for data collection through mobile application. Regarding NetCode questionnaires, the questions were adapted for NBCAL assessment. Foods and products considered in the questionnaire for the evaluation of stores observed the definitions contained in Law n. 11,265/20061111. Presidência da República. Lei nº 11.265, de 3 de janeiro de 2006. Regulamenta a comercialização de alimentos para lactentes e crianças de primeira infância e também a de produtos de puericultura correlatos. Diário Oficial da União 2006; 4 jan., art. 2. Although dairy compounds are not mentioned in the Law, they were included in the Multi-NBCAL assessment because they are for children aged 1 year or older. The questionnaires had sections, which will be detailed under the Results.

In each study center, the questionnaires were pre-tested in districts or municipalities other than those selected for the study, and changes were made to the questionnaire structure, i.e., question inclusion or removal, and skip pattern rules were created in this stage; then the final questionnaire was provided to each center.

The questionnaires were developed using MAGPI 2525. Abbot A. Paper, paper, everywhere... Nature 2005; 437:310. application (https://www.magpi.com/), which has an intuitive interface for data entry, electronic cloud hosting, online monitoring of data collection, and data export to common statistical packages.

This study was approved by the Research Ethics Committee of the Escola Politécnica de Saúde Joaquim Venâncio, Oswaldo Cruz Foundation (Fiocruz), process n. 2,912,729 of September 24, 2018, in agreement with CNS Resolution n. 466/20122626. Conselho Nacional de Saúde. Resolução CNS nº 466, de 12 de dezembro de 2012. Aprova diretrizes e normas regulamentadoras de pesquisas envolvendo seres humanos. Revoga a Resolução. 196/96, Resolução nº 303/00 e a Resolução nº 404/08. Diário Oficial da União 2013; 13 jun..

Results

Assessment of NBCAL compliance by stores and knowledge and practices of store managers

Study design and sample

The first study was an epidemiological investigation assessing NBCAL compliance by stores and knowledge and practices of managers or pharmacists from such stores. The study analyzed drugstores and supermarkets that sold baby food and products within the scope of NBCAL through interviews conducted with the managers or pharmacists in charge of the stores. Compounding pharmacies were excluded.

The study defined a minimum number of 200 stores for each center in a purposive sample; this sample number was calculated to ensure comparability between centers, based on 50% estimated prevalence of NBCAL violation, a 3% margin of error, and a 95% confidence interval (95%CI).

In the city of Rio de Janeiro, all stores in the south zone were evaluated using the same methodology as Silva et al. 2222. Silva KB, Oliveira MIC, Boccolini CS, Sally EOF. Illegal commercial promotion of products competing with breastfeeding. Rev Saúde Pública 2020; 54:10., which allows the assessment of stores in neighborhoods of different socioeconomic levels and a comparison of these stores over time.

In Belém and Ouro Preto, all drugstores and supermarkets were mapped, since 200 or fewer stores were found in these locations. In João Pessoa, São Paulo, Brasília, and Florianópolis, a region of contiguous districts was intentionally chosen to cover populations of different socioeconomic levels comprising at least 200 eligible stores. The selection of neighborhoods in these locations was discussed and planned in detail to ensure socioeconomic diversity. Prior mapping of stores in each center was conducted through internet search and knowledge of local researchers. Internet search was based on the standards of Silva et al. 2222. Silva KB, Oliveira MIC, Boccolini CS, Sally EOF. Illegal commercial promotion of products competing with breastfeeding. Rev Saúde Pública 2020; 54:10. using electronic search engine (http://www.google.com) and websites of main chains of stores in each center.

Team training and data collection

The interviewers selected at each center had a 20-hour course about NBCAL compliance assessment, followed by data collection training at stores and maternity hospitals. The course and training were provided at each study center by a member of the central coordination, the coordinator of each center, and was supported by a member from local IBFAN, when available.

Data were collected by interviewers in drugstores, markets and supermarkets in selected districts, assessing all eligible stores in that location, previously listed or not.

For data collection, interviewers used smartphones, displaying an electronic questionnaire developed in MAGPI (https://www.magpi.com/) 2525. Abbot A. Paper, paper, everywhere... Nature 2005; 437:310. application. The questionnaire had three sections: store identification; NBCAL compliance assessment; interview with store manager or pharmacist about their characteristics and knowledge about NBCAL and breastfeeding, visits from industry representatives, as well as questions about who organized products and foods on the shelves and their sales prices. Data collection was conducted in all centers between November and December 2018.

Assessment of stores

Compliance with NBCAL by stores was assessed considering items from Chapters I and II of Law n. 11,265/20061111. Presidência da República. Lei nº 11.265, de 3 de janeiro de 2006. Regulamenta a comercialização de alimentos para lactentes e crianças de primeira infância e também a de produtos de puericultura correlatos. Diário Oficial da União 2006; 4 jan. related to sales and advertising of products covered by NBCAL at points of sale (Box 1).

Box 1
Variables for Brazilian Code of Marketing of Infant and Toddlers Food, Nipples, Pacifiers and Baby Bottles (NBCAL) monitoring in stores (Multi-NBCAL, 2020).

The outcome of this study was NBCAL violation, characterized as non-compliance with one or more items related to the commercial promotion of products and foods within the scope of NBCAL in stores 99. Agência Nacional de Vigilância Sanitária. Resolução RDC nº 221, de 5 de agosto de 2002. Diário Oficial da União 2002; 6 ago.. Two main groups were considered: group 1 - foods and products of prohibited commercial promotion; and group 2 - foods of commercial promotion permitted with a warning phrase (Box 1). In group 1 products, the identification of any commercial promotion was considered an NBCAL violation. In group 2 products, commercial promotion was considered a violation without a warning phrase, as recommended by the Brazilian Ministry of Health 99. Agência Nacional de Vigilância Sanitária. Resolução RDC nº 221, de 5 de agosto de 2002. Diário Oficial da União 2002; 6 ago. (Box 1).

The unit of analysis was the product sold at stores and each product could or could not have one or more NBCAL violations. The total number of products with NBCAL violations in each store could range from zero (no violation) to the total distinct products sold at the store. The total number of products with NBCAL violations was continuously computed (counting). Then, violations were categorized in a binary manner, indicating whether the assessed store presented NBCAL violations or not (Box 2).

Box 2
Indicators for Brazilian Code of Marketing of Infant and Toddlers Food, Nipples, Pacifiers and Baby Bottles (NBCAL) monitoring in stores (Multi-NBCAL, 2020).

Script for store assessment

First, information was collected about the characteristics of each store, including address, type of store (drugstore or market/supermarket); whether it belonged to a chain of stores (yes or no); its location (GPS marking, city, district, address); and size (small, midsize, large, hyperstore) - subjectively assessed by the interviewer.

Then, all products sold at the stores covered by NBCAL were registered. Also, the assessment and identification of NBCAL violation type were performed (Box 1). As approved by the Research Ethics Committee, this stage of the study did not require an Informed Consent Form (ICF) to assess the public environment of stores in order to prevent the manager or pharmacist from modifying the environment when learning about the study objectives.

When a product was identified with an NBCAL violation, the manufacturer name and the product name were registered. Then, the type of commercial promotion (off-price, special display, gifts or samples, and others) was registered, with multiple answers for each product. For products belonging to group 2, with commercial promotion allowed, the presence of a warning phrase and its adequacy (Box 1) were checked. NBCAL violations were registered product by product until all infractions were exhausted.

After a complete assessment of the store environment, the interviewers introduced themselves to the manager or pharmacist in charge and applied the questionnaire after reading the informed consent form and obtaining a verbal consent. A copy of the ICF signed by the study coordinator was provided to the interviewees.

The interview also analyzed the knowledge of managers or pharmacists of Law n. 11,265/2006, the name of the companies that visited the store, the frequency of these visits, whether they provided guidance to customers about infant food, as well as commercial practices influenced by this industry such as bonus offered for better product display and incremental sales (Box 3).

Box 3
Indicators to assess knowledge and practices of managers or pharmacists in charge of stores (Multi-NBCAL, 2020).

Once the store assessment ended, the electronic questionnaire was sent over the internet and stored as an electronic file. Collected variables were then categorized (Box 1) and the indicators were calculated (Boxes 2 and 3). Multi-NBCAL developed five indicators to assess NBCAL compliance in stores and six indicators to assess the knowledge and practices of managers or pharmacists in charge of the store.

Assessment of marketing practices of manufacturers of products covered by NBCAL with health professionals, institutions and post-partum women

Study design and sample

The second Multi-NBCAL study was an epidemiological investigation conducted in maternity hospitals in six Brazilian cities: Rio de Janeiro, São Paulo, Ouro Preto, Florianópolis, Brasília, and João Pessoa, where aspects related to Chapters I, II, and IV of NBCAL were evaluated, regarding financial or material sponsorship by the food industry and products within the scope of NBCAL, and about education and public information, such as educational materials 1111. Presidência da República. Lei nº 11.265, de 3 de janeiro de 2006. Regulamenta a comercialização de alimentos para lactentes e crianças de primeira infância e também a de produtos de puericultura correlatos. Diário Oficial da União 2006; 4 jan..

In each city, all hospitals/maternity hospitals that performed more than 500 births a year were mapped via search on the National Registry of Health Centers (CNES, http://www.cnes.datasus.gov.br). Then, the hospitals were sorted by funding source as public, public accredited by the Baby-Friendly Hospital Initiative (BFHI), and private. The sample aimed to include two hospitals from each stratum in each city. However, Florianópolis and Outro Preto had only four and one maternity hospitals, respectively. In Rio de Janeiro and São Paulo, three private maternity hospitals refused to participate in the study, and obstacles to obtain Institutional Acceptance Terms (IAT) prevented to reach the previously planned sample.

Data collection

At each maternity hospital, the interview with health professionals took place on a single day and included up to 10 professionals, including pediatricians, nutritionists, and speech therapists. These professionals were selected either because they are included in Law n. 11,265/2006 (nutritionists and pediatricians), or because they use children’s products covered by NBCAL in their professional practice, such as speech therapists. For maternity hospitals with less than 10 professionals present on the interview day, all of them were interviewed. When more than 10 professionals were present, a list of pediatricians, nutritionists and speech therapists present on the interview day was requested, and up to 10 professionals were randomly selected per maternity hospital. In addition, the director of each maternity hospital was interviewed. The physical environment of the maternity was also observed, including the areas of patient circulation and the areas for pediatricians, nutritionists and speech therapists. All interviews were conducted with ICF signing.

The interviewers and fieldwork supervisors were the same as those from the store assessment stage. Data collection took place between November 2018 and November 2019.

Data were collected in 25 hospitals with maternity hospitals: 6 (1 private, 3 public, and 2 public BFHI hospitals) in Rio de Janeiro; 4 (1 private, 2 public, and 1 public BFHI hospitals) in São Paulo; 4 (2 private, 2 public BFHI hospitals) in Florianópolis; all 6 maternity hospitals (2 private and 4 public BFHI hospitals) in João Pessoa; 4 (2 private, 1 public, and 1 public BFHI hospitals) in Brasília; and the only public BFHI maternity hospital in Ouro Preto. In these maternity hospitals, 217 health professionals were interviewed: 14 directors, 31 heads of services or departments, 170 health care professionals, and 2 university professors.

Assessment of maternity hospitals

For the assessment of maternity hospitals, the study was conducted in three modules:

(1) Module I - environment: presence of educational and technical-scientific materials produced by manufacturers in environments restricted to health professionals and patients; presence of equipment, gifts or free samples provided by companies related to NBCAL in these environments.

(2) Module I - characterization of maternity hospitals and assessment of incentives offered by manufacturers to: (a) maternity hospitals; (b) health professionals; and (c) post-partum women.

(3) Module - characterization of health professionals, including age, sex, race/skin color, profession, time since graduation, and place of work in the maternity hospital. The interviews with the professionals also addressed their knowledge about NBCAL and their participation in a class or course about NBCAL. Finally, they were asked about their participation in courses and congresses about breastfeeding or infant food in the last two years, whether these events were sponsored by manufacturers, and whether they received personal incentive, such as congress registration, meal during the congress or another incentive.

The outcome of this study was NBCAL violation by manufacturers, characterized as non-compliance with articles 6, 8, and 18 of Law n. 11,265/2006, through the provision of incentives to health professionals, maternity hospitals, and post-partum women. Article 6 prohibits actions of commercial representatives in health facilities, except for the communication of technical and scientific aspects of products to pediatricians and nutritionists. Article 8 stipulates that manufacturers of products covered by Law n. 11,265/2006 are allowed to grant financial sponsorships only to scientific educational and research entities or nationally recognized pediatric and nutritionist associations, and any sponsorship to individuals is prohibited. Article 18 prohibits the production of educational materials for consumers addressing infant food by manufacturers or distributors of products covered by NBCAL.

Box 4 shows the indicators whose unit of analysis is maternity hospitals: one indicator measures the industry approach to maternity hospitals and three indicators assess industry infractions by offering incentives to maternity hospitals, health professionals who work in maternity hospitals, and assisted post-partum women. Box 5 shows indicators whose unit of analysis is health professionals, which measure their knowledge and training about NBCAL, their participation in scientific congresses or events sponsored by the food and infant product industry, and NBCAL violation by health professionals as they received personal incentive at these events.

Box 4
Indicators to assess the provision of incentive to maternity hospitals, health professionals and post-partum women from companies of infant foods and products (Multi-NBCAL, 2020).
Box 5
Indicators to assess the knowledge of health professionals about Brazilian Code of Marketing of Infant Toddlers Food, Nipples, Pacifiers and Baby Bottles (NBCAL) and NBCAL violations related to receiving personal incentives from companies of infant foods or products for congresses or events sponsored by such organizations (Multi-NBCAL, 2020).

Discussion

The Multi-NBCAL innovates by developing and systematizing a methodology for compliance assessment, with the standardization of questionnaires and indicators for the assessment of stores and maternity hospitals. The methodology and the indicators were applied in cities in the five Brazilian macro-regions and constitute an essential tool for the assessment of the magnitude and number of Law n. 11,265/2006 violations to monitor the temporal evolution of law compliance, revealing different promotion strategies in different regions.

The Multi-NBCAL methodology is in line with the food and nutrition surveillance and food control guidelines of the PNAN 1717. Departamento de Atenção Básica, Secretaria de Atenção à Saúde, Ministério da Saúde. Política Nacional de Alimentação e Nutrição. Brasília: Ministério da Saúde; 2012., and axis 2 of the PNAISC, as it protects breastfeeding 1818. Ministério da Saúde. Portaria nº 1.130, de 5 de agosto de 2015. Institui a Política Nacional de Atenção Integral à Saúde da Criança (PNAISC) no âmbito do Sistema Único de Saúde (SUS). Diário Oficial da União 2015; 6 ago., helping mothers and family members make informed decisions about the best foods for their children, free from marketing influence and corporate practices for products covered by NBCAL.

Studies addressing NBCAL compliance in stores have been conducted in different areas in Brazil. Some have reported discounts, special displays, gifts, and different presentation of baby bottles, nipples and pacifiers in drugstores and supermarkets, including an investigation conducted in Piracicaba (São Paulo State) in 2012 2727. Lopes AG, Pereira AC, Fonseca EP, Mialhe FL. Irregularidades sanitárias na promoção comercial em rótulos de produtos para lactentes e os riscos para a saúde. Saúde Debate 2017; 41:539-52.. Other studies have found violations in the commercialization of infant formulas, as observed in a chain of supermarkets in Teresina (Piauí State), in 2009 2828. Paula LO, Chagas LR, Ramos CV. Monitoramento da norma brasileira de comercialização de alimentos infantis. Nutrire Rev Soc Bras Alim Nutr 2010; 35:43-55.. Irregular promotion of these two types of products were also observed in 2016 in a study conducted in 54.3% of all supermarkets in Mossoró (Rio Grande do Norte State) 2929. Gurgel TEP. Monitoramento da promoção comercial de alimentos para lactentes e crianças de primeira infância e de produtos de puericultura em estabelecimentos comerciais de Mossoró. Nutrivisa - Revista de Nutrição e Vigilância em Saúde 2016; 1:21-5.. But what is the magnitude of violations reported in these scenarios? What parameters were used to measure them? How to respond to these issues without standard indicators developed for this purpose?

Regarding the NBCAL compliance assessment in health units, the observation of gifts with infant formula advertising displayed in doctor’s offices, as well as manufacturers’ funding to support research, equipment, travel, and congress registration, generate undeniable conflicts of interest in the relationship between manufacturers and health professionals 3030. Rea MF, Toma TS. Proteção do leite materno e ética. Rev Saúde Pública 2000; 34:388-95..

In the international literature, investigations in hospitals report sponsorship of health professionals to participate in scientific events and receive promotional gifts, such as pens, pads, measuring tapes and stethoscopes displaying logos of companies like Nestlé (Switzerland) and Danone (France), in Africa 3131. Aguayo VM, Ross JS, Kanon S, Ouedraogo AN. Monitoring compliance with the International Code of Marketing of Breastmilk Substitutes in West Africa: multisite cross sectional survey in Togo and Burkina Faso. BMJ 2003; 326:127., Pakistan 3232. Salasibew M, Kiani A, Faragher B, Garner P. Awareness and reported violations of the WHO International Code at Pakistan's national breastfeeding legislation: a descriptive cross-sectional survey. Int Breastfeed J 2008; 3:24., and China 3333. Liu A, Dai Y, Xie X, Chen L. Implementation of International Code of Marketing Breast-Milk Substitutes in China. Breastfeed Med 2014; 9:467-72.. The display and size of gifts with the logo of baby food companies in maternity hospitals can be perceived as an endorsement of these products by assisted mothers.

The Multi-NBCAL methodology was based on what IBFAN Brazil 2121. Rede Internacional em Defesa do Direito de Amamentar Brasil. Manual do curso de capacitação em monitoramento. http://www.ibfan.org.br/documentos/ibfan/doc-678.pdf (acessado em 21/Abr/2021).
http://www.ibfan.org.br/documentos/ibfan...
proposes and NetCode Toolkit 2424. World Health Organization, United Nations Children's Fund. NetCode toolkit. Monitoring the marketing of breast-milk substitutes: protocol for ongoing monitoring systems. Geneva: World Health Organization; 2017.. However, NetCode indicators for stores consider the product 2323. Silva LIMM, Thé PMP, Medeiros JO, Giacomini SGMO, Rodrigues RCB, Peixoto MMLV. Conhecimento de farmacêuticos sobre aleitamento materno: um estudo nas farmácias comerciais em Fortaleza - CE. Rev Bras Promoç Saúde 2012; 25:482-91. as their unit of analysis, not the store. Multi-NBCAL, in compliance with Law n. 11,265/2006, considers stores are responsible for complying with the law and proposes indicators to measure such compliance. It allows mapping and identifying corporate marketing approaches and practices of manufacturers with store managers, health professionals and maternity hospitals, exploring noncompliance of law and ethical issues.

The indicator assessing maternity hospitals and primary health units proposed by NetCode uses institutions as a unit of analysis, presenting the analysis by industry. Multi-NBCAL offers the same pattern of analysis, but additionally proposes indicators to analyze manufacturers as the unit of analysis and identify the scope of illegal incentive practices, that is, NBCAL violations. This way, such information can be analyzed in the perspective of maternity hospitals and manufacturers.

Regarding the sampling process, both NetCode and Multi-NBCAL use a purposive sample. In NetCode, a health unit (maternity hospital or outpatient clinic) is the starting point, and from then, a number of surrounding stores is mapped. In Multi-NBCAL, the sample of stores is mapped in an area with different social strata, with all stores selected in this area, ensuring sample representativeness. Likewise, the sample of maternity hospitals ensures representativeness by including the public and private sectors, with and without adherence to the BFHI.

IBFAN has performed an annual monitoring of NBCAL for over 40 years, identifying different NBCAL violations 3434. Instituto Brasileiro de Defesa do Consumidor. Amamentação desvalorizada. Revista do IDEC 2015; (195):22-24. http://www.idec.org.br/uploads/revistas_materias/pdfs/195-amamentacao1.pdf (acessado em 13/Set/2020).
http://www.idec.org.br/uploads/revistas_...
. However, it does not aim to offer a systematic analysis to identify the proportion of stores and hospitals that fail to comply with NBCAL. This aspect was developed in Multi-NBCAL, which enables to identify the magnitude and frequency of NBCAL infractions in a certain territory.

Lessons learned

The Multi-NBCAL methodology construction is part of a collective construction process, in which a dialogue between academia and organized civil society (in this case IBFAN) took place in an intense and productive manner. This research project involved different challenges. For instance, structuring a questionnaire that allowed a continuous flow of registration of NBCAL violations in stores and a questionnaire to assess maternity hospitals and health professionals, associated with the absence of national indicators for NBCAL monitoring, were intensely debated issues. These questionnaires were pre-tested in training sessions until they could be used in different scenarios.

For operational reasons, other aspects of NBCAL were not considered in the study scope, for instance, assessment of product labeling of the items covered by NBCAL, marketing in social media channels, content of educational materials, and adequacy of free sample delivery practices. Future studies may standardize specific methodologies and indicators to address these aspects.

Ethical issues constituted another challenging aspect. Although the project was approved by the Research Ethics Committee of the study institution and centers, some maternity hospitals did not authorize study participation and had to be replaced with others, delaying the study start by 8 months. Despite these issues, the multicenter configuration of Multi-NBCAL ensured autonomy of center coordinators, who managed their activities and use of data collected in their respective centers.

The use of smartphones with an application that contained all questionnaires allowed store assessment without attracting the attention of store managers. In a study conducted by Oliveira et al. 3535. Oliveira MIC, Boccolini CS, Sally EOF. Breast milk substitutes marketing violations and associated factors in Rio de Janeiro, Brazil. J Hum Lact 2021; 37:158-68., tablets and printed forms were used to assess compliance with NBCAL in stores, which occasionally generated curiosity of store managers at the assessment moment and an attempt to change the environment by removing commercial promotions that violated NBCAL (unpublished data). The smartphone application also allowed real-time virtual monitoring of data collection.

One of the effects of this study was the development by Fiocruz of a free NBCAL monitoring system that uses the Multi-NBCAL 3636. Boccolini C, Boccolini P, Mezzavilla R. National System to Monitor the International Code of Breast Milk Substitutes in Brazil: SisNBCAL. Curr Dev Nutr 2020; 4 Suppl 2:705. methodology, in addition to training courses for human resource with a focus on health and the engagement of undergraduate students, PIBIC scholarship holders and graduate students in Multi-NBCAL.

Study limitations refer to differences in store sampling between the centers. Purposive samples were used in most cities, choosing areas with different population segments, except in Ouro Preto and Belém, where censuses were performed due to the impossibility of obtaining an accurate list of all stores, either over the Internet or from government agencies, which did not allow a random selection.

Another limitation was the selection of maternity hospitals, with lower representativeness of private maternity hospitals due to their refusal to participate in Rio de Janeiro and São Paulo, which may have generated a selection bias. Due to a delayed processing in the Research Ethics Committee, maternity hospitals could not be assessed in Belém, thus not representing all five macro-regions in Brazil.

Conclusions

Although it is difficult to estimate the extent to which NBCAL compliance has contributed to increases in breastfeeding rates and indicators, NBCAL is considered one of the critical pillars for the breastfeeding policy as it supports effective breastfeeding actions. In this sense, monitoring actions that guarantee visibility of what is happening in territories, cities, and federated units, showing the extent of such issues, can be the first step of social awareness in order to enforce this important law.

Multi-NBCAL contributes to law monitoring by providing compliance indicators in stores and maternity hospitals. The application of this methodology can provide information to support actions from government sectors in charge of inspection and compliance with Law n. 11,265/2006.

Using this methodology in different scenarios will help support public policies to ensure NBCAL compliance, with broad dissemination of results to raise awareness of citizens to ensure infant food choices free from the influence of marketing actions from manufacturers of products and foods that compete with breastfeeding.

Finally, further studies seeking to expand the scope of the methodology, including monitoring of virtual environments, scientific events, labeling, and professional training aiming to standardize data collection practices and development of indicators, are essential for the creation of a continuous monitoring system that allows noncompliant cases reported by civil society and NBCAL enforcement by government agencies.

Acknowledgments

We would like to thank International Baby Food Action Network (IBFAN Brazil) for its contributions to the Multi-NBCAL study development, and Brazilian National Research Council (CNPq, process 408809/2017-5) for study funding.

References

  • 1
    United Nations Office of the High Commissioner for Human Rights. Breastfeeding a matter of human rights, say UN experts, urging action on formula milk. http://www.ohchr.org/EN/NewsEvents/Pages/DisplayNews.aspx?NewsID=20904&LangID=E%28 (acessado em 12/Set/2020).
    » http://www.ohchr.org/EN/NewsEvents/Pages/DisplayNews.aspx?NewsID=20904&LangID=E%28
  • 2
    Victora CG, Bahl R, Barros AJ, França GV, Horton S, Krasevec J, et al. Breastfeeding in the 21st century: epidemiology, mechanisms, and lifelong effect. Lancet 2016; 387:475-90.
  • 3
    Rollins NC, Bhandari N, Hajeebhoy N, Horton S, Lutter CK, Martines JC, et al. Why invest, and what it will take to improve breastfeeding practices? Lancet 2016; 387:491-504.
  • 4
    Piwoz EG, Huffman S. The impact of marketing of breast-milk substitutes on WHO-recommended breastfeeding practices. Food Nutr Bull 2015; 36:373-86.
  • 5
    World Health Assembly Resolution. Guidance on ending inappropriate promotion of foods for infants and young children. http://www.who.int/nutrition/topics/guidance-inappropriate-food-promotion-iyc/en/ (acessado em 10/Set/2020).
    » http://www.who.int/nutrition/topics/guidance-inappropriate-food-promotion-iyc/en/
  • 6
    World Health Organization. International code of marketing of breast-milk substitutes. Geneva: World Health Organization; 1981.
  • 7
    Ministério da Saúde. Resolução CNS nº 5, de 20 de dezembro de 1988. Normas para comercialização de alimentos para lactentes. Diário Oficial da União 1988; 23 dez.
  • 8
    Agência Nacional de Vigilância Sanitária. Portaria nº 2.051, de 8 de novembro de 2001. Diário Oficial da União 2001; 9 nov.
  • 9
    Agência Nacional de Vigilância Sanitária. Resolução RDC nº 221, de 5 de agosto de 2002. Diário Oficial da União 2002; 6 ago.
  • 10
    Agência Nacional de Vigilância Sanitária. Resolução RDC nº 222, de 5 de agosto de 2002. Diário Oficial da União 2002; 6 ago.
  • 11
    Presidência da República. Lei nº 11.265, de 3 de janeiro de 2006. Regulamenta a comercialização de alimentos para lactentes e crianças de primeira infância e também a de produtos de puericultura correlatos. Diário Oficial da União 2006; 4 jan.
  • 12
    Brasil. Decreto nº 8.552, de 3 de novembro de 2015. Regulamenta a Lei nº 11.265, de 3 de janeiro de 2006, que dispõe sobre a comercialização de alimentos para lactentes e crianças de primeira infância e de produtos de puericultura correlatos. Diário Oficial da da União 2015; 4 jan.
  • 13
    Brasil. Decreto nº 9.579, de 22 de novembro de 2018. Consolida atos normativos editados pelo Poder Executivo Federal que dispõem sobre a temática do lactente, da criança e do adolescente e do aprendiz, e sobre o Conselho Nacional dos Direitos da Criança e do Adolescente, o Fundo Nacional para a Criança e o Adolescente e os programas federais da criança e do adolescente, e dá outras providências. Diário Oficial da União 2018; 23 nov.
  • 14
    Grummer-Strawn LM, Zehner E, Stahlhofer M, Lutter C, Clark D, Sterken E, et al. New World Health Organization guidance helps protect breastfeeding as a human right. Matern Child Nutr 2017; 13:e12491.
  • 15
    Whitmee S, Haines A, Beyrer C, Boltz F, Capon AG, Dias AGS, et al. Safeguarding human health in the Anthropocene epoch: report of the Rockefeller Foundation-Lancet Commission on Planetary Health. Lancet 2015; 386:1973-2028.
  • 16
    Presidência da República. Emenda Constitucional nº 64, de 4 de fevereiro de 2010. Altera o art. 6º da Constituição Federal, para introduzir a alimentação como direito social. Diário Oficial da União 2010; 4 fev.
  • 17
    Departamento de Atenção Básica, Secretaria de Atenção à Saúde, Ministério da Saúde. Política Nacional de Alimentação e Nutrição. Brasília: Ministério da Saúde; 2012.
  • 18
    Ministério da Saúde. Portaria nº 1.130, de 5 de agosto de 2015. Institui a Política Nacional de Atenção Integral à Saúde da Criança (PNAISC) no âmbito do Sistema Único de Saúde (SUS). Diário Oficial da União 2015; 6 ago.
  • 19
    Prado I, Rinaldi AE. Compliance of infant formula promotion on websites of Brazilian manufacturers and drugstores. Rev Saúde Pública 2020; 54:12.
  • 20
    Pérez-Escamilla R, Curry L, Minhas D, Taylor L, Bradley E. Scaling up of breastfeeding promotion programs in low- and middle-income countries: the "breastfeeding gear" model. Adv Nutr 2012; 3:790-800.
  • 21
    Rede Internacional em Defesa do Direito de Amamentar Brasil. Manual do curso de capacitação em monitoramento. http://www.ibfan.org.br/documentos/ibfan/doc-678.pdf (acessado em 21/Abr/2021).
    » http://www.ibfan.org.br/documentos/ibfan/doc-678.pdf
  • 22
    Silva KB, Oliveira MIC, Boccolini CS, Sally EOF. Illegal commercial promotion of products competing with breastfeeding. Rev Saúde Pública 2020; 54:10.
  • 23
    Silva LIMM, Thé PMP, Medeiros JO, Giacomini SGMO, Rodrigues RCB, Peixoto MMLV. Conhecimento de farmacêuticos sobre aleitamento materno: um estudo nas farmácias comerciais em Fortaleza - CE. Rev Bras Promoç Saúde 2012; 25:482-91.
  • 24
    World Health Organization, United Nations Children's Fund. NetCode toolkit. Monitoring the marketing of breast-milk substitutes: protocol for ongoing monitoring systems. Geneva: World Health Organization; 2017.
  • 25
    Abbot A. Paper, paper, everywhere... Nature 2005; 437:310.
  • 26
    Conselho Nacional de Saúde. Resolução CNS nº 466, de 12 de dezembro de 2012. Aprova diretrizes e normas regulamentadoras de pesquisas envolvendo seres humanos. Revoga a Resolução. 196/96, Resolução nº 303/00 e a Resolução nº 404/08. Diário Oficial da União 2013; 13 jun.
  • 27
    Lopes AG, Pereira AC, Fonseca EP, Mialhe FL. Irregularidades sanitárias na promoção comercial em rótulos de produtos para lactentes e os riscos para a saúde. Saúde Debate 2017; 41:539-52.
  • 28
    Paula LO, Chagas LR, Ramos CV. Monitoramento da norma brasileira de comercialização de alimentos infantis. Nutrire Rev Soc Bras Alim Nutr 2010; 35:43-55.
  • 29
    Gurgel TEP. Monitoramento da promoção comercial de alimentos para lactentes e crianças de primeira infância e de produtos de puericultura em estabelecimentos comerciais de Mossoró. Nutrivisa - Revista de Nutrição e Vigilância em Saúde 2016; 1:21-5.
  • 30
    Rea MF, Toma TS. Proteção do leite materno e ética. Rev Saúde Pública 2000; 34:388-95.
  • 31
    Aguayo VM, Ross JS, Kanon S, Ouedraogo AN. Monitoring compliance with the International Code of Marketing of Breastmilk Substitutes in West Africa: multisite cross sectional survey in Togo and Burkina Faso. BMJ 2003; 326:127.
  • 32
    Salasibew M, Kiani A, Faragher B, Garner P. Awareness and reported violations of the WHO International Code at Pakistan's national breastfeeding legislation: a descriptive cross-sectional survey. Int Breastfeed J 2008; 3:24.
  • 33
    Liu A, Dai Y, Xie X, Chen L. Implementation of International Code of Marketing Breast-Milk Substitutes in China. Breastfeed Med 2014; 9:467-72.
  • 34
    Instituto Brasileiro de Defesa do Consumidor. Amamentação desvalorizada. Revista do IDEC 2015; (195):22-24. http://www.idec.org.br/uploads/revistas_materias/pdfs/195-amamentacao1.pdf (acessado em 13/Set/2020).
    » http://www.idec.org.br/uploads/revistas_materias/pdfs/195-amamentacao1.pdf
  • 35
    Oliveira MIC, Boccolini CS, Sally EOF. Breast milk substitutes marketing violations and associated factors in Rio de Janeiro, Brazil. J Hum Lact 2021; 37:158-68.
  • 36
    Boccolini C, Boccolini P, Mezzavilla R. National System to Monitor the International Code of Breast Milk Substitutes in Brazil: SisNBCAL. Curr Dev Nutr 2020; 4 Suppl 2:705.
  • 37
    Ministério da Agricultura, Pecuária e Abastecimento. Instrução Normativa nº 28, de 12 de junho de 2007. Diário Oficial da União 2007; 14 jun.

Publication Dates

  • Publication in this collection
    20 Apr 2022
  • Date of issue
    2021

History

  • Received
    16 Sept 2020
  • Reviewed
    19 Feb 2021
  • Accepted
    16 Mar 2021
Escola Nacional de Saúde Pública Sergio Arouca, Fundação Oswaldo Cruz Rio de Janeiro - RJ - Brazil
E-mail: cadernos@ensp.fiocruz.br