The 12 city HIV Surveillance Survey among MSM in Brazil 2016 using respondent-driven sampling: a description of methods and RDS diagnostics

Inquérito sobre o HIV entre HSH no Brasil em 2016 usando respondent-driven sampling (RDS): descrição dos métodos e dos diagnósticos do RDS

Carl Kendall Ligia Kerr Rosa Salani Mota Mark Drew Crosland Guimarães Andrea Fachel Leal Edgar Merchan-Hamann Inês Dourado Maria Amélia Veras Ana Maria de Brito Alexandre Kerr Pontes Ana Rita Coimbra Motta-Castro Raimunda Hermelinda Maia Macena Daniela Knauth Luana Nepomuceno Gondim Costa Lima Lisangela Cristina Oliveira Socorro Cavalcante Ana Cláudia Camillo Ximena Pamela Diaz Bermudez Regina Célia Moreira Adele Schwartz Benzaken Gerson Pereira Ana Roberta Pati Pascom Cristina Pimenta Lisa Grazina Johnston About the authors

ABSTRACT:

Introduction:

This paper details the methods used in the second national Biological and Behavioral Surveillance Survey (BBSS) of HIV, syphilis, and hepatitis B and C among men who have sex with men in Brazil.

Methods:

Respondent-driven sampling (RDS) was used in 12 cities in 2016. The targeted sample size was initiated with five to six seeds in each city. HIV, syphilis, and Hepatitis B and C rapid tests were offered to participants. RDS Analyst with Gile’s successive sampling (SS) estimator was used to adjust results as recommended and a weight for each individual was generated for further analysis. Data for the 12 cities were merged and analyzed using Stata 14.0 complex survey data tools with each city treated as its own stratum.

Results:

Duration of data collection varied from 5.9 to 17.6 weeks. 4,176 men were recruited in the 12 cities. Two sites failed to achieve targeted sample size due to a six-month delay in local IRB approval. No city failed to reach convergence in our major outcome variable (HIV).

Conclusion:

The comprehensive BBSS was completed as planned and on budget. The description of methods here is more detailed than usual, due to new diagnostic tools and requirements of the new STROBE-RDS guidelines.

Keywords:
HIV Antibodies,Sexual and Gender Minorities,Brazil; Statistics,Methods

RESUMO:

Introdução:

Este artigo detalha os métodos utilizados na segunda Pesquisa Nacional de Vigilância Biológica e Comportamental (BBSS) do HIV, sífilis e hepatite B e C entre os homens que fazem sexo com homens no Brasil.

Métodos:

O método Respondent-driven Sampling (RDS) foi utilizado em 12 cidades em 2016. A amostra foi iniciada com cinco a seis sementes em cada cidade. Testes rápidos para o HIV, sífilis e Hepatite B e C foram oferecidos aos participantes. O software RDS Analyst com o estimador de amostragem sucessiva (SS) de Gile foi utilizado para ajustar os resultados como recomendado, gerando um peso para cada indivíduo para análises. Osdados das 12cidades foram unidos em um único banco e analisados usando as ferramentas de dados complexos do Stata 14.0, com cada cidade sendo tratada como seu próprio estrato.

Resultados:

A duração da coleta de dados variou de 5,9 a 17,6 semanas e 4.176 homens foram recrutados nas 12 cidades. Dois sites não alcançaram o tamanho da amostra alvo devido a uma demora de seis meses na aprovação local do Comitê de Ética. Todas as cidades atingiram a convergência na principal variável estudada (HIV).

Conclusão:

O BBSS foi representativo e concluído conforme planejado e dentro do orçamento. A descrição dos métodos aqui é mais detalhada do que o habitual, devido às novas ferramentas e requisitos de diagnóstico das novas diretrizes do STROBE-RDS.

Palavras-chave:
Anticorpos Anti-HIV,Minorias Sexuais e de Gênero,Brasil; Estatística; Métodos

INTRODUCTION

Rising HIV seroprevalence rates for men who have sex with men (MSM) continue to be reported globally11. Baral S, Sifakis F, Cleghorn F, Beyrer C. Elevated risk for HIV infection among men who have sex with men in low- and middle-income countries 2000-2006: a systematic review. PLoS Med 2007; 4(12): e339. https://doi.org/10.1371/journal.pmed.0040339
https://doi.org/10.1371/journal.pmed.004...
,22. Beyrer C, Sullivan PS, Sanchez J, Dowdy D, Altman D, Trapence G, et al. A call to action for comprehensive HIV services for men who have sex with men. Lancet 2012; 380(9839): 424-38. https://doi.org/10.1016/S0140-6736(12)61022-8
https://doi.org/10.1016/S0140-6736(12)61...
,33. Beyrer C, Baral SD, Walker D, Wirtz AL, Johns B, Sifakis F. The expanding epidemics of HIV type 1 among men who have sex with men in low- and middle-income countries: diversity and consistency. Epidemiol Rev 2010; 32(1): 137-151. https://doi.org/10.1093/epirev/mxq011
https://doi.org/10.1093/epirev/mxq011...
,44. Beyrer C. Hidden yet happening: the epidemics of sexually transmitted infections and HIV among men who have sex with men in developing countries. Sex Transm Infect 2008; 84(6): 410-2. https://doi.org/10.1136/sti.2008.033290
https://doi.org/10.1136/sti.2008.033290...
,55. Beyrer C, Sullivan P, Sanchez J, Baral SD, Collins C, Wirtz AL, et al. The increase in global HIV epidemics in MSM. Aids 2013; 27(17): 2665-78. https://doi.org/10.1097/01.aids.0000432449.30239.fe
https://doi.org/10.1097/01.aids.00004324...
,66. Beyrer C, Baral SD, van Griensven F, Goodreau SM, Chariyalertsak S, Wirtz AL, et al. Global epidemiology of HIV infection in men who have sex with men. Lancet 2012; 380(9839): 367-77. https://doi.org/10.1016/S0140-6736(12)60821-6
https://doi.org/10.1016/S0140-6736(12)60...
,77. Baral S, Kizub D, Masenior NF, Peryskina A, Stachowiak J, Stibich M, et al. Male sex workers in Moscow, Russia: a pilot study of demographics, substance use patterns, and prevalence of HIV-1 and sexually transmitted infections. AIDS Care 2010; 22(1): 112-8. https://doi.org/10.1080/09540120903012551
https://doi.org/10.1080/0954012090301255...
,88. Beyrer C, Baral SD, Collins C, Richardson ET, Sullivan PS, Sanchez J, et al. The global response to HIV in men who have sex with men. Lancet 2016; 388(10040): 198-206. https://doi.org/10.1016/S0140-6736(16)30781-4
https://doi.org/10.1016/S0140-6736(16)30...
. Evidence can be found in Brazil as well. The incidence rate of AIDS cases among men 15 - 19 and 20 - 24 years of age increased from 3.7 to 6.9 and from 18.1 to 33.1 (/100,000) from 2008 to 2015 in the country99. Brasil. Ministério da Saúde. Departamento de Vigilância, Prevenção e Controle das IST, do HIV/Aids e das Hepatites Virais. Boletim Epidemiologico de DST/AIDS. Brasília: Ministério da Saúde; 2016.. While the estimated prevalence of HIV among the population of 15 years of age or older is 0.37%1010. Szwarcwald C, Pascom A, Souza Júnior PR. Estimation of the HIV incidence and of the number of people living with HIV/AIDS in Brazil, 2012. J AIDS Clin Res 2015; 6(3): 430. https://doi.org/10.4172/2155-6113.1000430
https://doi.org/10.4172/2155-6113.100043...
, recent data suggest HIV rates of 4.9, 5.9, and 12.1% among users of injectable drugs, sex workers, and MSM, respectively1111. Damacena GN, Szwarcwald CL, de Souza Júnior PR, Dourado I. Risk factors associated with HIV prevalence among female sex workers in 10 Brazilian cities. J Acquir Immune Defic Syndr 2011; 57(Suppl. 3): S144-52. https://doi.org/10.1097/QAI.0b013e31821e9bf6
https://doi.org/10.1097/QAI.0b013e31821e...
,1212. Kerr LRFS, Mota RS, Kendall C, Pinho A de A, Mello MB, Guimarães MD, et al. HIV among MSM in a large middle-income country. Aids 2013; 27(3): 427-35. https://doi.org/10.1097/QAD.0b013e32835ad504
https://doi.org/10.1097/QAD.0b013e32835a...
,1313. Bastos F. Taxas de infecção de HIV e sífilis e inventário de conhecimento, atitudes e práticas de risco relacionadas às infecções sexualmente transmissíveis entre usuários de drogas em 10 municípios brasileiros. In: Brasil. Ministério da Saúde. Departamento de Vigilância, Prevenção e Controle das IST, do HIV/Aids e das Hepatites Virais, ed. Brasília: Ministério da Saúde ; 2009.. These prevalence rates are 12, 15, and almost 30 times larger than in the general population. Groups at major risk of infection from HIV in Brazil play an important role in the spread of the epidemic1414. Boily MC, Lowndes C, Alary M. The impact of HIV epidemic phases on the effectiveness of core group interventions: insights from mathematical models. Sex Transm Infect 2002; 78(Suppl. 1): i78-90., and there is international evidence that interventions focused on high-risk groups have an impact on the incidence of HIV in the general population1515. Mishra RM, Dube M, Sahu D, Saggurti N, Pandey A. Changing epidemiology of HIV in Mumbai: an application of the Asian epidemic model. Glob J Health Sci 2012; 4(5): 100-12. https://dx.doi.org/10.5539%2Fgjhs.v4n5p100
https://dx.doi.org/10.5539%2Fgjhs.v4n5p1...
.

Current information about infection, behaviors and programs are essential for understanding the dynamics of the HIV epidemic and improving programs. In response, the Department of Sexually Transmitted Infections and Viral Hepatitis (DIAHV) supported a second national round of Biological and Behavioral Surveillance Survey (BBSS) among MSM in 2016 using Respondent-driven Sampling (RDS). RDS is a chain-link sampling method that permits participants to recruit their acquaintances and uses a mathematical model to adjust for this method of recruitment1616. Heckathorn DD. Respondent-driven sampling: A new approach to the study of hidden populations. Social Problems 1997; 44(2): 174-99. https://doi.org/10.2307/3096941
https://doi.org/10.2307/3096941...
,1717. Gile KJ. Improved inference for respondent-driven sampling data with application to HIV prevalence estimation. J Am Statist Assoc 2011; 106(493): 135-46. https://doi.org/10.1198/jasa.2011.ap09475
https://doi.org/10.1198/jasa.2011.ap0947...
,1818. Salganik MJ. Variance estimation, design effects, and sample size calculations for respondent-driven sampling. J Urban Health 2006; 83(6 Suppl.): i98-112. https://doi.org/10.1007/s11524-006-9106-x
https://doi.org/10.1007/s11524-006-9106-...
. RDS is used in hundreds of surveys1919. Johnston LG, Hakim AJ, Dittrich S, Burnett J, Kim E, White RG. A Systematic Review of Published Respondent-Driven Sampling Surveys Collecting Behavioral and Biologic Data. AIDS Behav 2016; 20(8): 1754-76. https://doi.org/10.1007/s10461-016-1346-5
https://doi.org/10.1007/s10461-016-1346-...
,2020. Malekinejad M, Johnston LG, Kendall C, Kerr LRFS, Rifkin MR, Rutherford GW. Using respondent-driven sampling methodology for HIV biological and behavioral surveillance in international settings: a systematic review. AIDS Behav 2008; 12(4 Suppl.): S105-30. https://doi.org/10.1007/s10461-008-9421-1
https://doi.org/10.1007/s10461-008-9421-...
,2121. Montealegre JR, Johnston LG, Murrill C, Monterroso E. Respondent driven sampling for HIV biological and behavioral surveillance in Latin America and the Caribbean. AIDS Behav 2013; 17(7): 2313-40. https://doi.org/10.1007/s10461-013-0466-4
https://doi.org/10.1007/s10461-013-0466-...
around the world and was recently identified as a major contribution to social, behavioral and economic research by the U.S. National Academies of Sciences2222. National Academies of Sciences, Engineering and Medicine. The Value of Social, Behavioral, and Economic Sciences to National Priorities: A Report for the National Science Foundation. Washington, D.C.: The National Academies Press; 2017..

RDS data collection is initiated by selecting a small number of initial participants (seeds) from the population of interest. Seeds are extensively briefed about the purpose and operation of the study, and the information to be provided recruiters. They initiate recruitment with a fixed number of coupons (often three) to provide to the participants recruited. When these recruits complete the survey, they are provided with coupons and both themselves and their recruiter receive incentives1616. Heckathorn DD. Respondent-driven sampling: A new approach to the study of hidden populations. Social Problems 1997; 44(2): 174-99. https://doi.org/10.2307/3096941
https://doi.org/10.2307/3096941...
,2323. Heckathorn DD. Extensions of Respondent-Driven Sampling: A new approach to the study of injection drug users aged 18-25. In: Semaan SBRSHJJ, ed. AIDS & Behav.2002:6(1):55-67.https://doi.org/10.1023/A:1014528612685
https://doi.org/10.1023/A:1014528612685...
,2424. Heckathorn DD. Extensions of Respondent Driven Sampling: Analyzing continuous variables and controlling for differential recruitment. Sociol Methodol 2007; 37(1): 151-207. https://doi.org/10.1111/j.1467-9531.2007.00188.x
https://doi.org/10.1111/j.1467-9531.2007...
. The fact that participants are recruited by their peers contributes to the success of RDS. Additionally, bias associated with interviewers selecting participants is avoided1616. Heckathorn DD. Respondent-driven sampling: A new approach to the study of hidden populations. Social Problems 1997; 44(2): 174-99. https://doi.org/10.2307/3096941
https://doi.org/10.2307/3096941...
,2323. Heckathorn DD. Extensions of Respondent-Driven Sampling: A new approach to the study of injection drug users aged 18-25. In: Semaan SBRSHJJ, ed. AIDS & Behav.2002:6(1):55-67.https://doi.org/10.1023/A:1014528612685
https://doi.org/10.1023/A:1014528612685...
,2424. Heckathorn DD. Extensions of Respondent Driven Sampling: Analyzing continuous variables and controlling for differential recruitment. Sociol Methodol 2007; 37(1): 151-207. https://doi.org/10.1111/j.1467-9531.2007.00188.x
https://doi.org/10.1111/j.1467-9531.2007...
. RDS requires researchers to understand some of the underlying networks of the population being recruited, to assess if seed selection unduly influenced the final sample, and to identify recruitment bottlenecks and missed subpopulations. Gile et al. do an excellent job describing these issues and recommending diagnostics2525. Gile KJ, Johnston LG, Salganik MJ. Diagnostics for respondent‐driven sampling. J Royal Statistical Soc 2015; 178(1): 241-69. https://doi.org/10.1111/rssa.12059
https://doi.org/10.1111/rssa.12059...
. Our paper describes the methods used to conduct the BBSS for HIV among MSM in 2016 and applies several diagnostic tools. This paper is STROBE-RDS compliant2626. White RG, Hakim AJ, Salganik MJ, Spiller MW, Johnston LG, Kerr L, et al. Strengthening the Reporting of Observational Studies in Epidemiology for Respondent-Driven Sampling Studies:‘STROBE-RDS’Statement. J Clin Epidemiol 2015; 68(12): 1463-71. https://doi.org/10.1016/j.jclinepi.2015.04.002
https://doi.org/10.1016/j.jclinepi.2015....
.

MATERIALS AND METHODS

The study took place in 12 Brazilian municipalities in the five regions of Brazil from June to December 2016: Manaus, Belém (North Region); Fortaleza, Recife, Salvador (Northeast Region); Brasília, Campo Grande (Central-West Region); Belo Horizonte, Rio de Janeiro, and São Paulo (Southeast Region); and Curitiba, Porto Alegre (South Region) (Figure 1).

Figure 1.
States and cities where respondent-driven sampling (RDS) among men who have sex with men (MSM) was conducted in 2016, Brazil.

All of the surveys were conducted in public health facilities except in Belo Horizonte and Salvador where private offices were used. Study working hours were adjusted to include evenings or weekends as required. The study sites consisted of a receptionist and waiting area, interview areas, and a private room for testing and counseling.

STUDY POPULATION

Our sample included men, 18 years of age or older, reporting oral or anal sex with another man or transgender woman (travesti) in the last 12 months; and living, studying, or working in the studied city. Respondents were asked about their relationship to the person providing the coupon. The questionnaire and biological testing were consented separately, and participants could opt out of testing. Individuals under the influence of drugs or alcohol, or who identified as a transgender woman, were excluded.

SAMPLE SIZE

RDS requires a design effect (DE) multiplier. DE is the ratio of the actual variance to the variance expected with simple random sampling. Wejnert et al. recommend a DE of 42727. Wejnert C, Pham H, Krishna N, Le B, DiNenno E. Estimating design effect and calculating sample size for respondent-driven sampling studies of injection drug users in the United States. AIDS Behav 2012; 16(4): 797-806. https://doi.org/10.1007/s10461-012-0147-8
https://doi.org/10.1007/s10461-012-0147-...
. Using this recommendation and HIV prevalence by city from previous studies we calculated sample sizes for each city: Belém (393), Manaus (204), Salvador (193) Fortaleza (131), Recife (116), Brasília (299), Campo Grande (143), Belo Horizonte (267), Rio de Janeiro (450), SãoPaulo (474), Porto Alegre (393), and Curitiba (210). Donors limited sample size to 350 per city, which complies with a DE of 2. Most RDS studies reach convergence at wave 5 or 6, much earlier than the sample size of 350 participants.

FORMATIVE RESEARCH

Before initiation of the BBSS we conducted formative research (FR) as recommended2828. Centers for Disease Control and Prevention. Round 4: Formative Research Manual. City: Centers for Disease Control and Prevention; 2013. using both individual interviews and Focus Group Discussions (FGD)2929. Johnston LG, Whitehead S, Simic-Lawson M, Kendall C. Formative research to optimize respondent-driven sampling surveys among hard-to-reach populations in HIV behavioral and biological surveillance: lessons learned from four case studies. AIDS Care 2010; 22(6): 784-92. https://doi.org/10.1080/09540120903373557
https://doi.org/10.1080/0954012090337355...
. The individual and FGD interview guides covered sexual identities, social and geographic organization of MSM in each city and perceived community acceptance, including violence, homophobia and stigma. Questions related to study logistics such as the site of the study, hours of operation, social network size questions, willingness to participate and to test in the study and potential to serve as seeds were explored until saturation of responses for that topic was achieved.

Data collection for the formative research was conducted between December 2015 and March, 2016. FR was conducted by a team consisting of the national and site coordinators, and a qualitative research expert to assure uniformity. Interviews for both individual and FGDs were recorded but not transcribed. Initial interview notes were expanded by reviewing the recordings.

A convenience sample of participants was recruited by Site Coordinators from local non-governmental organizations (NGOs) that work with MSM and from voluntary HIV testing and counseling centers. We conducted 58 one-on-one interviews and 17 FGDs. Atotal of 184 MSM formally participated in the FR. Overall enthusiasm for the FR and BBSS was high. The Project logo, Me Convida que Eu Vou (Invite me so that I will go) was also endorsed in the FR. Site coordinators were instructed to continue documenting community response and other issues during the study.

BIOLOGICAL AND BEHAVIORAL SURVEILLANCE SURVEY DATA COLLECTION

Five to six seeds were identified for each site. After completing the survey (interview and testing), each seed received three non-reproducible coupons to distribute. Each eligible participant completing the survey, including the seeds, received R$ 25 » US$10. Each participant was also eligible to receive an additional R$ 25 for each recruitee that completed the survey. When approximately 75% of the sample had been recruited, the instructions to recruiters emphasized that when sample size was reached coupons would not be redeemable. In some sites one or two seeds were added when the directors were concerned about the speed of recruitment (Table 1). Seeds and recruiters were trained to provide a detailed explanation of the study to their recruits. The need to recruit participants that met eligibility requirements from their personal networks was emphasized. If the participant allowed, telephone messaging was used to remind respondents of appointments and to contact recruiters if their recruitees did not show up.

Table 1.
2016 RDS survey duration, seeds and longest wave, and eligibility by site (sample size n = 4,176).

Arriving at the site, eligibility of the recruit was reviewed. Reasons for non-eligibility were explained and HIV educational materials and condoms given. Coupons and IDs were managed with an on-line coupon generator. The recruit was then read a description of the study, and risks and benefits of participation. After consenting, an interviewer using Computer Assisted Self Interview (CAPI) administered the social network size questionnaire. The social network question cascade (4 questions) is summarized in the following two questions: “How many men do you know and who also know you, who have sex with other men (oral or anal) in the last 12 months, live, study and/or work in [municipality], are 18 years old or older, you encountered or spoke with in the last two months? Of these how many would you invite to participate in this study?” Following completion of the questionnaire, the respondent was consented separately for each test and sent for testing.

Following counseling, two tubes of venous blood were drawn. For HIV, blood was first tested with the rapid test RT1 Anti-HIV (Alere/Bioeasy). If positive, the blood was tested with RT2 HIV (Abon). Two positive results fulfilled Ministry criteria for reporting HIV+ serostatus. Respondents were counseled and immediately referred if positive. For syphilis, blood was tested with RT Anti-Syphilis (Biomanguinhos). For syphilis and all positive tests, results were explained to respondents and they were referred for counseling and treatment. Remaining rapid tests included RT Hepatitis B - HBsAg (Vikia) RT Hepatitis C - Anti-HCV (Alere). After the rapid tests, the tubes were centrifuged and stored at -20ºC. Samples were sent to the national reference laboratory - Instituto Adolfo Lutz, São Paulo - for confirmatory testing. Transport complied with requirements of the Agência Nacional de Vigilância Sanitária (ANVISA).

TRAINING

A series of four workshops about RDS was held: introduction to RDS, organization of the study, data analysis, and write-up. Organization of the study was a 3-day workshop for site coordinators who then trained their local teams. Following the workshop, a pilot implementation was then conducted in each site. After this exercise, the teams met via videoconference to identify issues. Videoconferences were held repeatedly with sites until concerns were successfully addressed. When necessary, sites were visited in person. All study procedures including scheduling interviews, tablet use, pre-test counseling, referral, post-test counseling, coupon generation and incentive management were documented in standard operating procedure (SOP) manuals, provided to and approved by the DIAHV before initiation of fieldwork.

DATA COLLECTION INSTRUMENT

The questionnaire was adapted from the 2009 BBSS questionnaire used in Brazil and contained items to report international AIDS indicators for Brazil and other questions related to the national program. The questionnaire was modified to:

  • improve comparison with items in the 2009 survey;

  • collect information related to policy and program changes (e.g., PrEP and PEP) and to environments and HIV-linked behaviors in the MSM population.

The questionnaire was organized into blocks:

  • identification and eligibility;

  • social network;

  • socioeconomic and demographic information;

  • sexually transmitted infections, HIV/AIDS and hepatitis B and C;

  • access to health services in general and specifically for HIV prevention and treatment;

  • history of HIV, syphilis and hepatitis testing;

  • information about HIV/AIDS, PEP, PrEP and treatment for HIV/AIDS;

  • gay and homosexual visibility;

  • violence, stigma and discrimination;

  • sexual behavior including condom use;

  • drug and alcohol use;

  • social inclusion and participation;

  • mental health.

DATA MANAGEMENT

Each site was equipped with high-speed and reliable internet provided by the project. Information entered from computer or tablet was encrypted, uploaded and stored on the project website. The website was maintained by an Information Technology professional and copies of the database archived on the project server and on the PI’s own desktop computer stored in a locked office in Fortaleza. The reference laboratory stored their data locally and on mirrored backups offsite.

ETHICAL CONSIDERATIONS

The overall study was approved by the Committee on Research Ethics of the Federal University of Ceará (UFC), credentialed by the National Commission on Research. Asmentioned above, all participants signed a consent form to participate in the interview and separately consented for each test that was offered. There appeared to be little reluctance to participate in both parts of the study, confirming results obtained from formative research.

ANALYSIS

Analysis proceeded as follows. Gile’s successive sampling (SS) estimator3030. Gile KJ, Handcock MS. Respondent-driven sampling: an assessment of current methodology. Sociol Methodol 2010; 40(1): 285-327. https://dx.doi.org/10.1111%2Fj.1467-9531.2010.01223.x
https://dx.doi.org/10.1111%2Fj.1467-9531...
was used to produce weighted estimates using RDS Analyst version 1.7-16. The estimator assumes a finite population and requires a population size estimate for each sample. To calculate this, we used the proportion of men who self-reported as having had at least one same-sex relationship in the National Survey of Knowledge, Attitudes and Practices in the Brazilian population (18 - 64) conducted in 20133131. Brasil. Ministério da Saúde. Secretaria de Vigilância em Saúde. Departamento de Vigilância, Prevenção e Controle das IST, do HIV/Aids e das Hepatites Virais. Pesquisa de Conhecimentos, Atitudes e Práticas na População Brasileira. Brasília: Ministério da Saúde / Secretaria de Vigilância em Saúde / Departamento de Vigilância, Prevenção e Controle das IST, do HIV/Aids e das Hepatites Virais; 2016.. This survey was powered to provide regional estimates. We applied this regional estimate to the total male population 18 - 64 in each city of that region as provided by the Brazilian Institute of Geography and Statistics3232. Instituto Brasileiro de Geografia e Estatística. Projeção da população do Brasil e das Unidades da Federação [Internet]. IBGE; 2017 [accessed on Jul. 26, 2017]. Available at: Available at: http://www.ibge.gov.br/apps/populacao/projecao/
http://www.ibge.gov.br/apps/populacao/pr...
. To test the accuracy of this procedure, we compared the percentage of MSM population calculated with this method in São Paulo city with the percentage obtained from a separate municipal survey conducted in São Paulo that used the same sampling methodology as the national survey, powered for city estimates. Both results (3.9 and 3.6%) were not significantly different3333. Battaggia GE. Pesquisa de conhecimentos, atitudes e práticas na população residente no município de São Paulo. Brasília: Ministério da Saúde ; 2015..

To provide a national estimate, we merged the 12 cities to create a single dataset. Weused the Complex Analysis Survey tools in StataTR 14.0 with each city treated as its own stratum to weight the final results.

RESULTS

A total of 4,176 respondents was recruited (Table 1). Some summary sociodemographic details of the final sample are presented in Table 2.

Table 2.
Socioeconomic and demographic characteristics of participants in 12 cities.

DIAGNOSIS AND STUDY ISSUES

Diagnostic criteria for RDS includes simple inspection of recruitment data as well as tools provided by RDS Analyst. As proposed in Gile et al.2525. Gile KJ, Johnston LG, Salganik MJ. Diagnostics for respondent‐driven sampling. J Royal Statistical Soc 2015; 178(1): 241-69. https://doi.org/10.1111/rssa.12059
https://doi.org/10.1111/rssa.12059...
we reviewed reported personal network sizes for consistency and reasonableness. Additional tools include convergence and bottleneck plots. Convergence - a stable estimate of the true population proportion- should be achieved rapidly for major variables. Bottleneck plots visually demonstrate convergence by recruitment chain: widely different indicator estimates by chain would signal important differences by seed and a failure to find a true population proportion. In a non-technical sense, it demonstrates if a recruitment chain is ‘stuck’ in a sub-population or geographical region. Review of convergence and bottlenecks is through visual inspection and interpretation (Table 1 and Figure2). The convergence plot shows when in recruitment the outcome estimate is determinedThe bottleneck plot shows the estimate in terms of each recruitment chain. Visual inspection can show if one or several chains demonstrate different outcome values. Given 5 seeds, an ideal bottleneck plot would stabilize after a few waves producing a single horizontal line.

Figure 2.
Bottleneck and convergence for HIV by city in Brazil (respondent-driven sampling - RDS - among men who have sex with men - MSM - 2016).

Data were collected over a 4-month period, much faster than recruitment in 2009 and in many other surveys. Data collection times varied between 5.9 and 17.6 weeks. Thesurveys in Rio de Janeiro and Porto Alegre failed to achieve their sample size due to the late start of data collection in reason of a six-month delay in local IRB approval. Median network size reported (4 - 10) was both relatively small and uniform across sites, which is a positive sign. The length of the longest wave varied from 8 in Belém to 21 in Curitiba (Table 1). Convergence was achieved on major variables for all sites. We illustrate this with HIV serostatus (Figure 2) which presents results for selected cities in the 5 regions. While there was reason to assume from the formative research that there might be bottlenecks that interfered with completion of the survey due to age, identity, behavior and class, it did not appear to be the case. One site, Porto Alegre, that might have required a larger sample size, achieved convergence, and the recruitment chains do not demonstrate bottlenecks (Figure 2).

Two sites, Rio de Janeiro and São Paulo (not shown), that required sample sizes larger than 350, did show a delayed convergence. While convergence for HIV prevalence in Rio de Janeiro ultimately appeared stable, early results along with the bottleneck chart demonstrate wide differences and separation in the recruitment chains in the early weeks of the study.

LIMITATIONS

The study reported here suffers from some limitations. RDS, required for comparison to previous national estimates, remains controversial3434. Khatib A, Haji S, Khamis M, Said C, Khalid F, Dahoma M, et al. Reproducibility of Respondent-Driven Sampling (RDS) in Repeat Surveys of Men Who have Sex with Men, Unguja, Zanzibar. AIDS Behav 2016; 21(7); 2180-7. https://doi.org/10.1007/s10461-016-1632-2
https://doi.org/10.1007/s10461-016-1632-...
. Limitations inherent in RDS are well reviewed elsewhere2525. Gile KJ, Johnston LG, Salganik MJ. Diagnostics for respondent‐driven sampling. J Royal Statistical Soc 2015; 178(1): 241-69. https://doi.org/10.1111/rssa.12059
https://doi.org/10.1111/rssa.12059...
,3030. Gile KJ, Handcock MS. Respondent-driven sampling: an assessment of current methodology. Sociol Methodol 2010; 40(1): 285-327. https://dx.doi.org/10.1111%2Fj.1467-9531.2010.01223.x
https://dx.doi.org/10.1111%2Fj.1467-9531...
. Specific cities and sample size were determined by the donor. Thismay have affected our results in the cities where our calculations required a larger sample size. However, in São Paulo our results are not statistically different from a previous study conducted in the city using Time Location Sampling3535. Veras MASM, Calazans GJ, de Almeida Ribeiro MCS, Oliveira CAF, Giovanetti MR, Facchini R, et al. High HIV prevalence among men who have sex with men in a time-location sampling survey, São Paulo, Brazil. AIDS Behav 2015; 19(9): 1589-98. https://doi.org/10.1007/s10461-014-0944-3
https://doi.org/10.1007/s10461-014-0944-...
. The 2016 study took place at a time of both great political change in Brazil and with an HIV program focused on treatment. Both changes have led to a disengagement with MSM - and MSM serving organizations3636. Parker R. O Fim da AIDS? Rio de Janeiro: Associação Brasileira Interdisciplinar de Aids; 2015. with potential effects on participation and the survey.

DISCUSSION

Overall, we argue that the study was successful: large sample sizes were achieved in a relatively short period of time, with no evidence of difficulties with convergence and little evidence of bottlenecks in recruitment. One important reason is overall receptivity among MSM in our sample. This happened in spite of the changes in national support reported in limitations. Many NGOs that participated in the 2009 study had closed by 2016. In half of the cities, finding MSM organization partners for the study was initially difficult. Perhaps these closures served to motivate participants, but we have no direct evidence of that. Inthe FR, enthusiasm for the study appears to be associated with an enhanced willingness to test for HIV and other infections in studies directed to MSM. Many studies report health disparities and barriers to health care for MSM3737. Alvy LM, McKirnan D, Du Bois SN, Jones K, Ritchie N, Fingerhut D. Health care disparities and behavioral health among men who have sex with men. J Gay Lesbian Soc Serv 2011; 23(4): 507-22. https://doi.org/10.1080/10538720.2011.611114
https://doi.org/10.1080/10538720.2011.61...
,3838. Buchmueller T, Carpenter CS. Disparities in health insurance coverage, access, and outcomes for individuals in same-sex versus different-sex relationships, 2000-2007. Am J Public Health 2010; 100(3): 489-95. https://doi.org/10.2105/AJPH.2009.160804
https://doi.org/10.2105/AJPH.2009.160804...
,3939. McKirnan DJ, Du Bois SN, Alvy LM, Jones K. Health care access and health behaviors among men who have sex with men: the cost of health disparities. Health Educ Behav 2013; 40(1): 32-41. https://doi.org/10.1177/1090198111436340
https://doi.org/10.1177/1090198111436340...
. Willingness to participate in our study may also be associated with a reaction to these barriers and an understanding of the importance of these outcomes to direct resources to MSM. In our study 90.2% of participants tested. Theformative research exercise itself may also have encouraged participation. Finally, the rapid completion of the study and its quality may be associated with the experience theresearch team had with the method from the earlier RDS study.

One issue influencing the conduct of the study was the delay in the IRB approval process. In our study, using an almost entirely standardized instrument and testing procedures preapproved by national authorities, with review and approval by a national IRB, several local IRBs, rather than defer, demanded separate reviews, extending the project start by six months in those locations. As the pace of scientific innovation (and demands on local expertise) increases, the role of local IRBs in national studies needs to be reviewed. Routinized surveillance is especially vulnerable to localized responses; much can be hidden by simply not collecting the data.

Finally, there is some concern in the research community for reproducibility of RDS3434. Khatib A, Haji S, Khamis M, Said C, Khalid F, Dahoma M, et al. Reproducibility of Respondent-Driven Sampling (RDS) in Repeat Surveys of Men Who have Sex with Men, Unguja, Zanzibar. AIDS Behav 2016; 21(7); 2180-7. https://doi.org/10.1007/s10461-016-1632-2
https://doi.org/10.1007/s10461-016-1632-...
. It is a concern for the reliability of RDS and its lack challenges RDS as a valid sampling method. BBSS is also meant to be repeated regularly to monitor prevalence and program. If results from two survey rounds are different, can we attribute that to changes in study outcomes, characteristics of the study, or RDS itself? There are many potential answers to these questions, but certainly one would be to use diagnostic tools and formative research to assist interpretation and comparison. At least for surveillance, health authorities and community groups familiar with these tools are the best arbiters of what surveillance data means.

CONCLUSIONS

This paper reports on the methods used in the most recent national BBSS for MSM in Brazil. The comprehensive BBSS was completed as planned and on budget. Thedescription of methods is more detailed than usual, due to new diagnostic tools and requirements of the new STROBE-RDS guidelines2626. White RG, Hakim AJ, Salganik MJ, Spiller MW, Johnston LG, Kerr L, et al. Strengthening the Reporting of Observational Studies in Epidemiology for Respondent-Driven Sampling Studies:‘STROBE-RDS’Statement. J Clin Epidemiol 2015; 68(12): 1463-71. https://doi.org/10.1016/j.jclinepi.2015.04.002
https://doi.org/10.1016/j.jclinepi.2015....
. Because RDS requires strong statistical assumptions, these reports are increasingly important, both to assist in interpretation of findings and to encourage continued development of RDS. Reliable and precise estimates of HIV and other diseases in key populations over time are required if the true scope of the global HIV pandemic and conditions of vulnerability created by disparities is to be addressed.

ACKNOWLEDGEMENTS

We acknowledge the financial support provided by the Brazilian Ministry of Health, through the Secretariat for Health Surveillance and the Department of Prevention, Surveillance and Control of Sexually Transmitted Infections, HIV/AIDS and Viral Hepatitis (Projeto # 914BRZ1138, BRAZIL, AIDS-SUS). To the men who participated in the study and the organizations that support them, we have a debt that is difficult to repay. Facing - with the rest of the Brazilian population - an uncertain future for universal health coverage and a retreat from a positive and inclusive policy for sexual minorities makes it easy to overlook their contribution.

REFERENCES

  • 1
    Baral S, Sifakis F, Cleghorn F, Beyrer C. Elevated risk for HIV infection among men who have sex with men in low- and middle-income countries 2000-2006: a systematic review. PLoS Med 2007; 4(12): e339. https://doi.org/10.1371/journal.pmed.0040339
    » https://doi.org/10.1371/journal.pmed.0040339
  • 2
    Beyrer C, Sullivan PS, Sanchez J, Dowdy D, Altman D, Trapence G, et al. A call to action for comprehensive HIV services for men who have sex with men. Lancet 2012; 380(9839): 424-38. https://doi.org/10.1016/S0140-6736(12)61022-8
    » https://doi.org/10.1016/S0140-6736(12)61022-8
  • 3
    Beyrer C, Baral SD, Walker D, Wirtz AL, Johns B, Sifakis F. The expanding epidemics of HIV type 1 among men who have sex with men in low- and middle-income countries: diversity and consistency. Epidemiol Rev 2010; 32(1): 137-151. https://doi.org/10.1093/epirev/mxq011
    » https://doi.org/10.1093/epirev/mxq011
  • 4
    Beyrer C. Hidden yet happening: the epidemics of sexually transmitted infections and HIV among men who have sex with men in developing countries. Sex Transm Infect 2008; 84(6): 410-2. https://doi.org/10.1136/sti.2008.033290
    » https://doi.org/10.1136/sti.2008.033290
  • 5
    Beyrer C, Sullivan P, Sanchez J, Baral SD, Collins C, Wirtz AL, et al. The increase in global HIV epidemics in MSM. Aids 2013; 27(17): 2665-78. https://doi.org/10.1097/01.aids.0000432449.30239.fe
    » https://doi.org/10.1097/01.aids.0000432449.30239.fe
  • 6
    Beyrer C, Baral SD, van Griensven F, Goodreau SM, Chariyalertsak S, Wirtz AL, et al. Global epidemiology of HIV infection in men who have sex with men. Lancet 2012; 380(9839): 367-77. https://doi.org/10.1016/S0140-6736(12)60821-6
    » https://doi.org/10.1016/S0140-6736(12)60821-6
  • 7
    Baral S, Kizub D, Masenior NF, Peryskina A, Stachowiak J, Stibich M, et al. Male sex workers in Moscow, Russia: a pilot study of demographics, substance use patterns, and prevalence of HIV-1 and sexually transmitted infections. AIDS Care 2010; 22(1): 112-8. https://doi.org/10.1080/09540120903012551
    » https://doi.org/10.1080/09540120903012551
  • 8
    Beyrer C, Baral SD, Collins C, Richardson ET, Sullivan PS, Sanchez J, et al. The global response to HIV in men who have sex with men. Lancet 2016; 388(10040): 198-206. https://doi.org/10.1016/S0140-6736(16)30781-4
    » https://doi.org/10.1016/S0140-6736(16)30781-4
  • 9
    Brasil. Ministério da Saúde. Departamento de Vigilância, Prevenção e Controle das IST, do HIV/Aids e das Hepatites Virais. Boletim Epidemiologico de DST/AIDS. Brasília: Ministério da Saúde; 2016.
  • 10
    Szwarcwald C, Pascom A, Souza Júnior PR. Estimation of the HIV incidence and of the number of people living with HIV/AIDS in Brazil, 2012. J AIDS Clin Res 2015; 6(3): 430. https://doi.org/10.4172/2155-6113.1000430
    » https://doi.org/10.4172/2155-6113.1000430
  • 11
    Damacena GN, Szwarcwald CL, de Souza Júnior PR, Dourado I. Risk factors associated with HIV prevalence among female sex workers in 10 Brazilian cities. J Acquir Immune Defic Syndr 2011; 57(Suppl. 3): S144-52. https://doi.org/10.1097/QAI.0b013e31821e9bf6
    » https://doi.org/10.1097/QAI.0b013e31821e9bf6
  • 12
    Kerr LRFS, Mota RS, Kendall C, Pinho A de A, Mello MB, Guimarães MD, et al. HIV among MSM in a large middle-income country. Aids 2013; 27(3): 427-35. https://doi.org/10.1097/QAD.0b013e32835ad504
    » https://doi.org/10.1097/QAD.0b013e32835ad504
  • 13
    Bastos F. Taxas de infecção de HIV e sífilis e inventário de conhecimento, atitudes e práticas de risco relacionadas às infecções sexualmente transmissíveis entre usuários de drogas em 10 municípios brasileiros. In: Brasil. Ministério da Saúde. Departamento de Vigilância, Prevenção e Controle das IST, do HIV/Aids e das Hepatites Virais, ed. Brasília: Ministério da Saúde ; 2009.
  • 14
    Boily MC, Lowndes C, Alary M. The impact of HIV epidemic phases on the effectiveness of core group interventions: insights from mathematical models. Sex Transm Infect 2002; 78(Suppl. 1): i78-90.
  • 15
    Mishra RM, Dube M, Sahu D, Saggurti N, Pandey A. Changing epidemiology of HIV in Mumbai: an application of the Asian epidemic model. Glob J Health Sci 2012; 4(5): 100-12. https://dx.doi.org/10.5539%2Fgjhs.v4n5p100
    » https://dx.doi.org/10.5539%2Fgjhs.v4n5p100
  • 16
    Heckathorn DD. Respondent-driven sampling: A new approach to the study of hidden populations. Social Problems 1997; 44(2): 174-99. https://doi.org/10.2307/3096941
    » https://doi.org/10.2307/3096941
  • 17
    Gile KJ. Improved inference for respondent-driven sampling data with application to HIV prevalence estimation. J Am Statist Assoc 2011; 106(493): 135-46. https://doi.org/10.1198/jasa.2011.ap09475
    » https://doi.org/10.1198/jasa.2011.ap09475
  • 18
    Salganik MJ. Variance estimation, design effects, and sample size calculations for respondent-driven sampling. J Urban Health 2006; 83(6 Suppl.): i98-112. https://doi.org/10.1007/s11524-006-9106-x
    » https://doi.org/10.1007/s11524-006-9106-x
  • 19
    Johnston LG, Hakim AJ, Dittrich S, Burnett J, Kim E, White RG. A Systematic Review of Published Respondent-Driven Sampling Surveys Collecting Behavioral and Biologic Data. AIDS Behav 2016; 20(8): 1754-76. https://doi.org/10.1007/s10461-016-1346-5
    » https://doi.org/10.1007/s10461-016-1346-5
  • 20
    Malekinejad M, Johnston LG, Kendall C, Kerr LRFS, Rifkin MR, Rutherford GW. Using respondent-driven sampling methodology for HIV biological and behavioral surveillance in international settings: a systematic review. AIDS Behav 2008; 12(4 Suppl.): S105-30. https://doi.org/10.1007/s10461-008-9421-1
    » https://doi.org/10.1007/s10461-008-9421-1
  • 21
    Montealegre JR, Johnston LG, Murrill C, Monterroso E. Respondent driven sampling for HIV biological and behavioral surveillance in Latin America and the Caribbean. AIDS Behav 2013; 17(7): 2313-40. https://doi.org/10.1007/s10461-013-0466-4
    » https://doi.org/10.1007/s10461-013-0466-4
  • 22
    National Academies of Sciences, Engineering and Medicine. The Value of Social, Behavioral, and Economic Sciences to National Priorities: A Report for the National Science Foundation. Washington, D.C.: The National Academies Press; 2017.
  • 23
    Heckathorn DD. Extensions of Respondent-Driven Sampling: A new approach to the study of injection drug users aged 18-25. In: Semaan SBRSHJJ, ed. AIDS & Behav.2002:6(1):55-67.https://doi.org/10.1023/A:1014528612685
    » https://doi.org/10.1023/A:1014528612685
  • 24
    Heckathorn DD. Extensions of Respondent Driven Sampling: Analyzing continuous variables and controlling for differential recruitment. Sociol Methodol 2007; 37(1): 151-207. https://doi.org/10.1111/j.1467-9531.2007.00188.x
    » https://doi.org/10.1111/j.1467-9531.2007.00188.x
  • 25
    Gile KJ, Johnston LG, Salganik MJ. Diagnostics for respondent‐driven sampling. J Royal Statistical Soc 2015; 178(1): 241-69. https://doi.org/10.1111/rssa.12059
    » https://doi.org/10.1111/rssa.12059
  • 26
    White RG, Hakim AJ, Salganik MJ, Spiller MW, Johnston LG, Kerr L, et al. Strengthening the Reporting of Observational Studies in Epidemiology for Respondent-Driven Sampling Studies:‘STROBE-RDS’Statement. J Clin Epidemiol 2015; 68(12): 1463-71. https://doi.org/10.1016/j.jclinepi.2015.04.002
    » https://doi.org/10.1016/j.jclinepi.2015.04.002
  • 27
    Wejnert C, Pham H, Krishna N, Le B, DiNenno E. Estimating design effect and calculating sample size for respondent-driven sampling studies of injection drug users in the United States. AIDS Behav 2012; 16(4): 797-806. https://doi.org/10.1007/s10461-012-0147-8
    » https://doi.org/10.1007/s10461-012-0147-8
  • 28
    Centers for Disease Control and Prevention. Round 4: Formative Research Manual. City: Centers for Disease Control and Prevention; 2013.
  • 29
    Johnston LG, Whitehead S, Simic-Lawson M, Kendall C. Formative research to optimize respondent-driven sampling surveys among hard-to-reach populations in HIV behavioral and biological surveillance: lessons learned from four case studies. AIDS Care 2010; 22(6): 784-92. https://doi.org/10.1080/09540120903373557
    » https://doi.org/10.1080/09540120903373557
  • 30
    Gile KJ, Handcock MS. Respondent-driven sampling: an assessment of current methodology. Sociol Methodol 2010; 40(1): 285-327. https://dx.doi.org/10.1111%2Fj.1467-9531.2010.01223.x
    » https://dx.doi.org/10.1111%2Fj.1467-9531.2010.01223.x
  • 31
    Brasil. Ministério da Saúde. Secretaria de Vigilância em Saúde. Departamento de Vigilância, Prevenção e Controle das IST, do HIV/Aids e das Hepatites Virais. Pesquisa de Conhecimentos, Atitudes e Práticas na População Brasileira. Brasília: Ministério da Saúde / Secretaria de Vigilância em Saúde / Departamento de Vigilância, Prevenção e Controle das IST, do HIV/Aids e das Hepatites Virais; 2016.
  • 32
    Instituto Brasileiro de Geografia e Estatística. Projeção da população do Brasil e das Unidades da Federação [Internet]. IBGE; 2017 [accessed on Jul. 26, 2017]. Available at: Available at: http://www.ibge.gov.br/apps/populacao/projecao/
    » http://www.ibge.gov.br/apps/populacao/projecao/
  • 33
    Battaggia GE. Pesquisa de conhecimentos, atitudes e práticas na população residente no município de São Paulo. Brasília: Ministério da Saúde ; 2015.
  • 34
    Khatib A, Haji S, Khamis M, Said C, Khalid F, Dahoma M, et al. Reproducibility of Respondent-Driven Sampling (RDS) in Repeat Surveys of Men Who have Sex with Men, Unguja, Zanzibar. AIDS Behav 2016; 21(7); 2180-7. https://doi.org/10.1007/s10461-016-1632-2
    » https://doi.org/10.1007/s10461-016-1632-2
  • 35
    Veras MASM, Calazans GJ, de Almeida Ribeiro MCS, Oliveira CAF, Giovanetti MR, Facchini R, et al. High HIV prevalence among men who have sex with men in a time-location sampling survey, São Paulo, Brazil. AIDS Behav 2015; 19(9): 1589-98. https://doi.org/10.1007/s10461-014-0944-3
    » https://doi.org/10.1007/s10461-014-0944-3
  • 36
    Parker R. O Fim da AIDS? Rio de Janeiro: Associação Brasileira Interdisciplinar de Aids; 2015.
  • 37
    Alvy LM, McKirnan D, Du Bois SN, Jones K, Ritchie N, Fingerhut D. Health care disparities and behavioral health among men who have sex with men. J Gay Lesbian Soc Serv 2011; 23(4): 507-22. https://doi.org/10.1080/10538720.2011.611114
    » https://doi.org/10.1080/10538720.2011.611114
  • 38
    Buchmueller T, Carpenter CS. Disparities in health insurance coverage, access, and outcomes for individuals in same-sex versus different-sex relationships, 2000-2007. Am J Public Health 2010; 100(3): 489-95. https://doi.org/10.2105/AJPH.2009.160804
    » https://doi.org/10.2105/AJPH.2009.160804
  • 39
    McKirnan DJ, Du Bois SN, Alvy LM, Jones K. Health care access and health behaviors among men who have sex with men: the cost of health disparities. Health Educ Behav 2013; 40(1): 32-41. https://doi.org/10.1177/1090198111436340
    » https://doi.org/10.1177/1090198111436340

  • Financial support: Ministério da Saúde (Projeto no. 914BRZ1138)

  • ERRATUM / ERRATA
    DOI: 10.1590/1980-549720190004erratum In the manuscript “The 12 city HIV Surveillance Survey among MSM in Brazil 2016 using respondent-driven sampling: a description of methods and RDS diagnostics”, DOI: 10.1590/1980-549720190004, published in the Rev. bras. epidemiol. vol.22, Mar 14 2019. [ Epub ahead of print ] Where it reads: Luana Costa LindaXIII Ana Rita Coimbra Motta CastroX Inês Costa DouradoVII It should read: Luana Nepomuceno Gondim Costa LimaXIII Ana Rita Coimbra Motta-CastroX Inês DouradoVII

  • ERRATA / ERRATUM
    DOI: 10.1590/1980-549720190004erratum No artigo “Inquérito sobre o HIV entre HSH no Brasil em 2016 usando respondent-driven sampling (RDS): descrição dos métodos e dos diagnósticos do RDS”, DOI: 10.1590/1980-549720190004, publicado no periódico Rev. bras. epidemiol. vol.22, 14 Mar 2019. [ Epub ahead of print ] Onde se lia: Luana Costa LindaXIII Ana Rita Coimbra Motta CastroX Inês Costa DouradoVII Leia-se: Luana Nepomuceno Gondim Costa LimaXIII Ana Rita Coimbra Motta-CastroX Inês DouradoVII

Publication Dates

  • Publication in this collection
    14 Mar 2019
  • Date of issue
    2019

History

  • Received
    05 Nov 2017
  • Reviewed
    19 Apr 2018
  • Accepted
    15 May 2018
Associação Brasileira de Pós -Graduação em Saúde Coletiva São Paulo - SP - Brazil
E-mail: revbrepi@usp.br