Comparative effectiveness of single and dual rapid diagnostic tests for syphilis and HIV in antenatal care services in Colombia

Comparación de la eficacia de la prueba rápida individual y la prueba rápida dual para el diagnóstico de la sífilis y la infección por el VIH en los servicios de atención prenatal en Colombia

Hernando Guillermo Gaitán-Duarte Lori Newman Maura Laverty Ndema Abu Habib Lina María González-Gordon Edith Ángel-Müller Catleya Abella Esther Cristina Barros Carlos Rincón Sidia Caicedo ... About the authors

ABSTRACT

Objective

To assess the effectiveness of a dual rapid test compared to a single rapid test for syphilis and HIV screening.

Methods

A cluster-randomized open-label clinical trial was performed in 12 public antenatal care (ANC) centers in the cities of Bogotá and Cali, Colombia. Pregnant women who were over 14 years of age at their first antenatal visit and who had not been previously tested for HIV and syphilis during the current pregnancy were included. Pregnant women were randomized to single HIV and single syphilis rapid diagnostic tests (Arm A) or to dual HIV and syphilis rapid diagnostic tests (Arm B). The four main outcomes measured were: (1) acceptability of the test, (2) uptake in testing, (3) treatment on the same day (that is, timely treatment), and (4) treatment at any time for positive rapid test cases. Bivariate and multivariate analyses were calculated to adjust for the clustering effect and the period.

Results

A total of 1 048 patients were analyzed in Arm A, and 1 166 in Arm B. Acceptability of the rapid tests was 99.8% in Arm A and 99.6% in Arm B. The prevalence of positive rapid tests was 2.21% for syphilis and 0.36% for HIV. Timely treatment was provided to 20 of 29 patients (69%) in Arm A and 16 of 20 patients (80%) in Arm B (relative risk (RR), 1.10; 95% confidence interval (CI): (1.00 −1.20). Treatment at any time was given to 24 of 29 patients (83%) in Arm A and to 20 of 20 (100%) in Arm B (RR, 1.11; 95% CI: 1.01−1.22).

Conclusions

There were no differences in patient acceptability, testing and timely treatment between dual rapid tests and single rapid tests for HIV and syphilis screening in the ANC centers. Same-day treatment depends also on the interpretation of and confidence in the results by the health providers.

Key words
HIV; syphilis; pregnant women; point-of-care testing; effectiveness; patient acceptance of health care; Colombia

RESUMEN

Objetivo

Evaluar la eficacia de la prueba rápida dual en comparación con la prueba rápida individual para la detección sistemática de la sífilis y la infección por el VIH.

Métodos

Se realizó un ensayo clínico sin enmascaramiento y aleatorizado por grupos en 12 centros públicos de atención prenatal en las ciudades de Bogotá y Cali (Colombia). Se incluyó a las mujeres embarazadas de 14 o más años de edad que asistían a su primera consulta prenatal y no se habían realizado pruebas en este embarazo. Las embarazadas se dividieron de forma aleatoria para realizarles las pruebas rápidas individuales para el diagnóstico de sífilis y de infección por el VIH (Grupo A) o la prueba rápida dual para el diagnóstico de la sífilis y la infección por el VIH (Grupo B). Se midieron principalmente cuatro resultados: (1) aceptabilidad de la prueba, (2) uso de los servicios de prueba, (3) tratamiento el mismo día (es decir, tratamiento oportuno) y (4) tratamiento en cualquier momento en los casos con resultados positivos en las pruebas rápidas. Se realizaron análisis bifactoriales y multifactoriales para hacer los ajustes pertinentes por el efecto de la división en grupos y el período.

Resultados

Se estudió a 1 048 pacientes en el Grupo A y a 1 166 en el Grupo B. La aceptabilidad de las pruebas rápidas fue de 99,8% en el Grupo A y 99,6% en el Grupo B. La prevalencia de resultados positivos en las pruebas rápidas fue de 2,21% para la sífilis y 0,36% para la infección por el VIH. Se administró tratamiento oportuno a 20 de 29 pacientes (69%) del Grupo A y a 16 de 20 pacientes (80%) del Grupo B (riesgo relativo, 1,10; intervalo de confianza de 95% (IC): 1,00-1,20). Se administró tratamiento en cualquier momento a 24 de 29 pacientes (83%) del Grupo A y a 20 de 20 (100%) del Grupo B (riesgo relativo, 1,11; IC de 95%: 1,01-1,22).

Conclusiones

No hubo diferencias en cuanto a la aceptabilidad por parte de los pacientes, y el uso de los servicios de cribaje y el tratamiento oportuno entre las pruebas rápidas duales y las pruebas rápidas individuales para la detección sistemática de la sífilis y la infección por el VIH en los centros de atención prenatal. El tratamiento el mismo día depende también de la interpretación y la confianza en los resultados del trabajador de salud.

Palabras clave
VIH; sífilis; mujeres embarazadas; pruebas en el punto de atención; efectividad; aceptación de la atención de salud; Colombia

Manuscript received on 18 May 2016. Revised version accepted for publication on 24 October 2016.

Suggested citation Gaitan-Duarte HG, Newman L, Laverty M, Habib N, González-Gordon LM, Ángel-Müller E, et al. Comparative effectiveness of single and dual rapid diagnostic tests for syphilis and HIV in antenatal care services in Colombia. Rev Panam Salud Publica. 2016;40(6):455–62.

Syphilis is a sexually transmitted bacterial infection that is caused by Treponema pallidum and that can be transmitted from an infected mother to her unborn child. Infection from mother to fetus or to child is known as congenital syphilis (11 Workowski KA, Bolan GA, Centers for Disease Control and Prevention. Sexually transmitted diseases treatment guidelines, 2015. MMWR Recomm Rep. 2015;64(RR-03):1−137. Erratum in: MMWR Recomm Rep. 2015 Aug 28;64(33):924.). HIV can also be transmitted from mother to child during pregnancy, delivery, or breast-feeding (22 Luzuriaga K, Mofenson LM. Challenges in the Elimination of Pediatric HIV-1 Infection. N Engl J Med. 2016;374:761-70.).

In 2012 the World Health Organization (WHO) estimated that 930 000 syphilis infections occur among pregnant women worldwide annually, resulting in 350 000 adverse birth outcomes, including 143 000 early fetal deaths/stillbirths, 62 000 neonatal deaths, 44 000 preterm/low-weight births, and 102 000 infected infants (33 Wijesooriya NS, Rochat RW, Kamb ML, Turlapati P, Temmerman M, Broutet N, et al. Global burden of maternal and congenital syphilis in 2008 and 2012: a health systems modelling study. Lancet Glob Health. 2016;4(8):e525−3. doi:10.1016/S2214−109X(16)30135-8
https://doi.org/10.1016/S2214−109X(16)30...
). Early syphilis affects the fetus in 90% of cases, and late syphilis could affect it in up to 30% of cases (44 Berman SM. Maternal syphilis: pathophysiology and treatment. Bull World Health Organ. 2004;82(6):433−8.). In 2007 it was reported that a total of 36 000 pregnant women were living with HIV in Latin America, and that not treating them could result in 20% to 50% of their infants becoming infected (22 Luzuriaga K, Mofenson LM. Challenges in the Elimination of Pediatric HIV-1 Infection. N Engl J Med. 2016;374:761-70.).

It has been acknowledged for many years that prevention of mother-to-child transmission (PMTCT) is feasible, even in low-resource settings. Early screening and detection of syphilis and HIV and rapid treatment for positive pregnant women for syphilis (2.4 million units of benzathine penicillin) and HIV (initiation of antiretroviral therapy [ART]) will prevent mother-to-child transmission. Based on this, the WHO launched strategies for PMTCT of HIV and syphilis in 2011 (55 World Health Organization. The global elimination of congenital syphilis: rationale and strategy for action. Geneva: WHO; 2007. Available at: http://www.who.int/reproductivehealth/publications/rtis/9789241595858/en/index.html Accessed 20 January 2016.
http://www.who.int/reproductivehealth/pu...
, 66 Joint United Nations Programme on HIV/AIDS. Global plan towards the elimination of new HIV infections among children by 2015 and keeping their mothers alive. Geneva: UNAIDS; 2011 Available at: http://www.unaids.org/en/media/unaids/contentassets/documents/unaidspublication/2011/20110609_JC2137_Global-Plan-Elimination-HIV-Children_en.pdf Accessed 20 January 2016.
http://www.unaids.org/en/media/unaids/co...
). Both strategies include early diagnosis and timely treatment. The use of rapid diagnostic tests (RDTs) at point of care (POC) can help to implement these strategies.

In many settings, health workers are using laboratory tests for syphilis and HIV screening (VDRL or RPR for syphilis and enzyme-linked immunosorbent assay (ELISA) for HIV). Performing those tests requires laboratory equipment and facilities and in many cases same-day results are not provided. This leads to missed opportunities for timely syphilis and HIV treatment since it is not infrequent that patients either cannot or do not return to the clinic for results and treatment (55 World Health Organization. The global elimination of congenital syphilis: rationale and strategy for action. Geneva: WHO; 2007. Available at: http://www.who.int/reproductivehealth/publications/rtis/9789241595858/en/index.html Accessed 20 January 2016.
http://www.who.int/reproductivehealth/pu...
, 66 Joint United Nations Programme on HIV/AIDS. Global plan towards the elimination of new HIV infections among children by 2015 and keeping their mothers alive. Geneva: UNAIDS; 2011 Available at: http://www.unaids.org/en/media/unaids/contentassets/documents/unaidspublication/2011/20110609_JC2137_Global-Plan-Elimination-HIV-Children_en.pdf Accessed 20 January 2016.
http://www.unaids.org/en/media/unaids/co...
). RDTs applied at POC involve minimal technical training and do not require laboratory facilities. Results are available within 20 minutes of the testing, allowing for early detection of infection and same-day treatment for syphilis and referral for HIV treatment (77 Peeling RW, Ye H. Diagnostic tools for preventing and managing maternal and congenital syphilis: an overview. Bull World Health Organ. 2004;82(6):439–46.). Available studies on POC tests’ accuracy for syphilis show sensitivities of 74.3% to 90.0% and specificities of 94.2% to 99.6% (88 Jafari Y, Peeling RW, Shivkumar S, Claessens C, Joseph L, Pai NP. Are Treponema pallidum specific rapid and point-of-care tests for syphilis accurate enough for screening in resource limited settings? Evidence from a meta-analysis. PLoS One. 2013;8(2):e54695. doi: 10.1371/journal.pone.0054695.
https://doi.org/10.1371/journal.pone.005...
). It has been reported that diagnosis of gestational syphilis with RDTs and same-day treatment leads to a 98% reduction in congenital syphilis incidence (relative risk (RR), 0.02; 95% confidence interval (CI): 0.00–0.18) and a 47% decrease in the risk of perinatal death (RR, 0.50; 95% CI: 0.33–0.84) (99 Shahrook S, Mori R, Ochirbat T, Gomi H. Strategies of testing for syphilis during pregnancy. Cochrane Database Syst Rev. 2014 Oct 29;10:CD010385. doi:0.1002/14651858.CD010385.
https://doi.org/10.1002/14651858.CD01038...
). Evaluations on POC tests for HIV report sensitivities of 86.4% to 100% and specificities of 96.4% to 100% (1010 Pai NP, Tulsky JP, Cohan D, Colford JM Jr, Reingold AL. Rapid point-of-care HIV testing in pregnant women: a systematic review and meta-analysis. Trop Med Int Health. 2007;12(2):162−73.).

In Colombia, the Ministry of Health and Social Protection reported in its Boletín Epidemiológico weekly newsletter that over the 2011–2014 period the incidence of congenital syphilis decreased from 3.03 to 2.34 per 1 000 live-births (1111 Colombia, Instituto Nacional de Salud. Boletín Epidemiológico. Available at: http://www.ins.gov.co/boletin-epidemiologico/Paginas/default.aspx Accessed 21 January 2016.
http://www.ins.gov.co/boletin-epidemiolo...
). Until 2014, syphilis screening during antenatal care (ANC) visits was mainly done using nontreponemal tests, while HIV diagnosis was undertaken with two positive ELISA tests (or through a rapid test in settings where ELISA tests were not available) and confirmed by Western blot.

Based on the cited evidence, in 2014 the Ministry of Health and Social Protection launched national clinical practice guidelines (CPGs) for gestational and congenital syphilis and HIV in adult patients (1212 Ministerio de Salud y Protección Social y Fondo de Población de las Naciones Unidas. Guía de práctica clínica basada en la evidencia para la atención integral de la sífilis gestacional y congénita. Bogotá: Ministerio de Salud y Protección Social y Fondo de Población de las Naciones Unidas – UNFPA; 2014. Available at: http://gpc.minsalud.gov.co/guias/Documents/SifilisGestacional/gpc_guiacomple_sifilisant.pdf Accessed 20 January 2016.
http://gpc.minsalud.gov.co/guias/Documen...
, 1313 Ministerio de Salud y Protección Social y Fondo de Población de las Naciones Unidas. Guía de práctica clínica basada en la evidencia científica para la atención de la infección por VIH/Sida en adolescentes (con 13 años de edad o más) y adultos. Bogotá: Ministerio de Salud y Protección Social y Fondo de Población de las Naciones Unidas – UNFPA; 2014. Available at: http://gpc.minsalud.gov.co/guias/Documents/VIHSida/GPC_Comple_VIHADULTOS_web.pdf Accessed 20 January 2016.
http://gpc.minsalud.gov.co/guias/Documen...
). Those CPGs established clear recommendations for the use of POC rapid tests for HIV and gestational syphilis detection in order to reduce mother-to-child transmission.

Currently, two platforms are being used. One is a single rapid diagnostic test (SRDT) using two different cartridges, one for HIV and one for syphilis testing. The other is a dual rapid diagnostic test (DRDT) that contains tests for both HIV and syphilis in the same cartridge. Studies assessing the diagnostic performance of DRDTs have been published: sensitivity ranges from 97.9% to 100% for HIV and 81.0% to 100% for syphilis, and specificity from 94.2% to 99.5% for HIV and 96.9% to 100% for syphilis, depending on the test brand (1414 Yin YP, Ngige E, Anyaike C, Ijaodola G, Oyelade TA, Vaz RG, et al. Laboratory evaluation of three dual rapid diagnostic tests for HIV and syphilis in China and Nigeria. Int J Gynaecol Obstet. 2015;130 Suppl 1:S22−6. doi: 10.1016/j.ijgo.2015.04.004.
https://doi.org/10.1016/j.ijgo.2015.04.0...
, 1515 Omoding D, Katawera V, Siedner M, Boum Y 2nd. Evaluation of the SD BIOLINE HIV/syphilis Duo assay at a rural health center in Southwestern Uganda. BMC Res Notes. 2014;7−746. doi: 10.1186/1756-0500-7-746.
https://doi.org/10.1186/1756-0500-7-746...
), population (1616 Humphries RM, Woo JS, Chung JH, Sokovic A, Bristow CC, Klausner JD. Laboratory evaluation of three rapid diagnostic tests for dual detection of HIV and Treponema pallidum antibodies. J Clin Microbiol. 2014;52(12):4394−7. doi: 10.1128/JCM.02468-14.
https://doi.org/10.1128/JCM.02468-14...
), and testing setting (laboratory or field) (1717 Bristow CC, Leon SR, Huang E, Brown BJ, Ramos LB, Vargas SK, et al. Field evaluation of a dual rapid diagnostic test for HIV infection and syphilis in Lima, Peru. Sex Transm Infect. 2016;92(3): 182−5. doi: 10.1136/sextrans-2015-052326. Epub 2015 Dec 15.
https://doi.org/10.1136/sextrans-2015-05...
, 1818 Bristow CC, Leon SR, Huang E, Ramos LB, Vargas SK, Flores JA, et al. Field evaluation of a dual rapid immunodiagnostic test for HIV and syphilis infection in Peru. Sex Transm Dis. 2016;43(1):57−60. doi: 10.1097/OLQ.0000000000000387.
https://doi.org/10.1097/OLQ.000000000000...
).

Apart from the accuracy of the SRDTs and the DRDTs, it is relevant to compare the two platforms in terms of other characteristics that could affect the usage of the tests, including acceptability, feasibility, testing, timely treatment, and costs. For various of these characteristics, there are limited published data available. The objective of this study was to assess the effectiveness of DRDTs compared to SRDTs for HIV and syphilis in pregnant women in terms of: 1. acceptability of testing by patients, 2. uptake in testing, 3. timely treatment, and 4. any-time treatment of positive rapid-test cases.

MATERIALS AND METHODS

Study design

An open-label cluster-randomized controlled trial was designed for comparing SRDTs for HIV and syphilis with DRDTs for HIV and syphilis. The inclusion criteria were: pregnant women over 14 years of age who were attending a first ANC visit and who had not been previously tested in the current pregnancy or did not know their syphilis and/or HIV serostatus. All participants were required to provide a signed informed consent. There were no exclusion criteria. A cluster design was selected in order to guard against contamination bias between arms and to better standardize procedures at the health institutions.

The study was conducted in 12 public ANC clinics, six in the city of Bogotá and six in the city of Cali. Clusters were selected through convenience sampling by choosing institutions based on a history of having at least 250 first antenatal visits per month as well as similar numbers of reported cases of HIV and syphilis. No cluster sites were using RDTs prior to the study.

Procedures

Clusters were randomly allocated to Arm A (SRDTs) or to Arm B (DRDTs) with a 1:1 allocation ratio, using SAS software (1919 SAS Institute. The SAS system for Windows. Release 9.3. Cary, North Carolina: SAS Inst.; 2011.). The two rapid tests used in Arm A were the SD BIOLINE syphilis 3.0 and the SD BIOLINE HIV 3.0. In Arm B, the SD BIOLINE HIV/Syphilis Duo dual test was used. These tests were selected according to an assessment performed by the National Institute of Health of Colombia based on: test-devise characteristics, ease of use and interpretation of the results as well as availability in the country. The tests use an immunochromatographic assay for the simultaneous qualitative detection of antibodies to all isotypes (IgG, IgM, and IgA) specific to HIV-1 and HIV-2 and Treponema pallidum in human serum, plasma, or whole blood. In these tests, the recombinant HIV-1/2 antigen (gp41, gp36) - colloid gold conjugate or/and the recombinant Treponema pallidum antigens colloid gold conjugate (17KDa), specimen sample, and sample diluent move along the membrane chromatographically to the test region and form a visible line as the antigen-antibody-antigen gold particle complex forms, with a high degree of sensitivity and specificity (2020 Shimelis T, Tadesse E. The diagnostic performance evaluation of the SD BIOLINE HIV/syphilis Duo rapid test in southern Ethiopia: a cross-sectional study. BMJ Open. 2015 Apr 23;5(4):e007371. doi: 10.1136/bmjopen-2014-00737.
https://doi.org/10.1136/bmjopen-2014-007...
). Nurses were recruited, trained, and certified to perform and interpret the rapid tests (single or dual) and to deliver pre- and posttest counseling for HIV and syphilis, according to national CPGs (1212 Ministerio de Salud y Protección Social y Fondo de Población de las Naciones Unidas. Guía de práctica clínica basada en la evidencia para la atención integral de la sífilis gestacional y congénita. Bogotá: Ministerio de Salud y Protección Social y Fondo de Población de las Naciones Unidas – UNFPA; 2014. Available at: http://gpc.minsalud.gov.co/guias/Documents/SifilisGestacional/gpc_guiacomple_sifilisant.pdf Accessed 20 January 2016.
http://gpc.minsalud.gov.co/guias/Documen...
, 1313 Ministerio de Salud y Protección Social y Fondo de Población de las Naciones Unidas. Guía de práctica clínica basada en la evidencia científica para la atención de la infección por VIH/Sida en adolescentes (con 13 años de edad o más) y adultos. Bogotá: Ministerio de Salud y Protección Social y Fondo de Población de las Naciones Unidas – UNFPA; 2014. Available at: http://gpc.minsalud.gov.co/guias/Documents/VIHSida/GPC_Comple_VIHADULTOS_web.pdf Accessed 20 January 2016.
http://gpc.minsalud.gov.co/guias/Documen...
). The staff from the health institutions were also trained on study procedures and recommendations of the current Colombian CPGs regarding HIV and congenital and gestational syphilis management, but no specific training on this topic was provided to clinicians. Allocation of clusters was concealed until the cluster was ready to recruit patients.

After performing standard-of-care ANC visit procedures, study personnel invited women who fulfilled the eligibility criteria to participate, and the informed consent process was initiated. An initial survey collecting baseline characteristics of pregnant women was carried out by nurses in Bogotá and technical nurses in Cali and was entered into electronic forms. Women were then tested for HIV and syphilis according to the cluster allocation. Tests were performed according to the manufacturer’s guidelines, using blood collected by a finger stick, one drop per cartridge for the SRDTs and one drop for the DRDTs.

The rapid tests were conducted by study personnel and read within 15 minutes. All patients who tested positive for syphilis were immediately sent to the reference center in order to be evaluated by a clinician for treatment with benzathine penicillin G 2.4 million units IM in a single dose (early latent syphilis) or benzathine penicillin G 7.2 million units total, administered as three doses of 2.4 million units IM each at one-week intervals (late latent syphilis or latent syphilis of unknown duration) based on Colombian guidelines. All patients who tested positive for HIV received confirmatory HIV testing through a second different rapid test and Elisa test according to national HIV management CPGs. Patient management was recorded in the patient questionnaire as well as in clinical charts. Data from the data collection forms were verified and recorded in Excel spreadsheets at the study sites.

Variables measured

The variables that were measured were: age, education level, employment status, number of sexual partners in the last 6 months, pregnancies, previous miscarriage (less than 22 weeks of gestational age), previous stillborn (equal to or more than 22 weeks of gestational age), gestational age, proportion of patients who accepted the tests in the two arms, proportion of testing in the two arms, proportion of patients with positive results for syphilis who were treated in the first 24 hours, proportion of positive cases for syphilis who were treated at any time, and serious adverse events.

All the facilities were performing serological tests to screen for HIV and syphilis. During that period, the total number of first ANC visits was 1 159 in Arm A and 1 485 in Arm B.

Statistical methods

A sample size of 1 200 patients per arm was calculated, taking into account a 20% increase in syphilis testing uptake between both arms (the proportion of patients accepting syphilis testing in Arm A was assumed to be 55%), with type I error set at a 5% level, 80% power, and intra-cluster correlation of 0.05 assumed. An average cluster size of 200 was calculated. However, because the size of clusters during the evaluation period of about 4.5 months (18 weeks) ranged widely, a coefficient of variation of cluster size of 0.47 was estimated, using the method outlined by Eldridge et al. (2121 Eldridge SM, Ashby D, Kerry S. Sample size for cluster randomized trials: effect of coefficient of variation of cluster size and analysis method. Int J Epidemiol. 2006 Oct;35(5):1292−300.), and was used in the estimation of the design effect for the cluster design.

Patient flow in the study for the cluster-randomized trial is shown using a CONSORT diagram (Figure 1). Primary analyses were based on the intention-to-treat principle. Baseline characteristics of the women enrolled in each arm were provided. For qualitative variables, frequencies and percentages are reported. For quantitative normally distributed variables, the number of subjects, means, and standard errors are provided, while medians, interquartile ranges, minima, and maxima are reported for skewed nonnormal quantitative variables. Because of the clustered nature of outcomes due to randomization to interventions based on clusters (facilities), a Poisson regression model with robust standard errors was used to estimate the crude risk ratio and risk difference for syphilis treatment, with a 95% CI, adjusted for clustering, between interventions in Arm A and Arm B. This model was also adjusted for any important baseline factors, including historical performance for testing in syphilis or HIV at the participant institutions and regional differences. The SRDTs were regarded as the control group for this study. Two-sided tests and 5% significance levels and 95% CIs for all relevant parameters were calculated. In addition, SAS statistical packages that take into account dependency of outcomes were used for the analyses.

FIGURE 1
Flowchart of the institutions and patients in the study of single and dual HIV and syphilis rapid diagnostic tests (RDTs) in antenatal care in Colombia, 2014–2015

The study protocol was reviewed and approved by the Universidad Nacional de Colombia and the WHO Ethics Research Committee. The identifier in the international randomized controlled trials register is NCT02454816.

RESULTS

From 7 October 2014 to 30 March 2015, a total of 2 225 pregnant women were candidates to be enrolled in the study. Seven of the women (0.3%) did not accept having a rapid test performed and thus participating in the study. The acceptability level for the rapid tests was 99.8% in Arm A and 99.6% in Arm B. Thus, 2 218 patients were assigned to Arm A or Arm B and were tested for syphilis and HIV. There was a lack of adherence to the eligibility criteria considered in the protocol in a total of four patients (not pregnant), all of them in Arm A. Ultimately, 1 048 patients were analyzed in Arm A, and 1 166 patients were analyzed in Arm B (Figure 1). Patients in Arm A were recruited in the following clusters: Carlos Holmes Trujillo, a primary care hospital, El Vallado, and Marroquín health centers in Cali and three health centers in Bogotá: CAMI Altamira, UPA Porvenir, and UPA Primero de Mayo. Patients from Arm B were recruited in the Decepaz, El Diamante, and Potrero Grande health centers in Cali and the Suba, Vista Hermosa, and Pablo Sexto primary care hospitals in Bogotá. The intervention lasted 18 weeks in each cluster, with differing times for start of recruitment due to institutional issues.

With respect to the baseline characteristics of the patients, they had a similar age (22 years old in Arm A and 21 years old in Arm B), education level (secondary education, 78.3% and 80.2% in Arm A and Arm B, respectively), and work outside the home (Table 1). Concerning the number of sexual partners, a higher number of women from Arm B reported one partner in the last six months. Close to one-third reported a previous miscarriage, and less than 3% reported a stillborn. Pregnant women were seen for their first antenatal care visit during the first trimester in the majority of cases.

TABLE 1
Baseline characteristics of women recruited in the study of single and dual HIV and syphilis rapid diagnostic tests in antenatal care in Colombia, 2014–2015

Testing for syphilis and HIV in the intervention period was 100% in both arms of the study. In comparison to the period prior to the intervention, syphilis testing showed an increase of 9.7% in Arm A and of 6.6% in Arm B; for HIV testing, there was an increase of 16.2% in Arm A and 8.6% in Arm B.

The overall prevalence of positive rapid tests for syphilis was 2.21%. In Bogotá, it was 1.19% (13 of 1 091); in Cali, it was 3.2% (36 of 1 123). The prevalence of HIV-positive rapid test results was 0.26% in Cali and 0.47% in Bogotá.

We found that 29 of 1 048 pregnant women (2.7%) in Arm A and 20 of 1 166 women (1.7%) in Arm B showed positive rapid test results for syphilis. Regarding HIV rapid test results, 8 pregnant women were found with positive results, 5 in Arm B and 3 in Arm A.

The number of women treated for syphilis according to the protocol (same day of the test results) were 20 of 29 (69%) in Arm A and 16 of 20 (80%) in Arm B. When the comparison was adjusted for period effect, no differences were found in timely treatment between the single and dual tests (RR, 1.09; 95% CI: 0.92–1.28). Neither were differences found when the comparison was adjusted by region (Cali or Bogotá) and period. The treatment was superior in the dual test group (RR, 1.10; 95% CI: 1.00–1.20) ( Table 2 and Table 3).

TABLE 2
Syphilis treatment in patients with positive rapid tests in antenatal care services in Colombia, 2014–2015
TABLE 3
Syphilis treatment per intervention arm, with crude data and adjusted relative risk (RR) and risk difference (RD) estimates in antenatal care services in Colombia, 2014–2015

Regarding treatment at any time, 24 of 29 (82.8%) pregnant women from Arm A received treatment, in comparison to 20 of 20 (100%) in Arm B (RR, 1.11; 95% CI: 1.01–1.21), regardless of region adjustment or not (Table 2 and Table 3). These pregnant women were treated on a different day from the positive rapid test results or were not treated at all due to a lack of adherence to the protocol by physicians who had seen the patients prior to the prescription of benzathine penicillin. These physicians requested a nontreponemal test as a requirement for penicillin treatment; therefore, treatment was given only after the positive nontreponemal test results were available. In five cases, the physicians considered the rapid test results as false positives, so treatment was not provided.

Among pregnant women with HIV-positive rapid test results, nurses performed a second rapid test, with a different kit (dual instead of single and vice versa), from which they found six reactive tests, four in Arm B and two in Arm A. The confirmatory tests showed two women from Cali with positive Western blots, one in each arm of the study, and they were referred to the HIV management program. Therefore, there were six false-positive results from the HIV rapid tests.

There were no reports of serious adverse events in patients or health workers related to the use of SRDTs or DRDTs during the study.

DISCUSSION

Currently, there is an interest in improving care and management among patients in relation to HIV and syphilis screening in ANC. This is especially true in low-resource settings. This study showed that there are no differences between single and dual tests to be applied at POC for syphilis and HIV infections during pregnancy in terms of acceptability, testing and timely treatment (treatment on the same day of the rapid test results) when region and cluster effect are taken into account. However, treatment at any time for positive rapid test cases was superior in the Dual test. There were no serious adverse events during the application of the test. In addition, our descriptive findings suggest that the RDTs increase the uptake of syphilis and HIV testing in prenatal control compared to levels in the pre-intervention period.

Both arms of the study could be considered intervention groups based on the fact that in Colombia, no rapid tests were used before this research. Therefore, it is not surprising that acceptability and testing levels were high and very similar (close to 100%) in both comparison groups. On the other hand, this study highlights that in real life there are important issues related to treatment, beyond the availability of obtaining a positive rapid test result at the POC. We found that 13 of 49 physicians (26.5%) did not consider the rapid test results enough to prescribe treatment to prevent congenital syphilis. This situation has been described in other settings (2222 Myer L, Wilkinson D, Lombard C, Zuma K, Rotchford K, Karim SS. Impact of on-site testing for maternal syphilis on treatment delays, treatment rates, and perinatal mortality in rural South Africa: a randomised controlled trial. Sex Transm Infect. 2003 Jun;79(3):208−13.).

Our data also shows differences between SRDTs and DRDTs in terms of timely treatments when the effect was adjusted by region and period. However, these results need to be viewed with caution as they could be associated with preexisting differences in testing between the two cities in the pre-intervention period rather than to real differences between the two tests.

There are limited data that compare both SRDTs and DRDTs in terms of acceptability, testing, and timely treatment in the first 24 hours. Since 2014, studies have been published assessing dual rapid tests for HIV and syphilis. Those assessments have been done mainly from the point of view of accuracy (1414 Yin YP, Ngige E, Anyaike C, Ijaodola G, Oyelade TA, Vaz RG, et al. Laboratory evaluation of three dual rapid diagnostic tests for HIV and syphilis in China and Nigeria. Int J Gynaecol Obstet. 2015;130 Suppl 1:S22−6. doi: 10.1016/j.ijgo.2015.04.004.
https://doi.org/10.1016/j.ijgo.2015.04.0...
1818 Bristow CC, Leon SR, Huang E, Ramos LB, Vargas SK, Flores JA, et al. Field evaluation of a dual rapid immunodiagnostic test for HIV and syphilis infection in Peru. Sex Transm Dis. 2016;43(1):57−60. doi: 10.1097/OLQ.0000000000000387.
https://doi.org/10.1097/OLQ.000000000000...
, 2323 Bristow CC, Adu-Sarkodie Y, Ondondo RO, Bukusi EA, Dagnra CA, Oo KY, et al Multisite laboratory evaluation of a dual human immunodeficiency virus (HIV)/syphilis point-of-care rapid test for simultaneous detection of HIV and syphilis infection. Open Forum Infect Dis. 2014;1(1):ofu015. doi: 10.1093/ofid/ofu015. eCollection 2014 Mar.
https://doi.org/10.1093/ofid/ofu015...
, 2424 Leon SR, Ramos LB, Vargas SK, Kojima N, Perez DG, Caceres CF, et al. Laboratory evaluation of a dual-path platform assay for rapid point-of-care HIV and syphilis testing. J Clin Microbiol. 2016;54(2):492−4. doi: 10.1128/JCM.03152-15.
https://doi.org/10.1128/JCM.03152-15...
) and related costs (2525 Bristow CC, Larson E, Anderson LJ, Klausner JD. Cost-effectiveness of HIV and syphilis antenatal screening: a modelling study. Sex Transm Infect. 2016 Feb 26. pii: sextrans-2015−052367. doi: 10.1136/sextrans-2015-052367.
https://doi.org/10.1136/sextrans-2015-05...
). There is no available information about the comparative effectiveness of SRDTs versus DRDTs in real-life conditions.

A systematic review that examined the impact of introducing rapid tests for syphilis and HIV in pregnant women in low- and middle-income countries reported a larger increase in antenatal syphilis and HIV testing than our study did (2626 Swartzendruber A, Steiner RJ, Adler MR, Kamb ML, Newman LM. Introduction of rapid syphilis testing in antenatal care: a systematic review of the impact on HIV and syphilis testing uptake and coverage. Int J Gynaecol Obstet. 2015 Jun;130 Suppl 1:S15−21. doi: 10.1016/j.ijgo.2015.04.008. Epub 2015 Apr 29.
https://doi.org/10.1016/j.ijgo.2015.04.0...
). That review covered six studies done in various settings in countries in South America, Africa, and Asia. The review reported increases in the proportion of pregnant women screened for syphilis that ranged from 39% to 90% (2727 Flores EC, Lluque ME, Chiappe M, Lino R, Bayer AM. Operations research study to implement HIV and syphilis point-of-care tests and assess client perceptions in a marginalised area of Lima, Peru. Int J STD AIDS 2014 [pii: 0956462414552696, Epub ahead of print].3232 Pai NP, Kurji J, Singam A, Barick R, Jafari Y, Klein MB, et al. Simultaneous triple point of care testing for HIV, syphilis and hepatitis B virus to prevent mother-to-child transmission in India. Int J STD AIDS. 2012;23:319–24.). With HIV testing, the results were less consistent, ranging from an increase of 40% to a decrease of 10% (2727 Flores EC, Lluque ME, Chiappe M, Lino R, Bayer AM. Operations research study to implement HIV and syphilis point-of-care tests and assess client perceptions in a marginalised area of Lima, Peru. Int J STD AIDS 2014 [pii: 0956462414552696, Epub ahead of print].2929 Strasser S, Bitarakwate E, Gill M, Hoffman HJ, Musana O, Phiri A, et al. Introduction of rapid syphilis testing within prevention of mother-to-child transmission of HIV programs in Uganda and Zambia: a field acceptability and feasibility study. J Acquir Immune Defic Syndr. 2012;61(3):e40–6., 3131 Delvaux T, Samreth S, Barr-DiChiara M, Seguy N, Guerra K, Ngauv B, et al. Linked response for prevention, care, and treatment of HIV/AIDS, STIs, and reproductive health issues: results after 18 months of implementation in five operational districts in Cambodia. J Acquir Immune Defic Syndr. 2011;57:e47–55., 3232 Pai NP, Kurji J, Singam A, Barick R, Jafari Y, Klein MB, et al. Simultaneous triple point of care testing for HIV, syphilis and hepatitis B virus to prevent mother-to-child transmission in India. Int J STD AIDS. 2012;23:319–24.).

The use of RDTs has resulted in an improvement in the uptake of syphilis treatment of pregnant women, according to the majority of published studies, in comparison to the treatment rates for syphilis found in this study. In Mongolia (with 7 700 pregnant women), POC testing was found to be effective in increasing syphilis treatment uptake and thus reducing congenital syphilis (3333 Munkhuu B, Liabsuetrakul T, Chongsuvivatwong V, McNeil E, Janchiv R. One-stop service for antenatal syphilis screening and prevention of congenital syphilis in Ulaanbaatar, Mongolia: a cluster randomized trial. Sex Transm Dis. 2009 Nov;36(11):714−20. doi: 10.1097/OLQ.0b013e3181bc0960.
https://doi.org/10.1097/OLQ.0b013e3181bc...
). In contrast, in a study in South Africa (2222 Myer L, Wilkinson D, Lombard C, Zuma K, Rotchford K, Karim SS. Impact of on-site testing for maternal syphilis on treatment delays, treatment rates, and perinatal mortality in rural South Africa: a randomised controlled trial. Sex Transm Infect. 2003 Jun;79(3):208−13.), there was no evidence of an effect of POC syphilis testing in increasing adequate syphilis treatment rates (64.1% in the intervention cluster versus 68.6% in the control (OR 0.82; 95% CI: 0.57–1.17)), nor in reducing perinatal mortality (3.3% versus 5.1% (OR 0.63; 95% CI: 0.27–1.48)).

This study has the limitation of not having used gold standard tests for HIV and syphilis as a means of comparison of the SRDTs and DRDTs used; thus, it was not possible to establish the accuracy of the tests. Six patients were considered false positives for HIV when the DRTs were compared against confirmatory tests. This shows the important role of the second confirmatory test. Lack of adherence to the treatment protocol could have affected the effect estimation; nonetheless, this is considered in the effectiveness analysis. In terms of study strengths, we used a cluster-randomized trial to reduce the risks of bias and to control confounders. However, due to the design of an open trial, detection and performance biases could have occurred.

Conclusion

In practice, in Colombia, DRDTs are similar to SRDTs for HIV and syphilis screening in ANC, in terms of patient acceptability, testing and timely treatment. Nevertheless, same-day treatment depends not only on the type of RDTs, but also on the interpretation of and confidence in the results by the health providers who prescribe the tests. Based on our findings, care providers could choose either of these approaches for HIV and syphilis screening, but the above considerations should be taken into account when introducing POC RDTs.

Acknowledgments

To the health centers and the health professionals that collaborated with the study. In Bogotá: CAMI Suba, CAMI Vista Hermosa, CAMI Pablo Sexto, UPA Porvenir, UPA Primero de Mayo, and CAMI Altamira. In Cali: El Diamante, El Vallado, Decepaz, Potrero Grande, Marroquín, and Hospital Carlos Holmes Trujillo.

Funding

This study was funded by Pan American Health Organization and World Health Organization, WHO Reference 2014/460270-0 and 2013/390270-0. The study also received support from the Ministry of Health and Social Protection of Colombia. Rapid single and dual HIV and syphilis tests were donated by Standard Diagnostics, Inc.

Conflicts of Interest

Standard Diagnostics, Inc. donated the rapid single and dual tests for this study. However, neither the company nor its personnel participated in the preparation and conduct of the study or the analysis of the data.

Disclaimer

Authors hold sole responsibility for the views expressed in the manuscript, which may not necessarily reflect the opinion or policy of the RPSP/ PAJPH or PAHO.

REFERENCES

  • 1
    Workowski KA, Bolan GA, Centers for Disease Control and Prevention. Sexually transmitted diseases treatment guidelines, 2015. MMWR Recomm Rep. 2015;64(RR-03):1−137. Erratum in: MMWR Recomm Rep. 2015 Aug 28;64(33):924.
  • 2
    Luzuriaga K, Mofenson LM. Challenges in the Elimination of Pediatric HIV-1 Infection. N Engl J Med. 2016;374:761-70.
  • 3
    Wijesooriya NS, Rochat RW, Kamb ML, Turlapati P, Temmerman M, Broutet N, et al. Global burden of maternal and congenital syphilis in 2008 and 2012: a health systems modelling study. Lancet Glob Health. 2016;4(8):e525−3. doi:10.1016/S2214−109X(16)30135-8
    » https://doi.org/10.1016/S2214−109X(16)30135-8
  • 4
    Berman SM. Maternal syphilis: pathophysiology and treatment. Bull World Health Organ. 2004;82(6):433−8.
  • 5
    World Health Organization. The global elimination of congenital syphilis: rationale and strategy for action. Geneva: WHO; 2007. Available at: http://www.who.int/reproductivehealth/publications/rtis/9789241595858/en/index.html Accessed 20 January 2016.
    » http://www.who.int/reproductivehealth/publications/rtis/9789241595858/en/index.html
  • 6
    Joint United Nations Programme on HIV/AIDS. Global plan towards the elimination of new HIV infections among children by 2015 and keeping their mothers alive. Geneva: UNAIDS; 2011 Available at: http://www.unaids.org/en/media/unaids/contentassets/documents/unaidspublication/2011/20110609_JC2137_Global-Plan-Elimination-HIV-Children_en.pdf Accessed 20 January 2016.
    » http://www.unaids.org/en/media/unaids/contentassets/documents/unaidspublication/2011/20110609_JC2137_Global-Plan-Elimination-HIV-Children_en.pdf
  • 7
    Peeling RW, Ye H. Diagnostic tools for preventing and managing maternal and congenital syphilis: an overview. Bull World Health Organ. 2004;82(6):439–46.
  • 8
    Jafari Y, Peeling RW, Shivkumar S, Claessens C, Joseph L, Pai NP. Are Treponema pallidum specific rapid and point-of-care tests for syphilis accurate enough for screening in resource limited settings? Evidence from a meta-analysis. PLoS One. 2013;8(2):e54695. doi: 10.1371/journal.pone.0054695.
    » https://doi.org/10.1371/journal.pone.0054695.
  • 9
    Shahrook S, Mori R, Ochirbat T, Gomi H. Strategies of testing for syphilis during pregnancy. Cochrane Database Syst Rev. 2014 Oct 29;10:CD010385. doi:0.1002/14651858.CD010385.
    » https://doi.org/10.1002/14651858.CD010385
  • 10
    Pai NP, Tulsky JP, Cohan D, Colford JM Jr, Reingold AL. Rapid point-of-care HIV testing in pregnant women: a systematic review and meta-analysis. Trop Med Int Health. 2007;12(2):162−73.
  • 11
    Colombia, Instituto Nacional de Salud. Boletín Epidemiológico. Available at: http://www.ins.gov.co/boletin-epidemiologico/Paginas/default.aspx Accessed 21 January 2016.
    » http://www.ins.gov.co/boletin-epidemiologico/Paginas/default.aspx
  • 12
    Ministerio de Salud y Protección Social y Fondo de Población de las Naciones Unidas. Guía de práctica clínica basada en la evidencia para la atención integral de la sífilis gestacional y congénita. Bogotá: Ministerio de Salud y Protección Social y Fondo de Población de las Naciones Unidas – UNFPA; 2014. Available at: http://gpc.minsalud.gov.co/guias/Documents/SifilisGestacional/gpc_guiacomple_sifilisant.pdf Accessed 20 January 2016.
    » http://gpc.minsalud.gov.co/guias/Documents/SifilisGestacional/gpc_guiacomple_sifilisant.pdf
  • 13
    Ministerio de Salud y Protección Social y Fondo de Población de las Naciones Unidas. Guía de práctica clínica basada en la evidencia científica para la atención de la infección por VIH/Sida en adolescentes (con 13 años de edad o más) y adultos. Bogotá: Ministerio de Salud y Protección Social y Fondo de Población de las Naciones Unidas – UNFPA; 2014. Available at: http://gpc.minsalud.gov.co/guias/Documents/VIHSida/GPC_Comple_VIHADULTOS_web.pdf Accessed 20 January 2016.
    » http://gpc.minsalud.gov.co/guias/Documents/VIHSida/GPC_Comple_VIHADULTOS_web.pdf
  • 14
    Yin YP, Ngige E, Anyaike C, Ijaodola G, Oyelade TA, Vaz RG, et al. Laboratory evaluation of three dual rapid diagnostic tests for HIV and syphilis in China and Nigeria. Int J Gynaecol Obstet. 2015;130 Suppl 1:S22−6. doi: 10.1016/j.ijgo.2015.04.004.
    » https://doi.org/10.1016/j.ijgo.2015.04.004
  • 15
    Omoding D, Katawera V, Siedner M, Boum Y 2nd. Evaluation of the SD BIOLINE HIV/syphilis Duo assay at a rural health center in Southwestern Uganda. BMC Res Notes. 2014;7−746. doi: 10.1186/1756-0500-7-746.
    » https://doi.org/10.1186/1756-0500-7-746
  • 16
    Humphries RM, Woo JS, Chung JH, Sokovic A, Bristow CC, Klausner JD. Laboratory evaluation of three rapid diagnostic tests for dual detection of HIV and Treponema pallidum antibodies. J Clin Microbiol. 2014;52(12):4394−7. doi: 10.1128/JCM.02468-14.
    » https://doi.org/10.1128/JCM.02468-14
  • 17
    Bristow CC, Leon SR, Huang E, Brown BJ, Ramos LB, Vargas SK, et al. Field evaluation of a dual rapid diagnostic test for HIV infection and syphilis in Lima, Peru. Sex Transm Infect. 2016;92(3): 182−5. doi: 10.1136/sextrans-2015-052326. Epub 2015 Dec 15.
    » https://doi.org/10.1136/sextrans-2015-052326
  • 18
    Bristow CC, Leon SR, Huang E, Ramos LB, Vargas SK, Flores JA, et al. Field evaluation of a dual rapid immunodiagnostic test for HIV and syphilis infection in Peru. Sex Transm Dis. 2016;43(1):57−60. doi: 10.1097/OLQ.0000000000000387.
    » https://doi.org/10.1097/OLQ.0000000000000387
  • 19
    SAS Institute. The SAS system for Windows. Release 9.3. Cary, North Carolina: SAS Inst.; 2011.
  • 20
    Shimelis T, Tadesse E. The diagnostic performance evaluation of the SD BIOLINE HIV/syphilis Duo rapid test in southern Ethiopia: a cross-sectional study. BMJ Open. 2015 Apr 23;5(4):e007371. doi: 10.1136/bmjopen-2014-00737.
    » https://doi.org/10.1136/bmjopen-2014-00737
  • 21
    Eldridge SM, Ashby D, Kerry S. Sample size for cluster randomized trials: effect of coefficient of variation of cluster size and analysis method. Int J Epidemiol. 2006 Oct;35(5):1292−300.
  • 22
    Myer L, Wilkinson D, Lombard C, Zuma K, Rotchford K, Karim SS. Impact of on-site testing for maternal syphilis on treatment delays, treatment rates, and perinatal mortality in rural South Africa: a randomised controlled trial. Sex Transm Infect. 2003 Jun;79(3):208−13.
  • 23
    Bristow CC, Adu-Sarkodie Y, Ondondo RO, Bukusi EA, Dagnra CA, Oo KY, et al Multisite laboratory evaluation of a dual human immunodeficiency virus (HIV)/syphilis point-of-care rapid test for simultaneous detection of HIV and syphilis infection. Open Forum Infect Dis. 2014;1(1):ofu015. doi: 10.1093/ofid/ofu015. eCollection 2014 Mar.
    » https://doi.org/10.1093/ofid/ofu015
  • 24
    Leon SR, Ramos LB, Vargas SK, Kojima N, Perez DG, Caceres CF, et al. Laboratory evaluation of a dual-path platform assay for rapid point-of-care HIV and syphilis testing. J Clin Microbiol. 2016;54(2):492−4. doi: 10.1128/JCM.03152-15.
    » https://doi.org/10.1128/JCM.03152-15
  • 25
    Bristow CC, Larson E, Anderson LJ, Klausner JD. Cost-effectiveness of HIV and syphilis antenatal screening: a modelling study. Sex Transm Infect. 2016 Feb 26. pii: sextrans-2015−052367. doi: 10.1136/sextrans-2015-052367.
    » https://doi.org/10.1136/sextrans-2015-052367
  • 26
    Swartzendruber A, Steiner RJ, Adler MR, Kamb ML, Newman LM. Introduction of rapid syphilis testing in antenatal care: a systematic review of the impact on HIV and syphilis testing uptake and coverage. Int J Gynaecol Obstet. 2015 Jun;130 Suppl 1:S15−21. doi: 10.1016/j.ijgo.2015.04.008. Epub 2015 Apr 29.
    » https://doi.org/10.1016/j.ijgo.2015.04.008
  • 27
    Flores EC, Lluque ME, Chiappe M, Lino R, Bayer AM. Operations research study to implement HIV and syphilis point-of-care tests and assess client perceptions in a marginalised area of Lima, Peru. Int J STD AIDS 2014 [pii: 0956462414552696, Epub ahead of print].
  • 28
    Fleming E, Oremo J, O’Connor K, Odhiambo A, Ye T, Oswago S, et al. The impact of integration of rapid syphilis testing during routine antenatal services in rural Kenya. J Sex Transm Dis. 2013;2013674584.
  • 29
    Strasser S, Bitarakwate E, Gill M, Hoffman HJ, Musana O, Phiri A, et al. Introduction of rapid syphilis testing within prevention of mother-to-child transmission of HIV programs in Uganda and Zambia: a field acceptability and feasibility study. J Acquir Immune Defic Syndr. 2012;61(3):e40–6.
  • 30
    Mabey DC, Sollis KA, Kelly HA, Benzaken AS, Bitarakwate E, Changalucha J, et al. Point-of-care tests to strengthen health systems and save newborn lives: the case of syphilis. PLoS Med. 2012;9:e1001233.
  • 31
    Delvaux T, Samreth S, Barr-DiChiara M, Seguy N, Guerra K, Ngauv B, et al. Linked response for prevention, care, and treatment of HIV/AIDS, STIs, and reproductive health issues: results after 18 months of implementation in five operational districts in Cambodia. J Acquir Immune Defic Syndr. 2011;57:e47–55.
  • 32
    Pai NP, Kurji J, Singam A, Barick R, Jafari Y, Klein MB, et al. Simultaneous triple point of care testing for HIV, syphilis and hepatitis B virus to prevent mother-to-child transmission in India. Int J STD AIDS. 2012;23:319–24.
  • 33
    Munkhuu B, Liabsuetrakul T, Chongsuvivatwong V, McNeil E, Janchiv R. One-stop service for antenatal syphilis screening and prevention of congenital syphilis in Ulaanbaatar, Mongolia: a cluster randomized trial. Sex Transm Dis. 2009 Nov;36(11):714−20. doi: 10.1097/OLQ.0b013e3181bc0960.
    » https://doi.org/10.1097/OLQ.0b013e3181bc0960

History

  • Received
    18 May 2016
  • Accepted
    24 Oct 2016
  • Publication
    Dec 2016
Organización Panamericana de la Salud Washington - Washington - United States
E-mail: contacto_rpsp@paho.org