Abstract:

Objective:

To assess the assumption of ‘equity’ of Mexico’s resident-selection assessment tool, the Examen Nacional para Aspirantes a Residencias Médicas (ENARM).

Materials and methods:

Official ENARM-2016 and -2017 databases were analyzed. Differences in the absolute number of correct answers (multivariable linear regression) and the number of applicants reaching their specialty minimum score (SMS) per test day (odds ratio [OR]) were calculated. Applicants affected by test-day inequity were estimated.

Results:

There were 36 114 applicants in 2016, and 38 380 in 2017. In 2016, day-2 applicants had significantly higher scores and more reached the SMS than on days 1-3-4 (OR 1.55), and 5 (OR 3.8); 3 565 non-passing applicants were affected by inequity (equivalent to 44.64% of those selected). In 2017, day-1 and -2 applicants had significantly higher scores and more reached the SMS than on days 3-4 (OR 1.85), and 5 (OR 4.04); 3,155 non-passing applicants were affected by inequity (37.2% of those selected).

Conclusion:

Analysis of official ENARM databases does not support the official attribution of equity, suggesting the test should be redesigned.

Keywords:
education; graduate medical; specialty; residency and internship; personnel selection; academic test score; psychometrics

Resumen_

Objetivo:

Evaluar el atributo de “equidad” asignado al Examen Nacional para Aspirantes a Residencias Médicas (ENARM).

Material y métodos:

Se analizaron las bases de datos oficiales del ENARM 2016 y 2017. Se compararon las diferencias inter-día de respuestas correctas (regresión linear multivariable) y de sustentantes que alcanzaron el puntaje mínimo de su especialidad (PME) (razón de momios [RM]). Se estimó a los afectados por la inequidad.

Resultados:

Hubo 36 114 sustentantes en 2016 y 38 380 en 2017. Los días 2 (ENARM-2016) y 1-2 (ENARM-2017) registraron puntajes significativamente más altos, y más sustentantes alcanzaron el PME que en los días 1-3-4 (RM 1.55) y 5 (RM 3.8) en 2016, y los días 3-4 (RM 1.85) y 5 (RM 4.04) en 2017. Se estimó que cuatro de cada diez sustentantes que aprobaron el ENARM no lo hubieran hecho si el examen fuera equitativo.

Conclusión:

Los resultados sugieren que el atributo de equidad del ENARM está en duda.

Palabras clave:
educación de postgrado en medicina; especialización; internado y residencia; selección de personal; rendimiento académico; psicometría

Introduction

The availability and quality of facilities and human resources in healthcare is of utmost importance,¹1. Makary MA, Daniel M. Medical error-the third leading cause of death in the US. BMJ. 2016;353:i2139. https://doi.org/10.1136/bmj.i2139
https://doi.org/10.1136/bmj.i2139... with physicians comprising one of the main human resources. Most medical doctors (MDs) seek to continue their training within a specialty. However, not all MDs pursuing specialty training can enter an official program.

Most Western countries have regulatory agencies that are responsible for evaluating, selecting, and placing MDs to continue training. These agencies use multiple high-stakes processes to select the best candidates.²2. Roberts C, Khanna P, Rigby L, Bartle E, Llewellyn A, Gustavs J, et al. Utility of selection methods for specialist medical training: a BEME (best evidence medical education) systematic review: BEME guide no. 45. Med Teach. 2018;40(1):3-19. https://doi.org/10.1080/0142159X.2017.1367375
https://doi.org/10.1080/0142159X.2017.13... ^,33. mrcpuk.org [internet] London: Membership of the Royal Colleges of Physicians of the United Kingdom, 2018 [ cited 2018 Jun 4]. Available from:Available from:https://www.mrcpuk.org/
https://www.mrcpuk.org/... ^,44. Centre National de Gestion [internet]. Paris: CNG, 2018 . Épreuves Classantes Nationales (ECN) [cited 2018 Jun 4]. Available from:Available from:https://www.cng.sante.fr/concours-examens/epreuves-classantes-nationales-ecn
https://www.cng.sante.fr/concours-examen... For example, the selection process in the United States involves approving different components of the United States Medical Licensing Examination, interviewing, and participating in a national matching program. In France, students present a national ranking examination and choose the specialty they wish to pursue according to their rank.⁴4. Centre National de Gestion [internet]. Paris: CNG, 2018 . Épreuves Classantes Nationales (ECN) [cited 2018 Jun 4]. Available from:Available from:https://www.cng.sante.fr/concours-examens/epreuves-classantes-nationales-ecn
https://www.cng.sante.fr/concours-examen... In Mexico, MDs must first register on the registry of the Examen Nacional para Aspirantes a Residencias Médicas (National Exam for Applicants to Medical Residencies, ENARM).⁵5. Comisión Interinstitucional para la Formación de Recursos Humanos para la Salud [internet]. Mexico City: CIFRHS, 2018 . Examen Nacional para Aspirantes a Residencias Médicas [cited 2018 Jun 4]. Available from:Available from:http://cifrhs.salud.gob.mx/
http://cifrhs.salud.gob.mx/... This requires registration of an individual’s ID, the specialty for which they are competing (one of 27 options), and the preferred date to present the exam (one of five consecutive days on which the test is available yearly, for 2016 and 2017). An online registration platform is enabled at a predetermined day and time, and closes around 10 days later. As registration and date-selection occurs on a first-come, first-served basis, 36 000+ applicants typically register during the first 24 hours. The ENARM is a norm-referenced test,⁶6. Norcini J, Friedman Ben-David N. Concepts in assessment. In: Dent JA, Harden RM (eds). A practical guide for medical teachers. 4th ed. China: Churchill Livingstone, 2013:285-91.^,77. Comité de Posgrado y Educación ContinuaXL ENARM: Metodología proceso de selección. Mexico City: CIFRHS , 2016 [cited 2018 Jun 4]. Available from:Available from:https://cifrhs.salud.gob.mx/site1/enarm/docs/2016/E40_met_proceso_seleccion_2016.pdf
https://cifrhs.salud.gob.mx/site1/enarm/... ^,88. Comité de Posgrado y Educación Continua. XLI ENARM: Metodología proceso de selección. Mexico City: CIFRHS , 2017 [cited 2018 Jun 4]. Available from:Available from:http://cifrhs.salud.gob.mx/site1/enarm/docs/2017/E41_met_proceso_seleccion_2017.pdf
http://cifrhs.salud.gob.mx/site1/enarm/d... and is the assessment tool used in Mexico to select those who will enter specialty training. There are approximately 8 000 official training positions available nationwide. These positions vary by specialty, and are determined by the certified training programs of the healthcare institutions that provide them. Applicants for each specialty are ranked from highest to lowest according to their total ENARM score. Ranked applicants receive a ‘pass’ certificate until the quota is met according to that specialty’s available positions, and present it to the institution of their choice to apply for an appointment.

The quality and performance of such selection and placement processes cannot be overemphasized, as they have a beneficial or harmful impact on the candidate, patients, healthcare systems, and ultimately, the public health of a given country. The Comité de Posgrado y Educación Continua (Committee of Postgraduate and Continuous Education [CPEC]), the agency in charge of the ENARM, confers the attributes of “equitable, transparent, objective, and valid” to the ENARM.⁹9. Comisión Interinstitucional para la Formación de Recursos Humanos para la Salud. Convocatoria XLExamen Nacional para Aspirantes a Residencias Médicas . Mexico City: CIFRHS , 2016 [cited 2018 Jun 4]. Available from:Available from:http://cifrhs.salud.gob.mx/site1/enarm/docs/2016/E40_convo_2016.pdf
http://cifrhs.salud.gob.mx/site1/enarm/d... ^,1010. Comisión Interinstitucional para la Formación de Recursos Humanos para la Salud. Convocatoria XLI Examen Nacional para Aspirantes a Residencias Médicas . Mexico City: CIFRHS , 2017 [cited 2018 Jun 4]. Available from:Available from:http://cifrhs.salud.gob.mx/site1/enarm/docs/2017/E41_convo_2017.pdf
http://cifrhs.salud.gob.mx/site1/enarm/d... The CPEC declares that the five different test forms (one for each test day, each comprising an exclusive set of items) used for the ENARM in a particular year are equivalent because they are created with “the same objectivity, quality, balance, and academic level”.⁹9. Comisión Interinstitucional para la Formación de Recursos Humanos para la Salud. Convocatoria XLExamen Nacional para Aspirantes a Residencias Médicas . Mexico City: CIFRHS , 2016 [cited 2018 Jun 4]. Available from:Available from:http://cifrhs.salud.gob.mx/site1/enarm/docs/2016/E40_convo_2016.pdf
http://cifrhs.salud.gob.mx/site1/enarm/d... ^,1010. Comisión Interinstitucional para la Formación de Recursos Humanos para la Salud. Convocatoria XLI Examen Nacional para Aspirantes a Residencias Médicas . Mexico City: CIFRHS , 2017 [cited 2018 Jun 4]. Available from:Available from:http://cifrhs.salud.gob.mx/site1/enarm/docs/2017/E41_convo_2017.pdf
http://cifrhs.salud.gob.mx/site1/enarm/d... However, our group recently reported that the test development processes and the theoretical basis under which the ENARM (2016 and 2017) was built do not support the stated attributes of validity, reliability, and equity.¹¹11. Barajas-Ochoa A, Ramos-Remus C. Equidad, validez y confiabililidad del Examen Nacional para Aspirantes a Residencias Medicas (ENARM): oportunidades para mejorar. Salud Publica Mex. 2017;36:501-2. https://doi.org/10.21149/8769
https://doi.org/10.21149/8769... This contrasts with important advances that have resulted in a change in established standards for educational testing,¹²12. American Educational Research Association, American Psychological Association, National Council on Measurement in Education. Standards for educational and psychological testing. Washington, DC: American Educational Research Association, 2014.^,1313. Lane S, Raymond MR, Haladyna TM (eds). Handbook of test development. 2nd ed. New York: Routledge, 2016.^,1414. Bennett RE, von Davier M. Advancing human assessment: The methodological, psychological and policy contributions of ETS [eBook]. Springer, 2017 . https://doi.org/10.1007/978-3-319-58689-2
https://doi.org/10.1007/978-3-319-58689-... and supports the statement that, for some systems of educational assessment in medicine, tradition weighs more than evidence-based methods and best practices.¹⁵15. Schuwirt LWT, van der Vleuten CPM. General overview of the theories used in assessment: AMEE Guide No. 57. Med Teach . 2011;33(10):783-97. https://doi.org/10.3109/0142159X.2011.611022
https://doi.org/10.3109/0142159X.2011.61...

The CPEC does not precisely define ‘equity’, but the sense of this attribute is close to that of ‘fairness’ (preferred in the English language and in the literature related to education), as defined by Lane and colleagues,¹⁶16. Lane S, Raymond MR, Haladyna TM, Downing SM. Test development process. In: Lane S, Raymond MR, Haladyna TM (eds). Handbook of test development. 2nd ed. New York: Routledge , 2016:3-18. “fairness in testing is achieved if a given test score has the same meaning for all examinees and is not substantially influenced by factors not relevant to the examinee’s performance.”

This study aimed to assess whether the ENARM met the attribute of ‘equity’ (i.e. fairness) as ascribed by the CPEC, by analyzing the databases of the results for all applicants in 2016 and 2017.

Materials and methods

This study analyzed the official databases of ENARM 2016 and 2017. The study took place in Guadalajara, Mexico from February to May of 2018.

ENARM characteristics

The ENARM test is available yearly and measures knowledge in general medicine. Five test forms are created each year, each comprising 450 multiple-choice single-best answer items; no item is used in more than one test form. Each form is used nationwide for one of the five consecutive days on which applicants can present the test. All forms comprise the same number of items per area of knowledge (specialty/subspecialty), with an approximate item distribution of 37.5% internal medicine, 25% pediatrics, 22% gynecology-obstetrics, and 15% surgery. Furthermore, all test forms include 25% low-difficulty (n=113), 50% medium-difficulty (n=225), and 25% high-difficulty items (n=112).¹⁷17. Dirección General de Calidad y Educación en Salud. Departamento de Transparencia de la Secretaría de Salud: Reply to request No. 0001200055817 on public information regarding the development of the ENARM. Mexico City: DGCES, 2017 . Available from:https://drive.google.com/file/d/0B4Qe4F_sYmX0bHM0dHlpWFBObnM/view
https://drive.google.com/file/d/0B4Qe4F_... The processes of item development and test assembly have been the same since at least 2010.¹⁸18. Comisión Interinstitucional para la Formación de Recursos Humanos para la Salud. ENARM: Características y evolución a su formato electrónico. Mexico City: CIFRHS , 2012 [cited 2018 Jun 4]. Available from:Available from:http://cifrhs.salud.gob.mx/descargas/pdf/enarm_caracteristicas_evolucion.pdf
http://cifrhs.salud.gob.mx/descargas/pdf... These processes are performed by “at least eight expert professors, whom are selected according to specific profiles to participate in different steps of item-development such as creation, analysis and calibration, quality-control, decision of difficulty category, and final approval”.⁹9. Comisión Interinstitucional para la Formación de Recursos Humanos para la Salud. Convocatoria XLExamen Nacional para Aspirantes a Residencias Médicas . Mexico City: CIFRHS , 2016 [cited 2018 Jun 4]. Available from:Available from:http://cifrhs.salud.gob.mx/site1/enarm/docs/2016/E40_convo_2016.pdf
http://cifrhs.salud.gob.mx/site1/enarm/d... ^,1010. Comisión Interinstitucional para la Formación de Recursos Humanos para la Salud. Convocatoria XLI Examen Nacional para Aspirantes a Residencias Médicas . Mexico City: CIFRHS , 2017 [cited 2018 Jun 4]. Available from:Available from:http://cifrhs.salud.gob.mx/site1/enarm/docs/2017/E41_convo_2017.pdf
http://cifrhs.salud.gob.mx/site1/enarm/d... These experts assign items to the three difficulty categories, based on “assessing the number of correct answers of five expert clinicians and their experience,” without testing the items on the target population¹⁷17. Dirección General de Calidad y Educación en Salud. Departamento de Transparencia de la Secretaría de Salud: Reply to request No. 0001200055817 on public information regarding the development of the ENARM. Mexico City: DGCES, 2017 . Available from:https://drive.google.com/file/d/0B4Qe4F_sYmX0bHM0dHlpWFBObnM/view
https://drive.google.com/file/d/0B4Qe4F_... or using psychometric theories.

Applicants receive their ENARM score report immediately after finishing the test. The score report includes the number of correct answers (NoCA) per item difficulty-category, NoCA per area of knowledge, absolute number of correct answers (ANoCA: 0-450), and the total score (0-100). The total score is calculated by dividing the ANoCA by the total number of items (i.e., ANoCA/450). After all tests have been completed, applicants’ registers are clustered by nationality (Mexican or foreign medical graduate) and chosen specialty (the same 27 direct-entry specialties in 2016 and 2017), and ranked from best to worst according to total scores. Applicants are then selected until the quota for each specialty is met. When a tie in the total score occurs, the tiebreak process considers (in successive order) applicants’ scores in internal medicine, pediatrics, gynecology-obstetrics, surgery, and the level of difficulty of correctly-answered items.⁷7. Comité de Posgrado y Educación ContinuaXL ENARM: Metodología proceso de selección. Mexico City: CIFRHS , 2016 [cited 2018 Jun 4]. Available from:Available from:https://cifrhs.salud.gob.mx/site1/enarm/docs/2016/E40_met_proceso_seleccion_2016.pdf
https://cifrhs.salud.gob.mx/site1/enarm/... ^,88. Comité de Posgrado y Educación Continua. XLI ENARM: Metodología proceso de selección. Mexico City: CIFRHS , 2017 [cited 2018 Jun 4]. Available from:Available from:http://cifrhs.salud.gob.mx/site1/enarm/docs/2017/E41_met_proceso_seleccion_2017.pdf
http://cifrhs.salud.gob.mx/site1/enarm/d...

Characteristics of the ENARM 2016 and 2017 databases

The anonymized complete official ENARM databases for 2016 and 2017 were obtained from the Dirección General de Calidad y Educación en Salud (General Direction of Quality and Education in Health; DGCES), a federal institution that participates in ENARM development, through the Plataforma Nacional de Transparencia (National Transparency Platform). This platform is supported by the Instituto Nacional de Transparencia, Acceso a la Información Pública y Protección de Datos Personales (National Institute of Transparency, Access to Information and Protection of Personal Data). In Mexico, the federal law on transparency and access to public information (Ley General de Transparencia y Acceso a la Información Pública) allows citizens to obtain data from publicly funded institutions, provided the information is not considered confidential and does not affect the privacy of third persons.

The databases included all applicants of the 2016 and 2017 ENARMs. Available data per applicant included age, sex, medical school, nationality (Mexican, foreign), chosen specialty (the same 27 specialties for 2016 and 2017), test date (five days for each year), overall rank in the test, NoCA in low- (0-113), medium- (0-225), and high-difficulty (0-112) items, and total score (0-100).

Statistical analysis

This ecological study of the ENARM databases performed the same analysis separately for 2016 and 2017. The primary end-point was ‘equity,’ which was assumed if no significant inter-day differences in ANoCA existed. If equity was present, test forms were assumed to have the same difficulty. If lack of equity was suggested by the primary end-point, the secondary end-point was to estimate the number of applicants dubiously classified as not passing.

For the primary end-point, an exploratory analysis was performed to assess inter-day differences on the average ANoCA. To assess this, the variable ‘ANoCA’ (0-450) was created by adding applicants’ NoCA for low-, medium-, and high-difficulty items. The average ANoCA per day was calculated and compared using one-way ANOVA. If inter-day differences in the average ANoCA suggested a lack of equity, further analysis with estimation of effect size was performed.

The effect size for lack of equity was assessed by two methods. For the first method, the categorical variable ‘applicant reached the specialty minimum score’ (ARSMS) was created, as data for pass/not pass status was not available. Specialty minimum scores (SMS) were obtained from reports published by the Comisión Interinstitucional para la Formación de Recursos Humanos para la Salud (CIFRHS),¹⁹19. Comisión Interinstitucional para la Formación de Recursos Humanos para la Salud. XL Examen Nacional para Aspirantes a Residencias Médicas: Puntajes máximos y mínimos por especialidad. Mexico City: CIFRHS , 2016 [cited 2018 Jun 4]. Available from:Available from:http://cifrhs.salud.gob.mx/site1/enarm/docs/2016/E40_puntajes_max_min_2016.pdf
http://cifrhs.salud.gob.mx/site1/enarm/d... ^,2020. Comisión Interinstitucional para la Formación de Recursos Humanos para la Salud. XLI Examen Nacional para Aspirantes a Residencias Médicas : Puntajes máximos y mínimos por especialidad. Mexico City: CIFRHS , 2017 [cited 2018 Jun 4]. Available from:Available from:http://cifrhs.salud.gob.mx/site1/enarm/docs/2017 /E41_puntajes_max_min_2017 .pdf
http://cifrhs.salud.gob.mx/site1/enarm/d... the federal agency that reports ENARM outcomes. Each SMS was converted into its correspondent ANoCA, which represented the minimum ANoCA required to obtain a pass certificate for that specialty in that year (mANoCA). Next, whether an applicant’s ANoCA was equal or higher to their selected specialty mANoCA was coded as yes/no. The odds ratios (OR) for the proportion of ARSMS by specialty per day were calculated. The days with similar proportions of ARSMS were grouped, and the OR between the day/group with the higher proportion of ARSMS and the other days/groups were calculated.

The second effect size method involved constructing a multivariable linear regression model that calculated the inter-day differences in ANoCA and in NoCA per item’s ascribed difficulty. By themselves, these values represent an appropriate measure of the magnitude of the effect. The model included available variables that could explain the differences: sex, age (potential surrogate for years since graduation and the number of test attempts), nationality, and chosen specialty (because of auto-selection). A sensitivity analysis that also included the medical school ‘public/private’ status was performed.

A third effect size definition was used to assess the secondary end-point. The number of applicants dubiously classified as not passing (i.e., number of affected applicants) was estimated using the coefficients obtained in the regression model. Days on which the regression model showed similar results (differences of ≤2 items) or overlap of the confidence intervals (CI) for the ANoCA were grouped. A reference group (R) was defined when ≥3 days met the former condition, or when two days met the condition and their average ANoCA was similar to that year’s average ANoCA. The differences between the rounded averages of R and days i and j (‘i’ for day[s] with higher scores than R, and ‘j’ for day[s] with lower scores), and between days i and j were used to calculate the ANoCA range where affected applicants lay. For each specialty, the SMS¹⁹19. Comisión Interinstitucional para la Formación de Recursos Humanos para la Salud. XL Examen Nacional para Aspirantes a Residencias Médicas: Puntajes máximos y mínimos por especialidad. Mexico City: CIFRHS , 2016 [cited 2018 Jun 4]. Available from:Available from:http://cifrhs.salud.gob.mx/site1/enarm/docs/2016/E40_puntajes_max_min_2016.pdf
http://cifrhs.salud.gob.mx/site1/enarm/d... ^,2020. Comisión Interinstitucional para la Formación de Recursos Humanos para la Salud. XLI Examen Nacional para Aspirantes a Residencias Médicas : Puntajes máximos y mínimos por especialidad. Mexico City: CIFRHS , 2017 [cited 2018 Jun 4]. Available from:Available from:http://cifrhs.salud.gob.mx/site1/enarm/docs/2017 /E41_puntajes_max_min_2017 .pdf
http://cifrhs.salud.gob.mx/site1/enarm/d... was expressed as the mANoCA. To calculate the lower limits of each range, differences between i and R (ΔiR), i and j (Δij), and R and j (ΔRj) were subtracted from each specialty’s mANoCA. For the upper limits, 1 was subtracted from the mANoCA. Therefore, the limits of the ranges for each specialty were ([mANoCA - ΔiR] to [mANoCA − 1]), ([mANoCA - Δij] to [mANoCA - 1]), and ([mANoCA − ΔRj] to [mANoCA − 1]). The sum of the number of applicants per specialty that lay within the ranges ([mANoCA - ΔiR] to [mANoCA - 1]) and ([mANoCA - Δij] to [mANoCA − 1]) reflected the number of applicants affected by the lack of equity in different ENARM test forms; that is, those that could have obtained a pass score if they had completed the test on the easiest day.

Confidence intervals (95%) and effect size were used to define significance, as statistically significant p-values were assumed likely to appear because of the population size.²¹21. Ferguson CJ. An effect size primer: A guide for clinicians and researchers. Prof Psychol Res Pract. 2009:40:532-8. https://doi.org/10.1037/a0015808
https://doi.org/10.1037/a0015808... ^,2222. Sullivan GM, Feinn R. Using effect size-or why the P value is not enough. J Grad Med Educ. 2012;4:279-82. https://doi.org/10.4300/JGME-D-12-00156.1
https://doi.org/10.4300/JGME-D-12-00156....

This study used an audit approach with publicly-available, anonymized, official databases. Informed consent and institutional review board approval was not required.

Results

In the 2016 ENARM, 36 114 applicants from 112 medical schools applied for 27 specialties, mainly general surgery, internal medicine, gynecology-obstetrics, pediatrics, anesthesiology, and family medicine.Table I shows applicants’ demographics, NoCA by item difficulty, and ANoCA. Applicants’ age and sex distribution was similar across all ENARM days, except for day 5, in which applicants were on average 2 years older. Significant differences were found (according to the 95%CI) in scores for the low-, medium-, and high-difficulty questions; day 2 had higher scores (easier test form) and day 5 had lower scores (more difficult test form). These differences were also reflected in the ANoCA. Univariate linear regression showed that the ANoCA decreased significantly for each year of increase in age (−3.26 [95%CI −3.37 to −3.15], being female (−5.45 [95%CI −6.26 to −4.64]), being non-Mexican (−5.12 [95%CI −7.57 to −2.66]), and having studied in a private medical school (−3.36 [95%CI −4.3 to −2.42]).

In the 2017 ENARM, there were 6% more applicants and three more medical schools, but the same number of specialties. For most specialties, the number of applicants varied slightly. Table II shows demographics, NoCA by item difficulty, and the ANoCA. Applicants’ age and sex distribution was similar across all days. Significant differences were found (according to the 95%CI) in the scores for low-, medium-, and high-difficulty items. The NoCA and ANoCA showed patterns similar to 2016. Days 1 and 2 showed higher scores (easier test forms) and day 5 showed lower scores (more difficult test form). The univariate linear regression showed that ANoCA decreased significantly in similar way to the previous year: for each year of increase in age (−2.99 [95%CI −3.1 to −2.88]), being female (−5.29 [95%CI −4.47 to −6.1]), being non-Mexican (−4.97 [−7.29 to −2.65]), and having studied in a private medical school (−3.29 [95%CI −4.23 to −2.34]).

The identified inter-day differences influenced the number of applicants who achieved the SMS for the specialty for which they were competing. Table III highlights the number and proportion of ARSMS (overall and per day) for the 12 specialties with the greatest number of applicants (data on all specialties available upon request). Grouping days with similar proportions of ARSMS showed that applicants on day 2 were more likely to achieve their SMS when compared with days 1, 3, 4 (OR 1.55 [95%CI: 1.46 to 1.64]), and 5 (OR 3.8 [95%CI: 3.47 to 4.16]) for year 2016. The inter-day differences also had an effect for the 2017 ENARM. After grouping days with similar proportions of ARSMS, applicants on days 1 and 2 were more likely to reach their SMS when compared with days 3, 4 (OR 1.85 [95%CI 1.75 to 1.94]), and 5 (OR 4.04 [95%CI 3.72 to 4.38]).

The inter-day differences among item difficulty categories and the ANoCA were adjusted for age, gender, selected-specialty, and nationality using multivariable linear regression. The 2016 ENARM showed that days 1, 3, and 4 had similar coefficients (differences in the ANoCA), whereas day 2 showed an advantageous coefficient (e.g., 11.66 higher ANoCA than day 1) and day 5 showed a disadvantageous coefficient (16.18 lower ANoCA than day 1) (table IV). The 2017 ENARM showed days 3 and 4 had similar inter-day coefficients, days 1 and 2 had similar but higher (advantaged) coefficients, and day 5 had a lower coefficient (disadvantaged). Furthermore, the inter-day coefficients within the item difficulty categories also significantly differed (table IV).

Since the significant inter-day variability described above precludes the assumption of equity for the ENARM in 2016 and 2017, we estimated its impact as the number of applicants that could have been dubiously classified as not passing (secondary end-point). Table V shows that for 2016, i (day 2) presupposed a disadvantage for 2 070 (9.5%) applicants of R (days 1, 3, and 4) and for 1 495 (21.7%) applicants of j (day 5). In consequence, 3 565 (12.49%) applicants were affected by the lack of equity in the test forms. In other words, this last figure represents 44.64% of the 7 986 applicants that received a pass certificate that year.²³23. Comisión Interinstitucional para la Formación de Recursos Humanos para la Salud. XL Examen Nacional para Aspirantes a Residencias Médicas : Plazas para médicos seleccionados. Mexico City: CIFRHS , 2016 [cited 2018 Jun 4]. Available from:Available from:http://cifrhs.salud.gob.mx/site1/enarm/docs/2016/E40_plazas_mex_lugares_ext_2016.pdf
http://cifrhs.salud.gob.mx/site1/enarm/d... ^,2424. Comisión Interinstitucional para la Formación de Recursos Humanos para la Salud. XL Examen Nacional para Aspirantes a Residencias Médicas : Folios de médicos seleccionados - Médicos con categoría mexicana. Mexico City: CIFRHS , 2016 [cited 2018 Jun 4]. Available from:Available from:http://cifrhs.salud.gob.mx/site1/enarm/docs/2016/E40_folio_sel_mex_2016.pdf
http://cifrhs.salud.gob.mx/site1/enarm/d... ^,2525. Comisión Interinstitucional para la Formación de Recursos Humanos para la Salud. XLExamen Nacional para Aspirantes a Residencias Médicas : Folios de médicos seleccionados - Médicos con categoría extranjera. Mexico City: CIFRHS , 2016 [cited 2018 Jun 4]. Available from:Available from:http://cifrhs.salud.gob.mx/site1/enarm/docs/2016/E40_folio_sel_ext_2016.pdf
http://cifrhs.salud.gob.mx/site1/enarm/d...

Table V shows the estimate of affected applicants in 2017, where i (days 1 and 2) presupposed a disadvantage for 1 644 (13.6%) applicants of R (days 3 and 4) and 1 511 (19.79%) applicants of j (day 5), indicating 3 155 (13.64%) applicants were affected. This represents 37.2% of the 8 480 applicants that received a pass certificate in 2017.²⁶26. Comisión Interinstitucional para la Formación de Recursos Humanos para la Salud. XLI Examen Nacional para Aspirantes a Residencias Médicas : Plazas para médicos seleccionados. Mexico City: CIFRHS , 2017 [cited 2018 Jun 4]. Available from:Available from:http://cifrhs.salud.gob.mx/site1/enarm/docs/2017 /E41_plazas_mex_lugares_ext_2017 .pdf
http://cifrhs.salud.gob.mx/site1/enarm/d... ^,2727. Comisión Interinstitucional para la Formación de Recursos Humanos para la Salud. XLI Examen Nacional para Aspirantes a Residencias Médicas : Folios de médicos seleccionados - Médicos con categoría mexicana. Mexico City: CIFRHS , 2017 [cited 2018 Jun 4]. Available from:Available from:http://cifrhs.salud.gob.mx/site1/enarm/docs/2017 /E41_folio_sel_mex_2017 .pdf
http://cifrhs.salud.gob.mx/site1/enarm/d... ^,2828. Comisión Interinstitucional para la Formación de Recursos Humanos para la Salud. XLI Examen Nacional para Aspirantes a Residencias Médicas : Folios de médicos seleccionados - Médicos con categoría extranjera. Mexico City: CIFRHS , 2017 [cited 2018 Jun 4]. Available from:Available from:http://cifrhs.salud.gob.mx/site1/enarm/docs/2017 /E41_folio_sel_ext_2017 .pdf
http://cifrhs.salud.gob.mx/site1/enarm/d...

A sensitivity analysis for the multivariable linear regression model that also adjusted to the status of ‘public/private’ medical school in addition to the other variables was performed (data not shown, available upon request), but excluded approximately 1 200 and 1 400 applicants (2016 and 2017, respectively) from foreign medical schools for which the “public/private” status could not be identified. It yielded similar statistically significant results and identical grouping of the days, but a higher estimate of applicants affected by inequity. Considering these findings, the authors decided to opt for the more discrete estimate.

Additional data on the ANoCA for each specialty for both years is available upon request.

Discussion

The effort required for an MD to meet the selection requirements is vast. The selection process is based on different constructs. A construct of particular importance is fairness. This ecological study evaluated the equity (i.e. ‘fairness’)¹⁶16. Lane S, Raymond MR, Haladyna TM, Downing SM. Test development process. In: Lane S, Raymond MR, Haladyna TM (eds). Handbook of test development. 2nd ed. New York: Routledge , 2016:3-18. of the ENARM over two consecutive years.

Our first important finding was that there were significant inter-day differences in scores for items classified as low-, medium-, and high-difficulty, and thus the ANoCA. These differences, although significant, may seem small. However, a difference of just one correct item has an impact on an applicant’s pass or no-pass status. Consequently, the variable ‘day on which an applicant presented the test’ in both years gave applicants an advantage or disadvantage. For 2016, day 2 applicants were 55% more likely to reach their SMS than applicants on days 1, 3, and 4, and almost four times more likely than on day 5. For 2017, day 1 and 2 applicants were 85% more likely to reach their SMS than applicants on days 3 and 4, and four times more likely than on day 5. Therefore, the stated premise of equity was not sustained.

The methods used to create the different test forms may explain these inter-day differences. The agency that creates the ENARM does not perform item analysis with established psychometric theories, such as classical test theory or item response theory. Instead, experts classify an item’s difficulty into three broad categories defined a priori, without performing field testing on the target population, whereas item discrimination is not calculated (data on applicants’ answers are erased after the test).¹⁷17. Dirección General de Calidad y Educación en Salud. Departamento de Transparencia de la Secretaría de Salud: Reply to request No. 0001200055817 on public information regarding the development of the ENARM. Mexico City: DGCES, 2017 . Available from:https://drive.google.com/file/d/0B4Qe4F_sYmX0bHM0dHlpWFBObnM/view
https://drive.google.com/file/d/0B4Qe4F_... ^,2929. Dirección General de Calidad y Educación en Salud. Departamento de Transparencia de la Secretaría de Salud: Reply to request No. 0001200357817 on the petition of the official databases of the 2016 and 2017 ENARM. Mexico City: DGCES , 2017 . Available from:https://drive.google.com/open?id=1SkzaK20MXXShKTVzK42Alc4jbLWfB8kg
https://drive.google.com/open?id=1SkzaK2... The only consideration in assembling different test forms is including an exclusive set of items for each test form that accounts for the same number of items per area of knowledge and includes 25% low-difficulty, 50% medium-difficulty, and 25% high-difficulty items. Consequently, the problem is in the definition of difficulty. For example, the low-difficulty category has in itself a range of difficulty values; one test form may have low-difficulty items that are, on average, located at the top of this range, whereas another test form may contain items that are located, on average, at the bottom of the range. Furthermore, for a test where the score is computed as the sum of N dichotomously scored items, such as the ENARM, the test’s mean score (i.e., average ANoCA) directly relates to the average difficulty of item scores.³⁰30. Moses T. A review of developments and applications in item analysis. In: Bennett RE, von Davier M (eds). Advancing human assessment: The methodological, psychological and policy contributions of ETS [eBook]. Springer Open, 2017 :19-46. https://doi.org/10.1007/978-3-319-58689-2
https://doi.org/10.1007/978-3-319-58689-...

If research-supported methods in educational assessment already in use had been used for the ENARM, the attribution of equity would not have been compromised. Best practice involves performing item tryout and item analysis to assess item difficulty and discrimination, with the purpose (among others) of calibrating items to design alternate test forms as if they were on the same test score scale.³¹31. Haladyna TM. Item analysis for selected-response test items. In: Lane S, Raymond MR, Haladyna TM (eds). Handbook of test development . 2nd ed. New York : Routledge , 2016:392-409.Standard 5.12 of the Educational Standards for Educational and Psychological Testing ¹²12. American Educational Research Association, American Psychological Association, National Council on Measurement in Education. Standards for educational and psychological testing. Washington, DC: American Educational Research Association, 2014. states that “a clear rationale and supporting evidence should be provided for any claim that scale score earned on alternate forms of a test may be used interchangeably.” However, in the ENARM, although ‘equivalence among different test forms’ is stated, no supporting evidence is provided and no test-equating methods are used.¹⁷17. Dirección General de Calidad y Educación en Salud. Departamento de Transparencia de la Secretaría de Salud: Reply to request No. 0001200055817 on public information regarding the development of the ENARM. Mexico City: DGCES, 2017 . Available from:https://drive.google.com/file/d/0B4Qe4F_sYmX0bHM0dHlpWFBObnM/view
https://drive.google.com/file/d/0B4Qe4F_...

Our second finding concerns estimating the impact on applicants due to differences according to the day they took the exam. Since the ENARM is a selection exam for specialties and each specialty has a predetermined number of positions, it is expected that applicants with high scores will not be impacted. The expected impact would be on those scoring closely below and above each SMS. Our estimation showed that around four out of ten applicants that received a pass certificate might have not received a pass if the test forms were equitable. The impact might go beyond these estimates; that is, on applicants’ professional development, quality of healthcare provided by the training institutions, and ultimately, population health.

Regarding the variables associated with obtaining a lower ANoCA, we can only speculate. We have no explanation as to why female applicants obtained lower scores, but older applicants may most likely represent MDs that graduated years ago and may not be as updated as their recently-graduated counterparts. Regarding the private status of a medical school, a possible explanation is that acceptance into some private schools may not be as competitive and selective as that of public schools.

The strengths of this study that lend weight to the conclusions are that the official ENARM 2016 and 2017 databases were analyzed and the attribution of equity was assessed using different approaches. Nonetheless, several limitations should be noted. First, this was an ecological study with inherent limitations of that design. Second, differences in the average ANoCA among test forms do not necessarily prove a lack of equity, but do suggest that further review is needed.³²32. Zieky MJ. Developing fair tests. In: Lane S, Raymond MR, Haladyna TM (eds). Handbook of test development . 2nd ed. New York: Routledge , 2016:81-99. The ENARM is a norm-referenced test by which applicants are selected based on a quota. The inter-day differences in the proportions of ARSMS strongly suggest that the different test forms jeopardized the possibility that a better (or at least equally) prepared number of applicants entered specialty training. Third, the official databases did not include data on the correct/incorrect status for each answered item, which precluded calculating the standard error of measurement of each test form to assess its reliability.³³33. Tighe J, McManus IC, Dewhurst NG, Chis L, Mucklow J. The standard error of measurement is a more appropriate measure of quality for postgraduate medical assessments that is reliability: an analysis of MRCP(UK) examinations. BMC Med Educ. 2010;10:40. https://doi.org/10.1186/1472-6920-10-40
https://doi.org/10.1186/1472-6920-10-40... In addition, we could not use the method proposed by Raykov and Marcoulides³⁴34. Raykov T, Marcoulides GA. Introduction to psychometric theory. Classical test theory. New York, NY: Routledge, 2011:115-36. to assess whether the different ENARM test forms were consistent with the model of parallel tests in the classical test theory framework, as the way applicants’ scores are compared fits this model (i.e., no estimation of random error of measurement and no use of test-equating methods). Fourth, other relevant variables that might have influenced test scores were not available for analysis, such as applicants’ number of test attempts and the grade point average obtained during medical school. Fifth, it is assumed that applicants distributed in a more or less randomized manner among test days. This assumption is supported by the registration method. Test-registration occurs on a first-come, first-served basis; most applicants try to register immediately, which leads the web-server to saturate, delaying successful registration for applicants to a random fashion.

In many science-related areas, there is a concern that a gap between research and practice exists, even with the growing volume and quality of evidence along with the technological and organizational improvements in information management and synthesis.³⁵35. Green LW, Ottoson JM, Garcia C, Hiatt RA. Diffusion theory and knowledge dissemination, utilization, and integration in public health. Annu Rev Public Health. 2009;30:151-74. https://doi.org/10.1146/annurev.publhealth.031308.100049
https://doi.org/10.1146/annurev.publheal... The field of educational assessment seems to be no exception, as attested by the example of the ENARM. Furthermore, although this study aimed to assess equity among ENARM test forms for 2016 and 2017, it has to be considered that the exam methods have been the same since at least the 2010 ENARM.¹⁸18. Comisión Interinstitucional para la Formación de Recursos Humanos para la Salud. ENARM: Características y evolución a su formato electrónico. Mexico City: CIFRHS , 2012 [cited 2018 Jun 4]. Available from:Available from:http://cifrhs.salud.gob.mx/descargas/pdf/enarm_caracteristicas_evolucion.pdf
http://cifrhs.salud.gob.mx/descargas/pdf... Changes newly introduced for ENARM 2018 are likely to decrease the lack of equity by using the same test form for all applicants competing for the same specialty.³⁶36. Comisión Interinstitucional para la Formación de Recursos Humanos para la Salud. Convocatoria XLIIExamen Nacional para Aspirantes a Residencias Médicas . Mexico City: CIFRHS , 2018 [cited 2018 Jun 4]. Available from:Available from:http://cifrhs.salud.gob.mx/site1/enarm/docs/2018 /E42_convo_2018 .pdf
http://cifrhs.salud.gob.mx/site1/enarm/d... This may be the best immediate approach to improve the equity of the test. However, improvements in many other aspects of the ENARM are needed in the long term. Examples include: a) increasing transparency by creating a publicly-available test’s development technical report to provide evidence of validity, reliability and fairness; b) improving item-writing, as applicants’ opinions suggest that poorly-written items introduce construct-irrelevant variation (e.g., those written in English); c) improving item development by using item tryout and item analysis; d) assessing the effect on test scores of the nine different item arrangements used for each test form,¹⁸18. Comisión Interinstitucional para la Formación de Recursos Humanos para la Salud. ENARM: Características y evolución a su formato electrónico. Mexico City: CIFRHS , 2012 [cited 2018 Jun 4]. Available from:Available from:http://cifrhs.salud.gob.mx/descargas/pdf/enarm_caracteristicas_evolucion.pdf
http://cifrhs.salud.gob.mx/descargas/pdf... as the location of items on a test form can affect item statistics, particularly in testing programs that report immediately or shortly after test administration;¹⁶16. Lane S, Raymond MR, Haladyna TM, Downing SM. Test development process. In: Lane S, Raymond MR, Haladyna TM (eds). Handbook of test development. 2nd ed. New York: Routledge , 2016:3-18. e) addressing sources of systematic errors (applicants’ opinions suggest that a 2017’s test form included an item that required applicants to identify a clinical sign in an image, but the image was veiled); and f) estimating the standard error of measurement, which requires the non-deletion of answered items after the test.

Evidence points toward the need to use a robust, evidence-based test-development process on the ENARM if the test’s stated objectives and characteristics are to be achieved. Estimates suggest that average items for high-stakes tests may be valued around 300 USD, with some items valued above 1 000 USD (e.g., those that undergo extensive statistical analysis or that need complex development processes such as simulations); overall, an item bank with a few thousand items may be valued around 1 000 000 USD.³⁷37. Timothy JM. Web-based item development and banking. In: Lane S, Raymond MR, Haladyna TM (eds). Handbook of test development . 2nd ed. New York: Routledge , 2016: 241-56 The revenues obtained from the ENARM registration fees are around 110 000 000 MXP per year (approximately 6 000 000 USD). With these funds, it would be possible to redesign the test to improve its quality.

In summary, the ENARM has a paramount and multidimensional impact on Mexico’s healthcare system and on applicants wishing to undergo specialty training. However, the analysis of the official ENARM databases does not support the attribution of equity. For this reason, it is necessary to redesign the test using evidence-based test-development processes that support the fairness, validity, and reliability of the ENARM.

References

Publication Dates

History