Previous research has established that Step 1 scores correlate moderately with pre-admission factors (MCAT, undergraduate GPA) on the individual level. The following data, from 2011/2012 available via USNWR, demonstrate that this correlation holds at the school-wide level at the very top ranked institutions.
The graphs below show a linear regression of the top 20 ranked schools (minus Pitt, which did not have USMLE data available). The 95% confidence interval of that best-fit line appears dashed. Schools that perform better than predicted based on pre-admission factors, beyond the 95% best fit intervals, appear as green. Schools that do worse appear as red. I’ve labeled these stand-out schools as well as the most extreme points that fall within the prediction zone.
Even at the elite school level, aggregate MCAT and GPA are moderately positively correlated with Step 1 score (R-squared > 0.5). There is no ‘ceiling’ of diminishing returns seen in this model.
There was a significant correlation between pre-admission factors and Step 2 score, but the association is only weakly positive (R-squared >= 0.2).
On Step 1, Baylor stands out as an over-performer, and UCSF stands out as an under-performer compared to peer schools. On Step 2, Baylor and Vanderbilt stand out as over-performers; and UCSF, UCSD, Stanford, and Duke stand out as under-performers compared to their peers. Schools that have the highest admission standards and predictably high Step scores are the ones we traditionally consider as good schools: HMS, WashU, Penn, Hopkins.
There are many possible explanations for deviations from the prediction:
- A certain school might have different student characteristics that are not captured by MCAT/GPA statistics but are selected for during the admissions process. For instance, UCSF has a higher percentage of URM and in-state students than its peers. Other relevant factors might include % of science majors (who could have been exposed to preclinical material in college that is not reflected in the MCAT) and proportion of students aiming for less competitive specialties or non-clinical careers (who might value boards scores less highly). These are hard to tease out.
- A school requires students to take the boards at a different time during their training (i.e. a minority of schools have students take Step 1 after clinical clerkships).
- A school’s formal curriculum is aligned differently with boards-tested content.
- A school’s assessment policies (e.g. grading) motivates students to master boards-(ir)relevant content to different extents.
- A school permits different amounts of dedicated time for individual boards preparation.
With the exception of #1 above, these considerations give some insight into the effect of the school on boards scores. People often spew the wisdom that boards performance is entirely an individual function — it doesn’t matter what curriculum you have, your scores are solely dependent on how hard you push yourself to study. In this case, I want to make the logical hop that preadmission factors (MCAT and GPA) are an indicator of a person’s aptitude to take standardized tests (e.g. not freaking out and shutting down the day of the test), their study efficiency, and and their study ethic. These are all individual factors that carry forward into medical school. If we can assume that the pool of students that these top schools draw from are not vastly different (#1 above), and this is a big IF, we can conclude that any large deviations from the prediction are indicative of non-individual factors at play, i.e. factors in the school’s control.
If you consider boards scores as a reasonable partial assessment of the learning necessary to become a good doctor, or even as an important factor in achieving your future goals (e.g. get into a competitive residency program), then you can consider this analysis one of the ‘value added’ of a particular medical school education.
Three final caveats I want to point out:
- These data are from one year only. There are often fluctuations from year-to-year at each school up to 2-3 points in the absence of any significant changes in curriculum. Unfortunately, longitudinal data is not available to do a more thorough/stable analysis.
- These data are probably self-reported from the school to USNWR. Schools could lie, and there is rampant speculation on SDN that schools do manipulate their statistics when presenting them to applicants because publicly available data on Step scores from the NBME are not available. A number of schools have admitted fudging their undergraduate data. For all we know, Baylor could be pulling BS on everyone. But I trust people. =)
- This is only the top 20 schools (because I’m lazy). Below that could be a different picture.