“Value added” in top medical schools? MCAT/GPA as predictors of USMLE scores

Previous research has established that Step 1 scores correlate moderately with pre-admission factors (MCAT, undergraduate GPA) on the individual level. The following data, from 2011/2012 available via USNWR, demonstrate that this correlation holds at the school-wide level at the very top ranked institutions.

The graphs below show a linear regression of the top 20 ranked schools (minus Pitt, which did not have USMLE data available). The 95% confidence interval of that best-fit line appears dashed. Schools that perform better than predicted based on pre-admission factors, beyond the 95% best fit intervals, appear as green. Schools that do worse appear as red. I’ve labeled these stand-out schools as well as the most extreme points that fall within the prediction zone.

Even at the elite school level, aggregate MCAT and GPA are moderately positively correlated with Step 1 score (R-squared > 0.5). There is no ‘ceiling’ of diminishing returns seen in this model.

There was a significant correlation between pre-admission factors and Step 2 score, but the association is only weakly positive (R-squared >= 0.2).

On Step 1, Baylor stands out as an over-performer, and UCSF stands out as an under-performer compared to peer schools. On Step 2, Baylor and Vanderbilt stand out as over-performers; and UCSF, UCSD, Stanford, and Duke stand out as under-performers compared to their peers. Schools that have the highest admission standards and predictably high Step scores are the ones we traditionally consider as good schools: HMS, WashU, Penn, Hopkins.

There are many possible explanations for deviations from the prediction:

  1. A certain school might have different student characteristics that are not captured by MCAT/GPA statistics but are selected for during the admissions process. For instance, UCSF has a higher percentage of URM and in-state students than its peers. Other relevant factors might include % of science majors (who could have been exposed to preclinical material in college that is not reflected in the MCAT) and proportion of students aiming for less competitive specialties or non-clinical careers (who might value boards scores less highly). These are hard to tease out.
  2. A school requires students to take the boards at a different time during their training (i.e. a minority of schools have students take Step 1 after clinical clerkships).
  3. A school’s formal curriculum is aligned differently with boards-tested content.
  4. A school’s assessment policies (e.g. grading) motivates students to master boards-(ir)relevant content to different extents.
  5. A school permits different amounts of dedicated time for individual boards preparation.

With the exception of #1 above, these considerations give some insight into the effect of the school on boards scores. People often spew the wisdom that boards performance is entirely an individual function — it doesn’t matter what curriculum you have, your scores are solely dependent on how hard you push yourself to study. In this case, I want to make the logical hop that preadmission factors (MCAT and GPA) are an indicator of a person’s aptitude to take standardized tests (e.g. not freaking out and shutting down the day of the test), their study efficiency, and and their study ethic. These are all individual factors that carry forward into medical school. If we can assume that the pool of students that these top schools draw from are not vastly different (#1 above), and this is a big IF, we can conclude that any large deviations from the prediction are indicative of non-individual factors at play, i.e. factors in the school’s control.

If you consider boards scores as a reasonable partial assessment of the learning necessary to become a good doctor, or even as an important factor in achieving your future goals (e.g. get into a competitive residency program), then you can consider this analysis one of the ‘value added’ of a particular medical school education.

Three final caveats I want to point out:

  1. These data are from one year only. There are often fluctuations from year-to-year at each school up to 2-3 points in the absence of any significant changes in curriculum. Unfortunately, longitudinal data is not available to do a more thorough/stable analysis.
  2. These data are probably self-reported from the school to USNWR. Schools could lie, and there is rampant speculation on SDN that schools do manipulate their statistics when presenting them to applicants because publicly available data on Step scores from the NBME are not available. A number of schools have admitted fudging their undergraduate data. For all we know, Baylor could be pulling BS on everyone. But I trust people. =)
  3. This is only the top 20 schools (because I’m lazy). Below that could be a different picture.

MCAT vs Step 1

GPA vs Step 1

MCAT vs Step 2GPA vs Step 2

Advertisements

18 thoughts on ““Value added” in top medical schools? MCAT/GPA as predictors of USMLE scores

  1. Sorry. To clarify: I don’t see the data in the most recent USNWR hard-copy publication. Perhaps it is only available to the online subscribers of the “compass” tool thing?

    • I got them from a premium subscription to the US News compass for best grad schools rankings. I would be very interested to see such an analysis but have no desire to do the data abstraction/entry from USNWR to a spreadsheet for so many schools.

  2. Wow! Thank you! I I have just been searching for information about this topic for ages and yours is the greatest I’ve discovered so far. permanently wanted to inscribe lying on my website amazing like that. Can I implement a part of your post to my site? our site http://www.aboutthemcat.com/ preparation of Medical College Admission Test. here are good informative stuff mcat for study.

  3. I just stumbled upon this blog as I was feeling really low and overwhelmed with my step 1 prep. I grad. 2yrs ago, (IMG) and have finally buckled down to write the step 1. I have 3months which is more than ample time compared to AMGs and yourself. I am using the same materials which you listed, and am doin 8-10 hrs 6days a week. I often second guess myself and abilities but then keep focusing on my end goal.
    Thanks for this blog..USMLE Classes in chennai

  4. Pingback: Influencing thought « Anastomosed

  5. These types of analyses bother me because they have such little value for predicting ability to get into a good residency program, and, ultimately, becoming a good doctor. Step1 score, in itself, is not the outcome that gauges success of medical programs in training students. If that was the case, residencies in top programs will be filled with people with insanely high step 1 scores. But, as everyone knows, that is not the case. Ask any residency director and they will tell you how desirable UCSF trained students are despite their lower step 1 scores. UCSF does not teach exclusively for boards (like many other programs do) and they will tell you that openly. Step 1 score is important but it is only a component of the entire application and clinical performance matters just as much, if not more. Residencies will overlook a lower step 1 for excellent LORs and clinical performance, but not the other way around.

  6. USNWR does not appear to list Step 1 scores for medical schools. Is this a closely guarded secret?

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s