In my researches, I recently came across Malcolm Gladwell’s New Yorker article from 2001 on Stanley Kaplan, which also discusses the decision by the University of California to abandon its reliance on the SAT and move towards a more “holistic” admissions system.
Statistical analysis concluded that the SAT was “virtually useless as a tool for making admissions decisions.” The four year study of 78,000 freshmen demonstrated that the predictive validity of the student’s high school grade point average (always considered by the ETS as too subjective to be predictive) and the “SAT II” Achievement Tests (measuring what the student actually learned) was far greater than that of the “aptitude” based SAT!
Those findings rocked the academic world, and they should rock the world of business too. Millions of students have taken the SAT, thousands of colleges have used it for years, and tens of millions of dollars have been poured into research and development. So what could have gone wrong? Turns out the folks at ETS - and nearly all universities and colleges - have been making some bad assumptions all along.
The first assumption was that aptitude tests predict future performance more accurately than past performance does. That turns out to be wrong - hard work and a track record of achievement are much greater predictors of success than “aptitude.” When the University of Texas starting admitting all students in the top 10% of their high school classes - regardless of their SAT scores - these students proved to be amazingly successful, often more successful than those with SAT scores 300 or so points higher.
The second assumption was that people wouldn’t try to skew the results – and that, of course, was way off base. Whether it’s for getting into college or getting a job, most people in our world think that it pays to stack the deck in your favor if you possibly can. Millions attend test preparation courses, buy and read prep books, and practice test-taking in order to appear better than they really are, and hey, it works – in fact, it’s a primary reason for SAT score inflation – and it works even better in prepping for the standardized personality tests that we use to qualify and help select candidates. With all this work (call it preparation, or call it cheating) going on, it’s no wonder that nearly 50% of new hires in the US fail within 18 months!
So what’s the takeaway here? Simple: if we want to make better hires, we should work to understand and measure past success as the best predictor of future success. Unlike the SAT or a personality profile, you can’t cram to produce a strong four-year academic record, or a 10-year career track record. We must continue to explore and find better ways to determine how job candidates have actually behaved and performed throughout their careers. Their success – and ours – depends on it.
Comments