regression to the mean and face validity at forbes
i'm loathe to criticize an all-time great minnversity athlete, but i had to chuckle when i heard that forbes had ranked our own kevin mchale as the very best general manager in professional sports. this simply confirms the widely-held opinion that those great courtside seats are wasted on the rich.
in his 11 years as a general manager, mr. mchale has guided our timberwolves to a .539 regular season winning percentage, or about 54 wins per hundred. this represents a significant improvement over his predecessor’s abysmal are-you-sure-that's-not-a-misprint? percentage of .244. mr. mchale thus crushed his 97 competitors on the key "winning improvement" component of forbes' measure:
Forbes.com's first-ever proprietary look at GMs in the NFL, NBA, NHL and MLB grades each GM on two yardsticks. First, there's the performance (regular season winning percentage and postseason wins) during the GM's tenure versus the performance of his predecessor. Second, there's the GM's relative (to the league median) payroll compared with his predecessor's relative payroll.
so, they pretty much just look at changes in winning percentage and the ability to keep the payroll down (this is a business magazine, after all). i like the idea of somehow standardizing by resources or traditions and getting beyond championship trophies in ranking gms, but any sports fan can identify serious flaws in such a system. looking at this as a researcher, i'd probably base my critique on regression to the mean and basic validity concerns.
regression to the mean is the simple tendency for units with extremely high scores at time 1 to decline at time 2 (and for units with extremely low scores at time 1 to rise at time 2). this is based on the idea that measurements are comprised of both a "true score" and an error component. while the true score might be stable, the errors should even out over time and push the overall measure closer to the population mean. in this case, the woofies' winning percentage went from 24% -- c'mon, nobody can be that bad forever -- to somewhere around the population mean of 50%. the concept is also useful in explaining sports voodoo such as the sophomore slump and the sports illustrated jinx. with regard to the forbes example, one can also ask whether mr. mchale should really be judged against the standard of a long-forgotten predecessor. also, shouldn't a move from mediocrity to greatness count for more than a move from hopelessness to mediocrity?
we could challenge the validity of the rankings on many grounds, including their construct validity, content validity, and criterion validity. for me, however, forbes fails even the most cursory and subjective check of face validity -- does it even look like a measure of gm quality to the average sports rube? no, this is akin to devising a ranking of baseball players that puts bombo rivera as the all-time greatest and babe ruth at the 45th percentile.
doesn't the publication have an intern around who might say, "hmmm. mchale as #1, huh? this doesn't look right -- maybe we should throw a couple more variables into the mix." how on earth could a valid measure place the twins' always-doing-more-with-less gm terry ryan in 53rd place? didn't it catch anyone's eye when the top baseball exec on the list (billy beane) was ranked 26th out of 97? oh well, i suppose i wouldn't look to the sporting news for stock advice, so i shouldn't get too upset about forbes stubbing its collective toe on this one. on the other hand, would you really trust a magazine that comes up with such a measure to pick "four growth stocks at a discount" for your portfolio?