In light of the discussion here about the relative strength of different college football conferences, I decided to review the performance of each conference in BCS games since the BCS’s inception in 1998.
For comparing conference strength, this is a reasonable metric because it’s a focused sample where the best teams from different conferences play each other in meaningful games. This is something that happens too rarely in the regular season, where the polling system’s penalty for losses probably discourages strong teams from scheduling strong discretionary (non-conference) opponents. In addition, by focusing on the actual performance of the top teams, it avoids undervaluing the leading teams from conferences with high variance (both good and bad teams), as would happen from an evaluation of all interconference regular season games.
The results were interesting. In order of winning percentage:
Non-BCS conferences 3-1 (.750) (Mountain West 2-0, WAC 1-1)
SEC total 12-5 (.706), champions 8-3, at-large 4-2
Pac 10 total 9-4 (.692), champions 7-4, at-large 2-0
Big East champions 6-5 (.545), no at-large selections
Big 12 total 7-9 (.438), champions 4-7, at-large 3-2
Big 10 total 8-11 (.421), champions 4-7, at-large 4-4
ACC champions 2-9 (.182), no at-large selections
Independent (Notre Dame) 0-3 (.000)
Since at-large entrants from BCS conferences are presumably weaker (on average) than the champions from the same conferences, I felt that including at-large entrants might unfairly lower the apparent performance of their conferences, and I considered looking only at conference champions. However, in three conferences (Big 10, Big 12, and Pac 10) the inclusion of at-large teams actually improved their winning percentage, and in only one conference (SEC) did it slightly lower this value (from .727 to .706).
The strong performance of at-large teams from BCS conferences was notable. These teams had a 13-8 record (.619), in contrast to BCS conference champions, with a record of 31-35 (.470). This isn’t merely due to the “conference champion” pool being pulled down by poorly performing conferences (such as the ACC) while the at-large pool is not weighted down by the forced inclusion of poor teams. If that were true, we wouldn’t expect the at-large teams to boost the winning percentage of 3 of 4 BCS conferences with at-large invitations.
A stronger source of bias against conference champions may be the BCS championship game format, since one of the two supposedly best teams (usually both conference champions) will lose a game they might win against a random BCS opponent. Over this period, 9 BCS conference champions have lost in the BCS championship game.
However, for the purpose of comparing conference strength, this factor does not seem important. The main effect should be reducing the spread in winning percentages, since the better conferences are more likely to have a conference champion invited to (and have an opportunity to lose in) the BCS championship game. Except for small sample size effects, it would not seem to influence the relative ranking of the conferences.
Another possible concern is the non-random pairings of conference champions in the Rose Bowl’s Pac 10 – Big 10 matchup. If one conference champion routinely beats the other in the Rose Bowl, this would not tell us anything about how to place these conferences against other conferences. However, the Rose Bowl formula does not appear to have been much of an influence. Over this period, there have only been five matchups in the Rose Bowl between Pac 10 and Big 10 champions, with the Pac 10 leading the series 3-2. In other years, the matchup was disrupted by one of the conference champions going to the BCS Championship game, or the Rose Bowl itself being the BCS championship game.
In summary, the SEC and Pac 10 seem to be the strongest conferences, with a wide gap between them and everyone else. Given the performance of the Big East, the only other BCS conference with a winning record, the absence of any at-large invitations suggests that it’s undervalued. The extraordinary performance of non-BCS conferences — with a higher winning percentage than any BCS conference (albeit with a small sample size) — suggests that the BCS is too reluctant to extend invitations to these teams.
I believe my math is correct (and my knowledge of which teams are in which conferences), but if you want to check the numbers, here is my source material with raw data:
Pingback: Horned Frog « Grumpy Ant
Sorry but the sample size is too small to be even close to significant, and the selection of these teams is influenced by factors included “will they make us more money” and self-selection bias of the pollsters (if you assume the SEC is strong, and you rank them accordingly, then when they beat each other you punish them less, because well they are strong, oh look they are strong!).
David K, I think you may be taking my post too seriously – I don’t intend this as a definitive analysis of conference strength, but an interesting perspective that showed stronger trends than I had expected. I agree, the results aren’t statistically significant, but I’m not trying to prove anything here — the answer makes no possible difference to anything, and I was careful to only say that the results “suggest” certain conclusions, not prove them.
But to give a more detailed statistical commentary:
If we take the null hypothesis to be that conference membership is irrelevant to the outcome of a game, then we can model the games as a series of coin flips, and use the binomial distribution with a probability of 0.5. We’ll use a two-tailed result, since what matters is how far a conference falls from a .500 record, and (for example) a 2-9 and 9-2 record are equally surprising.
The ACC has a 2-9 record, and the probability of a conference having 0, 1, 2, 9, 10 or 11 wins over 11 games is 6.5 percent.
The SEC has a 12-5 record, and the probability of a conference having 0, 1, 2, 3, 4, 5, 12, 13, 14, 15, 16 or 17 wins over 17 games is 14.3 percent.
The Pac 10 has a 9-4 record, and the probability of a conference having 0, 1, 2, 3, 4, 9, 10, 11, 12 or 13 wins over 13 games is 26.7 percent.
If our working hypothesis is “the SEC and Pac 10 are strong, while the ACC is weak,” then we may be justified in using a one-tailed distribution, and cutting the P values in half – to 3, 7 and 13 percent, respectively. Again not statistically significant (the ACC is below 5, but we’re looking at multiple conferences, so a lower P value per test is appropriate) … but certainly interesting.
“if you assume the SEC is strong, and you rank them accordingly, then when they beat each other you punish them less, because well they are strong, oh look they are strong!”
I’m looking at how well conference teams play in interconference BCS games — not how many conference teams are selected for at-large invitations — to rank the conferences. So if the SEC is overvalued, the effect would be to lower the SEC’s winning percentage due to the inclusion of overvalued teams. Are you saying that the SEC is an even better conference than their BCS winning percentage of .706 would suggest?
“… the selection of these teams is influenced by factors included “will they make us more money” and self-selection bias of the pollsters …”
The BCS conference champions are not discretionary, so these sources of bias only apply to the at-large invitations, and the at-large teams seem to perform well.
Granted, I don’t believe that the BCS selection process identifies the best teams from each conference (using either the ‘conference champion’ or at-large methods). But as with the small sample size, it’s what I have to play with … so take it with a grain of salt, but the problem with 1-A college football is that there is so little meaningful data from each season (what does Florida d. FIU really tell you?) and it’s interesting to see what angles can be found/