By Kevin Meers
Almost everyone who makes predictions about the NFL tends to be overconfident: we think we are more accurate and more precise than we actually are. We like to think we know just how good every team is before the season even starts, when we really know almost nothing. Once week four is in the books, many were ready to declare the Arizona Cardinals a “Title Contender” and the fifth best team in football, and showered them with journalistic gems like “throw out the stats: This team just wins.” After halfway through the season, it’s hard to resist saying, “We know who that team is, and they’re [great/terrible].” This post examines just how confident we should actually be when evaluating NFL teams in the middle of the season.
Football Outsiders calculates every team’s Defense-adjusted Value Over Average (DVOA) after each week of the NFL season, which I use as an estimate of every team’s true skill level after a given week. Once we get more data on every team, we revise our estimates based on the new evidence we get every Sunday. After sixteen games and seventeen weeks, we get our final estimate of each team’s skill level, and that’s about as close as we get to knowing just how good a team really is. But how close are the estimates we make in week one to our final, best estimates at the end of the season?
This graph shows the 95% and 67% prediction intervals for a team’s end-of-season DVOA after a given week of the season (95% and 67% represent how confident we should be in our predictions). As the season goes on, we get a better idea of each team’s true talent level, and the prediction interval shrinks. These prediction intervals approach 0% in change in DVOA – if it ever hit 0%, we would have a perfect prediction.
These prediction intervals are wider than you might expect. For example, we can only predict a team’s final DVOA within +/- 10 percentage points with 95% confidence after fourteen weeks of NFL play. Now 95% confidence is really confident, but even if we relax our confidence to 67%, we still need seven weeks to get within +/- 10 percentage points.
Taking a concrete example from the 2012 season, our 67% prediction interval for Baltimore after week 7 would put them somewhere between the 21st and 7th highest ranked team in the league. So after almost two months of football, we might not know whether a team should be hosting a playoff game or picking in the top ten of next year’s draft. Of course, the more extreme a team’s DVOA, the less they we would expect their ranking to change: we could have been 67% confident that Kansas City would remain the last-ranked team from week seven to week seventeen (which they did).
So let’s make a deal: I won’t say we’re sure how good a team really is at the end of next October if you don’t. I’m over 95% confident the world would be a better place if we better understood our predictive limitations.