When Do We Know How Good an NFL Team Really Is?

By Kevin Meers

Photo: Grantland
Photo: Grantland

Almost everyone who makes predictions about the NFL tends to be overconfident: we think we are more accurate and more precise than we actually are. We like to think we know just how good every team is before the season even starts, when we really know almost nothing. Once week four is in the books, many were ready to declare the Arizona Cardinals a “Title Contender” and the fifth best team in football, and showered them with journalistic gems like “throw out the stats: This team just wins.” After halfway through the season, it’s hard to resist saying, “We know who that team is, and they’re [great/terrible].” This post examines just how confident we should actually be when evaluating NFL teams in the middle of the season.

Football Outsiders calculates every team’s Defense-adjusted Value Over Average (DVOA) after each week of the NFL season, which I use as an estimate of every team’s true skill level after a given week. Once we get more data on every team, we revise our estimates based on the new evidence we get every Sunday. After sixteen games and seventeen weeks, we get our final estimate of each team’s skill level, and that’s about as close as we get to knowing just how good a team really is. But how close are the estimates we make in week one to our final, best estimates at the end of the season?

This graph shows the 95% and 67% prediction intervals for a team’s end-of-season DVOA after a given week of the season (95% and 67% represent how confident we should be in our predictions). As the season goes on, we get a better idea of each team’s true talent level, and the prediction interval shrinks. These prediction intervals approach 0% in change in DVOA – if it ever hit 0%, we would have a perfect prediction.

These prediction intervals are wider than you might expect. For example, we can only predict a team’s final DVOA within +/- 10 percentage points with 95% confidence after fourteen weeks of NFL play. Now 95% confidence is really confident, but even if we relax our confidence to 67%, we still need seven weeks to get within +/- 10 percentage points.

Taking a concrete example from the 2012 season, our 67% prediction interval for Baltimore after week 7 would put them somewhere between the 21st and 7th highest ranked team in the league. So after almost two months of football, we might not know whether a team should be hosting a playoff game or picking in the top ten of next year’s draft. Of course, the more extreme a team’s DVOA, the less they we would expect their ranking to change: we could have been 67% confident that Kansas City would remain the last-ranked team from week seven to week seventeen (which they did).

So let’s make a deal: I won’t say we’re sure how good a team really is at the end of next October if you don’t. I’m over 95% confident the world would be a better place if we better understood our predictive limitations.

About the author

harvardsports

View all posts

5 Comments

  • I think you’re actually understating how little we know. By mid-season, we may know a lot about how a team will finish the season but only because half of the season is over! You really want to compare a team’s DVOA from the first half of the season to the second half to find out when we know how good an NFL team is.

    Here’s something I wrote suggesting we never really know all that much: http://www.footballperspective.com/we-dont-know-anything-and-we-never-will/

  • Chase’s comment is interesting. If we know that Vegas’ lines don’t tend to improve over the course of a season, then it would be interesting to know if FO’s premium picks improve over the course of the season ATS. If they don’t, then the increase in “what we know” that Kevin is writing about is really just reduced volatility in a backwards looking stat because the sample is growing.

  • Chase and Frank — thanks for your responses. It would be cool to see Chase’s methodology applied to FO’s picks. The Vegas spread relies on the wisdom of crowds, and I would read Chase’s study as showing that the wisdom of crowds fails for NFL game prediction (as others have suggested: http://econpapers.repec.org/article/ucpjconrs/doi_3a10.1086_2f658070.htm). So while we, the crowd, may not get better at prediction over the season, it may still be possible to improve our individual predictions. – Kevin

  • Congratulations on successfully year of gymnastics. We are so proud that you won 10 gold and 9 silver medals on National competitions his year! IKgymnastics is the best!

Leave a Reply to Frank DuPont Cancel reply

Your email address will not be published. Required fields are marked *