Poking holes in kicking studies

Grantland recently published an article by Football Outsiders' Bill Barnwell that states: "there's no year-to-year consistency with kickers". While Pigskintelligence believes that the volume of field goals kickers attempt on a yearly basis does not constitute a sample size large enough to demonstrate large year-to-year trends, it objects to Barnwell's certainty in proving a negative* as it sees clear methodological errors in the approach used to reach his conclusion:

The point is simple: There's no year-to-year consistency with kickers, so there's no point in paying a premium for an "accurate" kicker.

This isn't necessarily true.  Even if no year-to-year consistency could be statistically proven -- which we'll get to later -- this doesn't mean that a kicker's career field goal percentage, when properly regressed towards the population mean, isn't indicative of his future propensity to make field goals. And if this last point is true, it would mean that the number of seasons Barnwell used in his study was not a large enough sample to overcome the noise from individual season variance.

The problem with Barnwell's study, however, is how the data was chosen.  The selection criteria for inclusion into the dataset was "every kicker since 1999 who attempted 20 field goals or more in each of two consecutive seasons".  This is clear selection bias, as it removes any kicker who was sufficiently poor that he was failed to be brought back for a second season.  And, presumably, kickers who had a poor season who were brought back for a second season (and therefore qualify for inclusion) may have, in the past, better demonstrated their ability to make kicks moreso than those who weren't brought back for another try.  Additionally, some kickers may have been brought back for another try, but failed to reach 20 field goal attempts because of continued poor play.

Here are two charts that demonstrate the effect of this selection bias on the data Barnwell decided to use.  The top chart plots FG% from the first year of the back-to-back 20 FGA seasons versus FG% from the second year, recreating the precise dataset Barnwell used.  The bottom chart, however, makes no such caveat regarding field goal attempts in the second season, instead plotting FG% in the first season (minimum of 20 field goal attempts) versus FG% in the next season (without a minimum requirement).  With this methodology, we are including kickers who kicked sufficiently poorly (or otherwise got hurt, etc.) in season two that they would have been excluded from Barnwell's analysis.

As can be clearly seen, the year-to-year causation doubles (and the correlation increased from 0.085 to 0.119) with this more inclusive and non-biased methodology.  Unfortunately, we still have no way to measure how the kickers who kick so poorly in one season only to never get an opportunity to kick again perform in their next season, since (by definition) they don't get to perform.  However, simply including the kickers 'on a short leash' in the following season satisfactorily shows the bias in Barnwell's methodology.

Simpler proof that the method biases against the inclusion of poor kickers is plainly apparent when you consider that every percentage band that Barnwell segments the data into has a higher field goal percentage in the second season than the NFL in general over that time period.  From 1999 through 2011, NFL kickers have made 80.6% of their field goal attempts, while 81.1% and 82.8% were the lowest and highest second season field goal rate bands. (Note: the overall FG% in the league has increased by roughly half a percentage point per season over this period. However, as the samples are evenly distributed, this doesn't have a strong effect on the point just made.) 

The main flaw in Barnwell's entire argument, however, is that the Broncos are awarding Prater with his contract based upon his previous season...and then seeks to prove that there's little correlation between year and year+1 based on that assumption.  But, presumably, the Broncos gave Prater this contract based on the entire body of work in his career, since that is more information to use when making the decision.  This begs the question: how would the above charts look if the input variable was career kicking percentage?  The answer (using only players who have entered the league since 1999):
With a correlation of 21.3, we are presented with evidence that career kicking percentage -- even when we neglect to normalize against career attempts -- is indeed indicative of future performance. 

Whether or not Matt Prater is worth his contract is something I do not possess the immediate knowledge to answer, but will hope to formulate in the future.  However, Broncos fans should not rush to judgement based upon the evidence presented by Bill Barnwell.

* - Please do not take this as Pigskintelligence asserting that one cannot prove a negative. For more on this fallacy, please see Thinking Tools: You Can Prove A Negative by Steven D. Hales