Football Research Appendix

By Max Mulitz

I’ll keep this updated as I make new posts with a quick summary of the big idea of each post and links to posts by category:

Mission Statement:

What is football analytics?/Blueprint

Game Strategy Research:

The Role of Analytics in Game Planning

First down probability and when to accept/decline penalties.

Running out of shotgun on 3rd & 1 is more efficient than running from under center. Though under center plays have a higher average conversion percentage because teams run more often from under center.

Game theory can help tendency research with respect to the order in which teams make decisions for a given play

Solving the problem of the Two Point Conversion up 7 late in the game

When to call Timeout on defense at the end of the First Half

Individual Player Value:

A significant majority of NFL Teams will have to give at least 6 offensive linemen significant snaps each season. Teams will commonly give significant snaps to as many as 8 different offensive linemen per season.

Run Stops Added is our foundational metric for understanding how individual players contribute to run defense

Expected Points forms the basis of our fundamental metric for understanding how much value a receiver provided when he was targeted.

Projecting Quarterback success in the NFL Draft

Valuing Free Agent Receivers using Production and Age

How to use expected points to estimate Wide Receiver value

How valuable is Earl Thomas? He’s probably worth 2-3 points per game to the Seahawks

A mental model for evaluating risk in free agency

A Running backs’ expected success rate varies wildly depending on the backs role. Expected success rate is far more stable year to year than actual performance, meaning role is more consistent than rushing performance for running backs.

Injury Study: There’s about a 40-50% chance a given player will miss time with an injury in a given season. It’s likely some players are injury prone and have a 70%+ chance of being injured an some players are particularly resilient to injury and only have about a 25-30% chance of being injured in a given season.

The most valuable kicker you could reasonably expect to have is worth about 0.1 wins above average per season, which is  worth about 0.66% of the Salary Cap (about 1.1 Million dollars/season)

An extra 1% of the cap spent in a given year correlates with 0.15 more wins

NFL Draft/Scouting

Calculating Sparq for Edge Rushers

Arm Length almost always matters more than height

Jumping tests at the combine are a valid measure of leg power once you adjust for weight.

Measuring Grip Strength to gain an advantage selecting linemen

Using decision trees to form an inclusive model of which players are worth scouting

Indicators of Team Strengths and Weaknesses: 

How are different team efficiency statistics correlated with winning? Each extra sack per hundred snaps is worth about an extra quarter of a win. An increase of a yard per attempt passing or a decrease of a yard per attempt allowed each correlate with about 1.6 extra wins.

Passing efficiency is at least a little over twice as important as rushing efficiency, both offensively and defensively. Team passing efficiency predicts team offense quality far better than rushing efficiency.

Point differential is highly correlated with wins. Every 36 points of added point differential is worth an expected win

How to handle being in Field Goal attempt range when tied or trailing by 1-2 points at the end of a game

 

Advertisements

2017 NFL Draft Modeling: Quarterback Edition

by Max Mulitz

I built a simple model for projecting drafted FBS Quarterbacks to the NFL using only Age-Adjusted Collegiate Statistics. The model is fit on drafted players from 2005-2013 and uses Peak 3 seasons of Wins Over Average as the dependent variable.  Age-Adjusted collegiate Production predicted NFL Performance about the same as draft position in the sample, with a correlation of 0.36 compared to 0.35 for draft position. The model gave the following quarterbacks first round grades from the 2005-2016 drafts

Year Round Pick  Name Position Draft Age
2015 1 2 Marcus Mariota QB 21
2010 1 1 Sam Bradford QB 22
2014 1 22 Johnny Manziel QB 21
2011 1 1 Cam Newton QB 22
2005 1 1 Alex Smith QB 21
2016 1 1 Jared Goff QB 21
2014 1 32 Teddy Bridgewater QB 21
2009 1 5 Mark Sanchez QB 22
2012 3 75 Russell Wilson QB 23
2012 1 1 Andrew Luck QB 22
2009 1 1 Matthew Stafford QB 21
2012 1 2 Robert Griffin QB 22
2007 1 1 JaMarcus Russell QB 22
2005 1 24 Aaron Rodgers QB 21

It’s a fairly elite list. Mostly it’s young players who were dominant in college and went on to be drafted in Round 1, often first overall. Of the 14 players in the list, 7 have been to Pro Bowls.JaMarcus Russell and Johnny Manziel both busted badly, though arguably not due to lack of talent, and Mark Sanchez has settled in as more of a backup…The jury is probably still out on Goff, Bridgewater  and Mariota.

Here’s a list of the players taken in Round 1 from FBS schools that the model didn’t like as first round players.

Year Round Pick  Name Position Draft Age
2015 1 1 Jameis Winston QB 21
2008 1 3 Matt Ryan QB 23
2014 1 3 Blake Bortles QB 22
2006 1 3 Vince Young QB 23
2012 1 8 Ryan Tannehill QB 24
2011 1 8 Jake Locker QB 23
2011 1 10 Blaine Gabbert QB 21
2006 1 10 Matt Leinart QB 23
2006 1 11 Jay Cutler QB 23
2011 1 12 Christian Ponder QB 23
2013 1 16 EJ Manuel QB 23
2009 1 17 Josh Freeman QB 21
2012 1 22 Brandon Weeden QB 28
2007 1 22 Brady Quinn QB 22
2010 1 25 Tim Tebow QB 23
2005 1 25 Jason Campbell QB 23
2016 1 26 Paxton Lynch QB 22

Matt Ryan is elite. Vince Young, Jay Cutler, and Ryan Tannehill have had varying degrees of success. Jameis Winston looks like he is going to have a nice career.  The jury is still out on Paxton Lynch. Most of the players on this list ended up as busts.

This year the model considers three players First Round talents: Deshaun Watson, Patrick Mahomes and Mitch Trubisky. There have been multiple  players the model considered first round quarterbacks four times before.

In 2005, Alex Smith and Aaron Rodgers were both first round players per the model, Smith went first overall and Rodgers went 24th.In 2009, Matt Stafford went first overall and Mark Sanchez went fifth.  In 2012, the model liked Andrew Luck, Robert Griffin, and Russell Wilson in the first round. All three were successful, though Griffin has struggled since his knee injuries. Then in 2014, Johnny Manziel went 22nd and Bridgewater went ten picks later at 32. Bridgewater is currently dealing with injuries and Manziel is out of the league due to off-field issues.It does not appear as if the first player drafted in a given year has a significant advantage on the other qualifying players in the class, nor does it appear clear that draft position is a significant predictor  for qualifying players.

Right now, Mahomes appears undervalued. CBS has him 70th overall while ranking Mitch Trubisky 12th. Per the Jimmy Johnson Trade Chart, the 70th pick is only 1/5th as costly in terms of draft capital as the 12th pick. Given the data, it’s hard to argue Trubisky is twice as likely to become a top quarterback as Mahomes, let alone five times as likely. There simply aren’t prospects with Mahomes’ age adjusted production who fall out of the first round (other than Russell Wilson.)

The model also considers Deshone Kizer, Brad Kaaya, Nate Peterman, and Davis Webb draftable prospects based on their age and production, while Chad Kelly and Jerod Evans just miss the cut.

The Fundamental Metric For Valuing Individual Run Defenders: Introducing Run Stops Added

by Max Mulitz

Theory of Run Stops Added:

A run stop is a play where a defender  makes a tackle on a running play that is not efficient for the offense. Rushing Success Rate correlates with winning better than Yards Per Carry, so the measure of a run defenders value can be measured by his ability to prevent successful runs. By determining the value of a run stop and measuring the players’ ability to generate run stops, we can solve for the players’ value as a run defender.

How Valuable is a Run Stop?  A run stop is worth about 1.3 points. Runs that are unsuccessful have an expected point value of about 0.75 expected points, while unsuccessful runs have an expected points value of about -0.55 expected points, so a run stop is worth about 1.3 expected points on average. Of course, a 5 yard loss on 3rd & 1 at the Goal Line has a greater impact than a tackle for no gain on 1st & 10 at midfield, but in aggregate a run stop is worth about 1.3 points.

How does an individual player contribute to his team making Run Stops? Different positions have different base run stop percentages. A replacement level 4-3 defensive end may only make a run stop on 6% of his run plays while a replacement level MLB may make a run stop on closer to 7.5% of his plays. Players in different schemes will have different opportunity levels, but a middle linebacker on the Raiders is has a more similar job to a middle linebacker on the Chiefs in terms of run defense than he will have to a defensive tackle on his own team.

Modeling Individual Run Defense: A players’ contribution to stopping the run can be estimated as the percentage of run plays where he generates a run stop minus the expected run stop percentage for his position, multiplied by the number of snaps they played in a given season, as seen in the formula below:

Run Stops Added=(Run Stop %-Position Replacement Level Run Stop %)*Number of Snaps

For instance, a Defensive End who played 200 Run Defense Snaps and had a 8% Run Stop Rate would be responsible for 4 extra run stops above a 6% replacement level player. At 1.3 points per run stop, the players contribution could be estimated at 5.2 points (about 0.14 wins.)

 Run stops added is a reasonably consistent statistic from year to year, which is a necessity for it to be used as an indicator of defensive player value.  For example, for Defensive Ends, Run Stops Added for players with over 200 run snaps in both years were strongly correlated from 2014 to 2015 at R=.66.

Limitations of Run Stops Added:

  1. Certain players may improve their teams’ run defense statistics by making their teammates’ job easier. If a star defensive tackle demands a double team, thereby allowing the teams middle linebacker to often go unblocked and make more plays in the run game, the tackles contribution will reflect in the linebackers statistics, but not his own. Dominant defensive linemen/run stopping LBs are usually able to create for themselves often enough that we are unlikely to have many “No Stats All Stars” in primary run defense, though the stats obviously only tell most of the story. In general, the players who are considered top run defenders are able to translate that into a high run stop percentage.
  2. Scheme obviously matters. Some teams schemes are going to enable their weak side linebackers to make relatively more tackles in the run game than other teams irrespective of individual player ability.
  3. Run Stop Percentage is not appropriate for secondary run defenders, as it does not reward tackles in the open field (after the run has already “succeeded” in terms of improving first down probability.)

An Anecdote:

Damon Harrison is a good check on the calibration of our model. Harrison is perhaps the best run stopper in the NFL, though he brings nothing to the table as a pass rusher. In 2015, Harrison’s Run Stops Added would have valued him at almost exactly one win above replacement. With a 143 Million Dollar Salary Cap in 2016, this would imply an open market value of 9.53 million. In fact, Harrison signed with the Giants for an average of 9.25 million per season.

Why Run Stops Added instead of other metrics:

Tackles for loss are useful, but because tackles for loss are a subset of run stops, there is always a smaller number of TFLs than Run Stops in a given year, making them less stable over a sample of only a few hundred snaps.

Percentage Share of Run Game Tackles/Total Tackles can be useful, but doesn’t tell us if a player is making a lot of tackles because he’s making plays in the run game or because he’s chasing players down at the second level after they beat his teammates. An average run defender surrounded by replacement level teammates is going to have the tackle totals of a stud, for instance.

Expected Points Added is useful, but may overweight plays in certain situations. For instance, a stop on 4th & Goal at the 1 could be more valuable than five run stops over the course of a game in the middle of the field. In the long run, the player who made five stops in the run game has shown more of a repeatable skill set than the player who happened to make only one play at exactly the right time.

Conclusion:

Run Stop Percentage and Run Stops Added are the fundamental metrics of a defenders’ contribution in the run game. A players Run Stops Added can be converted to  expected points added and therefore wins added, which serves as a starting point to value an individual player’s contribution as a run defender. Run Stops Added can also be used for advanced scouting purposes to help discern who the opponents’ best and worst run defenders are.

Valuing Notable Free Agent Receivers Using Production and Age

A players’ previous performance statistics, age, health, film grade, team needs, and projected scheme fit all factor into determining how much money a team is willing to offer a free agent. Generally, a starting point for determining a players value is taking a players’ past production and extrapolating it into the future depending on the players age. This is by no means a complete analysis for any player. Sometimes very talented players lack production in one scheme and go on to produce at a high level in a different scheme where they are a better fit. Wes Welker and Emmanuel Sanders are just two examples of receivers who saw massive increases in production after changing teams. That said, on field production is a good place to start and a players’ past production is the most predictive single variable for future performance.

Below I list some notable Free Agent Receivers along with their production over the past three years and a single value representing the past three years using a 5/4/3 weighting structure to weight more recent years more heavily. Also included is a players’ free market value if they continue to play at the level of their Weighted Wins Added based on a 166 million dollar projected cap next year. I also included Spotrac’s estimate of a players free market value as a reference. Further included is the probability the player declines due to age based on Adam Harstad’s model of receiver career mortality. I do not include age related decline for players who will be 25 or younger next year or 35 or older, as the model is not well trained on these extremes.

Untitled.png

 

-Alshon Jeffery leads the pack. His value would be somewhat higher at 13.5 million if you prorate his stats to account for missed games.

-Kenny Britt has quietly been very good. My WR metric does not attempt to account for team quality, so it is impressive that Britt was 12th in the league last season in Wins Added in spite of the fact that the Rams were dead last in Offensive Passing NEP and second to last in passing yards per game. 29 next year, Britt is not at great risk of age related decline. Spotrac is low on Britt’s value and it is possible he finds a soft market in Free Agency because he played on such a poor offense. But the fact remains that very good things happened for the Rams when they targeted Britt, and it looks like he could be one of the more underrated players in free agency. For whatever it’s worth, PFF had Britt as the best receiver at running crossing routes in the league last season.

-Terrelle Pryor would projected  in the 9.5 million a year range if you only used the 2016 season. At age 28 he’s not likely to suffer severe age related decline. It may or may not be a concern that he only has one year of production.

-Kenny Stills and Desean Jackson have produced almost identically over the past three years, though Jackson could be starting to get old. Stills and Jackson had almost identical advanced stats last season, with Jackson receiving 18% of the Redskins targets but 30% of their Air Yards for 15.5 Air Yards/Target. Stills receiving 17% of the Dolphins Targets but a whole 31% of their Air Yards and averaged 15.9 Air Yards Per Target. Jackson has some risk of decline in the next couple years, but as recently as last season he was among the fastest players in the league. Both players are low-usage, high-efficiency deep threats so they may only make sense for a team that is looking for that type of player.

 

 

The Role of Analytics in Game Planning

 

by Max Mulitz

Data generated scouting reports famously involve many pages of data. A team does not gain an advantage by summarizing the most data. The goal is to get the most useful information to the coaches. What specific decisions are coaches looking to the data to help them make?

For example, say a defensive coordinator is deciding if he wants to play base defense or more nickel defense when his upcoming opponent is in 12 personnel. The coordinator expects his base defense to be above average against the run but below average against the pass but for his nickel defense to be above average against the pass but below average against the run. To make this decision the coordinator wants to know the probability the opponent will run or pass in normal down and distances from 12 personnel depending on the defensive personnel.

The problem is, this is a very specific question. A team may have only faced nickel defense from 12 personnel in normal situations a handful of times in a season, so there might not be a meaningful sample. The primary way to alleviate this is by using base rates. If NFL pass 60% of the time from 12 personnel in normal situations against base defenses but only 40% of the time against nickel defense that is useful information. If the base rates are very similar that is also useful information. How often teams run from 12 personnel in general is not that useful for this question, since we have good reason to believe defensive personnel will affect run/pass choices.

The other way to expand our sample is to look for similar situations that would fit our theory. Here that means looking for how the opponents’ run/pass balances changes depending on the defensive personnel in normal situations. Some teams run quite a bit more often vs. nickel than vs. base and some teams don’t change their offense very much based on defensive personnel. Knowledge is theory laden.

As important as what data you include is what data is excluded from the analysis. How often teams run from 12 personnel isn’t very useful because it doesn’t account for the fact that we are considering using a lighter defense that may cause teams to run more.

Also if you don’t constrain yourself to normal situations the sample will be contaminated with examples of teams playing nickel defense on 3rd & long which will inflate the probability of a pass.

If you don’t expand your sample size by looking at classes of situations a team can be fooled into thinking a team is 80% likely to run in a given situation just because they have done it 4/5 times, even if the base rate is just 33% runs.

Interpreting data for game planning is the domain of coaches. A football scientist’s job is to make sure coaches are seeing the correct data that helps them game plan and are not weighed down with irrelevant or misleading data.

Measuring Grip Strength to Gain an Advantage when Selecting Offensive Linemen

By Max Mulitz

At the 1:30:50 mark of this clinic,  Alex Gibbs, the godfather of the modern zone blocking scheme, explains that hand strength is one of the most important athletic traits for an offensive lineman. He says, “this is their tool right here, this is what they have to have, they have to have hand strength…” Gibbs goes on explain that hand strength is a more accurate indicator of functional strength than the bench press.

Former Pro Bowl Lineman LeCharles Bentley, who runs a premier offensive line school, also believes grip strength is very important.

If measurable grip strength is a key trait for offensive linemen, then any attempt to project offensive lineman athleticism using data without grip strength is going to be considerably less accurate than a model that includes a measure of grip strength. Unfortunately, none of the NFL combine tests directly measure this trait, so there’s no way to know the exact nature of the relationship between measured grip strength and offensive line performance.

NFL teams should collect grip strength measurements on any offensive linemen that comes through the franchise to help examine how grip strength affects performance. If some of the great minds of offensive line play are correct, then a team will be able to get an advantage on their offensive line by using data to study the relationship between grip strength and performance.

A Mental Model For Valuing Players In Free Agency

by Max Mulitz

Adam Harstad has a fantastic piece explaining that players do not  gradually decline as they age, they continue performing at more or less their average level of output until they hit a wall. This probably isn’t precisely true. For instance we know speed declines as players age, but they also improve by gaining experience and technical skill. Larry Fitzgerald is an example of a player whose style of play has changed as he’s gotten older and lost some of his speed, but who has maintained a very high level of production. Still, given that our model of player aging should be some balance of year to year decline and risk of falling off a cliff, Adam’s data is a strong case that the risk of “death” has a far larger impact.

When teams consider how to structure a players’ contract, they must consider the possibility the player maintains his approximate level of production through the life of the contract as well as the possibility the player stops producing after each year of the contract. Every year of a players’ career carries some risk that the player will, either due to injury, loss of athleticism, or off field issues, no longer be a viable NFL player. A teams’ risk in each year is simply how costly it would be to the franchise in terms of percentage of the salary cap paid over the life of the contract if the player stopped being productive in that year. Good contracts (for the franchise) balance saving salary cap space with protection in the event of a players’ drop off. There is some utility in estimating a players’ risk, but becoming overconfident in a specific individuals’ durability creates fragility.

This mental model is limited to established starters going on their 2nd or 3rd contract. Players who have not yet been given the opportunity to play significant snaps do not have an established baseline of production and are not likely to get relatively large guaranteed contracts.

 

How Valuable is Earl Thomas to the Seahawks’ Defense?

by Max Mulitz

Over the past couple seasons, numerous articles have been written explaining why Earl Thomas is the most important player in the Seahawks defense. Basically, the entire scheme of the Seahawks defense relies on Earl Thomas being able to cover the deep middle third of the defense on the majority of the snaps.

Passes to the deep middle of the field are the most dangerous and efficient passes in the game, so Thomas is charged with defending the most important area of the field on a down to down basis. As Numberfire points out, the Seahawks deep passing defense went from allowing only about one expected point per game with Thomas to about three per game without him, for a net increase of two points per game allowed. Football Outsiders data also evidences that the Seahawks defense of the deep middle of the field has gone from elite to very poor with the loss of Thomas.

Based on Numberfire’s analysis the Seahawks pass defense was about 2.5 points a game worse without Earl Thomas. This would value Thomas at a little over a win per season.

Thomas’ salary of  $10 Million per year (6% of the Cap) values him at about .9 win a year, though Numberfire’s valuation of him would put his value in the 12-14 million/year range, which seems reasonable.

What is Football Analytics/Blueprint for integrating Football Science into an NFL Franchise

by Max Mulitz

 

Analytics is the process of applying the scientific method to football. Scientific knowledge is knowledge that makes testable predictions and is hard to vary. All knowledge is conjectural and comes from within us through creativity to solve problems, not to summarize data. The scientific method is an endless iterative process of moving closer to the truth of the world, especially when studying complex systems such as NFL football.

Untitled.png

A successful Football Science department requires the following characteristics and priorities:

  1. Be centralized in such a way that the department can communicate with all areas of the franchise, because all areas of an NFL franchise overlap in their needs and uses for data.
  2. Integrate the football knowledge and theories of Scouts, General Management, and Coaches into testable theories that can enhance the total football knowledge of the organization. This framework can be used to create comprehensive theory of roster construction, projecting college players to the NFL, and understanding how different players contribute to winning games All models and knowledge created by the Football Science department should be rooted in the organizations’ foundational philosophy of football.
  3. Work with the coaches to create a system of game management that is agreed upon and understood by the Head Coach and play callers and can be continuously updated and improved as new situations are recognized and analyzed and more knowledge is created regarding the topics of game and clock management.
  4. Utilize the coaching staff’s knowledge of play calling and how coaches think to enhance pre-scouting opponents and game planning with the best, most relevant, and most useful data. The role of analytics in game preparation is to generate information that aids the coach in forming his gameplan, not to summarize the data as many ways as possible.
  5. Always be in the process of revising and building on foundational models and systems to continuously learn about each area of football the department interacts with. Perpetual growth and improvement in every area Football Science touches is a necessity.

A Simple Box Score Metric for Valuing Wide Receivers

One of the projects I’ve been working on is creating a model of player value by position based on converting box score based metrics to a single number of points added. This model is by no means the be all end all of player value, but it would provide an evidence-based starting point for estimating the value of players across positions. In a previous post, I explained why measures of a players contribution should always be in terms of Expected Points.

For now I’m only going to deal with the value a receiver adds by catching passes. It is undoubtedly true that an NFL wide receiver can contribute to his offense scoring points by drawing defensive attention away from other players or by blocking well in the running game, but these contribution are probably an order of magnitude less important to a receivers full contribution than his pass catching and they are much harder to measure, so I’m going to set these considerations aside for now.

Receivers with under 20 targets in a given season over 2011-2015 averaged about 0.5 NEP/Target, so this can be considered the per target value of a replacement level player. Adam Harstad has written about the fact that for receivers, simply being targeted is an efficient play for the receiver because receivers who can’t get open aren’t targeted at all. This implies that if we forced a replacement level player to see a high share of targets, their efficiency would suffer, so we would expect the NEP/Target for a replacement level player thrust into a starting role to be marginally lower than 0.5. The 25th percentile of receivers is about 0.45 NEP/Target over the past 5 years, which strikes me as being about right. To calculate the value a receiver added in a given season over a replacement player, simply take the players’ total Net Expected Points and then penalize the player 0.45 expected points for each target they receive.  This method leaves approximately 90-100 receivers a season more than 5 points above replacement level, which is probably about right as well, generally players don’t stick around the league too long if they aren’t one of their teams three best receivers.

A weakness of this method is it does not account for quarterback quality. I’ll get back to this in a later post though it is very difficult to separate out a quarterbacks’ influence on his receivers, so it is best to leave quarterback quality out of the initial model.The top 10 Receivers from the 2016 Season and their Value over Replacement are presented below:

Name Targets NEP/Target Expected Points Added
Mike Evans 171 0.84 67.27
T.Y. Hilton 155 0.84 60.09
Julio Jones 129 0.89 57.06
Jordy Nelson 152 0.81 54.75
Brandin Cooks 117 0.84 46.21
Adam Thielen 92 0.95 45.7
Pierre Garcon 114 0.84 43.93
Antonio Brown 154 0.71 40.4
Tyrell Williams 119 0.78 39.07
Doug Baldwin 125 0.75 37.78

For the most part, the players at the top of the list got their on high volume and efficiency and are all part of the NFL’s elite passing attacks. Adam Thielen and Pierre Garcon probably are not top 10 receivers at this point in their career, but we would expect there to be significant variation between the 10 most skilled receivers and the 10 receivers who add the most value in a given season just due to luck and random variation. In other words, the observation that Adam Thielen was extremely efficient when he was targeted last season is a fact, but it is not by itself an argument to extrapolate that he is one of the 10 best wide receivers in the NFL. As stated, this is a descriptive metric meant to summarize a receivers’ entire stat line into a single value, so it is by definition going to suffer the same problems of any analysis drawn from the box score. Nevertheless, it is a far more useful starting point for evaluating the value a receiver added when targeted than simply using Yards, Touchdowns, Yards/Target, raw Net Expected Points, or any other box score driven statistic.