2017 NFL Draft Modeling: Quarterback Edition

by Max Mulitz

I built a simple model for projecting drafted FBS Quarterbacks to the NFL using only Age-Adjusted Collegiate Statistics. The model is fit on drafted players from 2005-2013 and uses Peak 3 seasons of Wins Over Average as the dependent variable.  Age-Adjusted collegiate Production predicted NFL Performance about the same as draft position in the sample, with a correlation of 0.36 compared to 0.35 for draft position. The model gave the following quarterbacks first round grades from the 2005-2016 drafts

Year Round Pick  Name Position Draft Age
2015 1 2 Marcus Mariota QB 21
2010 1 1 Sam Bradford QB 22
2014 1 22 Johnny Manziel QB 21
2011 1 1 Cam Newton QB 22
2005 1 1 Alex Smith QB 21
2016 1 1 Jared Goff QB 21
2014 1 32 Teddy Bridgewater QB 21
2009 1 5 Mark Sanchez QB 22
2012 3 75 Russell Wilson QB 23
2012 1 1 Andrew Luck QB 22
2009 1 1 Matthew Stafford QB 21
2012 1 2 Robert Griffin QB 22
2007 1 1 JaMarcus Russell QB 22
2005 1 24 Aaron Rodgers QB 21

It’s a fairly elite list. Mostly it’s young players who were dominant in college and went on to be drafted in Round 1, often first overall. Of the 14 players in the list, 7 have been to Pro Bowls.JaMarcus Russell and Johnny Manziel both busted badly, though arguably not due to lack of talent, and Mark Sanchez has settled in as more of a backup…The jury is probably still out on Goff, Bridgewater  and Mariota.

Here’s a list of the players taken in Round 1 from FBS schools that the model didn’t like as first round players.

Year Round Pick  Name Position Draft Age
2015 1 1 Jameis Winston QB 21
2008 1 3 Matt Ryan QB 23
2014 1 3 Blake Bortles QB 22
2006 1 3 Vince Young QB 23
2012 1 8 Ryan Tannehill QB 24
2011 1 8 Jake Locker QB 23
2011 1 10 Blaine Gabbert QB 21
2006 1 10 Matt Leinart QB 23
2006 1 11 Jay Cutler QB 23
2011 1 12 Christian Ponder QB 23
2013 1 16 EJ Manuel QB 23
2009 1 17 Josh Freeman QB 21
2012 1 22 Brandon Weeden QB 28
2007 1 22 Brady Quinn QB 22
2010 1 25 Tim Tebow QB 23
2005 1 25 Jason Campbell QB 23
2016 1 26 Paxton Lynch QB 22

Matt Ryan is elite. Vince Young, Jay Cutler, and Ryan Tannehill have had varying degrees of success. Jameis Winston looks like he is going to have a nice career.  The jury is still out on Paxton Lynch. Most of the players on this list ended up as busts.

This year the model considers three players First Round talents: Deshaun Watson, Patrick Mahomes and Mitch Trubisky. There have been multiple  players the model considered first round quarterbacks four times before.

In 2005, Alex Smith and Aaron Rodgers were both first round players per the model, Smith went first overall and Rodgers went 24th.In 2009, Matt Stafford went first overall and Mark Sanchez went fifth.  In 2012, the model liked Andrew Luck, Robert Griffin, and Russell Wilson in the first round. All three were successful, though Griffin has struggled since his knee injuries. Then in 2014, Johnny Manziel went 22nd and Bridgewater went ten picks later at 32. Bridgewater is currently dealing with injuries and Manziel is out of the league due to off-field issues.It does not appear as if the first player drafted in a given year has a significant advantage on the other qualifying players in the class, nor does it appear clear that draft position is a significant predictor  for qualifying players.

Right now, Mahomes appears undervalued. CBS has him 70th overall while ranking Mitch Trubisky 12th. Per the Jimmy Johnson Trade Chart, the 70th pick is only 1/5th as costly in terms of draft capital as the 12th pick. Given the data, it’s hard to argue Trubisky is twice as likely to become a top quarterback as Mahomes, let alone five times as likely. There simply aren’t prospects with Mahomes’ age adjusted production who fall out of the first round (other than Russell Wilson.)

The model also considers Deshone Kizer, Brad Kaaya, Nate Peterman, and Davis Webb draftable prospects based on their age and production, while Chad Kelly and Jerod Evans just miss the cut.

Advertisements

The Fundamental Metric For Valuing Individual Run Defenders: Introducing Run Stops Added

by Max Mulitz

Theory of Run Stops Added:

A run stop is a play where a defender  makes a tackle on a running play that is not efficient for the offense. Rushing Success Rate correlates with winning better than Yards Per Carry, so the measure of a run defenders value can be measured by his ability to prevent successful runs. By determining the value of a run stop and measuring the players’ ability to generate run stops, we can solve for the players’ value as a run defender.

How Valuable is a Run Stop?  A run stop is worth about 1.3 points. Runs that are unsuccessful have an expected point value of about 0.75 expected points, while unsuccessful runs have an expected points value of about -0.55 expected points, so a run stop is worth about 1.3 expected points on average. Of course, a 5 yard loss on 3rd & 1 at the Goal Line has a greater impact than a tackle for no gain on 1st & 10 at midfield, but in aggregate a run stop is worth about 1.3 points.

How does an individual player contribute to his team making Run Stops? Different positions have different base run stop percentages. A replacement level 4-3 defensive end may only make a run stop on 6% of his run plays while a replacement level MLB may make a run stop on closer to 7.5% of his plays. Players in different schemes will have different opportunity levels, but a middle linebacker on the Raiders is has a more similar job to a middle linebacker on the Chiefs in terms of run defense than he will have to a defensive tackle on his own team.

Modeling Individual Run Defense: A players’ contribution to stopping the run can be estimated as the percentage of run plays where he generates a run stop minus the expected run stop percentage for his position, multiplied by the number of snaps they played in a given season, as seen in the formula below:

Run Stops Added=(Run Stop %-Position Replacement Level Run Stop %)*Number of Snaps

For instance, a Defensive End who played 200 Run Defense Snaps and had a 8% Run Stop Rate would be responsible for 4 extra run stops above a 6% replacement level player. At 1.3 points per run stop, the players contribution could be estimated at 5.2 points (about 0.14 wins.)

 Run stops added is a reasonably consistent statistic from year to year, which is a necessity for it to be used as an indicator of defensive player value.  For example, for Defensive Ends, Run Stops Added for players with over 200 run snaps in both years were strongly correlated from 2014 to 2015 at R=.66.

Limitations of Run Stops Added:

  1. Certain players may improve their teams’ run defense statistics by making their teammates’ job easier. If a star defensive tackle demands a double team, thereby allowing the teams middle linebacker to often go unblocked and make more plays in the run game, the tackles contribution will reflect in the linebackers statistics, but not his own. Dominant defensive linemen/run stopping LBs are usually able to create for themselves often enough that we are unlikely to have many “No Stats All Stars” in primary run defense, though the stats obviously only tell most of the story. In general, the players who are considered top run defenders are able to translate that into a high run stop percentage.
  2. Scheme obviously matters. Some teams schemes are going to enable their weak side linebackers to make relatively more tackles in the run game than other teams irrespective of individual player ability.
  3. Run Stop Percentage is not appropriate for secondary run defenders, as it does not reward tackles in the open field (after the run has already “succeeded” in terms of improving first down probability.)

An Anecdote:

Damon Harrison is a good check on the calibration of our model. Harrison is perhaps the best run stopper in the NFL, though he brings nothing to the table as a pass rusher. In 2015, Harrison’s Run Stops Added would have valued him at almost exactly one win above replacement. With a 143 Million Dollar Salary Cap in 2016, this would imply an open market value of 9.53 million. In fact, Harrison signed with the Giants for an average of 9.25 million per season.

Why Run Stops Added instead of other metrics:

Tackles for loss are useful, but because tackles for loss are a subset of run stops, there is always a smaller number of TFLs than Run Stops in a given year, making them less stable over a sample of only a few hundred snaps.

Percentage Share of Run Game Tackles/Total Tackles can be useful, but doesn’t tell us if a player is making a lot of tackles because he’s making plays in the run game or because he’s chasing players down at the second level after they beat his teammates. An average run defender surrounded by replacement level teammates is going to have the tackle totals of a stud, for instance.

Expected Points Added is useful, but may overweight plays in certain situations. For instance, a stop on 4th & Goal at the 1 could be more valuable than five run stops over the course of a game in the middle of the field. In the long run, the player who made five stops in the run game has shown more of a repeatable skill set than the player who happened to make only one play at exactly the right time.

Conclusion:

Run Stop Percentage and Run Stops Added are the fundamental metrics of a defenders’ contribution in the run game. A players Run Stops Added can be converted to  expected points added and therefore wins added, which serves as a starting point to value an individual player’s contribution as a run defender. Run Stops Added can also be used for advanced scouting purposes to help discern who the opponents’ best and worst run defenders are.

Valuing Notable Free Agent Receivers Using Production and Age

A players’ previous performance statistics, age, health, film grade, team needs, and projected scheme fit all factor into determining how much money a team is willing to offer a free agent. Generally, a starting point for determining a players value is taking a players’ past production and extrapolating it into the future depending on the players age. This is by no means a complete analysis for any player. Sometimes very talented players lack production in one scheme and go on to produce at a high level in a different scheme where they are a better fit. Wes Welker and Emmanuel Sanders are just two examples of receivers who saw massive increases in production after changing teams. That said, on field production is a good place to start and a players’ past production is the most predictive single variable for future performance.

Below I list some notable Free Agent Receivers along with their production over the past three years and a single value representing the past three years using a 5/4/3 weighting structure to weight more recent years more heavily. Also included is a players’ free market value if they continue to play at the level of their Weighted Wins Added based on a 166 million dollar projected cap next year. I also included Spotrac’s estimate of a players free market value as a reference. Further included is the probability the player declines due to age based on Adam Harstad’s model of receiver career mortality. I do not include age related decline for players who will be 25 or younger next year or 35 or older, as the model is not well trained on these extremes.

Untitled.png

 

-Alshon Jeffery leads the pack. His value would be somewhat higher at 13.5 million if you prorate his stats to account for missed games.

-Kenny Britt has quietly been very good. My WR metric does not attempt to account for team quality, so it is impressive that Britt was 12th in the league last season in Wins Added in spite of the fact that the Rams were dead last in Offensive Passing NEP and second to last in passing yards per game. 29 next year, Britt is not at great risk of age related decline. Spotrac is low on Britt’s value and it is possible he finds a soft market in Free Agency because he played on such a poor offense. But the fact remains that very good things happened for the Rams when they targeted Britt, and it looks like he could be one of the more underrated players in free agency. For whatever it’s worth, PFF had Britt as the best receiver at running crossing routes in the league last season.

-Terrelle Pryor would projected  in the 9.5 million a year range if you only used the 2016 season. At age 28 he’s not likely to suffer severe age related decline. It may or may not be a concern that he only has one year of production.

-Kenny Stills and Desean Jackson have produced almost identically over the past three years, though Jackson could be starting to get old. Stills and Jackson had almost identical advanced stats last season, with Jackson receiving 18% of the Redskins targets but 30% of their Air Yards for 15.5 Air Yards/Target. Stills receiving 17% of the Dolphins Targets but a whole 31% of their Air Yards and averaged 15.9 Air Yards Per Target. Jackson has some risk of decline in the next couple years, but as recently as last season he was among the fastest players in the league. Both players are low-usage, high-efficiency deep threats so they may only make sense for a team that is looking for that type of player.

 

 

The Role of Analytics in Game Planning

 

by Max Mulitz

Data generated scouting reports famously involve many pages of data. A team does not gain an advantage by summarizing the most data. The goal is to get the most useful information to the coaches. What specific decisions are coaches looking to the data to help them make?

For example, say a defensive coordinator is deciding if he wants to play base defense or more nickel defense when his upcoming opponent is in 12 personnel. The coordinator expects his base defense to be above average against the run but below average against the pass but for his nickel defense to be above average against the pass but below average against the run. To make this decision the coordinator wants to know the probability the opponent will run or pass in normal down and distances from 12 personnel depending on the defensive personnel.

The problem is, this is a very specific question. A team may have only faced nickel defense from 12 personnel in normal situations a handful of times in a season, so there might not be a meaningful sample. The primary way to alleviate this is by using base rates. If NFL pass 60% of the time from 12 personnel in normal situations against base defenses but only 40% of the time against nickel defense that is useful information. If the base rates are very similar that is also useful information. How often teams run from 12 personnel in general is not that useful for this question, since we have good reason to believe defensive personnel will affect run/pass choices.

The other way to expand our sample is to look for similar situations that would fit our theory. Here that means looking for how the opponents’ run/pass balances changes depending on the defensive personnel in normal situations. Some teams run quite a bit more often vs. nickel than vs. base and some teams don’t change their offense very much based on defensive personnel. Knowledge is theory laden.

As important as what data you include is what data is excluded from the analysis. How often teams run from 12 personnel isn’t very useful because it doesn’t account for the fact that we are considering using a lighter defense that may cause teams to run more.

Also if you don’t constrain yourself to normal situations the sample will be contaminated with examples of teams playing nickel defense on 3rd & long which will inflate the probability of a pass.

If you don’t expand your sample size by looking at classes of situations a team can be fooled into thinking a team is 80% likely to run in a given situation just because they have done it 4/5 times, even if the base rate is just 33% runs.

Interpreting data for game planning is the domain of coaches. A football scientist’s job is to make sure coaches are seeing the correct data that helps them game plan and are not weighed down with irrelevant or misleading data.