The Elephant in the Room
The latest sports data/analytics piece for KCSN from contributor, Joseph Hefner
I knew when I started this series on analytics that at some point, I was going to be writing this article. I know it’s a controversial topic, but I think it’s time. Let’s talk about PFF grades.
But really, what I want to talk about is standardization. Standardization is when we take a whole scattered set of systems, and try to build a single system that everyone can use. It’s absolutely essential in many businesses with different branches.
Standardized tests are something we’ve all likely had to deal with before. One test that everyone takes that measures everyone by a single yardstick. SAT’s, ACT’s. It works for a lot of people. It also completely misses the mark for many.
Standardized systems are designed to work for 60-80% of the target population. They work for the masses, and leave the rest behind. They can’t account for every variable, and don’t pretend to. They work very well in forming broad overarching statistics and conclusions that are actionable, even though they miss all along the margins.
The advanced stats I’ve already written articles about (EPA and CPOE) are both standardized systems. They take historical data and turn it into a standard that they then measure performance by. They are very good at it, too.
But Jimmy Garoppolo consistently shows up as a top 5-ish QB in EPA. Literally nobody thinks he’s a top 5 QB, not even his own coach, who gave up the world for his replacement. Standardized systems, by their very nature, do not work well for people who do not comfortably fit in that standard. People on the margins.
PFF grades are a standardized system. PFF hires people to come in to watch film and grade every player on every snap of every game. They train all of their people to watch for the same keys, to judge the same angles and blocks, and to spit out the same grade as everyone else would for that play. It’s all standardized.
The beauty of this kind of standardization is that every single player at every single position is graded on every single snap by the exact same standard. The catch, for PFF and for all standardized systems, is that their grades actually aren’t telling you how good a player is.
The grades describe how well a player meets a very specific standard. They are a reasonable proxy for the quality of player, but they’re not actually a statement of quality. The same way that an SAT score doesn’t actually describe how intelligent you are, just how well you meet that specific standard of intelligence.
PFF doesn’t publish their standards for grades for different positions, but we do know some things about how they grade players. We know that for QB’s, they mostly ignore scrambling around behind the line of scrimmage. Snap, drop back, pocket presence, and the throw itself are what they judge.
A huge part of the brilliance of Patrick Lavon Mahomes II is that he doesn’t play the game the way everyone else does. He’s great in structure, but he’s truly a work of art out of structure. He steps up, steps left, runs around, spins and breaks the ankles of 3 different rushers, runs toward the line of scrimmage to pull in the defense and throws to a now wide open receiver.
A PFF grader ignores all the brilliance between the snap and the throw, and simply grades Mahomes on how difficult the throw was. Since the receiver was wide open, it wasn’t difficult at all. Writes down 0.5 on a -2 to +2 scale, and moves on to the next play. Some of Pat’s greatest plays just don’t fit their standard. He’s a unicorn.
Additionally, any small sample size (i.e. a single game, or even a few games) can be very misleading, or prone to bias. Maybe all the receivers were wide open that day. Maybe Pat missed a couple layups and those counteract the brilliant plays. Maybe we got the harsher grader on staff.
Eric Eager, big Chiefs fan, formerly the VP of PFF’s R&D, now with SumerSports, tweeted this just a few days ago:
PFF knows that their game by game grade results can be wonky at times. Small sample sizes, human graders who come in with their own biases and expectations, etc., lead to some odd results. But over time, those biases even out. Over time, the cream generally rises to the top.
So what should we do when we see Mahomes game grade is lower than Geno Smith’s in week 1, after our evisceration of the Cardinals? Maybe, instead of getting fired up (PFF wants the engagement, folks! That’s why they tweet them!) about the grades, understand that small sample sizes make for weird stats, and take a look at the trend over time.
Take a look at the full season PFF grades for QB’s through week 12:
Lavon is in his rightful place as the King of the NFL. He does enough things that DO fit the standard that even though some of his most brilliant plays get diminished, he’s STILL the top QB without them. Absolute King.
And if he wasn’t rated as the top QB by PFF? We can just say that their standardized model misses on him, while still recognizing that it hits on the vast majority of players. As it’s designed to. It’s not designed for the unicorns.
Finally, I’ll leave you with this. I love the KC Lab guys: Kent, Matty, & Craig. I love the articles they put out reviewing players. I’ve bought their draft guide every year. It’s quality work, and very much appreciated. They’re good at it. We’re lucky to have them, and we shouldn’t take that for granted.
You know what I can’t get from them? A comparison of Isiah Pacheco vs every other running back in the NFL. Not just how well is he doing in KC, but how does his season compare to the season every other RB is having this year?
For that kind of comparison, you have to have reviewed all of the snaps from every single RB in the NFL this year. To my knowledge, PFF is the only publicly available system that does that. Yes, it misses on players. Yes, it’s an imperfect system. Yes, it’s a black box system that they don’t explain publicly.
But to go back to the first words I wrote for KCSN, “All models are wrong, but some are useful”. PFF’s model is wrong, but it still gives very useful information, especially in the aggregate. And if you disagree with them? Hey, maybe that player is just a unicorn.
You make a lifetime of difference for a child or teen.
Boys & Girls Clubs is looking for dedicated men and women to invest in the lives of youth through coaching. Did you know by participating in sports, children learn the value of teamwork, responsibility, good sportsmanship, and self-esteem? By becoming a volunteer coach, you are helping to power the dreams—and successful futures—of Kansas City’s kids! The Clubs are currently seeking baseball coaches. Click here to sign up today! All equipment will be provided. No prior coaching experience is needed.
 | A guest post by
|
Not that anyone will respond to the email side, but this was a great article.
Thank you!! That’s very much appreciated.