In the mean time, I have discovered two other ranking systems: Fivethirtyeight's NFL Forecasts (ELO) using the simple yet elegant ELO system and Jeff Sagarin's Ratings (JSR) on USA Today. I discovered these rating systems on Friday after the start of week one games. To shuffle in these predictions into the prediction model, I applied a weighed average to each (on Friday) of 9:5:2 FPI:ELO:JSR. It should be noted that New England had a greater than a 50% chance of winning from all three systems.
On Tuesday, I did my own post game analysis of the three rating systems plus my combo weight. Although I am using probabilities to determine risk and to defer risk, I still must ultimately make a deterministic choice. So my initial analysis is quite deterministic. If a team had a greater than 50% chance of winning and actually won the match, the score of 1 was awarded. Otherwise a score of 0 was awarded. The highest score is what is desired here.
Information
|
Week 1
|
Post Week 1
|
FPI Average Correctness
|
75%
|
88%
|
ELO Average Correctness
|
75%
|
88%
|
JSR Average Correctness
|
69%
|
88%
|
COMBO Average Correctness
|
69%
|
88%
|
While it is an interesting look, it doesn't tell the full picture. The other factor I am noting is the 'average incorrectness'. That is if a call was incorrect, what is the difference between the call and 50%? Were they close games or blowout mistakes?
Information
|
Week 1
|
Post Week 1
|
FPI Average Delta Incorrect
|
17%
|
8%
|
ELO Average Delta Incorrect
|
10%
|
7%
|
JSR Average Delta Incorrect
|
(2.14)
|
(1.35)
|
COMBO Average Delta Incorrect
|
16%
|
7%
|
FPI seemed to yield slightly worse results than ELO. Both ELO and FPI got the Atlanta, St. Louis, and Tennessee games incorrect. The ELO also got the Kansas City game incorrect, while FPI got the Buffalo game incorrect.
Looking deep into the games that were incorrect, one game stands out: the Tennessee game in the FPI system. Tennessee had just a 12% chance of winning the game. However they won the game, and they won the game in a relatively decisive fashion. However, upon further review Tennessee's FPI was rated at a -5.9 while Tampa Bay was rated at a -5.6. Except for Oakland and Jacksonville these two are the two weakest teams in the NFL. This is definitely a lesson in a wake-up call to myself, be wary if two-week teams are going at each other with low FPI's.
The only other game worth noting, would probably be the St. Louis versus Seattle game. It was probably extremely upset by any stretch of the imagination. Seattle came in with approximately a 70% chance of winning and ended up losing the game. Outside of these two outliers, I am pretty happy with the results.
My dad emailed me with the following question:
You may need to rethink strategy, Eagles made it look close but were not really in it, ELO appears better than FPI
to that I responded:
You are partly correct, ELO is marginally better than FPI. They both got correct 12 of 16 games. If we look at the "Average Incorrectness" - which I define as the average distance between the probability of a wrong pick and 50% of all incorrect picks (A pick is wrong if a team had less than a 50% chance of winning and actually won the game) - FPI had an average of 17% wrong and ELO was 10% wrong.
However if we look closely, FPI really screwed up the Tampa-Tennessee Game. Tampa was given an 88% chance of winning and lost. However, if we look closely at the raw FPI of the squad, Tampa had an FPI of 0.5 and Tennessee had 0.2. Both are absolute garbage. If we remove this outlier, the average incorrectness drops to a palatable 12%.
I'm actually pretty pleased with the results too. If we look at games with odds close to 50-50 (I went as high as 65-35, i.e. to exclude outliers), the number of incorrect choices was about 15% (depending on scheme), much better than the expected number of incorrect of... well... 35% as if I was deciding a game by 100 coin flips.
FPI, JSR, and ELO were all updated so I have tentative picks of Baltimore and New Orleans.
No comments:
Post a Comment