Measuring Analysts
We measure media analysts on TV by evaluating how many predictions they get correct over the course of the season. We have two main metrics: accuracy and quality. Accuracy is a simple measurement which looks at how many predictions are correct divided by how many predictions are made. Quality takes into account the difficulty of a prediction ( predicting who will win the Superbowl at the beginning of the year vs predicting who will win that game the week before) to determine which analyst is objectively better than others. Scroll down to see what qualifies as a prediction and learn more about how Quality is measured.
DEFINING A PREDICTION
The criteria for a opinion qualifying as a prediction is as follows:
- The forecaster gives a specific, exact goal that is achievable for a given player, team, or organization. (IE: The 49ers will win the Super Bowl).
- The prediction must have a specific end date. An end date is important because it limits the luck factor when making these forecasts. For example, a prediction stating a coach will eventually win a championship does not count as it does not specify the year, and a coach can eventually win a championship with a different team or in a different role as an assistant coach.
- The prediction must be binary. For example, stating that a player is a sleeper for the MVP is too general to have a specific yes or no outcome which makes this type of statement ineligible to be graded. Conversely, stating that the player will win MVP or will place second for MVP will qualify as a prediction.
- All predictions are listed with an associated video link or a quotation that includes the date, network, show and segment as a citation. Video links must be provided if the prediction is not outright stated but shown as a tv graphic.
Measuring Sports Predictions
Sports predictions are easier to make as time passes. It is much harder to predict the Eagles winning the Super Bowl in August before the season has started than it is to predict them winning the Super Bowl one week before the Super Bowl itself. As a result, there will be a scaled measurement where predictions made further from the end date are given more value than predictions closer to the end date.
This also helps with situations when an analyst changes their prediction. For example, if an analyst reasonably changes their championship pick due to an injury to a star athlete, this scale will allow them to still get credit but at a discounted rate due to them making this prediction later on in the season.
Therefore, we will have two types of measurements.
The first is a straight up measurement of accuracy where we take the number of correct predictions and divide it by the number of overall predictions for each analyst, network, or show within a season. This metric will simply be called accuracy.
The second metric will be called quality. Quality takes into account the scaled measurement mentioned above. Essentially, every correct prediction will be given a value equal to the amount of days separating the day the prediction was made to the end date of when that prediction came true. For each analyst or show, these values will be added up over the course of the season and then divided by the number of predictions to see how risky an analyst is in general. This riskiness is then multiplied by accuracy to balance both riskiness and accuracy and see the true quality of the analyst. The purpose of this metric is to reward the analysts who take the most risk when making predictions and give them a baseline comparison to analysts who are less risky when making predictions.
Methodology
When judging against contemporaries, we will have a baseline floor on the number of predictions an analyst must attempt over the course of the season. Predicting sports is hard and generally has a low percent chance of being correct, especially when taking into account overall luck, the variability of injuries, and unexpected life changes that any athlete or coach can experience but is not initially reported on. Therefore, the more predictions an analyst makes, the more likely they are at being wrong compared to an analyst that hypothetically makes two predictions and could post a 50% success rate due to one of them being right. The baseline on the number of predictions will be determined by the average number of predictions by analysts in each given sports league.
Joint predictions or predictions that include multiple components will be graded separately based on each statement. For example, if an analyst claims that the Los Angeles Lakers will go to the NBA Finals and face the Boston Celtics with the Lakers winning, that claim has three components to it:
- Lakers advancing to the NBA Finals.
- Celtics advancing to the NBA Finals.
- Lakers winning the NBA Finals.
As a result, this will qualify as three separate predictions. If the Celtics advance to the Finals but do not face the Lakers, the analyst can only get credit for one correct prediction. Conversely if the Lakers advance and win the NBA Finals, the analyst will get credit for two predictions with the possibility of three depending on if the Lakers face the Celtics.
An analyst does not get credit if they go off from their original prediction, but the original prediction still ends up being correct anyways. Any change in prediction will be marked accordingly within the dataset and the original prediction will automatically be marked as incorrect.
If an analyst makes a prediction on the same day as the game, they are automatically attributed a day, so that they are still given credit if they are correct.
Predictions involving mock drafts will be measured in a separate manner as misinformation by organizations plays a bigger role in this sporting event. To compensate for this, we will only select the most recent mock draft of each analyst before the draft of the relevant sport. There will be just one metric where we calculate the number of correct picks and divide it by the number of overall picks. For logistics purposes, we will only evaluate the first round of a given mock draft.