Prediction Accuracy: How to Measure It — Calibration and Brier Score

Prediction accuracy cannot be measured honestly with a single hit rate. A football prediction is a probability, so what you need to check is whether those probabilities match reality: do events given 70% happen roughly 70% of the time? That is what probability calibration, the Brier score and a backtest on past matches are for. Below is how each tool works and how to use them to check any tipster.

Why hit rate misleads

Picture two tipsters. The first only picks heavy favourites and gets most results right. The second predicts evenly matched games and is right less often. On hit rate the first looks stronger, but that says nothing about quality: they are simply choosing events that almost always happen anyway.

Hit rate has three problems:

• It ignores confidence. A "70% home win" and a "51% home win" count the same, although they are very different claims.
• It depends on match selection. A high percentage is easy to get by picking obvious outcomes.
• It says nothing about odds. For betting what matters is not only "did it land" but at what price. Hitting 60% of outcomes at odds that imply 70% means losing over the long run.

That is why prediction accuracy is judged with metrics that work with probabilities, not a binary yes/no.

Probability calibration

A model is calibrated when its probabilities mean exactly what they say. Checking it is simple: split all predictions into groups by stated probability (say 50–60%, 60–70%, 70–80%) and count what share actually happened in each group.

• If about 65% happened in the "60–70%" group, the model is well calibrated.
• If 50% happened, the model is overconfident: it promises more than it delivers.
• If 80% happened, the model is underconfident: it understates probabilities.

Probability calibration matters more than it seems. Even a correctly chosen favourite is useless for judging a bet if the stated probability is inflated, because that probability is what gets compared with the odds.

On xG.Football the calibrated probability is calculated exactly this way. The system looks at how often past predictions with the same market, line, score and model decision came true. It starts with matches from the same competition and needs at least 200 settled outcomes. If there are fewer, it uses all competitions, and then it needs at least 500. If no sample is large enough, the probability is not shown as calibrated. The criteria are described in how the model works.

Brier score: one number for probability quality

The Brier score is the mean squared error of a probability. For each prediction you take the stated probability p and the actual outcome o (1 if the event happened, 0 if not), compute (p − o)² and average over all predictions.

Example: a prediction gave an event 0.7 and it happened. The error is (0.7 − 1)² = 0.09. Had it not happened, the error would be (0.7 − 0)² = 0.49. A confident miss is punished hardest.

How to read the result for yes/no events:

• 0 — perfect predictions (unreachable in football);
• 0.25 — the level of predicting "50% on everything": the benchmark a model has to beat to be useful;
• lower is better.

The key property of the Brier score is that it cannot be gamed by picking easy matches as simply as hit rate can. It captures both whether the outcome was right and whether the confidence was right. Comparing Brier scores only makes sense on the same set of events: a market with frequent outcomes and one with rare outcomes have different baselines.

Backtesting predictions: testing on the past without peeking

A backtest runs the model over matches already played as if it had been working at the time. A backtest of predictions is only useful if the main rule holds: for each match, only data known before kick-off is used. If anything from the future leaks in (the final season table, stats from matches played later), the model looks more accurate than it really is. This is called data leakage or look-ahead bias.

On xG.Football the backtest is built from stored statistics, and calibration for each match only uses matches that kicked off before it. Even an honest backtest is still a test on the past, though. The real test is predictions saved before the match and checked against the result afterwards. That log is open on the page with prediction statistics.

How many predictions you need before drawing conclusions

Prediction accuracy on a small sample means almost nothing. Ten right out of fifteen is 67%, but with a sample that size the true accuracy could just as easily be noticeably lower or noticeably higher.

A confidence interval is used to express that uncertainty. Alongside each calibrated probability xG.Football stores a 95% Wilson interval: it shows the range the true probability most likely falls in, given the sample size. The smaller the sample, the wider the interval and the less weight a single number deserves.

How to check a tipster: a checklist

The same principles apply when you evaluate any prediction service or author:

1. Predictions are published before kick-off, and you can verify it. A prediction that appears after the fact is worthless.
2. The log is complete. Losing predictions are not deleted or "forgotten".
3. Probabilities or odds are stated. Without them you can judge neither calibration nor value.
4. The sample is large. Dozens of predictions are still luck or bad luck; hundreds start to say something.
5. The method is explained. It is clear what data the prediction is built on, and nothing is promised as "guaranteed".
6. Stated probability matches reality. If average confidence is high but the share that came true is clearly lower, that is overconfidence, not accuracy.

You can check the last point on the xG.Football statistics page: average estimated probability and accuracy on settled predictions are shown side by side, and the log can be filtered by market, status and date. Current football predictions for upcoming matches go into the same log and are checked the same way, after the final whistle.

The short version

Prediction accuracy is not measured by hit rate but by how well probabilities match reality. Calibration tells you whether the stated percentages can be trusted, the Brier score sums up probability quality in one number, an honest backtest tests the model on the past without peeking, and sample size tells you how much any of these numbers can be trusted.