Forecast Accuracy Formula: Measuring YES/NO Probabilities

For YES/NO probability forecasts, a useful forecast accuracy formula is the Brier score: the average squared difference between each forecast probability and its resolved outcome. Lower is better.

There is no single formula for every forecasting task. A probability forecast, a predicted winner and a sales-volume estimate require different measurements. For prediction markets, start by scoring the probability—not just whether the favorite won.

The Brier score formula

For binary events:

Brier score = (1/N) × Σ(pᵢ − yᵢ)²

• N is the number of scored forecasts.
• pᵢ is the forecast probability of YES, expressed from 0 to 1: 70% becomes 0.70.
• yᵢ is the resolved outcome: 1 for YES, 0 for NO.
• Σ means add the squared errors across all forecasts.

This binary convention ranges from 0, perfect forecasts, to 1, certain forecasts that are always wrong. Some implementations sum errors across both outcome classes and use a 0–2 scale, so check the convention before comparing published scores. Scikit-learn’s scoring documentation explains both versions.

Squaring the error penalizes confident misses more heavily. A 90% YES forecast resolving NO contributes 0.81; a 60% YES forecast resolving NO contributes 0.36. Brier is a strictly proper scoring rule: reporting your actual probability belief minimizes your expected loss, rather than rewarding exaggerated certainty. See the scores library’s Brier tutorial.

Worked example: four resolved markets

These are illustrative forecasts, not platform results.

Market	YES probability, p	Outcome, y	Squared error, (p − y)²
A	0.80	1	0.04
B	0.60	0	0.36
C	0.30	1	0.49
D	0.10	0	0.01

Brier score = (0.04 + 0.36 + 0.49 + 0.01) / 4 = 0.225.

Forecasting 50% on every event would produce 0.25, regardless of the outcomes. This sample therefore beats that benchmark, but four observations do not establish a reliable forecasting record. Nor is 50% necessarily the most useful benchmark: a historical base rate known before forecasting may be more informative.

A score of 0.225 does not mean “77.5% accurate.” Subtracting it from 1 does not turn squared probability error into a percentage of correct predictions.

Calculate it in a spreadsheet

Put probabilities in column B and outcomes in column C. In D2 enter:

=(B2-C2)^2

Copy down, then average the error column:

=AVERAGE(D2:D5)

Use decimals or percentage-formatted cells for probabilities—not whole numbers such as 80 when you mean 80%. Only score rows containing a valid probability and a resolved 0/1 outcome; do not let a blank outcome stand in for NO.

When other accuracy formulas apply

Hit rate answers a narrower question:

Accuracy (%) = correct YES/NO calls / total calls × 100

Choose a threshold in advance—for example, call YES above 50%, NO below 50%, with an explicit rule for exactly 50%. In the table, two of four calls are correct: 50% accuracy.

But 51% and 99% forecasts become the same YES call. Hit rate discards confidence, whereas Brier scores probability estimates. This distinction between probability prediction and a categorical decision is set out in scikit-learn’s metric-selection guidance.

MAPE applies to numerical point forecasts, such as units sold:

MAPE (%) = (100/N) × Σ |(Aᵢ − Fᵢ) / Aᵢ|

Here A is the actual value and F the forecast. A calculation labeled “100% − MAPE” is not a hit rate and can become negative. More importantly, MAPE is undefined when an actual value is zero—exactly what happens when a binary outcome resolves NO. Forecasting: Principles and Practice documents this limitation. Our guide to MAPE for YES/NO odds explains the mismatch.

Make the market comparison fair

Before averaging errors, define the evaluation:

• Timestamp: use comparable forecast horizons, such as 24 hours before the event, and exclude prices observed after the outcome became known.
• Price measure: specify whether p comes from the last trade, bid, ask or midpoint. Keep that choice consistent; record spreads and liquidity alongside it.
• Outcome rules: score the contract’s actual resolution definition. State how voided, disputed or nonbinary settlements are handled.
• Sample: report dates, categories and counts. For head-to-head comparisons, use the same events wherever possible. One forecast per contract avoids silently giving frequently sampled markets more weight; identify related contracts that share an outcome.

Finally, check calibration separately: do forecasts near 70% resolve YES roughly 70% of the time? Compare each probability bucket’s average forecast with its observed YES rate, and show the bucket count. A lower Brier score does not necessarily mean better calibration, because it also reflects discrimination—the ability to distinguish more likely events from less likely ones—and outcome uncertainty. Scikit-learn’s calibration guide explains that distinction; see also Brier score versus calibration in prediction markets.