Forecast Bias: Why Zero Average Error Can Still Mislead

Forecast bias is a systematic tendency for forecasts to run too high or too low. In prediction markets, the measurable question is whether forecast YES probabilities exceed—or fall short of—the frequency of YES outcomes across a defined sample.

A single losing favorite does not establish bias. An 80% forecast allows for NO to happen 20% of the time. The test requires a collection of forecasts and their corresponding outcomes, not a list of surprising losses. Forecast-verification guidance distinguishes overall bias from reliability within particular probability ranges.

The forecast bias formula

For binary events, use this signed estimate:

Mean forecast bias = average forecast probability − observed YES rate

Equivalently:

Bias = (1/N) × Σ(pᵢ − yᵢ)

Here, pᵢ is the forecast probability from 0 to 1, yᵢ is 1 for a YES outcome and 0 for NO, and N is the number of forecast–outcome pairs. Each pair receives equal weight.

Under this convention:

• Positive bias: YES probabilities exceeded the observed YES rate in the sample.
• Negative bias: YES probabilities fell below the observed YES rate in the sample.
• Zero bias: the average probability matched the observed YES rate—not necessarily at every probability level.

Multiply the result by 100 to express it in percentage points, not relative percent error. If probabilities occupy spreadsheet cells B2:B201 and binary outcomes occupy C2:C201, the formula is =AVERAGE(B2:B201)-AVERAGE(C2:C201). Use only rows with a complete numeric forecast–outcome pair; blanks can otherwise make the two averages cover different samples.

Always state the sign convention. Some forecasting references define error as actual minus forecast, which reverses the signs; Forecasting: Principles and Practice uses that convention.

How zero bias can hide overconfidence

Consider this hypothetical sample—not a platform performance record:

Forecast group	Number of forecasts	Mean YES probability	Actual YES outcomes	Observed YES rate	Bias: forecast minus observed
Low-probability events	100	20%	30	30%	−10 points
High-probability events	100	80%	70	70%	+10 points
Combined	200	50%	100	50%	0 points

The overall bias is zero, yet both groups miss by 10 points. Low probabilities are too low; high probabilities are too high. That is an overconfidence pattern in this sample: forecasts are more extreme than the observed frequencies justify.

Positive overall bias alone does not mean overconfidence. It means overforecasting YES. Overconfidence concerns excessive certainty on either side, so it can disappear in an overall average.

A calibration table or reliability diagram exposes this cancellation by comparing the mean forecast with the observed YES rate within probability ranges. On a diagram with forecast probability on the horizontal axis, points below the diagonal indicate overforecasting; points above it indicate underforecasting. See the verification guidance on reliability diagrams.

Test market bias on a defined sample

Before interpreting a bias estimate, specify what was measured:

1. Fix the forecast horizon. Choose a rule such as one snapshot per sports contract, 24 hours before the scheduled game start. Do not mix early forecasts with prices recorded after the outcome became known.
2. Define the price. Record whether the forecast uses the last trade, bid, ask or midpoint. Keep timestamps and spread information so stale trades and wide quotes are visible.
3. Match the outcome to the contract. Score the event as its rules define it. Document cancellations, voids and non-binary settlements rather than silently coding them as NO.
4. Report both overall and bucket-level bias. Include sample dates, counts, mean probabilities and YES rates. Split by category and horizon where the sample permits.
5. Show uncertainty. A small observed gap may be sampling noise rather than systematic bias. Contracts tied to the same underlying event should not be treated as independent evidence; uncertainty estimates should account for that dependence.

The forecast-verification guidance recommends error bounds, especially for small samples and rare events. Weighting also changes the question: an equal-contract estimate describes the average contract, while a volume-weighted estimate emphasizes heavily traded contracts.

Bias is not the whole accuracy test

Mean bias measures direction; the Brier score measures average squared probability error. Neither replaces a calibration table. Use Brier score versus calibration to distinguish the tasks.

Nor does measured bias identify its cause. Wolfers and Zitzewitz’s 2006 research describes conditions under which prices correspond to traders’ mean beliefs—and models where they are useful but sometimes biased estimates of those beliefs. That is different from proving that traders’ beliefs match real-world frequencies.

If historical results suggest a correction, fit it on earlier data and test it on later, untouched forecasts. A correction that removes bias in the sample used to build it is not evidence that future forecasts improve; forecast-evaluation methodology calls for testing on data not used for fitting. A measured gap is a forecasting diagnostic, not by itself a profitable trade.