Forecast error is not the same as a wrong prediction

Forecast error is the difference between an observed outcome and the forecast made before it. For prediction-market probabilities, that means comparing a forecast between 0 and 1 with a resolved YES/NO outcome—not simply counting whether the favorite won.

An 80% forecast that resolves NO incurs a large error. But that result alone does not establish that 80% was a bad probability: the forecast also allowed a 20% chance of NO.

The forecast error formula

A common convention is error = actual − forecast. Forecasting: Principles and Practice defines error this way and distinguishes it from an in-sample residual: forecast errors evaluate predictions against test data, rather than fitted values against training data. Hyndman and Athanasopoulos

For a standard binary contract, define:

• p: the forecast probability of YES, expressed as a decimal.
• y: the resolved outcome, coded 1 for YES and 0 for NO.
• e = y − p: the signed forecast error.

Under this convention, a positive error means the outcome was above the forecast; a negative error means it was below. Check the subtraction order before interpreting forecast bias.

Here is an illustrative 80% forecast scored against either possible outcome:

The signed error measures direction. Squaring it prevents positive and negative errors from cancelling and penalizes larger misses more heavily.

Across N forecasts, the binary Brier score = Σ(yᵢ − pᵢ)² / N. On this convention it ranges from 0 to 1, with lower scores better. It is a strictly proper scoring rule: in expectation, reporting the true probability minimizes the loss. Scikit-learn Brier-score documentation

Why a good probability still has error

Suppose an illustrative batch contains 100 forecasts, all at 80%, and exactly 80 resolve YES. The forecasts match the observed frequency, yet their Brier score is not zero:

(80 × 0.04 + 20 × 0.64) / 100 = 0.16.

That is not a contradiction. Calibration asks whether events assigned a probability occur at approximately that frequency. An individual outcome must be either 0 or 1; it cannot resolve at 0.80. Well-calibrated probabilities therefore still incur realized forecast error. Scikit-learn calibration guide

This also explains why outcome-based error is not the same as the difference between a forecast and the underlying probability at forecast time. A NO resolution lets you calculate the 80% forecast’s squared loss of 0.64. It does not prove the event’s probability was actually zero when the forecast was made.

Use Brier score to summarize probabilistic performance, then inspect calibration separately. A lower Brier score does not necessarily mean better calibration: the score also reflects how effectively forecasts distinguish more likely events from less likely ones and the uncertainty of the outcomes being evaluated. Methodological explanation

For a market evaluation, define the price measure, forecast horizon, resolution rules and sample before averaging errors. Our forecast accuracy formula guide covers the calculation and comparison checklist; Brier score versus calibration explains how to interpret the results. One losing favorite establishes neither a platform’s average performance nor its calibration.