Why Forecast Accuracy and Calibration Are Not the Same Thing

Brier score and calibration answer different questions. The Brier score summarizes the squared error of resolved probability forecasts; calibration tests whether stated probabilities correspond to observed frequencies. A credible prediction-market evaluation should report both, using comparable contracts and consistently timed forecast snapshots.

The short answer: use Brier score and calibration together

Measure	Question and calculation	Output and strength	Main blind spot
Brier score	How far were the probabilities from the resolved outcomes? For binary forecasts, BS=mean((p-y)^2).	One overall score; lower realized values mean less squared error.	A pooled average can conceal bias within probability ranges, categories, or horizons.
Calibration	Did events assigned a given probability occur at approximately that rate?	A curve or table showing systematic over- or underprediction.	Forecasts can be calibrated but uninformative if they do not distinguish likely events from unlikely ones.

In the Brier formula, p is the forecast probability and y is the resolved outcome, coded as 1 if the event occurred and 0 if it did not. For n forecasts:

BS = 1 ÷ n\sumi₌₁ⁿ(p_i-y_i)²

Lower realized scores indicate less squared prediction error over the observations evaluated. Calibration instead asks whether, across a suitable collection of forecasts, events assigned 70% occur about 70% of the time and events assigned 20% occur about 20% of the time. A methodological analysis of binary prediction models stresses that the Brier score measures total probabilistic error rather than calibration alone (“On misconceptions about the Brier score in binary prediction models”).

The core distinction is simple: Brier score measures overall probabilistic error, whereas calibration examines one property of a forecasting system. Neither is universally superior. Report an overall proper score alongside a calibration table, plot, or comparable diagnostic.

Worked example: calculate the score, then check calibration

Consider this illustrative example, not actual prediction-market data:

Forecast p	Outcome y	Squared error
0.80	1	(0.80-1)^2=0.04
0.60	0	(0.60-0)^2=0.36
0.20	0	(0.20-0)^2=0.04

The mean squared error is:

BS = 0.04 + 0.36 + 0.04 ÷ 3 = 0.1467

This is a valid realized Brier score for the three observations. Because errors are squared, the incorrect 60% forecast contributes much more to the total than either of the other forecasts.

The same data cannot support a persuasive calibration conclusion. Each probability level appears only once, so there is no repeated evidence showing whether 80%, 60%, or 20% forecasts resolve at corresponding rates. One successful 80% forecast does not establish calibration at 80%.

In practice, analysts can group many comparable forecasts into probability ranges chosen under a disclosed rule. For each range, report:

1. The number of observations.
2. The mean forecast probability.
3. The observed proportion of events that occurred.
4. An appropriate indication of uncertainty.

A range with a mean forecast of 0.72 and an observed event rate near 0.72 would appear calibrated, subject to its observation count and uncertainty.

A user-generated prediction-market discussion provides an example of the basic self-tracking approach: log forecasts, calculate Brier scores, and compare confidence ranges with observed frequencies. It is an informal community discussion rather than a validated statistical method, and it does not establish a universal minimum sample size (prediction-market community post).

What each metric can miss

A low Brier score does not by itself prove good calibration. An aggregate score may look favorable because most questions were easy while forecasts in a particular probability range, category, or horizon were consistently too high or too low.

Calibration has a different weakness: it does not necessarily show useful discrimination. If 30% of a comparable set of events occur, a forecaster who assigns 0.30 to every event may be calibrated in aggregate. Yet those forecasts do nothing to distinguish relatively likely events from relatively unlikely ones. The system is calibrated but unsharp.

Realized performance must also be distinguished from expected performance. Even a forecaster who knows and reports the true underlying probability will generally receive a nonzero realized score because the binary outcome remains uncertain. A truthful 70% forecast incurs squared error of 0.09 if the event occurs and 0.49 if it does not.

A perfect realized Brier score of zero requires every event that occurs to have received a forecast of exactly 1 and every event that does not occur to have received a forecast of exactly 0. The Brier score is nevertheless strictly proper in expectation: reporting the true probability uniquely minimizes expected loss, although randomness can make truthful forecasts look better or worse in a finite sample (journal article on Brier-score interpretation).

A practical workflow for evaluating prediction-market probabilities

Set the evaluation rules before examining the results:

• Define the contracts eligible for inclusion.
• Document objective resolution criteria and the authoritative resolution source.
• Decide how canceled, ambiguous, annulled, or unresolved questions will be handled.
• Choose a consistent forecast horizon or timestamp rule.
• Record timestamped probability observations.
• Identify each input as a traded market price, individual probability submission, or aggregated forecast.
• State the market-price convention used, if applicable.
• Code resolved binary outcomes as 0 or 1.
• Calculate the overall Brier score.
• Inspect calibration by probability range, including counts and uncertainty.
• Segment by category or horizon only where the sample supports it.

Traded prices and directly submitted probabilities are different inputs even though both can be evaluated against binary outcomes. Market prices emerge from trading, whereas some forecasting platforms solicit and aggregate probability estimates directly. The Metaculus FAQ, for example, distinguishes its submitted and aggregated forecasts from prediction-market shares. An evaluation should identify its input rather than treating every probability-like number as interchangeable.

The analyst should state the chosen convention and apply it consistently.

Multiple observations from one contract should not silently be treated as independent forecasts. Use one standardized snapshot per contract, report separate horizons, or employ an analysis that explicitly handles repeated observations. The weighting rule should also be clear so that frequently sampled contracts do not gain influence by accident.

Finally, forecast accuracy and trading profitability are separate questions. A statistical evaluation of probabilities does not establish that an actionable or profitable strategy existed. This framework is educational, not trading advice.

How to make Brier-score comparisons fair

Raw Brier scores are most informative when calculated on the same—or closely comparable—sets of questions. Important dimensions include the contracts evaluated, time period, category, event prevalence, forecast horizon, sampling convention, resolution rules, and weighting method.

Question difficulty, the distribution of underlying risks, and random outcome variation all affect realized scores. A system forecasting many highly predictable events may obtain a lower score than one tackling genuinely uncertain questions, even if the latter uses available information more skillfully.

For binary outcomes, constant 50% forecasts always produce:

(0.5-y)² = 0.25

Thus, 0.25 is a convenient reference for a system that always says “even odds.” It is not a universal definition of random guessing or a universal line between good and bad forecasting. Depending on the evaluated questions, a historical event rate, category-specific base rate, or competing system tested on the same observations may be more informative (analysis of why Brier scores depend on the evaluated population).

For a fair head-to-head comparison, freeze the eligible contract set and timestamp rules, then evaluate each candidate system against the same resolved outcomes. Report the pooled score and, where observation counts permit, results by probability range, category, or horizon. Avoid ranking unrelated markets or forecasters solely from raw scores calculated on different datasets.

What a Kalshi case study illustrates—and what it does not

It reports an analysis of 594 selected Kalshi macroeconomic markets, comprising 1,860 observations across nine economic series from 2025 through June 2026. The reported overall Brier score is 0.0987, compared with 0.0812 at a one-hour horizon and 0.1587 at a 30-day horizon. The abstract also reports overconfidence and a bias of -0.0721 for contracts priced below $0.30 (SSRN abstract for the Kalshi study).

Taken at face value, those figures illustrate the central distinction: a favorable pooled score can coexist with localized bias in a particular price range. They also show why horizon-specific results can be more informative than a single aggregate number.

The findings are not independently verified. The abstract does not provide enough methodological detail to confirm the calculations, selection process, price convention, or statistical treatment of repeated observations. Its claimed improvement over “random guessing” should not be treated as established because the benchmark and calculation are not defined. Likewise, a very low Brier score for one economic series would not, by itself, prove near-perfect calibration.

The appropriate conclusion is narrow: prediction-market results should be examined by horizon, series, and probability range rather than reduced to one pooled score. This selected macroeconomic case study cannot establish the performance of Kalshi as a whole or of prediction markets generally.

Is a Brier score of 0.25 good or bad?

It depends on the questions and baseline. Constant 50% forecasts produce a Brier score of 0.25 for binary outcomes, making it a useful reference when the alternative is always predicting even odds.

It is not a universal quality threshold. A simple base-rate forecast may score much better than 0.25 on a dataset where one outcome is common without offering much event-specific insight. Judge a score against suitable baselines and competing forecasts evaluated on equivalent contracts and snapshots.

How many resolved forecasts are needed to assess calibration?

The requirement depends on the probability ranges examined, the distribution of forecasts and outcomes, the desired precision, and whether observations are independent.

Extreme-probability ranges may contain few instances of the rarer outcome, while dividing a modest dataset into many narrow buckets can leave each range too sparse for stable conclusions. Report counts and uncertainty for every range, disclose any rule used to combine ranges, and do not count repeated snapshots from one contract as automatically independent evidence.

The compact rule is: use the Brier score to summarize overall probability error and calibration analysis to identify where forecast probabilities diverge from observed frequencies. Compare systems on equivalent questions and snapshots, disclose resolution and sampling choices, and keep forecast accuracy separate from trading profitability.