3 min read ·
Forecast Accuracy Metrics: How to Evaluate YES/NO Odds
Choose Brier score, log loss, calibration and skill scores for YES/NO forecasts—and compare market odds without mistaking hit rate for accuracy.
For YES/NO prediction markets, start with Brier score for overall probability accuracy, a calibration curve for reliability, and a benchmark for context. Add log loss when you want a sharper penalty for confident mistakes. Hit rate alone cannot tell you whether the probabilities were good.
These metrics evaluate forecasts, not trading returns. First define p as the forecast probability of YES, between 0 and 1, and y as the resolved outcome: 1 for YES, 0 for NO. A quoted price is an input to that forecast, not a guarantee; see how market prices become probabilities.
Which metric answers which question?
Brier score and log loss are strictly proper scoring rules: reporting your true probability estimate minimizes expected loss. That makes them suitable for evaluating probabilities rather than just picking winners. Neither isolates calibration on its own. Scikit-learn’s calibration guide explains that these scores also reflect how well forecasts distinguish outcomes and the uncertainty in the sample.
| Metric | What it measures | How to read it |
|---|---|---|
| Brier score | Average squared error: mean of (p − y)² |
Lower is better. Under this binary convention, 0 is perfect and 1 is worst. Definition |
| Log loss | Average negative natural logarithm of the probability assigned to the outcome that occurred | Lower is better; 0 is perfect. It penalizes near-certain mistakes much more sharply than Brier score. Definition |
| Calibration curve | Average forecast probability versus observed YES frequency within probability buckets | Points near the diagonal indicate agreement. Show the number of observations in each bucket. Method |
| Brier skill score | Relative improvement over a specified reference: 1 − Brier / reference Brier |
Higher is better: 1 is perfect, 0 matches the reference, and negative values are worse. Requires a positive reference Brier score. Verification guidance |
Choose the scoring rule before comparing results. Brier gives each observation a bounded loss, making it a straightforward default. Log loss is useful when you especially want to expose forecasts that assign almost no chance to outcomes that subsequently occur.
For log loss, an exactly certain forecast that proves wrong has infinite mathematical loss. Software may clip probabilities to avoid that result; disclose the clipping rule when comparing implementations. Scikit-learn’s implementation, for example, clips at bounds determined by numerical precision.
Same hit rate, different forecast quality
Consider a hypothetical set of ten contracts, eight resolving YES and two NO.
A forecaster assigning 80% YES to every contract gets:
- Brier score:
(8 × 0.2² + 2 × 0.8²) / 10 = 0.16. - Log loss: approximately 0.500, using natural logarithms.
A second forecaster assigning 99% YES to every contract gets a Brier score of 0.1961 and log loss of approximately 0.929.
Both have an 80% hit rate if probabilities above 50% count as YES calls. Yet the second forecast performs worse on both probability scores: its extra confidence was not justified by these outcomes.
The first forecast also matches the observed YES frequency in this tiny sample. That illustrates calibration, but ten outcomes do not establish reliable long-run calibration. Nor does assigning 80% to every contract distinguish which events are more likely. For that distinction, see Brier score versus calibration.
Skill depends on the benchmark
A constant 50% forecast has a binary Brier score of 0.25, whatever the outcomes. Against that reference, the first forecast’s Brier skill score is 1 − 0.16 / 0.25 = 0.36: a 36% reduction in squared probability error, not “36% more correct predictions.”
That is a useful reference calculation, not a universal boundary between good and bad forecasts. If a defensible pre-event reference also assigned 80% to every contract, it would score 0.16 on this sample, making the first forecast’s skill score zero. The forecast beats even odds but does not beat that reference.
Name the benchmark and say how it was constructed. For an operational comparison, use a historical base rate available before forecasting or a competing forecast recorded at the same horizon—not a baseline presented as predictive despite using the test outcomes. Some software instead uses the observed class proportions of the evaluation sample as its reference; scikit-learn’s Brier skill implementation does this. Those are different comparisons.
Keep the contracts, resolution rules, timestamps, price fields and weighting consistent across candidates. Report sample dates and counts alongside scores; unrelated event mixes do not support a clean platform ranking. The forecast accuracy formula guide covers calculations and the market-data comparison checklist.