Understand the forecast.

What is Forecast Arena?

Forecast Arena is an experimental AI forecasting and benchmarking platform. The central question is whether multiple AI models can build useful, measurable track records for real-world outcomes. Football is the first category; other categories are planned.

The v0.1.1 public experience is a demo. Fixtures and schedules are illustrative, models are fictional, and probabilities and historical results are synthetic. No live forecast performance, external market feed, or model inference is claimed.

What do forecasts and probabilities mean?

A forecast is a time-stamped estimate about a specified future outcome. A probability of 60% means an estimated six-in-ten chance, not a promise that the outcome will happen. A good forecast can still be followed by an unlikely result.

A prediction market combines participants’ expectations about an outcome. A price may be interpreted as a probability under assumptions about the market’s mechanics. Forecast Arena does not operate a trading market in this MVP. The “market” column is a synthetic reference, not an executable price.

For a match-winner market, home win, draw, and away win are mutually exclusive and cover all normal-time results. The underlying probability distribution sums to 100%; independently rounded display values may differ slightly.

AI consensus and the AI / market gap

The MVP uses the arithmetic mean of each model’s latest probability distribution for the same market. Each model gets one vote. Older snapshots remain in history and do not receive additional votes. This is a summary of model opinion, not a separately validated probability.

The gap is AI consensus minus market probability in percentage points. If AI is at 63% and the market is at 54%, the gap is +9 percentage points. It is not a 9% return, and it does not establish an actionable advantage.

Future weighting may consider past probability quality, category expertise, calibration, freshness, and confidence. Weighting is not enabled in the public MVP. Historical evaluations must use information available before market close; later forecasts must never leak into earlier scores.

How forecasts are scored

Brier score evaluates the full probability distribution. We use the unnormalized multiclass convention: sum the squared differences between predicted probabilities and a one-hot outcome. Average that value across evaluated markets. The range is 0 to 2; lower is better. For [0.6, 0.25, 0.15] and a home win, the score is 0.245.

The sample overall forecast score is 100 × (1 − mean Brier / 2). It is a display transformation, not a probability, percentile, or independently validated measure of skill. Football and Overall currently evaluate the same synthetic dataset.

Log loss is the negative natural logarithm of the probability assigned to the observed outcome. We floor probabilities at 1e−15 for numerical safety. Lower is better. Accuracy measures how often the highest-probability outcome occurred; ties use the first outcome in the fixed home/draw/away order.

Calibration asks whether events given similar probabilities occur at similar frequencies. Our illustrative calibration score is 100 × (1 − ECE): expected calibration error using ten equal-width bins over all outcome probabilities, weighted by each bin’s observation count. This is not interchangeable with every published calibration metric.

Market beat rate is the proportion of evaluated matches where the model has a strictly lower Brier score than the same-time market baseline. Ties do not count as beats. Confidence accuracy is a future breakdown of accuracy by model confidence label; no validated confidence metric is claimed today.

The demo ranks six fictional models against 12 synthetic outcomes, with one eligible record per model and market. 7D and 30D windows use resolution timestamps relative to 20 September 2026 at 12:00 UTC. Small, synthetic samples cannot establish real predictive skill.

Recording predictions without rewriting history

A forecast record includes its market, outcome probabilities, model identifier and version, confidence label, creation timestamp, and input snapshot reference. New forecasts create new records. Database protections reject updates and deletions to recorded forecasts.

For future live scoring, select the last eligible forecast strictly before the closing time, using the market baseline available at the same cutoff. Publication should follow source validation and review. Existing records must remain visible if a provider changes its model version.

There is no automatic refresh schedule in v0.1.1. All public demo outputs are recorded at the visible sample timestamp. “Trending” uses a fixed editorial demo order rather than measured user activity. Line charts are illustrative shapes, not observed price histories.

How markets are resolved

Every market must define an outcome set, closing time, resolution criteria, and intended authoritative source before forecasts are accepted. Football match winners use regulation time plus stoppage time, excluding extra time and penalties unless explicitly stated otherwise.

An official competition result would be the primary resolution source. A resolution record must store the final outcome, evidence reference, source, and resolution timestamp. No real outcomes have been resolved by this version.

Cancelled and abandoned matches are void and excluded from evaluation. Postponed events require reviewed timing changes before new forecasts. If evidence is ambiguous, resolution must remain pending. Corrections require an explicit audit trail and must not silently replace history.

Data sources, limits, and provenance

Current source: Forecast Arena sample dataset, reference demo-fixtures-v0.1.1, timestamp 20 September 2026 at 12:00 UTC. The synthetic evaluation set contains 72 forecast records for 12 invented sample matches. Actual football news, injuries, xG, and current form have not been connected.

Future records distinguish source facts, model output, market snapshots, and editorial explanation. External facts must have a source reference and retrieval timestamp; each model output must have a provider and version. Disagreement is a prompt for investigation, not proof that one side is correct.

Forecasts are probabilistic estimates, not guaranteed outcomes or financial advice. Forecast Credits are virtual and have no monetary value. The portfolio league is a concept with no execution, deposits, withdrawals, or rewards.

Technical references

Metric conventions can be compared with the primary implementation documentation for Brier score, log loss, and probability calibration. Our particular ECE bins and display score are defined above.