Event forecasting with an LLM ensemble: calibration and Brier scoring

The gate was set before the first forecast

The Brier score is the squared difference between the stated probability and the outcome (1 or 0). Zero means a perfect forecast, a permanent 50% scores exactly 0.25, and a confident miss costs close to one. The same quantity is computed for the market price at forecast time, which gives a natural benchmark: the market is a forecaster too. The gate was written into the report code before the first forecast was recorded:

• at least 30 resolved markets and our Brier below the market's
• until both conditions hold, real-money trading is forbidden; the system lives in paper mode
• the paper result is computed at the ask with a 0.015 half-spread, so the paper run cannot flatter itself
• the geopolitics category was picked partly because the venue charges it no fee: the spread stays the only execution cost in the model

Two paths to a forecast: the model over the API, and the model as operator

The full Python pipeline calls the model over the API: five agents in parallel, then the supervisor. The server had no API key, so the second path became the working one: a Claude Code session with web search does the research by the recipe written into a skill, and the recording script takes its raw probability, fetches the live market price itself, applies the calibration, computes the divergence and writes the row. A daily batch looks like this: a queue of 13–14 markets, 8–14 Claude Sonnet sub-agents through a workflow, then the operator's arbitration before anything is recorded.

The split of roles is deliberate. The model owns reasoning and search. Python owns what has to survive a change of model or prompt: discovery, the database, calibration, outcome tracking and the report.

The outcome counter stayed silent for seven weeks

The first real failure happened in the most important place. The venue's public API stopped returning closed markets on the endpoint the system used to check outcomes: the response came back empty, with no error. The outcomes table stayed empty from late May until 16 July, the report honestly printed "no resolved forecasts yet", and it all looked like slow markets. Same failure class as in the case on reliability of background services: the process is alive, the data stops arriving, nobody complains.

The fix: outcome checks moved to the exchange-core endpoint keyed by condition id, where every outcome carries an explicit winner flag; the lookup by market slug stayed as a fallback. The very first run recorded 4 outcomes: our Brier 0.499 against the market's 0.235. The market led from the first row, and from that day the measurement became daily.

Every row keeps what a recomputation will need

Each forecast stores the ensemble's raw probability, the calibrated one, the blend with the price, the price itself, the number of agents, their agreement, the signal side, the reasoning and the model version. That decided the project's fate later: when the question came up whether Platt extremization helps, the answer was computed from the rows already accumulated, with no re-runs. Queue hygiene rules came from incidents:

• a market enters the queue only if discovery saw it within the last 48 hours: May markets with a frozen "1.6 days to resolve" were still sitting in the queue in July
• divergence is computed against the live price only: a queued price once lagged the real one tenfold (0.04 against 0.389)
• the discovery horizon was cut from 120 to 45 days: short-dated markets produce resolved pairs faster

Where exactly the accuracy leaks

The signature is stable from checkpoint to checkpoint: on most markets we beat the price, on the total we lose. At n = 66, four misses produced 84% of the entire deficit (3.646 of 4.322); without them, 0.068 against 0.057 on 62 pairs, close to level. Three of the four were confident disagreements with the price of more than 75 points, and in all four the market was right. Two misses share one question template about warships transiting a strait: raw 0.98 and 0.82 against prices of 0.010 and 0.037, both resolved "no". Another confused one conflict with a different one: 0.03 against 0.895, resolved "yes". A breakdown by price bucket at n = 203 showed the leak is concentrated:

• price below 0.10 — n = 132, our Brier 0.042 against 0.022, with a realised base rate of 2.3%: we overestimated rare events
• the middle, 0.25–0.75 — n = 20, 0.297 against 0.220: the one bucket where our disagreement with the price hurts statistically
• price above 0.90 — n = 21, 0.048 against 0.047: level with the market, nothing to fix

The paper's correction made the score worse

Platt extremization stretches probabilities toward the edges: the log-odds are multiplied by 1.73. In the paper it improved the Brier score because models tend to huddle around 50%. On our series the effect ran the other way, and it is visible in the same rows where the raw probability is stored separately: at n = 53 the raw Brier was 0.108, with the correction 0.115, the market's 0.048. In the tails the correction flips the sign of the divergence: raw 0.13 at a price of 0.07 means "we are above the market"; after stretching it becomes 0.036 and a signal in the opposite direction. In one batch of 13 markets the sign flipped on 9.

The practical consequence: the signal field in the database is unreliable for cheap and expensive markets, and divergence is read only as "raw probability minus price". The general lesson for any AI evaluation: a coefficient from a paper gets verified on your own outcomes, and the pre-correction value is what you store.

What changed after the checkpoints

Every change to the recipe was dated and verified on the following checkpoints, over the same table:

• The market price is hidden from the agent. Despite an explicit ban in the prompt, all five forecasts on one date ladder landed within 1.5 points of the price, and the reasoning said so in plain words. Since 23 August an agent receives only the question, the criteria and the horizon
• One market family, one agent. Independent agents produced logically impossible pairs on date ladders: 0.25 for "by 31 August" and 0.15 for "by 15 September". A ladder is now derived from a single survival model, and an arbiter checks monotonicity
• A skeptic pass and an arbiter pass. When the divergence from the price reaches 15 points, a second agent tries to refute the load-bearing facts: on 29 August it rewrote 5 of 14 forecasts. From 20 points, a third agent in the role of "the person who will resolve this market" walks the criteria clause by clause: on 1 September it flipped 3 tails out of 5 (0.94 → 0.33; 0.93 → 0.32; 0.78 → 0.55) and confirmed two
• Family base rates from the outcomes table. "Ceasefire holds through date" — 14 of 14 "yes" against our average of 0.847; "town captured by date" — 0 of 4 against our 0.268 and a price of 0.095; "blockade lifted by date" — 10 of 10 "no", ours 0.011 against a price of 0.101, Brier 0.0003 against 0.0161, the one family where we are confidently more accurate than the market

The effect shows up in the same buckets. The cheap tail stopped growing: from n = 183 to n = 203 it gained 12 outcomes, and its cumulative gap moved from 2.605 to 2.579; the overestimation of rare events shrank from 2.3× to 1.7× the realised rate. The 0.25–0.75 middle keeps leaking: two new outcomes added 0.371 to the gap. That is where the recording discipline now points: a forecast whose market price falls in the middle is recorded more conservatively than the agents' conclusion.

A sibling line: a numerical ensemble on weather markets

In parallel, the same discipline was applied to daily-temperature markets, where the probability came from an ensemble of numerical weather forecasts (30–50 members). The first lesson from there: a win rate on its own means nothing. A bot with a 91.6% win rate over 1,089 markets stayed in the red: buying favourites at a high price, one loss eats many wins. The extreme bucket showed it cleanly: at an average price of 0.995 the break-even is 99.5%, and the actual 97.54% over 285 markets fell short. The second lesson: the bot picked favourites by price and never looked at the forecast at all; the ensemble gate was connected on 31 May, first in an observe mode where it only logged its decision. Minimum-temperature markets were excluded after a 61% win rate, and the venue fee, once discovered, was added to the paper accounting. On 28 July the line was closed by its own A/B gate: negative expectancy in the tested variants.