Forecasting events with several language models: calibration and Brier scoring

The quality bar was set before the first forecast

The Brier score is the squared difference between the stated probability and the outcome (1 or 0). Zero is a perfect forecast, always saying 50% scores exactly 0.25, and a confident miss costs close to 1. The same number is calculated for the market price at the time of the forecast, which gives a natural benchmark: the market is a forecaster too. The bar was written into the report code before the first forecast was saved:

• at least 30 resolved markets, and the system's Brier score lower than the market's
• until both conditions are met, trading with real money is forbidden, and the system runs in paper mode
• the paper result is calculated at the ask price with a 0.015 half-spread, so the paper run can't flatter itself
• the geopolitics category was chosen partly because the platform charges no fee on it, so the spread is the only trading cost in the model

Two ways to make a forecast: through the API, or with the model as an operator

The full Python pipeline calls the model through the API: five agents in parallel, then the supervisor. The server had no API key, so the second way became the working one. A Claude Code session with web search does the research, following a recipe written up as a Claude Code skill. A recording script then takes its raw probability, fetches the live market price itself, applies the calibration, calculates the gap and saves the row. A daily batch looks like this: a queue of 13–14 markets, 8–14 Claude Sonnet sub-agents run through a workflow, then the operator's review before anything is saved.

The split of roles is deliberate. The model does the reasoning and the search. Python handles everything that has to survive a change of model or prompt: finding markets, the database, calibration, tracking outcomes and the report.

The outcome counter stayed silent for seven weeks

The first real failure happened in the most important place. The platform's public API stopped returning closed markets at the address the system used to check outcomes: the response came back empty, with no error. The outcomes table stayed empty from late May until 16 July, the report honestly printed "no resolved forecasts yet", and it all looked like slow markets. It's the same kind of failure as in the case on reliability of background services: the process is alive, data stops arriving, and nobody complains.

The fix: outcome checks moved to the exchange's core API, where each market is looked up by its condition ID and every outcome has an explicit winner flag. The lookup by market address stayed as a backup. The very first run recorded 4 outcomes: the system's Brier score was 0.499 against the market's 0.235. The market led from the first row, and from that day the measurement ran daily.

Every row keeps what's needed to recalculate it

Each forecast stores the agents' raw probability, the calibrated one, the blend with the price, the price itself, the number of agents, how far they agreed, the direction of the signal, the reasoning and the model version. That later decided the project's course: when the question came up whether Platt extremization helps, the answer was calculated from the rows already saved, with no re-runs. The rules for keeping the queue clean came from incidents:

• a market enters the queue only if the market search saw it within the last 48 hours: May markets with a frozen "1.6 days to resolve" were still in the queue in July
• the gap is calculated only against the live price: a price stored in the queue was once ten times off the real one (0.04 against 0.389)
• the search window was cut from 120 to 45 days: short-dated markets produce resolved pairs faster

Where exactly the accuracy is lost

The pattern holds from one check to the next: the system beats the price on most markets and loses on the total. At n = 66, four misses made up 84% of the whole gap (3.646 of 4.322); without them the score was 0.068 against 0.057 on 62 pairs, close to even. Three of the four were confident disagreements with the price by more than 75 points, and in all four the market was right. Two misses came from the same question template about warships passing through a strait: raw 0.98 and 0.82 against prices of 0.010 and 0.037, and both resolved "no". Another mixed up one conflict with a different one: 0.03 against 0.895, resolved "yes". A breakdown by price range at n = 203 showed where the losses are concentrated:

• price below 0.10: n = 132, the system's Brier 0.042 against 0.022, with an actual rate of 2.3%: the system overrated unlikely events
• the middle, 0.25–0.75: n = 20, 0.297 against 0.220: the one range where disagreeing with the price clearly costs accuracy
• price above 0.90: n = 21, 0.048 against 0.047: level with the market, nothing to fix

The paper's correction made the score worse

Platt extremization pushes probabilities toward the edges: the log-odds are multiplied by 1.73. In the paper it improved the Brier score because models tend to cluster around 50%. On this series it worked the other way. The same rows show it, since the raw probability is stored separately: at n = 53 the raw Brier score was 0.108, with the correction 0.115, and the market's 0.048. At the extremes the correction flips the direction of the gap. A raw 0.13 at a price of 0.07 means "the forecast is above the market". After stretching it becomes 0.036, a signal in the opposite direction. In one batch of 13 markets the direction flipped on 9.

The practical result: the signal field in the database can't be trusted for very cheap and very expensive markets, and the gap is read only as "raw probability minus price". The general lesson for any AI evaluation: check a coefficient from a paper on your own outcomes, and store the value before correction.

What changed after the checks

Every change to the recipe was dated and then measured at the next checks, on the same table:

• The market price is hidden from the agent. Despite an explicit ban in the prompt, all five forecasts on one series of dates landed within 1.5 points of the price, and the reasoning said so openly. Since 23 August an agent gets only the question, the rules and the time left
• One family of markets, one agent. Independent agents gave logically impossible pairs on series of dates: 0.25 for "by 31 August" and 0.15 for "by 15 September". A series is now built from a single survival model, and an arbiter checks that the probability never falls for a later date
• A sceptic pass and an arbiter pass. When the gap from the price reaches 15 points, a second agent tries to disprove the key facts: on 29 August it rewrote 5 of 14 forecasts. From 20 points, a third agent plays "the person who will resolve this market" and goes through the rules clause by clause: on 1 September it pulled 3 of 5 extreme forecasts back toward the middle (0.94 → 0.33; 0.93 → 0.32; 0.78 → 0.55) and confirmed two
• Base rates by question family, from the outcomes table. "Ceasefire holds until a date": 14 of 14 "yes" against the system's average of 0.847. "Town captured by a date": 0 of 4 against the system's 0.268 and a price of 0.095. "Blockade lifted by a date": 10 of 10 "no", with the system at 0.011 against a price of 0.101, and a Brier score of 0.0003 against 0.0161. That is the one family where the system is clearly more accurate than the market

The effect shows in the same price ranges. The cheap end stopped getting worse: from n = 183 to n = 203 it gained 12 outcomes, and its total gap moved from 2.605 to 2.579; the overrating of unlikely events fell from 2.3× to 1.7× the actual rate. The 0.25–0.75 middle is still losing: two new outcomes added 0.371 to the gap. That's where the recording rules now point: when the market price is in the middle, the forecast is recorded more cautiously than the agents' conclusion.

A related experiment: weather markets and numerical forecasts

In parallel, the same discipline was applied to daily temperature markets, where the probability came from a set of numerical weather forecasts (30–50 runs). The first lesson there: a win rate on its own says little. A bot that won 91.6% of 1,089 markets still lost money: it bought favourites at a high price, so one loss wiped out many wins. The most expensive range showed it clearly: at an average price of 0.995 you need to win 99.5% to break even, and the actual 97.54% over 285 markets fell short. The second lesson: the bot picked favourites by price and never looked at the forecast. The forecast filter was connected on 31 May, first in a watch-only mode where it just logged its decision. Minimum-temperature markets were dropped after a 61% win rate, and the platform's fee, once discovered, was added to the paper accounting. On 28 July the experiment was closed by its own A/B test: the tested versions lost money on average.