Cyan Xiao
Back to projects
•
3 min read
ApexBench

Disclaimer: Contains AI generated content.

Can an AI model tell when a squad is on its way to winning an Apex Legends match? ApexBench reconstructs the observations available at a moment in a historical game, asks a model for a probability for every final placement, then scores that forecast against the result.

This first experiment follows Alliance through a single Storm Point match. The squad ultimately finished first out of 20 teams. Five models made forecasts every 60 seconds, from the start of gameplay through 19:00, before the match ended at 19:17.

Single-match experiment · October 30, 2026

Five models. One winning squad.

Claude Opus 5.5 had the lowest Brier score across the 16 checkpoints where every model returned a valid forecast. Jev 1.14 cost the least at $0.0064 for its full run.

Checkpoints per model
20
Valid forecasts
96 / 100
Reported inference cost
$2.244

Forecast comparison

Lower scores are better

Ranked by Brier score on the same 16 valid checkpoints. Coverage and cost include all 20 attempts per model.

Alliance placement forecasting results. All models are scored on identical shared valid checkpoints.
ModelBrier ↓Log loss ↓ValidCost
Claude Opus 5.5Lowest Brier in this match 0.8552.23616/20$0.730
GPT-6.1 Sol 0.9162.58020/20$0.315
Grok 4.7 0.9232.76120/20$0.862
Gemini 4 Argon 0.9352.59820/20$0.330
Jev 1.14 0.9563.32420/20$0.0064
Uniform baselineAmong remaining squads0.9012.494—$0

The baseline assigns equal probability to each still-possible placement, from 1 to the number of squads remaining. It uses no model. Costs are in USD, including invalid responses, and exclude taxes and credit purchase fees.

What do the scores measure?

Brier score measures error across the full placement distribution: Σ(pₖ − 1[k = actual])², from 0 to 2. A perfect forecast scores 0.

Log loss is −ln(p of the actual placement). It penalizes assigning very little probability to the outcome that happened. Probabilities below 10⁻¹⁵ use that floor.

Every shared checkpoint has equal weight. Missing and invalid forecasts are excluded from scores and remain visible in coverage.

When did they see the win?

Win probability · 0–40%

Probability of Alliance finishing first, at every scheduled checkpoint. Gaps indicate unavailable forecasts. This chart includes every valid forecast, including checkpoints excluded from the shared score comparison.

Alliance's predicted win probability through the match Five model series and a uniform baseline from 0:00 to 19:00, with a y-axis from 0 to 40 percent. Alliance won the match. Exact values are available in the checkpoint table. 0%10%20%30%40% 0:005:0010:0015:0019:00 Claude Opus 5.5 at 0:00: 7.0% chance of first placeClaude Opus 5.5 at 1:00: 8.5% chance of first placeClaude Opus 5.5 at 2:00: 7.5% chance of first placeClaude Opus 5.5 at 3:00: 7.5% chance of first placeClaude Opus 5.5 at 4:00: 7.5% chance of first placeClaude Opus 5.5 at 5:00: 8.0% chance of first placeClaude Opus 5.5 at 9:00: 6.0% chance of first placeClaude Opus 5.5 at 10:00: 5.0% chance of first placeClaude Opus 5.5 at 11:00: 6.0% chance of first placeClaude Opus 5.5 at 13:00: 13.0% chance of first placeClaude Opus 5.5 at 14:00: 13.0% chance of first placeClaude Opus 5.5 at 15:00: 15.0% chance of first placeClaude Opus 5.5 at 16:00: 18.0% chance of first placeClaude Opus 5.5 at 17:00: 22.0% chance of first placeClaude Opus 5.5 at 18:00: 32.0% chance of first placeClaude Opus 5.5 at 19:00: 25.0% chance of first place GPT-6.1 Sol at 0:00: 5.0% chance of first placeGPT-6.1 Sol at 1:00: 5.0% chance of first placeGPT-6.1 Sol at 2:00: 4.7% chance of first placeGPT-6.1 Sol at 3:00: 4.5% chance of first placeGPT-6.1 Sol at 4:00: 4.8% chance of first placeGPT-6.1 Sol at 5:00: 4.5% chance of first placeGPT-6.1 Sol at 6:00: 4.7% chance of first placeGPT-6.1 Sol at 7:00: 3.8% chance of first placeGPT-6.1 Sol at 8:00: 2.5% chance of first placeGPT-6.1 Sol at 9:00: 3.5% chance of first placeGPT-6.1 Sol at 10:00: 4.9% chance of first placeGPT-6.1 Sol at 11:00: 4.2% chance of first placeGPT-6.1 Sol at 12:00: 5.0% chance of first placeGPT-6.1 Sol at 13:00: 11.5% chance of first placeGPT-6.1 Sol at 14:00: 11.5% chance of first placeGPT-6.1 Sol at 15:00: 10.5% chance of first placeGPT-6.1 Sol at 16:00: 11.5% chance of first placeGPT-6.1 Sol at 17:00: 17.0% chance of first placeGPT-6.1 Sol at 18:00: 24.0% chance of first placeGPT-6.1 Sol at 19:00: 22.0% chance of first place Grok 4.7 at 0:00: 5.0% chance of first placeGrok 4.7 at 1:00: 4.2% chance of first placeGrok 4.7 at 2:00: 3.2% chance of first placeGrok 4.7 at 3:00: 4.5% chance of first placeGrok 4.7 at 4:00: 2.6% chance of first placeGrok 4.7 at 5:00: 4.3% chance of first placeGrok 4.7 at 6:00: 3.8% chance of first placeGrok 4.7 at 7:00: 2.0% chance of first placeGrok 4.7 at 8:00: 0.8% chance of first placeGrok 4.7 at 9:00: 2.0% chance of first placeGrok 4.7 at 10:00: 2.0% chance of first placeGrok 4.7 at 11:00: 4.3% chance of first placeGrok 4.7 at 12:00: 3.2% chance of first placeGrok 4.7 at 13:00: 10.5% chance of first placeGrok 4.7 at 14:00: 11.2% chance of first placeGrok 4.7 at 15:00: 6.2% chance of first placeGrok 4.7 at 16:00: 12.5% chance of first placeGrok 4.7 at 17:00: 16.0% chance of first placeGrok 4.7 at 18:00: 28.0% chance of first placeGrok 4.7 at 19:00: 29.0% chance of first place Gemini 4 Argon at 0:00: 5.0% chance of first placeGemini 4 Argon at 1:00: 5.2% chance of first placeGemini 4 Argon at 2:00: 5.2% chance of first placeGemini 4 Argon at 3:00: 5.4% chance of first placeGemini 4 Argon at 4:00: 5.6% chance of first placeGemini 4 Argon at 5:00: 5.4% chance of first placeGemini 4 Argon at 6:00: 5.6% chance of first placeGemini 4 Argon at 7:00: 5.2% chance of first placeGemini 4 Argon at 8:00: 3.8% chance of first placeGemini 4 Argon at 9:00: 4.2% chance of first placeGemini 4 Argon at 10:00: 7.5% chance of first placeGemini 4 Argon at 11:00: 4.5% chance of first placeGemini 4 Argon at 12:00: 5.5% chance of first placeGemini 4 Argon at 13:00: 10.5% chance of first placeGemini 4 Argon at 14:00: 10.5% chance of first placeGemini 4 Argon at 15:00: 6.0% chance of first placeGemini 4 Argon at 16:00: 10.5% chance of first placeGemini 4 Argon at 17:00: 11.5% chance of first placeGemini 4 Argon at 18:00: 22.0% chance of first placeGemini 4 Argon at 19:00: 16.0% chance of first place Jev 1.14 at 0:00: 1.0% chance of first placeJev 1.14 at 1:00: 1.0% chance of first placeJev 1.14 at 2:00: 2.0% chance of first placeJev 1.14 at 3:00: 2.0% chance of first placeJev 1.14 at 4:00: 1.0% chance of first placeJev 1.14 at 5:00: 1.0% chance of first placeJev 1.14 at 6:00: 2.0% chance of first placeJev 1.14 at 7:00: 1.0% chance of first placeJev 1.14 at 8:00: 1.0% chance of first placeJev 1.14 at 9:00: 1.0% chance of first placeJev 1.14 at 10:00: 2.0% chance of first placeJev 1.14 at 11:00: 2.0% chance of first placeJev 1.14 at 12:00: 2.0% chance of first placeJev 1.14 at 13:00: 9.0% chance of first placeJev 1.14 at 14:00: 8.0% chance of first placeJev 1.14 at 15:00: 12.0% chance of first placeJev 1.14 at 16:00: 7.0% chance of first placeJev 1.14 at 17:00: 12.0% chance of first placeJev 1.14 at 18:00: 30.0% chance of first placeJev 1.14 at 19:00: 23.0% chance of first place
Uniform baseline

Inspect a checkpoint

0:00
0:0019:00

20 squads remainingIncluded in shared scores

ModelWin chanceExpected placeBrier
Claude Opus 5.57.0%9.320.912
GPT-6.1 Sol5.0%10.500.950
Grok 4.75.0%10.500.950
Gemini 4 Argon5.0%10.500.950
Jev 1.141.0%13.911.099

Invalid responses are part of the result

4 of 100 attempts produced no valid forecast. They were neither rescaled nor retried. All reported charges are included above.

View the 4 failed checkpoints
  • Claude Opus 5.5 at 6:00: placement probabilities do not sum to 1 (sum: 1.023000).
  • Claude Opus 5.5 at 7:00: placement probabilities do not sum to 1 (sum: 0.975000).
  • Claude Opus 5.5 at 8:00: placement probabilities do not sum to 1 (sum: 0.955000).
  • Claude Opus 5.5 at 12:00: placement probabilities do not sum to 1 (sum: 1.030000).

What the models could see

Every model received the same reconstructed team-mode observations: Alliance’s last observed positions, health, shields, weapons, survival state, combat statistics, and timestamped legend identities; public kill counts, squad eliminations, and observed ring updates were also available. Opponent private telemetry was withheld.

Final placements, winner metadata, cumulative match-summary statistics, future events, and the exact match end time were excluded from prediction inputs. Forecasts at or after a resolved outcome were ineligible. Team and player names remained visible, so recognition of a historical match is a possible source of contamination.

How to read this experiment

This is a case study of one team in one match. The checkpoints share the same outcome and are correlated; the results do not establish general model quality or calibration. An accurate forecast in this game does not guarantee the same performance across maps, tournaments, or teams.

The generative models used their provider-default reasoning settings. Jev used the native Decisions API, while the other models generated probabilities under a JSON Schema. These are different inference methods and reasoning budgets. The comparison uses the same observation policy and scoring rules, rather than equal compute.

One request was made per model per checkpoint. Invalid probability distributions were retained as failures, without normalization or retries. Scores in the main table use only checkpoints with valid forecasts from all five models; valid coverage and the cost of every attempt are reported separately. See the benchmark methodology and source code.