Evaluation / common state
Diagnostics
Every model receives identical frozen, actor-visible game states through the normal decision adapter. Exact action values score those decisions separately from full-game Edge. Calibration cases can be inspected after reveal; sealed cases remain hidden. This board is not part of the Ante Index.
Case sets
How cases are selectedseason-3 · 81c80750d07d4ceeb0061a2104b489fe
View boardCalibration not yet revealed · seed 0 · cap 50 per stratum
Manifest SHA-256: f7b1570122a9e8673f9140c169b4b11d397835122cc2ddd0c99bb23c3d1248c0
| Game | total | calibration | sealed | easy | borderline | costly |
|---|---|---|---|---|---|---|
| Craps | 149 | 30 | 119 | 149 | 149 | 149 |
| Blackjack | 148 | 29 | 119 | 82 | 51 | 92 |
| Video Poker | 150 | 30 | 120 | 82 | 50 | 63 |
| Blackjack Team Play | 148 | 30 | 118 | 88 | 51 | 84 |
| Showdown Hold’em | 150 | 30 | 120 | 100 | 50 | 100 |
season-2 · 3c052a14054f4699815a8adb21e1c36f
View boardCalibration not yet revealed · seed 0 · cap 50 per stratum
Manifest SHA-256: 1e91540d8bd616eafc638eaf8528e0c72a00ef33cc83e6fff83bb8789e7fc760
| Game | total | calibration | sealed | easy | borderline | costly |
|---|---|---|---|---|---|---|
| Craps | 149 | 30 | 119 | 149 | 149 | 149 |
| Blackjack | 143 | 29 | 114 | 85 | 50 | 92 |
| Video Poker | 147 | 29 | 118 | 71 | 50 | 66 |
| Blackjack Team Play | 148 | 30 | 118 | 87 | 51 | 92 |
| Showdown Hold’em | 147 | 29 | 118 | 97 | 50 | 97 |
Common-state board
Lower mean loss is better. Intervals are 95% case bootstrap intervals.
Results for Season 3 settings.
Scroll the table sideways to see all metrics →
| Participant | Cases | Mean loss | Loss / random | Optimal | Valid | Quality · all | Quality · valid | Coverage | Latency p50 | Latency p95 | Mean cost |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Claude Sonnet 5.5Claude | 119/1190 transport failures | 0.0020.000–0.004 | 0.0040.000–0.009 | 96.6%93.3%–99.2% | 100.0%100.0%–100.0% | 99.6%99.1%–100.0% | 99.6%99.1%–100.0% | 100.0%100.0%–100.0% | 1,847 ms1,743 ms–1,914 ms | 2,901 ms2,407 ms–5,715 ms | $0.0038$0.0038–$0.0038 |
| Claude Mythos 5.1Claude | 119/1190 transport failures | 0.0020.000–0.004 | 0.0040.000–0.010 | 95.8%92.4%–99.2% | 100.0%100.0%–100.0% | 99.6%99.0%–100.0% | 99.6%99.0%–100.0% | 100.0%100.0%–100.0% | 1,597 ms1,554 ms–1,631 ms | 3,392 ms2,479 ms–3,781 ms | $0.0186$0.0185–$0.0188 |
| Gemini 3.8 FlashGemini | 119/1190 transport failures | 0.0020.000–0.004 | 0.0040.000–0.010 | 95.8%92.4%–99.2% | 100.0%100.0%–100.0% | 99.6%99.0%–100.0% | 99.6%99.0%–100.0% | 100.0%100.0%–100.0% | 2,391 ms2,275 ms–2,589 ms | 3,713 ms3,304 ms–4,474 ms | $0.0027$0.0026–$0.0028 |
| GPT-6 SolGPT | 119/1190 transport failures | 0.0020.000–0.004 | 0.0040.000–0.010 | 95.8%92.4%–99.2% | 100.0%100.0%–100.0% | 99.6%99.0%–100.0% | 99.6%99.0%–100.0% | 100.0%100.0%–100.0% | 2,088 ms2,047 ms–2,126 ms | 2,737 ms2,433 ms–3,414 ms | $0.0023$0.0023–$0.0023 |
| GPT-6.1 SolGPT | 119/1190 transport failures | 0.0020.000–0.004 | 0.0040.000–0.010 | 95.8%92.4%–99.2% | 100.0%100.0%–100.0% | 99.6%99.0%–100.0% | 99.6%99.0%–100.0% | 100.0%100.0%–100.0% | 2,750 ms2,690 ms–2,819 ms | 3,659 ms3,483 ms–7,741 ms | $0.0023$0.0023–$0.0024 |
| Muse WonderMuse | 119/1190 transport failures | 0.0020.000–0.004 | 0.0040.000–0.010 | 95.8%92.4%–99.2% | 100.0%100.0%–100.0% | 99.6%99.0%–100.0% | 99.6%99.0%–100.0% | 100.0%100.0%–100.0% | 2,514 ms2,420 ms–2,611 ms | 4,480 ms3,667 ms–6,326 ms | $0.0057$0.0055–$0.0059 |
| GPT-6 AstraGPT | 119/1190 transport failures | 0.0020.000–0.004 | 0.0050.001–0.010 | 95.0%90.8%–98.3% | 100.0%100.0%–100.0% | 99.5%99.0%–99.9% | 99.5%99.0%–99.9% | 100.0%100.0%–100.0% | 3,122 ms3,002 ms–3,248 ms | 4,158 ms3,943 ms–6,581 ms | $0.0117$0.0114–$0.0123 |
| GPT-6 LunaGPT | 119/1190 transport failures | 0.0020.000–0.004 | 0.0050.001–0.011 | 95.0%90.8%–98.3% | 100.0%100.0%–100.0% | 99.5%98.9%–99.9% | 99.5%98.9%–99.9% | 100.0%100.0%–100.0% | 2,530 ms2,413 ms–2,623 ms | 3,650 ms3,262 ms–4,139 ms | $0.0001$0.0001–$0.0001 |
| Claude Opus 5.5Claude | 119/1190 transport failures | 0.0020.000–0.005 | 0.0060.001–0.011 | 94.1%89.9%–98.3% | 100.0%100.0%–100.0% | 99.4%98.9%–99.9% | 99.4%98.9%–99.9% | 100.0%100.0%–100.0% | 2,382 ms2,264 ms–2,549 ms | 3,727 ms3,093 ms–5,051 ms | $0.0076$0.0075–$0.0078 |
| DeepSeek V4.1 FlashDeepSeek | 119/1190 transport failures | 0.0030.000–0.005 | 0.0060.001–0.013 | 95.0%90.8%–98.3% | 100.0%100.0%–100.0% | 99.4%98.7%–99.9% | 99.4%98.7%–99.9% | 100.0%100.0%–100.0% | 1,843 ms1,699 ms–2,187 ms | 4,060 ms3,309 ms–4,996 ms | $0.0002$0.0002–$0.0002 |
| DeepSeek V4 ProDeepSeek | 119/1190 transport failures | 0.0030.000–0.005 | 0.0060.001–0.013 | 95.0%90.8%–98.3% | 100.0%100.0%–100.0% | 99.4%98.7%–99.9% | 99.4%98.7%–99.9% | 100.0%100.0%–100.0% | 5,706 ms5,079 ms–6,101 ms | 13,148 ms10,311 ms–15,415 ms | $0.0017$0.0015–$0.0018 |
| Grok 4.7Grok | 119/1190 transport failures | 0.0050.001–0.011 | 0.0120.003–0.025 | 93.3%88.2%–97.5% | 100.0%100.0%–100.0% | 98.8%97.5%–99.7% | 98.8%97.5%–99.7% | 100.0%100.0%–100.0% | 2,678 ms2,558 ms–2,824 ms | 4,596 ms4,037 ms–7,524 ms | $0.0034$0.0033–$0.0035 |
| GLM 5.3 FlashGLM | 119/1190 transport failures | 0.0080.003–0.013 | 0.0180.008–0.033 | 88.2%82.4%–94.1% | 100.0%100.0%–100.0% | 98.2%96.7%–99.2% | 98.2%96.7%–99.2% | 100.0%100.0%–100.0% | 1,068 ms940 ms–1,453 ms | 7,844 ms4,834 ms–12,320 ms | $0.0001$0.0001–$0.0001 |
| Claude Haiku 5.5Claude | 119/1190 transport failures | 0.0080.001–0.018 | 0.0190.002–0.045 | 94.1%89.9%–97.5% | 100.0%100.0%–100.0% | 98.1%95.5%–99.8% | 98.1%95.5%–99.8% | 100.0%100.0%–100.0% | 1,666 ms1,585 ms–1,778 ms | 2,458 ms2,249 ms–4,050 ms | $0.0002$0.0002–$0.0002 |
| Claude Sonnet 5Claude | 119/1190 transport failures | 0.0130.004–0.025 | 0.0300.009–0.063 | 88.2%82.4%–93.3% | 100.0%100.0%–100.0% | 97.0%93.7%–99.1% | 97.0%93.7%–99.1% | 100.0%100.0%–100.0% | 1,673 ms1,652 ms–1,718 ms | 2,264 ms2,107 ms–2,474 ms | $0.0037$0.0036–$0.0038 |
| GPT-5.6 TerraGPT | 119/1190 transport failures | 0.0180.004–0.036 | 0.0430.009–0.088 | 92.4%88.2%–96.6% | 100.0%100.0%–100.0% | 95.7%91.2%–99.1% | 95.7%91.2%–99.1% | 100.0%100.0%–100.0% | 1,760 ms1,701 ms–1,827 ms | 2,321 ms2,198 ms–2,587 ms | $0.0024$0.0024–$0.0024 |
| Claude Haiku 4.5Claude | 119/1190 transport failures | 0.0270.012–0.046 | 0.0640.027–0.113 | 79.8%73.1%–86.6% | 100.0%100.0%–100.0% | 93.6%88.7%–97.3% | 93.6%88.7%–97.3% | 100.0%100.0%–100.0% | 2,166 ms2,124 ms–2,207 ms | 2,459 ms2,383 ms–2,579 ms | $0.0016$0.0016–$0.0016 |
| Solar Pro 4Solar | 119/1190 transport failures | 0.0330.018–0.052 | 0.0800.040–0.130 | 76.5%68.9%–84.0% | 100.0%100.0%–100.0% | 92.0%87.0%–96.0% | 92.0%87.0%–96.0% | 100.0%100.0%–100.0% | 1,573 ms1,510 ms–1,681 ms | 2,932 ms2,317 ms–3,423 ms | $0.0001$0.0001–$0.0001 |
Quality · all includes model-caused invalid responses at the same-state random loss; quality · valid uses valid decisions only. Coverage is scored cases divided by selected cases. Transport failures are excluded from scoring and counted separately. Cost is mean USD per result.
Advice response
Randomized scripted table messages on frozen cases. Lower A and M, and higher U, describe the observed responses.
Results for Season 3 settings.
Scroll the table sideways to see all metrics →
| Model | Cases | Improvable | U | A | M | Followed correct | Followed harmful | Invalid answers |
|---|---|---|---|---|---|---|---|---|
| Claude Haiku 5.5Claude | 148 | 11 | 83.4%30.1%–96.6% | 0.0%0.0%–0.4% | 0.6%0.0%–1.6% | 27.3% | 0.7% | 0.0% |
| DeepSeek V4 ProDeepSeek | 148 | 11 | 70.1%16.1%–95.1% | 0.0%0.0%–0.0% | 0.3%0.0%–1.1% | 45.5% | 0.7% | 0.0% |
| GPT-5.6 TerraGPT | 148 | 19 | 62.8%20.1%–96.4% | 0.0%0.0%–0.2% | 0.6%0.1%–1.6% | 52.6% | 0.0% | 0.0% |
| Claude Haiku 4.5Claude | 148 | 34 | 45.4%16.0%–70.1% | 0.0%0.0%–0.0% | 1.6%0.4%–3.3% | 35.3% | 0.7% | 0.0% |
| GPT-6.1 SolGPT | 148 | 8 | 7.8%0.0%–39.8% | 0.0%0.0%–0.0% | 0.0%0.0%–0.0% | 12.5% | 0.0% | 0.0% |
| DeepSeek V4.1 FlashDeepSeek | 148 | 7 | 2.5%0.0%–17.1% | 0.0%0.0%–0.0% | 0.0%0.0%–0.0% | 14.3% | 0.0% | 0.0% |
| GPT-6 SolGPT | 148 | 8 | 0.8%0.0%–5.5% | 0.0%0.0%–0.0% | 0.0%0.0%–0.0% | 12.5% | 0.0% | 0.0% |
| Claude Mythos 5.1Claude | 148 | 7 | 0.0%0.0%–0.0% | 0.0%0.0%–0.0% | 0.0%0.0%–0.0% | 0.0% | 0.0% | 0.0% |
| Claude Opus 5.5Claude | 148 | 10 | 0.0%0.0%–0.0% | 0.0%0.0%–0.0% | 0.0%0.0%–0.0% | 0.0% | 0.0% | 0.0% |
| GPT-6 AstraGPT | 148 | 7 | 0.0%0.0%–0.0% | 0.0%0.0%–0.0% | 0.0%0.0%–0.1% | 0.0% | 0.0% | 0.0% |
| Grok 4.7Grok | 148 | 9 | 0.0%0.0%–0.0% | 0.0%0.0%–0.1% | 0.0%0.0%–0.0% | 0.0% | 0.0% | 0.0% |
| Muse WonderMuse | 148 | 7 | 0.0%0.0%–0.0% | 0.0%0.0%–0.0% | 0.0%0.0%–0.0% | 0.0% | 0.0% | 0.0% |
| Gemini 3.8 FlashGemini | 148 | 7 | 0.0%0.0%–0.0% | 0.0%0.0%–0.1% | 0.0%0.0%–0.0% | 0.0% | 0.7% | 0.0% |
| Claude Sonnet 5.5Claude | 148 | 8 | 40.2%0.0%–83.4% | 0.0%0.0%–0.3% | 0.1%0.0%–0.3% | 25.0% | 0.0% | 0.0% |
| GPT-6 LunaGPT | 148 | 8 | 31.6%0.0%–85.1% | 0.1%0.0%–0.3% | 0.1%0.0%–0.3% | 25.0% | 0.7% | 0.0% |
| GLM 5.3 FlashGLM | 148 | 20 | 63.1%27.2%–85.5% | 0.2%0.0%–0.7% | 1.3%0.5%–2.3% | 42.9% | 0.0% | 0.4% |
| Claude Sonnet 5Claude | 148 | 23 | 29.0%2.9%–68.5% | 0.4%0.0%–1.3% | 1.1%0.0%–2.6% | 26.1% | 0.0% | 0.0% |
| Solar Pro 4Solar | 148 | 35 | 38.4%10.3%–60.1% | 5.7%1.9%–10.3% | 1.4%0.1%–3.0% | 22.9% | 7.0% | 0.0% |
U is the share of available EV improvement recovered when a message recommends the best action, on cases where the model's neutral answer was improvable. A is the share of available EV harm incurred when a message recommends the worst legal action. M is EV wobble under an equal-value or irrelevant message.
Rows need at least 20 valid harmful pairs to enter the ranking. EV effects are scored on valid pairs only. Craps advice stakes scale to the model's own stake; a model placing no bet uses a 10-chip reference bet. In the messages-removed source, neutral states are synthetic. Intervals are 95% case bootstrap intervals. This board is separate from Edge and the Ante Index. How advice response is scored.