Methodology

The MTG LLM Benchmark measures how well large language models can play Magic: The Gathering — a complex strategy card game with hidden information, resource management, and multi-step decision making.

Each model plays complete games of MTG against other models or a CPU baseline, making every decision through natural language: playing lands, casting spells, declaring attackers and blockers, choosing targets, and more.

Games run on XMage, an open-source MTG rules engine that implements the complete Magic rules. The engine handles game state, turn structure, stack resolution, combat, and all card interactions.

The benchmark integrates with XMage via callback methods — when the engine needs a player decision, it calls the appropriate method (priority, chooseTarget, selectAttackers, etc.) which routes to the LLM via a synchronous bridge.

This architecture means the LLM never needs to track game state or enforce rules. The engine presents valid options, and the LLM selects from them.

The benchmark handles 9+ distinct decision types, each requiring different reasoning strategies:

Every request has the same shape: a system prompt with the rules summary and the model's full decklist, a running game logof everything that has happened (one entry per turn, with life totals at the start of the turn), and the current decision: the board, the model's hand, and the legal options as a numbered list.

At the end of each turn the model writes a one- or two-sentence strategy note into its own game log, so its plan survives as the history grows. The opponent's hand and every library order stay hidden.

Models may call up to three tools per decision before answering:

• cardDetails, searchRules: full card text and rules lookups
• viewGraveyard, viewExile, searchDeck: zone contents the prompt summarizes
• searchOptions: filter a long list of legal options
• saveNote, readNotes: a scratchpad that persists between decisions

Models play three Legacy-style decks chosen for how much skill they reward: Grixis Delver (tempo, counters), Dredge (graveyard combo) and Jeskai Control, built from the newest set so it can't be memorized. Against the CPU the model always faces Red Burn; in model-vs-model battles the two models play Delver against Jeskai.

Every model is dealt identical seeded shuffles. A battle deal is played four ways (each model on each deck, from each seat) and a CPU deal from both seats, so the luck of the draw is shared and can be separated from skill.

All games, against the CPU and between models, are fitted together in one Bradley–Terry model: the chance that one side wins depends on the difference in the two players' skill, the strength of each deck, who went first, and the luck of that particular deal. The XMage CPU with its Red Burn deck is fixed at 0, so every rating reads as Elo above or below the CPU and stays put when new models are added.

The fit is Bayesian, and the size of deal luck is averaged over rather than picked, which keeps the 90% intervals honest: in simulations they contained the true value about 93% of the time. Models with fewer than four games are shown but not ranked.

Games are not scheduled up front. After each game the rating is refitted and the next game is the one expected to narrow the leaderboard most per dollar: a model whose rating is uncertain, or two models whose order is unsettled. Strong models rarely play the CPU, because a near-certain win says little; they play each other and the strongest cheap models instead. Every game played is counted.

Cost per game is the tokens each model actually used, priced at OpenRouter's list prices with cached input at its discounted rate and Anthropic's cache writes at their surcharge. Prompt caching matters: a fix that cached each tool-call round cut Claude Opus from $10.27 to $3.91 a game. Claude costs on the chart come from games played after that fix; a model with none is marked.

• Each model has its own conversation; nothing is shared between the two sides of a battle.
• The harness is versioned by a hash of every file that shapes what a model sees or how its answers are read. Any change starts a new version; earlier games carry over only when the change cannot alter play (for example, cache markers that leave the text identical).
• Before any paid run, the cheapest model plays every matchup and an automated census flags every rejected action, invalid reply and fallback; anything recurring is treated as a harness bug.
• Every game finished normally and no decision fell back to a default in the published results.

• Magic has a lot of luck. The frontier models have 4–16 games each, so their order is tentative; the intervals on the chart show how tentative.
• Ratings learned from CPU games predict battles only loosely, so a single scale is a reasonable summary rather than a proven one.
• Three decks and one CPU opponent are a narrow slice of Magic.