JudgeStack stores today's ban and the dated announcement that imposed it as separate documents

Ask JudgeStack when a card became illegal in Modern and two records can answer. A legality claim says it is banned today. A dated format announcement says when the ban began. According to the dev.to write-up, Magic has no single source hierarchy that works for every question, so the two rows are modelled separately ^[28]. Depending on what is asked, the system may need the card's current Oracle wording, an older printing, a rule, a ruling or a format announcement ^[2]. When the evidence is missing it is supposed to say so instead of filling the gap from memory ^[3]. "If the system treats those as the same question, it can invent an effective date while sounding completely certain," JudgeStack's developer wrote ^[5].

The corpus is 496 documents across ten types ^[6]. The counts that are itemised add to 405, which leaves 91 documents spread across glossaryTerm, formatEvent, decision, adjudicationCase and authoritySource ^[8]. formatEvent is the type that carries the dated announcements, and one of the three evaluation categories is historical rules changes ^[14]. Legality claims alone are about 42 percent of the corpus ^[9]. Ten document types for a card game looks like over-modelling until you notice the corpus records 74 printed-versus-Oracle text differences against 77 printings ^[30].

The Comprehensive Rules are not in the dataset. A second Sanity Context endpoint serves them as a file source, while the dataset endpoint takes filtered GROQ queries ^[10]. Only the sixteen rule paragraphs cited by reviewed cases exist as dataset documents. That is enough to verify those citations against exact text, and it keeps the developer from republishing the full rules as hundreds of documents ^[12].

Both conditions ran deepseek-flash through api.deepseek.com with the same answer prompt, so the answer stage was held constant ^[20]. The condition is a flag on one command, `npm run eval -- --condition structured`, with the provider set by an environment variable ^[29]. The step budget also changed: the structured condition got ten times as many model steps as the lexical one ^[19]. "This is not a clean comparison of BM25 against GROQ," the developer wrote ^[22].

For a structured win to transfer, two things about your workload would have to hold. Your questions would have to route to more than one kind of authority the way Magic's do ^[28], and your agent would need a step budget in the same range as the ten steps the structured condition could spend on GROQ queries, the rules knowledge base and reference following ^[18]. Twenty of the thirty questions were run during development ^[16], so the held-out ten are the part of the result nobody tuned against ^[15].

The code both conditions ran from is tagged eval-2026-09-22, and the frozen outputs were committed afterwards under a separate artifacts tag ^[24]. The post says the method, the limitations and one metric the developer had to throw out are set out further down; the results themselves are not in the text reviewed here ^[27].