Three domains, three tiers, one protocol
Each scenario is an evolving-environment streaming dataset: a chronological task stream split into windows, each governed by a latent environment state that may change at the next boundary. Serving and test tasks of a window cover the same policy with disjoint instances.
Retail
L1 · L2 · L3Multi-turn customer support over an order database: cancel or modify pending orders, return or exchange delivered items. Correct only if the final database state is the intended one, or the request is refused with the right reason code.
Current timestamp: 129 Customer: Olivia Moreau (zip 21938) writes: "Could I return item 2645252359 — the Water Bottle — off order #W9000498? No longer needed. I'd like the refund on my gift card (gift_card_8859872)."
Banking
L1 · L2 · L3Case review: authorize card transactions, review credit-limit increases, screen outbound transfers. Harder tiers add ranked alternatives and payment batches. Hidden policies are thresholds, caps and corridors.
Current timestamp: 134 Credit request LIM-107892: account ACC-184760 proposes a new ceiling of $5857.25. Record the decision.
Pitch
L1 · L2 · L3Write a 40–80 word sales pitch from a factual product sheet for a customer whose taste is never stated. A rubric rewards liked attributes and penalizes disliked, false or invented claims.
Current timestamp: 99 Product to pitch: Product: Orin Go Max (backpack) Facts: - already has a large user base - premium price - standard materials - provides only the essential functions, with few extras - a trending, much-talked-about item Write your pitch.
| Scenario | Latent structure | Windows | Serving | Test |
|---|---|---|---|---|
| retail_l1 | Categorical rules | 6 | 720 | 384 |
| retail_l2 | Categorical rules | 7 | 810 | 765 |
| retail_l3 | Categorical rules with reversals | 8 | 624 | 537 |
| banking_l1 | Numeric boundaries | 4 | 336 | 178 |
| banking_l2 | Composed numeric boundaries | 5 | 500 | 387 |
| banking_l3 | Numeric boundaries with reversals | 6 | 600 | 551 |
| pitch_l1 | Customer preferences | 4 | 216 | 96 |
| pitch_l2 | Preference drift | 6 | 324 | 144 |
| pitch_l3 | Preferences with reversals | 7 | 378 | 168 |
| Total | 53 | 4,508 | 3,210 |
Difficulty tiers
- L1 · acquisition
- Previously unseen states; mostly single-policy requests; Pitch profiles never repeat.
- L2 · revision and composition
- Earlier states are revised; multi-policy Retail requests, ranked or batched Banking decisions, recurring Pitch profiles.
- L3 · non-monotonic evolution
- Reversals, removals and re-entry of earlier states on top of L2's composition.
What the learner sees
- Every task: the task, prefixed with its stream position (Current timestamp: t), and the domain documents, which may be incomplete or out of date.
- After a serving task: its own trajectory and one line, Score: n/100.
- Never: the hidden policy, the correct answer, the task's category, window boundaries, or any explanation. Test tasks return nothing.
- Budget: one attempt per task, 50 tool calls, 100 turns.
How it is scored
- Hidden-Dependent tasks need the hidden policy; Fully Specified tasks can be solved from visible information. A good learner raises the first without losing the second.
- Retail and Banking are graded by rule (database state or decision and reason code); Pitch by an LLM-judged rubric.
- A setting score is the mean reward × 100 over a scenario's test tasks of one category; averages weight every setting equally.