🤗 Models & Data | ♠ SPADE on Hugging Face | 📄 Paper
- 20/08/2026: 🎉 We release our paper on arXiv, together with the SPADE checkpoints (4B / 8B / 30B-A3B, games and tool use), the grounding corpora, and the generated environments on Hugging Face.
- 11/08/2026: 🎉 We release our self-play codebase and the static GPT-5.5 environment corpus.
Recent advances in reinforcement learning have shown that language models can develop sophisticated reasoning through training on tasks with verifiable rewards, but these approaches draw their reward signal from fixed, hand-built pools of environments that stop adapting once the learner masters them.
We introduce SPADE, a self-play framework where a single language model learns in two roles: an environment designer that writes complete multi-turn environments as executable Python with reset() and step() interfaces, and a reasoning agent that learns by acting in them. The designer is trained with hint-based regret, the gap between the agent's return with and without a privileged hint, which steers generation toward environments at the agent's capability frontier while keeping them feasible. Through this loop, SPADE generates an adaptive curriculum that keeps moving with the learner instead of saturating.
Applying SPADE to Qwen3 models at 4B, 8B, and 30B-A3B scale in two settings, cognitive games and multi-turn tool use, we observe the designer produce progressively harder, more interactive environments and the agent improve on held-out math, science, code, and procedural-reasoning benchmarks past the saturation point of fixed-environment baselines. These results suggest that making environment design itself a learnable component is a promising direction for open-ended self-improvement.
SPADE trains one shared policy that plays both roles. Each cycle, the environment designer samples grounding context from a pretraining corpus and an environment memory, then writes a complete executable environment with a privileged hint; generated code passes structural and runtime validation before entering the training pool. The reasoning agent plays each environment with and without the hint: task return trains the agent role, hint-based regret trains the designer role, and per-role advantage normalization keeps the joint update stable. The backend-independent orchestration lives in spade/core/; distributed training uses the Slime/SGLang integration under spade/slime/ (SGLang inference, Megatron-LM policy updates, Ray orchestration), with a Tinker integration under spade/tinker/.
SPADE is model-agnostic: the training loop only needs a chat-capable policy, so any model your backend can fine-tune works. The table lists the popular families; bold entries are the ones we train in the paper.
| Model family | Models |
|---|---|
| Qwen3 | Qwen/Qwen3-4B-Instruct-2507, Qwen/Qwen3-8B, Qwen/Qwen3-30B-A3B-Instruct-2507, Qwen/Qwen3-32B |
| Qwen3.5 | Qwen/Qwen3.5-4B, Qwen/Qwen3.5-9B, Qwen/Qwen3.5-35B-A3B |
| GPT-OSS | openai/gpt-oss-20b, openai/gpt-oss-120b |
| Nemotron | nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16, nvidia/NVIDIA-Nemotron-3-Super-120B-A12B |
| GLM | GLM-5.2, GLM-5.3 |
For the complete set, see the Miles model list for full fine-tuning and the Tinker model docs for Tinker-backed training.
# clone codebase with pinned submodules (slime, tinker-cookbook)
git clone --recurse-submodules git@github.com:spade-rl/spade.git && cd spade
# prepare environment
python -m venv .venv && source .venv/bin/activate
# install dependencies
python -m pip install --upgrade pip
python -m pip install -e .Python 3.10 through 3.12 are supported; python -m pip install -e ".[dev]" adds test and lint tooling. GEM support on Python 3.12.1+ needs python -m pip install --ignore-requires-python gem-llm; see cmd/README.md for details.
The launchers read paths and credentials from the environment:
export MODEL_ROOT=/path/to/model/checkpoints # HF checkpoints + Megatron conversions
export WORKSPACE_DIR=/path/to/spade/workspace # external eval data (aime-*/, bfcl/)
export CORPUS_FILE=/path/to/grounding.jsonl # designer grounding corpus (adaptive recipes)
export WANDB_API_KEY=... # Weights & Biases logging
export WANDB_ENTITY=your-wandb-entitybash cmd/games/train_spade_30b.shThis training script runs SPADE games self-play for 400 rollouts on a single 8-GPU node, training both roles of Qwen3-30B-A3B-Instruct with GRPO. The full paper matrix is organized by setting:
| Setting | Models | Commands |
|---|---|---|
| SPADE games | 4B, 8B, 30B-A3B | cmd/games/train_spade_{4b,8b,30b}.sh |
| Fixed-env GRPO, GPT-5.5 corpus | 4B, 8B, 30B-A3B | cmd/games/train_fixed_gpt55_{4b,8b,30b}.sh |
| Fixed-env RLVE | 4B, 8B, 30B-A3B | cmd/games/train_fixed_rlve_{4b,8b,30b}.sh |
| SPADE tool use | 4B, 8B, 30B-A3B | cmd/tool_use/train_spade_{4b,8b,30b}.sh |
| Paper ablations | 30B-A3B | cmd/ablations/*.sh |
The fixed-env GRPO recipes need no CORPUS_FILE: they train on the released static GPT-5.5 corpus (7,872 validated Python environments across six cognitive skills, pinned revision with per-environment SHA-256 checksums, Apache-2.0). Checkpoint and data prerequisites are documented in cmd/README.md, and the setting-specific READMEs under cmd/games/, cmd/tool_use/, and cmd/ablations/ document the corpora and overrides each recipe expects.
eval_offline/ scores a trained checkpoint, or an OpenAI-compatible endpoint, against the paper's benchmark suites:
# run the offline benchmark suites
python -m eval_offline.run_offline_eval --help
# format the results
python -m eval_offline.render_table --helpThe runner needs the [eval] extra and per-benchmark data setup; both, plus the benchmark matrix, are documented in eval_offline/README.md. eval_configs/ is separate: those YAML files drive the in-loop evaluations the training launchers run during a job.
SPADE also supports training with Thinking Machines' Tinker distributed training framework through the integration under spade/tinker/.
# Install with Tinker dependencies (Python 3.11+)
python -m pip install -e ".[tinker]"| Model | Launchers |
|---|---|
| Qwen3-4B-Instruct | cmd/tinker/qwen3_4b_instruct/ |
| Qwen3-8B | cmd/tinker/qwen3_8b/ |
| Qwen3-8B-Base | cmd/tinker/qwen3_8b_base/ |
| Qwen3-30B-A3B-Instruct | cmd/tinker/qwen3_30b_instruct/ |
| GPT-OSS-20B | cmd/tinker/gpt_oss_20b/ |
The full set of Tinker-trainable models is listed in the Tinker model docs. See cmd/tinker/README.md for setup, launchers, and advanced usage. For more information on the Tinker framework, see the tinker-cookbook repository.
If you find our work useful for your research, please consider citing:
@article{liu2026spade,
title={SPADE: Self-Play in Adaptive Synthetic Executable Environments},
author={Liu, Bo and Yu, Simon and Jiang, Yiding and Qu, Ao and Zhao, Andrew and Liu, Zichen and Kim, Junsu and Zhou, Zijian and Kim, Seungone and Ren, Tongzheng and Liu, Mickel and Yu, Hanfei and Chen, Zhaorun and Shi, Weiyan and Liang, Paul Pu and Zettlemoyer, Luke and Choi, Yejin and Jaques, Natasha},
journal={arXiv preprint arXiv:2608.19197},
year={2026}
}- The distributed RL training is implemented with Slime, pairing SGLang inference with Megatron-LM policy updates, and informed by the Miles team's RL post-training framework.
- We thank Thinking Machines for the Tinker framework and tinker-cookbook, an alternative training backend.
- We thank Modal for compute and model serving during development.
- The evaluation stack builds on RLVE, the Berkeley Function Calling Leaderboard, and PRIME evaluation utilities.
- The base models are from Qwen3.
SPADE source code is released under the MIT License; datasets, vendored code, and adapted evaluation components retain their respective licenses as documented in THIRD_PARTY_NOTICES.md.