Cooperative Multi-Agent RL with Intra-Episode Task Switching

English | 中文

---

ComplexOvercooked is a cooperative multi-agent reinforcement learning benchmark for Intra-Episode Task Switching (IETS). Agents must coordinate cooking, processing, combining, transferring, and delivering food while the active orders can change within an episode. The benchmark provides two- and four-agent layouts, vector and grid state encodings, configurable recipes and rewards, and training configurations for MARL algorithms such as IPPO, HAPPO, QMIX, VDN and QATTEN.

Duo-Open

Duo-Split

Quad-Open

Quad-Split

Introduction

ComplexOvercooked advances cooperative multi-agent benchmarks with the following key features:

• (a) Intra-Episode Task Switching (IETS): Most importantly, compared to prior cooperative game benchmarks, ComplexOvercooked introduces the IETS challenge. Within each episode, multiple orders switch randomly, meaning the environment's reward function is non-stationary. Agents must continuously adapt their cooperation strategy to the current active orders.
• (b) Multi-step Orders with Varying Difficulty: The environment provides four types of orders. Each order requires multiple sequential steps to complete (e.g., gathering ingredients, processing, cooking, combining, and serving), and the difficulty differs across recipes.
• (c) Heterogeneous Control Interfaces: ComplexOvercooked supports three categories of control interfaces — MARL agents, LLM-based agents, and human players — enabling research across reinforcement learning, language-agent coordination, and human-AI collaboration.
• (d) Scalable Team Size and Rich Interactions: The benchmark supports both 2-player and 4-player cooperative settings, with no fewer than 18 types of interactive objects.

State and observation

Each agent receives a player-centric observation. The environment supports:

• Flatten encoding (lossless_obs=false): compact features for players, held items, nearby objects, equipment states, active orders, and timers.
• Grid encoding (lossless_obs=true): an H x W x C tensor containing terrain, equipment, items, players, order channels, and global time.

enable_state_norm scales coordinates, progress, and time features to stable ranges. The current global state is the first agent's observation.

The insertion order of task_config is part of the observation schema. Keep it unchanged between training and evaluation. Saved checkpoints preserve this order in model_configs.json.

Action

Every agent selects one of six discrete actions at each timestep:

ID	Action
0	Stay
1	Move down
2	Move right
3	Move up
4	Move left
5	Interact

Interaction covers picking up, placing, cutting, cooking, combining, serving, and discarding items according to the adjacent station.

Reward

The training reward is:

reward = sparse delivery reward + shaping_factor * shaped reward

• The sparse reward is the score of the completed order.
• Shaped rewards cover useful intermediate events such as cutting, cooking, collecting a prepared item, taking a useful dish, and transferring items.
• reward_shaping_fraction controls linear shaping decay during training.
• enable_reward_clip_norm optionally clips and scales training rewards.
• Evaluation reports sparse return only.

Installation

Python 3.12 or newer and uv are required.

uv sync --frozen
source .venv/bin/activate

Run the quick environment checks:

pytest tests/test_env.py tests/test_main_entrypoints.py

Launch a one-episode render demo to verify the GUI works:

python examples/render_demo.py

CUDA is optional. Training automatically falls back to CPU when no CUDA device is available.

How to Run

Train

Experiment presets are defined in main/presets.yaml.

# List presets
python main/main_train.py --list

# Inspect the resolved command
python main/main_train.py --preset ippo_duo_open --dry-run

# Train both flatten and grid encodings, using five consecutive seeds
python main/main_train.py \
  --preset ippo_duo_open \
  --sweep \
  --repeat-time 5

Any setting can be overridden from the command line:

python main/main_train.py \
  --preset qmix_quad_open \
  --set seed=42 \
  --set use_swanlab=false \
  --set t_max=2000000

The lower-level entrypoint is also available:

python src/main.py \
  --config=ippo \
  --env-config=overcooked2 \
  --set env_args.map_name=duo_open \
  --set env_args.lossless_obs=false

Configuration priority is:

default < environment < algorithm < preset < sweep < --set

Models are saved with their complete runtime configuration:

results/models/<map>/<experiment>/best_model/
├── agent.th
├── ...
└── model_configs.json

Evaluate

main_eval.py restores the saved environment, model dimensions, normalization statistics, and task order. It evaluates 20 episodes by default and prints the sparse-return mean and variance.

A pretrained IPPO checkpoint on duo_open is bundled under examples/checkpoints/ for a quick out-of-the-box evaluation:

python main/main_eval.py examples/checkpoints/ippo_duo_open ippo --episodes 20

To evaluate your own trained models:

python main/main_eval.py \
  results/models/duo_open/<experiment> \
  ippo \
  --episodes 20

Supported evaluator names are ippo, happo, and qmix.

Customize the benchmark

Maps. Add an entry to envs/maps.json with a rectangular layout and player count. Player spawn points use 1-4.

Symbol	Station	Symbol	Station
X	Counter	C	Pot
U	Cutting board	D	Dish dispenser
F	Fish	B	Beef
L	Lemon	M	Tomato
H	Bun	T	Trash
E	Serving	space	Walkable floor

Order mix. Configure orders under env_args.task_config. Each value is [delivery_reward, sampling_probability], and probabilities must sum to 1:

env_args:
  task_config:
    AClemoncookedfish: [20, 0.5]
    cookedbeefhamburger: [15, 0.5]
  task_num: 2
  task_time: 200

New combinations. Add composition rules to RECIPE in envs/overcook_class.py, reward semantics to TASK_VALUE in envs/overcook_mdp.py, and register new item types in the ordered self.ITEMS registry so both encodings remain stable.

Acknowledgement

ComplexOvercooked builds on ideas and implementations from:

• Overcooked-AI for cooperative cooking environments and featurization.
• PyMARL2 for multi-agent reinforcement learning infrastructure and algorithms.
• ProAgent for LLM-agent design.

This project is released under the MIT License.