Agent Orchestration Playground (Robot Monorepo)

A deterministic, inspectable multi-agent orchestration core that drives a physical LeRobot SO-101 arm. This is a monorepo: the orchestration layer, the robot agent layer, and the robot SDK all live together in one tree.

The orchestration core runs a fixed governance loop — accept goal, recall memory, plan, build/review/retry, remember — and stays deterministic and traceable. The robot swarm plugs into that loop by swapping three workers, then executes real skills on the arm through a LangGraph supervisor and a thread-safe hardware service (with a safe dry_run default).

Monorepo layout

agent-orchestration-playground/
├── orchestration/        # core governance loop + robot orchestrator
│   ├── swarm.py              # SwarmOrchestrator (the reusable run() loop)
│   ├── robot_swarm.py        # RobotSwarmOrchestrator (swaps in robot workers)
│   ├── robot_cli.py          # CLI entry point: python -m orchestration.robot_cli
│   ├── event_bus.py, tracing.py, trace_store.py, retry_policy.py,
│   ├── run_history.py, structured_logger.py, tenant.py, task_graph.py
├── agents/               # role-separated agents
│   ├── commander.py, memory_agent.py            # shared
│   ├── planner.py, builder.py, reviewer.py      # content workers (imported by swarm)
│   └── robot_planner.py, robot_executor.py, robot_verifier.py  # robot workers
├── memory/               # persistence (JSON default, SQLite optional)
│   ├── memory_store.py, base.py, sqlite_store.py
├── lerobot_agents/       # multi-agent robot control layer (LangGraph)
│   ├── orchestration/supervisor.py              # routes goal to specialists
│   ├── agents/  (motion, state, gripper specs)
│   ├── capabilities/  (framework-agnostic skills)
│   ├── adapters/  (the only LangChain/LangGraph bindings)
│   └── robot_service.py                         # single, thread-safe hardware owner
├── lerobot_sdk/          # self-contained SO-101 SDK (kinematics, motors, poses)
├── docs/                 # architecture + robot orchestration docs and diagrams
├── requirements-robot.txt
└── pyproject.toml

Because everything is under one root, the packages import each other as plain top-level packages (orchestration, agents, memory, lerobot_agents, lerobot_sdk) with the repo root on PYTHONPATH — no sys.path juggling.

How it works

RobotSwarmOrchestrator subclasses SwarmOrchestrator and reuses its run() governance loop verbatim, swapping in three robot-aware workers:

Core role	Default	Robot swap
planner	PlannerAgent	RobotPlannerAgent
builder	BuilderAgent	RobotExecutorAgent
reviewer	ReviewerAgent	RobotStateVerifierAgent

The run loop:

User goal
  -> CommanderAgent        normalize / validate
  -> MemoryAgent.recall     prior feedback
  -> RobotPlannerAgent      robot task graph (read state -> plan -> position -> grip -> verify)
  -> build -> review -> retry loop (RetryPolicy, max 2):
       RobotExecutorAgent.stream(goal)
         -> lerobot_agents Supervisor delegates to state / motion / gripper specialists
         -> RobotService (sanitize/clamp, single lock) -> LeRobotArm -> SO-101 (or dry_run)
       RobotStateVerifierAgent.review  (approved if >=1 skill ran, no errors)
  -> MemoryAgent.remember   persist feedback + run record
  -> result (goal, task_graph, draft, review, event_log, trace, attempts)

Full details and diagrams: docs/robot_orchestration.md.

Quickstart

cd agent-orchestration-playground
export PYTHONPATH=$(pwd)

# Install robot dependencies (LangGraph supervisor + SDK)
pip install -r requirements-robot.txt
pip install -r lerobot_agents/requirements.txt
pip install -r lerobot_sdk/requirements.txt

# The inner robot supervisor needs an LLM
export OPENAI_API_KEY=...

# Run a goal (dry-run simulation is the safe default; no hardware opened)
python -m orchestration.robot_cli --goal "open the gripper and report the pose"

# Full JSON result
python -m orchestration.robot_cli --goal "..." --json

# Interactive loop (prompts for goals)
python -m orchestration.robot_cli

# Drive real hardware (opens the serial port)
python -m orchestration.robot_cli --goal "..." --no-dry-run --port /dev/ttyACM0

The console script robot-swarm (see pyproject.toml) maps to the same entry point.

CLI options

Flag	Purpose
--goal	Natural-language robot goal (omit for interactive loop)
--json	Print the full JSON result
--dry-run / --no-dry-run	Simulate (safe default) vs. drive the real arm
--port	Serial port, e.g. /dev/ttyACM0
--id	LeRobot calibration id
--model / --provider	LLM model / provider for the supervisor

Design notes

• Deterministic, inspectable core: every step is logged to an EventBus and a span-based trace; runs are persisted to history.
• The only seam between core and robot is RobotExecutorAgent, which needs an object exposing stream(goal). Inject any compatible system (e.g. a fake) to run the loop offline without an LLM or hardware.
• Safety: RobotService clamps joint targets and defaults to dry_run.

Not production

This is an experimental architecture playground for reliable, memory-aware, inspectable multi-agent coordination that reaches all the way down to a physical arm. It is not a production robotics framework.