Motivation
Switchyard’s existing benchmark example shows how to evaluate routing with Harbor and coding-agent benchmarks. Switchyard also integrates with NeMo Gym, which supports a broader collection of benchmarks and agent harnesses, but the Switchyard repository does not currently include a runnable example of that workflow.
Adding one would help users:
- Evaluate routing strategies on benchmarks beyond Harbor’s coding-agent workloads.
- Compare a routed strategy with a fixed-model baseline using the same tasks.
- Reproduce the workflow directly from the Switchyard repository.
- Discover and validate the NeMo Gym integration without assembling commands from external documentation.
Proposed solution
Add benchmark/nemo_gym/ with a small, executable, version-pinned example that:
- Defines a fixed-model baseline and one routed strategy.
- Runs both against the same benchmark and task subset.
- Writes each condition to a separate output directory.
- Pairs rollouts before comparing rewards.
- Compares model tokens, classifier tokens, latency, and routing statistics.
- Explains the generated provenance and statistics artifacts.
- Links to the full NeMo Gym integration documentation.
Motivation
Switchyard’s existing benchmark example shows how to evaluate routing with Harbor and coding-agent benchmarks. Switchyard also integrates with NeMo Gym, which supports a broader collection of benchmarks and agent harnesses, but the Switchyard repository does not currently include a runnable example of that workflow.
Adding one would help users:
Proposed solution
Add benchmark/nemo_gym/ with a small, executable, version-pinned example that: