EvoObserver

What if an agent's harness improves, but the component that decides what was learned from each run stays frozen?

EvoObserver is an idea-first research framework for co-evolving an agent harness H with an executable Observer O. The Harness controls how a frozen base model acts. The Observer reads frozen traces and evaluator evidence, then produces structured diagnoses that guide the next Harness update.

The key boundary is deliberate:

• the verifier is fixed and is the only authority for task performance;
• the Observer may evolve, but it cannot edit evidence or award itself a score;
• Observer quality is measured by downstream improvement: did its diagnosis produce a better next Harness under the fixed verifier?

This repository is a public research preview. It contains an auditable engine, a no-API demo, a budgeted text-classification adapter, a proposed benchmark protocol, and a paper draft. It does not yet claim benchmark superiority.

Paper: Learning to Observe: Co-Evolving the Feedback Loop of an Agent Harness

The idea in one loop

Most harness optimizers search over H while keeping the trace-analysis and feedback machinery fixed. EvoObserver treats (H, O) as the candidate, while keeping V outside the evolutionary boundary.

For a Harness H, frozen evidence E_H, Observer O, mutator M, and sealed validation tasks D_val, the proposed one-step Observer utility is:

U(O; H) = V(M(H, O(E_H)), D_val) - V(H, D_val)

This turns "the diagnosis sounded useful" into a falsifiable question: did it cause improvement?

Quick start

Requirements: Python 3.10+ and uv.

git clone https://github.com/haochen2115/evo-observer.git
cd evo-observer
uv sync --extra dev
uv run pytest -q
uv run evo-observer demo --iterations 20 --seeds 20 --output runs/toy-demo

The demo compares bounded raw evidence, a fixed budgeted Observer, and a co-evolving budgeted Observer across paired random seeds. A score-only search arm is retained as a sanity lower bound. It uses no model API and is explicitly a synthetic mechanism study, not a real-task benchmark result.

cat runs/toy-demo/comparison.json

With a shared 600-character mutation-view budget, raw evidence contains 32 records and occupies about 3.5k characters. Bounded raw context fits at most five records. The fixed and evolving Observers scan all records in code while forwarding no raw cases. Over 30 paper seeds, their final mean scores are .868, .951, and .992, respectively; co-evolution also has the narrowest confidence band. See the research report.

What is implemented

• versioned (Harness, Observer) candidates;
• fixed-verifier scoring separated from Observer diagnoses;
• frozen evidence with before/after SHA-256 integrity checks;
• causal credit from a child's score gain to the parent Observer;
• replayable JSON artifacts and append-only event records;
• score-only, bounded-raw, fixed-Observer, and co-evolution ablations;
• fixed Observer-view budgets and auditable compression statistics;
• API-backed score-only/fixed/joint matched arms;
• H x O counterfactual cross-play with incremental checkpoints;
• a vendor-neutral coding-executor seam for future worktree-based agents;
• deterministic tests and CI;
• a paper draft containing the concept, prior-art map, and experiment plan.

The current core is intentionally small. Production-grade sandboxing, parallel search, coding-agent execution, and benchmark adapters are next milestones.

OpenAI-compatible model configuration

Real adapters read credentials only from process environment variables:

export EVO_OBSERVER_API_BASE="https://your-provider.example/v1"
export EVO_OBSERVER_API_KEY="your-key"
export EVO_OBSERVER_MODEL="your-model"

Never commit a real endpoint, key, or .env file. Each user must obtain credentials from their own model provider. See .env.example for the complete non-secret template.

Repository map

src/evo_observer/   engine, budgeted Observer, provider, coding executor
tests/              behavioral and integrity tests
data/               public support-intent mini dataset
results/            machine-readable multi-seed synthetic result
REPORT.md           experiment contract, result, and benchmark path
paper/              editable draft sources and generated PDF

Why this may matter

As the Harness changes, its failure distribution changes too. A fixed Observer can become stale, compress away the wrong parts of a trace, or keep recommending edits for errors that no longer dominate. Co-evolution gives the feedback path permission to adapt - without giving it permission to redefine success.

The strongest version of the claim is not "reflection helps." It is:

Under a fixed verifier and matched evidence, context, tool, and search budgets, jointly evolving an executable Observer improves sample efficiency, final held-out quality, or transfer relative to bounded raw evidence and a strong fixed-Observer baseline.

That claim still needs multi-seed real-task experiments. See the research report and paper draft.

Relation to current work

This project is adjacent to Meta-Harness, Agentic Harness Engineering, ECHO, MetaSkill-Evolve, TextGrad, and the Darwin Godel Machine.

The intended research gap is narrower than any general claim about recursive self-improvement: executable observer evolution for harness search, evaluated through downstream causal lift and cross-play under a sealed verifier. The paper draft contains the detailed comparison and overlap risks.

Research status

• Public thesis and falsifiable objective
• Minimal auditable reference loop
• Fixed-Observer ablation
• Paper draft and reproducible budgeted toy
• Generic command-based Observer sandbox
• H x O counterfactual cross-play implementation
• Provider-neutral LLM proposer adapter
• Multi-seed public benchmark results
• arXiv-ready manuscript

Citation

The project is preprint-stage. Citation metadata is provided in CITATION.cff and will be updated with an arXiv identifier after submission.

License

Apache-2.0. See LICENSE.