English | 中文
era — a CLI that evolves a program toward a better score on a metric you define. Each iteration, it selects a program from its search tree, asks a coding agent to improve it, and scores the result — your metric, your eval, measured by ERA, never by the agent. The winning program is a file you can run.
Built on the FUTS (Flat-UCB Tree Search) algorithm from "An AI system to help scientists write expert-level empirical software" — a single global selection over the whole tree, no per-node value averaging. The mutation operator is a coding agent, pi, driven over its stdio RPC protocol. The engine adapter is pluggable.
select flatten the WHOLE tree into one set of arms
rank-score + PUCT bonus -> single global argmax
|
propose an operator turns the selected node into an instruction:
mutate = improve this one program
recombine = merge it with an unrelated high-scoring node
(+ optionally one idea injected from an idea book)
|
run one bounded engine turn: fresh process, cwd = the child node
directory, era counts LLM steps and aborts at the cap
|
score era runs YOUR --eval command and parses the last stdout line
as a float. Optionally N times, aggregated, with a lower
confidence bound. A held-out command can be measured but
never optimised.
|
backprop +1 num_visits on the new node and its ancestors.
No value averaging. That's it.
| 🧬 Program evolution | One command turns a starting program into a search tree of variants; the best one is exported ready to run. |
| 🎯 Your metric, your eval | ERA runs --eval inside each variant and parses the last stdout line as a score. The agent never reports its own score. |
| 🛡️ Overfitting is measured, not assumed | Hold-out eval (--holdout-eval) measures a score the search never optimises; the run summary prints the selection gap. |
| 🪵 Everything is a log | Every event is appended to trace.jsonl — the single source of truth. tree.json is a projection. Interrupted runs stay inspectable. |
| 📊 Built-in analysis | era report turns the log into a markdown report: breakthroughs, diversity, and whether the search stalled or the metric ran out. |
| 🔌 Pluggable engine | The engine adapter is a small layer. Today it ships with pi; era doctor --probe verifies the model actually answers before your first real run. |
| 🪶 Zero runtime dependencies | Pure stdlib (Python ≥ 3.10). pytest is only for the tests. |
| ⚖️ No reward hacking | The scorer is restored from the pristine seed before every evaluation; a mutation cannot improve its score by rewriting eval.py. |
| 🧪 Tested without a network | A stub engine (tests/fake_pi.py) speaks the real RPC protocol, so the whole loop runs offline. |
git clone https://github.com/kirie123/era-evolve.git
cd era-evolve
pip install -e .
era --helpRequires Python ≥ 3.10. No third-party packages.
python scripts/install_pi.py # installs bun (if missing) + pi, then prints
# where to configure the modelscripts/install_pi.py is pure stdlib and works on Windows / macOS / Linux.
Already have pi? python scripts/install_pi.py --check, or
--agent-bin <path> to verify a specific binary. Manual alternative:
bun install -g @earendil-works/pi-coding-agent.
Point pi at a provider in ~/.pi/agent/models.json — Ollama
or a llama.cpp server work fully local,
any OpenAI-compatible endpoint works too. ERA never touches models or
credentials: it passes no --model / --api-key / --provider, and the agent
reads its own configuration.
era doctor # engine binary found, no secrets printed
era doctor --probe # also asks the model one minimal prompt — catches
# "engine found, model misconfigured" before the first runera run --task "improve solve() so the score gets better" \
--seed examples/poly/seed \
--eval "python eval.py" \
--metric-goal max \
--budget 5 \
--out ./run1This fits a hidden quadratic from a deliberately bad seed; one or two mutations usually solve it, so it exercises the plumbing fast. A full run, end to end:
era replay ./run1 # rebuild the tree from the log and verify it
era best ./run1 # the winning program
era report ./run1 --svg # markdown report + best-score-so-far chartexamples/ ships three tasks:
| example | task | why |
|---|---|---|
examples/poly/seed |
fit a hidden quadratic; score = -SSE |
fast smoke test — one or two mutations, good for your first run |
examples/toy/seed |
60-city Euclidean TSP × 8 fixed instances; score = -mean tour length |
the acceptance task — real head-room: seed ≈ −32900, nearest-neighbour ≈ −6900, 2-opt ≈ −5800 |
examples/noisy/seed |
fit y = 0.5x + 3·sin(x) + ε from 60 rows; score = -MSE on a random bootstrap resample |
where the noise protections matter — unseeded, so every measurement differs; a memoriser scores −0.000000 on the searched metric and −3.911 on the held-out set, worse than the trivial baseline |
era run --task "<what a better program would do>" \
--seed <dir with the starting program> \
--eval "<command whose last stdout line is a float>" \
--metric-goal max|min \
--budget <number of expansions> \
--out <fresh output dir> \
[--engine pi] [--agent-bin PATH] [--engine-source DIR] \
[--c-puct 1.0] [--max-steps 20] \
[--eval-timeout 300] [--mutate-timeout 1200] \
\
[--eval-repeats 1] [--eval-aggregate mean|median|min|max] \
[--best-by mean|lcb] [--lcb-z 1.0] \
[--holdout-eval CMD] [--holdout-when best|always|never] \
\
[--recombine-every 4] \
[--ideas PATH] [--idea-policy least-used|round-robin|off]
era replay <out> # rebuild the tree from trace.jsonl and verify it
era best <out> [--export DIR] # print / copy out the best node
era report <out> [--out-file report.md] [--svg] [--sections ...] [--json]
era ideas list <book>
era ideas generate --task ... --from <paper|dir> --out <book> [-n 12]
era doctor [--engine pi] [--probe] # --probe also verifies the model answers
era serve <run flags...> # same run, with a live event stream on stdoutEvery default is the previous behaviour: --eval-repeats 1 and no
--holdout-eval is exactly a single measurement per node, and
--recombine-every 0 --idea-policy off is pure mutation.
era serve takes the exact same flags as era run and streams the run to
stdout as JSON lines, one event per line, as it happens — no waiting for the
run to finish, no re-reading trace.jsonl at the end. The trace file is still
written; the stream is a live copy of it, so the same run is both watchable
and replayable.
era serve --task "<...>" --seed <dir> --eval "python eval.py" \
--metric-goal max --budget 20 --out <fresh dir> \
[--agent-bin PATH] [--heartbeat 15.0] [--no-erase-cwd]
-
stdout is the protocol. Every event is one JSON object per line, flushed immediately. The search's human-readable chatter goes to stderr so the stream stays parseable.
-q/--quietsilences it. -
One vocabulary, two transports. The stream reuses the exact same event kinds and payloads as
trace.jsonl— a consumer sees precisely whatera replay <out>would verify afterwards. There is no second schema to drift. -
Raw engine frames are slimmed on the stream. The full RPC events the engine emitted stay in the trace (
nodes/<id>/.era/engine-trace.jsonl) and on disk; the stream carries a reduced frame (the eventtypeplus a few scalar fields), so a dashboard does not have to swallow the engine's entiremessageIndexhistory.era_metarecords whicheraversion, engine, budget and output dir produced the stream. -
A minimal control channel on stdin. The client sends JSON lines back:
client → era meaning {"type": "era_ping"}emit an era_heartbeatnow (a consumer can also set--heartbeat Nto get one every N seconds){"type": "era_abort"}stop after the current evaluation; the run ends as an interrupted-but-replayable run ( run_interrupted, exit code 0)Anything else on stdin is ignored. Unknown kinds in either direction are ignored — an older era and a newer UI stay compatible.
-
--no-erase-cwdkeeps absolute paths in the stream as-is; by default paths are rewritten relative to the working directory so the stream does not leak the machine's filesystem layout. -
Exit codes match
era run: 0 finished, 2 error.
The event vocabulary — the contract a visualizer is built against:
| kind | when | payload (besides kind, seq, ts) |
|---|---|---|
era_hello |
stream opens | — |
era_meta |
stream opens | era_version, python, engine, budget, heartbeat_s, out |
era_heartbeat |
every --heartbeat s / on era_ping |
running (nodes so far) |
era_exit |
stream closes | code, message |
run_started |
search begins | config, engine, fixtures, hidden, ideas |
config_warning |
a config assumption was relaxed | message |
select |
a node was chosen for expansion | iteration, selected_node_id, c_puct, candidates (full PUCT table) |
node_created |
a node directory was materialised | node_id, parent_id, name, path, source, operator, donors, idea_id, cost_usd |
mutation_started |
one bounded engine turn begins | node_id, parent_id, iteration, engine, engine_command, operator, donors, idea_id, operator_meta, instruction_chars |
engine_event |
an RPC event from the engine | slimmed frame: type, sessionId, timestamp, done, success, error, model, reason, cost, messageIndex, usage (input/output/cacheRead/cacheWrite/reasoning) |
mutation_finished |
the turn ends | node_id, ok, reason, cost_usd, usage, tool_calls, llm_steps, wall_s |
fixtures_restored |
protected files re-copied before a score | node_id, files |
evaluated |
a score was measured | node_id, score, eval_ok, reason, returncode, samples, score_std, score_sem, selection_score |
holdout_evaluated |
a held-out measurement | node_id, why, score, eval_ok, reason |
operator_fallback |
recombination was scheduled but could not run | iteration, reason |
backprop |
visit counts updated | node_id, touched |
run_interrupted |
the run was stopped (abort / Ctrl-C) | nodes |
run_finished |
the run completed | best_node_id, best_score, root_score, improved, num_nodes, total_cost_usd, selection_gap, operators |
A tree visualizer needs only node_created (the nodes), select + backprop
(which arm was taken, how visits moved), and evaluated (the scores) — the
rest is provenance. Because tree.json is a projection of this same log, a UI
can also just watch the stream and verify its rendering against era replay
afterwards.
run/
trace.jsonl append-only event log — the single source of truth
tree.json end-of-run projection: every node, score, visits, cost
best.json pointer to the winning node
best/ a copy of the winning node's workspace
nodes/
n0000/ the root = a copy of --seed
n0001/ one full program variant per node
.era/
instruction.md the exact prompt this expansion received
engine-trace.jsonl every RPC event the engine emitted, raw
engine-stderr.log the engine's stderr
turn.json usage / cost_usd / stop reason
A node is a complete program variant in out/nodes/<id>/. The root is a
copy of --seed; expanding node P copies P's directory to a new child and
mutates the copy. Build artefacts (__pycache__, .git, .venv, …) are never
copied, and stale bytecode is purged before every evaluation.
Per expansion, exactly one engine process is spawned with cwd = the child
directory. Over pi's RPC dialect (JSON lines on stdio):
| step | direction | message |
|---|---|---|
| ① | host → engine | {"id":"era-turn-1","type":"prompt","message":<instruction>} |
| ② | engine → host | agent_start / turn_start / message_end / tool_execution_* … (all logged) |
| ③ | host → engine | {"type":"abort"} once turn_end count reaches --max-steps |
| ④ | engine → host | agent_settled |
| ⑤ | host → engine | get_session_stats → authoritative cost |
| ⑥ | host → engine | close stdin, process exits |
The step cap is enforced by ERA, not by the engine. pi has no --max-steps;
ERA counts turn_end events and sends abort itself. That turned out to be the
right place for it anyway — it is engine-agnostic, so a future adapter inherits
the bound for free. A turn stopped this way is not a failure: it returns
ok=True, reason="step_cap", and its cost is still collected.
Why bound it at all: on an early run, one mutation spent 15 minutes running its own search loop inside a single turn, which makes ERA's outer loop redundant. The cap plus a prompt that states the budget keeps a mutation a single deliberate edit.
Honest cost accounting. get_session_stats returns the session cost even
for a turn ERA aborted, so cost_known is normally true. When the engine
dies before answering, ERA falls back to summing what the event stream
reported, marks the node cost_known: false, and labels the run total a
lower bound in tree.json (cost_complete), in the run log, in era best
and in era replay. Missing spend is never quietly reported as $0.
The operator decides what the next instruction is. The scheduler is deterministic, because replay depends on it:
if recombine_every > 0 and iteration % recombine_every == 0 and recombine.applicable():
recombine
else:
mutate
mutate— parent source + its score + the last evaluation's stdout/stderr tail + the best score in the tree + "improve it to {maximise|minimise} the metric".recombine— the selected node stays the primary parent (the workspace is copied from it, and visits back-propagate along that edge). A donor is chosen from the scored nodes that are neither its ancestors nor its descendants: highest score, ties broken by lowest id. Both sources and both scores go into the prompt, with an explicit instruction not to simply pick one. The extra edge is recorded asdonorson the node — FUTS still sees a single-parent tree. When no unrelated scored node exists, the expansion falls back to mutation and anoperator_fallbackevent records why.- Idea injection is a decorator, not a third operator: an idea from
--ideasis spliced into whichever instruction the scheduler produced, so mutation and recombination both get it. Selection is deterministic (least-used, ties by index). Theidea_idlands in the trace, which is what lets the analysis layer answer "which idea preceded a breakthrough".
An idea book is a directory of .md/.txt files, or one file with one idea
per line. era ideas generate drafts one from a paper, a directory of
reference material, or just a task description, using one bounded engine turn.
Recombination is our extension, not the paper's. The paper recombines out-of-tree: it takes 11 methods, forms 55 pairs, and seeds 87 independent searches from them. Here recombination is in-tree — an extra donor on one expansion,
donorsrecorded, FUTS unchanged.
After the turn, ERA runs --eval inside the child directory and parses the
last line of stdout as a float. Non-zero exit, timeout, or unparseable
output ⇒ the node keeps a worst-possible sentinel score (-1e18, oriented by
--metric-goal) and stays in the tree — a failed branch is information, not
garbage. nan and inf are rejected.
Fixture protection. If the seed contains a .era-fixtures manifest, the
files it lists are restored byte-for-byte from the pristine seed before every
evaluation. The scorer is outside the mutable search space: a mutation cannot
raise its score by rewriting eval.py. See Reward hacking below.
A score is a random variable. --eval-repeats N measures each node N
times; --eval-aggregate folds the samples into one number; the spread is
kept (score_samples, score_std, score_sem per node) rather than
averaged away. --best-by lcb ranks on mean − z·sem, so a candidate that
scored well once and wildly otherwise cannot dominate selection on luck. FUTS
still reads one field, NodeRecord.score, so noise handling attaches at
exactly one point.
Held-out data the candidate never sees. .era-fixtures protects the
scorer's bytes; a held-out set has to be protected from being read. So
there is a second manifest, .era-hidden:
copy_workspace()skips those files → they were never in a node directory;- the code snapshot therefore cannot see them → they never enter a prompt;
Scorer.holdout()stages them in for the duration of one measurement and deletes them again.
--holdout-eval CMD measures that score and never optimises it;
--holdout-when best|always|never says how often to pay for it. The run
summary then reports a selection gap: how much the searched score improved
versus how much the held-out score improved. Overfitting becomes a printed
number instead of an assumption. A test asserts that after a full run no hidden
filename appears in any node directory or any instruction.md.
era/futs.py is a faithful re-implementation of the algorithm released with
the paper, verified against it by tests/test_futs.py, which runs both through
a 60-step simulated search and asserts identical rank scores, PUCTs, selections
and visit counts at every step. It is not modified by any of the work above.
The three things that make it flat UCB rather than MCTS:
- No descent from the root. Every node in the tree is an arm. All arms are flattened into one set and a single global argmax picks what to expand.
- Rank scores, not min-max. The exploitation term is the node's rank among
all nodes normalised to
[0,1]. Scale-free and outlier-proof — one wild score cannot squash the rest of the tree into a corner. - Visits only. Back-propagation increments
num_visitson the node and its ancestors. No value is averaged upward.
prior = 1 / N
rank_score = rank(node) / (N - 1) # 0 = worst, 1 = best
puct = rank_score + c_puct * prior * sqrt(total_visits) / (1 + visits)
--metric-goal min is handled by ranking on -score while reporting the raw
value, which is the only addition over the reference implementation.
Every node creation, selection decision (with the full PUCT table), raw engine
event, evaluation, held-out measurement, operator fallback and
back-propagation is appended to out/trace.jsonl. tree.json is a
projection of that log, not a parallel source of truth. era serve is the
same log with a live transport: the exact same kinds and payloads, flushed to
stdout as each event happens (see the table in
Live stream).
era replay <out> folds the log back into a tree and cross-checks it against
tree.json (ids, parents, scores, visit counts), printing verify: OK or a
diff. A truncated log rebuilds a valid smaller tree, which is what makes an
interrupted run inspectable.
era report <out> reads only the trace and the node directories:
| section | answers |
|---|---|
| overview | task, engine, operator mix, cost, selection gap |
| breakthrough | which nodes set records, which were jumps, what the engine said it did, the diff against the parent, and which idea/operator preceded each |
| diversity | are the programs actually different — textual (difflib on normalised source), structural (cosine of Python AST node-type histograms), behavioural (number of distinct scores), plus which nodes re-derived a program the tree already had |
| saturation | did the search stall, or did the metric stop discriminating — the two look identical in a log and call for opposite responses |
--svg writes a hand-drawn best-score-so-far chart (no plotting dependency).
--json emits the same analysis objects the prose was rendered from, so a
consumer never re-derives numbers that could drift.
Every verdict prints the rule that produced it. For example the jump threshold
is stated inline, including the admission that with --eval-repeats 1 the
standard-error term is zero and the report therefore cannot separate a real
jump from luck.
The diversity numbers are standard-library proxies. The paper uses Gemini embedding cosine distance. Ours are comparable across runs of
eraand are not comparable with the paper's figures.
Two real runs, reported as measured. Nothing below is estimated or reconstructed.
An early full-length run on the TSP task, driven by a coding agent. It is kept
because it is the only long real run, and its evidence is checkable rather than
quoted: a redacted copy lives in
archive/tsp-16-expansions/ and replays.
era replay archive/tsp-16-expansions
# verify: OK — replayed tree matches tree.json (17 nodes: ids, parents, scores, visits all equal)
era report archive/tsp-16-expansions --svgroot score (n0000, = the seed) |
−32932.939820 |
best score (n0004) |
−6061.613464 |
| improvement | 5.43× shorter mean tour |
| nodes | 17 (1 root + 16 expansions) |
| total cost | $0.11417 — a lower bound |
| wall clock | 16781.75 s (4.66 h) |
| LLM steps / tool calls | 250 / 265 across all mutations |
n0000 score=-32932.9 visits=16
`-- n0001 score=-6096.59 visits=16 $0.00329
`-- n0002 score=-6070.49 visits=15 $0.00798
`-- n0003 score=-6065.44 visits=14 $0.01029
`-- n0004 score=-6061.61 visits=13 $0.01584 <= BEST
`-- n0005 score=-6061.61 visits=12 $0.01866
|-- n0006 score=-1e+09 visits=1 $? MUT-FAIL
`-- n0007 score=-6061.61 visits=10 $0.02069
|-- n0008 score=-6086.23 visits=1 $? MUT-FAIL
`-- n0009 score=-6061.61 visits=8 $? MUT-FAIL
`-- n0010 score=-6061.61 visits=7 $? MUT-FAIL
`-- n0011 score=-6061.61 visits=6 $? MUT-FAIL
`-- n0012 score=-6061.61 visits=5 $0.02145
|-- n0013 score=-32932.9 visits=1 $? MUT-FAIL
`-- n0014 score=-6061.61 visits=3 $0.01597
`-- n0015 score=-6061.61 visits=2 $? MUT-FAIL
`-- n0016 score=-1e+09 visits=1 $? MUT-FAIL
The four steps that mattered. The algorithm the agent invented was not there at the start — it was built mutation by mutation:
| expansion | change | score |
|---|---|---|
n0000 → n0001 |
identity tour (points in input order) → nearest-neighbour construction + 2-opt, tried from every starting city | −32932.9 → −6096.6 |
n0001 → n0002 |
+ Iterated Local Search: double-bridge (4-opt) perturbation, re-apply 2-opt, accept if better | −6096.6 → −6070.5 |
n0002 → n0003 |
nearest-neighbour construction → farthest-insertion; perturb + accept if within 2% of best | −6070.5 → −6065.4 |
n0003 → n0004 |
2-opt local search → VND (2-opt + relocate) inside the ILS walk | −6065.4 → −6061.6 |
The jump from −32932.9 to −6096.6 in one mutation is the paper's whole point:
one turn went from a placeholder that returned the input order to a real
nearest-neighbour + 2-opt solver. The last three expansions were fine-tuning on
top of it — each bought a little, and after n0004 nothing could be bought at
all. The mutations after that all reproduced one of these methods or broke on
timeout; they are the plateau the saturation section describes.
Why the cost is a lower bound. 8 of the 16 turns hit --mutate-timeout and
so never emitted the event that carried cost. Their spend is unknown, not zero:
they are marked cost_known: false, rendered $? above, and $0.11417 is the
sum over the 8 turns that did report. The true total is higher. (This is the
failure mode pi's get_session_stats removes — see run B.)
A caveat about the archive. The published copy was produced by
scripts/redact_run.py, which keeps only the event kinds that describe the
search — 100 of 449,758 events, taking the directory from 103.6 MB to 263 KB,
because everything else was raw engine chatter — and scrubs identifying
strings. So the report prints engine: ?: that field was scrubbed, and
inventing a value would be worse than an honest question mark. Per-node engine
logs are gone; solution.py, eval.py, the scores, the tree and the selection
decisions are all intact, which is exactly the set era replay verifies.
The smoke run that proves the adapter against a real engine.
era run --task "<improve solve() to maximise eval.py>" \
--seed examples/poly/seed --eval "python eval.py" \
--metric-goal max --budget 3 --max-steps 8 --out ./smoke-pi| engine | pi 0.84.1, RPC over stdio |
root score (n0000) |
−463715 |
best score (n0002) |
−0.0 (exact fit) |
| nodes | 4 (1 root + 3 expansions) |
| wall clock | 75.44 s |
cost_known |
true on every node |
| tokens | 12,393 in / 2,856 out / 8,192 cache-read |
| total cost | $0.00000 — see below |
[ 1/3] select n0000 (rank=0.500 puct=0.500 visits=0) -> n0001 score=-463715 (no change)
[ 2/3] select n0001 (rank=1.000 puct=1.354 visits=1) -> n0002 score=-0 (+463715)
[ 3/3] select n0002 (rank=1.000 puct=1.373 visits=1) -> n0003 score=-0 (no change)
Three honest notes:
$0.00000withcost_known: trueis not a bug and not a free lunch. The proxy this run was pointed at reports usage but carries no price table, so the engine's own cost field is genuinely0. The token counts above are real. Printing a dollar figure computed from prices ERA guessed would be inventing data, so ERA reports what it was told.- The first expansion improved nothing, and the third reproduced the optimum. On a task one mutation can solve, that is the expected shape — this run tests the plumbing, not the search.
- Two identical invocations produced different trees. The engine is stochastic; ERA is deterministic given the same scores. Reproducibility here means the replay is exact, not that a rerun repeats.
python scripts/check_acceptance.py ./smoke-pi passes 6 of the 7 criteria.
The one that fails is "≥ 8 nodes", which a 3-expansion smoke cannot satisfy by
construction; the criterion that mattered for the engine — "RPC protocol
visible (prompt / turn_end / agent_settled), no credential commands" — passes
against the real engine.
An early acceptance run used the polynomial task. The agent solved it exactly
on mutation #1 (-463715 → -0.0, a perfect fit). With the metric saturated
and 15 expansions of budget left, the next mutation started reasoning —
visibly, in its trace — about intercepting print so the last stdout line
would read better than the truth.
Fixture protection did not cover that: eval.py was pristine, but eval.py
imports the candidate, so the candidate could have hijacked the output
stream.
All harnesses were hardened in response, and tests/test_harness_integrity.py
pins the defences against four concrete attacks (print an extra final line,
rebind builtins.print, print during solve(), flood stdout):
- the candidate is imported and called with
stdoutredirected into a buffer; - the verdict is written to the stdout object captured before the import,
and via
write()rather thanprint(), so rebindingbuiltins.printachieves nothing; - the TSP harness additionally validates that the returned tour is a genuine permutation, and caps total wall-clock.
Two general lessons:
- A saturated metric is an incentive to cheat. Budget past the point where
the metric can still improve and the search will look for slack elsewhere.
This is also why
era report's saturation section exists. .era-fixturesprotects the scorer's bytes, not its process. For untrusted candidates, an eval harness should run the candidate out-of-process (or in a sandbox) and read the score from a channel the candidate cannot write to. The in-process hardening here raises the bar; it does not seal it — a candidate can still write to fd 1 directly or callos._exit.
era/
cli.py argparse -> config -> dispatch. Thin; no algorithm.
config.py RunConfig / ScoringConfig / OperatorConfig
node.py NodeRecord
search.py the loop. Orchestration only.
futs.py PINNED. selection + visit backprop.
prompt.py instruction text, in composable sections
evaluate.py run one command, parse one float
scoring.py repeats / aggregation / LCB / held-out measurement
workspace.py node copy, .era-fixtures restore, .era-hidden staging
ideas.py the idea book: load, pick deterministically, account
idea_gen.py [detachable] one engine turn that drafts ideas
trace.py TraceWriter, read_trace, rebuild_tree
engines/ base.py (Engine protocol) / pi.py / __init__.py (registry)
operators/ base.py (Proposal) / mutate.py / recombine.py / scheduler.py
analysis/ load / breakthrough / diversity / saturation / report
The four newer layers do not import each other. They exchange data through
search.py's orchestration and through trace.jsonl:
| layer | may import | must not import |
|---|---|---|
engines/ |
stdlib | anything else in era except engines.base |
operators/ |
node, config, prompt, ideas |
search, engines |
scoring.py |
evaluate, workspace, config |
search, operators |
analysis/ |
stdlib | search, engines, operators, scoring |
analysis/ is the strict one: it reads trace.jsonl and the node directories
and nothing else, so a run directory can be analysed by someone who does not
have the code that produced it. Two tests assert this by parsing the module
ASTs — one for the forbidden era layers, one for third-party imports.
Both were built to be removable, because neither is part of the search:
era/idea_gen.py(the "deep research" step). Delete the file.cli.pyregistersera ideas generatebehind atry: import ... except ImportError, so the subcommand simply disappears;era ideas listandera run --ideaskeep working, because they only needera/ideas.py, which is pure stdlib and never talks to an engine.era/analysis/(reporting). Delete the package and theera reportbranch incli.py. Nothing else imports it. Conversely, the package can be copied out on its own and pointed at any run directory.
| paper | here | |
|---|---|---|
| selection | FUTS | same algorithm, pinned by a differential test |
| mutation | an LLM rewrites the code | same, via a pluggable engine adapter |
| recombination | out of tree: 55 pairs of 11 methods seeded into 87 independent searches | in tree: an extra donor on one expansion, donors recorded, FUTS unchanged. Our extension, not evaluated against theirs. |
| idea injection | expert ideas / a "brainstorm" tree in the generation prompt | an idea book file plus a detachable generator; deterministic selection |
| diversity metric | Gemini embedding cosine | difflib + ast + score-bucket proxies |
| scale | many parallel searches, large budgets | one sequential search |
pip install pytest
python -m pytest tests -q # no network, no API cost
python scripts/offline_demo.py # run -> replay -> best, end to endtests/fake_pi.py is a real RPC engine — same JSON-lines wire format, same
handshake, honours abort, answers get_session_stats — whose "intelligence"
is a deterministic local search. The entire ERA loop runs over the same code
path as the real binary.
| file | covers |
|---|---|
test_futs.py |
Flat-UCB vs. the paper's reference implementation, step by step; rank-not-min-max; ancestor-only backprop; that it does not degenerate into best-of-N |
test_engines.py |
no model/credential flags or RPCs; the step cap aborts with ok=True; cost comes from get_session_stats; a killed process yields cost_known=False |
test_scoring.py |
aggregation and standard error; LCB selection; that .era-hidden files never reach a node directory or an instruction |
test_operators.py |
scheduler determinism; the recombination prompt carries both sources; donors survive replay; fallback when no unrelated node exists |
test_analysis.py |
breakthrough detection; "search stalled" vs "metric exhausted"; that the layer imports neither the search nor any third-party package |
test_e2e_offline.py |
a full run: tree shape, visit-count consistency, fixture restoration, RPC presence, absence of credential ops, replay round-trip, truncated-log replay, cost accounting |
test_harness_integrity.py |
the four reward-hacking attacks above, seed determinism, tour validation, that examples run standalone |
| script | what it does |
|---|---|
scripts/check_acceptance.py <run> |
audits a finished run against all seven acceptance criteria, re-running --eval on the root and best nodes rather than trusting the run's own summary |
scripts/acceptance_run.py [example] [budget] [out] |
launches a real run; exists only to keep non-ASCII arguments out of the Windows shell |
scripts/offline_demo.py |
run → replay → best against the stub engine, no network |
scripts/redact_run.py <run> <dest> --scrub RE --drop-key RE |
shrink and scrub a run for publication, then re-scan every byte written and fail loudly if a pattern survived |
MIT — see LICENSE.