Scalable Dashboards (Wave 1): coordinated upstream PR set — tracking + reproduction + evidence

[speediedan]

Thank you so much to the SAEDashboard, SAELens, Neuronpedia and circuit-tracer maintainers and
communities for making these foundationally valuable contributions to the open-source
interpretability ecosystem! I'm using them extensively in the downstream world-model analysis
framework I'm building (Interpretune) and couldn't appreciate the work more. This coordinated PR
set comes entirely out of that heavy use, and each PR carries the same note.

Overview

Steady-state seconds per batch (stacked substages, left axis) and generation throughput (right
axis) across three batch shapes. The legacy lane is flat — 1,280 / 1,287 / 1,037 features/min — because
it is bound by a per-feature CPU cost. The columnar lane climbs 11,874 → 17,852 → 25,218 as the
feature axis grows. That divergence, not any single multiple, is the point: the advantage grows with
the shape and with the hardware. Measured on a consumer RTX 4090 (24 GiB), gemma-3-1b-it,
262k-width transcoders.

Scalable Dashboards (Wave 1) — long form "Scalable & Example-Aligned Dashboards" — is a
coordinated set of upstream PRs addressing three structural limits that currently compound to slow
dashboard experimentation, and with it growth in Neuronpedia / SAEDashboard / SAELens usage:
generation cost that scales badly with feature count; publishing a novel dashboard requiring
maintainer intervention; and specialized-generation code having no seam to upstream into. These
projects are foundational infrastructure for open-source mechanistic interpretability — the aim here
is to release a brake on the ecosystem they anchor, not to redesign it.

Wave 1 makes the existing DB-backed dashboards substantially more scalable and eases/accelerates
specialized generation (arbitrary HuggingFace prompt datasets, reusable custom pretokenization,
streamlined custom source sets), while keeping the DB as the dashboard source of truth and the
webapp unchanged.

The speedup is not a single constant. On a consumer 24 GiB card (RTX 4090) the columnar lane
runs ~3.7x faster at a small shape and 24.3x at the largest swept (4096 features x 256 prompts).
Legacy throughput is FLAT across shapes — a per-feature CPU cost floor — whereas the columnar lane is
GPU-batched, so its advantage grows with the feature axis and with available VRAM. Larger devices
should exceed these numbers. Only BASIC multi-GPU generation has been explored (scope/limitations:
multi-GPU generation notes); further distributed generation gains should be readily
available but were not pursued here.

Wave 1 is also the foundation for Wave 2. The schema-aligned Arrow/Parquet format landing here is
what would later allow dashboards to be published to and streamed from a hub — read directly from
hub-resident columnar data alongside the DB path, so publishing no longer requires maintainer
intervention. Wave 2 is a PROPOSAL, not a commitment: Wave 1 preserves the legacy path at functional
parity, and any deprecation decision is entirely the maintainers'. But reviewing the storage format
with that possibility in mind is the most useful lens, because it is the piece that keeps the option
open.

Important

Ownership framing: the upstream repos — principally Neuronpedia, together with SAEDashboard —
own the core dashboard APIs and generation protocol and will continue to own them. Interpretune
provides one example generation/import orchestration pipeline that exercises the proposed
scalable, example-aligned-customizable, and (in the next PR wave) streamable dashboard generation.
The PRs demonstrate improvements; they do not redefine or encroach on upstream interfaces or
ownership. And while Wave 2 proposes a transition toward a streaming architecture, these
scalability PRs keep the existing legacy generation path available and at functional parity
(deprecated, with the removal timeline entirely at the maintainers' discretion).

The PR set

#	Repo	PR	Role
1	SAELens ✅ MERGED (v6.49.0)	decoderesearch/SAELens#721	Supporting: gemma-scope-2 Neuronpedia aliases + registry repair; in-place decoder-norm fold (storage-preserving); get_batch_tokens(move_to_model_device=…) seam
2	SAEDashboard	jbloomAus/SAEDashboard#74	Core: opt-in columnar (Arrow/Parquet) GPU-batched generation lane; prompt/pretokenization/windowing contracts; parity + golden + import-parity suites; opt-in dashboard hygiene flags; peak-memory controls
3	neuronpedia	hijohnnylin/neuronpedia#217	neuronpedia-utils Python columnar import tooling (+ webapp robustness for large local source sets + localhost/gemma inference env); body carries the ecosystem overview
4	circuit-tracer	#79 (pre-existing, refreshed)	transformers-v5 support lineage the ecosystem stack builds on, plus a correctness fix (double-BOS position drift in the feature-intervention path); independent of merge order, but see the note below

SAELens#721 merged 2026-08-09 and shipped in v6.49.0, so the remaining soft merge order is
SAEDashboard → neuronpedia (SD is the core; NP's importer consumes SD's columnar contract).
Nothing hard-fails out of order. The released SAELens is what NP's temporary inference pin can now
flip to.

Note

circuit-tracer#79 stays outside the merge order but is now the wave's longest pole for
dependency hygiene. Its refresh carries a genuine correctness fix, not just the transformers-v5
lineage: released circuit-tracer re-tokenizes a raw str prompt with add_special_tokens=True,
double-prefixing an explicit BOS and shifting every position index in the feature-intervention
path. Nothing in the SL/SD/NP PRs depends on it, so the merge order is unchanged — but until it
lands upstream, any downstream consumer of that path needs a SHA pin rather than a release. That
is a tolerable state for interpretune (which already pins the fork) and NOT something we ask of
Neuronpedia or SAEDashboard: neither imports the affected surface. We are carrying the pin
ourselves precisely so the other three PRs stay release-clean.

Interpretune's own Wave 1 surface (already on main here): the example orchestration pipeline
(interpretune.utils.neuronpedia_dashboard_pipeline — multi-GPU workers, batch-level resume,
overlap orchestration), the neuronpedia_db_utils import wrappers, the benchmark/parity/profiling
suite (scripts/run_dashboard_benchmark_suite.py + regenerable reviewer artifacts), and the custom
pretokenization examples (src/it_examples/utils/dashboard_pretokenization_rte.py). Operator docs:
docs/neuronpedia_dashboard_pipeline.md, scripts/dashboard_benchmark_suite_usage.md.

Reproduction quickstart

Important

Prerequisites (the setup script checks these itself): git and uv on PATH; docker only
if the local Neuronpedia DB needs bring-up; bash >= 4.3 for the env build (macOS:
brew install bash); HuggingFace access to the gated google/gemma-3-1b-it model — accept
the license on the model page, then hf auth login or export HF_TOKEN=...
(HF_GATED_PUBLIC_REPO_AUTH_KEY is honored as a fallback). Root is never required; nothing is
pushed; no existing checkout is modified.

1. Environment — one guided command (Linux/macOS):

   git clone https://github.com/speediedan/interpretune.git && cd interpretune
   python scripts/setup_dashboard_benchmark_env.py --worktrees-dir <dir-for-baseline-worktrees>

   Only four repos are involved — interpretune plus the three PR branches (SAEDashboard,
   SAELens, neuronpedia), located or cloned for you. TransformerLens (3.5.1), nnsight (0.7.0),
   and circuit-tracer install from interpretune's dependency pins during the env build — no
   checkouts needed. The script transparently: recreates the preserved pre-PR baseline
   worktrees (SD-7886eaa+benchmark_patches / SL-3eea6552 / NP-5a33f17) by checking out the
   pinned commits (stable fork refs benchmark-baseline-7886eaa / benchmark-baseline-3eea6552)
   and applying the audited benchmark-enablement patch set vendored at
   scripts/benchmark_baseline_patches/ (its README classifies every patch: profiling
   instrumentation vs the three small functional alignments parity requires — the resulting tree
   state is verified against pinned expectations); ensures the neuronpedia local-stack .env
   defaults (Postgres host port/data dir + HF cache paths, appended only when missing) and checks
   the local Postgres (offering the docker compose bring-up); builds the integrated venv via
   scripts/build_it_env.sh (SAEDashboard + SAELens editable from source); offers to build any
   missing benchmark prompt datasets (the pretokenization commands of record — tokenizer-only,
   CPU, a few minutes per set); and writes a benchmark_env.sh capturing the required
   environment variables. Non-destructive by design: --dry-run prints the full plan without
   executing anything, existing checkouts are never switched or modified (dirty trees prompt
   stash/continue/abort), existing worktrees are verified rather than recreated, and an existing
   venv is only cleared after explicit confirmation. --help lists a flag for every prompt;
   --yes runs non-interactively. On completion it reports the detected GPU against the
   reference benchmark hardware (NVIDIA GeForce RTX 4090, 24 GiB).

2. Benchmarks: source <worktrees-dir>/benchmark_env.sh, activate the built venv, then
   python scripts/run_dashboard_benchmark_suite.py --mode threeway (~25 min on the reference
   RTX 4090) reproduces the accepted three-way benchmark (preserved pre-PR baseline vs in-tree
   legacy vs columnar) and packages the reviewer artifacts. --mode full (~2 h on the reference
   GPU) adds the batch-shape scaling sweeps and the n-prompts scaling curve — long enough that it
   is worth backgrounding:

   nohup python scripts/run_dashboard_benchmark_suite.py --mode full \
     --session-root /tmp/np_dashboard_generation_profiles/full_$(date +%Y%m%d) \
     --package-root /tmp/dashboard_benchmark_packages/full_$(date +%Y%m%d) \
     --run-tag dashboard-bench \
     --local-db-url "postgres://postgres:postgres@127.0.0.1:5433/postgres" \
     > /tmp/full_wave.log 2>&1 &

   The setup script prints this exact command (with your local DB URL) on completion. Full usage:
   scripts/dashboard_benchmark_suite_usage.md.

3. Parity: SAEDashboard scripts/run_parity_tests.py (unit mechanism + CUDA integration +
   golden-batch + three-way import parity).

4. Production-scale example: the multi-GPU 26-layer generation/import flow in
   docs/neuronpedia_dashboard_pipeline.md.

Performance evidence (regenerable)

Reviewer package full_20260727 — Evidence: rendered profiling notebook · summary tables · annotated flow diagram · artifact index · full package — (summary tables, unified flow diagram,
executed profiling notebook incl. the by-config legacy-vs-columnar charts and the n-prompts scaling
curves; manifest records the exact four-repo lineage and refuses dirty trees). Link the notebook's
rendered HTML export as the primary artifact (charts/data only — the .ipynb's collapsed code
cells are not honored by every viewer, e.g. GitHub's renderer / some VS Code builds), and inline the
Monology: in-tree legacy vs columnar_gpu by config chart image (the 1,037 → 25,218 f/min curve at
4096x256) up front as the attention-catcher.

Headlines (RTX 4090 24 GiB / CUDA 13 / torch 2.13, gemma-3-1b-it, 262k transcoders; package
full_20260727, manifest lineage SD-1c36394/SL-990b6b37/NP-f8b13ef2/IT-9bf79d4, dirty repos none):

• Columnar vs legacy generation: 3.7x at the accepted RTE example-aligned shape (3,140 vs 852
  f/min), 9.3x at the accepted Monology shape (11,874 vs 1,280), up to 24.3x at the largest swept
  feature-axis shape (25,218 vs 1,037 at 4096x256) — legacy is flat across configs (per-feature
  CPU cost floor); columnar scales with batch shape.
• Prompt-dimension scaling curve packaged (n_prompts {2490, 4096, 24576} at 4096x256, with peak-GPU
  memory tracked against the 24 GiB device ceiling — 15.2→21.7 GiB under the opt-in peak-memory
  controls).
• In-tree legacy ≈ preserved pre-PR baseline within noise (the deprecated lane is faithfully
  preserved).
• Production-scale: two full 262,144-feature layers generated + imported in 91 min total at
  100.00% per-feature parity; all 26 layers of gemma-3-1b-it/262k built (6.8M neurons, ~250M
  activations).

Roadmap: Wave 1 → Wave 2

• Wave 1 (this set): scalable + example-aligned generation, schema-aligned columnar artifacts,
  Python import utilities alongside the TypeScript admin importer. DB remains the source of truth;
  webapp unchanged.
• Wave 2 (future set, to be co-designed with the Neuronpedia maintainers): hub-resident,
  streamable dashboards — publish bundles as HF Hub datasets with a manifest/versioning
  convention and a new front-end interface/API; a streaming read mode ALONGSIDE the DB mode across
  the existing services (remote Parquet range-reads or locally cached Arrow chunks); potentially
  migrating much DB-resident dashboard data to the hub format (DB reduced toward core metadata —
  the local Activation table is ~670 GB after one full 262k source set); the importer's future
  (refactor/adopt/retire) decided with real usage data. User-generated feature explanations and
  future J-lens artifacts ride the same convention. The Wave 1 Parquet bundles are deliberately
  shaped to make this possible without committing anyone to it now.

Tracking

• SAELens PR merged 2026-08-09, released in v6.49.0. Follow-ups now unblocked: NP inference
  pin and the interpretune git-dep can both move to a released SAELens floor
• SAEDashboard PR opened → reviewed → merged; then: interpretune SD git-dep → released floor
• neuronpedia PR opened → reviewed → merged
• circuit-tracer#79 refreshed (rebase + validated additions + one ping) — elevated priority:
  carries the double-BOS correctness fix, so until it merges every downstream consumer of the
  feature-intervention path is on a SHA pin rather than a release. Track it as the wave's
  longest-lived pin even though nothing blocks on it.
• Upstream issues filed with the wave: TransformerLens bridge processed-weights device
  migration; nnsight /status bare-entry client crash; NP production re-import of
  gemma-3-4b-it/16k layers 13-33
• Evidence package links current at every rebase (the benchmark suite regenerates the whole
  package; see the reproduction quickstart above)
• Interpretune-side review enablement landed — nothing here blocks the upstream PRs, but it
  is what makes them cheap to review. The shared verification guide linked above is live on
  ReadTheDocs, and both published corpora are downloadable and importable in one command, so a
  reviewer can populate a local Neuronpedia and exercise these PRs end to end without spending
  GPU hours. Both corpora carry multiple Parquet row groups, which is what actually makes a
  single feature's dashboard cheap to range-read (the page index alone does not — with one row
  group per file a reader still pays for the whole file). Each corpus records the layout it was
  written with in its own dashboards.json.

@hijohnnylin @chanind

---

Reviewing this wave

Order

With SAELens#721 merged, the remaining order is SAEDashboard → neuronpedia — but nothing
hard-fails out of order and each PR can be reviewed and tested on its own.

Cheapest checks first

To sanity-check the headline claims without installing anything:

Question	Where to look	Cost
Is it actually faster, and by how much?	docs/benchmark_artifacts/full_20260727/benchmark_summary.md — committed tables, plus tables/*.md and manifest.json	seconds, nothing installed
Did legacy output change?	The parity tables in the same directory: 100.00% raw and value-bearing parity, 0 mismatches	seconds
What was measured on?	manifest.json — records the exact lineage (SD-40bdc62 / SL-978a9654 / NP-6484b342) and that no repo was dirty	seconds

Headline figures (RTX 4090 24 GiB, CUDA 13 / torch 2.13, gemma-3-1b-it, 262k transcoders):

Shape	Legacy s/batch	Columnar s/batch	Speedup
RTE 512x128	36.06	9.78	~3.7x
RTE 2048x128	142.47	10.95	~13x
Monology 1024x256	47.99	5.17	~9.3x
Monology 4096x256	237.02	9.75	~24x (1,037 → 25,218 features/min)

Per-PR verification

Each PR body carries its own verification table, either "Verifying this PR yourself" or "Manual Verification for this PR" table — the commands, expected counts, and the known upstream gotchas for that repo.

Dashboard Review Options: Notebook Example, Pre-generated Hub Download, Local Generation

All three are documented once for the whole wave here: Verifying dashboard generation and local usage.

The pre-generated hub download is the cheapest of the three and needs no GPU: two 26-layer
gemma-3-1b-it / gemmascope-2-transcoder-16k corpora are published as public HF Storage Buckets —
monology (24,576 × 128 tok) and RTE (2,490 × 319 tok, example-aligned) — and one command downloads
and imports either into a local Neuronpedia. Cross-PR review order and the shared caveats are under
Order and Cheapest checks first above.

This umbrella issue lives in the interpretune repo for now; happy to move it somewhere more central
(neuronpedia, say) if that suits the maintainers better.

What this wave does not cover

• Hardware. Every measurement is one consumer RTX 4090 (24 GiB). Nothing was run on multi-node or
  datacenter-class hardware.
• Multi-GPU generation is basic only — scope and limitations are in the pipeline guide.
• Models. gemma-3-1b-it and gemma-3-4b-it only; 16k and 262k widths only.
• Production Neuronpedia. All import figures are against a local Postgres.
• The "40-test import contract suite" is not selectable as a single pytest invocation at HEAD —
  no marker or path isolates it. The evidence that it is unmodified is the additive-only diff
  recorded in the neuronpedia PR body.