Conversation
main to dev sync v0.8.4
The Phase 2 plan existed only in conversation and was lost to a context compaction, leaving two comments in store.js pointing at a design nobody could read. This writes it down as a specification. The substance: panel *settings* become per-panel with a global link toggle defaulting to linked, rather than global-plus-overrides (two sources of truth for every value, and no clean answer for what a slider shows) or always-independent (breaks the default case and reintroduces the drift linkColorScale exists to prevent). The push-to-other-panel button that prompted this work falls out of per-panel state as an object copy. Surface is measured rather than guessed: 4 files, 17 bare useStore() destructure sites, 25 selector calls, 19 LayerPanel sections — and notably 0 data hooks, since they take settings as props from ViewerPanel rather than reading the store. Staged 2a–2e so the large mechanical migration lands while behaviour is still frozen and any regression is unambiguous. Also recorded, both found while checking the plan against the code: - CLAUDE.md described edgeFile as global in three places. Phase 1 moved it to panels[i].edgeFile and left no global behind, so the docs contradicted the code. Corrected here rather than deferred — a stale architecture note is worse than a missing one. - regions and measurements carry no panelIndex and Viewer renders them unfiltered, so every region draws in both panels at identical image-pixel coordinates. Harmless with one dataset in both panels, wrong with two: a polygon on a 6.5 mm Visium capture area redraws at those coordinates on a 55 µm seqFISH ROI. Logged as a Phase 1 bug to fix separately, not folded into this plan. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Split screen shipped with `regions` and `measurements` still global and carrying no panel index. The visible symptom is that every annotation draws in both panels at identical image-pixel coordinates — a polygon over a 6.5 mm Visium capture area reappearing over a 55 µm seqFISH ROI, where it means nothing. Two consequences were worse, because they produce confident wrong answers rather than obvious breakage: - CSV export read `region.panelIndex ?? 0` while nothing ever *wrote* panelIndex, so every export resolved against panel 0's dataset. Exporting a region drawn in panel 1 posted panel 1's cell ids to panel 0's endpoint. The read side was written for a field the write side never set. - Measurement labels compute `distPx * pixelSize` using the *rendering* panel's pixel size, so one measurement read as two different distances. Verified on CosMx (1.0 µm/px) beside MERSCOPE (0.108 µm/px): the same 100 px line reads 100.0 µm and 10.8 µm. Every annotation now carries `panelIndex`, stamped at creation; ViewerPanel renders only its own; the in-progress polygon tracks which panel is drawing so its dashed outline does not shadow the other; and clearAnnotations is scoped, since the Clear button lives in each panel's own toolbar. Omitting the index still clears everything. Anything created before this has no panelIndex and is read as panel 0, the only place it could have come from. The sidebar region list is shared, so in split mode each entry is labelled with its panel — a bare cell count does not say which tissue it came from when the panels hold different datasets. Adds Vitest and the repo's first frontend tests (10, in src/store.annotations.test.js), written failing before the fix. Store logic is plain JS, so these need no DOM and no jsdom. This is a small down payment on the risk named in docs/split_screen_phase2.md: that plan migrates ~30 state keys across 19 components with no automated check behind it. Verified in a two-dataset split: annotation scope 0/1 per panel, export resolves to merscope-vpt-smallset rather than cosmx-mousebrain, and the measurement label renders only in its own panel. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… 2a) First stage of docs/split_screen_phase2.md, and deliberately behaviour-frozen: the ~30 display settings move off the store root onto each panel, but every write still lands on every panel, so one sidebar drives both exactly as before. Landing the large mechanical diff while behaviour is fixed means any regression is unambiguously a bug in the move rather than a disagreement about intent. - makeSettings() builds the defaults. A factory rather than a constant because `layers` and `hiddenLrms` are containers; one shared object would alias the panels, which is the exact bug the structure exists to prevent. - patchSettings(patch, panelIndex = null) is the only writer. A null index writes to all panels — the 2a behaviour — and is the single place where 2b's link toggle will decide "one panel or all". The named setters are kept as thin wrappers so no write call site had to change. - usePanelSettings() returns the store merged with one panel's settings, with the panel coming from PanelIndexContext or passed explicitly by ViewerPanel. That shape makes the 17 bare useStore() sites a one-identifier swap with their destructuring untouched. - The data hooks are unchanged, as the plan predicted: they take settings as props from ViewerPanel rather than reading the store. The one real regression risk was setPanelDataset. It rebuilt the panel with makePanel(), which now carries fresh settings — that would have wiped edge width, palettes and layer visibility, none of which a dataset change has ever reset. It now applies a named RESET patch and keeps the rest, with a test pinning both halves. Verified behaviour-preserving by capturing all 28 effective setting values from the running app before the change and diffing after: no differences. Also checked live on two panels holding different datasets (cosmx-mousebrain, merscope-vpt-smallset) that layer toggles, edge width, palette and colour-by all still move in lockstep, with a clean console in both the dev server and the production container. Adds store.settings.test.js (13 tests); 23 frontend tests total. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Second stage of docs/split_screen_phase2.md. The sidebar gains tabs and a link checkbox, so the two panels can be styled independently — two renderings of one dataset side by side (cluster colouring against gene expression, two palettes, two filters) was previously impossible, since every setting was one value. Linked stays the default, which is the pre-2b behaviour exactly. That is the same figure-integrity argument as linkColorScale: two panels that each drifted to their own palette and clamp look comparable and are not. - activePanel and linkSettings live in the store. patchSettings resolves the write target — all panels when linked, the active tab when not — and is the only place that decision is made, so no setter knows about tabs. - getSetting reads the *active* panel. Read-modify-write setters depend on this: unlinked, toggleLrm must build its next Set from the panel it is about to write, not panel 0, or it drops whatever that panel had hidden. - Re-linking adopts the active panel's settings rather than merely resuming propagation. Leaving the divergence in place would put a control labelled "linked" over two visibly different panels and converge them only partially on the next edit. cloneSettings deep-copies the containers; a shallow copy would leave the panels aliasing, so the next unlinked edit would write to both. - CellInfoPanel now reads the clicked panel's settings rather than the active tab's, and AnnotationToolbar is bound to its own panel. Also folds in 2d, which the plan had sequenced last. That ordering was wrong: 2b promises "editing panel 1 only", and a global dataset reset breaks it — an action on panel 2 still wipes panel 1's filter, so 2b alone would ship a control that lies. The panel whose dataset or edge file changed is always reset; the others only while linked, where they share one set of values and a stale filter would 400 on every viewport change. Verified live on two panels over one dataset: viridis and inferno rendering side by side with transcripts on in one panel only, re-link collapsing both to the active tab's palette with no shared containers, and — unlinked — panel 1 keeping its colour-by across a panel 2 dataset change while panel 2 is cleared. Linked, both still reset. Clean console in the dev server and the production container. 11 new tests; 33 frontend tests total. The fixture now resets activePanel and linkSettings between tests — they are what the targeting rule reads, so leaving them set silently redirected a later test's writes to the wrong panel. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…y dataset Two things, found together: 2c is the last functional stage of docs/split_screen_phase2.md, and verifying it surfaced a regression I shipped in ab9d739 that had broken every /edges endpoint. ## The regression `DUCKDB_MEMORY_LIMIT` lost its fixed 8GB default in v0.8.4 so the cap could be sized from memory actually available, and both compose files now pass it through empty. But `edge_reader.py` kept its own connection setup reading `os.getenv("DUCKDB_MEMORY_LIMIT", "8GB")` — and that default applies only when the variable is *absent*. Set-but-empty yields "", so DuckDB got `SET memory_limit=''` and raised ParserException: Parser Error: Memory limit must have a number on every edge query, in the dev server and the container alike. Edges, the tissue graph and the LRM catalogue were dead on all nine datasets. It did not show up earlier because the failure needs the variable *set to empty*, which only the compose files do — a bare `python3 tests/golden_snapshot.py` inherits an unset variable and passes. The real defect is two modules defaulting one environment variable two different ways, so EdgeReader now goes through `duck.connect()`. It also picks up the temp_directory it never had, so a large edge query can spill rather than fail. `duck.py` additionally strips the value, since a whitespace-only setting is truthy and would reach DuckDB unchanged — a hole the new check found. `backend/tests/duckdb_config_check.py` covers unset / empty / blank / explicit against a real edge query. The golden snapshot structurally cannot catch this. ## 2c `pushSettings(from, to, allowed)` copies one panel's settings onto the other, sanitised for the target. The button sits under the link toggle, unlinked only, labelled with its direction. The guard is the substance: a filter naming a column the target lacks 400s on every viewport change and the panel silently stops rendering, so those settings are dropped rather than copied. Column names come from /cells/schema and /edges/schema, which the store never fetches — the component gathers the vocabulary so the store action stays pure and testable. A gene allowlist with no overlap becomes null ("no filter") rather than an empty Set ("show no species"), matching a dataset change; colour clamps reset across different datasets, since [0, 4000] onto data topping out at 70 paints everything one colour. Verified across MERSCOPE → CosMx (zero gene overlap) through the real button: geometry and palette copied, gene allowlist and clamp dropped, source untouched, target still rendering 5000/38996 cells. 46 frontend tests, golden guard 238/238. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Last stage of docs/split_screen_phase2.md, which is now closed out. The hosted manual needed correcting rather than extending. It said "Both panels share all layer settings (visibility, color-by, LRM filter, etc.) but have independent pan and zoom positions" — untrue since Phase 1 gave each panel its own dataset, and wrong through two releases. Someone reading it would not have known the feature they were looking for existed. The Split-Screen section now covers two datasets side by side, the panel tabs and link checkbox, the copy button and what it declines to copy, micron-based zoom matching, and the shared colour scale. Two points get called out as notes because they are surprising rather than discoverable: re-linking re-syncs to the tab you are on, so pick that tab first; and Match works in microns, which is what makes 20% of a 6.5 mm Visium capture area comparable to 20% of a 55 µm seqFISH region rather than fifty-fold apart. Also corrected nearby claims that the same work invalidated: Clear removes that panel's annotations rather than everything, annotations belong to the panel that drew them, and the dataset/image/edge pickers move into the panel headers in split mode. The dataset-picker section gained a note on what a dataset change clears and what it keeps, since that surprised me while writing the tests. Verified rendered in a browser rather than by reading the source: all five headings present, the old claim gone, anchors resolve, HTML balanced. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Fix edges being dead in v0.8.4, and finish split-screen per-panel settings
Writes up #59 before touching code, as with split-screen Phase 2. Three lab directives shape it beyond the issue text, and two of them change current behaviour: - The tissue graph is ground truth: shown or hidden, never subset by a filter. Today one useEdges request feeds both the graph and the edge layer, so filtering edges thins the structural graph too. - Cell and edge filtration are completely independent. Today query_grouped(cell_ids=…) emits `sending IN S AND receiving IN S`, so filtering cells removes edges. - Density is applied last, being a rendering control rather than a selection. This one already holds — the grouped query samples an outer select wrapping the filtered, grouped subquery — and the plan records it as an invariant to preserve rather than work to do. Findings that shaped the design, verified rather than assumed: - Half of #59 already ships. sending_type/receiving_type are populated columns and are already offered in the edge filter dropdown; on mouse_ileum_tiny, sending_type=Fibroblast gives 55 of 223 edges with one sending type and all five receiving types. The blocker is that edgeFilter holds exactly one filter — the composition gap deferred in #45 — not the sending/receiving distinction. - The tissue graph needs its own fetch rather than a passes_filter flag on the shared result, and density-last is why: the two layers want opposite orderings of the same pipeline, and a flag would be read after sampling had already thinned the rows it describes. - Efficiency, which the issue asks about directly: one hash semi-join becomes two, on a query that already performs one. filter_cell_ids() and duck.register_ids() already exist. Sizing measured across the bundled datasets — 169,219 unique edges on cosmx-mousebrain is the largest. Rendering that is not the constraint; the ~15 fields per edge in the payload are, hence a lean projection staged as optional work to be measured first. CLAUDE.md indexes the plan and flags the both-endpoints rule as slated for reversal, keeping the original reasoning since it still explains the code. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ound it I claimed "half of #59 already ships", demonstrating a sending_type filter returning 55 of 223 edges on mouse_ileum_tiny. The mechanism works, but the demonstration was against invented data and the claim was wrong in substance. Tracing the column: - sample_data/make_edges.py writes it, and its own docstring says `cell type (simulated)` — the tidy Endothelial/Immune/Fibroblast labels in every bundled fixture are the generator's invention. - Of the six export scripts in r/, only niches_xenium.R can populate it, and only when the user passes --celltype. niches_cosmx, merscope, seqfish, visium and visium_hd all pass celltype.col = NULL, so on those platforms the column does not exist. So on real output it is absent five times out of six, opt-in on the sixth, and carries one label where the issue asks for any cell metadata column — mouse_ileum_tiny's cells table has cluster, region, pseudotime and seurat_clusters, none of which is in the edge file. The plan now resolves both dropdowns through the cells table uniformly, with no fast path: a fast path for the one case where the column happens to exist would mean two code paths answering the same question differently depending on which ran, and the frozen-at-scoring-time value can disagree with a re-annotated cells table. The testing section no longer leans on the simulated column either. Whether the column should exist at all is raised as an open question rather than answered — it is a NICHESv2-side decision, and whether it is duplication or provenance depends on whether NICHESv2 used the label when scoring, which is not answerable from this repo. docs/data_format.md and CLAUDE.md both presented it as a plain optional field, which reads as "usually there". Both now say what populates it and that it should not be relied on. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ssue graph (#59) Implements docs/edge_filter_independence.md. The pipeline is now explicit, and the order is the contract: all edges in viewport → density filter deterministic, spatially random → EDGESET A → tissue-graph layer (query_structure) → sending filter → receiving filter → edge-table filters → EDGESET B → edge-data layer (query_grouped) Three lab directives drove it, two of which reversed existing behaviour. The tissue graph is ground truth. It gets its own query, and query_structure takes no filter arguments at all — not "they default to none", but no parameter to pass, so nothing can wire one in later. It is a separate request rather than a flag on the shared one because the two layers want opposite things from sampling, and a passes_filter column would be read after the sample had already thinned the rows it describes. The projection is lean (edge + four coordinates), measured 2.2–2.4x smaller than the grouped payload. Cell and edge filtering are independent. cell_filter no longer reaches the edge request at all; sending_ids and receiving_ids constrain the two endpoints separately, so both set gives the intersection and one set leaves the other end free. An edge may now terminate on a cell that is not drawn, which is intended. Both resolve from the *cells table* rather than the edge file's own sending_type/receiving_type — those are absent on five of six platforms as the r/ scripts stand, carry one label where any cell column is wanted, and are frozen at scoring time. Density is one slider over both layers. A second slider was built and removed: the argument for it does not hold, since unfiltered the two layers draw exactly the same number of lines and the graph already has an opacity control, which is the better lever for clutter. The real bug in the first cut was that the two queries sampled independently. Two bernoulli draws at 10% overlap only ~1%, so edge data appeared where the graph beneath it had been sampled away. density_predicate replaces USING SAMPLE with a deterministic hash of the edge id: the same edge gets the same verdict in every query, so the predicate commutes with the filters and B is a subset of A at every density. It is also stable across re-fetches, where bernoulli flickered on each pan. Also: edge_filter becomes edge_filters, a list and-ed together, closing the composition gap deferred in #45. The old cell_filter and edge_filter fields are still accepted so an older frontend against a newer backend keeps working. Verified rather than assumed: - The intersection matches ground truth computed independently from the parquet and the metadata CSV — a full 3x3 sending/receiving matrix, zero mismatches. - backend/tests/edge_pipeline_check.py asserts the graph is unmoved by any filter, that edge data is always a subset of it, and that sampling is stable: 9 datasets across 6 platforms, at densities 1.0 / 0.5 / 0.1. - Visually on Xenium: applying a sending filter leaves the grey graph pixel-for- pixel identical while the coloured edge layer thins; at 15% density every coloured edge sits on a grey line. - Split screen with two datasets and per-panel endpoint filters, clean console. - Golden snapshot 249 probes, frontend 52 tests, duckdb config check. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Independent sending/receiving edge filters, and unfilter the tissue graph
Click a cell, press "show local neighbourhood", and see what it is connected to in the tissue graph — drawn on the canvas and summarised in the info panel: neighbour and edge counts, the enclosing radius in µm, composition by any cell metadata column, and the strongest LRMs across its incident edges. The design note is docs/neighborhood_summary.md. Computed server-side, and that is the point rather than an implementation detail. The obvious version filters the `edges` array the frontend already holds; it would appear to work and be wrong twice over, since that array is density-sampled (nine of ten neighbours missing at the default) and viewport-bounded (a neighbour just off-screen does not exist, and the answer changes as you pan). A neighbourhood is a property of the tissue, not of the current view, so the query ignores density, the viewport, the endpoint filters and the LRM checklist. Cost was measured first and is not a reason to avoid it: 15 ms on the 3.8M-row CosMx file, no index, no cache. Two marks are drawn because the radius alone would mislead. Connectivity is anisotropic — a cell at a tissue boundary has neighbours on one side only — so a disc around it encloses many cells it is not connected to. The highlighted cells are the honest answer; the circle is the spatial scale the issue asked for. Composition resolves against the cells table rather than the edge file's sending_type, for the reasons recorded in v0.8.6: that column is absent on five of six platforms, carries one label, and is frozen at scoring time. It keys on the frame's cell_id *column* — the frame carries a plain RangeIndex, and my first version indexed by position, which matched nothing and reported every neighbour as missing. Also fixed, found while testing this: CellInfoPanel rendered stale `detail` for one frame after the selection cleared, because `detail` is state and outlives `selectedCell`. The new section dereferenced it and took the whole app to the error boundary on a dataset change. Guarded on both. Verified: - Counts and radius match ground truth computed independently from the parquet, on all 9 bundled datasets across 6 platforms; 3–125 ms each. - End to end through the real button on Xenium: 11 neighbours, 18 edges, 12.8 µm, the highlight landing exactly on the tissue graph vertices joined to the clicked cell — and visibly enclosing unconnected cells the circle would have claimed. - 56 frontend tests, including that the highlight carries its panel and is dropped on selection change or a dataset change in its own panel. - Golden snapshot 249 probes, edge pipeline check, DuckDB config check. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Local neighbourhood highlighting and summaries
…sion in README
Three tidy-ups, plus a real hole found in my own test while doing them.
Empty metadata columns. `fov` and `transcript_count` are entirely null on the
bundled MERSCOPE dataset, but color-values reported them as a continuous 0–0
range. The UI then offered a range slider that did nothing, a legend with no
span, and a filter that correctly matched no cells while looking broken. They now
come back with `empty: True` and the filter section says "no values in this
column". `type` is still set so nothing switching on categorical-vs-continuous
needs a third case, and the cross-panel merge calls a column empty only when it
is empty in every panel.
The pan re-render storm. `usePanelSettings` subscribed to the whole store, so
every sidebar section re-rendered on each OpenSeadragon viewport-change event.
It now ignores `viewports` and `viewportActual`, which are the only continuously
rewritten keys and which no consumer of that hook reads — ViewerPanel takes
`viewports[panelIndex]` through its own selector and Match zoom reads
`viewportActual` via getState(). Measured over one simulated pan of 120 writes:
121 re-renders before, 0 after, with ordinary settings changes still delivered.
Narrowing the rest means a selector per section and is not worth it without
component tests.
Version in the README, and the bump list recorded in CLAUDE.md: three files move
together — package.json, main.py, README — while every other v0.x.y in the tree
is a historical reference that must not be swept along. Audited: the two
authoritative files agreed, and all other mentions were correctly historical.
The hole. tests/edge_pipeline_check.py called
`color_values("metadata", None, field)`, but the signature is
`(mode, field, genes, categorical)` — so the field went in as `genes`, the call
returned the empty result, `spec_for` returned None, and the filtered-subset
assertion was silently skipped on every dataset. The check printed OK while not
testing its own core property. Fixed, and it now says so out loud when no filter
resolves rather than passing over it. With the assertion actually running the
property still holds, on all 9 datasets and 3 densities — so the earlier
hand-verification was right, but the automated claim was weaker than stated.
Golden snapshot 249 probes, edge pipeline check, DuckDB config check, 56
frontend tests.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Report empty metadata columns, stop the pan re-render storm, version in the README
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.