Conversation
Adds the platform with the largest published corpus in spatial
transcriptomics, which was the one obvious gap: a Space Ranger `outs/`
folder previously gave "Cannot detect spatial platform".
Spots render as 55 um circle polygons on their hex lattice, with
expression, both colour-value modes, the metadata filter, and NICHESv2
edges. Nothing renders cells() centroids, so synthesising polygons is
what makes fill, colour-by, picking and region selection work — the same
reasoning that made Visium HD bins squares. Circles rather than squares
here because a bin really is square and a spot really is round.
Shared base — readers/spaceranger.py
Classic Visium and Visium HD agree on scalefactors, tissue_positions,
the feature-matrix h5, the cells table, gene-set colouring and the
sample-then-slice boundary logic. They differ in layout, unit shape and
where the pixel size comes from. Only the differences stay in the two
readers.
Fixes an unapplied tissue_hires_scalef
visium_hd_reader computed the factor and reported it in info() but
never applied it to coordinates, while its own docstring said it did.
Space Ranger reports positions in FULL-RESOLUTION pixels and ships only
the hires PNG, which is what the tile pyramid is built from — so
everything landed off the image by 1/scalef on any real dataset. It was
invisible because visium_hd_tiny has the factor set to exactly 1.0.
The factor is now applied in the shared positions loader and folded
into pixel_size, which matters beyond distance labels: EdgeReader
divides edges.parquet micron coordinates by pixel_size, so a
fullres-based value would scatter the connectivity layer off the
tissue. Behaviour on the bundled HD fixture is byte-identical (scalef
is 1.0 there), which is precisely why the fixture could not catch it.
Three classic-Visium traps, all handled
- No microns_per_pixel key. 10x does not record image pixel size for
classic Visium, so it is derived from the one guaranteed constant
(55 um / spot_diameter_fullres) and flagged pixel_size_derived in
info(), since 10x calls spot diameters estimates.
- tissue_positions_list.csv (Space Ranger < 2.0) has no header row;
default header inference eats the first spot and mislabels the rest.
- tissue_hires_scalef really is ~0.08 here, not ~1.0.
Fixture that can actually fail
sample_data/make_visium.py generates a non-identity scalef of 0.08 on
purpose. With the multiply, 252/252 spots land on the image; without
it, 0/252. The derived pixel size recovers 0.25 um/px exactly, and
niches_visium.R's independent lattice check reports a median
nearest-neighbour distance of 100.0 um — the Visium pitch.
Also
- r/niches_visium.R, run end to end: 252 spots, 1642 edges, 30 LRMs,
avg 6.5 neighbours per spot, which is the hex lattice's 6.
- The gene-set hint no longer points at the Transcript Species section
on platforms that hide it, and the colour-by mode is unit-labelled.
- README platform table corrected: MERSCOPE and CosMx boundaries have
been implemented since v0.7.1 but were still marked absent.
Guard: 217 probes across 8 datasets, verified to reproduce without the
gitignored metrics_summary.csv.
Not yet verified against a real Space Ranger outs/ tree.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Testing the reader against real Space Ranger output (V1_Mouse_Kidney and V1_Adult_Mouse_Brain) found the previous commit's `pixel_size` derivation wrong by 18%, and turned up an unrelated autocrine bug on the way. pixel_size now comes from the lattice pitch, not the spot diameter Deriving it as `55 um / spot_diameter_fullres` looked obvious and was wrong. Measured independently on both datasets: the in-row lattice pitch is exactly 138.00 fullres px while spot_diameter_fullres is 89.45 — a ratio of 0.648. That field is Space Ranger's DETECTED spot footprint (~64.8 um), not the 55 um nominal capture spot; 10x's own docs describe classic Visium spot diameters as "approximately 60-70 um" and warn they are estimates. It now derives from the 100 um centre-to-centre pitch, measured off the positions table. That reconstructs 10x's specified 6.5 x 6.5 mm capture area where the old route gave 5.4 x 5.7 mm, and it is better conditioned — a median over thousands of positions in a rigid array template rather than one estimate of a fuzzy edge. `info()` reports `pixel_size_source` so which route was taken is visible. This matters beyond distance labels: EdgeReader divides edges.parquet micron coordinates by pixel_size, so the whole connectivity layer was placed 18% off. The fixture could not have caught it, and now can make_visium.py set spot_diameter_fullres to exactly 55/mpp, making the fixture a tautology — it confirmed whatever derivation the reader used. It now emits the real 0.648 ratio, so the bad derivation yields an 85 um nearest-neighbour spacing against the 100 um truth, the same failure signature the real datasets show. Verified against real data 1438/1438 spots land on the hires image (0/1438 without the scale multiply). Registration: 98.1% of in_tissue spots sit on stained tissue and 99.5% of out-of-tissue spots on bare slide, measured as mean inverse intensity under each spot polygon. NICHESv2 end to end: 9,666 edges, 2,479 LRMs, 3.2M rows, top mechanisms Apoe|Lrp2 and Cst3|Lrp2 — megalin-mediated proximal tubule uptake, the expected kidney biology. Autocrine signalling Autocrine edges were excluded from the lrm_set colour computation, so the rings stayed default orange whatever the colour control said — the one edge type that ignored "colour by LRM set". They also skipped the visible_lrm_count test, so hiding every mechanism removed the lines but left a ring on every cell. Both fixed. Sharing the colour scale is safe: autocrine score medians run 1.00-1.33x the directed medians across the bundled datasets. Verified on Visium, Xenium, seqFISH and Visium HD. User manual (docs/index.html, published to GitHub Pages) Listed three platforms; now lists six with their sentinels and units. Adds sections for metadata filtering and the treat-as-categorical toggle, rewrites the autocrine section, de-Xenium-ifies the intro, and is now linked from the README, which never pointed at it. Also: r/niches_visium.R uses the same corrected derivation with an independent lattice-pitch check; docs/public_datasets.md records the verification downloads; README platform table corrected (MERSCOPE and CosMx have had boundaries since v0.7.1 but were still marked absent). v0.8.2. Guard: 217 probes across 8 datasets. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Classic Visium (10x v1/v2) support — v0.8.2 + autocrine fix
All nine demo datasets now carry edges.parquet and an edge-metadata/
folder, so the connectivity layer and the annotation workflow have
something to show whichever platform you load. Previously six of nine had
edges and exactly one had annotations.
Two new NICHESv2 scripts complete the set — one per platform
r/niches_merscope.R The easy case for coordinates: cell_metadata.csv is
already microns. Two traps instead: EntityID is a 19-digit integer
that R silently mangles if read as numeric, and small targeted panels
usually carry no complete LR pair.
r/niches_cosmx.R Cell identity is the (fov, cell_ID) pair, and
coordinates are slide-frame PIXELS that must be shifted to the
reader's origin (min - 50) and THEN scaled to microns. Skip the shift
and edges land tens of thousands of pixels away; skip the scale and
they are off by ~8x.
Results: CosMx 169,219 edges across 228 LRMs, top mechanisms App|Aplp2 and
Prnp|Ncam1 — neural adhesion, as expected for mouse brain. MERSCOPE 14,503
edges but only 2 LRMs, because that 130-gene human panel contains just
THBS1->LRP1 and CEMIP->LRP1.
Why targeted panels often cannot be scored
seqfish_instrument2 has 12 genes and ZERO complete ligand-receptor
pairs, so no real run is possible. check_lr_coverage() catches that in a
second rather than ten minutes. It gets synthetic edges from
make_edges.py instead, which is now platform-agnostic: it reads
centroids through the reader and converts image pixels back to microns,
so it works on any of the six platforms rather than only Xenium.
Fixes a Visium HD bin mismatch that shipped in v0.7.0
niches_visium_hd.R defaulted to square_016um while visium_hd_reader
served square_008um. Barcodes are bin-size specific, so the fixture had
ZERO ids in common between its edges and its bins. The edges still drew,
because they carry their own coordinates, but nothing joined: clicking a
bin found no edge, the metadata filter dropped every edge, and the
tissue graph floated free of the bins beneath it.
_bin_from_edges() now reads one barcode from the edge file and serves
the matching bin, falling back to Space Ranger's default when a dataset
has no edges. That is also right on the merits — 8 um bins are usually
too sparse to score (233,531 edges of which 83 scored, against 66,001
edges across 69 LRMs at 16 um), and whoever ran NICHESv2 already made
that call.
Verified: all nine datasets now have 100% barcode overlap between edges
and units, with a median endpoint-to-centroid offset of 0.000 px.
sample_data/make_edge_metadata.py
Derives interaction_class, score_tier and score_percentile from the edge
file itself, plus one clearly-named invented demo_review_flag so the
categorical filter has something to bite on. score_percentile is
continuous on purpose — every other column is categorical, and the
gradient path needs exercising too.
Each folder gets a README naming which columns are derived and which is
fabricated. A CSV of plausible-looking annotations sitting next to real
kidney or brain data is a hazard; someone will eventually read
"score_tier: high" as a result. None of it is analysis output.
Also: the four LayerPanel sections that fetch per-dataset endpoints now
wait for `dataset` to resolve. They were firing five 404s against
/spatial/null/... on every cold load, which is noise that buries real
errors.
v0.8.3. Guard: 238 probes across 8 datasets.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
nginx served index.html with only Last-Modified/ETag and no Cache-Control,
so browsers fell back to heuristic caching and could reuse it for hours
without revalidating.
index.html is the only file that names the current bundle. Serving a stale
one after `docker compose up --build` leaves the tab running the previous
JS: the app looks healthy, reports the new version from the backend — that
comes over the API, not the bundle — and silently lacks every frontend fix
in the update.
That is what "I rebuilt and my change isn't there" looks like from the
outside, and it is what made the autocrine colour fix appear absent on
:3000 when the container was in fact serving the corrected bundle all
along.
index.html -> Cache-Control: no-cache
Revalidate before use. Not "do not store" — the usual
response is a 304, so the cost is one round trip on the
smallest file we serve.
/assets/ -> Cache-Control: public, max-age=31536000, immutable
Vite fingerprints these by content, so the name changes
whenever the bytes do. Caching them forever is what makes
revalidating the HTML cheap.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
fine tuning, operational, 9 in-repo datasets with edge data
Split screen showed two views of one dataset — a single global `dataset`,
a single dataset picker, a single `imageSize`. Each panel can now show a
different dataset, including a different platform.
Per-panel state
Everything bound to *which dataset a panel shows* moves into
panels[panelIndex]: dataset, activeImage, imageSize,
platformCapabilities, pixelSize, edgeFile, lrmCatalogue, allGenes, the
colour ranges, and the shown/total stats. Each differs between datasets,
so none of them could stay global.
Style and choice settings stay shared on purpose. One sidebar driving
both panels is what makes a comparison comparable, and it avoids the
focus model — "which panel is the sidebar editing?" — that is the
genuinely hard part of full independence. Phase 2 splits these behind
sidebar tabs.
Reconciling one sidebar across two datasets — hooks/usePanels.js
The rule is union, then degrade per panel: offer a control when either
panel can use it, and let the panel that cannot render nothing.
Intersecting would hide controls that work fine on one side, which is
worse when the whole point is comparing unlike things. Gene panels,
metadata columns, edge columns and LRM catalogues are all unioned;
unit_label becomes the neutral "unit" when panels disagree, because
calling a Visium spot a "cell" would be worse than saying nothing.
Match zoom now matches PHYSICAL scale
It divided both viewports by the local image width — "the same
proportion of the picture". Identical behaviour when both panels showed
one dataset, and meaningless across two: 20% of a 6.5 mm Visium capture
area and 20% of a 55 µm seqFISH ROI differ by 55x. It now converts
through each panel's own pixel_size, so the same number of microns spans
the same screen width, as a scalebar would.
Verified Visium <-> seqFISH: 2996.9 µm and 54.9 µm across, matched to
54.9 µm both — at which point one 55 µm Visium spot fills the same field
as the entire seqFISH ROI.
Shared colour scale, on by default
Two viridis panels that each auto-ranged to their own data look
comparable and are not: one's yellow might be 40 counts and the other's
4,000. Linked, both clamp to the union of the two ranges so the single
legend is true for everything on screen. Verified [131,547] and [66,447]
unioning to [66,547]. Unlockable, with the panel saying plainly that
colours are then not comparable. An explicit slider clamp always wins.
Also
- Dataset / image / edge-source pickers move into each panel's header in
split mode; an image name or edge file only means something relative to
one dataset. Single-panel mode is unchanged, pickers still in the
sidebar.
- Selection carries its panel index, so CellInfoPanel, EdgeInfoPanel and
region export resolve against the dataset that was actually clicked.
EdgeInfoPanel used to be pinned to panel 0.
- DatasetPicker is its own module, breaking the cycle it would otherwise
close: Viewer -> LayerPanel -> App -> Viewer.
Known trade: changing either panel's dataset resets the shared
column-named settings (filters, colour-by field, gene allowlist, hidden
LRMs), because a filter naming a column the new dataset lacks 400s on
every viewport change. That clears the other panel's filter too, and goes
away in Phase 2.
v0.9.0. Guard unaffected (238 probes, reader-level).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two independent fixes surfaced while bringing up a full-size Spatial
Genomics seqFISH dataset (91250x44667 DAPI, ~4 gigapixels).
Tile pyramid — large JPEG2000 OME-TIFFs (pyramid.py, tiles.py, compose):
libvips 8.16 ships OpenJPEG but still cannot decode this file's tiles
(TIFF compression tag 34712) because it goes through libtiff, which has
no codec for it; only tifffile+imagecodecs can. The tifffile fallback
only builds levels up to MAX_TIFFFILE_DIM yet used to advertise the
native size in the DZI, so the viewer requested deeper tiles that 404'd
and rendered black on deep zoom.
- Keep the descriptor at NATIVE size so overlay coordinates (cells,
edges, transcripts, emitted as um / pixel_size) stay aligned with the
image; a capped descriptor had misplaced every overlay by the scale
factor.
- Synthesise the un-built top levels on demand by upscaling the deepest
built level (get_or_synth_tile), so deep zoom is smooth-blurry instead
of black, with no large pre-build and bounded memory.
- Log the previously-silent pyvips -> tifffile fallback so codec
failures are visible.
- Expose MAX_TIFFFILE_DIM as an env knob (default 16384) to trade RAM
for a sharper deepest real level.
seqFISH per-cell counts (seqfish_reader.py):
Derive transcript_counts / total_counts from the CellxGene row sum so
the cell-info panel shows real numbers instead of blanks (seqFISH v2 has
no transcript->cell assignment; the row sum is the authoritative total).
Also makes the counts available as a color-by / filter field.
Existing datasets (Xenium, small images, the pyvips path) are unaffected:
synthesis only engages above the deepest built level.
fix: support large JPEG2000 OME-TIFFs and add seqFISH per-cell counts
…nly it The gene picker renders every checkbox as ticked when `selectedGenes` is null (`checked = selectedGenes === null || selectedGenes.has(gene)`), but `toggleSelectedGene` treated a click in that state as "start an allowlist with just this gene". So the first click on a ticked box did the opposite of what the box said: it selected that one gene rather than excluding it. On a small demo panel that is merely surprising. On real data it does not look like a filter bug at all — on a 480-gene Xenium run one click took the transcript layer from 200,000 rendered dots to 363 over the same viewport, which presents as "transcripts are broken", with nothing in the UI obviously wrong beyond a "1 / 480 genes selected" line that reads as intentional. A click on a ticked box now excludes that gene, building the allowlist from the union of the visible panels' gene lists. Toggling the last gene back on collapses to null so "no filter" stays single-valued rather than becoming an explicit 480-name allowlist on every request. Behaviour predates v0.3.0 — `git tag --contains c52dbc0` puts it in v0.2.0. Verified on the reporting dataset (Rat-PPLR-Run1, 480 genes, 132M transcripts): clicking Abca3 now yields 479/480 with Abca3 excluded and 200,000 transcripts still rendering; clicking it again returns to no filter. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Both version strings claimed 0.9.0, but no 0.9.0 release exists anywhere. The v0.9.0 tag is local-only: `git ls-remote --tags` shows origin carries no tags at all, and upstream's newest is v.0.8.3. So the browser tab, the sidebar badge and GET /health all advertised a version nobody could obtain, and the badge's frontend-vs-backend mismatch check compared two equally wrong strings. The frontend reads frontend/package.json at build time via vite's __APP_VERSION__; the backend has its own constant. They have to be edited together or the badge turns red. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… index build Reported as "toggle transcripts on and nothing shows up" on a real Xenium run (132,789,299 transcripts, 1736 MB). The layer was empty because the request never returned data, for two independent reasons that compounded. 1. The backend was being killed. DUCKDB_MEMORY_LIMIT defaulted to 8GB in both compose files, but the Docker Desktop VM has 7.8 GB, so the spatial-index build's global ORDER BY over 132M rows ran the VM out of memory and the kernel killed uvicorn. This is invisible in the obvious places: the container limit is 12 GB so `OOMKilled` stays false and the exit code is 0 — only RestartCount climbing (5, here) shows it. It also cannot be caught by the build's own `except Exception`, since SIGKILL is not an exception. The limit is now derived from 60% of the minimum of the cgroup limit and physical RAM, and connect() sets a temp_directory under CACHE_DIR so DuckDB can spill — an in-memory DuckDB cannot go out-of-core without one, which is what made a sort this size impossible rather than merely slow. 2. The build blocked the request. Even surviving, it takes ~94 s against nginx's 120 s proxy_read_timeout. It now runs in a daemon thread while queries continue against the unsorted file, which answers them correctly, just slower. Either failure reached the frontend as a non-ok response, and the hooks return [] on non-ok — so the canvas showed nothing, exactly as if the platform served no transcripts. Small datasets are under the 64 MB cache threshold, which is why this never appeared on the bundled fixtures. Verified on the reporting dataset: 200,000 transcripts render across all sections, index builds in 94 s with 0 restarts, viewport query 2.3 s -> 0.37 s once adopted. Golden guard: 238 probes across 8 datasets, all matching. The baseline also absorbs the two seqFISH per-cell count columns added in e668eda, which was never re-recorded; verified by stashing this change that the drift is unrelated to it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A .DS_Store was committed to the repo root in e668eda alongside unrelated tiling work. It is macOS Finder view metadata — per-machine, meaningless to anyone else, and rewritten whenever the folder is opened in Finder, so it shows up as spurious churn in every status and diff. `git rm --cached` untracks it without deleting it, so local Finder view settings survive. The pattern is left unanchored (`.DS_Store`, not `/.DS_Store`) so it also covers the copies already sitting in r/, docs/ and sample_data/, plus any directory browsed in future. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…rter Review follow-up to f4b2f2b. That fix made "uncheck one gene" produce an allowlist of everything else, which is correct behaviour but put the whole panel into the URL of a GET. Measured against this stack: 479 of 480 genes is a 7,780-byte URL, ~220 bytes below nginx's 8 KB request-line limit, and 600 genes returns 414. So it worked on the reporting dataset by luck, and a Xenium Prime 5K panel would have failed on the first click. The complement is short exactly when the allowlist is long, so the client now sends whichever is smaller. `exclude_genes` is resolved back into an allowlist in the router against reader.gene_list(), leaving the reader interface — six implementations — with a single gene argument. Unchecking one gene now sends 157 bytes instead of 7,780. nginx's large_client_header_buffers goes to 64k to cover the worst case the rule can still produce, half a panel. Verified equivalent, not just shorter: exclude_genes=Abca3 and the explicit 479-gene allowlist both return total=34884 on the same viewport. Also in this pass: - An empty selectedGenes Set now short-circuits the fetch. It means "show no species", but omitting the parameter means "no filter", so the request returned the full 200K-row cap for Viewer's client-side filter to throw away: nothing drew, which looked correct, while ~20 MB moved per pan and the layer badge showed the unfiltered total against an empty canvas. - spatial_cache's _resolved memo is keyed on the sort columns, not just the path. The manifest check refuses a cache sorted on a different column pair, but the memo sat in front of it and would have returned that exact file on every later call. Not reachable today — each source is queried with one fixed pair — but it silently defeated the check written to prevent it. - Tidied a comment in _start_background_build that had been split mid-sentence across two lines. Golden guard: 238 probes across 8 datasets, all matching. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
useUnionList joins dataset names on a NUL to build its effect key, which is sound (a NUL cannot appear in a directory name, so two datasets can never collide into one key), but the byte was written literally into the source rather than as an escape. Git classifies any file containing a NUL as binary, so the file showed as "Bin 0 -> 3418 bytes" in every diff: no line-level review, no blame, no meaningful conflict resolution. The escape form is the identical value at runtime and keeps the file text. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Split-screen dataset comparison, and fix transcripts never rendering on large datasets; v0.8.4
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.