Perturbation analysis - #11
Draft
aarmey wants to merge 8 commits into
Draft
Conversation
Adds analysis/ (uv project, py>=3.12) wiring fastpidc + vsparse together to load /opt/IVCSC/hESC.h5ad, since fastpidc's own requires-python floor is lower than vsparse's. Verifies the Python FastPIDC API against both plan documents' open questions (raw PUC access and calibration-disable both already supported; the interventional plan's pid_triple primitive does not exist yet and is the first real blocker for its Stage 6). Runs the interventional plan's Checkpoint 0 (reciprocal-pair count) against the real data: 150 perturbation targets, not the assumed ~300, but the gate passes comfortably (11,175 candidate pairs vs. a ~2,000 floor) after cell-count and phenotype-detectability screens. Findings and caveats logged in LOG.md; STATE.md tracks per-checkpoint status against both plans going forward. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Implements the interventional plan's actual Stage 4 (not Checkpoint 0's mean-shift proxy): per-gene energy distance between each of the 150 perturbation targets and NTC control, via a vectorized O((m+n)log(m+n))-per-gene merge-rank identity (verified against scipy.stats.energy_distance), parallelized across targets with chunked multiprocessing. Significance calibrated against NTC/NTC splits at representative sample sizes (not a hardcoded parametric null, per the plan), BH-FDR across all (target, gene) pairs. Two real infrastructure problems hit and fixed along the way, both logged in detail since they'll recur at Stage 3's much larger PUC computation: an OOM from over-parallelizing at full gene width (fixed by chunking genes + downcasting sort-order dtype + fewer workers), and an I/O bottleneck from scattered column reads against a cell-major array (fixed by precomputing a gene-major contiguous layout). Results: companion-plan Checkpoint D0 passes literally (measured effective out-degree has real spread, 1,003-11,559 of 11,942 genes, no target near zero) but the top of that range is implausibly large for a single gene's direct regulatory targets -- almost certainly hESC cell-state/cell-cycle confounding, which the interventional plan's own Checkpoint 0 flagged in advance as a risk. Interventional Checkpoint 3's pluripotency-edge check passes cleanly (SOX2 -> POU5F1 and SOX2 -> NANOG both q~0). STATE.md updated with both results and the cell-cycle conditioning this surfaces as the next concrete blocker before the companion plan's degree-prior work can trust k_out.csv. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014b66DKG4wgzFCcJYmPLsWA
Interventional plan's Stage 2 needs a discretizer that gives exact zeros their own bin (dropout-dominated single-cell data), then equal-frequency bins the nonzero remainder. Wires in cleanly as a new get_bin_ids mode, usable directly through Node.from_raw_values without needing to bypass the public discretizer API. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014b66DKG4wgzFCcJYmPLsWA
Implements the pid_triple primitive the interventional plan calls for before Stage 6 (H2): given (source1, source2, target), decomposes MI(source1,source2 ; target) into redundancy/unique1/unique2/synergy via the Williams & Beer I_min lattice, built from the existing get_mi_and_si / apply_redundancy_formula pieces plus a new joint-binned combined_node helper for the synergy term. Verified against known-analytic cases (XOR -> pure synergy, identical sources -> pure redundancy, independent -> all zero) rather than just structural checks. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014b66DKG4wgzFCcJYmPLsWA
Installs cupy-cuda12x in analysis/ and validates fastpidc's CUDA PUC backend against the CPU backend (exact match) plus a timing/memory profile at real cell count, extrapolating to ~14 GB / ~70 min for the full 11,942-gene Stage-3 run on the RTX 4090. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014b66DKG4wgzFCcJYmPLsWA
Adds cell_cycle.py (Tirosh/Seurat S/G2M module scoring + vectorized per-gene OLS regression of those scores out of the filtered expression matrix) and rerun_stage4_residualized.py (reruns the Stage-4 effect matrix, NTC-split null calibration, and BH-FDR against the residualized data), parameterizing effect_matrix.py/null_calibration.py /calibrate_effects.py to support an alternate input array and output paths without duplicating them. Result: cell-cycle regression does not fix Checkpoint D0's implausible out-degree inflation -- residualized k^out_g went up for 135/150 targets (median +72.5), and SOX2's own out-degree got worse (~75% -> ~80%). Only ~1.5% of per-gene variance was explained by the two cell-cycle scores, consistent with this null result. Full numbers and next-step candidates in LOG.md; D0 is a re-opened blocker, not closed. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014b66DKG4wgzFCcJYmPLsWA
…nfound Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014b66DKG4wgzFCcJYmPLsWA
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request introduces a new document outlining a companion plan for degree-aware recalibration of PIDC, focusing on improvements to the calibration layer rather than the core PUC score. It proposes several new calibration strategies to address topological assumptions in PIDC, particularly regarding gene degree distributions, and details a careful experimental and evaluation framework. Additionally, a new Python CI workflow is added to automate testing for the Python codebase.
Key changes:
1. Documentation: Degree-aware PIDC recalibration plan
Degree-aware recalibration of PIDC—a companion plan.md, which:2. CI/CD: Python workflow automation
.github/workflows/python-CI.ymlto enable automated CI for the Python codebase, including:These changes collectively improve both the methodological planning for PIDC recalibration and the development workflow for Python code.