Skip to content

Add run_nmf_batch(), opt-in parallel NMF, and g:Profiler hardening - #9

Merged
MilesGithub merged 1 commit into
mainfrom
feature/v0.4.0-batch
Jul 9, 2026
Merged

Add run_nmf_batch(), opt-in parallel NMF, and g:Profiler hardening#9
MilesGithub merged 1 commit into
mainfrom
feature/v0.4.0-batch

Conversation

@MilesGithub

Copy link
Copy Markdown
Owner

Batch-and-scale features. Everything is additive; the sequential default is unchanged.

Changes

  • run_nmf_batch() — runs NMFprofileR() over a named list of cohorts, each to its own <name>_Results/ subdirectory, and returns one consolidated per-factor summary tagged by cohort Run_ID. It is resilient (on_error = "continue" records failures and carries on), resumable (skip_existing reloads finished cohorts from their .rds bundle or summary), and writes Batch_Consolidated_Summary.tsv.
  • Opt-in parallel NMFnmf_parallel = TRUE (with optional nmf_cores) spreads the consensus runs across a local cluster. NMF selects its worker packages from the attached search path, so nmf_fit() attaches NMF for the duration of a parallel fit (via attachNamespace(), restored afterwards); otherwise workers hit the old "none of the packages are loaded" failure when NMF is called from a package namespace. Verified that for a fixed seed the parallel result is byte-identical to the sequential path (basis, coef, objective). Default remains sequential.
  • g:Profiler hardening — an oversized-query guard (max_query_size), automatic retry with exponential backoff on transient failures, and an optional on-disk result cache (enrichment_cache) keyed by a stable query hash so re-runs and batches skip the network. Exposed on NMFprofileR() and nmf_enrichment().

Testing

  • New network-free tests: test-batch (consolidation, Run_ID tagging, skip_existing, on_error, arg guards, via a mocked fit), test-enrichment-hardening (hash key, retry, oversized guard, cache hit), and test-parallel (parallel reproduces sequential; asserts it did not fall back).
  • Repointed the stale determinism parallel test at the working nmf_fit() path (it previously always skipped).
  • Full suite green on R 4.6.0 (0 failures/0 warnings). R CMD check clean (0 errors/warnings/notes), both vignettes build. New "Batch analysis across many cohorts" vignette. Version 0.4.0.

- run_nmf_batch(): drive NMFprofileR() over a named list of cohorts, writing
  each to its own subdirectory and returning one consolidated per-factor summary
  tagged by cohort Run_ID. Resilient (on_error = "continue"), resumable
  (skip_existing reloads finished cohorts from their bundle/summary), and writes
  a Batch_Consolidated_Summary.tsv.
- Opt-in parallel NMF via nmf_parallel (+ nmf_cores), default off. NMF selects
  worker packages from the attached search path, so nmf_fit() attaches NMF (via
  attachNamespace, restored afterwards) for the duration of a parallel fit --
  otherwise workers fail with "none of the packages are loaded" when NMF is
  called from a package namespace. Verified that for a fixed seed the parallel
  result is identical to the sequential path (basis, coef, objective).
- g:Profiler hardening in the enrichment path: an oversized-query guard
  (max_query_size), automatic retry with exponential backoff on transient
  failures, and an optional on-disk cache (enrichment_cache) keyed by a stable
  query hash so re-runs and batches skip the network. Exposed on NMFprofileR()
  and nmf_enrichment().

Adds a generic "Batch analysis across many cohorts" vignette and network-free
tests (test-batch, test-enrichment-hardening, test-parallel); repoints the stale
determinism parallel test at the working nmf_fit() path. digest added to
Suggests (optional cache hashing). R CMD check passes clean; version 0.4.0.
@MilesGithub
MilesGithub merged commit 0c9f558 into main Jul 9, 2026
4 checks passed
@MilesGithub
MilesGithub deleted the feature/v0.4.0-batch branch July 9, 2026 04:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant