Add run_nmf_batch(), opt-in parallel NMF, and g:Profiler hardening - #9
Merged
Conversation
- run_nmf_batch(): drive NMFprofileR() over a named list of cohorts, writing each to its own subdirectory and returning one consolidated per-factor summary tagged by cohort Run_ID. Resilient (on_error = "continue"), resumable (skip_existing reloads finished cohorts from their bundle/summary), and writes a Batch_Consolidated_Summary.tsv. - Opt-in parallel NMF via nmf_parallel (+ nmf_cores), default off. NMF selects worker packages from the attached search path, so nmf_fit() attaches NMF (via attachNamespace, restored afterwards) for the duration of a parallel fit -- otherwise workers fail with "none of the packages are loaded" when NMF is called from a package namespace. Verified that for a fixed seed the parallel result is identical to the sequential path (basis, coef, objective). - g:Profiler hardening in the enrichment path: an oversized-query guard (max_query_size), automatic retry with exponential backoff on transient failures, and an optional on-disk cache (enrichment_cache) keyed by a stable query hash so re-runs and batches skip the network. Exposed on NMFprofileR() and nmf_enrichment(). Adds a generic "Batch analysis across many cohorts" vignette and network-free tests (test-batch, test-enrichment-hardening, test-parallel); repoints the stale determinism parallel test at the working nmf_fit() path. digest added to Suggests (optional cache hashing). R CMD check passes clean; version 0.4.0.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Batch-and-scale features. Everything is additive; the sequential default is unchanged.
Changes
run_nmf_batch()— runsNMFprofileR()over a named list of cohorts, each to its own<name>_Results/subdirectory, and returns one consolidated per-factor summary tagged by cohortRun_ID. It is resilient (on_error = "continue"records failures and carries on), resumable (skip_existingreloads finished cohorts from their.rdsbundle or summary), and writesBatch_Consolidated_Summary.tsv.nmf_parallel = TRUE(with optionalnmf_cores) spreads the consensus runs across a local cluster. NMF selects its worker packages from the attached search path, sonmf_fit()attaches NMF for the duration of a parallel fit (viaattachNamespace(), restored afterwards); otherwise workers hit the old "none of the packages are loaded" failure when NMF is called from a package namespace. Verified that for a fixed seed the parallel result is byte-identical to the sequential path (basis, coef, objective). Default remains sequential.max_query_size), automatic retry with exponential backoff on transient failures, and an optional on-disk result cache (enrichment_cache) keyed by a stable query hash so re-runs and batches skip the network. Exposed onNMFprofileR()andnmf_enrichment().Testing
test-batch(consolidation, Run_ID tagging, skip_existing, on_error, arg guards, via a mocked fit),test-enrichment-hardening(hash key, retry, oversized guard, cache hit), andtest-parallel(parallel reproduces sequential; asserts it did not fall back).nmf_fit()path (it previously always skipped).R CMD checkclean (0 errors/warnings/notes), both vignettes build. New "Batch analysis across many cohorts" vignette. Version 0.4.0.