Add lazy DATA helpers (only_text docs, as_memmap, read_events) - #39
Open
jahnzh wants to merge 2 commits into
Open
Add lazy DATA helpers (only_text docs, as_memmap, read_events)#39jahnzh wants to merge 2 commits into
jahnzh wants to merge 2 commits into
Conversation
Document only_text as the supported metadata probe, expose DATA byte range / numpy dtype helpers, and add as_memmap() plus read_events() for uniform F/D layouts (with clear errors for ASCII and variable bit-widths). Co-authored-by: Cursor <cursoragent@cursor.com>
Add preprocess=True default to read_events() so lazy subsampling returns the same gain/log/timestep-corrected values as as_array(). Extract shared _apply_preprocessing() and warn on ignore_offset_error in the lazy mmap path. Document raw access via preprocess=False or as_memmap() in README and tests. Co-authored-by: Cursor <cursoragent@cursor.com>
There was a problem hiding this comment.
Pull request overview
This PR adds a supported “metadata-only” (only_text=True) workflow and new lazy DATA access helpers to FlowData, enabling mmap-backed access and indexed row reads for uniform IEEE float FCS layouts while keeping lazy and eager preprocessing consistent.
Changes:
- Documented and stabilized
only_text=Trueas a metadata probe (README +FlowDatadocstring) while retaining DATA offsets/metadata for later access. - Added lazy DATA helpers:
data_byte_range,numpy_dtype(),as_memmap(),read_events(indices=..., preprocess=...), plus shared_apply_preprocessing()to keep behavior aligned withas_array(). - Added unit tests covering memmap parity, endian handling, preprocessing defaults, and unsupported layouts/handles.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| tests/test_lazy_data.py | New tests for lazy/mmap helpers, preprocessing parity, and rejection paths. |
| src/flowio/flowdata.py | Implements DATA byte-range/dtype helpers, memmap + indexed reads, and shared preprocessing. |
| src/flowio/exceptions.py | Adds UnsupportedLazyDataError for unsupported lazy access scenarios. |
| README.md | Documents only_text=True as the metadata probe and introduces lazy DATA access usage. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+713
to
+716
| return ( | ||
| self._dataset_offset + self.data_start, | ||
| self._dataset_offset + self.data_stop, | ||
| ) |
Comment on lines
+862
to
+866
| if self.events is not None and indices is None: | ||
| return self.as_array(preprocess=preprocess) | ||
|
|
||
| if self.events is not None: | ||
| events_2d = np.reshape( |
whitews
approved these changes
Aug 15, 2026
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## develop #39 +/- ##
============================================
- Coverage 93.56% 83.10% -10.46%
============================================
Files 7 7
Lines 497 586 +89
============================================
+ Hits 465 487 +22
- Misses 32 99 +67 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
only_text=Trueas the supported metadata probe (README +FlowDatadocstring); retain DATA offsets / dtype metadata for later lazy access.data_byte_range,numpy_dtype(),as_memmap(), andread_events(indices=...)for uniform$DATATYPE=F/Dlayouts; raiseUnsupportedLazyDataErrorfor ASCII, integer, variable bit-width, and in-memory handles without a path.read_events(..., preprocess=True)— default matchesas_array()(gain/log/timestep). Shared_apply_preprocessing()keeps lazy and eager paths aligned. Usepreprocess=Falseoras_memmap()for raw stored DATA.ignore_offset_errorwarning parity on the lazy mmap offset correction path.data/fcs_files/fixtures (little- and big-endian float, integer rejection, variable-bit rejection, preprocess default vsas_array()).Three-tier lazy access model
as_memmap()read_events(..., preprocess=False)read_events()/as_array()Closes / related: #38
Test plan
PYTHONPATH=src python3 -m pytest tests/test_lazy_data.py tests/test_flowdata.py -q(30 passed)read_events()default preprocess matchesas_array(preprocess=True)on lazy pathonly_text+ lazy helpersMade with Cursor