Introduce a small internal buffer for I/O. - #81
Merged
Merged
Conversation
ashleyz
approved these changes
Apr 22, 2026
Wrap nestegg_io in an internal ne_io struct and change all pure I/O function signatures from nestegg_io * to ne_io *. This separates I/O state from parser state, so I/O functions cannot access the full nestegg context. Purely mechanical, no behavioral change.
Capture the logical stream position after each packet is fully read and expose it via nestegg_packet_end_offset(). For SimpleBlock packets this is the end of the block; for BlockGroup packets this is the end of the entire group. This lets callers query the stream position without relying on the raw I/O tell callback, which may not reflect the logical position when internal buffering is active.
Change nestegg_io read callback to return number of bytes read, allowing short reads. nestegg now uses an internal 8KB buffer to service reads internally, refilling the buffer via the updated read callback. This significantly reduces I/O callback overhead for the many single-byte reads the EBML and lace parsers require. The max_offset parse fence is enforced during init only; it is cleared after init succeeds so that subsequent cue loading, seeking, and packet reading can access the full stream. Refactor the test suite to run all tests through legacy read, buffered readn, and short-read (16 byte cap) readn I/O paths. Add a seek-then-read test and a max_offset fencing test. Enable sanitizer halt_on_error in the test harness.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rework nestegg_io with a new ne_io wrapper that provides an 8KB internal buffer to significantly reduce the I/O callback overhead for single-byte reads from the EBML and lacing parsers.
nestegg_packet_end_offset is added to provide an accurate packet end offset based on the parser's logical stream position since the buffered I/O can cause the raw stream position to be further ahead. This is intended to replace Gecko's Tell() after a nestegg_read_packet with an API at the appropriate abstraction level.
It's tricky to give a definitive performance improvement ratio for this change, since it's so dependent on the cost of the user-supplied I/O read callback. Testing locally with a trivial read callback, there's around a 1.4x improvement for typical files and up to 7x improvement for crafted files that trigger many single-byte reads. Gecko's read callback is fairly complex, with differing performance based on lock contention and caching, but simulating a moderately expensive read callback (by adding a CPU yield to the trivial callback mentioned earlier) shows a 4x improvement for typical files and 49x for crafted files.