From 7b418b56102dc92c21b8dd4aac1df56e088ba26d Mon Sep 17 00:00:00 2001 From: Glenn Hickey Date: Tue, 25 Aug 2026 07:46:49 -0400 Subject: [PATCH] Zero the external array buffer before it is written to the file Not every byte of an Hdf5ExternalArray buffer gets set before the buffer is flushed. The bottom segment "length" field never is -- lengths are derived from the next segment's start position, so nothing writes it and nothing reads it -- and the buffer comes from new char[], so whatever the allocator last had there is serialised into the file. The result is that two runs producing the same alignment write different files. It showed up as heap garbage in element 0 of a genome's BOTTOM_ARRAY: one run had 0, another had 7378429340001763376, whose bytes spell out a fragment of a sequence name left over in freed memory. Only the first buffer is affected in a fresh file, since page() reads every later window back and hdf5 returns its fill value for regions never written. Zero it in initBuf() so it covers load() as well as create(). A file opened for modification reaches its arrays through Hdf5Genome::read(), which loads them, and segments are then written through those buffers -- zeroing only on the create() path leaves uninitialised bytes in the merged output of halAppendSubtree. Beyond reproducibility this stops process memory leaking into a shared file, and lets h5diff and checksums be used to compare two hal files meaningfully. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_017E9w7KrbFEfqNRtkam4u1F --- api/hdf5_impl/hdf5ExternalArray.cpp | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/api/hdf5_impl/hdf5ExternalArray.cpp b/api/hdf5_impl/hdf5ExternalArray.cpp index 37d52aa5..32fcad85 100644 --- a/api/hdf5_impl/hdf5ExternalArray.cpp +++ b/api/hdf5_impl/hdf5ExternalArray.cpp @@ -7,6 +7,7 @@ #include "hdf5ExternalArray.h" #include +#include #include using namespace hal; @@ -30,6 +31,17 @@ void Hdf5ExternalArray::initBuf() { _bufEnd = _bufSize - 1; delete[] _buf; _buf = new char[_bufSize * _dataSize]; + // Zero it. Not every byte of the buffer is written before it is flushed -- the + // bottom segment "length" field never is, since lengths are derived from the next + // segment's start -- and whatever the allocator handed back would otherwise be + // serialised into the file, leaving two runs that produced the same alignment + // differing byte for byte. + // + // This has to cover load() as well as create(): a file opened for modification + // reaches its arrays through Hdf5Genome::read(), which loads them, and segments + // are then written through those buffers. Zeroing only on create() leaves + // uninitialised bytes in the merged output of halAppendSubtree. + memset(_buf, 0, _bufSize * _dataSize); } // Create a new dataset in specifed location