Measuring memory usage of Zarr array storage operations using memray.
In an ideal world array storage operations would be zero-copy, but many libraries do not achieve this in practice. The scripts here measure what the actual empirical behaviour is across different filesystems (local/cloud), Zarr stores (local/s3fs/obstore/zarrs), compression settings (using numcodecs), Zarr implementations (Zarr Python v2/v3, and zarrs via Zarrista), and Zarr formats (2/3).
- Writes with the latest Zarr Python are best achievable (0 copies uncompressed, 1 compressed), for both the local store and obstore
- Reads still need a lot of work, see: Codec pipeline memory usage zarr-developers/zarr-python#2904, but the new Zarrista library looks very promising here
- 24 August 2026. obstore 0.11.1 was released, which included developmentseed/obstore#771, removing the buffer copy that the multipart upload path made for in-memory buffers. Uncompressed obstore writes are now zero-copy.
- 17 August 2026. Add icechunk as a current target.
- 15 August 2026. Added zarrs (via Zarrista) to the comparison - the first library tested to achieve zero-copy uncompressed reads - and split the results into current and legacy tables.
- 19 May 2025. Zarr Python 3.0.8 was released, which included the fix for zarr-developers/zarr-python#2972
- 14 May 2025. zarr-developers/zarr-python#2972 was merged, reducing the number of buffer copies for obstore writes using Zarr v3 by one (local files and S3).
- 25 April 2025. Progress in making fsspec
pipeavoid a copy by using cramjam: fsspec/s3fs#959 (comment). Path forward for Zarr is not clear though. - 21 April 2025. Zarr Python 3.0.7 was released, which included the fix for zarr-developers/zarr-python#2944
- 8 April 2025. Numcodecs 0.16.0 was released which fixed zarr-developers/numcodecs#717, reducing the number of buffer copies in compressed writes by one.
- 3 April 2025. zarr-developers/zarr-python#2944 was merged, reducing the number of buffer copies for local writes using Zarr v3 by one.
- 6 March 2025. First commit in this repo.
The workload is simple: create a random 100MB NumPy array and write it to Zarr storage in a single chunk. Then (in a separate process) read it back from storage into a new NumPy array.
- Writes with no compression should not incur any buffer copies.
- Writes with compression incur a buffer copy, since implementations first write the compressed bytes into another buffer, which has to be around the size of the uncompressed bytes (since it is not known in advance how compressible the original is).
- Reads with no compression should be able to incur no buffer copies by reading directly into the array buffer. zarrs (via Zarrista) achieves this; Zarr Python does not yet, for either v2 or v3. See zarr-developers/zarr-python#2904
- Reads with compression incur a second buffer copy for a separate decompress step, except for zarrs (via Zarrista) and Zarr v2 reading from the local filesystem, both of which decompress directly into the array buffer.
Local filesystem with current library versions. S3 is not tracked yet, but for obstore the numbers for S3 and local should be the same.
These tables are regenerated by make. The Libraries column reports the versions actually exercised: configs/ pins zarr and zarrista but leaves the store libraries floating, so each profile records its own versions in a profiles/*.versions.json sidecar as it runs, and the table is built from those.
zarrista appears in the reads table but not the writes table for icechunk: it can read an icechunk repository, but not write to one yet (its session bridge reconstructs a separate session internally, so writes are silently lost). Those reads are of data written by the zarr-python + icechunk target.
Number of extra copies needed to write an array to storage using Zarr. (Links are to memray flamegraphs. Bold indicates best achievable.)
| Filesystem | Store | Libraries | Zarr format | Uncompressed | Compressed |
|---|---|---|---|---|---|
| Local | local | zarr 3.3.0 | 3 | 0 | 1 |
| obstore | zarr 3.3.0, obstore 0.11.1 | 3 | 0 | 1 | |
| icechunk | zarr 3.3.0, icechunk 2.1.2 | 3 | 1 | 2 | |
| local | zarrista 0.1.0 | 3 | 1 | 1 | |
| obstore | zarrista 0.1.0, obstore 0.11.1 | 3 | 1 | 1 |
Number of extra copies needed to read an array from storage using Zarr. (Links are to memray flamegraphs. Bold indicates best achievable.)
| Filesystem | Store | Libraries | Zarr format | Uncompressed | Compressed |
|---|---|---|---|---|---|
| Local | local | zarr 3.3.0 | 3 | 1 | 2 |
| obstore | zarr 3.3.0, obstore 0.11.1 | 3 | 1 | 2 | |
| icechunk | zarr 3.3.0, icechunk 2.1.2 | 3 | 2 | 2 | |
| local | zarrista 0.1.0 | 3 | 0 | 1 | |
| obstore | zarrista 0.1.0, obstore 0.11.1 | 3 | 0 | 1 | |
| icechunk | zarrista 0.1.0, icechunk 2.1.2 | 3 | 1 | 1 |
Kept for historical interest: older zarr-python versions and S3. These are frozen and not regenerated - as new library versions are released, the current rows above are promoted here by hand. These rows predate the version sidecars, so they name only the zarr and numcodecs versions that were recorded at the time.
| Filesystem | Store | Libraries | Zarr format | Uncompressed | Compressed |
|---|---|---|---|---|---|
| Local | local | zarr 2.18.7, numcodecs 0.15.1 | 2 | 0 | 2 |
| zarr 3.0.6, numcodecs 0.15.1 | 3 | 1 | 2 | ||
| zarr 3.0.8, numcodecs 0.16.1 | 3 | 0 | 1 | ||
| obstore | zarr 3.0.8, numcodecs 0.16.1 | 3 | 0 | 1 | |
| S3 | s3fs | zarr 2.18.7, numcodecs 0.15.1 | 2 | 1 | 2 |
| zarr 3.0.6, numcodecs 0.15.1 | 3 | 1 | 2 | ||
| obstore | zarr 3.0.8, numcodecs 0.16.1 | 3 | 0 | 1 |
| Filesystem | Store | Libraries | Zarr format | Uncompressed | Compressed |
|---|---|---|---|---|---|
| Local | local | zarr 2.18.7, numcodecs 0.15.1 | 2 | 1 | 1 |
| zarr 3.0.6, numcodecs 0.15.1 | 3 | 1 | 2 | ||
| obstore | zarr 3.0.8, numcodecs 0.16.1 | 3 | 1 | 2 | |
| S3 | s3fs | zarr 2.18.7, numcodecs 0.15.1 | 2 | 2 | 2 |
| zarr 3.0.6, numcodecs 0.15.1 | 3 | 2 | 2 | ||
| obstore | zarr 3.0.8, numcodecs 0.16.1 | 3 | 1 | 2 |
Update: some of these paths have been fixed now, so the below may not represent the code in the latest releases.
This delves into what is happening for the different code paths, and suggests some remedies to reduce the number of buffer copies.
-
Local uncompressed writes (v2 only) - actual copies 0, desired copies 0
- This is the only zero-copy case. The numpy array is passed directly to the file's
write()method (inDirectoryStore), and since arrays implement the buffer protocol, no copy is made.
- This is the only zero-copy case. The numpy array is passed directly to the file's
-
S3 uncompressed writes (v2 only) - actual copies 1, desired copies 0
- A copy of the numpy array is made by this code in fsspec (in
maybe_convert, called fromFSMap.setitems()):bytes(memoryview(value)). - Remedy: it might be possible to use the memory view in fsspec and avoid the copy (see fsspec/s3fs#959), but it's probably better to focus on improvements to v3 (see below)
- A copy of the numpy array is made by this code in fsspec (in
-
Uncompressed writes (v3 only) - actual copies 1, desired copies 0
- A copy of the numpy array is made by this code (in Zarr's
LocalStore):memoryview(value.as_numpy_array().tobytes()). A similar thing happens inFsspecStoreand obstore. - Remedy: this could be fixed with zarr-developers/zarr-python#2925, so the
valueBufferis exposed via the buffer protocol without making a copy.
- A copy of the numpy array is made by this code (in Zarr's
-
Compressed writes - actual copies 2, desired copies 1
- It is surprising that there are two copies, not one, given that the uncompressed case has zero copies (for local v2, at least). What's happening is that the numcodecs blosc compressor is making an extra copy when it resizes the compressed buffer. A similar thing happens for lz4 and zstd.
- Remedy: the issue is tracked in numcodecs in zarr-developers/numcodecs#717.
-
Local reads (v2 only) - actual copies 1, desired copies 0
- The Zarr Python v2 read pipeline separates reading the bytes from storage, and filling the output array - see
_process_chunk(). So there is necessarily a buffer copy, since the bytes are never read directly into the output array. - Remedy: Zarr Python v2 is in bugfix mode now so there is no point in trying to change it to make fewer buffer copies. The changes would be quite invasive anyway.
- The Zarr Python v2 read pipeline separates reading the bytes from storage, and filling the output array - see
-
Local reads (v3 only), plus obstore local and S3 - actual copies 1 (2 for compressed), desired copies 0 (1 for compressed)
- The Zarr Python v3
CodecPipelinehas aread()method that separates reading the bytes from storage, and filling the output array (just like v2). TheByteGetterclass has no way of reading directly into an output array. - Remedy: this could be fixed by zarr-developers/zarr-python#2904, but it is potentially a major change to Zarr's internals
- The Zarr Python v3
-
S3 reads (s3fs only) - actual copies 2, desired copies 0
- Both the Python asyncio SSL library and aiohttp introduce a buffer copy when reading from S3 (using s3fs).
- Remedy: unclear
- [cubed] Improve memory model by explicitly modelling buffer copies - cubed-dev/cubed#701 (fixed)
- [zarr-python] Codec pipeline memory usage - zarr-developers/zarr-python#2904
- [zarr-python] Add
Buffer.as_buffer_likemethod - zarr-developers/zarr-python#2925 (fixed) - [zarr-python] Avoid memory copy in local store write - zarr-developers/zarr-python#2944 (fixed)
- [zarr-python] Avoid memory copy in obstore write - zarr-developers/zarr-python#2972 (fixed)
- [numcodecs] Extra memory copies in blosc, lz4, and zstd compress functions - zarr-developers/numcodecs#717 (fixed)
- [numcodecs] Switch
Buffers tomemoryviews - zarr-developers/numcodecs#656 (fixed) - [obstore] Reduce memory overhead for multipart uploads - developmentseed/obstore#771 (fixed, released in 0.11.1)
- [s3fs] Using the Python buffer protocol in
pipe- fsspec/s3fs#959
Requires uv. Per-configuration dependencies are declared in configs/. uv resolves and caches each environment automatically - no manual pip install or virtual env activation is needed.
The actively tracked matrix (local filesystem, latest zarr-python, and zarrista):
make current-profilesTo run a single profile:
make profiles/write-local-zarr-3.3.0-fsspec-compressed.binFor read profiles, the corresponding write is run first automatically (the data must exist).
When a new zarr-python is released, bump CURRENT_ZARR in the Makefile, add a matching configs/zarr-<version>.txt, regenerate, and move the outgoing rows from the current table into the "Legacy results" table by hand (their flamegraphs are already committed).
Older versions and S3 are frozen in the "Legacy results" table and are not part of the default build. They can still be regenerated for reproducibility:
make legacy-profiles
export AWS_DEFAULT_REGION=...
export AWS_ACCESS_KEY_ID=...
export AWS_SECRET_ACCESS_KEY=...
make s3-profiles S3_PREFIX=s3://your-bucket/mem-arraymakeThis generates a temporal flamegraph for each current profile and then regenerates the current-results tables in this README.
To regenerate the tables without rebuilding flamegraphs:
make tableFlamegraph links point at the GitHub Pages copy (https://tomwhite.github.io/memray-array/), because github.com serves committed .html files as source rather than rendering them. To check flamegraphs that haven't been pushed yet, either open flamegraphs/*.html directly in a browser, or generate the tables with relative links:
make table FLAMEGRAPH_BASE=flamegraphs(Don't commit the relative-link version — it renders as unclickable source on github.com.)