diff --git a/.claude/CLAUDE.md b/.claude/CLAUDE.md index a25221afe..6b2a821af 100644 --- a/.claude/CLAUDE.md +++ b/.claude/CLAUDE.md @@ -297,7 +297,77 @@ Semantics (all probed against NumPy 2.4.2, pinned in `NDEvaluateTests.cs`): `bernoulli`, `beta`, `binomial`, `chisquare`, `choice`, `dirichlet`, `exponential`, `f`, `gamma`, `geometric`, `gumbel`, `hypergeometric`, `laplace`, `logistic`, `lognormal`, `logseries`, `multinomial`, `multivariate_normal`, `negative_binomial`, `noncentral_chisquare`, `noncentral_f`, `normal`, `pareto`, `permutation`, `poisson`, `power`, `rand`, `randint`, `randn`, `random_sample`, `rayleigh`, `seed`, `shuffle`, `standard_cauchy`, `standard_exponential`, `standard_gamma`, `standard_normal`, `standard_t`, `triangular`, `uniform`, `vonmises`, `wald`, `weibull`, `zipf` ### File I/O -`fromfile`, `load`, `save`, `tofile` +`fromfile`, `load`, `load_npy`, `load_npz`, `save`, `savez`, `savez_compressed`, `tofile` + +The `.npy`/`.npz` stack is a **port of NumPy 2.4.2's `numpy/lib/_format_impl.py`** (NEP-01) in +`IO/{NpyFormat,NpzFile,PyLiteral}.cs`, and the writer is **byte-for-byte identical to NumPy's own +`np.save`** — not merely readable by it. Format versions **1.0 / 2.0 / 3.0** (2- vs 4-byte header +length; latin-1 vs UTF-8), **64-byte `ARRAY_ALIGN`** data alignment (so the data is mmap-ready), +**`fortran_order`** both ways (write via the transpose; read via reshape-reversed + transpose), and +**big-endian** files byte-swapped to native on read (per-component: `>c16` swaps in 8s, `>U1` in 4s). +Headers parse through a real Python-literal parser (`PyLiteral`, standing in for `ast.literal_eval`) +rather than `IndexOf`/regex, so key order, whitespace, quoting, trailing commas and the Python 2 `3L` +suffix all work; `max_header_size` (default 10000) bounds hostile input and `allow_pickle` lifts it. + +`np.load` returns `object` — `NDArray` for `.npy`, `NpzFile` for `.npz` — mirroring NumPy's +content-dependent return; **`np.load_npy` / `np.load_npz` are the typed forms** and need no cast. +`NpzFile` is lazy + cached, takes `"w"` or `"w.npy"` as keys, exposes `.Files` stripped, supports +`npz.f.weights` dot access (NumPy's `BagObj`) and **must be disposed**. + +| Dtype | Maps to | Notes | +|-------|---------|-------| +| Boolean/SByte/Byte | `\|b1` / `\|i1` / `\|u1` | single-byte types take the `\|` prefix, never `<` | +| Int16…UInt64, Half, Single, Double | `1/`\|V`, ` 4 GiB** — verified at +5 GiB against every 32-bit boundary (`int.MaxValue`, `uint.MaxValue`) with NumPy reading the result, +and the 5 GiB header is byte-identical to NumPy's. Array data lives in unmanaged memory and both IO +paths stream in 256 KB chunks, so no managed buffer ever scales with the array. Zip64 comes from +.NET's `ZipArchive` on demand (NumPy forces `force_zip64=True` for the same reason, gh-10776). The +only 2 GB ceilings left are the inherently `byte[]`-returning conveniences — `np.save(arr)`, +`np.savez(...)` → `byte[]`, and `NpzFile.GetRawBytes` — which is a `byte[]` limit, not a format one; +use the path/stream overloads above 2 GB. Tests: `NpyLargeFileTests` (`[HighMemory][LongIndexing]`, +excluded from CI; needs ~8 GB RAM + ~6 GB disk). + +**Gates:** `test/oracle/gen_npy_oracle.py` → `IO/corpus/npy_oracle.zip` (286 committed cases of real +NumPy output) replayed by `NpyOracleTests` under the **`NpyOracle`** category — read / byte-exact +write / header-only write / verbatim error / round-trip / npz / live-view write / hostile-allocation. +Reverse interop (NumPy reading NumSharp, the direction byte-equality can't cover for `.npz`) is the +manual gate `python test/oracle/verify_npy_interop.py`. ### Printing / Formatting `array2string`, `array_repr`, `array_str`, `format_float_positional`, `format_float_scientific`, `get_printoptions`, `printoptions` (IDisposable context), `set_printoptions` @@ -342,6 +412,7 @@ Semantics (all probed against NumPy 2.4.2, pinned in `NDEvaluateTests.cs`): | ILKernelGenerator | `Backends/Kernels/ILKernelGenerator*.cs` (per-chunk, NDIter-driven) | | DirectILKernelGenerator | `Backends/Kernels/Direct/DirectILKernelGenerator.*.cs` (whole-array, 63 partials) | | Kernel shared infra | `Backends/Kernels/{VectorMethodCache,ScalarMethodCache,KernelOp,StrideDetector,SimdMatMul.*}.cs` | +| .npy/.npz format (NEP-01) | `IO/NpyFormat.cs` (port of NumPy's `_format_impl.py`), `IO/NpzFile.cs` (lazy archive), `IO/PyLiteral.cs` (header parser ≙ `ast.literal_eval`), `APIs/np.{save,load}.cs` | | Type info | `Utilities/InfoOf.cs` | | Generic NDArray | `Generics/NDArray\`1.cs` | @@ -561,6 +632,10 @@ test/oracle/ corpus generators (NumPy 2.4.2 — run by varstd,matmul,astype,stat,where,sort,manip}.jsonl gen_index_oracle.py getter/setter index oracle (token index over 15 base recipes) fuzz_random.py seeded random fuzzer (imports the other two) + gen_npy_oracle.py .npy/.npz format oracle: REAL np.save/savez output + a manifest -> + NumSharp.UnitTest/IO/corpus/npy_oracle.zip (286 cases) + interop_write.cs the reverse direction: NumSharp writes, verify_npy_interop.py reads + verify_npy_interop.py manual gate — real NumPy reads NumSharp's output (needs Python + SDK) test/NumSharp.UnitTest/Fuzz/ C# replay harness (no Python) FuzzCorpus.cs rebuilds EXACT NDArray views from (dtype,shape,strides,offset,bytes) OpRegistry.cs op-name → NumSharp call (pairs 1:1 with gen_oracle.py) @@ -571,6 +646,20 @@ test/NumSharp.UnitTest/Fuzz/ C# replay harness (no Python) MetamorphicTests.cs oracle-free invariants (round-trips / involutions / identities — no NumPy) HarnessSelfTests.cs proves the harness has teeth (BitDiff detects value/NaN/-0 diffs; non-vacuous) corpus/*.jsonl committed corpus (~68K cases / 40 tiers; op corpus ~53K incl. 3.7K Char woven + 579 Decimal), copied to test output via the csproj glob +test/NumSharp.UnitTest/IO/ .npy/.npz format gate (no Python) + NpyOracleCorpus.cs opens npy_oracle.zip, rebuilds arrays from the manifest + NpyOracleTests.cs one [NpyOracle] test per claim: read / byte-exact write / header-only + write / verbatim error / round-trip / npz / live-view write / + hostile-allocation / corpus non-vacuity + NpyLargeFileTests.cs the 2 GB and 4 GB walls; the 5 GiB pair is [HighMemory] (not in CI) + AllocationTests.cs MinAllocated: best-of-N, since GetTotalAllocatedBytes is process-wide + corpus/npy_oracle.zip committed real-NumPy output (286 cases), copied via the csproj glob + NumpyCompatibilityTests.cs hand-made NumPy fixtures (test_compat/) — v2.0/v3.0/fortran/scalar/ + NpyFormatVersionTests.cs empty/bool/multi/compressed. They resolve test_compat RELATIVE to the + working dir and must fail loudly when it is absent: they used to + `return` silently, and nothing copied the fixtures, so on a clean + checkout all 13 passed while asserting NOTHING. The csproj now copies + test_compat\* — do not drop that rule. ``` - **Generators live in `test/oracle/`** and write the corpus into `test/NumSharp.UnitTest/Fuzz/corpus/` (path resolved relative to `test/oracle/`). CI replays the committed corpus, never the generators. @@ -579,6 +668,27 @@ test/NumSharp.UnitTest/Fuzz/ C# replay harness (no Python) - **Regenerate** (deterministic; needs `numpy==2.4.2`): `python test/oracle/gen_oracle.py ` (modes: `smoke astype_full binary divmod_power comparison unary reduce where place matmul bitwise unary_extra nanreduce scan stat logic modf manip sort tail params aliasing copyto errors`) + `python test/oracle/gen_index_oracle.py` (the `index_*` tiers) + `python test/oracle/fuzz_random.py 1234 2000 random_smoke.jsonl` + `dotnet run test/oracle/gen_decimal_oracle.cs` (the `decimal_*` tiers), then `dotnet build` (copies the corpus to test output). - **Run the gate**: `dotnet test --filter "TestCategory=FuzzMatrix"`. Each case is bit-exact (pass), a documented difference in `MisalignedRegistry` (excused, never silent), or a failure (red). Full divergence ledger: `test/NumSharp.UnitTest/Fuzz/README.md`. +### The `.npy`/`.npz` format oracle (same philosophy, separate corpus) + +Same shape as above — NumPy is the oracle, the corpus is committed, no Python at test time — but the +claim is stronger: NumSharp's writer must be **byte-identical** to `np.save`, not merely readable. + +- **Regenerate**: `python test/oracle/gen_npy_oracle.py` (deterministic; needs `numpy==2.4.2`), then `dotnet build`. +- **Run the gate**: `dotnet test --filter "TestCategory=NpyOracle"` — 286 cases across every dtype × + {0-d, empty, 1-D, 2-D, 3-D, unit, empty-2d/3d} × {C, F, strided, reversed, offset, broadcast, + transposed} × versions {1.0, 2.0, 3.0} × {little, big, native} endian, plus value fidelity (NaN + payloads incl. sNaN, subnormals, signed zero, integer extremes, BMP seams) and 42 + malformed / unsupported / hostile files whose messages must match NumPy verbatim. +- **`kind: "header"` cases exist because two writer branches are unreachable from a real array.** The + growth padding is normally INVISIBLE — shrink the body by 5 chars and the alignment padding grows by + 5, leaving the file identical — so a wrong growth axis (`shape[0]` vs `shape[-1]`) passes every + ordinary test; it only shows on shapes that tip the header across a 64-byte bucket, and those need + 10¹⁷ elements to allocate. The v1.0→v2.0 auto-selection boundary likewise sits at 21,817 dimensions. + Both are driven through the header dict, exactly as NumPy's `_write_array_header` was. +- **Reverse interop** (manual; needs Python + the SDK): `python test/oracle/verify_npy_interop.py` has + NumSharp write 28 files and real NumPy read them. This is the only way to prove `.npz` output, + whose ZIP framing legitimately differs from Python's `zipfile` and so cannot be byte-compared. + ## CI Pipeline `.github/workflows/build-and-release.yml` — test on 3 OSes (Windows/Ubuntu/macOS), build NuGet on tag push, create GitHub Release, publish to nuget.org. The `FuzzMatrix` gate runs here (replays the committed corpora; no Python). @@ -759,7 +869,10 @@ A: No external runtime dependencies. `System.Memory` and `System.Runtime.Compile A: TensorFlow.NET, ML.NET integrations, Gym.NET, Pandas.NET, and various scientific computing projects. **Q: Can I save/load NumPy files?** -A: Yes. `np.save()` writes `.npy` files, `np.load()` reads both `.npy` and `.npz` archives. Compatible with Python NumPy files. +A: Yes, and byte-exactly. `np.save` writes `.npy`, `np.savez`/`np.savez_compressed` write `.npz`, and `np.load` reads both (returning `object`; `np.load_npy`/`np.load_npz` are the typed forms). `IO/NpyFormat.cs` is a port of NumPy 2.4.2's `_format_impl.py`, so a saved file is byte-for-byte what NumPy's own `np.save` would produce — versions 1.0/2.0/3.0, 64-byte alignment, `fortran_order`, and big-endian reads included. See the File I/O section above for the dtype map and the gates. + +**Q: Why does `np.load` return `object` instead of `NDArray`?** +A: Because NumPy's does: `np.load` yields an `ndarray` for a `.npy` and an `NpzFile` for a `.npz`, dispatching on the file's magic bytes rather than its extension, and C# has no union type to express that. `np.load_npy` and `np.load_npz` are the typed escapes — prefer them when the kind is known; each also reports a directed error if handed the other kind. **Q: What random distributions are supported?** A: An extensive NumPy-matching set (40+ distributions and samplers — see the Supported np.* APIs → Random list), all on the `NumPyRandom` class in `RandomSampling/`, with 1-to-1 seed/state parity to NumPy. diff --git a/docs/website-src/docs/NDArray.md b/docs/website-src/docs/NDArray.md index 625562d1b..8e5de96f9 100644 --- a/docs/website-src/docs/NDArray.md +++ b/docs/website-src/docs/NDArray.md @@ -155,14 +155,14 @@ See the [Buffering & Memory](buffering.md) page for the full story: memory archi | `@base` | `NDArray?` | `ndarray.base` | Owner array if this is a view, else `null` | ```csharp -var a = np.arange(12).reshape(3, 4); +var a = np.arange(12).reshape(3, 4); // arange defaults to int64 (NumPy 2.x) a.shape; // [3, 4] a.ndim; // 2 a.size; // 12 -a.dtype; // typeof(int) -a.typecode; // NPTypeCode.Int32 +a.dtype; // typeof(long) +a.typecode; // NPTypeCode.Int64 a.T.shape; // [4, 3] -a.@base; // null (arange owns its data) +a.@base; // the 1-D arange buffer — reshape returns a view (like NumPy) var b = a["1:, :2"]; b.@base; // wraps a's Storage (b is a view) ``` @@ -209,7 +209,7 @@ a[a > 10] = -1; // masked write > **View / copy summary for indexing:** > - Plain slices (`a["1:3"]`, `a[0]`, `a[..., -1]`): **writeable view** — shares memory with the parent. > - Fancy indexing (`a[indexArray]`): **writeable copy** — independent memory (matches NumPy). -> - Boolean masking (`a[mask]`): **read-only copy** — independent memory; mutation via `a[mask] = value` still works as an *assignment* because it goes through the setter, not by writing into the returned array. +> - Boolean masking (`a[mask]`): **writeable copy** — independent memory (matches NumPy). Writing into the returned array is allowed but does not touch the parent; to modify the parent use `a[mask] = value`, which goes through the setter. --- @@ -228,7 +228,7 @@ c[0] = 0; a[2]; // still 999 ``` -Detect views with `arr.@base != null`. Force a copy with `.copy()` or `np.copy(arr)`. +Detect views with `arr.@base is not null` (use the C# `is` pattern, not `!= null` — NDArray's `==`/`!=` operators are element-wise and return an array, not a bool). Force a copy with `.copy()` or `np.copy(arr)`. Broadcasted arrays are a special case: they're views with stride=0 dimensions, and they're **read-only** (`Shape.IsWriteable == false`) to prevent cross-row corruption. See [Broadcasting](broadcasting.md#memory-behavior). @@ -378,26 +378,26 @@ To unwrap a 0-d result to a raw C# scalar, cast: `(int)a[i]` or `a.item(i)` Four ways to touch individual elements, picked based on how many indices you have and whether you already know the dtype: ```csharp -var a = np.arange(12).reshape(3, 4); +var a = np.arange(12).reshape(3, 4); // int64 (NumPy 2.x default integer) // 1. Indexer — returns NDArray (0-d for a single element) NDArray elem = a[1, 2]; -int v = (int)elem; // explicit cast to scalar +long v = (long)elem; // explicit cast to scalar (converts) -// 2. .item() — direct scalar extraction (NumPy parity) -int v2 = a.item(6); // flat index 6 → row 1, col 2 -object box = a.item(6); // untyped form returns object +// 2. .item() — direct scalar extraction (NumPy parity; converts if T differs) +long v2 = a.item(6); // flat index 6 → row 1, col 2 +object box = a.item(6); // untyped form returns object (boxed long) -// 3. GetValue — N-D coordinates, typed -int v3 = a.GetValue(1, 2); +// 3. GetValue — N-D coordinates, typed. T must match the dtype exactly. +long v3 = a.GetValue(1, 2); -// 4. GetAtIndex — flat index, typed, no Shape math (fastest) -int v4 = a.GetAtIndex(6); +// 4. GetAtIndex — flat index, typed, no Shape math (fastest). T must match the dtype. +long v4 = a.GetAtIndex(6); // Writes mirror the reads: -a[1, 2] = 99; // indexer assignment -a.SetValue(99, 1, 2); // N-D coordinates -a.SetAtIndex(99, 6); // flat index +a[1, 2] = 99; // indexer assignment (converts to the array dtype) +a.SetValue(99L, 1, 2); // N-D coordinates (value must match the dtype) +a.SetAtIndex(99L, 6); // flat index ``` **Rule of thumb:** use `.item()` when porting NumPy code, `GetAtIndex` in a hot loop, and the indexer (`a[i, j]`) when you want NumPy-like ergonomics and don't mind the 0-d NDArray detour. @@ -484,11 +484,11 @@ a[0] = 3.14; Three ways to get a typed wrapper: -| Method | Allocates? | When to use | -|--------|------------|-------------| -| `MakeGeneric()` | never (same storage) | You know the dtype matches | -| `AsGeneric()` | never; throws if dtype mismatch | Defensive typing | -| `AsOrMakeGeneric()` | only if dtype differs (then `astype`) | Accept any dtype, convert if needed | +| Method | Allocates? | On dtype mismatch | When to use | +|--------|------------|-------------------|-------------| +| `MakeGeneric()` | never (same storage) | throws `ArgumentException` | You already know the dtype matches | +| `AsGeneric()` | never (same storage) | returns `null` (like C# `as`) | Defensive typing — check for `null` | +| `AsOrMakeGeneric()` | only if dtype differs (then `astype`) | converts via `astype` | Accept any dtype, convert if needed | `NDArray` wraps the same storage; use the untyped `NDArray` when dtype is dynamic. @@ -496,18 +496,33 @@ Three ways to get a typed wrapper: ## Saving, Loading, and Interop -NumSharp reads and writes NumPy's `.npy` / `.npz` formats and raw binary — files saved in Python open in NumSharp, and vice versa. To wrap an existing in-memory byte buffer (file bytes, a network packet, a native pointer) see [`np.frombuffer`](#wrapping-existing-buffers--npfrombuffer) above. +NumSharp reads **and** writes NumPy's `.npy` / `.npz` formats and raw binary. The `.npy`/`.npz` stack is a port of NumPy 2.4.2's format code (NEP-01), so a file written by `np.save` is **byte-for-byte identical** to what NumPy itself writes — data moves between Python and C# in both directions, losslessly. To wrap an existing in-memory byte buffer (file bytes, a network packet, a native pointer) see [`np.frombuffer`](#wrapping-existing-buffers--npfrombuffer) above. ```csharp -// .npy round-trip +// .npy — a single array (byte-identical to NumPy's np.save) np.save("arr.npy", arr); -var loaded = np.load("arr.npy"); // also handles .npz archives +NDArray a = np.load_npy("arr.npy"); // typed load -// Raw binary +// .npz — many arrays in one archive +np.savez("bundle.npz", x, y); // positional → "arr_0", "arr_1" +np.savez("bundle.npz", new Dictionary { ["w"] = w, ["b"] = b }); +np.savez_compressed("bundle.npz", w, b); // same, Deflate-compressed + +using NpzFile npz = np.load_npz("bundle.npz"); // lazy + cached; dispose it (holds the file handle) +NDArray w1 = npz["w"]; // "w.npy" also works as a key +NDArray w2 = npz.f.w; // dot access, like NumPy's npz.f +foreach (string name in npz.Files) { } // "w", "b" — the ".npy" is stripped + +// np.load dispatches on the file's magic bytes and returns `object` +object any = np.load("bundle.npz"); // NDArray for .npy, NpzFile for .npz + +// Raw binary — element bytes only, no header arr.tofile("data.bin"); var raw = np.fromfile("data.bin", np.float64); ``` +Format versions 1.0 / 2.0 / 3.0, C- and Fortran-order, and big-endian files all load; the writer emits the byte-exact, 64-byte-aligned, mmap-ready layout NumPy produces. `np.load` returns `object` because — like NumPy — it yields an array for a `.npy` and an archive for a `.npz`, decided by the file's contents rather than its name; prefer the typed `np.load_npy` / `np.load_npz` when you know the kind. `allow_pickle` defaults to `false` (NumPy's security default), so object-array files are rejected with a clear message instead of executed. See [NumPy Compliance](compliance.md) for the full dtype map and the handful of unsupported types. + Interop with standard .NET arrays: ```csharp @@ -550,7 +565,7 @@ Views can be non-contiguous (sliced, transposed, broadcasted). Use `arr.Shape.Is | `np.array_equal(a, b)` | `bool` | same shape AND all elements equal | | `np.allclose(a, b)` | `bool` | same shape AND all elements within tolerance (good for floats) | | `ReferenceEquals(a, b)` | `bool` | same C# object (rarely what you want) | -| `a.@base != null` | `bool` | `a` is a view (shares memory with some owner) | +| `a.@base is not null` | `bool` | `a` is a view (shares memory with some owner); use `is`, not `!= null` | > Caveat: NumSharp does not expose a direct "do these two arrays share memory?" check from user code. `a.@base` returns a fresh wrapper on every call and the underlying `Storage` is `protected internal`, so strict memory-identity testing is only available inside the assembly. @@ -562,9 +577,9 @@ Views can be non-contiguous (sliced, transposed, broadcasted). Use `arr.Shape.Is That's views. `a["1:3"]` shares memory with `a`. Force a copy: `a["1:3"].copy()`. -### "ReadOnlyArrayException writing to my slice" +### "NumSharpException: assignment destination is read-only" -You're writing to a broadcasted view (stride=0 dimension). Copy first: `b.copy()[...] = value`. +You're writing to a broadcasted view (stride=0 dimension), which is read-only. Copy first: `b.copy()["..."] = value`. ### "ScalarConversionException on `(int)arr`" @@ -576,7 +591,7 @@ C# requires the declaring type on the left of shift operators. Use `np.left_shif ### "a += 1 didn't update another reference" -C# compound assignment reassigns the variable; it doesn't mutate. See [Compound assignment](#compound-assignment) above. For in-place modification, write directly: `a[...] = a + 1`. +C# compound assignment reassigns the variable; it doesn't mutate. See [Compound assignment](#compound-assignment) above. For in-place modification, write directly: `a["..."] = a + 1` (or `a[":"] = a + 1`). --- @@ -617,8 +632,8 @@ C# compound assignment reassigns the variable; it doesn't mutate. See [Compound | `SetAtIndex(value, i)` | Write element at flat index | | `GetValue(indices)` | Read at N-D coordinates | | `SetValue(value, indices)` | Write at N-D coordinates | -| `MakeGeneric()` | Wrap as `NDArray` (same storage) | -| `AsGeneric()` | Wrap as `NDArray`; throws if dtype mismatch | +| `MakeGeneric()` | Wrap as `NDArray` (same storage); throws if dtype differs | +| `AsGeneric()` | Wrap as `NDArray`; returns `null` if dtype differs | | `AsOrMakeGeneric()` | Wrap as `NDArray`; `astype` if dtype differs | | `Data()` | Get the underlying `ArraySlice` handle | | `ToMuliDimArray()` | Copy to a rank-N .NET array | @@ -647,8 +662,12 @@ C# compound assignment reassigns the variable; it doesn't mutate. See [Compound | Call | Format | View / copy | Notes | |------|--------|-------------|-------| -| `np.save(path, arr)` | `.npy` | — | NumPy-compatible; writes header + data | -| `np.load(path)` | `.npy` / `.npz` | — | Also accepts a `Stream` | +| `np.save(path, arr)` | `.npy` | — | Byte-identical to NumPy's `np.save`; `Stream` / `byte[]` overloads | +| `np.savez(path, …)` | `.npz` | — | Many arrays; positional (`arr_0`, `arr_1`, …) or `IDictionary` | +| `np.savez_compressed(path, …)` | `.npz` | — | Same as `savez`, Deflate-compressed | +| `np.load(path)` | `.npy` / `.npz` | — | Returns `object` (`NDArray` or `NpzFile`); `Stream` / `byte[]` overloads | +| `np.load_npy(path)` | `.npy` | copy | Typed → `NDArray` | +| `np.load_npz(path)` | `.npz` | lazy | Typed → `NpzFile` (`IDisposable`; `.Files`, `["w"]` / `.f.w` access) | | `arr.tofile(path)` | raw | — | Element bytes only, no header | | `np.fromfile(path, dtype)` | raw | copy | Pair with `tofile` | | `np.frombuffer(byte[], …)` | in-memory | view (pins array) | Endian-prefix dtype strings trigger a copy | diff --git a/docs/website-src/docs/compliance.md b/docs/website-src/docs/compliance.md index 8ddebb0a5..972b6ba9a 100644 --- a/docs/website-src/docs/compliance.md +++ b/docs/website-src/docs/compliance.md @@ -251,25 +251,43 @@ NumSharp can read and write NumPy's `.npy` file format. This means you can: 3. Share data files between Python and C# applications ```csharp -// Save +// Save — the bytes are identical to what NumPy's own np.save would write var arr = np.arange(100).reshape(10, 10); np.save("mydata.npy", arr); // Load -var loaded = np.load("mydata.npy"); +NDArray loaded = np.load_npy("mydata.npy"); ``` +Format versions 1.0, 2.0 and 3.0 all load, in either byte order and in either C or Fortran order. +Object arrays, structured dtypes and `datetime64`/`timedelta64` are not supported — those files are +rejected with a message naming the reason rather than loading incorrectly. + ### .npz Archives -NumPy's `.npz` format stores multiple arrays in a ZIP archive. NumSharp can **read** `.npz` files but not write them yet. +NumPy's `.npz` format stores multiple arrays in a ZIP archive. NumSharp reads **and** writes them, +compressed or not. ```csharp -// Load multiple arrays from .npz -var archive = np.load("data.npz") as NpzDictionary; -var weights = archive["weights"]; -var biases = archive["biases"]; +// Write multiple arrays +np.savez("data.npz", new Dictionary { + ["weights"] = weights, + ["biases"] = biases, +}); +np.savez_compressed("data.npz", ...); // same, deflated + +// Read them back. Arrays load lazily and are cached; the archive holds a +// file handle, so dispose it. +using var archive = np.load_npz("data.npz"); +var weights = archive["weights"]; // "weights.npy" works too +var biases = archive.f.biases; // dot access, like NumPy's npz.f +archive.Files; // ["weights", "biases"] — ".npy" stripped ``` +`np.load` also opens `.npz` archives — it detects the format from the file's magic bytes and returns +either an `NDArray` or an `NpzFile`, mirroring NumPy. Prefer the typed `np.load_npy` / `np.load_npz` +when you know which one you have. + --- ## Linear Algebra: Partial Support diff --git a/docs/website-src/docs/ndarray.md b/docs/website-src/docs/ndarray.md index 625562d1b..8e5de96f9 100644 --- a/docs/website-src/docs/ndarray.md +++ b/docs/website-src/docs/ndarray.md @@ -155,14 +155,14 @@ See the [Buffering & Memory](buffering.md) page for the full story: memory archi | `@base` | `NDArray?` | `ndarray.base` | Owner array if this is a view, else `null` | ```csharp -var a = np.arange(12).reshape(3, 4); +var a = np.arange(12).reshape(3, 4); // arange defaults to int64 (NumPy 2.x) a.shape; // [3, 4] a.ndim; // 2 a.size; // 12 -a.dtype; // typeof(int) -a.typecode; // NPTypeCode.Int32 +a.dtype; // typeof(long) +a.typecode; // NPTypeCode.Int64 a.T.shape; // [4, 3] -a.@base; // null (arange owns its data) +a.@base; // the 1-D arange buffer — reshape returns a view (like NumPy) var b = a["1:, :2"]; b.@base; // wraps a's Storage (b is a view) ``` @@ -209,7 +209,7 @@ a[a > 10] = -1; // masked write > **View / copy summary for indexing:** > - Plain slices (`a["1:3"]`, `a[0]`, `a[..., -1]`): **writeable view** — shares memory with the parent. > - Fancy indexing (`a[indexArray]`): **writeable copy** — independent memory (matches NumPy). -> - Boolean masking (`a[mask]`): **read-only copy** — independent memory; mutation via `a[mask] = value` still works as an *assignment* because it goes through the setter, not by writing into the returned array. +> - Boolean masking (`a[mask]`): **writeable copy** — independent memory (matches NumPy). Writing into the returned array is allowed but does not touch the parent; to modify the parent use `a[mask] = value`, which goes through the setter. --- @@ -228,7 +228,7 @@ c[0] = 0; a[2]; // still 999 ``` -Detect views with `arr.@base != null`. Force a copy with `.copy()` or `np.copy(arr)`. +Detect views with `arr.@base is not null` (use the C# `is` pattern, not `!= null` — NDArray's `==`/`!=` operators are element-wise and return an array, not a bool). Force a copy with `.copy()` or `np.copy(arr)`. Broadcasted arrays are a special case: they're views with stride=0 dimensions, and they're **read-only** (`Shape.IsWriteable == false`) to prevent cross-row corruption. See [Broadcasting](broadcasting.md#memory-behavior). @@ -378,26 +378,26 @@ To unwrap a 0-d result to a raw C# scalar, cast: `(int)a[i]` or `a.item(i)` Four ways to touch individual elements, picked based on how many indices you have and whether you already know the dtype: ```csharp -var a = np.arange(12).reshape(3, 4); +var a = np.arange(12).reshape(3, 4); // int64 (NumPy 2.x default integer) // 1. Indexer — returns NDArray (0-d for a single element) NDArray elem = a[1, 2]; -int v = (int)elem; // explicit cast to scalar +long v = (long)elem; // explicit cast to scalar (converts) -// 2. .item() — direct scalar extraction (NumPy parity) -int v2 = a.item(6); // flat index 6 → row 1, col 2 -object box = a.item(6); // untyped form returns object +// 2. .item() — direct scalar extraction (NumPy parity; converts if T differs) +long v2 = a.item(6); // flat index 6 → row 1, col 2 +object box = a.item(6); // untyped form returns object (boxed long) -// 3. GetValue — N-D coordinates, typed -int v3 = a.GetValue(1, 2); +// 3. GetValue — N-D coordinates, typed. T must match the dtype exactly. +long v3 = a.GetValue(1, 2); -// 4. GetAtIndex — flat index, typed, no Shape math (fastest) -int v4 = a.GetAtIndex(6); +// 4. GetAtIndex — flat index, typed, no Shape math (fastest). T must match the dtype. +long v4 = a.GetAtIndex(6); // Writes mirror the reads: -a[1, 2] = 99; // indexer assignment -a.SetValue(99, 1, 2); // N-D coordinates -a.SetAtIndex(99, 6); // flat index +a[1, 2] = 99; // indexer assignment (converts to the array dtype) +a.SetValue(99L, 1, 2); // N-D coordinates (value must match the dtype) +a.SetAtIndex(99L, 6); // flat index ``` **Rule of thumb:** use `.item()` when porting NumPy code, `GetAtIndex` in a hot loop, and the indexer (`a[i, j]`) when you want NumPy-like ergonomics and don't mind the 0-d NDArray detour. @@ -484,11 +484,11 @@ a[0] = 3.14; Three ways to get a typed wrapper: -| Method | Allocates? | When to use | -|--------|------------|-------------| -| `MakeGeneric()` | never (same storage) | You know the dtype matches | -| `AsGeneric()` | never; throws if dtype mismatch | Defensive typing | -| `AsOrMakeGeneric()` | only if dtype differs (then `astype`) | Accept any dtype, convert if needed | +| Method | Allocates? | On dtype mismatch | When to use | +|--------|------------|-------------------|-------------| +| `MakeGeneric()` | never (same storage) | throws `ArgumentException` | You already know the dtype matches | +| `AsGeneric()` | never (same storage) | returns `null` (like C# `as`) | Defensive typing — check for `null` | +| `AsOrMakeGeneric()` | only if dtype differs (then `astype`) | converts via `astype` | Accept any dtype, convert if needed | `NDArray` wraps the same storage; use the untyped `NDArray` when dtype is dynamic. @@ -496,18 +496,33 @@ Three ways to get a typed wrapper: ## Saving, Loading, and Interop -NumSharp reads and writes NumPy's `.npy` / `.npz` formats and raw binary — files saved in Python open in NumSharp, and vice versa. To wrap an existing in-memory byte buffer (file bytes, a network packet, a native pointer) see [`np.frombuffer`](#wrapping-existing-buffers--npfrombuffer) above. +NumSharp reads **and** writes NumPy's `.npy` / `.npz` formats and raw binary. The `.npy`/`.npz` stack is a port of NumPy 2.4.2's format code (NEP-01), so a file written by `np.save` is **byte-for-byte identical** to what NumPy itself writes — data moves between Python and C# in both directions, losslessly. To wrap an existing in-memory byte buffer (file bytes, a network packet, a native pointer) see [`np.frombuffer`](#wrapping-existing-buffers--npfrombuffer) above. ```csharp -// .npy round-trip +// .npy — a single array (byte-identical to NumPy's np.save) np.save("arr.npy", arr); -var loaded = np.load("arr.npy"); // also handles .npz archives +NDArray a = np.load_npy("arr.npy"); // typed load -// Raw binary +// .npz — many arrays in one archive +np.savez("bundle.npz", x, y); // positional → "arr_0", "arr_1" +np.savez("bundle.npz", new Dictionary { ["w"] = w, ["b"] = b }); +np.savez_compressed("bundle.npz", w, b); // same, Deflate-compressed + +using NpzFile npz = np.load_npz("bundle.npz"); // lazy + cached; dispose it (holds the file handle) +NDArray w1 = npz["w"]; // "w.npy" also works as a key +NDArray w2 = npz.f.w; // dot access, like NumPy's npz.f +foreach (string name in npz.Files) { } // "w", "b" — the ".npy" is stripped + +// np.load dispatches on the file's magic bytes and returns `object` +object any = np.load("bundle.npz"); // NDArray for .npy, NpzFile for .npz + +// Raw binary — element bytes only, no header arr.tofile("data.bin"); var raw = np.fromfile("data.bin", np.float64); ``` +Format versions 1.0 / 2.0 / 3.0, C- and Fortran-order, and big-endian files all load; the writer emits the byte-exact, 64-byte-aligned, mmap-ready layout NumPy produces. `np.load` returns `object` because — like NumPy — it yields an array for a `.npy` and an archive for a `.npz`, decided by the file's contents rather than its name; prefer the typed `np.load_npy` / `np.load_npz` when you know the kind. `allow_pickle` defaults to `false` (NumPy's security default), so object-array files are rejected with a clear message instead of executed. See [NumPy Compliance](compliance.md) for the full dtype map and the handful of unsupported types. + Interop with standard .NET arrays: ```csharp @@ -550,7 +565,7 @@ Views can be non-contiguous (sliced, transposed, broadcasted). Use `arr.Shape.Is | `np.array_equal(a, b)` | `bool` | same shape AND all elements equal | | `np.allclose(a, b)` | `bool` | same shape AND all elements within tolerance (good for floats) | | `ReferenceEquals(a, b)` | `bool` | same C# object (rarely what you want) | -| `a.@base != null` | `bool` | `a` is a view (shares memory with some owner) | +| `a.@base is not null` | `bool` | `a` is a view (shares memory with some owner); use `is`, not `!= null` | > Caveat: NumSharp does not expose a direct "do these two arrays share memory?" check from user code. `a.@base` returns a fresh wrapper on every call and the underlying `Storage` is `protected internal`, so strict memory-identity testing is only available inside the assembly. @@ -562,9 +577,9 @@ Views can be non-contiguous (sliced, transposed, broadcasted). Use `arr.Shape.Is That's views. `a["1:3"]` shares memory with `a`. Force a copy: `a["1:3"].copy()`. -### "ReadOnlyArrayException writing to my slice" +### "NumSharpException: assignment destination is read-only" -You're writing to a broadcasted view (stride=0 dimension). Copy first: `b.copy()[...] = value`. +You're writing to a broadcasted view (stride=0 dimension), which is read-only. Copy first: `b.copy()["..."] = value`. ### "ScalarConversionException on `(int)arr`" @@ -576,7 +591,7 @@ C# requires the declaring type on the left of shift operators. Use `np.left_shif ### "a += 1 didn't update another reference" -C# compound assignment reassigns the variable; it doesn't mutate. See [Compound assignment](#compound-assignment) above. For in-place modification, write directly: `a[...] = a + 1`. +C# compound assignment reassigns the variable; it doesn't mutate. See [Compound assignment](#compound-assignment) above. For in-place modification, write directly: `a["..."] = a + 1` (or `a[":"] = a + 1`). --- @@ -617,8 +632,8 @@ C# compound assignment reassigns the variable; it doesn't mutate. See [Compound | `SetAtIndex(value, i)` | Write element at flat index | | `GetValue(indices)` | Read at N-D coordinates | | `SetValue(value, indices)` | Write at N-D coordinates | -| `MakeGeneric()` | Wrap as `NDArray` (same storage) | -| `AsGeneric()` | Wrap as `NDArray`; throws if dtype mismatch | +| `MakeGeneric()` | Wrap as `NDArray` (same storage); throws if dtype differs | +| `AsGeneric()` | Wrap as `NDArray`; returns `null` if dtype differs | | `AsOrMakeGeneric()` | Wrap as `NDArray`; `astype` if dtype differs | | `Data()` | Get the underlying `ArraySlice` handle | | `ToMuliDimArray()` | Copy to a rank-N .NET array | @@ -647,8 +662,12 @@ C# compound assignment reassigns the variable; it doesn't mutate. See [Compound | Call | Format | View / copy | Notes | |------|--------|-------------|-------| -| `np.save(path, arr)` | `.npy` | — | NumPy-compatible; writes header + data | -| `np.load(path)` | `.npy` / `.npz` | — | Also accepts a `Stream` | +| `np.save(path, arr)` | `.npy` | — | Byte-identical to NumPy's `np.save`; `Stream` / `byte[]` overloads | +| `np.savez(path, …)` | `.npz` | — | Many arrays; positional (`arr_0`, `arr_1`, …) or `IDictionary` | +| `np.savez_compressed(path, …)` | `.npz` | — | Same as `savez`, Deflate-compressed | +| `np.load(path)` | `.npy` / `.npz` | — | Returns `object` (`NDArray` or `NpzFile`); `Stream` / `byte[]` overloads | +| `np.load_npy(path)` | `.npy` | copy | Typed → `NDArray` | +| `np.load_npz(path)` | `.npz` | lazy | Typed → `NpzFile` (`IDisposable`; `.Files`, `["w"]` / `.f.w` access) | | `arr.tofile(path)` | raw | — | Element bytes only, no header | | `np.fromfile(path, dtype)` | raw | copy | Pair with `tofile` | | `np.frombuffer(byte[], …)` | in-memory | view (pins array) | Endian-prefix dtype strings trigger a copy | diff --git a/src/NumSharp.Core/APIs/np.load.cs b/src/NumSharp.Core/APIs/np.load.cs index ae07c2bf3..680e6a120 100644 --- a/src/NumSharp.Core/APIs/np.load.cs +++ b/src/NumSharp.Core/APIs/np.load.cs @@ -1,496 +1,350 @@ -using System; -using System.Collections; -using System.Collections.Generic; +using System; using System.IO; -using System.Linq; -using NumSharp.Utilities; +using NumSharp.IO; namespace NumSharp { public static partial class np { - #region NpyFormat - - //Signature from numpy doc: - // numpy.load(file, mmap_mode=None, allow_pickle=True, fix_imports=True, encoding='ASCII')[source] - public static NDArray load(string path) - { - using (var stream = new FileStream(path, FileMode.Open)) - return load(stream); - } - - public static NDArray load(Stream stream) - { - using (var reader = new BinaryReader(stream, System.Text.Encoding.ASCII - , leaveOpen: true - )) + /// The encodings NumPy accepts; anything else can silently corrupt numeric data. + private static readonly string[] ValidPickleEncodings = { "ASCII", "latin1", "bytes" }; + + /// The memory-map modes NumPy accepts. + private static readonly string[] ValidMmapModes = { "r", "r+", "w+", "c" }; + + #region load + + /// + /// Load an array or archive from a .npy or .npz file. + /// + /// + /// Path to the file. The type is detected from its magic bytes, not its extension. + /// + /// + /// Memory-map mode: "r", "r+", "w+" or "c"; null (default) reads + /// normally. Not yet implemented — see . + /// + /// + /// Whether to trust the file. Default false, as in NumPy since 1.16.3: an object array is a + /// Python pickle, and unpickling untrusted data can execute arbitrary code. NumSharp cannot + /// unpickle at all, so this only selects the error message — and, per NumPy, lifts + /// . + /// + /// + /// Present for NumPy parity. Only affects unpickling Python 2 files, which NumSharp does not do. + /// + /// + /// Present for NumPy parity; validated but otherwise unused. Must be "ASCII", + /// "latin1" or "bytes". + /// + /// + /// Reject headers larger than this (default 10000). Guards against a header crafted to make + /// parsing pathologically expensive. Ignored when is true. + /// + /// + /// An for a .npy file, or an for a + /// .npz archive — mirroring NumPy, whose return type also depends on the content. + /// Prefer or + /// when the kind is known: they are typed and need no cast. + /// An owns a file handle and must be disposed. + /// + /// The file is empty. + /// The content is not a .npy or .npz file, or is malformed. + /// + /// + /// var arr = (NDArray)np.load("data.npy"); + /// + /// if (np.load("data.npz") is NpzFile npz) + /// using (npz) + /// { NDArray w = npz["weights"]; } + /// + /// + /// https://numpy.org/doc/stable/reference/generated/numpy.load.html + public static object load(string file, string mmap_mode = null, bool allow_pickle = false, + bool fix_imports = true, string encoding = "ASCII", + long max_header_size = NpyFormat.MaxHeaderSize) + { + if (file == null) throw new ArgumentNullException(nameof(file)); + CheckEncoding(encoding); + CheckMmapMode(mmap_mode); + + // The stream is handed to NpzFile on the .npz branch, which then owns it; otherwise it is + // closed here. + var stream = new FileStream(file, FileMode.Open, FileAccess.Read); + bool transferred = false; + try { - int bytes; - Type type; - int[] shape; - if (!parseReader(reader, out bytes, out type, out shape)) - throw new FormatException(); - - Array array = Arrays.Create(type, shape.Aggregate((dims, dim) => dims * dim)); - - var result = new NDArray(readValueMatrix(reader, array, bytes, type, shape)); - return result.reshape(shape); + object result = LoadCore(stream, ownStream: true, allow_pickle, max_header_size); + transferred = result is NpzFile; + return result; + } + finally + { + if (!transferred) + stream.Dispose(); } } - public static T Load(byte[] bytes) - where T : class, - ICloneable, - IList, ICollection, IEnumerable - , IStructuralComparable, IStructuralEquatable - { - if (typeof(T).IsArray && (typeof(T).GetElementType().IsArray || typeof(T).GetElementType() == typeof(string))) - return LoadJagged(bytes) as T; - return LoadMatrix(bytes) as T; - } - - public static T Load(byte[] bytes, out T value) - where T : class, - ICloneable, - IList, ICollection, IEnumerable - , IStructuralComparable, IStructuralEquatable - { - return value = Load(bytes); - } - - - public static T Load(string path, out T value) - where T : class, - ICloneable, - IList, ICollection, IEnumerable - , IStructuralComparable, IStructuralEquatable - { - return value = Load(path); - } - - public static T Load(Stream stream, out T value) - where T : class, - ICloneable, - IList, ICollection, IEnumerable - , IStructuralComparable, IStructuralEquatable - { - return value = Load(stream); - } - - - public static T Load(string path) - where T : class, - ICloneable, - IList, ICollection, IEnumerable - , IStructuralComparable, IStructuralEquatable - { - using (var stream = new FileStream(path, FileMode.Open)) - return Load(stream); - } - - - public static T Load(Stream stream) - where T : class, - ICloneable, - IList, ICollection, IEnumerable - , IStructuralComparable, IStructuralEquatable - { - if (typeof(T).IsArray && (typeof(T).GetElementType().IsArray || typeof(T).GetElementType() == typeof(string))) - return LoadJagged(stream) as T; - return LoadMatrix(stream) as T; - } - - public static Array LoadMatrix(byte[] bytes) + /// + /// Load an array or archive from an open stream. The stream must be readable and seekable, + /// and is left open — the caller owns it. + /// + /// + /// https://numpy.org/doc/stable/reference/generated/numpy.load.html + public static object load(Stream file, string mmap_mode = null, bool allow_pickle = false, + bool fix_imports = true, string encoding = "ASCII", + long max_header_size = NpyFormat.MaxHeaderSize) { - using (var stream = new MemoryStream(bytes)) - return LoadMatrix(stream); - } + if (file == null) throw new ArgumentNullException(nameof(file)); + CheckEncoding(encoding); + CheckMmapMode(mmap_mode); - - public static Array LoadMatrix(string path) - { - using (var stream = new FileStream(path, FileMode.Open)) - return LoadMatrix(stream); + return LoadCore(file, ownStream: false, allow_pickle, max_header_size); } - - public static Array LoadJagged(byte[] bytes) + /// Load an array or archive from an in-memory .npy/.npz image. + /// + public static object load(byte[] bytes, string mmap_mode = null, bool allow_pickle = false, + bool fix_imports = true, string encoding = "ASCII", + long max_header_size = NpyFormat.MaxHeaderSize) { - using (var stream = new MemoryStream(bytes)) - return LoadJagged(stream); - } - - public static Array LoadJagged(string path) - { - using (var stream = new FileStream(path, FileMode.Open)) - return LoadJagged(stream); - } + if (bytes == null) throw new ArgumentNullException(nameof(bytes)); + CheckEncoding(encoding); + CheckMmapMode(mmap_mode); - public static Array LoadMatrix(Stream stream) - { - using (var reader = new BinaryReader(stream, System.Text.Encoding.ASCII - , leaveOpen: true - )) + var stream = new MemoryStream(bytes, writable: false); + bool transferred = false; + try { - int bytes; - Type type; - int[] shape; - if (!parseReader(reader, out bytes, out type, out shape)) - throw new FormatException(); - - Array matrix = Arrays.Create(type, shape); - - if (type == typeof(String)) - return readStringMatrix(reader, matrix, bytes, type, shape); - return readValueMatrix(reader, matrix, bytes, type, shape); + object result = LoadCore(stream, ownStream: true, allow_pickle, max_header_size); + transferred = result is NpzFile; + return result; } - } - - - public static Array LoadJagged(Stream stream, bool trim = true) - { - using (var reader = new BinaryReader(stream, System.Text.Encoding.ASCII - , leaveOpen: true - )) + finally { - int bytes; - Type type; - int[] shape; - if (!parseReader(reader, out bytes, out type, out shape)) - throw new FormatException(); - - Array matrix = Arrays.Create(type, shape); - - if (type == typeof(String)) - { - Array result = readStringMatrix(reader, matrix, bytes, type, shape); - - //if (trim) - // return result.Trim(); - return result; - } - - return readValueJagged(reader, matrix, bytes, type, shape); + if (!transferred) + stream.Dispose(); } } - private static Array readValueMatrix(BinaryReader reader, Array matrix, int bytes, Type type, int[] shape) - { - long total = 1; - for (int i = 0; i < shape.Length; i++) - total *= shape[i]; - var buffer = new byte[bytes * total]; - - reader.Read(buffer, 0, buffer.Length); - Buffer.BlockCopy(buffer, 0, matrix, 0, buffer.Length); - - return matrix; - } + #endregion - static IEnumerable GetIndices(Array array, int[] current, int pos) - { - if (pos == current.Length) - yield return current; - else - { - for (int i = 0; i < array.GetLength(pos); i++) - { - current[pos] = i; - foreach (var ind in GetIndices(array, current, pos + 1)) - yield return ind; - current[pos] = 0; - } - } - } + #region load_npy - static IEnumerable GetIndices(Array array) + /// + /// Load a single array from a .npy file — + /// without the cast. + /// + /// Path to a .npy file. + /// Whether the file is trusted; see . + /// Reject headers larger than this. + /// The file is not a .npy file, or is malformed. + public static NDArray load_npy(string file, bool allow_pickle = false, long max_header_size = NpyFormat.MaxHeaderSize) { - return GetIndices(array, new int[array.Rank], 0); - } + if (file == null) throw new ArgumentNullException(nameof(file)); - private static Array readValueJagged(BinaryReader reader, Array matrix, int bytes, Type type, int[] shape) - { - int last = shape[shape.Length - 1]; - byte[] buffer = new byte[bytes * last]; - - int[] firsts = new int[shape.Length - 1]; - for (int i = 0; i < firsts.Length; i++) - firsts[i] = -1; - - foreach (var p in GetIndices(matrix)) - { - bool changed = false; - for (int i = 0; i < firsts.Length; i++) - { - if (firsts[i] != p[i]) - { - firsts[i] = p[i]; - changed = true; - } - } - - if (!changed) - continue; - - Array arr = (Array)matrix.GetValue(indices: firsts); - - reader.Read(buffer, 0, buffer.Length); - Buffer.BlockCopy(buffer, 0, arr, 0, buffer.Length); - } - - return matrix; + using (var stream = new FileStream(file, FileMode.Open, FileAccess.Read)) + return LoadNpyChecked(stream, file, allow_pickle, max_header_size); } - private static Array readStringMatrix(BinaryReader reader, Array matrix, int bytes, Type type, int[] shape) + /// + /// Read one array from a stream positioned at a .npy magic string. The stream is left + /// just past that array's data, so successive calls read successively saved arrays. + /// + /// An open, readable stream. + /// Whether the file is trusted. + /// Reject headers larger than this. + /// The stream is already at its end. + public static NDArray load_npy(Stream file, bool allow_pickle = false, long max_header_size = NpyFormat.MaxHeaderSize) { - var buffer = new byte[bytes]; - - unsafe - { - fixed (byte* b = buffer) - { - foreach (var p in GetIndices(matrix)) - { - reader.Read(buffer, 0, bytes); - if (buffer[0] == byte.MinValue) - { - bool isNull = true; - for (int i = 1; i < buffer.Length; i++) - { - if (buffer[i] != byte.MaxValue) - { - isNull = false; - break; - } - } - - if (isNull) - { - matrix.SetValue(value: null, indices: p); - continue; - } - } - - String s = new String((sbyte*)b); - matrix.SetValue(value: s, indices: p); - } - } - } + if (file == null) throw new ArgumentNullException(nameof(file)); - return matrix; + return LoadNpyChecked(file, "stream", allow_pickle, max_header_size); } - private static bool parseReader(BinaryReader reader, out int bytes, out Type t, out int[] shape) + /// Load a single array from an in-memory .npy image. + /// + public static NDArray load_npy(byte[] bytes, bool allow_pickle = false, long max_header_size = NpyFormat.MaxHeaderSize) { - bytes = 0; - t = null; - shape = null; - - // The first 6 bytes are a magic string: exactly "x93NUMPY" - if (reader.ReadChar() != 63) return false; - if (reader.ReadChar() != 'N') return false; - if (reader.ReadChar() != 'U') return false; - if (reader.ReadChar() != 'M') return false; - if (reader.ReadChar() != 'P') return false; - if (reader.ReadChar() != 'Y') return false; - - byte major = reader.ReadByte(); // 1 - byte minor = reader.ReadByte(); // 0 - - if (major != 1 || minor != 0) - throw new NotSupportedException(); - - ushort len = reader.ReadUInt16(); - - string header = new String(reader.ReadChars(len)); - string mark = "'descr': '"; - int s = header.IndexOf(mark) + mark.Length; - int e = header.IndexOf("'", s + 1); - string type = header.Substring(s, e - s); - bool? isLittleEndian; - t = GetType(type, out bytes, out isLittleEndian); - - if (isLittleEndian.HasValue && isLittleEndian.Value == false) - throw new Exception(); - - mark = "'fortran_order': "; - s = header.IndexOf(mark) + mark.Length; - e = header.IndexOf(",", s + 1); - bool fortran = bool.Parse(header.Substring(s, e - s)); - - if (fortran) - throw new Exception(); - - mark = "'shape': ("; - s = header.IndexOf(mark) + mark.Length; - e = header.IndexOf(")", s + 1); - shape = header.Substring(s, e - s).Split(',').Where(v => !String.IsNullOrEmpty(v)).Select(Int32.Parse).ToArray(); - - return true; - } + if (bytes == null) throw new ArgumentNullException(nameof(bytes)); - private static Type GetType(string dtype, out int bytes, out bool? isLittleEndian) - { - isLittleEndian = IsLittleEndian(dtype); - bytes = Int32.Parse(dtype.Substring(2)); - - string typeCode = dtype.Substring(1); - - if (typeCode == "b1") - return typeof(bool); - if (typeCode == "i1") - return typeof(Byte); - if (typeCode == "i2") - return typeof(Int16); - if (typeCode == "i4") - return typeof(Int32); - if (typeCode == "i8") - return typeof(Int64); - if (typeCode == "u1") - return typeof(Byte); - if (typeCode == "u2") - return typeof(UInt16); - if (typeCode == "u4") - return typeof(UInt32); - if (typeCode == "u8") - return typeof(UInt64); - if (typeCode == "f4") - return typeof(Single); - if (typeCode == "f8") - return typeof(Double); - if (typeCode.StartsWith("S")) - return typeof(String); - - throw new NotSupportedException(); + using (var stream = new MemoryStream(bytes, writable: false)) + return LoadNpyChecked(stream, "bytes", allow_pickle, max_header_size); } - private static bool? IsLittleEndian(string type) + // Report a .npz handed to a .npy reader by name rather than letting the magic check produce + // "the magic string is not correct", which would not tell the caller what to do instead. + private static NDArray LoadNpyChecked(Stream stream, string what, bool allowPickle, long maxHeaderSize) { - bool? littleEndian = null; - - switch (type[0]) + if (stream.CanSeek) { - case '<': - littleEndian = true; - break; - case '>': - littleEndian = false; - break; - case '|': - littleEndian = null; - break; - default: - throw new Exception(); + if (stream.Position >= stream.Length) + throw new EndOfStreamException("No data left in file"); + if (NpyFormat.IsNpzFile(stream)) + throw new FormatException($"'{what}' is a .npz archive, not a .npy file. Use np.load_npz to open it."); } - return littleEndian; + return NpyFormat.ReadArray(stream, allowPickle, maxHeaderSize); } #endregion - #region NpzFormat - - public static void Load_Npz(byte[] bytes, out T value) - where T : class, - ICloneable, - IList, ICollection, IEnumerable, IStructuralComparable, IStructuralEquatable - { - using (var dict = Load_Npz(bytes)) + #region load_npz + + /// + /// Open a .npz archive — + /// without the cast. + /// + /// Path to a .npz archive. + /// Whether members are trusted. + /// Reject member headers larger than this. + /// + /// A lazily-loading, dictionary-like archive. It holds an open file handle — dispose it: + /// using var npz = np.load_npz("m.npz"); + /// + /// The file is not a ZIP archive. + public static NpzFile load_npz(string file, bool allow_pickle = false, long max_header_size = NpyFormat.MaxHeaderSize) + { + if (file == null) throw new ArgumentNullException(nameof(file)); + + var stream = new FileStream(file, FileMode.Open, FileAccess.Read); + try { - value = dict.Values.First(); + CheckIsNpz(stream, file); + return new NpzFile(stream, ownStream: true, allow_pickle, max_header_size); } - } - - public static void Load_Npz(string path, out T value) - where T : class, - ICloneable, - IList, ICollection, IEnumerable, IStructuralComparable, IStructuralEquatable - { - using (var dict = Load_Npz(path)) - { - value = dict.Values.First(); - } - } - - public static void Load_Npz(Stream stream, out T value) - where T : class, - ICloneable, - IList, ICollection, IEnumerable, IStructuralComparable, IStructuralEquatable - { - using (var dict = Load_Npz(stream)) + catch { - value = dict.Values.First(); + stream.Dispose(); + throw; } } - public static NpzDictionary Load_Npz(byte[] bytes) - where T : class, - ICloneable, - IList, ICollection, IEnumerable, IStructuralComparable, IStructuralEquatable - { - return Load_Npz(new MemoryStream(bytes)); - } - - public static NpzDictionary Load_Npz(string path, out NpzDictionary value) - where T : class, - ICloneable, - IList, ICollection, IEnumerable, IStructuralComparable, IStructuralEquatable - { - return value = Load_Npz(new FileStream(path, FileMode.Open)); - } - - public static NpzDictionary Load_Npz(Stream stream, out NpzDictionary value) - where T : class, - ICloneable, - IList, ICollection, IEnumerable, IStructuralComparable, IStructuralEquatable - { - return value = Load_Npz(stream); - } - - public static NpzDictionary Load_Npz(string path) - where T : class, - ICloneable, - IList, ICollection, IEnumerable, IStructuralComparable, IStructuralEquatable - { - return Load_Npz(new FileStream(path, FileMode.Open)); - } - - public static NpzDictionary Load_Npz(Stream stream) - where T : class, - ICloneable, - IList, ICollection, IEnumerable, IStructuralComparable, IStructuralEquatable + /// + /// Open a .npz archive over a stream. + /// + /// A readable, seekable stream. + /// When true, disposing the archive also disposes the stream. + /// Whether members are trusted. + /// Reject member headers larger than this. + public static NpzFile load_npz(Stream file, bool own_stream = false, bool allow_pickle = false, + long max_header_size = NpyFormat.MaxHeaderSize) { - return new NpzDictionary(stream); - } + if (file == null) throw new ArgumentNullException(nameof(file)); - public static NpzDictionary LoadMatrix_Npz(byte[] bytes) - { - return LoadMatrix_Npz(new MemoryStream(bytes)); + CheckIsNpz(file, "stream"); + return new NpzFile(file, own_stream, allow_pickle, max_header_size); } - public static NpzDictionary LoadMatrix_Npz(string path) + /// Open a .npz archive from an in-memory image. + /// + public static NpzFile load_npz(byte[] bytes, bool allow_pickle = false, long max_header_size = NpyFormat.MaxHeaderSize) { - return LoadMatrix_Npz(new FileStream(path, FileMode.Open)); - } + if (bytes == null) throw new ArgumentNullException(nameof(bytes)); - public static NpzDictionary LoadMatrix_Npz(Stream stream) - { - return new NpzDictionary(stream, jagged: false); + var stream = new MemoryStream(bytes, writable: false); + try + { + CheckIsNpz(stream, "bytes"); + return new NpzFile(stream, ownStream: true, allow_pickle, max_header_size); + } + catch + { + stream.Dispose(); + throw; + } } - public static NpzDictionary LoadJagged_Npz(byte[] bytes) + private static void CheckIsNpz(Stream stream, string what) { - return LoadJagged_Npz(new MemoryStream(bytes)); + if (stream.CanSeek && stream.Position >= stream.Length) + throw new EndOfStreamException("No data left in file"); + if (!NpyFormat.IsNpzFile(stream)) + throw new FormatException( + NpyFormat.IsNpyFile(stream) + ? $"'{what}' is a .npy file, not a .npz archive. Use np.load_npy to load it." + : $"'{what}' is not a .npz archive (no ZIP signature)."); } - public static NpzDictionary LoadJagged_Npz(string path) - { - return LoadJagged_Npz(new FileStream(path, FileMode.Open)); - } + #endregion - public static NpzDictionary LoadJagged_Npz(Stream stream, bool trim = true) - { - return new NpzDictionary(stream, jagged: true); + #region internals + + // File-type detection by magic, in NumPy's order: ZIP, then NPY, then "must be a pickle". + private static object LoadCore(Stream stream, bool ownStream, bool allowPickle, long maxHeaderSize) + { + if (!stream.CanSeek) + throw new ArgumentException( + "Stream must be seekable so the file type can be detected from its magic bytes.", nameof(stream)); + + if (stream.Position >= stream.Length) + throw new EndOfStreamException("No data left in file"); + + if (NpyFormat.IsNpzFile(stream)) + return new NpzFile(stream, ownStream, allowPickle, maxHeaderSize); + + if (NpyFormat.IsNpyFile(stream)) + return NpyFormat.ReadArray(stream, allowPickle, maxHeaderSize); + + // Neither magic matched. NumPy assumes a bare pickle here and tries to unpickle it; + // NumSharp has no pickle reader, so both branches end in an error — but the allow_pickle=False + // one is NumPy's verbatim message, which is what a user porting code will recognize. + if (!allowPickle) + throw new FormatException( + "This file contains pickled (object) data. If you trust the file you can load it unsafely " + + "using the `allow_pickle=` keyword argument or `pickle.load()`."); + + throw new NotSupportedException( + "This file is not a .npy or .npz file. NumPy would try to unpickle it, which NumSharp cannot do: " + + "the Python pickle protocol is not implemented. Re-save the data from NumPy with np.save."); + } + + private static void CheckEncoding(string encoding) + { + if (Array.IndexOf(ValidPickleEncodings, encoding) < 0) + throw new ArgumentException("encoding must be 'ASCII', 'latin1', or 'bytes'", nameof(encoding)); + } + + /// + /// Validate mmap_mode and reject it as unimplemented. + /// + /// + /// Implementing this means backing an NDArray with a mapped view instead of owned + /// unmanaged memory. Sketch, if someone picks it up: + /// + /// Read the header via and keep the data + /// offset — it is 64-byte aligned precisely so the view can start there. + /// Map with MemoryMappedFile.CreateFromFile and take a + /// MemoryMappedViewAccessor at that offset. Note the view offset must be a + /// multiple of the system allocation granularity (64 KB on Windows), so map from 0 + /// and index past the header rather than mapping at the data offset. + /// Wrap the view's pointer in an UnmanagedMemoryBlock whose free-callback + /// releases the view and the map, so the NDArray keeps the mapping alive exactly as + /// long as it lives. This is the load-bearing part: today + /// UnmanagedMemoryBlock assumes it owns its allocation. + /// Honor the mode: "r" must produce a non-writeable Shape; "c" needs + /// MemoryMappedFileAccess.CopyOnWrite; "r+" and "w+" write + /// through, and "w+" must pre-size the file and write a header first. + /// Fortran-order files need the reshape-and-transpose from + /// applied to the mapped buffer. + /// Byte-swapped (big-endian) files cannot be mapped at all — NumSharp converts on + /// read, which a zero-copy view rules out. Reject them explicitly. + /// + /// + /// The mode is not one of NumPy's. + /// A valid mode was requested. + private static void CheckMmapMode(string mmap_mode) + { + if (mmap_mode == null) + return; + + if (Array.IndexOf(ValidMmapModes, mmap_mode) < 0) + throw new ArgumentException( + $"mode must be one of {{'r+', 'r', 'w+', 'c'}} (got '{mmap_mode}')", nameof(mmap_mode)); + + throw new NotImplementedException( + $"mmap_mode='{mmap_mode}' is not implemented yet — NumSharp arrays own their unmanaged memory and " + + "cannot yet be backed by a mapped file view. Load without mmap_mode to read the file normally."); } #endregion diff --git a/src/NumSharp.Core/APIs/np.save.cs b/src/NumSharp.Core/APIs/np.save.cs index c4c249b64..b082e08c8 100644 --- a/src/NumSharp.Core/APIs/np.save.cs +++ b/src/NumSharp.Core/APIs/np.save.cs @@ -1,319 +1,284 @@ -using System; +using System; using System.Collections.Generic; using System.IO; using System.IO.Compression; -using System.Linq; -using System.Runtime.InteropServices; +using NumSharp.IO; namespace NumSharp { public static partial class np { - #region NpyFormat - - public static void save(string filepath, Array arr) - { - Save(arr, filepath); - } - - public static void save(string filepath, NDArray arr) + #region save (.npy) + + /// + /// Save an array to a .npy binary file. + /// + /// + /// Target path. .npy is appended if the name does not already end with it, matching + /// NumPy. + /// + /// The array to save. Any layout; a Fortran-contiguous array is stored as such. + /// + /// Present for NumPy parity. NumSharp has no object dtype, so nothing can reach the pickle + /// path and this never changes the outcome. + /// + /// + /// The dtype has no NumPy equivalent — . + /// + /// + /// The file is byte-for-byte what NumPy 2.4.2's own np.save writes for the same array. + /// https://numpy.org/doc/stable/reference/generated/numpy.save.html + /// + public static void save(string file, NDArray arr, bool allow_pickle = true) { - Save((Array)arr, filepath); - } - - public static byte[] Save(Array array) - { - using (var stream = new MemoryStream()) - { - Save(array, stream); - return stream.ToArray(); - } - } - + if (file == null) throw new ArgumentNullException(nameof(file)); + if (arr is null) throw new ArgumentNullException(nameof(arr)); - public static ulong Save(Array array, string path) - { - if (Path.GetExtension(path) != ".npy") - { - path += ".npy"; - } + if (!file.EndsWith(".npy", StringComparison.Ordinal)) + file += ".npy"; - using (var stream = new FileStream(path, FileMode.Create)) - return Save(array, stream); + using (var stream = new FileStream(file, FileMode.Create, FileAccess.Write)) + NpyFormat.WriteArray(stream, arr, null, allow_pickle); } - public static ulong Save(Array array, Stream stream) + /// + /// Save an array to a .npy file, converting with + /// first. + /// + /// https://numpy.org/doc/stable/reference/generated/numpy.save.html + public static void save(string file, Array arr, bool allow_pickle = true) + => save(file, asanyarray(arr), allow_pickle); + + /// + /// Write an array in .npy format to an open stream. + /// + /// + /// An open, writable stream. Written from its current position and left open, so successive + /// calls append — several arrays can share one file and be read back in order by + /// . + /// + /// The array to save. + /// Present for NumPy parity; see . + /// https://numpy.org/doc/stable/reference/generated/numpy.save.html + public static void save(Stream file, NDArray arr, bool allow_pickle = true) { - using (var writer = new BinaryWriter(stream - , System.Text.Encoding.ASCII, leaveOpen: true - )) - { - Type type; - int maxLength; - string dtype = GetDtypeFromType(array, out type, out maxLength); + if (file == null) throw new ArgumentNullException(nameof(file)); + if (arr is null) throw new ArgumentNullException(nameof(arr)); - int[] shape = Enumerable.Range(0, array.Rank).Select(d => array.GetLength(d)).ToArray(); - - ulong bytesWritten = (ulong)writeHeader(writer, dtype, shape); - - if (array.GetType().GetElementType().IsArray || array.GetType().GetElementType() == typeof(string)) - { - if (type == typeof(String)) - return bytesWritten + writeStringMatrix(writer, array, maxLength, shape); - return bytesWritten + writeValueJagged(writer, array, maxLength, shape); - } - else - { - if (type == typeof(String)) - return bytesWritten + writeStringMatrix(writer, array, maxLength, shape); - return bytesWritten + writeValueMatrix(writer, array, maxLength, shape); - } - } + NpyFormat.WriteArray(file, arr, null, allow_pickle); } - private static ulong writeValueMatrix(BinaryWriter reader, Array matrix, int bytes, int[] shape) + /// + /// Encode an array as .npy and return the bytes. + /// + /// A NumSharp convenience; NumPy has no in-memory equivalent. + public static byte[] save(NDArray arr, bool allow_pickle = true) { - long total = 1; - for (int i = 0; i < shape.Length; i++) - total *= shape[i]; - var buffer = new byte[bytes * total]; - - Buffer.BlockCopy(matrix, 0, buffer, 0, buffer.Length); - reader.Write(buffer, 0, buffer.Length); + if (arr is null) throw new ArgumentNullException(nameof(arr)); - return (ulong)buffer.LongLength; - } - - static IEnumerable Enumerate(Array a, int[] dimensions, int pos) - { - if (pos == dimensions.Length - 1) - { - for (int i = 0; i < dimensions[pos]; i++) - yield return (T)a.GetValue(i); - } - else + using (var stream = new MemoryStream()) { - for (int i = 0; i < dimensions[pos]; i++) - foreach (var subArray in Enumerate(a.GetValue(i) as Array, dimensions, pos + 1)) - yield return subArray; + NpyFormat.WriteArray(stream, arr, null, allow_pickle); + return stream.ToArray(); } } - private static ulong writeValueJagged(BinaryWriter reader, Array matrix, int bytes, int[] shape) + /// + /// Write an array in .npy format using an explicit format version — NumPy's + /// numpy.lib.format.write_array. + /// + /// An open, writable stream. + /// The array to save. + /// + /// (1,0), (2,0), (3,0), or null to use the oldest that can hold the header. Version 2.0 + /// widens the header-length field to 4 bytes; 3.0 also switches the header to UTF-8. + /// + /// Present for NumPy parity; see . + /// The version is not one of (1,0), (2,0) or (3,0). + public static void save_version(Stream file, NDArray arr, NpyFormat.FormatVersion? version, bool allow_pickle = true) { - int last = shape[shape.Length - 1]; - byte[] buffer = new byte[bytes * last]; - int[] first = shape.Take(shape.Length - 1).ToArray(); - - ulong writtenBytes = 0; - foreach (Array arr in Enumerate(matrix, first, 0)) - { - Array.Clear(buffer, arr.Length, buffer.Length - buffer.Length); - Buffer.BlockCopy(arr, 0, buffer, 0, buffer.Length); - reader.Write(buffer, 0, buffer.Length); - writtenBytes += (ulong)buffer.LongLength; - } + if (file == null) throw new ArgumentNullException(nameof(file)); + if (arr is null) throw new ArgumentNullException(nameof(arr)); - return writtenBytes; + NpyFormat.WriteArray(file, arr, version, allow_pickle); } - private static ulong writeStringMatrix(BinaryWriter reader, Array matrix, int bytes, int[] shape) - { - var buffer = new byte[bytes]; - var empty = new byte[bytes]; - empty[0] = byte.MinValue; - for (int i = 1; i < empty.Length; i++) - empty[i] = byte.MaxValue; - - ulong writtenBytes = 0; - - unsafe - { - fixed (byte* b = buffer) - { - foreach (String s in Enumerate(matrix, shape, 0)) - { - if (s != null) - { - int c = 0; - for (int i = 0; i < s.Length; i++) - b[c++] = (byte)s[i]; - for (; c < buffer.Length; c++) - b[c] = byte.MinValue; - - reader.Write(buffer, 0, bytes); - } - else - { - reader.Write(empty, 0, bytes); - } - - writtenBytes += (ulong)buffer.LongLength; - } - } - } - - return writtenBytes; - } - - private static int writeHeader(BinaryWriter writer, string dtype, int[] shape) - { - // The first 6 bytes are a magic string: exactly "x93NUMPY" - - char[] magic = {'N', 'U', 'M', 'P', 'Y'}; - writer.Write((byte)147); - writer.Write(magic); - writer.Write((byte)1); // major - writer.Write((byte)0); // minor; - - string tuple = String.Join(", ", shape.Select(i => i.ToString()).ToArray()); - if (shape.Length == 1) - tuple += ","; // 1-dim array's shape is (R,) - string header = "{{'descr': '{0}', 'fortran_order': False, 'shape': ({1}), }}"; - header = String.Format(header, dtype, tuple); - int preamble = 10; // magic string (6) + 4 - - int len = header.Length + 1; // the 1 is to account for the missing \n at the end - int headerSize = len + preamble; + #endregion - int pad = 16 - (headerSize % 16); - header = header.PadRight(header.Length + pad); - header += "\n"; - headerSize = header.Length + preamble; + #region savez (.npz) + + /// + /// Save several arrays into an uncompressed .npz archive, named arr_0, + /// arr_1, … in order. + /// + /// Target path. .npz is appended if not already present. + /// The arrays, in order. + /// https://numpy.org/doc/stable/reference/generated/numpy.savez.html + public static void savez(string file, params NDArray[] args) + => savez(file, args, null); + + /// + /// Save named arrays into an uncompressed .npz archive — NumPy's keyword form, + /// np.savez(file, weights=w, biases=b). + /// + /// Target path. .npz is appended if not already present. + /// Name/array pairs. Each becomes <name>.npy in the archive. + /// https://numpy.org/doc/stable/reference/generated/numpy.savez.html + public static void savez(string file, IDictionary kwds) + => savez(file, null, kwds); + + /// + /// Save positional and named arrays into an uncompressed .npz archive. + /// + /// Target path. .npz is appended if not already present. + /// Positional arrays, stored as arr_0, arr_1, … + /// Named arrays. + /// A name in collides with a generated arr_N. + /// https://numpy.org/doc/stable/reference/generated/numpy.savez.html + public static void savez(string file, NDArray[] args, IDictionary kwds) + => SaveZip(file, args, kwds, CompressionLevel.NoCompression); + + /// Write an uncompressed .npz archive to an open stream. + /// https://numpy.org/doc/stable/reference/generated/numpy.savez.html + public static void savez(Stream file, params NDArray[] args) + => SaveZip(file, args, null, CompressionLevel.NoCompression); + + /// Write an uncompressed .npz archive of named arrays to an open stream. + /// https://numpy.org/doc/stable/reference/generated/numpy.savez.html + public static void savez(Stream file, IDictionary kwds) + => SaveZip(file, null, kwds, CompressionLevel.NoCompression); + + /// Write an uncompressed .npz archive of positional and named arrays to an open stream. + /// https://numpy.org/doc/stable/reference/generated/numpy.savez.html + public static void savez(Stream file, NDArray[] args, IDictionary kwds) + => SaveZip(file, args, kwds, CompressionLevel.NoCompression); + + /// Encode an uncompressed .npz archive of arr_0… arrays and return the bytes. + /// A NumSharp convenience; NumPy has no in-memory equivalent. + public static byte[] savez(params NDArray[] args) + => SaveZipBytes(args, null, CompressionLevel.NoCompression); + + /// Encode an uncompressed .npz archive of named arrays and return the bytes. + /// A NumSharp convenience; NumPy has no in-memory equivalent. + public static byte[] savez(IDictionary kwds) + => SaveZipBytes(null, kwds, CompressionLevel.NoCompression); - if (headerSize % 16 != 0) - throw new Exception(); + #endregion - writer.Write((ushort)header.Length); - for (int i = 0; i < header.Length; i++) - writer.Write((byte)header[i]); + #region savez_compressed (.npz, deflated) + + /// + /// Save several arrays into a compressed .npz archive, named arr_0, + /// arr_1, … in order. + /// + /// Target path. .npz is appended if not already present. + /// The arrays, in order. + /// https://numpy.org/doc/stable/reference/generated/numpy.savez_compressed.html + public static void savez_compressed(string file, params NDArray[] args) + => savez_compressed(file, args, null); + + /// Save named arrays into a compressed .npz archive. + /// https://numpy.org/doc/stable/reference/generated/numpy.savez_compressed.html + public static void savez_compressed(string file, IDictionary kwds) + => savez_compressed(file, null, kwds); + + /// Save positional and named arrays into a compressed .npz archive. + /// https://numpy.org/doc/stable/reference/generated/numpy.savez_compressed.html + public static void savez_compressed(string file, NDArray[] args, IDictionary kwds) + => SaveZip(file, args, kwds, CompressionLevel.Optimal); + + /// Write a compressed .npz archive to an open stream. + /// https://numpy.org/doc/stable/reference/generated/numpy.savez_compressed.html + public static void savez_compressed(Stream file, params NDArray[] args) + => SaveZip(file, args, null, CompressionLevel.Optimal); + + /// Write a compressed .npz archive of named arrays to an open stream. + /// https://numpy.org/doc/stable/reference/generated/numpy.savez_compressed.html + public static void savez_compressed(Stream file, IDictionary kwds) + => SaveZip(file, null, kwds, CompressionLevel.Optimal); + + /// Write a compressed .npz archive of positional and named arrays to an open stream. + /// https://numpy.org/doc/stable/reference/generated/numpy.savez_compressed.html + public static void savez_compressed(Stream file, NDArray[] args, IDictionary kwds) + => SaveZip(file, args, kwds, CompressionLevel.Optimal); + + /// Encode a compressed .npz archive of arr_0… arrays and return the bytes. + /// A NumSharp convenience; NumPy has no in-memory equivalent. + public static byte[] savez_compressed(params NDArray[] args) + => SaveZipBytes(args, null, CompressionLevel.Optimal); + + /// Encode a compressed .npz archive of named arrays and return the bytes. + /// A NumSharp convenience; NumPy has no in-memory equivalent. + public static byte[] savez_compressed(IDictionary kwds) + => SaveZipBytes(null, kwds, CompressionLevel.Optimal); - return headerSize; - } + #endregion - static Type GetInnerMostType(Type arrayType) - { - if (arrayType.GetElementType().IsArray) - return GetInnerMostType(arrayType.GetElementType()); - return arrayType.GetElementType(); - } + #region npz internals - private static string GetDtypeFromType(Array array, out Type type, out int bytes) + private static void SaveZip(string file, NDArray[] args, IDictionary kwds, CompressionLevel level) { - type = GetInnerMostType(array.GetType()); + if (file == null) throw new ArgumentNullException(nameof(file)); - bytes = 1; + if (!file.EndsWith(".npz", StringComparison.Ordinal)) + file += ".npz"; - if (type == typeof(String)) - { - int[] shape = Enumerable.Range(0, array.Rank).Select(d => array.GetLength(d)).ToArray(); - foreach (String s in Enumerate(array, shape, 0)) - { - if (s.Length > bytes) - bytes = s.Length; - } - } - else if (type == typeof(bool)) - { - bytes = 1; - } - else - { - bytes = System.Runtime.InteropServices.Marshal.SizeOf(type); - } - - if (type == typeof(bool)) - return "|b1"; - if (type == typeof(Byte)) - return "|i1"; - if (type == typeof(Int16)) - return " arrays, CompressionLevel compression = DEFAULT_COMPRESSION) + private static byte[] SaveZipBytes(NDArray[] args, IDictionary kwds, CompressionLevel level) { using (var stream = new MemoryStream()) { - Save_Npz(arrays, stream, compression, leaveOpen: true); + SaveZip(stream, args, kwds, level); return stream.ToArray(); } } - public static byte[] Save_Npz(Array array, CompressionLevel compression = DEFAULT_COMPRESSION) + private static void SaveZip(Stream stream, NDArray[] args, IDictionary kwds, CompressionLevel level) { - using (var stream = new MemoryStream()) - { - Save_Npz(array, stream, compression, leaveOpen: true); - return stream.ToArray(); - } - } + if (stream == null) throw new ArgumentNullException(nameof(stream)); + // NumPy's _savez: keyword arrays keep their names and come first, then positional arrays + // take arr_0, arr_1, …; a positional name that collides with a keyword is an error rather + // than a silent overwrite. + var namedict = new List>(); + var seen = new HashSet(StringComparer.Ordinal); - public static void Save_Npz(Dictionary arrays, string path, CompressionLevel compression = DEFAULT_COMPRESSION) - { - using (var stream = new FileStream(path, FileMode.Create)) + if (kwds != null) { - Save_Npz(arrays, stream, compression); - } - } - - public static void Save_Npz(Array array, string path, CompressionLevel compression = DEFAULT_COMPRESSION) - { - using (var stream = new FileStream(path, FileMode.Create)) - { - Save_Npz(array, stream, compression); + foreach (KeyValuePair kv in kwds) + { + if (kv.Value is null) throw new ArgumentNullException(nameof(kwds), $"Array '{kv.Key}' is null."); + namedict.Add(kv); + seen.Add(kv.Key); + } } - } - public static void Save_Npz(Dictionary arrays, Stream stream, CompressionLevel compression = DEFAULT_COMPRESSION, bool leaveOpen = false) - { - using (var zip = new ZipArchive(stream, ZipArchiveMode.Create, leaveOpen: leaveOpen)) + if (args != null) { - foreach (KeyValuePair p in arrays) + for (int i = 0; i < args.Length; i++) { - var entry = zip.CreateEntry(p.Key, compression); - using (Stream s = entry.Open()) - { - Save(p.Value, s); - } + string key = $"arr_{i}"; + if (seen.Contains(key)) + throw new ArgumentException($"Cannot use un-named variables and keyword {key}", nameof(kwds)); + if (args[i] is null) throw new ArgumentNullException(nameof(args), $"Array at index {i} is null."); + namedict.Add(new KeyValuePair(key, args[i])); + seen.Add(key); } } - } - public static void Save_Npz(Array array, Stream stream, CompressionLevel compression = DEFAULT_COMPRESSION, bool leaveOpen = false) - { - using (var zip = new ZipArchive(stream, ZipArchiveMode.Create, leaveOpen: leaveOpen)) + // ZipArchive writes Zip64 records as needed, so archives above 4 GB work — NumPy passes + // force_zip64=True for the same reason (numpy gh-10776). + using (var zip = new ZipArchive(stream, ZipArchiveMode.Create, leaveOpen: true)) { - var entry = zip.CreateEntry("arr_0"); - using (Stream s = entry.Open()) + foreach (KeyValuePair kv in namedict) { - Save(array, s); + ZipArchiveEntry entry = zip.CreateEntry(kv.Key + ".npy", level); + using (Stream s = entry.Open()) + NpyFormat.WriteArray(s, kv.Value); } - } } diff --git a/src/NumSharp.Core/IO/NpyFormat.cs b/src/NumSharp.Core/IO/NpyFormat.cs new file mode 100644 index 000000000..b51c49ca0 --- /dev/null +++ b/src/NumSharp.Core/IO/NpyFormat.cs @@ -0,0 +1,1054 @@ +using System; +using System.Buffers.Binary; +using System.Collections.Generic; +using System.Globalization; +using System.IO; +using System.Linq; +using System.Runtime.InteropServices; +using System.Text; + +namespace NumSharp.IO +{ + /// + /// The NumPy .npy binary format (NEP-01) — reader and writer for versions 1.0, 2.0 and 3.0. + /// + /// + /// A port of numpy/lib/_format_impl.py (NumPy 2.4.2), structured to mirror it function for + /// function so the two can be diffed. Byte-for-byte identical output is a tested contract: see + /// test/oracle/gen_npy_oracle.py, which has real NumPy write every case this class claims + /// to support, and NpyOracleTests, which replays them. + /// + /// File layout: + /// + /// \x93NUMPY 6 bytes magic (\x93 is byte 147, NOT the character '?') + /// major, minor 2 bytes format version + /// HEADER_LEN 2 bytes uint16 little-endian (v1.0) + /// 4 bytes uint32 little-endian (v2.0 / v3.0) + /// header HEADER_LEN bytes — a Python dict literal, space-padded, '\n'-terminated, + /// sized so the data starts on a 64-byte (ARRAY_ALIGN) boundary + /// data shape-product * itemsize raw bytes, C- or Fortran-order + /// + /// + /// Divergences from NumPy, all forced by NumSharp's type system and each covered by a test: + /// + /// Big-endian files are byte-swapped to native on read. NumPy instead keeps the raw bytes + /// behind a byte-swapped dtype; NumSharp has no such dtype, so it converts. Values match; + /// the byte-order attribute is not preserved. + /// Char (2-byte UTF-16) maps to NumPy's '<U1' (4-byte UCS-4), converting in + /// both directions. '<U' widths above 1 have no NumSharp analog. + /// '<c8' (complex64) widens to on read; + /// writing always emits '<c16'. + /// Decimal has no NumPy dtype and cannot be written. + /// Object arrays, structured dtypes, datetime64/timedelta64, byte strings and void are + /// parsed but rejected with a precise message (see the issue's "What We're Not Doing"). + /// + /// + public static class NpyFormat + { + #region Constants + + /// The magic string every .npy file starts with: \x93NUMPY. + public static ReadOnlySpan MagicPrefix => new byte[] { 0x93, (byte)'N', (byte)'U', (byte)'M', (byte)'P', (byte)'Y' }; + + /// Length of the magic string plus the two version bytes. + public const int MagicLen = 8; + + /// + /// The header is padded so the data begins on a multiple of this. 64 bytes lets the data be + /// memory-mapped and read with aligned SIMD loads. + /// + public const int ArrayAlign = 64; + + /// Chunk size for streamed reads, matching NumPy's BUFFER_SIZE (256 KB). + public const int BufferSize = 1 << 18; + + /// + /// Spare spaces reserved after the dict so the shape can be rewritten in place when a file is + /// grown along its first (C-order) or last (F-order) axis. len(str(8 * 2**64 - 1)). + /// + public const int GrowthAxisMaxDigits = 21; + + /// + /// Default cap on header size. Parsing an arbitrarily large header is a denial-of-service + /// vector, and no legitimate array needs more than this. + /// + public const int MaxHeaderSize = 10000; + + /// The only keys a valid header may contain. + private static readonly string[] ExpectedKeys = { "descr", "fortran_order", "shape" }; + + /// The first bytes of a ZIP archive — a .npz. PK\x05\x06 marks an empty one. + private static ReadOnlySpan ZipPrefix => new byte[] { 0x50, 0x4B, 0x03, 0x04 }; + private static ReadOnlySpan ZipSuffix => new byte[] { 0x50, 0x4B, 0x05, 0x06 }; + + // latin1 that THROWS instead of silently substituting '?', so header encoding can detect + // characters that need version 3.0 — mirroring Python's UnicodeEncodeError. + private static readonly Encoding StrictLatin1 = + Encoding.GetEncoding("ISO-8859-1", EncoderFallback.ExceptionFallback, DecoderFallback.ExceptionFallback); + + private static readonly Encoding Latin1 = Encoding.Latin1; + private static readonly Encoding Utf8 = new UTF8Encoding(false); + + #endregion + + #region Format version + + /// A .npy format version. Only (1,0), (2,0) and (3,0) exist. + public readonly struct FormatVersion : IEquatable + { + public readonly byte Major; + public readonly byte Minor; + + public FormatVersion(byte major, byte minor) + { + Major = major; + Minor = minor; + } + + /// Version 1.0 — 2-byte header length, latin-1. The default and most portable. + public static FormatVersion V1_0 => new FormatVersion(1, 0); + + /// Version 2.0 (NumPy 1.9) — 4-byte header length, for headers above 64 KB. + public static FormatVersion V2_0 => new FormatVersion(2, 0); + + /// Version 3.0 (NumPy 1.17) — 4-byte header length, UTF-8, for Unicode field names. + public static FormatVersion V3_0 => new FormatVersion(3, 0); + + /// Size of the header-length field: 2 bytes for v1.0, 4 for v2.0 and v3.0. + public int HeaderLengthSize => Major >= 2 ? 4 : 2; + + /// The header's text encoding: UTF-8 from v3.0, latin-1 before it. + public Encoding HeaderEncoding => Major >= 3 ? Utf8 : Latin1; + + /// The encoding used when writing, which throws rather than substituting characters. + internal Encoding StrictHeaderEncoding => Major >= 3 ? Utf8 : StrictLatin1; + + /// Largest value the header-length field can hold. + public uint MaxHeaderLength => Major >= 2 ? uint.MaxValue : ushort.MaxValue; + + public bool IsSupported => + (Major == 1 || Major == 2 || Major == 3) && Minor == 0; + + public bool Equals(FormatVersion other) => Major == other.Major && Minor == other.Minor; + public override bool Equals(object obj) => obj is FormatVersion v && Equals(v); + public override int GetHashCode() => (Major << 8) | Minor; + + /// Formatted as Python's tuple repr, e.g. (1, 0), so it can go verbatim into messages. + public override string ToString() => $"({Major}, {Minor})"; + + public static bool operator ==(FormatVersion l, FormatVersion r) => l.Equals(r); + public static bool operator !=(FormatVersion l, FormatVersion r) => !l.Equals(r); + public static bool operator <=(FormatVersion l, FormatVersion r) => + l.Major < r.Major || (l.Major == r.Major && l.Minor <= r.Minor); + public static bool operator >=(FormatVersion l, FormatVersion r) => + l.Major > r.Major || (l.Major == r.Major && l.Minor >= r.Minor); + } + + #endregion + + #region dtype descriptors + + /// + /// How a file's elements must be transformed to become NumSharp elements. Everything except + /// exists because NumSharp's type has a different width than NumPy's. + /// + internal enum ElementConversion + { + /// File bytes are NumSharp bytes (after any byte-swap). + None, + + /// NumPy 4-byte UCS-4 code point ↔ NumSharp 2-byte UTF-16 . + Ucs4, + + /// NumPy 'c8' (two float32) → (two float64). + Complex64, + } + + /// A parsed descr: what the file holds, and what NumSharp will hold. + internal readonly struct DtypeInfo + { + /// The dtype NumSharp materializes. + public readonly NPTypeCode TypeCode; + + /// Bytes per element as stored in the FILE (may differ from NumSharp's item size). + public readonly int FileItemSize; + + /// + /// Size of one byte-swappable unit within an element. An element is not always one unit: + /// complex swaps per component, so '>c16' is 2 units of 8 bytes, not 1 of 16. + /// + public readonly int SwapUnit; + + /// The file's byte order. Native for '|' and '='. + public readonly bool LittleEndian; + + public readonly ElementConversion Conversion; + + /// True for '|O', whose data is a Python pickle rather than raw elements. + public readonly bool HasObject; + + public DtypeInfo(NPTypeCode typeCode, int fileItemSize, int swapUnit, bool littleEndian, + ElementConversion conversion = ElementConversion.None, bool hasObject = false) + { + TypeCode = typeCode; + FileItemSize = fileItemSize; + SwapUnit = swapUnit; + LittleEndian = littleEndian; + Conversion = conversion; + HasObject = hasObject; + } + + /// True when the file's byte order differs from this machine's. + public bool NeedsSwap => LittleEndian != BitConverter.IsLittleEndian && SwapUnit > 1; + } + + /// + /// The descr string for a NumSharp dtype — the inverse of + /// and the equivalent of NumPy's dtype_to_descr for the types NumSharp has. + /// + /// + /// Single-byte types take the '|' (not-applicable) prefix, everything else '<': + /// NumSharp is native-order internally and every platform it runs on is little-endian. + /// + /// The dtype has no NumPy equivalent. + public static string DtypeToDescr(NPTypeCode typeCode) + { + switch (typeCode) + { + case NPTypeCode.Boolean: return "|b1"; + case NPTypeCode.SByte: return "|i1"; + case NPTypeCode.Byte: return "|u1"; + case NPTypeCode.Int16: return " + /// Turn a header's descr into the dtype NumSharp will materialize — NumPy's + /// descr_to_dtype. + /// + /// The parsed value of the header's descr key. + /// The descriptor is not a valid NumPy dtype. + /// A valid NumPy dtype NumSharp cannot represent. + internal static DtypeInfo DescrToDtype(object descr) + { + // A list of field tuples is a structured (record) dtype; a tuple is a subarray dtype. Both + // are valid NumPy and both are out of scope, so report them precisely rather than as + // "invalid descriptor". + if (descr is List) + throw new NotSupportedException( + $"Structured dtypes are not supported by NumSharp: {PyLiteral.Repr(descr)}"); + if (descr is PyTuple) + throw new NotSupportedException( + $"Subarray dtypes are not supported by NumSharp: {PyLiteral.Repr(descr)}"); + + if (!(descr is string s) || s.Length == 0) + throw new FormatException($"descr is not a valid dtype descriptor: {PyLiteral.Repr(descr)}"); + + // [extra] + int i = 0; + bool littleEndian; + switch (s[0]) + { + case '<': littleEndian = true; i = 1; break; + case '>': littleEndian = false; i = 1; break; + case '|': + case '=': littleEndian = BitConverter.IsLittleEndian; i = 1; break; + default: littleEndian = BitConverter.IsLittleEndian; break; // no prefix: native + } + + if (i >= s.Length) + throw new FormatException($"descr is not a valid dtype descriptor: {PyLiteral.Repr(descr)}"); + + char kind = s[i]; + string rest = s.Substring(i + 1); + + // Object: parses fine; read_array decides based on allow_pickle. + if (kind == 'O') + return new DtypeInfo(NPTypeCode.Empty, 0, 1, littleEndian, ElementConversion.None, hasObject: true); + + // datetime64/timedelta64 carry a unit suffix, e.g. 'c16' is two 8-byte units, not one 16-byte unit. + case 'c' when size == 8: return new DtypeInfo(NPTypeCode.Complex, 8, 4, littleEndian, ElementConversion.Complex64); + case 'c' when size == 16: return new DtypeInfo(NPTypeCode.Complex, 16, 8, littleEndian); + case 'c' when size == 32: + throw new NotSupportedException( + "complex256 ('" + s + "') is not supported — .NET has no 256-bit complex. " + + "Convert to complex128 in NumPy before saving."); + default: + throw new FormatException($"descr is not a valid dtype descriptor: {PyLiteral.Repr(descr)}"); + } + } + + #endregion + + #region Magic and version + + /// The 8 magic bytes for a version — NumPy's magic(). + public static byte[] Magic(FormatVersion version) + { + var magic = new byte[MagicLen]; + MagicPrefix.CopyTo(magic); + magic[6] = version.Major; + magic[7] = version.Minor; + return magic; + } + + /// + /// Read and validate the magic string, returning the file's format version — NumPy's + /// read_magic(). Leaves the stream on the header-length field. + /// + /// The magic string is wrong or the stream ends inside it. + public static FormatVersion ReadMagic(Stream stream) + { + byte[] magic = ReadBytes(stream, MagicLen, "magic string"); + if (!magic.AsSpan(0, MagicPrefix.Length).SequenceEqual(MagicPrefix)) + throw new FormatException( + "the magic string is not correct; expected " + PyBytesRepr(MagicPrefix) + + ", got " + PyBytesRepr(magic.AsSpan(0, MagicPrefix.Length))); + + return new FormatVersion(magic[6], magic[7]); + } + + /// Reject versions this implementation does not know — NumPy's _check_version(). + /// The version is not (1,0), (2,0) or (3,0). + public static void CheckVersion(FormatVersion version) + { + if (!version.IsSupported) + throw new FormatException( + $"we only support format version (1,0), (2,0), and (3,0), not {version}"); + } + + /// Render bytes the way Python renders a bytes repr, for verbatim error messages. + private static string PyBytesRepr(ReadOnlySpan bytes) + { + var sb = new StringBuilder("b'"); + foreach (byte b in bytes) + { + if (b == (byte)'\\') sb.Append("\\\\"); + else if (b == (byte)'\'') sb.Append("\\'"); + else if (b >= 0x20 && b < 0x7F) sb.Append((char)b); + else sb.Append("\\x").Append(b.ToString("x2", CultureInfo.InvariantCulture)); + } + return sb.Append('\'').ToString(); + } + + #endregion + + #region Header — writing + + /// + /// Build the header dict for an array — NumPy's header_data_from_array_1_0(). + /// + /// + /// C-contiguity is tested first because a 1-D, 0-d or empty array is BOTH C- and + /// F-contiguous, and those must come out as fortran_order: False. An array that is + /// neither is copied to C-order at write time, so it too reports False. + /// + public static Dictionary HeaderDataFromArray(NDArray array) + { + bool fortranOrder = !array.Shape.IsContiguous && array.Shape.IsFContiguous; + + return new Dictionary(StringComparer.Ordinal) + { + ["descr"] = DtypeToDescr(array.typecode), + ["fortran_order"] = fortranOrder, + ["shape"] = array.shape, + }; + } + + // The dict as a Python literal, keys sorted — NumPy: "For repeatability and readability, the + // dictionary keys are sorted in alphabetic order. A writer SHOULD implement this if possible. + // A reader MUST NOT depend on this." (Alphabetical order happens to be descr, fortran_order, + // shape.) Values go through repr(), which is why a 1-tuple keeps its trailing comma: (3,). + private static string FormatHeaderDict(Dictionary d) + { + var sb = new StringBuilder("{"); + foreach (string key in d.Keys.OrderBy(k => k, StringComparer.Ordinal)) + { + sb.Append('\'').Append(key).Append("': "); + AppendRepr(sb, d[key]); + sb.Append(", "); // NumPy emits a trailing comma after every entry, including the last + } + return sb.Append('}').ToString(); + } + + private static void AppendRepr(StringBuilder sb, object value) + { + switch (value) + { + case string s: + sb.Append('\'').Append(s).Append('\''); + return; + case bool b: + sb.Append(b ? "True" : "False"); + return; + case long[] shape: + AppendShapeTuple(sb, shape); + return; + case int[] ishape: + AppendShapeTuple(sb, Array.ConvertAll(ishape, x => (long)x)); + return; + default: + sb.Append(Convert.ToString(value, CultureInfo.InvariantCulture)); + return; + } + } + + private static void AppendShapeTuple(StringBuilder sb, long[] shape) + { + sb.Append('('); + for (int i = 0; i < shape.Length; i++) + { + if (i > 0) sb.Append(", "); + sb.Append(shape[i].ToString(CultureInfo.InvariantCulture)); + } + if (shape.Length == 1) sb.Append(','); // Python's 1-tuple repr: (3,) — never (3) + sb.Append(')'); + } + + // Attach magic + length field + padding — NumPy's _wrap_header(). + private static byte[] WrapHeader(string header, FormatVersion version) + { + byte[] headerBytes; + try + { + headerBytes = version.StrictHeaderEncoding.GetBytes(header); + } + catch (EncoderFallbackException) + { + // Not encodable in latin-1 -> the caller falls through to version 3.0 (UTF-8), the same + // way NumPy catches UnicodeEncodeError. + throw new UnicodeEncodeException(); + } + + int hlen = headerBytes.Length + 1; // + the terminating newline + int prefixSize = MagicLen + version.HeaderLengthSize; + + // Pad so magic + length + header lands on an ARRAY_ALIGN boundary. Note this is never zero: + // an already-aligned header gets a FULL 64 bytes. NumPy does exactly this, and matching it + // is what makes the output byte-identical. + int padlen = ArrayAlign - ((prefixSize + hlen) % ArrayAlign); + long totalHeaderLen = (long)hlen + padlen; + + if (totalHeaderLen > version.MaxHeaderLength) + throw new InvalidOperationException($"Header length {hlen} too big for version={version}"); + + var result = new byte[prefixSize + totalHeaderLen]; + int pos = 0; + + MagicPrefix.CopyTo(result.AsSpan(pos)); + pos += MagicPrefix.Length; + result[pos++] = version.Major; + result[pos++] = version.Minor; + + if (version.HeaderLengthSize == 4) + { + BinaryPrimitives.WriteUInt32LittleEndian(result.AsSpan(pos), (uint)totalHeaderLen); + pos += 4; + } + else + { + BinaryPrimitives.WriteUInt16LittleEndian(result.AsSpan(pos), (ushort)totalHeaderLen); + pos += 2; + } + + headerBytes.CopyTo(result, pos); + pos += headerBytes.Length; + + result.AsSpan(pos, padlen).Fill((byte)' '); + pos += padlen; + result[pos] = (byte)'\n'; + + return result; + } + + /// Signals a header that latin-1 cannot encode, mirroring Python's UnicodeEncodeError. + private sealed class UnicodeEncodeException : Exception { } + + // Pick the oldest version that can hold this header — NumPy's _wrap_header_guess_version(). + private static byte[] WrapHeaderGuessVersion(string header) + { + try + { + return WrapHeader(header, FormatVersion.V1_0); // most compatible + } + catch (InvalidOperationException) + { + // Header exceeds the 2-byte length field -> needs v2.0. + } + catch (UnicodeEncodeException) + { + // Not latin-1 encodable -> v2.0 would fail the same way; fall through to v3.0. + return WrapHeader(header, FormatVersion.V3_0); + } + + try + { + return WrapHeader(header, FormatVersion.V2_0); + } + catch (UnicodeEncodeException) + { + return WrapHeader(header, FormatVersion.V3_0); // UTF-8 + } + } + + /// + /// Write an array header — NumPy's _write_array_header(). Leaves the stream exactly at + /// the (64-byte aligned) start of the data. + /// + /// Target stream. + /// Header dict: descr, fortran_order and shape. + /// The version to write, or null to pick the oldest that fits. + public static void WriteArrayHeader(Stream stream, Dictionary d, FormatVersion? version = null) + { + string header = FormatHeaderDict(d); + + // Reserve room to rewrite the growth axis in place when the file is later extended: the + // first axis in C-order, the last in F-order. A 0-d array has no axis to grow. + long[] shape = ShapeOf(d); + if (shape.Length > 0) + { + bool fortranOrder = d.TryGetValue("fortran_order", out object fo) && fo is bool b && b; + long growthAxisValue = shape[fortranOrder ? shape.Length - 1 : 0]; + int padding = GrowthAxisMaxDigits - growthAxisValue.ToString(CultureInfo.InvariantCulture).Length; + if (padding > 0) + header += new string(' ', padding); + } + + byte[] wrapped; + if (version.HasValue) + { + CheckVersion(version.Value); + try + { + wrapped = WrapHeader(header, version.Value); + } + catch (UnicodeEncodeException) + { + throw new InvalidOperationException( + $"Header cannot be encoded in latin-1 for version={version.Value}; use version (3, 0)."); + } + } + else + { + wrapped = WrapHeaderGuessVersion(header); + } + + stream.Write(wrapped, 0, wrapped.Length); + } + + private static long[] ShapeOf(Dictionary d) + { + if (!d.TryGetValue("shape", out object s)) + throw new ArgumentException("Header dict is missing the 'shape' key.", nameof(d)); + switch (s) + { + case long[] l: return l; + case int[] i: return Array.ConvertAll(i, x => (long)x); + default: throw new ArgumentException($"Header 'shape' must be long[] or int[], got {s?.GetType().Name ?? "null"}.", nameof(d)); + } + } + + #endregion + + #region Header — reading + + /// A validated header: everything needed to materialize the array that follows. + internal readonly struct HeaderData + { + public readonly long[] Shape; + public readonly bool FortranOrder; + public readonly DtypeInfo Dtype; + + public HeaderData(long[] shape, bool fortranOrder, DtypeInfo dtype) + { + Shape = shape; + FortranOrder = fortranOrder; + Dtype = dtype; + } + + /// Element count. A 0-d array holds exactly one element, not zero. + public long Count + { + get + { + if (Shape.Length == 0) return 1; + long count = 1; + foreach (long d in Shape) + count = checked(count * d); + return count; + } + } + } + + /// + /// Read and validate a header — NumPy's _read_array_header(). Leaves the stream at the + /// start of the data. + /// + /// A stream positioned just past the magic and version bytes. + /// The version read from the magic. + /// + /// Reject headers longer than this. The default guards against a header crafted to make + /// parsing pathologically expensive; raise it only for files you trust. + /// + internal static HeaderData ReadArrayHeader(Stream stream, FormatVersion version, long maxHeaderSize = MaxHeaderSize) + { + CheckVersion(version); + + byte[] lenBytes = ReadBytes(stream, version.HeaderLengthSize, "array header length"); + uint headerLength = version.HeaderLengthSize == 2 + ? BinaryPrimitives.ReadUInt16LittleEndian(lenBytes) + : BinaryPrimitives.ReadUInt32LittleEndian(lenBytes); + + byte[] headerBytes = ReadBytes(stream, headerLength, "array header"); + string header = version.HeaderEncoding.GetString(headerBytes); + + if (header.Length > maxHeaderSize) + throw new FormatException( + $"Header info length ({header.Length}) is large and may not be safe to load securely.\n" + + "To allow loading, adjust `maxHeaderSize` or fully trust the `.npy` file using `allow_pickle=True`.\n" + + "For safety against large resource use or crashes, sandboxing may be necessary."); + + object parsed; + try + { + // The Python 2 'L' integer suffix only ever appeared in v1.0/v2.0 files; NumPy's + // _filter_header retry is likewise gated on version <= (2, 0). + parsed = PyLiteral.Parse(header, allowLongSuffix: version <= FormatVersion.V2_0); + } + catch (PyLiteral.SyntaxException e) + { + throw new FormatException($"Cannot parse header: {PyLiteral.Repr(header)}", e); + } + + if (!(parsed is Dictionary d)) + throw new FormatException($"Header is not a dictionary: {PyLiteral.Repr(parsed)}"); + + if (d.Count != ExpectedKeys.Length || !ExpectedKeys.All(d.ContainsKey)) + { + string keys = PyLiteral.Repr(d.Keys.OrderBy(k => k, StringComparer.Ordinal).Cast().ToList()); + throw new FormatException($"Header does not contain the correct keys: {keys}"); + } + + if (!(d["shape"] is PyTuple shapeTuple) || !shapeTuple.All(x => x is long)) + throw new FormatException($"shape is not valid: {PyLiteral.Repr(d["shape"])}"); + + if (!(d["fortran_order"] is bool fortranOrder)) + throw new FormatException($"fortran_order is not a valid bool: {PyLiteral.Repr(d["fortran_order"])}"); + + var shape = new long[shapeTuple.Count]; + for (int i = 0; i < shapeTuple.Count; i++) + shape[i] = (long)shapeTuple[i]; + + return new HeaderData(shape, fortranOrder, DescrToDtype(d["descr"])); + } + + #endregion + + #region Writing + + /// + /// Write an array with its header — NumPy's write_array(). + /// + /// An open, writable stream. Written from its current position. + /// The array to write. Any layout; non-contiguous is materialized C-order. + /// The format version, or null (default) for the oldest that fits. + /// + /// Present for NumPy parity. NumSharp has no object dtype, so no array can reach the pickle + /// path and this parameter never changes the outcome. + /// + /// The dtype has no NumPy equivalent (Decimal). + public static void WriteArray(Stream stream, NDArray array, FormatVersion? version = null, bool allowPickle = true) + { + if (stream == null) throw new ArgumentNullException(nameof(stream)); + if (array is null) throw new ArgumentNullException(nameof(array)); + if (version.HasValue) CheckVersion(version.Value); + + WriteArrayHeader(stream, HeaderDataFromArray(array), version); + + // NumPy writes an F-contiguous array as its transpose in C-order — the transpose of an + // F-contiguous array is C-contiguous, so this streams the same bytes the file needs. + bool fortran = !array.Shape.IsContiguous && array.Shape.IsFContiguous; + WriteArrayData(stream, fortran ? array.T : array); + } + + private static void WriteArrayData(Stream stream, NDArray array) + { + if (array.size == 0) + return; + + if (array.typecode == NPTypeCode.Char) + { + WriteUcs4Data(stream, array); + return; + } + + // Only an offset-0 contiguous array can stream straight from its buffer; anything else + // (strided / broadcast / sliced) is materialized C-order first, which is what NumPy's + // nditer copy does. Matches NDArray.tofile's WriteBinary. + NDArray src = array.Shape.IsContiguous && array.Shape.offset == 0 ? array : array.copy('C'); + + unsafe + { + long len = checked(src.size * src.dtypesize); + using (var ums = new UnmanagedMemoryStream((byte*)src.Storage.Address, len)) + ums.CopyTo(stream, (int)Math.Min(len, BufferSize)); + } + + GC.KeepAlive(src); + } + + // NumSharp's Char is one 2-byte UTF-16 code unit; NumPy's ' 0) + stream.Write(buffer, 0, filled); + + GC.KeepAlive(src); + } + + #endregion + + #region Reading + + /// + /// Read an array — NumPy's read_array(). + /// + /// A stream positioned at the magic string. + /// + /// Whether the caller trusts this file. NumSharp cannot unpickle either way, but this + /// selects which error an object array produces and, per NumPy, lifts + /// . + /// + /// Reject headers larger than this. Ignored when . + public static NDArray ReadArray(Stream stream, bool allowPickle = false, long maxHeaderSize = MaxHeaderSize) + { + if (stream == null) throw new ArgumentNullException(nameof(stream)); + + // allow_pickle means the caller has declared the file trusted, so the header guard — which + // exists only to bound the cost of parsing hostile input — is moot. + if (allowPickle) + maxHeaderSize = long.MaxValue; + + FormatVersion version = ReadMagic(stream); + CheckVersion(version); + + HeaderData header = ReadArrayHeader(stream, version, maxHeaderSize); + + if (header.Dtype.HasObject) + { + if (!allowPickle) + throw new FormatException("Object arrays cannot be loaded when allow_pickle=False"); + throw new NotSupportedException( + "Object arrays are stored as a Python pickle stream, which NumSharp cannot read. " + + "Re-save the array from NumPy with a concrete numeric dtype."); + } + + foreach (long dim in header.Shape) + if (dim < 0) + throw new FormatException("negative dimensions are not allowed"); + + long count = header.Count; + var result = new NDArray(header.Dtype.TypeCode, new Shape(header.Shape)); + + if (count > 0) + ReadArrayData(stream, result, header.Dtype, count); + + // The file holds F-order data: read it flat, give it the reversed shape, then transpose — + // NumPy's `array.reshape(shape[::-1]).transpose()`. + if (header.FortranOrder && header.Shape.Length > 1) + { + var reversed = new long[header.Shape.Length]; + for (int i = 0; i < header.Shape.Length; i++) + reversed[i] = header.Shape[header.Shape.Length - 1 - i]; + result = result.reshape(reversed).T; + } + + return result; + } + + private static unsafe void ReadArrayData(Stream stream, NDArray array, DtypeInfo dtype, long count) + { + long fileBytes = checked(count * dtype.FileItemSize); + if (fileBytes == 0) + return; + + // Read in BUFFER_SIZE chunks: a ZipExtFile or network stream can return short reads, and + // NumPy notes crc32 breaks past 2**32 bytes on gzip streams. Chunking also caps peak memory. + // The chunk is a whole number of elements so no element ever straddles two chunks. + int chunkElements = Math.Max(1, BufferSize / dtype.FileItemSize); + int chunkBytes = chunkElements * dtype.FileItemSize; + var buffer = new byte[(int)Math.Min(fileBytes, chunkBytes)]; + + byte* dst = (byte*)array.Storage.Address; + long written = 0; + long remaining = fileBytes; + + while (remaining > 0) + { + int toRead = (int)Math.Min(remaining, buffer.Length); + int got = ReadInto(stream, buffer, toRead); + if (got != toRead) + throw new FormatException($"EOF: reading array data, expected {fileBytes} bytes got {fileBytes - remaining + got}"); + + if (dtype.NeedsSwap) + SwapBytes(buffer, got, dtype.SwapUnit); + + written += ConvertChunk(buffer, got, dst + written, dtype); + remaining -= got; + } + + GC.KeepAlive(array); + } + + // Copy one chunk of file bytes into the array, converting where NumPy's element width differs + // from NumSharp's. Returns the number of destination bytes written. + private static unsafe long ConvertChunk(byte[] buffer, int length, byte* dst, DtypeInfo dtype) + { + switch (dtype.Conversion) + { + case ElementConversion.None: + Marshal.Copy(buffer, 0, (IntPtr)dst, length); + return length; + + case ElementConversion.Ucs4: + { + // 4-byte UCS-4 code point -> 2-byte UTF-16 code unit. Anything outside the BMP needs + // a surrogate pair and cannot fit a single Char. + int n = length / 4; + char* c = (char*)dst; + for (int i = 0; i < n; i++) + { + uint cp = BinaryPrimitives.ReadUInt32LittleEndian(buffer.AsSpan(i * 4)); + if (cp > 0xFFFF) + throw new NotSupportedException( + $"'= 0xD800 && cp <= 0xDFFF) + throw new FormatException($"' two float64 (System.Numerics.Complex is real-then-imaginary, the + // same order NumPy stores). + int n = length / 4; // float32 components, not elements + double* d = (double*)dst; + for (int i = 0; i < n; i++) + d[i] = BitConverter.Int32BitsToSingle(BinaryPrimitives.ReadInt32LittleEndian(buffer.AsSpan(i * 4))); + return (long)n * 8; + } + + default: + throw new NotSupportedException($"Unknown element conversion {dtype.Conversion}."); + } + } + + // Reverse each `unit`-sized group in place. The unit is a component, not an element: '>c16' is + // two 8-byte doubles, so it swaps in 8s. + private static void SwapBytes(byte[] buffer, int length, int unit) + { + for (int i = 0; i + unit <= length; i += unit) + buffer.AsSpan(i, unit).Reverse(); + } + + #endregion + + #region Stream helpers + + /// + /// Read exactly bytes — NumPy's _read_bytes(). + /// + /// + /// is attacker-controlled — it is the length field straight out of + /// the file, so a 28-byte file can claim a 4 GB header — and is therefore never handed to the + /// allocator. A seekable stream states exactly how many bytes remain, which is a hard upper + /// bound on what any claim can deliver; otherwise the buffer starts small and doubles as the + /// bytes actually arrive. Python's fp.read(n) has the same property, which is why + /// NumPy rejects such a file in ~2 KB instead of reserving the claim. The message still + /// reports the CLAIMED size, exactly as NumPy's does. + /// + /// The stream ended early. + private static byte[] ReadBytes(Stream stream, long count, string what) + { + if (count == 0) + return Array.Empty(); + + // What could plausibly be delivered — never more than what is really left. + long plausible = count; + if (stream.CanSeek) + plausible = Math.Min(plausible, Math.Max(0L, stream.Length - stream.Position)); + + // A well-formed header (~118 bytes) lands here exactly sized, in one allocation. + var buffer = new byte[Math.Clamp(plausible, 1, BufferSize)]; + long total = 0; + + while (total < count) + { + if (total == buffer.Length) + { + // Only grow once the bytes we already hold prove the claim is being honored. + long next = Math.Min(count, Math.Min(buffer.Length * 2L, int.MaxValue)); + if (next == buffer.Length) + break; + Array.Resize(ref buffer, (int)next); + } + + int got = stream.Read(buffer, (int)total, (int)Math.Min(count - total, buffer.Length - total)); + if (got == 0) + break; // EOF — Stream.Read only returns 0 at the end + total += got; + } + + if (total != count) + throw new FormatException($"EOF: reading {what}, expected {count} bytes got {total}"); + + if (buffer.Length != total) + Array.Resize(ref buffer, (int)total); + return buffer; + } + + // Loop until the buffer is full or the stream ends: Stream.Read may legally return fewer bytes + // than asked for, and ZipExtFile routinely does. + private static int ReadInto(Stream stream, byte[] buffer, int count) + { + int total = 0; + while (total < count) + { + int read = stream.Read(buffer, total, count - total); + if (read == 0) break; // EOF + total += read; + } + return total; + } + + /// Whether the stream begins with the .npy magic. The position is restored. + public static bool IsNpyFile(Stream stream) => StartsWith(stream, MagicPrefix); + + /// Whether the stream begins with a ZIP signature — a .npz. The position is restored. + public static bool IsNpzFile(Stream stream) => StartsWith(stream, ZipPrefix) || StartsWith(stream, ZipSuffix); + + private static bool StartsWith(Stream stream, ReadOnlySpan prefix) + { + if (stream == null) throw new ArgumentNullException(nameof(stream)); + if (!stream.CanSeek) throw new ArgumentException("Stream must be seekable to detect its type.", nameof(stream)); + + long pos = stream.Position; + try + { + Span head = stackalloc byte[8]; + head = head.Slice(0, prefix.Length); + int got = 0; + while (got < prefix.Length) + { + int read = stream.Read(head.Slice(got)); + if (read == 0) break; + got += read; + } + return got == prefix.Length && head.SequenceEqual(prefix); + } + finally + { + stream.Position = pos; + } + } + + #endregion + } +} diff --git a/src/NumSharp.Core/IO/NpzFile.cs b/src/NumSharp.Core/IO/NpzFile.cs new file mode 100644 index 000000000..6ffb68835 --- /dev/null +++ b/src/NumSharp.Core/IO/NpzFile.cs @@ -0,0 +1,369 @@ +using System; +using System.Collections; +using System.Collections.Generic; +using System.Dynamic; +using System.IO; +using System.IO.Compression; +using System.Linq; + +namespace NumSharp.IO +{ + /// + /// A lazily-loaded .npz archive — NumPy's NpzFile. + /// + /// + /// A .npz is a ZIP archive of .npy members. Nothing is decoded until a key is + /// accessed, and each array is cached from then on, so a huge archive costs only what is read + /// out of it. + /// + /// Keys work with or without the .npy suffix — npz["weights"] and + /// npz["weights.npy"] are the same member — while reports the stripped + /// names, matching NumPy. + /// + /// The archive holds an open file handle, so dispose it: + /// + /// using var npz = np.load_npz("model.npz"); + /// NDArray w = npz["weights"]; + /// NDArray b = npz.f.biases; // dot access, NumPy's BagObj + /// + /// + public sealed class NpzFile : IReadOnlyDictionary, IDisposable + { + /// How many keys lists before eliding — NumPy's _MAX_REPR_ARRAY_COUNT. + private const int MaxReprArrayCount = 5; + + private readonly bool _ownStream; + private readonly string _name; + + /// + /// Maps every accepted key — stripped AND suffixed — to its zip entry. + /// + /// + /// Holds the rather than its name because a zip may contain + /// duplicate names, and the two runtimes disagree on which one wins: + /// returns the FIRST match, while Python's zipfile builds a + /// name→info dict as it scans, so the LAST wins. Building this map the same way — later + /// entries overwrite earlier ones — reproduces NumPy's choice. + /// + private readonly Dictionary _keyToEntry; + + /// Cache keyed by entry identity, so duplicate names cannot alias each other. + private readonly Dictionary _cache; + + private readonly List _files; + + private Stream _stream; + private ZipArchive _archive; + private bool _disposed; + + /// + /// Open an archive over a stream. + /// + /// A readable, seekable stream holding the ZIP archive. + /// When true, disposing this also disposes . + /// Whether members are trusted; see . + /// Per-member header size cap. + public NpzFile(Stream stream, bool ownStream = false, bool allowPickle = false, + long maxHeaderSize = NpyFormat.MaxHeaderSize) + { + _stream = stream ?? throw new ArgumentNullException(nameof(stream)); + _ownStream = ownStream; + AllowPickle = allowPickle; + MaxHeaderSize = maxHeaderSize; + _name = (stream as FileStream)?.Name ?? "object"; + + try + { + _archive = new ZipArchive(stream, ZipArchiveMode.Read, leaveOpen: true); + } + catch + { + if (ownStream) stream.Dispose(); + throw; + } + + _keyToEntry = new Dictionary(StringComparer.Ordinal); + _cache = new Dictionary(); + _files = new List(_archive.Entries.Count); + + // Mirrors NumPy's NpzFile.__init__ exactly: + // self.files = [name.removesuffix(".npy") for name in _files] + // self._files = dict(zip(self.files, _files)) # pass 1: stripped -> entry + // self._files.update(zip(_files, _files)) # pass 2: full -> entry + // + // The TWO passes are load-bearing, not a stylistic detail. A zip may hold both 'a' and + // 'a.npy', and both strip to the key 'a'; pass 2 then re-points 'a' at the entry literally + // NAMED 'a'. A single interleaved loop resolves 'a' to whichever came last instead — + // for entries ['a', 'a.npy'] NumPy answers with the first, one loop answers with the second. + // Within each pass later entries overwrite earlier ones, because a zip may legally repeat a + // name and Python's dict keeps the last. + foreach (ZipArchiveEntry entry in _archive.Entries) + { + string key = StripNpy(entry.FullName); + + // Files lists every entry, duplicates included — it is a plain list built from + // namelist(), so a repeated name appears twice here while the lookup keeps only one. + _files.Add(key); + _keyToEntry[key] = entry; + } + + foreach (ZipArchiveEntry entry in _archive.Entries) + _keyToEntry[entry.FullName] = entry; + + F = new BagObj(this); + } + + /// Open an archive from a file path. The file handle is owned and closed on dispose. + public NpzFile(string path, bool allowPickle = false, long maxHeaderSize = NpyFormat.MaxHeaderSize) + : this(new FileStream(path, FileMode.Open, FileAccess.Read), ownStream: true, allowPickle, maxHeaderSize) + { + _name = path; + } + + /// The array names in the archive, with .npy stripped — NumPy's .files. + public IReadOnlyList Files => _files; + + /// Whether members are loaded as trusted input. + public bool AllowPickle { get; } + + /// The per-member header size cap. + public long MaxHeaderSize { get; } + + /// + /// Dot-notation access to members — NumPy's npz.f.weights. Requires a dynamic + /// receiver: NDArray w = npz.f.weights;. + /// + public dynamic F { get; private set; } + + /// Lower-case alias of , spelled as NumPy spells it. + public dynamic f => F; + + /// The underlying archive, for callers that need entry metadata. + public ZipArchive Zip + { + get { ThrowIfDisposed(); return _archive; } + } + + /// Number of members. + public int Count => _files.Count; + + /// The member names — same as . + public IEnumerable Keys => _files; + + /// Every member's array. Enumerating this loads and caches all of them. + public IEnumerable Values + { + get + { + foreach (string key in _files) + yield return this[key]; + } + } + + /// + /// The array stored under , with or without the .npy suffix. + /// Loaded on first access and cached. + /// + /// No such member. + /// The member is not a .npy file — use . + public NDArray this[string key] + { + get + { + ThrowIfDisposed(); + + if (!_keyToEntry.TryGetValue(key, out ZipArchiveEntry entry)) + throw new KeyNotFoundException($"{key} is not a file in the archive"); + + if (_cache.TryGetValue(entry, out NDArray cached)) + return cached; + + // NumPy checks the magic and hands back raw bytes for anything that is not a .npy. + // NumSharp's indexer is typed, so route those to GetRawBytes instead of widening every + // access to object. + if (!StartsWithNpyMagic(entry)) + throw new FormatException( + $"'{entry.FullName}' is not a .npy member (its magic string is missing), so it has no " + + $"array to return. Use GetRawBytes(\"{key}\") to read it as bytes."); + + // Stream the member straight into the reader. Buffering it first would cap members at + // 2 GB (MemoryStream's limit) and double the peak cost of every load — see OpenMember. + using (Stream member = entry.Open()) + { + NDArray array = NpyFormat.ReadArray(member, AllowPickle, MaxHeaderSize); + _cache[entry] = array; + return array; + } + } + } + + /// + /// A member's raw bytes, whatever it holds. NumPy returns these from its indexer for + /// non-.npy members; for a .npy member this is the encoded file itself. + /// + /// + /// Capped at ~2 GB by itself. A member larger than that can still be read + /// as an array through the indexer, which streams instead of buffering. + /// + /// No such member. + public byte[] GetRawBytes(string key) + { + ThrowIfDisposed(); + + if (!_keyToEntry.TryGetValue(key, out ZipArchiveEntry entry)) + throw new KeyNotFoundException($"{key} is not a file in the archive"); + + var buffer = new MemoryStream(entry.Length > 0 && entry.Length <= int.MaxValue ? (int)entry.Length : 0); + using (Stream member = entry.Open()) + member.CopyTo(buffer); + return buffer.ToArray(); + } + + /// Whether names a member that holds a .npy array. + public bool IsArray(string key) + { + ThrowIfDisposed(); + + if (!_keyToEntry.TryGetValue(key, out ZipArchiveEntry entry)) + return false; + if (_cache.ContainsKey(entry)) + return true; + + return StartsWithNpyMagic(entry); + } + + /// NumPy's name.removesuffix(".npy"). + private static string StripNpy(string entryName) => + entryName.EndsWith(".npy", StringComparison.Ordinal) + ? entryName.Substring(0, entryName.Length - 4) + : entryName; + + /// + /// Peek at a member's magic string without consuming the stream the reader will use. + /// + /// + /// NumPy sniffs the magic and then rewinds (bytes.seek(0)), which Python's ZipExtFile + /// supports but .NET's entry stream does not — it is forward-only. So sniff on one stream and + /// read on a fresh one; in Read mode an entry may be opened any number of times, and the + /// sniff only ever pulls the 6 magic bytes. + /// + /// The obvious alternative — buffer the member into a MemoryStream and seek within it — is + /// what this code used to do, and it was wrong twice over: MemoryStream tops out at 2 GB, so + /// a >2 GB member NumSharp had happily WRITTEN failed to load with "Stream was too long", + /// and every ordinary load paid double (the buffer plus the array built from it). + /// + private static bool StartsWithNpyMagic(ZipArchiveEntry entry) + { + ReadOnlySpan magic = NpyFormat.MagicPrefix; + Span head = stackalloc byte[6]; + + using (Stream sniff = entry.Open()) + { + int got = 0; + while (got < head.Length) + { + int read = sniff.Read(head.Slice(got)); + if (read == 0) break; + got += read; + } + return got == head.Length && head.SequenceEqual(magic); + } + } + + /// Whether the archive has this member (with or without the .npy suffix). + public bool ContainsKey(string key) + { + ThrowIfDisposed(); + return _keyToEntry.ContainsKey(key); + } + + /// The array under , or false if there is no such member. + public bool TryGetValue(string key, out NDArray value) + { + ThrowIfDisposed(); + + if (!_keyToEntry.ContainsKey(key)) + { + value = null; + return false; + } + + value = this[key]; + return true; + } + + /// Enumerate every member as a name/array pair, loading each in turn. + public IEnumerator> GetEnumerator() + { + ThrowIfDisposed(); + + foreach (string key in _files) + yield return new KeyValuePair(key, this[key]); + } + + IEnumerator IEnumerable.GetEnumerator() => GetEnumerator(); + + /// Close the archive and release the file handle — NumPy's close(). + public void Close() => Dispose(); + + /// + public void Dispose() + { + if (_disposed) + return; + + _disposed = true; + F = null; + + _archive?.Dispose(); + _archive = null; + + if (_ownStream) + _stream?.Dispose(); + _stream = null; + + _cache.Clear(); + } + + private void ThrowIfDisposed() + { + if (_disposed) + throw new ObjectDisposedException(nameof(NpzFile)); + } + + /// Formatted as NumPy's repr: NpzFile 'model.npz' with keys: a, b, c. + public override string ToString() + { + string keys = string.Join(", ", _files.Take(MaxReprArrayCount)); + if (_files.Count > MaxReprArrayCount) + keys += "..."; + return $"NpzFile '{_name}' with keys: {keys}"; + } + + /// + /// Turns member lookups into property reads — NumPy's BagObj, reached via + /// . + /// + private sealed class BagObj : DynamicObject + { + // NumPy uses a weakref here so the NpzFile stays collectable despite the cycle. .NET's GC + // collects cycles, so a direct reference is fine. + private readonly NpzFile _owner; + + public BagObj(NpzFile owner) => _owner = owner; + + public override bool TryGetMember(GetMemberBinder binder, out object result) + { + if (!_owner.ContainsKey(binder.Name)) + throw new KeyNotFoundException($"{binder.Name} is not a file in the archive"); + + result = _owner[binder.Name]; + return true; + } + + public override IEnumerable GetDynamicMemberNames() => _owner.Files; + + public override string ToString() => _owner.ToString(); + } + } +} diff --git a/src/NumSharp.Core/IO/PyLiteral.cs b/src/NumSharp.Core/IO/PyLiteral.cs new file mode 100644 index 000000000..0cefedbfc --- /dev/null +++ b/src/NumSharp.Core/IO/PyLiteral.cs @@ -0,0 +1,430 @@ +using System; +using System.Collections; +using System.Collections.Generic; +using System.Globalization; +using System.Text; + +namespace NumSharp.IO +{ + /// + /// A Python tuple. Distinct from because the .npy header validator has to + /// tell (3,) from [3] — NumPy rejects a header whose shape is not a tuple. + /// + internal sealed class PyTuple : IReadOnlyList + { + private readonly List _items; + + public PyTuple(List items) => _items = items; + + public object this[int index] => _items[index]; + public int Count => _items.Count; + public IEnumerator GetEnumerator() => _items.GetEnumerator(); + IEnumerator IEnumerable.GetEnumerator() => _items.GetEnumerator(); + } + + /// + /// Parses the Python literal subset that appears in a .npy header, standing in for NumPy's + /// ast.literal_eval (numpy/lib/_format_impl.py). + /// + /// + /// Only the literal grammar is accepted — dict, list, tuple, str, int, float, complex-free + /// numerics, True/False/None. There is no name lookup, no operators and no + /// calls, so (like literal_eval) parsing a hostile header cannot execute anything. The + /// size guard that protects against pathological input lives in the caller + /// (), matching NumPy's max_header_size check. + /// + /// Python values map to CLR as: str→, int→, + /// float→, bool→, None→null, tuple→, + /// list→, dict→. + /// + internal static class PyLiteral + { + /// Thrown when the input is not a well-formed Python literal (≙ Python's SyntaxError). + internal sealed class SyntaxException : Exception + { + public SyntaxException(string message) : base(message) { } + } + + /// + /// Parse a complete Python literal expression. Trailing whitespace is ignored; any other + /// trailing content is an error. + /// + /// The literal source. + /// + /// Accept the Python 2 L integer suffix (3L). NumPy only retries with its + /// _filter_header for format versions ≤ (2, 0), so v3.0 headers reject it. + /// + public static object Parse(string s, bool allowLongSuffix = false) + { + var p = new Parser(s, allowLongSuffix); + object v = p.ParseValue(); + p.SkipWhitespace(); + if (!p.AtEnd) + throw new SyntaxException($"unexpected trailing content at position {p.Position}"); + return v; + } + + /// + /// Render a parsed value the way Python's repr() would. NumPy interpolates + /// repr() output into its header error messages, so error text matches verbatim. + /// + public static string Repr(object v) + { + var sb = new StringBuilder(); + Repr(v, sb); + return sb.ToString(); + } + + private static void Repr(object v, StringBuilder sb) + { + switch (v) + { + case null: + sb.Append("None"); + return; + case bool b: + sb.Append(b ? "True" : "False"); + return; + case string s: + ReprString(s, sb); + return; + case long l: + sb.Append(l.ToString(CultureInfo.InvariantCulture)); + return; + case double d: + ReprDouble(d, sb); + return; + case PyTuple t: + sb.Append('('); + for (int i = 0; i < t.Count; i++) + { + if (i > 0) sb.Append(", "); + Repr(t[i], sb); + } + if (t.Count == 1) sb.Append(','); // Python's 1-tuple repr: (3,) + sb.Append(')'); + return; + case List list: + sb.Append('['); + for (int i = 0; i < list.Count; i++) + { + if (i > 0) sb.Append(", "); + Repr(list[i], sb); + } + sb.Append(']'); + return; + case Dictionary dict: + sb.Append('{'); + bool first = true; + foreach (var kv in dict) + { + if (!first) sb.Append(", "); + first = false; + ReprString(kv.Key, sb); + sb.Append(": "); + Repr(kv.Value, sb); + } + sb.Append('}'); + return; + default: + sb.Append(v); + return; + } + } + + // Python prefers single quotes, switching to double only when the value contains a single + // quote but no double quote. + private static void ReprString(string s, StringBuilder sb) + { + char quote = s.Contains('\'') && !s.Contains('"') ? '"' : '\''; + sb.Append(quote); + foreach (char c in s) + { + switch (c) + { + case '\\': sb.Append("\\\\"); break; + case '\n': sb.Append("\\n"); break; + case '\r': sb.Append("\\r"); break; + case '\t': sb.Append("\\t"); break; + default: + if (c == quote) sb.Append('\\'); + sb.Append(c); + break; + } + } + sb.Append(quote); + } + + private static void ReprDouble(double d, StringBuilder sb) + { + if (double.IsNaN(d)) { sb.Append("nan"); return; } + if (double.IsPositiveInfinity(d)) { sb.Append("inf"); return; } + if (double.IsNegativeInfinity(d)) { sb.Append("-inf"); return; } + + // "R" is shortest-round-trip, which is what Python's float repr produces. Python always + // shows a decimal point or exponent so the value reads as a float; .NET does not. + string s = d.ToString("R", CultureInfo.InvariantCulture); + if (s.IndexOf('.') < 0 && s.IndexOf('e') < 0 && s.IndexOf('E') < 0 && + s.IndexOf("Inf", StringComparison.Ordinal) < 0 && s.IndexOf("NaN", StringComparison.Ordinal) < 0) + s += ".0"; + sb.Append(s); + } + + private struct Parser + { + private readonly string _s; + private readonly bool _allowLongSuffix; + private int _i; + + public Parser(string s, bool allowLongSuffix) + { + _s = s; + _allowLongSuffix = allowLongSuffix; + _i = 0; + } + + public bool AtEnd => _i >= _s.Length; + public int Position => _i; + + public void SkipWhitespace() + { + while (_i < _s.Length && (_s[_i] == ' ' || _s[_i] == '\t' || _s[_i] == '\n' || _s[_i] == '\r')) + _i++; + } + + private char Peek() + { + if (_i >= _s.Length) + throw new SyntaxException("unexpected EOF while parsing"); + return _s[_i]; + } + + private void Expect(char c) + { + SkipWhitespace(); + if (_i >= _s.Length || _s[_i] != c) + throw new SyntaxException($"expected '{c}' at position {_i}"); + _i++; + } + + public object ParseValue() + { + SkipWhitespace(); + char c = Peek(); + switch (c) + { + case '{': return ParseDict(); + case '[': return ParseList(); + case '(': return ParseTuple(); + case '\'': + case '"': return ParseString(); + } + if (c == '-' || c == '+' || char.IsDigit(c) || c == '.') + return ParseNumber(); + if (char.IsLetter(c) || c == '_') + return ParseKeyword(); + throw new SyntaxException($"invalid syntax at position {_i}: '{c}'"); + } + + private object ParseKeyword() + { + int start = _i; + while (_i < _s.Length && (char.IsLetterOrDigit(_s[_i]) || _s[_i] == '_')) + _i++; + string word = _s.Substring(start, _i - start); + switch (word) + { + case "True": return true; + case "False": return false; + case "None": return null; + // Deliberately NOT nan/inf: they are Names, not literals, so ast.literal_eval + // rejects them too. Accepting them here would make this parser more permissive + // than the thing it stands in for. + default: throw new SyntaxException($"unknown name '{word}' at position {start}"); + } + } + + private object ParseDict() + { + Expect('{'); + var dict = new Dictionary(StringComparer.Ordinal); + SkipWhitespace(); + if (Peek() == '}') { _i++; return dict; } + + while (true) + { + SkipWhitespace(); + if (Peek() != '\'' && Peek() != '"') + throw new SyntaxException($"dict keys must be strings, at position {_i}"); + string key = ParseString(); + Expect(':'); + dict[key] = ParseValue(); + + SkipWhitespace(); + char c = Peek(); + if (c == ',') + { + _i++; + SkipWhitespace(); + if (Peek() == '}') { _i++; return dict; } // trailing comma — NumPy always writes one + continue; + } + if (c == '}') { _i++; return dict; } + throw new SyntaxException($"expected ',' or '}}' at position {_i}"); + } + } + + private object ParseList() + { + Expect('['); + return ParseSequence(']', out _); // brackets always build a list, comma or not + } + + private object ParseTuple() + { + Expect('('); + List items = ParseSequence(')', out bool sawComma); + + // It is the COMMA that makes a tuple in Python, not the parentheses: `(1)` is just the + // int 1 in grouping parens, while `(1,)` is a 1-tuple and `()` is the empty tuple. The + // .npy header validator leans on the difference — NumPy rejects `'shape': (1)` with + // "shape is not valid: 1" because it never saw a tuple. + if (items.Count == 1 && !sawComma) + return items[0]; + + return new PyTuple(items); + } + + private List ParseSequence(char close, out bool sawComma) + { + var items = new List(); + sawComma = false; + + SkipWhitespace(); + if (Peek() == close) { _i++; return items; } + + while (true) + { + items.Add(ParseValue()); + SkipWhitespace(); + char c = Peek(); + if (c == ',') + { + _i++; + sawComma = true; + SkipWhitespace(); + if (Peek() == close) { _i++; return items; } // trailing comma: (3,) / [1, 2,] + continue; + } + if (c == close) { _i++; return items; } + throw new SyntaxException($"expected ',' or '{close}' at position {_i}"); + } + } + + private string ParseString() + { + char quote = Peek(); + _i++; + var sb = new StringBuilder(); + while (true) + { + if (_i >= _s.Length) + throw new SyntaxException("unterminated string literal"); + char c = _s[_i++]; + if (c == quote) + return sb.ToString(); + if (c != '\\') + { + sb.Append(c); + continue; + } + if (_i >= _s.Length) + throw new SyntaxException("unterminated escape sequence"); + char e = _s[_i++]; + switch (e) + { + case 'n': sb.Append('\n'); break; + case 't': sb.Append('\t'); break; + case 'r': sb.Append('\r'); break; + case '0': sb.Append('\0'); break; + case '\\': sb.Append('\\'); break; + case '\'': sb.Append('\''); break; + case '"': sb.Append('"'); break; + case 'x': sb.Append(ParseHexEscape(2)); break; + case 'u': sb.Append(ParseHexEscape(4)); break; + case 'U': sb.Append(char.ConvertFromUtf32(ParseHexCode(8))); break; + default: sb.Append('\\').Append(e); break; + } + } + } + + private char ParseHexEscape(int digits) => (char)ParseHexCode(digits); + + private int ParseHexCode(int digits) + { + if (_i + digits > _s.Length) + throw new SyntaxException("truncated hex escape"); + int v = 0; + for (int k = 0; k < digits; k++) + { + int d = HexVal(_s[_i + k]); + if (d < 0) throw new SyntaxException("invalid hex escape"); + v = (v << 4) | d; + } + _i += digits; + return v; + } + + private static int HexVal(char c) + { + if (c >= '0' && c <= '9') return c - '0'; + if (c >= 'a' && c <= 'f') return c - 'a' + 10; + if (c >= 'A' && c <= 'F') return c - 'A' + 10; + return -1; + } + + private object ParseNumber() + { + int start = _i; + if (_i < _s.Length && (_s[_i] == '-' || _s[_i] == '+')) + _i++; + + bool isFloat = false; + while (_i < _s.Length) + { + char c = _s[_i]; + if (char.IsDigit(c) || c == '_') { _i++; continue; } + if (c == '.') { isFloat = true; _i++; continue; } + if (c == 'e' || c == 'E') + { + // An exponent only continues the number if a digit/sign follows, otherwise + // this 'e' begins a following name. + int j = _i + 1; + if (j < _s.Length && (_s[j] == '+' || _s[j] == '-')) j++; + if (j < _s.Length && char.IsDigit(_s[j])) { isFloat = true; _i = j + 1; continue; } + } + break; + } + + string text = _s.Substring(start, _i - start).Replace("_", ""); + if (text.Length == 0 || text == "-" || text == "+") + throw new SyntaxException($"invalid number at position {start}"); + + // Python 2 wrote longs as `3L`. NumPy strips the suffix via _filter_header, but only + // when retrying a v1.0/v2.0 header; v3.0 post-dates Python 2 and rejects it. + if (!isFloat && _i < _s.Length && (_s[_i] == 'L' || _s[_i] == 'l')) + { + if (!_allowLongSuffix) + throw new SyntaxException($"invalid decimal literal at position {_i}"); + _i++; + } + + if (!isFloat && long.TryParse(text, NumberStyles.AllowLeadingSign, CultureInfo.InvariantCulture, out long l)) + return l; + if (double.TryParse(text, NumberStyles.Float, CultureInfo.InvariantCulture, out double d)) + return d; + throw new SyntaxException($"invalid number '{text}' at position {start}"); + } + } + } +} diff --git a/src/NumSharp.Core/Utilities/NpzDictionary.cs b/src/NumSharp.Core/Utilities/NpzDictionary.cs deleted file mode 100644 index f9bf2d3ec..000000000 --- a/src/NumSharp.Core/Utilities/NpzDictionary.cs +++ /dev/null @@ -1,206 +0,0 @@ -using System; -using System.Collections; -using System.Collections.Generic; -using System.Data; -using System.IO; -using System.IO.Compression; -using System.Linq; - -// ReSharper disable once CheckNamespace -namespace NumSharp -{ - public class NpzDictionary : IDisposable, IReadOnlyDictionary, ICollection - where T : class, - ICloneable, - IList, ICollection, IEnumerable, IStructuralComparable, IStructuralEquatable - { - Stream stream; - ZipArchive archive; - - bool disposedValue = false; - - Dictionary entries; - Dictionary arrays; - - - public NpzDictionary(Stream stream) - { - this.stream = stream; - this.archive = new ZipArchive(stream, ZipArchiveMode.Read, leaveOpen: true); - - this.entries = new Dictionary(); - foreach (var entry in archive.Entries) - this.entries[entry.FullName] = entry; - - this.arrays = new Dictionary(); - } - - - public IEnumerable Keys - { - get { return entries.Keys; } - } - - - public IEnumerable Values - { - get { return entries.Values.Select(OpenEntry); } - } - - public int Count - { - get { return entries.Count; } - } - - - public object SyncRoot - { - get { return ((ICollection)entries).SyncRoot; } - } - - - public bool IsSynchronized - { - get { return ((ICollection)entries).IsSynchronized; } - } - - public bool IsReadOnly - { - get { return true; } - } - - public T this[string key] - { - get { return OpenEntry(entries[key]); } - } - - private T OpenEntry(ZipArchiveEntry entry) - { - T array; - if (arrays.TryGetValue(entry.FullName, out array)) - return array; - - using (Stream s = entry.Open()) - { - array = Load_Npz(s); - arrays[entry.FullName] = array; - return array; - } - } - - protected virtual T Load_Npz(Stream s) - { - return np.Load(s); - } - - public bool ContainsKey(string key) - { - return entries.ContainsKey(key); - } - - public bool TryGetValue(string key, out T value) - { - value = default(T); - ZipArchiveEntry entry; - if (!entries.TryGetValue(key, out entry)) - return false; - value = OpenEntry(entry); - return true; - } - - public IEnumerator> GetEnumerator() - { - foreach (var entry in archive.Entries) - yield return new KeyValuePair(entry.FullName, OpenEntry(entry)); - } - - IEnumerator IEnumerable.GetEnumerator() - { - foreach (var entry in archive.Entries) - yield return new KeyValuePair(entry.FullName, OpenEntry(entry)); - } - - IEnumerator IEnumerable.GetEnumerator() - { - foreach (var entry in archive.Entries) - yield return OpenEntry(entry); - } - - public void CopyTo(Array array, int arrayIndex) - { - foreach (var v in this) - array.SetValue(v, arrayIndex++); - } - - public void CopyTo(T[] array, int arrayIndex) - { - foreach (var v in this) - array.SetValue(v, arrayIndex++); - } - - public void Add(T item) - { - throw new ReadOnlyException(); - } - - public void Clear() - { - throw new ReadOnlyException(); - } - - public bool Contains(T item) - { - foreach (var v in this) - if (Object.Equals(v.Value, item)) - return true; - return false; - } - - public bool Remove(T item) - { - throw new ReadOnlyException(); - } - - protected virtual void Dispose(bool disposing) - { - if (!disposedValue) - { - if (disposing) - { - archive.Dispose(); - stream.Dispose(); - } - - archive = null; - stream = null; - entries = null; - arrays = null; - - disposedValue = true; - } - } - - public void Dispose() - { - Dispose(true); - } - } - - public class NpzDictionary : NpzDictionary - { - bool jagged; - - public NpzDictionary(Stream stream, bool jagged) - : base(stream) - { - this.jagged = jagged; - } - - protected override Array Load_Npz(Stream s) - { - if (jagged) - return np.LoadJagged(s); - return np.LoadMatrix(s); - } - } -} diff --git a/test/NumSharp.UnitTest/APIs/np.load.Test.cs b/test/NumSharp.UnitTest/APIs/np.load.Test.cs deleted file mode 100644 index e4786079e..000000000 --- a/test/NumSharp.UnitTest/APIs/np.load.Test.cs +++ /dev/null @@ -1,47 +0,0 @@ -using Microsoft.VisualStudio.TestTools.UnitTesting; -using NumSharp; -using System; -using System.Collections.Generic; -using System.Text; -using System.Linq; - -namespace NumSharp.UnitTest.APIs -{ - [DoNotParallelize] - [TestClass] - public class NumpyLoad - { - [TestMethod] - public void NumpyLoadTest() - { - int[] a = {1, 2, 3, 4, 5}; - byte[] mem = np.Save(a); - - int[] b = np.Load(mem); - } - - [TestMethod] - public void NumpyLoad1DimTest() - { - int[] arr = np.Load(@"data/1-dim-int32_4_comma_empty.npy"); - Assert.IsTrue(arr[0] == 0); - Assert.IsTrue(arr[1] == 1); - Assert.IsTrue(arr[2] == 2); - Assert.IsTrue(arr[3] == 3); - } - - [TestMethod] - public void NumpyNPZRoundTripTest() - { - int[] arr = np.Load(@"data/1-dim-int32_4_comma_empty.npy"); - var d = new Dictionary(); - d.Add("A/A",arr); - d.Add("B/A",arr); // Tests zip entity.Name - var ms = new System.IO.MemoryStream(); - np.Save_Npz(d,ms,leaveOpen:true); - ms.Position = 0L; - var d2 = np.Load_Npz(ms); - Assert.IsTrue(d2.Count == 2); - } - } -} diff --git a/test/NumSharp.UnitTest/Documentation/NDArrayDocExamplesTests.cs b/test/NumSharp.UnitTest/Documentation/NDArrayDocExamplesTests.cs new file mode 100644 index 000000000..4cd473d51 --- /dev/null +++ b/test/NumSharp.UnitTest/Documentation/NDArrayDocExamplesTests.cs @@ -0,0 +1,796 @@ +using System; +using System.Collections.Generic; +using System.IO; +using System.Linq; +using System.Numerics; +using System.Runtime.InteropServices; +using AwesomeAssertions; +using Microsoft.VisualStudio.TestTools.UnitTesting; +using NumSharp; +using NumSharp.Generic; +using NumSharp.IO; + +namespace NumSharp.UnitTest.Documentation +{ + /// + /// Executable coverage for every code example in docs/website-src/docs/ndarray.md + /// (the "NumSharp's ndarray is NDArray!" guide). Each test mirrors one documented snippet and + /// asserts the behaviour the doc claims, so the page cannot silently drift from the library. + /// Section headers below match the doc's headings; the observed values were captured by running + /// the snippets against this branch. + /// + [TestClass] + public class NDArrayDocExamplesTests + { + private string _dir; + + [TestInitialize] + public void Setup() + { + _dir = Path.Combine(Path.GetTempPath(), "ns_ndarray_doc_" + Guid.NewGuid().ToString("N")); + Directory.CreateDirectory(_dir); + } + + [TestCleanup] + public void Cleanup() + { + try { Directory.Delete(_dir, recursive: true); } + catch (IOException) { /* a leaked handle fails the test that leaked it, not this one */ } + } + + private string At(string name) => Path.Combine(_dir, name); + + // ── Creating an NDArray ──────────────────────────────────────────────────────────────────── + + [TestMethod] + public void Creating_TheUsualWays() + { + np.array(new[] { 1, 2, 3 }).shape.Should().Equal(new long[] { 3 }); + np.array(new[] { 1, 2, 3 }).typecode.Should().Be(NPTypeCode.Int32); + np.array(new int[,] { { 1, 2 }, { 3, 4 } }).shape.Should().Equal(new long[] { 2, 2 }); + + np.zeros((3, 4)).shape.Should().Equal(new long[] { 3, 4 }); + np.zeros((3, 4)).typecode.Should().Be(NPTypeCode.Double); + np.ones(5).shape.Should().Equal(new long[] { 5 }); + np.full((2, 2), 7).ToArray().Should().Equal(7, 7, 7, 7); + np.full(new Shape(2, 2), 7).shape.Should().Equal(new long[] { 2, 2 }); + np.empty((3, 3)).shape.Should().Equal(new long[] { 3, 3 }); + np.eye(4).shape.Should().Equal(new long[] { 4, 4 }); + np.identity(4).shape.Should().Equal(new long[] { 4, 4 }); + + np.arange(10).shape.Should().Equal(new long[] { 10 }); + np.arange(10).typecode.Should().Be(NPTypeCode.Int64, "arange defaults to int64 (NumPy 2.x)"); + np.arange(0, 1, 0.1).typecode.Should().Be(NPTypeCode.Double); + np.linspace(0, 1, 11).shape.Should().Equal(new long[] { 11 }); + ((double)np.linspace(0, 1, 11)[0]).Should().Be(0.0); + ((double)np.linspace(0, 1, 11)[10]).Should().Be(1.0); + + np.random.rand(3, 4).shape.Should().Equal(new long[] { 3, 4 }); + np.random.randn(100).shape.Should().Equal(new long[] { 100 }); + } + + [TestMethod] + public void Creating_ShapeConversionForms_AllProduce3x4() + { + np.zeros((3, 4)).shape.Should().Equal(new long[] { 3, 4 }); + np.zeros(new[] { 3, 4 }).shape.Should().Equal(new long[] { 3, 4 }); + np.zeros(new Shape(3, 4)).shape.Should().Equal(new long[] { 3, 4 }); + np.zeros(new Shape(new[] { 3L, 4L })).shape.Should().Equal(new long[] { 3, 4 }); + + np.zeros(5).shape.Should().Equal(new long[] { 5 }, "a bare int is the 1-D length overload"); + } + + [TestMethod] + public void Creating_ScalarsFlowInImplicitly() + { + NDArray a = 42; + NDArray b = 3.14; + NDArray c = Half.One; + NDArray d = NDArray.Scalar(100.123m); + NDArray e = NDArray.Scalar(1); + + a.typecode.Should().Be(NPTypeCode.Int32); + a.ndim.Should().Be(0); + b.typecode.Should().Be(NPTypeCode.Double); + c.typecode.Should().Be(NPTypeCode.Half); + d.typecode.Should().Be(NPTypeCode.Decimal); + e.typecode.Should().Be(NPTypeCode.Int64); + } + + // ── Wrapping Existing Buffers — np.frombuffer ────────────────────────────────────────────── + + [TestMethod] + public void Frombuffer_ByteBuffer_OffsetAndCount() + { + byte[] buffer = new byte[16]; + Buffer.BlockCopy(new[] { 1f, 2f, 3f, 4f }, 0, buffer, 0, 16); + + np.frombuffer(buffer, typeof(float)).ToArray().Should().Equal(1f, 2f, 3f, 4f); + // offset is in BYTES + np.frombuffer(buffer, typeof(float), offset: 4).ToArray().Should().Equal(2f, 3f, 4f); + // count is in ELEMENTS + np.frombuffer(buffer, typeof(float), count: 2, offset: 4).ToArray().Should().Equal(2f, 3f); + } + + [TestMethod] + public void Frombuffer_ReinterpretTypedArrayAsBytes() + { + int[] ints = { 1, 2, 3, 4 }; + var bytes = np.frombuffer(ints, typeof(byte)); // little-endian on x86/x64 + + bytes.size.Should().Be(16); + bytes.ToArray().Take(8).Should().Equal((byte)1, 0, 0, 0, 2, 0, 0, 0); + } + + [TestMethod] + public void Frombuffer_SegmentMemoryAndSpan() + { + byte[] buffer = new byte[12]; + Buffer.BlockCopy(new[] { 11, 22, 33 }, 0, buffer, 0, 12); + + np.frombuffer(new ArraySegment(buffer, 0, 12), typeof(int)).ToArray() + .Should().Equal(11, 22, 33); + np.frombuffer((Memory)buffer, typeof(int)).ToArray() + .Should().Equal(11, 22, 33); + + Span tmp = stackalloc byte[8]; + BitConverter.TryWriteBytes(tmp, 7); + BitConverter.TryWriteBytes(tmp.Slice(4), 9); + ReadOnlySpan span = tmp; + np.frombuffer(span, typeof(int)).ToArray().Should().Equal(7, 9); // spans always copy + } + + [TestMethod] + public void Frombuffer_NativeMemory_BorrowedAndOwned() + { + // Borrowed — caller frees after the view is done. + IntPtr borrowed = Marshal.AllocHGlobal(sizeof(float) * 3); + try + { + Marshal.Copy(new[] { 1f, 2f, 3f }, 0, borrowed, 3); + np.frombuffer(borrowed, sizeof(float) * 3, typeof(float)).ToArray() + .Should().Equal(1f, 2f, 3f); + } + finally { Marshal.FreeHGlobal(borrowed); } + + // Owned — NumSharp frees on GC via the dispose callback (we must NOT free it too). + IntPtr owned = Marshal.AllocHGlobal(sizeof(float) * 2); + Marshal.Copy(new[] { 5f, 6f }, 0, owned, 2); + var arr1 = np.frombuffer(owned, sizeof(float) * 2, typeof(float), + dispose: () => Marshal.FreeHGlobal(owned)); + arr1.ToArray().Should().Equal(5f, 6f); + } + + [TestMethod] + public void Frombuffer_Endianness_BigVsLittle() + { + byte[] networkData = { 0, 0, 0, 1 }; + np.frombuffer(networkData, ">i4").GetInt32(0).Should().Be(1, "big-endian reads 0x00000001"); + np.frombuffer(networkData, "().Should().Equal(4L, 9L, 14L, 19L); + } + + [TestMethod] + public void Indexing_BooleanAndFancy() + { + var arr = np.array(new[] { 10, 20, 30, 40, 50 }); + + var mask = arr > 20; + mask.ToArray().Should().Equal(false, false, true, true, true); + arr[mask].ToArray().Should().Equal(30, 40, 50); + + var idx = np.array(new[] { 0, 2, 4 }); + arr[idx].ToArray().Should().Equal(10, 30, 50); + } + + [TestMethod] + public void Indexing_Assignment() + { + var a = np.arange(20).reshape(4, 5); + + a[1, 2] = 99; // scalar write + ((long)a[1, 2]).Should().Be(99L); + + a[0] = np.zeros(5); // row write (double broadcast into the int64 row) + a["0"].ToArray().Should().Equal(0L, 0L, 0L, 0L, 0L); + + a[a > 10] = -1; // masked write + a.ToArray().Should().OnlyContain(x => x <= 10); + } + + // ── Views vs Copies ──────────────────────────────────────────────────────────────────────── + + [TestMethod] + public void Views_ShareMemory_CopyDoesNot() + { + var a = np.arange(10); + var v = a["2:5"]; // view — shares memory + v[0] = 999; + ((long)a[2]).Should().Be(999L, "mutating the view mutates the parent"); + + var c = a["2:5"].copy(); // explicit copy — independent + c[0] = 0; + ((long)a[2]).Should().Be(999L, "the copy does not touch the parent"); + } + + // ── Operators ────────────────────────────────────────────────────────────────────────────── + + [TestMethod] + public void Operators_Arithmetic() + { + var a = np.array(new[] { 1, 2, 3 }); + var b = np.array(new[] { 4, 5, 6 }); + + (a + b).ToArray().Should().Equal(5, 7, 9); + (b - a).ToArray().Should().Equal(3, 3, 3); + (a * b).ToArray().Should().Equal(4, 10, 18); + + var div = np.array(new[] { 1, 2, 3 }) / np.array(new[] { 2, 2, 2 }); + div.typecode.Should().Be(NPTypeCode.Double, "int / int returns a float dtype"); + div.ToArray().Should().Equal(0.5, 1.0, 1.5); + + // result sign follows the divisor (Python/NumPy convention) + (np.array(new[] { -7, 7 }) % np.array(new[] { 3, 3 })).ToArray().Should().Equal(2, 1); + + (-np.array(new[] { 1, -2 })).ToArray().Should().Equal(-1, 2); + + var p = +a; // unary plus returns a copy + p.ToArray().Should().Equal(1, 2, 3); + ReferenceEquals(p, a).Should().BeFalse(); + + // object on either side works + (10 - np.array(new[] { 1, 2, 3 })).ToArray().Should().Equal(9, 8, 7); + } + + [TestMethod] + public void Operators_BitwiseAndShift() + { + (np.array(new[] { 12 }) & np.array(new[] { 10 })).GetInt32(0).Should().Be(8); + (np.array(new[] { 12 }) | np.array(new[] { 10 })).GetInt32(0).Should().Be(14); + (np.array(new[] { 12 }) ^ np.array(new[] { 10 })).GetInt32(0).Should().Be(6); + (~np.array(new[] { 0, -1 })).ToArray().Should().Equal(-1, 0); + + (np.array(new[] { 1, 2, 3 }) << 2).ToArray().Should().Equal(4, 8, 12); + (np.array(new[] { 4, 8, 16 }) >> 2).ToArray().Should().Equal(1, 2, 4); + + // bool arrays: & is logical AND, | is logical OR + var t = np.array(new[] { true, true, false }); + var u = np.array(new[] { true, false, false }); + (t & u).ToArray().Should().Equal(true, false, false); + (t | u).ToArray().Should().Equal(true, true, false); + } + + [TestMethod] + public void Operators_Comparison() + { + var a = np.array(new[] { 1, 2, 3 }); + var b = np.array(new[] { 3, 2, 1 }); + + (a == b).ToArray().Should().Equal(false, true, false); + (a != b).ToArray().Should().Equal(true, false, true); + (a < b).ToArray().Should().Equal(true, false, false); + (a <= b).ToArray().Should().Equal(true, true, false); + (a > b).ToArray().Should().Equal(false, false, true); + (a >= b).ToArray().Should().Equal(false, true, true); + + // comparisons with NaN are always false (IEEE 754) + var n = np.array(new[] { double.NaN }); + ((bool)(n == n)[0]).Should().BeFalse(); + } + + [TestMethod] + public void Operators_LogicalNot() + { + var mask = np.array(new[] { 1, 3 }) > 2; // [false, true] + (!mask).ToArray().Should().Equal(true, false); + } + + [TestMethod] + public void Operators_FunctionForms_ForMissingCSharpOperators() + { + // a ** b + np.power(np.array(new[] { 2, 3 }), 2).astype(NPTypeCode.Int64).ToArray() + .Should().Equal(4L, 9L); + // a // b + np.floor_divide(np.array(new[] { 7 }), np.array(new[] { 3 })).astype(NPTypeCode.Int64) + .ToArray().Should().Equal(2L); + // a @ b — 1-D dot product + ((int)np.dot(np.array(new[] { 1, 2, 3 }), np.array(new[] { 4, 5, 6 }))).Should().Be(32); + // a @ b — 2-D matmul against the identity is a no-op + var m = np.array(new int[,] { { 5, 6 }, { 7, 8 } }); + np.array_equal(np.matmul(np.array(new int[,] { { 1, 0 }, { 0, 1 } }), m), m).Should().BeTrue(); + // abs(a) + np.abs(np.array(new[] { -1, 2, -3 })).ToArray().Should().Equal(1, 2, 3); + // divmod(a, b) == (floor_divide, mod) + var (q, r) = (np.floor_divide(np.array(new[] { 7 }), np.array(new[] { 3 })), + np.array(new[] { 7 }) % np.array(new[] { 3 })); + q.astype(NPTypeCode.Int64).ToArray().Should().Equal(2L); + r.ToArray().Should().Equal(1); + } + + [TestMethod] + public void Operators_ShiftQuirk_RhsForms() + { + var arr = np.array(new[] { 1, 2, 3 }); + (arr << 2).ToArray().Should().Equal(4, 8, 12); // int RHS + + object rhs = 2; + (np.array(new[] { 1, 2, 3 }) << rhs).ToArray().Should().Equal(4, 8, 12); // object RHS + + // 2 << arr is a C# compile error; the function form is the way + np.left_shift(2, np.array(new[] { 1, 2, 3 })).ToArray().Should().Equal(4, 8, 16); + } + + [TestMethod] + public void Operators_CompoundAssignment_IsNotInPlace() + { + var x = np.array(new[] { 1, 2, 3 }); + var alias = x; + x += 10; // synthesized as x = x + 10 → new array + x.ToArray().Should().Equal(11, 12, 13); + // the alias still sees the original — unlike NumPy + alias.ToArray().Should().Equal(1, 2, 3); + } + + // ── Dtype Conversion ─────────────────────────────────────────────────────────────────────── + + [TestMethod] + public void DtypeConversion_AstypeAndScalarCasts() + { + var a = np.array(new[] { 1, 2, 3 }); + a.astype(np.float64).typecode.Should().Be(NPTypeCode.Double); + a.astype(NPTypeCode.Int64).typecode.Should().Be(NPTypeCode.Int64); + + NDArray scalar = NDArray.Scalar(42); + ((int)scalar).Should().Be(42); + ((double)scalar).Should().Be(42.0); + ((Half)scalar).Should().Be((Half)42); + var cx = (Complex)scalar; + cx.Real.Should().Be(42); + cx.Imaginary.Should().Be(0); + } + + [TestMethod] + public void DtypeConversion_ComplexToNonComplex_Throws() + { + // "Complex → non-complex throws TypeError (mirroring Python's int(1+2j))" + NDArray c = NDArray.Scalar(new Complex(1, 2)); + Action act = () => { int _ = (int)c; }; + act.Should().Throw(); + } + + // ── Scalars (0-d Arrays) ─────────────────────────────────────────────────────────────────── + + [TestMethod] + public void Scalars_ZeroDimensional() + { + var s1 = NDArray.Scalar(42); + NDArray s2 = 42; + s1.ndim.Should().Be(0); + s1.size.Should().Be(1); + ((int)s1).Should().Be(42); + ((int)s2).Should().Be(42); + } + + // ── Reading & Writing Elements ───────────────────────────────────────────────────────────── + + [TestMethod] + public void ReadWrite_FourWays() + { + var a = np.arange(12).reshape(3, 4); // int64 + + // reads + NDArray elem = a[1, 2]; + ((long)elem).Should().Be(6L); + a.item(6).Should().Be(6L); + a.item(6).Should().Be(6L); // boxed long + a.GetValue(1, 2).Should().Be(6L); + a.GetAtIndex(6).Should().Be(6L); + + // writes + a[1, 2] = 99; + ((long)a[1, 2]).Should().Be(99L); + a.SetValue(99L, 2, 0); + a.GetValue(2, 0).Should().Be(99L); + a.SetAtIndex(99L, 0); + a.GetAtIndex(0).Should().Be(99L); + } + + [TestMethod] + public void ReadWrite_ItemNoArgs_OnSize1() + { + NDArray.Scalar(5).item().Should().Be(5); // 0-d + np.array(new[] { 7 }).item().Should().Be(7); // 1-element 1-d + + Action act = () => np.array(new[] { 1, 2 }).item(); + act.Should().Throw("item() requires a size-1 array"); + } + + // ── Iterating (foreach) ──────────────────────────────────────────────────────────────────── + + [TestMethod] + public void Iterating_ForeachIsAxis0_FlatIsElementwise() + { + var m = np.arange(6).reshape(2, 3); + + int rows = 0; + foreach (NDArray row in m) + { + row.shape.Should().Equal(new long[] { 3 }, "each row is a (3,) view"); + rows++; + } + rows.Should().Be(2); + + int flat = 0; + foreach (var _ in m.flat) flat++; + flat.Should().Be(6); + } + + // ── Common Patterns ──────────────────────────────────────────────────────────────────────── + + [TestMethod] + public void Patterns_FlattenReshapeTranspose() + { + var a = np.arange(12).reshape(3, 4); + + a.ravel().shape.Should().Equal(new long[] { 12 }); + a.flatten().shape.Should().Equal(new long[] { 12 }); + + np.arange(12).reshape(3, 4).shape.Should().Equal(new long[] { 3, 4 }); + np.arange(12).reshape(-1).shape.Should().Equal(new long[] { 12 }); + np.arange(12).reshape(-1, 4).shape.Should().Equal(new long[] { 3, 4 }); + + var t = np.arange(24).reshape(2, 3, 4); + t.T.shape.Should().Equal(new long[] { 4, 3, 2 }); + t.transpose(new[] { 1, 0, 2 }).shape.Should().Equal(new long[] { 3, 2, 4 }); + np.swapaxes(t, 0, 1).shape.Should().Equal(new long[] { 3, 2, 4 }); + np.moveaxis(t, 0, -1).shape.Should().Equal(new long[] { 3, 4, 2 }); + } + + // ── Generic NDArray ───────────────────────────────────────────────────────────────────── + + [TestMethod] + public void Generic_TypedWrappers() + { + NDArray a = np.zeros(10).MakeGeneric(); + double first = a[0]; + first.Should().Be(0.0); + a[0] = 3.14; + a[0].Should().Be(3.14); + + np.zeros(3).AsGeneric()[0].Should().Be(0.0); // never allocates; returns null on mismatch + np.array(new[] { 1, 2, 3 }).AsOrMakeGeneric()[0].Should().Be(1.0); // astype when dtype differs + } + + // ── Saving, Loading, and Interop ─────────────────────────────────────────────────────────── + + [TestMethod] + public void SaveLoad_Npy_RoundTrip() + { + var arr = np.arange(10).reshape(2, 5); + np.save(At("arr.npy"), arr); + NDArray a = np.load_npy(At("arr.npy")); + np.array_equal(a, arr).Should().BeTrue(); + } + + [TestMethod] + public void SaveLoad_Npz_Positional() + { + var x = np.arange(3); + var y = np.arange(4.0); + np.savez(At("bundle.npz"), x, y); // positional → "arr_0", "arr_1" + + using NpzFile npz = np.load_npz(At("bundle.npz")); + npz.Files.Should().Equal("arr_0", "arr_1"); + npz["arr_0"].ToArray().Should().Equal(0L, 1L, 2L); + npz["arr_1"].ToArray().Should().Equal(0.0, 1.0, 2.0, 3.0); + } + + [TestMethod] + public void SaveLoad_Npz_DictionaryKeys() + { + var w = np.arange(6).reshape(2, 3); + var b = np.zeros(3); + np.savez(At("bundle.npz"), new Dictionary { ["w"] = w, ["b"] = b }); + + using NpzFile npz = np.load_npz(At("bundle.npz")); + npz.Files.Should().BeEquivalentTo(new[] { "w", "b" }); + npz["w"].shape.Should().Equal(new long[] { 2, 3 }); + } + + [TestMethod] + public void SaveLoad_Npz_Compressed_IsSmaller() + { + var big = np.zeros(20_000); + byte[] stored = np.savez(big); + byte[] deflated = np.savez_compressed(big); + deflated.Length.Should().BeLessThan(stored.Length, "zeros compress well"); + } + + [TestMethod] + public void SaveLoad_NpzFile_LazyCachedDisposableAccess() + { + np.savez(At("bundle.npz"), new Dictionary { ["w"] = np.arange(3), ["b"] = np.zeros(2) }); + + using NpzFile npz = np.load_npz(At("bundle.npz")); + npz["w"].Should().BeSameAs(npz["w.npy"], "\"w\" and \"w.npy\" name the same member"); + npz["w"].Should().BeSameAs(npz["w"], "a member is cached after first access"); + + NDArray w2 = npz.f.w; // dot access, like NumPy's npz.f + w2.ToArray().Should().Equal(0L, 1L, 2L); + + npz.Files.Should().BeEquivalentTo(new[] { "w", "b" }, "'.npy' is stripped from Files"); + } + + [TestMethod] + public void SaveLoad_Load_ReturnsObject_DispatchedOnMagicBytes() + { + np.save(At("one.npy"), np.arange(3)); + np.savez(At("many.npz"), np.arange(3)); + + object fromNpy = np.load(At("one.npy")); + object fromNpz = np.load(At("many.npz")); + + fromNpy.Should().BeOfType(); + fromNpz.Should().BeOfType(); + ((NpzFile)fromNpz).Dispose(); + } + + [TestMethod] + public void SaveLoad_RawBinary_TofileFromfile() + { + var arr = np.array(new[] { 1.0, 2.0, 3.0 }); + arr.tofile(At("data.bin")); + np.fromfile(At("data.bin"), np.float64).ToArray().Should().Equal(1.0, 2.0, 3.0); + } + + [TestMethod] + public void Interop_DotNetArrays() + { + var arr = np.array(new int[,] { { 1, 2 }, { 3, 4 } }); + + int[,] md = (int[,])arr.ToMuliDimArray(); + md[1, 1].Should().Be(4); + + int[][] jag = (int[][])arr.ToJaggedArray(); + jag[1][0].Should().Be(3); + + np.array(md).shape.Should().Equal(new long[] { 2, 2 }); + } + + // ── When Two Arrays Are "The Same" ───────────────────────────────────────────────────────── + + [TestMethod] + public void Sameness_ComparisonHelpers() + { + var a = np.array(new[] { 1, 2, 3 }); + var b = np.array(new[] { 1, 2, 3 }); + + (a == b).ToArray().Should().Equal(true, true, true); // element-wise + np.array_equal(a, b).Should().BeTrue(); // same shape AND all equal + np.allclose(np.array(new[] { 1.0, 2.0 }), np.array(new[] { 1.0, 2.0 })).Should().BeTrue(); + ReferenceEquals(a, b).Should().BeFalse(); // different C# objects + + var v = a["1:"]; + (v.@base is not null).Should().BeTrue("v is a view"); + (a.@base is not null).Should().BeFalse("a owns its data"); + } + + // ── Troubleshooting ──────────────────────────────────────────────────────────────────────── + + [TestMethod] + public void Troubleshooting_Snippets() + { + // "My array changed when I modified a slice!" → copy to detach + var a = np.arange(10); + var detached = a["1:3"].copy(); + detached[0] = -5; + ((long)a[1]).Should().Be(1L, "the copy is independent"); + + // "ReadOnlyArrayException writing to my slice" → copy the broadcast view first + var b = np.arange(3); + var writable = b.copy(); + writable[0] = 100; + ((long)writable[0]).Should().Be(100L); + + // "ScalarConversionException on (int)arr" → index/GetAtIndex first + var arr = np.array(new[] { 10, 20, 30 }); + arr.GetAtIndex(0).Should().Be(10); + + // "10 << arr doesn't compile" → np.left_shift + np.left_shift(10, np.array(new[] { 1, 2 })).ToArray().Should().Equal(20, 40); + + // "a += 1 didn't update another reference" → write directly for in-place + var c = np.array(new[] { 1, 2, 3 }); + c["..."] = c + 1; + c.ToArray().Should().Equal(2, 3, 4); + + // "NumSharpException: assignment destination is read-only" → copy the broadcast view first + var bc = np.broadcast_to(np.arange(3), new Shape(2, 3)); + Action write = () => bc[0, 0] = 9; + write.Should().Throw().WithMessage("*read-only*"); + } + + // ── Second pass: the view / copy / read-only semantics the tables claim ───────────────────── + + [TestMethod] + public void Frombuffer_ByteArray_IsAView_SeesSourceMutation() + { + byte[] buf = new byte[4]; + BitConverter.TryWriteBytes(buf, 5); + var view = np.frombuffer(buf, typeof(int)); + view.GetInt32(0).Should().Be(5); + + BitConverter.TryWriteBytes(buf, 9); // mutate the source after wrapping + view.GetInt32(0).Should().Be(9, "frombuffer(byte[]) is a view — it sees source mutations"); + } + + [TestMethod] + public void Frombuffer_BigEndian_IsACopy_IgnoresSourceMutation() + { + byte[] be = { 0, 0, 0, 5 }; + var copy = np.frombuffer(be, ">i4"); + copy.GetInt32(0).Should().Be(5); + + be[3] = 9; // mutate the source + copy.GetInt32(0).Should().Be(5, "big-endian frombuffer copies (byte-swapped), so source mutation is invisible"); + } + + [TestMethod] + public void Frombuffer_LengthNotMultipleOfElementSize_Throws() + { + Action act = () => np.frombuffer(new byte[6], typeof(int)); // 6 is not a multiple of 4 + act.Should().Throw(); + } + + [TestMethod] + public void Indexing_PlainSlice_IsWriteableView() + { + var a = np.arange(10); + var slice = a["2:5"]; + slice.Shape.IsWriteable.Should().BeTrue(); + slice[0] = -1; + ((long)a[2]).Should().Be(-1L, "a plain slice is a writeable view of the parent"); + } + + [TestMethod] + public void Indexing_Fancy_IsWriteableIndependentCopy() + { + var arr = np.array(new[] { 10, 20, 30, 40, 50 }); + var fancy = arr[np.array(new[] { 0, 2, 4 })]; + fancy.Shape.IsWriteable.Should().BeTrue(); + fancy[0] = 999; + ((int)arr[0]).Should().Be(10, "fancy indexing returns an independent copy"); + } + + [TestMethod] + public void Indexing_BooleanMask_IsWriteableIndependentCopy() + { + var arr = np.array(new[] { 10, 20, 30, 40, 50 }); + + var masked = arr[arr > 20]; + masked.Shape.IsWriteable.Should().BeTrue("the boolean-mask result is a writeable copy (NumPy parity)"); + masked[0] = 999; + ((int)masked[0]).Should().Be(999); + // the mask copy is independent of the parent + arr.ToArray().Should().Equal(10, 20, 30, 40, 50); + + // the setter form DOES reach the parent (goes through the indexer, not the returned copy) + arr[arr > 20] = -7; + arr.ToArray().Should().Equal(10, 20, -7, -7, -7); + } + + [TestMethod] + public void CopySemantics_ViewVsCopy_AtAGlance() + { + // views — share memory with the source + var m = np.arange(6).reshape(2, 3); + m.T[0, 0] = 100; + ((long)m[0, 0]).Should().Be(100L, "transpose is a view"); + + var contig = np.arange(4); + contig.ravel()[0] = 77; + ((long)contig[0]).Should().Be(77L, "ravel of a contiguous array is a view"); + + var reshaped = np.arange(6); + reshaped.reshape(2, 3)[0, 0] = 55; + ((long)reshaped[0]).Should().Be(55L, "reshape of a contiguous array is a view"); + + // copies — independent memory + var src = np.arange(4); + src.flatten()[0] = 88; + ((long)src[0]).Should().Be(0L, "flatten always copies"); + + var orig = np.arange(4); + orig.copy()[0] = 88; + ((long)orig[0]).Should().Be(0L, "copy() is independent"); + } + + [TestMethod] + public void Generic_MismatchBehaviour_AndStorageSharing() + { + // MakeGeneric shares storage with the source + var z = np.zeros(3); + NDArray g = z.MakeGeneric(); + g[0] = 3.5; + ((double)z[0]).Should().Be(3.5, "MakeGeneric wraps the same storage"); + + // AsGeneric returns null on a dtype mismatch (like C# 'as'); MakeGeneric throws + (np.zeros(3).AsGeneric() is null).Should().BeTrue("AsGeneric returns null when the dtype differs"); + Action make = () => np.zeros(3).MakeGeneric(); + make.Should().Throw("MakeGeneric throws when the dtype differs"); + } + + [TestMethod] + public void Scalars_IntegerIndexReducesOneDimension() + { + np.arange(5)[2].ndim.Should().Be(0, "1-D a[i] → 0-d"); + np.arange(20).reshape(4, 5)[1].shape.Should().Equal(new long[] { 5 }, "2-D a[i] → 1-D row"); + np.arange(24).reshape(2, 3, 4)[0].shape.Should().Equal(new long[] { 3, 4 }, "3-D a[i] → 2-D slab"); + } + + [TestMethod] + public void Broadcast_IsReadOnly() + { + var b = np.broadcast_to(np.arange(3), new Shape(2, 3)); + b.Shape.IsWriteable.Should().BeFalse("broadcast views are read-only"); + Action act = () => b[0, 0] = 99; + act.Should().Throw().WithMessage("*read-only*"); + } + + [TestMethod] + public void Operators_AllCompoundAssignmentsWork() + { + var a = np.array(new[] { 12 }); + a += 3; a.GetInt32(0).Should().Be(15); + a -= 5; a.GetInt32(0).Should().Be(10); + a *= 4; a.GetInt32(0).Should().Be(40); + a %= 7; a.GetInt32(0).Should().Be(5); + a &= np.array(new[] { 6 }); a.GetInt32(0).Should().Be(4); + a |= np.array(new[] { 1 }); a.GetInt32(0).Should().Be(5); + a ^= np.array(new[] { 3 }); a.GetInt32(0).Should().Be(6); + a <<= 2; a.GetInt32(0).Should().Be(24); + a >>= 1; a.GetInt32(0).Should().Be(12); + + // /= promotes to double, exactly like the a / b operator + var d = np.array(new[] { 10 }); + d /= 4; + d.typecode.Should().Be(NPTypeCode.Double); + ((double)d[0]).Should().Be(2.5); + } + } +} diff --git a/test/NumSharp.UnitTest/IO/AllocationTests.cs b/test/NumSharp.UnitTest/IO/AllocationTests.cs new file mode 100644 index 000000000..755459e56 --- /dev/null +++ b/test/NumSharp.UnitTest/IO/AllocationTests.cs @@ -0,0 +1,40 @@ +using System; + +namespace NumSharp.UnitTest.IO +{ + /// + /// Helper for tests that assert on how much an operation allocates. + /// + internal static class AllocationTests + { + /// + /// The smallest managed allocation made across several runs. + /// + /// + /// is process-wide, not per-operation, so a + /// background or finalizer thread allocating inside the measurement window inflates a single + /// reading — the difference between a real regression and a neighbour's garbage is invisible + /// from one sample. Noise can only ever ADD, so the minimum of several runs is the closest + /// thing to the operation's true cost and cannot be pushed over a threshold by unrelated + /// work. This is the same best-of-rounds rule the repo's benchmarks use. + /// + /// The first run is discarded: it pays for JIT and first-touch, which are real but one-off. + /// + public static long MinAllocated(Action action, int rounds = 5) + { + action(); // warm: JIT + first-touch + + long min = long.MaxValue; + for (int i = 0; i < rounds; i++) + { + long before = GC.GetTotalAllocatedBytes(precise: true); + action(); + long allocated = GC.GetTotalAllocatedBytes(precise: true) - before; + if (allocated < min) + min = allocated; + } + + return min; + } + } +} diff --git a/test/NumSharp.UnitTest/IO/NpyFormatEdgeCaseTests.cs b/test/NumSharp.UnitTest/IO/NpyFormatEdgeCaseTests.cs new file mode 100644 index 000000000..284df5aac --- /dev/null +++ b/test/NumSharp.UnitTest/IO/NpyFormatEdgeCaseTests.cs @@ -0,0 +1,394 @@ +using System; +using System.IO; +using System.Linq; +using Microsoft.VisualStudio.TestTools.UnitTesting; +using NumSharp.IO; + +namespace NumSharp.UnitTest.IO +{ + /// + /// Battle-tested edge cases for .npy/.npz format handling. + /// + [TestClass] + public class NpyFormatEdgeCaseTests + { + #region Sliced and Non-Contiguous Arrays + + [TestMethod] + public void Save_Load_SlicedArray_WithOffset() + { + // Regression test for Shape.offset bug + var arr = np.arange(20).reshape(4, 5); + var sliced = arr["1:3", "1:4"]; // Shape: [2, 3], offset: 6 + + Assert.IsFalse(sliced.Shape.IsContiguous); + Assert.AreEqual(6, sliced.Shape.offset); + + using var ms = new MemoryStream(); + np.save(ms, sliced); + ms.Position = 0; + var loaded = np.load_npy(ms); + + Assert.IsTrue(new long[] { 2, 3 }.SequenceEqual(loaded.shape)); + // np.arange yields Int64, so read the element back at that dtype. + var slicedFlat = sliced.flatten(); + var loadedFlat = loaded.flatten(); + Assert.AreEqual(6L, slicedFlat.GetAtIndex(0)); + Assert.AreEqual(6L, loadedFlat.GetAtIndex(0)); + Assert.IsTrue(np.array_equal(sliced, loaded)); + } + + [TestMethod] + public void Save_Load_ColumnSlice() + { + var arr = np.arange(20).reshape(4, 5); + var colSlice = arr[":, 1:3"]; // All rows, columns 1-2 + + Assert.IsFalse(colSlice.Shape.IsContiguous); + + using var ms = new MemoryStream(); + np.save(ms, colSlice); + ms.Position = 0; + var loaded = np.load_npy(ms); + + Assert.IsTrue(np.array_equal(colSlice, loaded)); + } + + [TestMethod] + public void Save_Load_StepSlice() + { + var arr = np.arange(10); + var stepped = arr["::2"]; // Every other element + + Assert.AreEqual(5, stepped.size); + + using var ms = new MemoryStream(); + np.save(ms, stepped); + ms.Position = 0; + var loaded = np.load_npy(ms); + + Assert.IsTrue(np.array_equal(stepped, loaded)); + } + + [TestMethod] + public void Save_Load_TransposedArray() + { + var arr = np.arange(12).reshape(3, 4); + var transposed = arr.T; + + Assert.IsFalse(transposed.Shape.IsContiguous); + + using var ms = new MemoryStream(); + np.save(ms, transposed); + ms.Position = 0; + var loaded = np.load_npy(ms); + + Assert.IsTrue(new long[] { 4, 3 }.SequenceEqual(loaded.shape)); + Assert.IsTrue(np.array_equal(transposed, loaded)); + } + + [TestMethod] + public void Save_Load_BroadcastView() + { + var arr = np.arange(3); + var broadcasted = np.broadcast_to(arr, (3, 3)); + + Assert.IsTrue(broadcasted.Shape.IsBroadcasted); + + using var ms = new MemoryStream(); + np.save(ms, broadcasted); + ms.Position = 0; + var loaded = np.load_npy(ms); + + Assert.IsTrue(np.array_equal(broadcasted, loaded)); + } + + #endregion + + #region Special Float Values + + [TestMethod] + public void Save_Load_SpecialFloatValues() + { + var arr = np.array(new double[] { + double.NaN, + double.PositiveInfinity, + double.NegativeInfinity, + 0.0, + double.MaxValue, + double.MinValue, + double.Epsilon + }); + + using var ms = new MemoryStream(); + np.save(ms, arr); + ms.Position = 0; + var loaded = np.load_npy(ms); + + Assert.IsTrue(double.IsNaN(loaded.GetDouble(0))); + Assert.IsTrue(double.IsPositiveInfinity(loaded.GetDouble(1))); + Assert.IsTrue(double.IsNegativeInfinity(loaded.GetDouble(2))); + Assert.AreEqual(0.0, loaded.GetDouble(3)); + Assert.AreEqual(double.MaxValue, loaded.GetDouble(4)); + Assert.AreEqual(double.MinValue, loaded.GetDouble(5)); + Assert.AreEqual(double.Epsilon, loaded.GetDouble(6)); + } + + #endregion + + #region Header Format Compliance + + [TestMethod] + public void Header_KeysSortedAlphabetically() + { + var arr = np.arange(6).reshape(2, 3); + using var ms = new MemoryStream(); + np.save(ms, arr); + var bytes = ms.ToArray(); + + int headerEnd = Array.IndexOf(bytes, (byte)'\n', 10); + string header = System.Text.Encoding.ASCII.GetString(bytes, 10, headerEnd - 10); + + int descrPos = header.IndexOf("'descr'"); + int fortranPos = header.IndexOf("'fortran_order'"); + int shapePos = header.IndexOf("'shape'"); + + Assert.IsTrue(descrPos < fortranPos); + Assert.IsTrue(fortranPos < shapePos); + } + + [TestMethod] + public void Header_HasTrailingComma() + { + var arr = np.arange(6).reshape(2, 3); + using var ms = new MemoryStream(); + np.save(ms, arr); + var bytes = ms.ToArray(); + + int headerEnd = Array.IndexOf(bytes, (byte)'\n', 10); + string header = System.Text.Encoding.ASCII.GetString(bytes, 10, headerEnd - 10).Trim(); + + Assert.IsTrue(header.EndsWith(", }")); + } + + [TestMethod] + public void Header_OneElementTupleHasTrailingComma() + { + var arr = np.arange(5); // 1D array + using var ms = new MemoryStream(); + np.save(ms, arr); + var bytes = ms.ToArray(); + + int headerEnd = Array.IndexOf(bytes, (byte)'\n', 10); + string header = System.Text.Encoding.ASCII.GetString(bytes, 10, headerEnd - 10); + + Assert.IsTrue(header.Contains("(5,)"), $"1D shape should be (5,) not (5): {header}"); + } + + [TestMethod] + public void Header_DataStartsAt64ByteBoundary() + { + var arr = np.arange(10); + using var ms = new MemoryStream(); + np.save(ms, arr); + var bytes = ms.ToArray(); + + int headerEnd = Array.LastIndexOf(bytes, (byte)'\n', bytes.Length - (int)(arr.size * arr.dtypesize) - 1); + int dataStart = headerEnd + 1; + + Assert.AreEqual(0, dataStart % 64, $"Data starts at byte {dataStart}, not 64-byte aligned"); + } + + #endregion + + #region True Scalars + + [TestMethod] + public void Save_Load_TrueScalar() + { + var scalar = new NDArray(NPTypeCode.Int32, Shape.NewScalar()); + scalar.SetAtIndex(42, 0); + + Assert.AreEqual(0, scalar.ndim); + Assert.AreEqual(0, scalar.shape.Length); + Assert.IsTrue(scalar.Shape.IsScalar); + + using var ms = new MemoryStream(); + np.save(ms, scalar); + + // Verify header has empty shape () + var bytes = ms.ToArray(); + int headerEnd = Array.IndexOf(bytes, (byte)'\n', 10); + string header = System.Text.Encoding.ASCII.GetString(bytes, 10, headerEnd - 10); + Assert.IsTrue(header.Contains("'shape': ()"), $"Scalar shape should be (): {header}"); + + // Verify round-trip + ms.Position = 0; + var loaded = np.load_npy(ms); + + Assert.AreEqual(0, loaded.ndim); + Assert.IsTrue(loaded.Shape.IsScalar); + Assert.AreEqual(42, loaded.GetInt32(0)); + } + + #endregion + + #region Multi-dimensional Shapes + + [DataTestMethod] + [DataRow(new long[] { 1 })] + [DataRow(new long[] { 1, 1, 1 })] + [DataRow(new long[] { 10 })] + [DataRow(new long[] { 2, 3, 4 })] + [DataRow(new long[] { 2, 3, 4, 5 })] + public void Save_Load_VariousShapes(long[] shape) + { + long size = shape.Aggregate(1L, (a, b) => a * b); + var arr = np.arange(size).reshape(shape); + + using var ms = new MemoryStream(); + np.save(ms, arr); + ms.Position = 0; + var loaded = np.load_npy(ms); + + Assert.IsTrue(shape.SequenceEqual(loaded.shape)); + Assert.IsTrue(np.array_equal(arr, loaded)); + } + + #endregion + + #region Empty Arrays + + [TestMethod] + public void Save_Load_Empty1D() + { + var empty = np.array(Array.Empty()); + + using var ms = new MemoryStream(); + np.save(ms, empty); + ms.Position = 0; + var loaded = np.load_npy(ms); + + Assert.AreEqual(0, loaded.size); + } + + [TestMethod] + public void Save_Load_EmptyMultiDimensional() + { + var empty = np.zeros(new long[] { 0, 3, 4 }, NPTypeCode.Double); + + using var ms = new MemoryStream(); + np.save(ms, empty); + ms.Position = 0; + var loaded = np.load_npy(ms); + + Assert.IsTrue(new long[] { 0, 3, 4 }.SequenceEqual(loaded.shape)); + Assert.AreEqual(0, loaded.size); + } + + #endregion + + #region Multiple Arrays in Stream + + [TestMethod] + public void Save_Load_MultipleArraysInOneStream() + { + var arr1 = np.array(new long[] { 1, 2, 3 }); + var arr2 = np.array(new long[] { 4, 5, 6, 7 }); + var arr3 = np.array(new long[] { 8, 9 }); + + using var ms = new MemoryStream(); + np.save(ms, arr1); + np.save(ms, arr2); + np.save(ms, arr3); + + ms.Position = 0; + var loaded1 = np.load_npy(ms); + var loaded2 = np.load_npy(ms); + var loaded3 = np.load_npy(ms); + + Assert.IsTrue(np.array_equal(arr1, loaded1)); + Assert.IsTrue(np.array_equal(arr2, loaded2)); + Assert.IsTrue(np.array_equal(arr3, loaded3)); + } + + #endregion + + #region Error Handling + + [TestMethod] + public void Load_InvalidMagic_ThrowsFormatException() + { + using var ms = new MemoryStream(new byte[] { 0, 1, 2, 3, 4, 5, 6, 7 }); + Assert.ThrowsException(() => np.load(ms)); + } + + /// + /// A file that starts correctly but stops short is a FORMAT error, not an EOF. + /// + /// + /// NumPy draws this line deliberately and we follow it: a truncated-but-present file raises + /// ValueError ("EOF: reading magic string, expected 8 bytes got 6" for these exact bytes), + /// while EOFError is reserved for "No data left in file" — a wholly empty read at the + /// dispatch level. Mapping both onto EndOfStreamException, as this test used to expect, + /// throws that distinction away: "this file is corrupt" and "there is nothing here" are + /// different answers, and only the second is a normal end-of-input. + /// + [TestMethod] + public void Load_TruncatedFile_ThrowsFormatException() + { + using var ms = new MemoryStream(new byte[] { 0x93, (byte)'N', (byte)'U', (byte)'M', (byte)'P', (byte)'Y' }); + + var e = Assert.ThrowsException(() => np.load(ms)); + StringAssert.Contains(e.Message, "EOF: reading magic string, expected 8 bytes got 6"); + } + + /// An EMPTY file is the one case that really is an end-of-stream. + [TestMethod] + public void Load_EmptyFile_ThrowsEndOfStreamException() + { + using var ms = new MemoryStream(Array.Empty()); + + var e = Assert.ThrowsException(() => np.load(ms)); + StringAssert.Contains(e.Message, "No data left in file"); + } + + [TestMethod] + public void Save_DecimalType_ThrowsNotSupportedException() + { + var arr = np.array(new decimal[] { 1.5m, 2.5m }); + using var ms = new MemoryStream(); + Assert.ThrowsException(() => np.save(ms, arr)); + } + + #endregion + + #region All Supported dtypes + + [DataTestMethod] + [DataRow(NPTypeCode.Boolean)] + [DataRow(NPTypeCode.Byte)] + [DataRow(NPTypeCode.Int16)] + [DataRow(NPTypeCode.UInt16)] + [DataRow(NPTypeCode.Int32)] + [DataRow(NPTypeCode.UInt32)] + [DataRow(NPTypeCode.Int64)] + [DataRow(NPTypeCode.UInt64)] + [DataRow(NPTypeCode.Single)] + [DataRow(NPTypeCode.Double)] + public void Save_Load_AllSupportedDtypes(NPTypeCode typeCode) + { + var arr = np.zeros(new long[] { 3 }, typeCode); + + using var ms = new MemoryStream(); + np.save(ms, arr); + ms.Position = 0; + var loaded = np.load_npy(ms); + + Assert.AreEqual(typeCode, loaded.typecode); + Assert.AreEqual(arr.size, loaded.size); + } + + #endregion + } +} diff --git a/test/NumSharp.UnitTest/IO/NpyFormatVersionTests.cs b/test/NumSharp.UnitTest/IO/NpyFormatVersionTests.cs new file mode 100644 index 000000000..38d1d31ac --- /dev/null +++ b/test/NumSharp.UnitTest/IO/NpyFormatVersionTests.cs @@ -0,0 +1,132 @@ +using System; +using System.IO; +using Microsoft.VisualStudio.TestTools.UnitTesting; +using NumSharp.IO; + +namespace NumSharp.UnitTest.IO +{ + /// + /// Tests for .npy format version handling (v1.0, v2.0, v3.0). + /// + [TestClass] + public class NpyFormatVersionTests + { + private const string TestDir = "test_compat"; + + [TestMethod] + public void ParseVersion2_LargeHeader() + { + var path = Path.Combine(TestDir, "version2_large_header.npy"); + // Fail loudly rather than skipping: without the fixture this test asserts nothing, + // and it used to pass silently on any checkout that lacked it. + Assert.IsTrue(File.Exists(path), $"the NumPy fixture 'version2_large_header.npy' is missing from {Path.GetFullPath(TestDir)}; this test asserts nothing without it."); + + // Read and verify the version + using var stream = File.OpenRead(path); + var version = NpyFormat.ReadMagic(stream); + + Assert.AreEqual(2, version.Major, "Should be version 2.0"); + Assert.AreEqual(0, version.Minor); + + // Try to read the header (will fail on dtype, but should parse header correctly) + try + { + var header = NpyFormat.ReadArrayHeader(stream, version, maxHeaderSize: 100000); + Console.WriteLine($"Header parsed successfully!"); + Console.WriteLine($" Shape: [{string.Join(", ", header.Shape)}]"); + Console.WriteLine($" FortranOrder: {header.FortranOrder}"); + // If we get here, the v2.0 header was parsed (4-byte length field worked) + Assert.Fail("Expected dtype parse error for structured array"); + } + catch (FormatException ex) when (ex.Message.Contains("descr")) + { + // Expected - structured dtypes not supported, but header was parsed! + Console.WriteLine($"Header parsing reached dtype: {ex.Message}"); + Assert.IsTrue(true, "Version 2.0 header length field (4 bytes) was parsed correctly"); + } + catch (NotSupportedException ex) + { + // Also acceptable - means we got past header parsing + Console.WriteLine($"Dtype not supported (expected): {ex.Message}"); + Assert.IsTrue(true, "Version 2.0 format parsed, dtype not supported"); + } + } + + [TestMethod] + public void ParseVersion3_UnicodeFieldNames() + { + var path = Path.Combine(TestDir, "version3_unicode.npy"); + // Fail loudly rather than skipping: without the fixture this test asserts nothing, + // and it used to pass silently on any checkout that lacked it. + Assert.IsTrue(File.Exists(path), $"the NumPy fixture 'version3_unicode.npy' is missing from {Path.GetFullPath(TestDir)}; this test asserts nothing without it."); + + // Read and verify the version + using var stream = File.OpenRead(path); + var version = NpyFormat.ReadMagic(stream); + + Assert.AreEqual(3, version.Major, "Should be version 3.0"); + Assert.AreEqual(0, version.Minor); + + // Try to read the header (UTF-8 encoded) + try + { + var header = NpyFormat.ReadArrayHeader(stream, version, maxHeaderSize: 10000); + Console.WriteLine($"Header parsed successfully!"); + Console.WriteLine($" Shape: [{string.Join(", ", header.Shape)}]"); + // If we get here, UTF-8 decoding worked + Assert.Fail("Expected dtype parse error for structured array"); + } + catch (FormatException ex) when (ex.Message.Contains("descr")) + { + // Expected - structured dtypes have complex descr format + Console.WriteLine($"Header parsing reached dtype: {ex.Message}"); + Assert.IsTrue(true, "Version 3.0 UTF-8 decoding worked correctly"); + } + catch (NotSupportedException ex) + { + Console.WriteLine($"Dtype not supported (expected): {ex.Message}"); + Assert.IsTrue(true, "Version 3.0 format parsed, dtype not supported"); + } + } + + [TestMethod] + public void Version1_SimpleArray_RoundTrip() + { + // Verify we write version 1.0 by default for simple arrays + var arr = np.arange(100); + + using var ms = new MemoryStream(); + np.save(ms, arr); + + ms.Position = 0; + var version = NpyFormat.ReadMagic(ms); + + Assert.AreEqual(1, version.Major, "Simple arrays should use version 1.0"); + Assert.AreEqual(0, version.Minor); + + // Verify round-trip + ms.Position = 0; + var loaded = np.load_npy(ms); + Assert.IsTrue(np.array_equal(arr, loaded)); + } + + [TestMethod] + public void HeaderAlignment_Is64Bytes() + { + var arr = np.arange(10); + + using var ms = new MemoryStream(); + np.save(ms, arr); + + // Data should start at a 64-byte aligned offset + var bytes = ms.ToArray(); + + // Find where the header ends (newline before data) + int headerEnd = Array.LastIndexOf(bytes, (byte)'\n', bytes.Length - (int)(arr.size * arr.dtypesize) - 1); + int dataStart = headerEnd + 1; + + Console.WriteLine($"Header ends at byte {headerEnd}, data starts at byte {dataStart}"); + Assert.AreEqual(0, dataStart % 64, $"Data should start at 64-byte aligned offset, but starts at {dataStart}"); + } + } +} diff --git a/test/NumSharp.UnitTest/IO/NpyLargeFileTests.cs b/test/NumSharp.UnitTest/IO/NpyLargeFileTests.cs new file mode 100644 index 000000000..c0e783df1 --- /dev/null +++ b/test/NumSharp.UnitTest/IO/NpyLargeFileTests.cs @@ -0,0 +1,172 @@ +using System; +using System.IO; +using Microsoft.VisualStudio.TestTools.UnitTesting; +using NumSharp; +using NumSharp.IO; + +namespace NumSharp.UnitTest.IO +{ + /// + /// Files past the 2 GB and 4 GB walls. + /// + /// + /// Three separate 32-bit limits sit under this format and each bites at a different size: + /// + /// 2 GB — MemoryStream and byte[] cannot exceed it, so any code that + /// buffers a whole member or file into managed memory caps out here. NumSharp's array + /// data is unmanaged, so nothing on the .npy path needs a managed buffer bigger than one + /// 256 KB chunk. + /// 4 GB — a zip entry above 0xFFFFFFFF needs Zip64 records. NumPy passes + /// force_zip64=True for exactly this (numpy gh-10776); .NET's ZipArchive emits + /// them on demand. + /// int.MaxValue elements — every offset, count and stride in the read/write loops + /// must be 64-bit. + /// + /// The multi-gigabyte tests are (excluded from CI) and want + /// ~8 GB of RAM plus ~6 GB of scratch disk. + /// is the cheap proxy that runs everywhere: it guards the same mechanism the 2 GB wall exposed. + /// + [TestClass] + public class NpyLargeFileTests + { + private string _dir; + + [TestInitialize] + public void Setup() + { + _dir = Path.Combine(Path.GetTempPath(), "numsharp_big_" + Guid.NewGuid().ToString("N")); + Directory.CreateDirectory(_dir); + } + + [TestCleanup] + public void Cleanup() + { + try { Directory.Delete(_dir, recursive: true); } + catch (IOException) { } + } + + /// + /// Reading an npz member must stream it, not buffer it. + /// + /// + /// This is the cheap guard for a bug that only showed at 5 GB: members used to be copied into + /// a MemoryStream so the magic could be sniffed by seeking, which capped members at + /// 2 GB — NumSharp would happily WRITE a 5 GiB npz and then fail to read it back with + /// "Stream was too long", while NumPy read the same file fine. Buffering also doubled the + /// peak cost of every ordinary load. A 16 MB member is enough to catch a regression: if the + /// member is being buffered, the managed allocation tracks its size. + /// + [TestMethod] + [TestCategory("NpyOracle")] + [DoNotParallelize] + public void Npz_StreamsMembers_RatherThanBufferingThem() + { + const int n = 16 * 1024 * 1024; + byte[] archive = np.savez(np.zeros(new Shape(n), NPTypeCode.Byte)); + + long allocated = AllocationTests.MinAllocated(() => + { + using NpzFile npz = np.load_npz(archive); + Assert.AreEqual(n, npz["arr_0"].size); + }); + + Assert.IsTrue(allocated < n / 4, + $"loading a {n / 1024 / 1024} MB npz member allocated {allocated:N0} managed bytes " + + $"({allocated / (double)n:F2}x the member). The member must be streamed into the reader: " + + "buffering it caps members at MemoryStream's 2 GB limit and doubles the cost of every load."); + } + + /// A .npy larger than 4 GB round-trips, and every 32-bit boundary inside it survives. + [TestMethod] + [TestCategory("NpyOracle")] + [HighMemory] + [LongIndexing] + [DoNotParallelize] + public void Npy_5GiB_RoundTrips() + { + const long n = 5L * 1024 * 1024 * 1024; + string file = Path.Combine(_dir, "big.npy"); + + (long Offset, byte Value)[] probes = Probes(n); + + Stamp(file, n, probes, npz: false); + + NDArray loaded = np.load_npy(file); + Assert.AreEqual(NPTypeCode.Byte, loaded.typecode); + Assert.AreEqual(n, loaded.size); + CollectionAssert.AreEqual(new[] { n }, loaded.shape); + AssertProbes(loaded, probes); + + // 128-byte header + the data, exactly. + Assert.AreEqual(128 + n, new FileInfo(file).Length); + } + + /// + /// An npz whose member exceeds 4 GB round-trips — the Zip64 case the issue calls out. + /// + [TestMethod] + [TestCategory("NpyOracle")] + [HighMemory] + [LongIndexing] + [DoNotParallelize] + public void Npz_5GiBMember_RoundTrips() + { + const long n = 5L * 1024 * 1024 * 1024; + string file = Path.Combine(_dir, "big.npz"); + + (long Offset, byte Value)[] probes = Probes(n); + + Stamp(file, n, probes, npz: true); + + Assert.IsTrue(new FileInfo(file).Length > uint.MaxValue, + "the archive must exceed 4 GiB, or it is not exercising Zip64 at all"); + + using NpzFile npz = np.load_npz(file); + NDArray loaded = npz["big"]; + Assert.AreEqual(NPTypeCode.Byte, loaded.typecode); + Assert.AreEqual(n, loaded.size); + AssertProbes(loaded, probes); + } + + // Offsets that straddle every 32-bit boundary a narrowing cast would trip over. + private static (long, byte)[] Probes(long n) => new[] + { + (0L, (byte)0x11), + (int.MaxValue - 1L, (byte)0x22), + ((long)int.MaxValue, (byte)0x33), + (int.MaxValue + 1L, (byte)0x44), // int overflow + (3_000_000_000L, (byte)0x55), + (uint.MaxValue - 1L, (byte)0x66), + (uint.MaxValue + 1L, (byte)0x77), // uint overflow + (n - 1, (byte)0x88), // last element + }; + + // Build the array, stamp the probes, save, then drop it so the load does not need 2x the RAM. + private static unsafe void Stamp(string file, long n, (long Offset, byte Value)[] probes, bool npz) + { + var a = new NDArray(NPTypeCode.Byte, new Shape(n), fillZeros: false); + byte* p = (byte*)a.Storage.Address; + foreach ((long off, byte val) in probes) + p[off] = val; + + if (npz) + np.savez(file, new System.Collections.Generic.Dictionary { ["big"] = a }); + else + np.save(file, a); + + GC.KeepAlive(a); + a = null; + GC.Collect(2, GCCollectionMode.Forced, blocking: true); + GC.WaitForPendingFinalizers(); + GC.Collect(2, GCCollectionMode.Forced, blocking: true); + } + + private static unsafe void AssertProbes(NDArray a, (long Offset, byte Value)[] probes) + { + byte* p = (byte*)a.Storage.Address; + foreach ((long off, byte val) in probes) + Assert.AreEqual(val, p[off], $"element at flat index {off:N0} did not survive the round-trip"); + GC.KeepAlive(a); + } + } +} diff --git a/test/NumSharp.UnitTest/IO/NpyOracleCorpus.cs b/test/NumSharp.UnitTest/IO/NpyOracleCorpus.cs new file mode 100644 index 000000000..dbb0e6c79 --- /dev/null +++ b/test/NumSharp.UnitTest/IO/NpyOracleCorpus.cs @@ -0,0 +1,239 @@ +using System; +using System.Collections.Generic; +using System.IO; +using System.IO.Compression; +using System.Linq; +using System.Runtime.InteropServices; +using System.Text.Json; +using NumSharp; +using NumSharp.Backends; + +namespace NumSharp.UnitTest.IO +{ + /// + /// Loads the committed NumPy oracle (IO/corpus/npy_oracle.zip) and rebuilds arrays from it. + /// + /// + /// The zip holds real NumPy 2.4.2 np.save / np.savez output plus a manifest saying, + /// for each case, what NumSharp must load, what bytes it must write, or which error it must + /// raise. Written by test/oracle/gen_npy_oracle.py; no Python runs at test time. + /// + internal static class NpyOracleCorpus + { + private static readonly Lazy<(NpyCase[] Cases, string NumpyVersion)> _corpus = new(Load); + + public static NpyCase[] Cases => _corpus.Value.Cases; + public static string NumpyVersion => _corpus.Value.NumpyVersion; + + public static IEnumerable OfKind(params string[] kinds) => + Cases.Where(c => kinds.Contains(c.Kind, StringComparer.Ordinal)); + + private static (NpyCase[], string) Load() + { + string path = Path.Combine(AppContext.BaseDirectory, "IO", "corpus", "npy_oracle.zip"); + if (!File.Exists(path)) + throw new FileNotFoundException( + $"NumPy oracle corpus not found at '{path}'. It is committed; rebuild the test project to " + + "copy it, or regenerate it with: python test/oracle/gen_npy_oracle.py", path); + + // Buffer the whole zip: it is ~1 MB and every case needs random access to it. + byte[] raw = File.ReadAllBytes(path); + using var zip = new ZipArchive(new MemoryStream(raw), ZipArchiveMode.Read); + + string json; + using (var reader = new StreamReader(zip.GetEntry("manifest.json")!.Open())) + json = reader.ReadToEnd(); + + using var doc = JsonDocument.Parse(json); + JsonElement root = doc.RootElement; + + var cases = new List(); + foreach (JsonElement e in root.GetProperty("cases").EnumerateArray()) + cases.Add(NpyCase.Parse(e, zip)); + + return (cases.ToArray(), root.GetProperty("numpy_version").GetString()); + } + + /// Rebuild an array holding exactly , in the requested layout. + /// + /// is the manifest's canonical form: logical values, native byte order, + /// C-order. Writing it into a C-contiguous array and then copying to 'F' produces an + /// F-contiguous array with the same logical values — which is what a fortran_order case needs. + /// + public static unsafe NDArray Build(NPTypeCode typeCode, long[] shape, byte[] data, bool fortranOrder) + { + var arr = new NDArray(typeCode, new Shape(shape)); + if (data.Length > 0) + Marshal.Copy(data, 0, (IntPtr)arr.Storage.Address, data.Length); + return fortranOrder ? arr.copy('F') : arr; + } + + /// The array's logical values as raw C-order bytes — the form the manifest stores. + public static unsafe byte[] RawBytes(NDArray a) + { + NDArray src = a.Shape.IsContiguous && a.Shape.offset == 0 ? a : a.copy('C'); + long len = src.size * src.dtypesize; + var bytes = new byte[len]; + if (len > 0) + Marshal.Copy((IntPtr)src.Storage.Address, bytes, 0, checked((int)len)); + GC.KeepAlive(src); + return bytes; + } + } + + /// One oracle case. See gen_npy_oracle.py's module docstring for the schema. + internal sealed class NpyCase + { + public string Name { get; private init; } + public string Kind { get; private init; } + public string Note { get; private init; } + + /// The exact bytes NumPy produced. + public byte[] Bytes { get; private init; } + + public NPTypeCode? NsDtype { get; private init; } + public long[] Shape { get; private init; } + public bool FortranOrder { get; private init; } + public string Descr { get; private init; } + public byte[] Version { get; private init; } + + /// Logical values, native byte order, C-order — what NumSharp must hold after loading. + public byte[] NsBytes { get; private init; } + + /// Whether NumSharp's writer must reproduce exactly. + public bool WriteExact { get; private init; } + + /// Text the load error must contain, or null if the case must load successfully. + public string LoadError { get; private init; } + + /// Which entry point the error case exercises: load or load_npy. + public string LoadVia { get; private init; } + + /// Per-case header-size cap, or null for the default. + public long? MaxHeaderSize { get; private init; } + + /// For npz cases: member name → expected array. + public Dictionary Entries { get; private init; } + + /// For npz cases: member name → expected raw bytes (non-.npy members). + public Dictionary RawEntries { get; private init; } + + /// + /// For npz cases: the exact expected .files, when it cannot be derived from + /// — duplicate member names appear twice in .files but resolve + /// to a single entry. + /// + public string[] FilesOverride { get; private init; } + + /// + /// Whether npz["k"] and npz["k.npy"] must be the same array. False for archives + /// with ambiguous names, where they intentionally resolve to different entries. + /// + public bool SuffixAlias { get; private init; } + + /// For the multi-array stream case: the arrays in the order they were appended. + public NpyMember[] Sequence { get; private init; } + + public bool Compressed { get; private init; } + + public override string ToString() => Name ?? ""; + + public static NpyCase Parse(JsonElement e, ZipArchive zip) + { + byte[] ReadEntry(string entry) + { + using Stream s = zip.GetEntry(entry)!.Open(); + using var ms = new MemoryStream(); + s.CopyTo(ms); + return ms.ToArray(); + } + + return new NpyCase + { + Name = e.GetProperty("name").GetString(), + Kind = e.GetProperty("kind").GetString(), + Note = Str(e, "note"), + Bytes = ReadEntry(e.GetProperty("file").GetString()), + NsDtype = Str(e, "ns_dtype") is { } d ? Enum.Parse(d) : null, + Shape = Longs(e, "shape"), + FortranOrder = Bool(e, "fortran_order") ?? false, + Descr = Str(e, "descr"), + Version = Bytes8(e, "version"), + NsBytes = Hex(Str(e, "ns_bytes")), + WriteExact = Bool(e, "write_exact") ?? false, + LoadError = Str(e, "load_error"), + LoadVia = Str(e, "load_via") ?? "load", + MaxHeaderSize = Long(e, "max_header_size"), + Compressed = Bool(e, "compressed") ?? false, + Entries = Members(e, "entries"), + RawEntries = RawMembers(e, "raw_entries"), + FilesOverride = e.TryGetProperty("files", out JsonElement fo) && fo.ValueKind == JsonValueKind.Array + ? fo.EnumerateArray().Select(x => x.GetString()).ToArray() + : null, + SuffixAlias = Bool(e, "suffix_alias") ?? true, + Sequence = e.TryGetProperty("sequence", out JsonElement s) && s.ValueKind == JsonValueKind.Array + ? s.EnumerateArray().Select(NpyMember.Parse).ToArray() + : null, + }; + } + + private static string Str(JsonElement e, string name) => + e.TryGetProperty(name, out JsonElement v) && v.ValueKind == JsonValueKind.String ? v.GetString() : null; + + private static bool? Bool(JsonElement e, string name) => + e.TryGetProperty(name, out JsonElement v) && (v.ValueKind == JsonValueKind.True || v.ValueKind == JsonValueKind.False) + ? v.GetBoolean() : null; + + private static long? Long(JsonElement e, string name) => + e.TryGetProperty(name, out JsonElement v) && v.ValueKind == JsonValueKind.Number ? v.GetInt64() : null; + + private static long[] Longs(JsonElement e, string name) => + e.TryGetProperty(name, out JsonElement v) && v.ValueKind == JsonValueKind.Array + ? v.EnumerateArray().Select(x => x.GetInt64()).ToArray() : null; + + private static byte[] Bytes8(JsonElement e, string name) => + e.TryGetProperty(name, out JsonElement v) && v.ValueKind == JsonValueKind.Array + ? v.EnumerateArray().Select(x => (byte)x.GetInt32()).ToArray() : null; + + private static Dictionary Members(JsonElement e, string name) + { + if (!e.TryGetProperty(name, out JsonElement v) || v.ValueKind != JsonValueKind.Object) + return null; + var d = new Dictionary(StringComparer.Ordinal); + foreach (JsonProperty p in v.EnumerateObject()) + d[p.Name] = NpyMember.Parse(p.Value); + return d; + } + + private static Dictionary RawMembers(JsonElement e, string name) + { + if (!e.TryGetProperty(name, out JsonElement v) || v.ValueKind != JsonValueKind.Object) + return null; + var d = new Dictionary(StringComparer.Ordinal); + foreach (JsonProperty p in v.EnumerateObject()) + d[p.Name] = Hex(p.Value.GetString()); + return d; + } + + internal static byte[] Hex(string hex) + { + if (hex == null) return null; + return System.Convert.FromHexString(hex); + } + } + + /// One array inside an npz archive or an appended stream. + internal sealed class NpyMember + { + public NPTypeCode NsDtype { get; private init; } + public long[] Shape { get; private init; } + public byte[] NsBytes { get; private init; } + + public static NpyMember Parse(JsonElement e) => new() + { + NsDtype = Enum.Parse(e.GetProperty("ns_dtype").GetString()!), + Shape = e.GetProperty("shape").EnumerateArray().Select(x => x.GetInt64()).ToArray(), + NsBytes = NpyCase.Hex(e.GetProperty("ns_bytes").GetString()), + }; + } +} diff --git a/test/NumSharp.UnitTest/IO/NpyOracleTests.cs b/test/NumSharp.UnitTest/IO/NpyOracleTests.cs new file mode 100644 index 000000000..5c1377373 --- /dev/null +++ b/test/NumSharp.UnitTest/IO/NpyOracleTests.cs @@ -0,0 +1,494 @@ +using System; +using System.Collections.Generic; +using System.IO; +using System.Linq; +using System.Text; +using Microsoft.VisualStudio.TestTools.UnitTesting; +using NumSharp; +using NumSharp.IO; + +namespace NumSharp.UnitTest.IO +{ + /// + /// The .npy/.npz differential gate: replay every committed NumPy 2.4.2 case through NumSharp. + /// + /// + /// Three independent claims per case, as recorded in the manifest: + /// + /// read — loading the file yields NumPy's dtype, shape and exact bytes. + /// write — saving the same array reproduces NumPy's file BYTE FOR BYTE. This is the + /// strong claim: not "NumPy can read it" but "it is indistinguishable from NumPy's output". + /// error — an unsupported or malformed file fails with NumPy's verbatim message. + /// + /// Each test reports every divergence at once rather than stopping at the first, so a regression + /// shows its full blast radius. + /// + [TestClass] + public class NpyOracleTests + { + /// Every .npy case loads to NumPy's dtype, shape and bytes. + [TestMethod] + [TestCategory("NpyOracle")] + public void Read_AllCases() + { + var f = new Failures("read"); + + foreach (NpyCase c in NpyOracleCorpus.OfKind("npy").Where(c => c.LoadError == null)) + { + try + { + var loaded = (NDArray)np.load(c.Bytes); + + if (loaded.typecode != c.NsDtype) + f.Add(c, $"dtype: expected {c.NsDtype}, got {loaded.typecode}"); + else if (!loaded.shape.SequenceEqual(c.Shape)) + f.Add(c, $"shape: expected ({string.Join(", ", c.Shape)}), got ({string.Join(", ", loaded.shape)})"); + else + { + byte[] got = NpyOracleCorpus.RawBytes(loaded); + if (!got.SequenceEqual(c.NsBytes)) + f.Add(c, "data: " + Diff(c.NsBytes, got)); + } + } + catch (Exception e) + { + f.Add(c, $"threw {e.GetType().Name}: {First(e.Message)}"); + } + } + + f.Assert(); + } + + /// + /// Saving the array produces the exact bytes NumPy produced — header, padding, alignment and + /// data. Covers every supported dtype, C and Fortran order, all three format versions and the + /// 0-d / empty / strided edges. + /// + [TestMethod] + [TestCategory("NpyOracle")] + public void Write_IsByteIdenticalToNumPy() + { + var f = new Failures("write"); + + foreach (NpyCase c in NpyOracleCorpus.OfKind("npy").Where(c => c.WriteExact)) + { + try + { + NDArray arr = NpyOracleCorpus.Build(c.NsDtype.Value, c.Shape, c.NsBytes, c.FortranOrder); + + // The corpus records which version NumPy chose. Pass it through only when it is not + // the default, so the common path exercises NumSharp's own version auto-selection. + var version = new NpyFormat.FormatVersion(c.Version[0], c.Version[1]); + using var ms = new MemoryStream(); + np.save_version(ms, arr, version == NpyFormat.FormatVersion.V1_0 ? null : version); + + byte[] got = ms.ToArray(); + if (!got.SequenceEqual(c.Bytes)) + f.Add(c, Diff(c.Bytes, got)); + } + catch (Exception e) + { + f.Add(c, $"threw {e.GetType().Name}: {First(e.Message)}"); + } + } + + f.Assert(); + } + + /// + /// Header-only cases: writer logic that no real array can reach. + /// + /// + /// Two branches are only observable here. The growth padding + /// (21 - len(repr(shape[0 or -1]))) is normally invisible — shrink the body and the + /// alignment padding grows to compensate — so a wrong growth axis passes every ordinary + /// test; it only shows on shapes that tip the header across a 64-byte bucket, and those need + /// 10^17 elements to allocate. Likewise the v1.0→v2.0 auto-selection boundary sits at ~21817 + /// dimensions. Both are driven through the header dict directly, exactly as NumPy's + /// _write_array_header was to produce the expected bytes. + /// + [TestMethod] + [TestCategory("NpyOracle")] + public void WriteHeader_MatchesNumPy() + { + var f = new Failures("header"); + + foreach (NpyCase c in NpyOracleCorpus.OfKind("header")) + { + try + { + var d = new Dictionary + { + ["descr"] = c.Descr, + ["fortran_order"] = c.FortranOrder, + ["shape"] = c.Shape, + }; + + using var ms = new MemoryStream(); + NpyFormat.WriteArrayHeader(ms, d); // no version => auto-select, as NumPy did + + byte[] got = ms.ToArray(); + if (!got.SequenceEqual(c.Bytes)) + f.Add(c, Diff(c.Bytes, got)); + else if (got[6] != c.Version[0]) + f.Add(c, $"version: expected {c.Version[0]}.{c.Version[1]}, got {got[6]}.{got[7]}"); + } + catch (Exception e) + { + f.Add(c, $"threw {e.GetType().Name}: {First(e.Message)}"); + } + } + + f.Assert(); + } + + /// + /// Saving a LIVE view — strided, reversed, offset, broadcast, transposed — produces NumPy's + /// bytes. + /// + /// + /// rebuilds each array from the manifest's + /// canonical C-order bytes, so the array it saves is always freshly contiguous (or an F-copy). + /// That never exercises the branch that matters most here: deciding what to do with an array + /// whose memory is not laid out the way the file needs. These recipes mirror the NumPy + /// expressions in gen_npy_oracle.py exactly, and are compared against the same files. + /// + [TestMethod] + [TestCategory("NpyOracle")] + public void Write_FromLiveViews_MatchesNumPy() + { + var f = new Failures("live-view write"); + + // case name -> the NumSharp equivalent of the NumPy expression that produced it + var recipes = new Dictionary> + { + // np.arange(12, dtype=np.int32)[::2] + ["strided_step2"] = () => Int32Range(12)["::2"], + // np.arange(6, dtype=np.float64)[::-1] + ["strided_reversed"] = () => np.arange(6).astype(typeof(double))["::-1"], + // np.arange(12, dtype=np.int32).reshape(3, 4)[:, ::2] + ["strided_2d_col"] = () => Int32Range(12).reshape(3, 4)[":, ::2"], + // np.arange(12, dtype=np.int32).reshape(3, 4)[1:, :] + ["strided_offset_row"] = () => Int32Range(12).reshape(3, 4)["1:, :"], + // np.broadcast_to(np.arange(3, dtype=np.int32), (4, 3)) + ["broadcast_view"] = () => np.broadcast_to(Int32Range(3), new Shape(4, 3)), + // np.arange(12, dtype=np.int32).reshape(3, 4).T -> F-contiguous, fortran_order: True + ["fortran_transposed_view"] = () => Int32Range(12).reshape(3, 4).T, + // np.asfortranarray(np.arange(5, dtype=np.int32)) -> 1-D is C-contig first: False + ["fortran_1d_is_c"] = () => Int32Range(5).copy('F'), + // np.asfortranarray(np.array(7, dtype=np.int32)) -> promoted to shape (1,), fortran_order: False + ["fortran_scalar_promoted_to_1d"] = () => np.array(new[] { 7 }), + }; + + foreach (KeyValuePair> r in recipes) + { + NpyCase c = NpyOracleCorpus.Cases.SingleOrDefault(x => x.Name == r.Key); + if (c == null) + { + f.Add(new NpyCase(), $"corpus case '{r.Key}' is missing — the recipe has no oracle to check against"); + continue; + } + + try + { + byte[] got = np.save(r.Value()); + if (!got.SequenceEqual(c.Bytes)) + f.Add(c, Diff(c.Bytes, got)); + } + catch (Exception e) + { + f.Add(c, $"threw {e.GetType().Name}: {First(e.Message)}"); + } + } + + f.Assert(); + } + + // np.arange yields Int64 (matching NumPy 2.x); the oracle recipes are int32. + private static NDArray Int32Range(int n) => np.arange(n).astype(typeof(int)); + + /// Malformed and unsupported files fail with NumPy's message. + [TestMethod] + [TestCategory("NpyOracle")] + public void Errors_MatchNumPy() + { + var f = new Failures("error"); + + foreach (NpyCase c in NpyOracleCorpus.Cases.Where(c => c.LoadError != null)) + { + long max = c.MaxHeaderSize ?? NpyFormat.MaxHeaderSize; + try + { + if (c.LoadVia == "load_npy") + np.load_npy(c.Bytes, max_header_size: max); + else + np.load(c.Bytes, max_header_size: max); + + f.Add(c, $"expected an error containing \"{c.LoadError}\", but the load succeeded"); + } + catch (Exception e) when (!(e is AssertFailedException)) + { + if (!e.Message.Contains(c.LoadError, StringComparison.Ordinal)) + f.Add(c, $"expected message to contain\n \"{c.LoadError}\"\n got {e.GetType().Name}:\n \"{First(e.Message)}\""); + } + } + + f.Assert(); + } + + /// Every .npy case survives a NumSharp save/load round-trip unchanged. + [TestMethod] + [TestCategory("NpyOracle")] + public void RoundTrip_PreservesEverything() + { + var f = new Failures("roundtrip"); + + foreach (NpyCase c in NpyOracleCorpus.OfKind("npy").Where(c => c.LoadError == null)) + { + try + { + var original = (NDArray)np.load(c.Bytes); + var reloaded = (NDArray)np.load(np.save(original)); + + if (reloaded.typecode != original.typecode) + f.Add(c, $"dtype: {original.typecode} -> {reloaded.typecode}"); + else if (!reloaded.shape.SequenceEqual(original.shape)) + f.Add(c, $"shape: ({string.Join(", ", original.shape)}) -> ({string.Join(", ", reloaded.shape)})"); + else if (!NpyOracleCorpus.RawBytes(reloaded).SequenceEqual(c.NsBytes)) + f.Add(c, "data changed across the round-trip: " + Diff(c.NsBytes, NpyOracleCorpus.RawBytes(reloaded))); + // A Fortran-order file must still be Fortran-order after a round-trip: the layout is + // part of what the format preserves. + else if (c.FortranOrder && !(reloaded.Shape.IsFContiguous && !reloaded.Shape.IsContiguous)) + f.Add(c, "fortran_order was not preserved across the round-trip"); + } + catch (Exception e) + { + f.Add(c, $"threw {e.GetType().Name}: {First(e.Message)}"); + } + } + + f.Assert(); + } + + /// NumPy's .npz archives load: lazily, by either key spelling, with .files stripped. + [TestMethod] + [TestCategory("NpyOracle")] + public void Npz_ReadsNumPyArchives() + { + var f = new Failures("npz"); + + foreach (NpyCase c in NpyOracleCorpus.OfKind("npz")) + { + try + { + using NpzFile npz = np.load_npz(c.Bytes); + + // .files normally follows from the members, except where the corpus pins it + // explicitly — a duplicated member name is listed twice but resolves once. + var expected = c.FilesOverride?.ToList() + ?? new List((c.Entries?.Keys ?? Enumerable.Empty()) + .Concat(c.RawEntries?.Keys ?? Enumerable.Empty())); + if (npz.Files.Count != expected.Count || expected.Any(k => !npz.Files.Contains(k))) + f.Add(c, $".files: expected [{string.Join(", ", expected)}], got [{string.Join(", ", npz.Files)}]"); + + foreach (KeyValuePair kv in c.Entries ?? new Dictionary()) + { + NDArray arr = npz[kv.Key]; + + if (arr.typecode != kv.Value.NsDtype) + f.Add(c, $"['{kv.Key}'] dtype: expected {kv.Value.NsDtype}, got {arr.typecode}"); + else if (!arr.shape.SequenceEqual(kv.Value.Shape)) + f.Add(c, $"['{kv.Key}'] shape: expected ({string.Join(", ", kv.Value.Shape)}), got ({string.Join(", ", arr.shape)})"); + else if (!NpyOracleCorpus.RawBytes(arr).SequenceEqual(kv.Value.NsBytes)) + f.Add(c, $"['{kv.Key}'] data: " + Diff(kv.Value.NsBytes, NpyOracleCorpus.RawBytes(arr))); + + // NumPy accepts the member's full name too, and caches: the same instance comes + // back. Archives with ambiguous names opt out — there '' and '.npy' + // deliberately resolve to DIFFERENT entries, which is the whole point of them. + if (c.SuffixAlias && !ReferenceEquals(npz[kv.Key + ".npy"], arr)) + f.Add(c, $"['{kv.Key}.npy'] did not resolve to the same cached array as ['{kv.Key}']"); + } + + foreach (KeyValuePair kv in c.RawEntries ?? new Dictionary()) + { + if (!npz.GetRawBytes(kv.Key).SequenceEqual(kv.Value)) + f.Add(c, $"['{kv.Key}'] raw bytes differ"); + if (npz.IsArray(kv.Key)) + f.Add(c, $"['{kv.Key}'] is not a .npy member but IsArray said it was"); + } + } + catch (Exception e) + { + f.Add(c, $"threw {e.GetType().Name}: {First(e.Message)}"); + } + } + + f.Assert(); + } + + /// + /// Arrays appended to one stream read back one at a time, in order, and the stream then + /// reports EOF — NumPy's np.save(f, a); np.save(f, b) idiom. + /// + [TestMethod] + [TestCategory("NpyOracle")] + public void Stream_MultipleArraysPerFile() + { + NpyCase c = NpyOracleCorpus.OfKind("sequence").Single(); + var f = new Failures("sequence"); + + using (var ms = new MemoryStream(c.Bytes)) + { + for (int i = 0; i < c.Sequence.Length; i++) + { + NDArray arr = np.load_npy(ms); + NpyMember want = c.Sequence[i]; + + if (arr.typecode != want.NsDtype) + f.Add(c, $"[{i}] dtype: expected {want.NsDtype}, got {arr.typecode}"); + else if (!arr.shape.SequenceEqual(want.Shape)) + f.Add(c, $"[{i}] shape: expected ({string.Join(", ", want.Shape)}), got ({string.Join(", ", arr.shape)})"); + else if (!NpyOracleCorpus.RawBytes(arr).SequenceEqual(want.NsBytes)) + f.Add(c, $"[{i}] data: " + Diff(want.NsBytes, NpyOracleCorpus.RawBytes(arr))); + } + + Assert.ThrowsException(() => np.load_npy(ms), + "reading past the last array must report EOF, as NumPy does"); + } + + // ...and NumSharp writes that same multi-array stream byte-for-byte. + using (var ms = new MemoryStream()) + { + foreach (NpyMember m in c.Sequence) + np.save(ms, NpyOracleCorpus.Build(m.NsDtype, m.Shape, m.NsBytes, false)); + + if (!ms.ToArray().SequenceEqual(c.Bytes)) + f.Add(c, "writing the three arrays to one stream: " + Diff(c.Bytes, ms.ToArray())); + } + + f.Assert(); + } + + /// + /// A hostile header-length field must not be trusted with the allocator: rejecting a 28-byte + /// file must not reserve the gigabytes it claims. + /// + /// + /// The error-message cases in pass either way — they would go + /// green even while allocating 1 GB per file — so the resource behaviour needs its own test. + /// NumPy handles this in ~2 KB because Python's fp.read(n) only allocates what it + /// returns. NumSharp gets the same property from the seekable stream's remaining length, + /// which is a hard bound on what any claim can deliver, and measures ~1.2 KB. The 64 KB + /// bound below leaves room for allocation noise while still being thousands of times below + /// any claim — enough to catch a regression to new byte[claimedLength]. + /// + [TestMethod] + [TestCategory("NpyOracle")] + [DoNotParallelize] + public void HostileHeaderLength_DoesNotAllocateTheClaim() + { + foreach ((string name, long claim) in new[] + { + ("hostile_header_len_1gb", 1_000_000_000L), + ("hostile_header_len_4gb", 4_294_967_280L), + ("hostile_header_len_64k_v1", 65_535L), + }) + { + NpyCase c = NpyOracleCorpus.Cases.Single(x => x.Name == name); + + long allocated = AllocationTests.MinAllocated( + () => Assert.ThrowsException(() => np.load_npy(c.Bytes), name)); + + Assert.IsTrue(allocated < 64 * 1024, + $"rejecting '{name}' — a {c.Bytes.Length}-byte file claiming a {claim:N0}-byte header — " + + $"allocated {allocated:N0} bytes. The header length comes straight from the file and must " + + "never size an allocation: read incrementally, bounded by what the stream actually holds."); + } + } + + /// The corpus is present and non-vacuous — a silent zero-case run would prove nothing. + [TestMethod] + [TestCategory("NpyOracle")] + public void Corpus_IsLoadedAndNonVacuous() + { + Assert.AreEqual("2.4.2", NpyOracleCorpus.NumpyVersion, "corpus should be generated by NumPy 2.4.2"); + Assert.IsTrue(NpyOracleCorpus.Cases.Length >= 200, $"expected 200+ cases, got {NpyOracleCorpus.Cases.Length}"); + Assert.IsTrue(NpyOracleCorpus.Cases.Count(c => c.WriteExact) >= 150, "expected 150+ byte-exact write cases"); + Assert.IsTrue(NpyOracleCorpus.Cases.Count(c => c.LoadError != null) >= 25, "expected 25+ error cases"); + Assert.IsTrue(NpyOracleCorpus.OfKind("npz").Count() >= 5, "expected 5+ npz cases"); + + // Every NumSharp dtype that can round-trip must actually appear in the corpus, or a dtype + // could silently lose coverage. + var covered = NpyOracleCorpus.OfKind("npy").Where(c => c.LoadError == null) + .Select(c => c.NsDtype.Value).Distinct().ToHashSet(); + var expected = new[] + { + NPTypeCode.Boolean, NPTypeCode.Byte, NPTypeCode.SByte, NPTypeCode.Int16, NPTypeCode.UInt16, + NPTypeCode.Int32, NPTypeCode.UInt32, NPTypeCode.Int64, NPTypeCode.UInt64, NPTypeCode.Char, + NPTypeCode.Half, NPTypeCode.Single, NPTypeCode.Double, NPTypeCode.Complex, + }; + CollectionAssert.AreEquivalent(expected, covered.ToArray(), + "every NumPy-representable NumSharp dtype must be exercised by the corpus " + + "(Decimal is excluded: it has no NumPy dtype)"); + } + + #region helpers + + private static string First(string message) => message.Split('\n')[0].TrimEnd('\r'); + + // Locate the first differing byte and show a window around it — a raw dump of two 800 KB + // buffers would bury the signal. + private static string Diff(byte[] expected, byte[] actual) + { + int at = 0; + int min = Math.Min(expected.Length, actual.Length); + while (at < min && expected[at] == actual[at]) + at++; + + var sb = new StringBuilder(); + if (expected.Length != actual.Length) + sb.Append($"length {actual.Length} != expected {expected.Length}; "); + sb.Append($"first difference at byte {at}"); + + int from = Math.Max(0, at - 8); + sb.Append($"\n expected[{from}..] {Window(expected, from)}"); + sb.Append($"\n actual [{from}..] {Window(actual, from)}"); + return sb.ToString(); + } + + private static string Window(byte[] b, int from) + { + if (from >= b.Length) return ""; + byte[] slice = b.Skip(from).Take(24).ToArray(); + return System.Convert.ToHexString(slice) + $" {Printable(slice)}"; + } + + private static string Printable(byte[] b) => + new string(b.Select(x => x >= 0x20 && x < 0x7F ? (char)x : '.').ToArray()); + + /// Collects every divergence so one run shows the whole picture, not just the first. + private sealed class Failures + { + private readonly List _items = new(); + private readonly string _what; + + public Failures(string what) => _what = what; + + public void Add(NpyCase c, string detail) => + _items.Add($" {c.Name}\n {detail}\n (case: {c.Note})"); + + public void Assert() + { + if (_items.Count == 0) + return; + + throw new AssertFailedException( + $"{_items.Count} {_what} divergence(s) from NumPy {NpyOracleCorpus.NumpyVersion}:\n\n" + + string.Join("\n\n", _items.Take(25)) + + (_items.Count > 25 ? $"\n\n ... and {_items.Count - 25} more" : "")); + } + } + + #endregion + } +} diff --git a/test/NumSharp.UnitTest/IO/NumpyCompatibilityTests.cs b/test/NumSharp.UnitTest/IO/NumpyCompatibilityTests.cs new file mode 100644 index 000000000..1fe6982d8 --- /dev/null +++ b/test/NumSharp.UnitTest/IO/NumpyCompatibilityTests.cs @@ -0,0 +1,196 @@ +using System; +using System.IO; +using System.Runtime.CompilerServices; +using System.Linq; +using Microsoft.VisualStudio.TestTools.UnitTesting; +using NumSharp.IO; + +namespace NumSharp.UnitTest.IO +{ + /// + /// Tests loading files created by actual NumPy to verify cross-compatibility. + /// Test files are in test_compat/ directory, created by Python/NumPy. + /// + [TestClass] + public class NumpyCompatibilityTests + { + private const string TestDir = "test_compat"; + + /// + /// Fail loudly when the fixtures are missing, rather than skipping. + /// + /// + /// These used to `return` silently when test_compat/ was absent — and nothing copied it to + /// the output directory, so on a clean checkout all of them passed while asserting NOTHING. + /// A test that cannot fail is worse than no test: it reports coverage that does not exist. + /// The csproj now copies the fixtures; this turns a regression in that copy rule into a red + /// test instead of silent green. + /// + private static void RequireTestFiles([CallerMemberName] string caller = null) + { + Assert.IsTrue(Directory.Exists(TestDir) && File.Exists(Path.Combine(TestDir, "int32_1d.npy")), + $"{caller}: the NumPy fixtures are missing from '{Path.GetFullPath(TestDir)}'. They are " + + "committed under test/NumSharp.UnitTest/test_compat/ and copied to the output by the " + + "csproj — this test asserts nothing without them."); + } + + [TestMethod] + public void Load_NumPy_Int32_1D() + { + RequireTestFiles(); + + var arr = np.load_npy(Path.Combine(TestDir, "int32_1d.npy")); + + Assert.AreEqual(NPTypeCode.Int32, arr.typecode); + Assert.AreEqual(1, arr.ndim); + Assert.AreEqual(5, arr.size); + Assert.IsTrue(new int[] { 1, 2, 3, 4, 5 }.SequenceEqual(arr.Data())); + } + + [TestMethod] + public void Load_NumPy_Float64_2D() + { + RequireTestFiles(); + + var arr = np.load_npy(Path.Combine(TestDir, "float64_2d.npy")); + + Assert.AreEqual(NPTypeCode.Double, arr.typecode); + Assert.AreEqual(2, arr.ndim); + Assert.IsTrue(new long[] { 3, 4 }.SequenceEqual(arr.shape)); + Assert.AreEqual(0.0, arr.GetDouble(0, 0)); + Assert.AreEqual(11.0, arr.GetDouble(2, 3)); + } + + [TestMethod] + public void Load_NumPy_Boolean() + { + RequireTestFiles(); + + var arr = np.load_npy(Path.Combine(TestDir, "bool.npy")); + + Assert.AreEqual(NPTypeCode.Boolean, arr.typecode); + Assert.AreEqual(3, arr.size); + Assert.IsTrue(arr.GetBoolean(0)); + Assert.IsFalse(arr.GetBoolean(1)); + Assert.IsTrue(arr.GetBoolean(2)); + } + + [TestMethod] + public void Load_NumPy_Scalar() + { + RequireTestFiles(); + + var arr = np.load_npy(Path.Combine(TestDir, "scalar.npy")); + + // NumPy scalars have shape () and ndim 0 + // NumSharp SHOULD load this as a true scalar + Console.WriteLine($"Scalar loaded: ndim={arr.ndim}, shape=[{string.Join(",", arr.shape)}], size={arr.size}, IsScalar={arr.Shape.IsScalar}"); + + Assert.AreEqual(1, arr.size); + Assert.AreEqual(42, arr.GetInt64(0)); // NumPy default int is int64 + + // Verify it's a TRUE scalar (ndim=0, shape=[]) + // This is the NumPy-compatible behavior + Assert.AreEqual(0, arr.ndim, "NumPy scalar should have ndim=0"); + Assert.AreEqual(0, arr.shape.Length, "NumPy scalar should have empty shape"); + Assert.IsTrue(arr.Shape.IsScalar, "Should be marked as scalar"); + } + + [TestMethod] + public void Load_NumPy_Empty() + { + RequireTestFiles(); + + var arr = np.load_npy(Path.Combine(TestDir, "empty.npy")); + + Assert.AreEqual(NPTypeCode.Double, arr.typecode); + Assert.AreEqual(0, arr.size); + } + + [TestMethod] + public void Load_NumPy_FortranOrder() + { + RequireTestFiles(); + + var arr = np.load_npy(Path.Combine(TestDir, "fortran.npy")); + + Console.WriteLine($"Fortran array loaded: shape=[{string.Join(",", arr.shape)}]"); + Console.WriteLine($"Data: {arr}"); + + // Should be 2x3 with values 0-5 in C-order view + Assert.AreEqual(2, arr.ndim); + Assert.IsTrue(new long[] { 2, 3 }.SequenceEqual(arr.shape)); + + // Verify data is correct (F-order [0,3,1,4,2,5] should appear as C-order [[0,1,2],[3,4,5]]) + // NumPy uses int64 by default + Assert.AreEqual(0L, arr.GetInt64(0, 0)); + Assert.AreEqual(1L, arr.GetInt64(0, 1)); + Assert.AreEqual(2L, arr.GetInt64(0, 2)); + Assert.AreEqual(3L, arr.GetInt64(1, 0)); + Assert.AreEqual(4L, arr.GetInt64(1, 1)); + Assert.AreEqual(5L, arr.GetInt64(1, 2)); + } + + [TestMethod] + public void Load_NumPy_Npz() + { + RequireTestFiles(); + + using var npz = np.load_npz(Path.Combine(TestDir, "multi.npz")); + + Assert.AreEqual(2, npz.Count); + Assert.IsTrue(npz.ContainsKey("a")); + Assert.IsTrue(npz.ContainsKey("b")); + + var a = npz["a"]; + var b = npz["b"]; + + Assert.IsTrue(new long[] { 1, 2, 3 }.SequenceEqual(a.Data())); + Assert.IsTrue(np.allclose(np.array(new double[] { 4.0, 5.0 }), b)); + } + + [TestMethod] + public void Load_NumPy_NpzCompressed() + { + RequireTestFiles(); + + using var npz = np.load_npz(Path.Combine(TestDir, "compressed.npz")); + + Assert.IsTrue(npz.ContainsKey("data")); + var data = npz["data"]; + + Assert.AreEqual(1000, data.size); + Assert.AreEqual(0, data.GetInt64(0)); + Assert.AreEqual(999, data.GetInt64(999)); + } + + [TestMethod] + public void RoundTrip_NumSharpToNumPy() + { + RequireTestFiles(); + + // Save from NumSharp + var original = np.arange(12).reshape(3, 4); + var path = Path.Combine(TestDir, "numsharp_created.npy"); + np.save(path, original); + + // Verify file exists and has correct header + var bytes = File.ReadAllBytes(path); + Assert.AreEqual(0x93, bytes[0], "Magic byte 0"); + Assert.AreEqual((byte)'N', bytes[1], "Magic byte 1"); + Assert.AreEqual((byte)'U', bytes[2], "Magic byte 2"); + Assert.AreEqual((byte)'M', bytes[3], "Magic byte 3"); + Assert.AreEqual((byte)'P', bytes[4], "Magic byte 4"); + Assert.AreEqual((byte)'Y', bytes[5], "Magic byte 5"); + Assert.AreEqual(1, bytes[6], "Version major"); + Assert.AreEqual(0, bytes[7], "Version minor"); + + // Header should contain shape info + var headerEnd = Array.IndexOf(bytes, (byte)'\n'); + var header = System.Text.Encoding.ASCII.GetString(bytes, 10, headerEnd - 10); + Console.WriteLine($"Header: {header}"); + Assert.IsTrue(header.Contains("'shape': (3, 4)"), $"Header should contain shape: {header}"); + Assert.IsTrue(header.Contains("'fortran_order': False"), $"Header should contain fortran_order: {header}"); + } + } +} diff --git a/test/NumSharp.UnitTest/IO/corpus/npy_oracle.zip b/test/NumSharp.UnitTest/IO/corpus/npy_oracle.zip new file mode 100644 index 000000000..d830c5b0c Binary files /dev/null and b/test/NumSharp.UnitTest/IO/corpus/npy_oracle.zip differ diff --git a/test/NumSharp.UnitTest/NumSharp.UnitTest.csproj b/test/NumSharp.UnitTest/NumSharp.UnitTest.csproj index 6eda1d31e..63396cbb5 100644 --- a/test/NumSharp.UnitTest/NumSharp.UnitTest.csproj +++ b/test/NumSharp.UnitTest/NumSharp.UnitTest.csproj @@ -51,6 +51,22 @@ + + + + PreserveNewest + + + + + + + PreserveNewest + + + diff --git a/test/NumSharp.UnitTest/View/OrderSupport.OpenBugs.Tests.cs b/test/NumSharp.UnitTest/View/OrderSupport.OpenBugs.Tests.cs index 8fef9e0c0..a88b589a1 100644 --- a/test/NumSharp.UnitTest/View/OrderSupport.OpenBugs.Tests.cs +++ b/test/NumSharp.UnitTest/View/OrderSupport.OpenBugs.Tests.cs @@ -2474,25 +2474,20 @@ public void MoveAxis_FContig2D_Effectively_Transposes() // save(F-contig) writes header 'fortran_order': True and F-strided bytes. // load(fortran_order=True .npy) returns an F-contig NDArray. // Round-trip through np.save + np.load preserves both values AND layout. - // NumSharp (current state, all documented gaps): - // 1. np.save.cs:172 hardcodes "'fortran_order': False" in header. - // 2. np.save writes C-order bytes (via (Array)nd → ToMuliDimArray()). - // 3. np.load.cs:322 throws if header says 'fortran_order': True. - // 4. np.load always returns C-contig (matches #1/#2 but not NumPy). + // NumSharp: all four gaps closed by the NEP-01 rewrite (NpyFormat/NpzFile). The header now + // reports the real layout, F-order bytes are written via the transpose, fortran_order: True + // loads as reshape-reversed + transpose, and the layout survives the round-trip. The full + // matrix lives in NpyOracleTests; these stay as the layout-focused regression. // ============================================================================ [TestMethod] public void NpSave_FContig_RoundTrip_Values_Preserved() { - // Values must match after round-trip, regardless of layout. - // NumSharp: (Array)nd materializes a C-order copy, so save writes - // C-order bytes + "fortran_order: False" header. The values survive; - // only the layout flag diverges from NumPy. var f = np.arange(12).reshape(3, 4).T.astype(typeof(double)); using var stream = new System.IO.MemoryStream(); - np.Save((Array)f, stream); + np.save(stream, f); stream.Position = 0; - var loaded = np.load(stream); + var loaded = np.load_npy(stream); loaded.shape.Should().Equal(new long[] { 4, 3 }); for (int i = 0; i < 4; i++) @@ -2501,15 +2496,11 @@ public void NpSave_FContig_RoundTrip_Values_Preserved() } [TestMethod] - [OpenBugs] // NumPy: save(F-contig) writes 'fortran_order': True in header. - // NumSharp: np.save.cs:172 hardcodes "'fortran_order': False" — the - // header is a lie when the caller passes an F-contig NDArray, and - // also loses round-trip layout info. public void NpSave_FContig_Header_ContainsFortranOrderTrue() { var f = np.arange(12).reshape(3, 4).T.astype(typeof(double)); using var stream = new System.IO.MemoryStream(); - np.Save((Array)f, stream); + np.save(stream, f); // Read just the header bytes — magic(6) + version(2) + header_len(2) + header. var bytes = stream.ToArray(); @@ -2521,16 +2512,13 @@ public void NpSave_FContig_Header_ContainsFortranOrderTrue() } [TestMethod] - [OpenBugs] // NumPy: loading a .npy with fortran_order: True yields an F-contig array. - // NumSharp: np.load.cs:322 throws Exception on fortran_order: True. - public void NpLoad_NumPyFortranOrderTrue_DoesNotThrow() + public void NpLoad_NumPyFortranOrderTrue_LoadsAsFContig() { - // Synthesize a minimal .npy header with 'fortran_order': True. - // dtype ' np.load(stream); - act.Should().NotThrow( - "NumPy saves F-contig arrays with fortran_order:True — NumSharp must accept them"); + var loaded = np.load_npy(stream); + + loaded.shape.Should().Equal(new long[] { 4, 3 }); + loaded.Shape.IsFContiguous.Should().BeTrue( + "NumPy saves F-contig arrays with fortran_order:True — NumSharp must load them as F-contig"); + + // F-order means the first axis varies fastest: element (i,j) is payload[j*4 + i]. + for (int i = 0; i < 4; i++) + for (int j = 0; j < 3; j++) + ((double)loaded[i, j]).Should().Be(j * 4 + i); } [TestMethod] - [OpenBugs] // NumPy: round-trip of F-contig preserves layout flag. - // NumSharp: load always returns C-contig (even if bytes/layout could - // be preserved, the loader discards that info). public void NpSave_FContig_RoundTrip_PreservesFContigFlag() { var f = np.arange(12).reshape(3, 4).T.astype(typeof(double)); using var stream = new System.IO.MemoryStream(); - np.Save((Array)f, stream); + np.save(stream, f); stream.Position = 0; - var loaded = np.load(stream); + var loaded = np.load_npy(stream); loaded.Shape.IsFContiguous.Should().BeTrue( "NumPy: round-tripping an F-contig array via save+load preserves layout"); diff --git a/test/NumSharp.UnitTest/np.save_load.Test.cs b/test/NumSharp.UnitTest/np.save_load.Test.cs index 2fca34675..2af3a91d4 100644 --- a/test/NumSharp.UnitTest/np.save_load.Test.cs +++ b/test/NumSharp.UnitTest/np.save_load.Test.cs @@ -1,73 +1,470 @@ -using System; +using System; using System.Collections.Generic; -using System.Text; +using System.IO; +using System.Linq; +using AwesomeAssertions; using Microsoft.VisualStudio.TestTools.UnitTesting; -using NumSharp.UnitTest.Creation; using NumSharp; -using System.IO; +using NumSharp.IO; namespace NumSharp.UnitTest { + /// + /// Behaviour of the public save/load surface — signatures, path handling, overloads and + /// round-trips. Byte-level conformance to NumPy is proven separately by NpyOracleTests, + /// which replays real NumPy output. + /// [TestClass] public class NumpySaveLoad { + private string _dir; + + [TestInitialize] + public void Setup() + { + _dir = Path.Combine(Path.GetTempPath(), "numsharp_npy_" + Guid.NewGuid().ToString("N")); + Directory.CreateDirectory(_dir); + } + + [TestCleanup] + public void Cleanup() + { + try { Directory.Delete(_dir, recursive: true); } + catch (IOException) { /* a leaked handle fails the test that leaked it, not this one */ } + } + + private string At(string name) => Path.Combine(_dir, name); + + #region round-trips + + [TestMethod] + public void Save_Load_Int1D() + { + var x = np.array(new[] { 1, 2, 3, 4, 5 }); + string file = At("ints.npy"); + + np.save(file, x); + var loaded = np.load_npy(file); + + loaded.typecode.Should().Be(NPTypeCode.Int32); + loaded.shape.Should().Equal(new long[] { 5 }); + loaded.ToArray().Should().Equal(1, 2, 3, 4, 5); + } + + [TestMethod] + public void Save_Load_Float1D() + { + var x = np.array(new[] { 1.0f, 1.5f, 2.0f, 2.5f, 3.0f }); + string file = At("floats.npy"); + + np.save(file, x); + np.load_npy(file).ToArray().Should().Equal(1.0f, 1.5f, 2.0f, 2.5f, 3.0f); + } + + [TestMethod] + public void Save_Load_Double1D() + { + var x = np.array(new[] { 1.0, 1.5, 2.0, 2.5, 3.0 }); + string file = At("doubles.npy"); + + np.save(file, x); + np.load_npy(file).ToArray().Should().Equal(1.0, 1.5, 2.0, 2.5, 3.0); + } + + [TestMethod] + public void Save_Load_MultiDim() + { + var x = np.arange(24).reshape(2, 3, 4); + string file = At("md.npy"); + + np.save(file, x); + var loaded = np.load_npy(file); + + loaded.shape.Should().Equal(new long[] { 2, 3, 4 }); + np.array_equal(x, loaded).Should().BeTrue(); + } + [TestMethod] - public void Run() + public void Save_Load_SystemArrayOverload() { - int[] x = {1, 2, 3, 4, 5}; - np.Save(x, @"test.npy"); - np.Save_Npz(x, @"test1.npz"); - np.Load(@"test.npy"); - np.Load_Npz(@"test1.npz"); + int[,] x = { { 1, 2 }, { 3, 4 } }; + string file = At("sysarray.npy"); + + np.save(file, x); + var loaded = np.load_npy(file); + + loaded.shape.Should().Equal(new long[] { 2, 2 }); + loaded.ToArray().Should().Equal(1, 2, 3, 4); } [TestMethod] - public void Float1DimArray() + public void Save_Load_Bytes_NoFile() { - float[] x = {1.0f, 1.5f, 2.0f, 2.5f, 3.0f}; - np.Save(x, @"test_Float1DimArray.npy"); - np.Save_Npz(x, @"test_Float1DimArray.npz"); - np.Load(@"test_Float1DimArray.npy"); - np.Load_Npz(@"test_Float1DimArray.npz"); + var x = np.arange(6).reshape(2, 3); + + byte[] encoded = np.save(x); + + np.array_equal(x, np.load_npy(encoded)).Should().BeTrue(); } + #endregion + + #region path handling + [TestMethod] - public void Double1DimArray() + public void Save_AppendsNpyExtension_WhenMissing() { - double[] x = {1.0, 1.5, 2.0, 2.5, 3.0}; - np.Save(x, @"test_Double1DimArray.npy"); - np.Save_Npz(x, @"test_Double1DimArray.npz"); - np.Load(@"test_Double1DimArray.npy"); - np.Load_Npz(@"test_Double1DimArray.npz"); + string stem = At("noext"); + np.save(stem, np.arange(9.0).reshape(3, 3)); + + File.Exists(stem + ".npy").Should().BeTrue("np.save appends '.npy' when the path lacks it"); + File.Exists(stem).Should().BeFalse(); } [TestMethod] - public void SaveAndLoadMultiDimArray() + public void Save_DoesNotDoubleAppendExtension() { - int[,] x = {{1, 2}, {3, 4}}; - np.Save(x, @"test_SaveAndLoadMultiDimArray.npy"); - np.Save_Npz(x, @"test_SaveAndLoadMultiDimArray.npz"); - np.Load(@"test_SaveAndLoadMultiDimArray.npy"); - np.Load_Npz(@"test_SaveAndLoadMultiDimArray.npz"); + string file = At("withext.npy"); + np.save(file, np.arange(4.0)); + + File.Exists(file).Should().BeTrue(); + File.Exists(file + ".npy").Should().BeFalse("'.npy' is already present, so nothing is appended"); } + [TestMethod] + public void Savez_AppendsNpzExtension_WhenMissing() + { + string stem = At("archive"); + np.savez(stem, np.arange(3)); + + File.Exists(stem + ".npz").Should().BeTrue(); + } [TestMethod] public void SaveAndLoadWithNpyFileExt() { - // float - string fTestFile = @"test_" + nameof(SaveAndLoadWithNpyFileExt); - string fTestFileWithExt = fTestFile + ".npy"; + string stem = At("ext_roundtrip"); + var f1 = np.arange(9.0f).reshape(3, 3); - np.save(fTestFile, f1); - var f2 = np.load(fTestFileWithExt); - Assert.IsTrue(np.all(f1 == f2)); + np.save(stem, f1); + np.all(f1 == np.load_npy(stem + ".npy")).Should().BeTrue(); - // double var d1 = np.arange(9.0d).reshape(3, 3); - np.save(fTestFile, d1); - var d2 = np.load(fTestFileWithExt); - Assert.IsTrue(np.all(d1 == d2)); + np.save(stem, d1); + np.all(d1 == np.load_npy(stem + ".npy")).Should().BeTrue(); + } + + #endregion + + #region npz + + [TestMethod] + public void Savez_PositionalArrays_GetArrNNames() + { + string file = At("positional.npz"); + np.savez(file, np.arange(3), np.arange(4.0)); + + using var npz = np.load_npz(file); + + npz.Files.Should().Equal("arr_0", "arr_1"); + npz["arr_0"].ToArray().Should().Equal(0L, 1L, 2L); + npz["arr_1"].ToArray().Should().Equal(0.0, 1.0, 2.0, 3.0); + } + + [TestMethod] + public void Savez_NamedArrays_KeepTheirNames() + { + string file = At("named.npz"); + np.savez(file, new Dictionary + { + ["weights"] = np.arange(6.0).reshape(2, 3), + ["biases"] = np.arange(3.0), + }); + + using var npz = np.load_npz(file); + + npz.Files.Should().BeEquivalentTo(new[] { "weights", "biases" }); + npz["weights"].shape.Should().Equal(new long[] { 2, 3 }); + npz["biases"].ToArray().Should().Equal(0.0, 1.0, 2.0); + } + + [TestMethod] + public void Savez_Compressed_IsSmallerAndReadsBack() + { + var big = np.zeros(new Shape(20_000), NPTypeCode.Double); // compresses well + + byte[] stored = np.savez(big); + byte[] deflated = np.savez_compressed(big); + + deflated.Length.Should().BeLessThan(stored.Length / 4, + "20k zeroed doubles should deflate to a small fraction of their stored size"); + + using var npz = np.load_npz(deflated); + npz["arr_0"].shape.Should().Equal(new long[] { 20_000 }); + np.array_equal(big, npz["arr_0"]).Should().BeTrue(); + } + + [TestMethod] + public void Savez_PositionalCollidingWithKeyword_Throws() + { + Action act = () => np.savez(At("collide.npz"), + new[] { np.arange(3) }, + new Dictionary { ["arr_0"] = np.arange(3) }); + + act.Should().Throw() + .WithMessage("*Cannot use un-named variables and keyword arr_0*", + "NumPy rejects a positional array whose generated name is already taken"); + } + + [TestMethod] + public void Npz_KeysWorkWithAndWithoutNpySuffix() + { + using var npz = np.load_npz(np.savez(np.arange(3))); + + npz.ContainsKey("arr_0").Should().BeTrue(); + npz.ContainsKey("arr_0.npy").Should().BeTrue(); + npz["arr_0.npy"].Should().BeSameAs(npz["arr_0"], "both spellings name the same member"); + npz.Files.Should().Equal(new[] { "arr_0" }, "'.files' reports names with '.npy' stripped"); + } + + [TestMethod] + public void Npz_IsLazyAndCaches() + { + using var npz = np.load_npz(np.savez(np.arange(3), np.arange(4))); + + var first = npz["arr_0"]; + + npz["arr_0"].Should().BeSameAs(first, "a loaded member is cached, not re-read"); + } + + [TestMethod] + public void Npz_MissingKey_Throws() + { + using var npz = np.load_npz(np.savez(np.arange(3))); + + Action act = () => { var _ = npz["nope"]; }; + act.Should().Throw().WithMessage("*nope is not a file in the archive*"); + } + + [TestMethod] + public void Npz_DotAccessViaF() + { + using var npz = np.load_npz(np.savez(new Dictionary + { + ["weights"] = np.arange(3.0), + })); + + NDArray w = npz.f.weights; + w.ToArray().Should().Equal(0.0, 1.0, 2.0); + } + + [TestMethod] + public void Npz_EnumerationYieldsEveryMember() + { + using var npz = np.load_npz(np.savez(np.arange(3), np.arange(4), np.arange(5))); + + npz.Count.Should().Be(3); + npz.Select(kv => kv.Key).Should().Equal("arr_0", "arr_1", "arr_2"); + npz.Sum(kv => kv.Value.size).Should().Be(3 + 4 + 5); + } + + [TestMethod] + public void Npz_UseAfterDispose_Throws() + { + var npz = np.load_npz(np.savez(np.arange(3))); + npz.Dispose(); + + Action act = () => { var _ = npz["arr_0"]; }; + act.Should().Throw(); + } + + [TestMethod] + public void Npz_DisposeReleasesTheFileHandle() + { + string file = At("handle.npz"); + np.savez(file, np.arange(3)); + + using (var npz = np.load_npz(file)) + npz["arr_0"].size.Should().Be(3); + + // A leaked handle would make this throw IOException. + Action act = () => File.Delete(file); + act.Should().NotThrow("np.load_npz owns the FileStream it opened and must close it on dispose"); + } + + [TestMethod] + public void Npz_NestedNamesRoundTrip() + { + // Zip entry names may contain '/', which NumPy neither forbids nor treats as a directory. + using var npz = np.load_npz(np.savez(new Dictionary + { + ["A/A"] = np.arange(4), + ["B/A"] = np.arange(4), + })); + + npz.Count.Should().Be(2); + npz.Files.Should().BeEquivalentTo(new[] { "A/A", "B/A" }); + npz["A/A"].ToArray().Should().Equal(0L, 1L, 2L, 3L); + } + + #endregion + + #region streams + + [TestMethod] + public void Save_ToStream_AppendsSoOneFileHoldsManyArrays() + { + using var ms = new MemoryStream(); + np.save(ms, np.arange(3)); + np.save(ms, np.arange(4.0).reshape(2, 2)); + + ms.Position = 0; + np.load_npy(ms).ToArray().Should().Equal(0L, 1L, 2L); + np.load_npy(ms).shape.Should().Equal(new long[] { 2, 2 }); + + Action act = () => np.load_npy(ms); + act.Should().Throw("there is no third array"); + } + + [TestMethod] + public void Load_LeavesCallerOwnedStreamOpen() + { + using var ms = new MemoryStream(np.save(np.arange(3))); + + np.load(ms); + + ms.CanRead.Should().BeTrue("np.load(Stream) does not own the caller's stream"); + } + + #endregion + + #region load dispatch + + [TestMethod] + public void Load_ReturnsNDArrayForNpy_AndNpzFileForNpz() + { + np.load(np.save(np.arange(3))).Should().BeOfType(); + + object archive = np.load(np.savez(np.arange(3))); + archive.Should().BeOfType(); + ((NpzFile)archive).Dispose(); + } + + [TestMethod] + public void Load_DetectsTypeByMagic_NotByExtension() + { + // A .npz archive under a .npy name is still an archive: NumPy dispatches on magic bytes. + string misnamed = At("actually_an_archive.npy"); + File.WriteAllBytes(misnamed, np.savez(np.arange(3))); + + object loaded = np.load(misnamed); + loaded.Should().BeOfType(); + ((NpzFile)loaded).Dispose(); } + + [TestMethod] + public void LoadNpy_OnNpzArchive_SaysUseLoadNpz() + { + Action act = () => np.load_npy(np.savez(np.arange(3))); + + act.Should().Throw().WithMessage("*is a .npz archive*np.load_npz*", + "the error should name the function that would work"); + } + + [TestMethod] + public void LoadNpz_OnNpyFile_SaysUseLoadNpy() + { + Action act = () => np.load_npz(np.save(np.arange(3))); + + act.Should().Throw().WithMessage("*is a .npy file*np.load_npy*"); + } + + [TestMethod] + public void Load_EmptyFile_ThrowsEof() + { + Action act = () => np.load(Array.Empty()); + act.Should().Throw().WithMessage("*No data left in file*"); + } + + #endregion + + #region parameter validation + + [TestMethod] + public void Load_RejectsEncodingsThatCorruptData() + { + // NumPy allows only these three; anything else can silently corrupt numeric data. + foreach (string ok in new[] { "ASCII", "latin1", "bytes" }) + { + Action valid = () => np.load(np.save(np.arange(3)), encoding: ok); + valid.Should().NotThrow($"'{ok}' is one of NumPy's three allowed encodings"); + } + + Action act = () => np.load(np.save(np.arange(3)), encoding: "utf-8"); + act.Should().Throw().WithMessage("*encoding must be 'ASCII', 'latin1', or 'bytes'*"); + } + + [TestMethod] + public void Load_EncodingIsValidatedBeforeTheFileIsEvenRead() + { + // NumPy checks encoding first, so an empty file with a bad encoding reports the encoding. + Action act = () => np.load(Array.Empty(), encoding: "utf-8"); + act.Should().Throw().WithMessage("*encoding must be*"); + } + + [TestMethod] + public void Load_MmapMode_ValidatesThenReportsNotImplemented() + { + byte[] data = np.save(np.arange(3)); + + foreach (string mode in new[] { "r", "r+", "w+", "c" }) + { + Action valid = () => np.load(data, mmap_mode: mode); + valid.Should().Throw() + .WithMessage("*not implemented yet*", $"'{mode}' is a real NumPy mode, just unsupported here"); + } + + Action bad = () => np.load(data, mmap_mode: "z"); + bad.Should().Throw().WithMessage("*mode must be one of*"); + } + + [TestMethod] + public void Load_MaxHeaderSize_GuardsAgainstOversizedHeaders() + { + byte[] data = np.save(np.arange(3)); // a normal 118-byte header + + Action tiny = () => np.load(data, max_header_size: 10); + tiny.Should().Throw().WithMessage("*is large and may not be safe to load securely*"); + + Action trusted = () => np.load(data, max_header_size: 10, allow_pickle: true); + trusted.Should().NotThrow("allow_pickle declares the file trusted, which lifts the header guard"); + } + + [TestMethod] + public void Save_Decimal_ThrowsWithTheFix() + { + var dec = np.array(new[] { 1.5m, 2.5m }); + + Action act = () => np.save(At("dec.npy"), dec); + act.Should().Throw() + .WithMessage("*Decimal has no NumPy dtype*astype*", + "the error should name the workaround, since no NumPy dtype can hold a Decimal"); + } + + #endregion + + #region legacy fixture + + [TestMethod] + public void Load_LegacyNumPyFixture() + { + var arr = np.load_npy(@"data/1-dim-int32_4_comma_empty.npy"); + + arr.typecode.Should().Be(NPTypeCode.Int32); + arr.shape.Should().Equal(new long[] { 4 }); + arr.ToArray().Should().Equal(0, 1, 2, 3); + } + + #endregion } } diff --git a/test/NumSharp.UnitTest/test_compat/bool.npy b/test/NumSharp.UnitTest/test_compat/bool.npy new file mode 100644 index 000000000..662667893 Binary files /dev/null and b/test/NumSharp.UnitTest/test_compat/bool.npy differ diff --git a/test/NumSharp.UnitTest/test_compat/compressed.npz b/test/NumSharp.UnitTest/test_compat/compressed.npz new file mode 100644 index 000000000..12bdfea0d Binary files /dev/null and b/test/NumSharp.UnitTest/test_compat/compressed.npz differ diff --git a/test/NumSharp.UnitTest/test_compat/empty.npy b/test/NumSharp.UnitTest/test_compat/empty.npy new file mode 100644 index 000000000..1b7ce3f1d Binary files /dev/null and b/test/NumSharp.UnitTest/test_compat/empty.npy differ diff --git a/test/NumSharp.UnitTest/test_compat/float64_2d.npy b/test/NumSharp.UnitTest/test_compat/float64_2d.npy new file mode 100644 index 000000000..fae1c7a8a Binary files /dev/null and b/test/NumSharp.UnitTest/test_compat/float64_2d.npy differ diff --git a/test/NumSharp.UnitTest/test_compat/fortran.npy b/test/NumSharp.UnitTest/test_compat/fortran.npy new file mode 100644 index 000000000..cc62a0a52 Binary files /dev/null and b/test/NumSharp.UnitTest/test_compat/fortran.npy differ diff --git a/test/NumSharp.UnitTest/test_compat/int32_1d.npy b/test/NumSharp.UnitTest/test_compat/int32_1d.npy new file mode 100644 index 000000000..9415363cc Binary files /dev/null and b/test/NumSharp.UnitTest/test_compat/int32_1d.npy differ diff --git a/test/NumSharp.UnitTest/test_compat/multi.npz b/test/NumSharp.UnitTest/test_compat/multi.npz new file mode 100644 index 000000000..ebe261f9e Binary files /dev/null and b/test/NumSharp.UnitTest/test_compat/multi.npz differ diff --git a/test/NumSharp.UnitTest/test_compat/scalar.npy b/test/NumSharp.UnitTest/test_compat/scalar.npy new file mode 100644 index 000000000..c635ae331 Binary files /dev/null and b/test/NumSharp.UnitTest/test_compat/scalar.npy differ diff --git a/test/NumSharp.UnitTest/test_compat/scalar_check.npy b/test/NumSharp.UnitTest/test_compat/scalar_check.npy new file mode 100644 index 000000000..c635ae331 Binary files /dev/null and b/test/NumSharp.UnitTest/test_compat/scalar_check.npy differ diff --git a/test/NumSharp.UnitTest/test_compat/version2_large_header.npy b/test/NumSharp.UnitTest/test_compat/version2_large_header.npy new file mode 100644 index 000000000..0fe1e753a Binary files /dev/null and b/test/NumSharp.UnitTest/test_compat/version2_large_header.npy differ diff --git a/test/NumSharp.UnitTest/test_compat/version3_unicode.npy b/test/NumSharp.UnitTest/test_compat/version3_unicode.npy new file mode 100644 index 000000000..b6025e241 Binary files /dev/null and b/test/NumSharp.UnitTest/test_compat/version3_unicode.npy differ diff --git a/test/oracle/gen_npy_oracle.py b/test/oracle/gen_npy_oracle.py new file mode 100644 index 000000000..790ee034c --- /dev/null +++ b/test/oracle/gen_npy_oracle.py @@ -0,0 +1,860 @@ +""" +gen_npy_oracle.py — emit a committed, bytes-exact NumPy 2.4.2 oracle for the .npy/.npz format. + +NumPy is the oracle. This script writes REAL `np.save` / `np.savez` / `np.savez_compressed` +output into a single zip; the C# harness (`test/NumSharp.UnitTest/IO/NpyOracleTests.cs`) +replays it with NO Python at test time or in CI. + +Output: test/NumSharp.UnitTest/IO/corpus/npy_oracle.zip + manifest.json the case list (schema below) + cases/.npy|.npz the exact bytes NumPy produced + +Each case asserts up to three independent things: + + read — NumSharp loads the file; dtype + shape + raw C-order buffer must equal `ns_bytes`. + `ns_bytes` is the array's LOGICAL values in NumSharp's native in-memory form, so + big-endian files must byte-swap and fortran_order files must transpose to match. + write — NumSharp rebuilds the array from `ns_bytes` and saves it; the produced file must be + BYTE-IDENTICAL to NumPy's. Only set where NumSharp can natively represent the dtype + and layout (see `write_exact` below). + error — NumSharp must throw, and the message must contain `load_error` (verbatim NumPy text). + +Case schema: + { + "name": "float64_2d_c", + "file": "cases/float64_2d_c.npy", # entry inside this zip + "kind": "npy" | "npz" | "raw", + "descr": " error case) + "shape": [2, 3], + "fortran_order": false, + "version": [1, 0], + "data_offset": 128, # where the data starts (always 64-aligned) + "ns_bytes": "", # logical values, native-endian, C-order + "write_exact": true, # if true, NumSharp's writer must reproduce `file` byte-for-byte + "load_error": null, # if set, loading MUST throw containing this text + "note": "..." # why this case exists + } + +NPZ cases carry `entries`: {name -> {ns_dtype, shape, ns_bytes}} instead of a single array. + +Regenerate (deterministic; needs numpy==2.4.2): + python test/oracle/gen_npy_oracle.py +""" +import io +import json +import os +import struct +import sys +import warnings +import zipfile + +import numpy as np +from numpy.lib import _format_impl as fmt + +np.seterr(all="ignore") +warnings.simplefilter("ignore") # the 2.0/3.0 "stored array in format X" warnings are expected here + +OUT = os.path.join( + os.path.dirname(os.path.abspath(__file__)), + "..", "NumSharp.UnitTest", "IO", "corpus", "npy_oracle.zip", +) + +# NumPy dtype -> NPTypeCode. Char (UTF-16) and Decimal have no NumPy analog and are handled +# specially: Char rides NumPy's 4-byte ' System.Numerics.Complex + return np.ascontiguousarray(arr.astype(native)).tobytes("C").hex() + + +def _ns_dtype_of(arr): + if arr.dtype.kind == "U": + return "Char" + native = arr.dtype.newbyteorder("=") + if native == np.dtype("complex64"): + return "Complex" # NumSharp has no 64-bit complex; c8 widens on read + return NS_DTYPE[native.name] + + +def add_npy(name, arr, *, version=None, write_exact=True, note=""): + """Save `arr` with real NumPy and register it as a case.""" + buf = io.BytesIO() + fmt.write_array(buf, arr, version=version) + raw = buf.getvalue() + + major = raw[6] + hlen_size = 2 if major == 1 else 4 + hlen = int.from_bytes(raw[8:8 + hlen_size], "little") + + CASES.append({ + "name": name, + "file": f"cases/{name}.npy", + "kind": "npy", + "bytes": raw, # popped before manifest serialization + "descr": fmt.dtype_to_descr(arr.dtype), + "np_dtype": arr.dtype.name, + "ns_dtype": _ns_dtype_of(arr), + "shape": [int(d) for d in arr.shape], + "fortran_order": bool(arr.flags.f_contiguous and not arr.flags.c_contiguous), + "version": [raw[6], raw[7]], + "data_offset": 8 + hlen_size + hlen, + "ns_bytes": _ns_bytes(arr), + "write_exact": write_exact, + "load_error": None, + "note": note, + }) + + +def add_raw(name, raw, *, load_error, note="", ext="npy", load_via="load", max_header_size=None): + """A hand-built (usually malformed) file that must fail to load with `load_error`. + + `load_via` picks the entry point, because the two disagree on purpose: np.load dispatches on + magic bytes and treats anything unrecognized as a pickle, so a CORRUPT magic never reaches + read_magic and surfaces the pickle message instead. numpy.lib.format.read_array (NumSharp's + np.load_npy) is handed a .npy directly and does report the magic error. + """ + CASES.append({ + "name": name, + "file": f"cases/{name}.{ext}", + "kind": "raw", + "bytes": raw, + "descr": None, "np_dtype": None, "ns_dtype": None, + "shape": None, "fortran_order": None, "version": None, "data_offset": None, + "ns_bytes": None, "write_exact": False, + "load_error": load_error, + "load_via": load_via, + "max_header_size": max_header_size, + "note": note, + }) + + +def add_npz(name, arrays, *, compressed=False, note="", write_exact=False): + buf = io.BytesIO() + (np.savez_compressed if compressed else np.savez)(buf, **arrays) + CASES.append({ + "name": name, + "file": f"cases/{name}.npz", + "kind": "npz", + "bytes": buf.getvalue(), + "compressed": compressed, + "descr": None, "np_dtype": None, "ns_dtype": None, + "shape": None, "fortran_order": None, "version": None, "data_offset": None, + "ns_bytes": None, + "write_exact": write_exact, + "load_error": None, + "entries": { + k: {"ns_dtype": _ns_dtype_of(v), "shape": [int(d) for d in v.shape], "ns_bytes": _ns_bytes(v)} + for k, v in arrays.items() + }, + "note": note, + }) + + +# --------------------------------------------------------------------------------------------- +# 1. Every NumSharp dtype x representative shapes, C-order, version 1.0 (the default write path) +# --------------------------------------------------------------------------------------------- +SHAPES = { + "scalar": (), # 0-d: count == 1, no growth padding (len(shape) == 0) + "empty": (0,), # zero elements, still a full 128-byte header + "1d": (5,), # trailing comma in the shape tuple: (5,) + "2d": (2, 3), + "3d": (2, 3, 4), + "unit": (1,), + "empty2d": (0, 4), # zero-size but multi-dim + "empty3d": (5, 0, 3), +} + + +def _values(dt, n): + """Deterministic, dtype-appropriate values spanning the interesting range.""" + d = np.dtype(dt) + if d.kind == "b": + return np.arange(n) % 3 == 0 + if d.kind in "iu": + info = np.iinfo(d) + pool = [0, 1, info.max, info.min, info.max - 1] + list(range(2, 40)) + return np.array([pool[i % len(pool)] for i in range(n)], dtype=d) + if d.kind == "f": + finfo = np.finfo(d) + pool = [0.0, -0.0, 1.0, -1.0, np.nan, np.inf, -np.inf, + float(finfo.max), float(finfo.tiny), 0.5, -2.25, 3.14159] + return np.array([pool[i % len(pool)] for i in range(n)], dtype=d) + if d.kind == "c": + re = _values(d.type(0).real.dtype, n) + im = _values(d.type(0).real.dtype, n)[::-1] + return (re + 1j * im).astype(d) + if d.kind == "U": + pool = list("aZ0 ~é中") + return np.array([pool[i % len(pool)] for i in range(n)], dtype=d) + raise AssertionError(dt) + + +for dt in NATIVE_DTYPES: + for sname, shape in SHAPES.items(): + n = int(np.prod(shape)) if shape else 1 + arr = _values(dt, n).reshape(shape) + add_npy(f"{dt}_{sname}", arr, note=f"{dt} {sname} C-order v1.0 — the default np.save path") + +# Char rides ' 4-byte UCS-4 in NumPy) +for sname, shape in SHAPES.items(): + n = int(np.prod(shape)) if shape else 1 + add_npy(f"char_{sname}", _values(" fortran_order: True") + +# A transposed view is F-contiguous: NumPy stores fortran_order=True and writes the base bytes. +add_npy("fortran_transposed_view", np.arange(12, dtype=np.int32).reshape(3, 4).T, + note="transposed view is F-contiguous -> fortran_order: True") + +# 1-D and 0-d are BOTH C- and F-contiguous; the C check runs first, so fortran_order stays False. +add_npy("fortran_1d_is_c", np.asfortranarray(np.arange(5, dtype=np.int32)), + note="1-D is both C- and F-contiguous; C is tested first -> fortran_order: False") +# NOTE: asfortranarray is documented "ndim >= 1", so it PROMOTES a 0-d input to shape (1,). This case +# therefore pins that promotion, not 0-d contiguity — the true 0-d case is `int32_scalar`. +_promoted = np.asfortranarray(np.array(7, dtype=np.int32)) +assert _promoted.shape == (1,) +add_npy("fortran_scalar_promoted_to_1d", _promoted, + note="asfortranarray promotes 0-d to shape (1,) (it is documented ndim>=1); the result is both " + "C- and F-contiguous -> fortran_order: False. True 0-d coverage is the *_scalar cases.") + +# --------------------------------------------------------------------------------------------- +# 3. Non-contiguous sources — NumPy copies them to C-order on write (fortran_order: False) +# --------------------------------------------------------------------------------------------- +add_npy("strided_step2", np.arange(12, dtype=np.int32)[::2], + note="strided slice is neither C- nor F-contiguous -> written as a C-order copy") +add_npy("strided_reversed", np.arange(6, dtype=np.float64)[::-1], + note="negative-stride view -> written as a C-order copy") +add_npy("strided_2d_col", np.arange(12, dtype=np.int32).reshape(3, 4)[:, ::2], + note="column-strided 2-D view -> C-order copy") +add_npy("strided_offset_row", np.arange(12, dtype=np.int32).reshape(3, 4)[1:, :], + note="sliced view with a non-zero base offset") +add_npy("broadcast_view", np.broadcast_to(np.arange(3, dtype=np.int32), (4, 3)), + note="stride-0 broadcast view -> materialized C-order on write") + +# --------------------------------------------------------------------------------------------- +# 4. Format versions 2.0 and 3.0 — rejected outright by the old implementation +# --------------------------------------------------------------------------------------------- +for ver in [(1, 0), (2, 0), (3, 0)]: + tag = f"v{ver[0]}_{ver[1]}" + add_npy(f"version_{tag}_int32", np.arange(6, dtype=np.int32).reshape(2, 3), version=ver, + note=f"explicit format version {ver} (4-byte header length for 2.0/3.0; utf8 for 3.0)") + add_npy(f"version_{tag}_fortran", np.asfortranarray(np.arange(6, dtype=np.float64).reshape(2, 3)), + version=ver, note=f"format {ver} + fortran_order") + +# A header big enough to FORCE version 2.0 (> 65535 bytes) via a many-field structured dtype is +# out of scope (structured dtypes unsupported), so force the version explicitly instead and also +# ship a genuine >64KB-header file for the 4-byte length-field path. +big_names = [(f"f{i:04d}", "u1") for i in range(12000)] +big_struct = np.zeros(1, dtype=np.dtype(big_names)) +buf = io.BytesIO() +fmt.write_array(buf, big_struct) +assert buf.getvalue()[6] == 2, "12000 fields should force an auto-selected version 2.0 header" +# Raise the limit past this 218100-byte header so the load gets far enough to reject the dtype: +# this is the case that proves the 4-byte v2.0 length field is parsed at all. +add_raw("version_2_0_auto_large_header", buf.getvalue(), + load_error="Structured dtypes are not supported", max_header_size=500_000, + note="12000-field structured dtype forces an auto-selected v2.0 header (>65535 bytes). With the " + "size guard raised, NumSharp must parse the 4-byte length and THEN reject the structured descr") +add_raw("version_2_0_auto_large_header_default_limit", buf.getvalue(), + load_error="Header info length (218100) is large and may not be safe to load securely", + note="the same file at the DEFAULT max_header_size: the size guard fires before the dtype is even parsed") + +# --------------------------------------------------------------------------------------------- +# 5. Byte order — big-endian files must byte-swap to native on read +# --------------------------------------------------------------------------------------------- +for dt in ["int16", "int32", "int64", "uint16", "uint32", "uint64", "float16", "float32", "float64", "complex128"]: + be = np.dtype(dt).newbyteorder(">") + arr = _values(dt, 6).astype(be) + add_npy(f"bigendian_{dt}", arr, write_exact=False, + note=f"'>{np.dtype(dt).str[1:]}' big-endian -> byte-swapped to native on read; " + f"NumSharp always writes native, so it cannot reproduce these bytes") + +# '=' (native) and '|' (not-applicable) descriptors must parse as native +add_raw_native = np.arange(3, dtype=np.int32) +for prefix, dt in [("=", "i4"), ("<", "i4")]: + raw = fmt.magic(1, 0) + body = ("{'descr': '%s%s', 'fortran_order': False, 'shape': (3,), }" % (prefix, dt)).encode() + pad = 64 - ((8 + 2 + len(body) + 1) % 64) + raw += struct.pack(" widened to System.Numerics.Complex (read-only; NumSharp writes 'f2"), + write_exact=False, note="'>f2' half NaN payloads survive the swap") +add_npy("value_be_f8_nan", _f8_nans.astype(">f8"), write_exact=False, + note="'>f8' NaN payload survives the swap") +add_npy("value_be_bool", np.array([True, False, True], dtype=">b1"), write_exact=False, + note="'>b1' is 1 byte: the swap must be a no-op, not a reversal") +add_npy("value_be_i1", np.array([-128, 0, 127], dtype=">i1"), write_exact=False, + note="'>i1' is 1 byte: no-op swap") +add_npy("value_be_U1", np.array(["a", chr(0xFFFF), chr(0xE9), "Z"], dtype=">U1"), write_exact=False, + note="'>U1': 4-byte code points swapped, then narrowed to UTF-16") + +# Char/UCS-4 seams. U+0000 is the interesting one: NumPy's ' v2.0 auto-selection boundary: NumSharp must switch at the same byte NumPy does. +# --------------------------------------------------------------------------------------------- +def add_header(name, d, note=""): + buf = io.BytesIO() + fmt._write_array_header(buf, d, None) # None => auto-select the oldest version that fits + raw = buf.getvalue() + CASES.append({ + "name": name, "file": f"cases/{name}.hdr", "kind": "header", "bytes": raw, + "descr": d["descr"], "np_dtype": None, "ns_dtype": None, + "shape": [int(x) for x in d["shape"]], "fortran_order": d["fortran_order"], + "version": [raw[6], raw[7]], "data_offset": None, "ns_bytes": None, + "write_exact": True, "load_error": None, "note": note, + }) + + +def _hlen(shape, fortran): + b = io.BytesIO() + fmt._write_array_header(b, {"descr": "|i1", "fortran_order": fortran, "shape": shape}, None) + return int.from_bytes(b.getvalue()[8:10], "little") + + +_observable = [] +for _ndim in range(1, 12): + for _df in range(1, 20): + for _dl in range(1, 20): + if _ndim == 1 and _df != _dl: + continue + _s = tuple(([10 ** (_df - 1)] + [1] * (_ndim - 2) + ([10 ** (_dl - 1)] if _ndim > 1 else []))[:_ndim]) + if _hlen(_s, False) != _hlen(_s, True): + _observable.append(_s) + +assert _observable, "expected shapes where the growth axis tips the header across a 64-byte bucket" +for _i, _s in enumerate(_observable[:12]): + for _fo in (False, True): + add_header(f"header_growth_axis_{_i}_{'F' if _fo else 'C'}", + {"descr": "|i1", "fortran_order": _fo, "shape": _s}, + note=f"growth axis is OBSERVABLE here: shape {_s} in {'F' if _fo else 'C'}-order tips the " + f"header across a 64-byte bucket (C hlen={_hlen(_s, False)} vs F hlen={_hlen(_s, True)}), " + f"so using the wrong axis changes the file") + +# The exact v1.0 -> v2.0 auto-selection boundary (found by bisection: 21817 dims is the last v1.0). +_lo, _hi = 1, 60000 +while _lo + 1 < _hi: + _mid = (_lo + _hi) // 2 + b = io.BytesIO() + fmt._write_array_header(b, {"descr": "|u1", "fortran_order": False, "shape": (1,) * _mid}, None) + if b.getvalue()[6] == 1: + _lo = _mid + else: + _hi = _mid +for _nd, _tag in [(1, "tiny"), (_lo - 1, "just_under"), (_lo, "last_v1_0"), (_hi, "first_v2_0"), (_hi + 500, "well_into_v2_0")]: + add_header(f"header_version_boundary_{_tag}", + {"descr": "|u1", "fortran_order": False, "shape": (1,) * _nd}, + note=f"{_nd} dims: NumSharp's version auto-selection must flip to 2.0 at exactly the byte " + f"NumPy does (the last v1.0 header is {_lo} dims)") + +# --------------------------------------------------------------------------------------------- +# 10. NPZ archives +# --------------------------------------------------------------------------------------------- +add_npz("npz_single", {"arr_0": np.arange(6, dtype=np.int32).reshape(2, 3)}, + note="single array, ZIP_STORED") +add_npz("npz_multi", { + "arr_0": np.arange(6, dtype=np.int32).reshape(2, 3), + "arr_1": _values("float64", 4), + "weights": _values("float32", 8).reshape(2, 4), + "flag": np.array(True), +}, note="mixed dtypes/shapes; .files strips '.npy'; both 'weights' and 'weights.npy' must resolve") +add_npz("npz_compressed", { + "arr_0": np.arange(1000, dtype=np.int64), + "text": _values("info dict as it scans, so the +# LAST wins; .NET's ZipArchive.GetEntry returns the FIRST. NumPy's .files still lists both. +# * NumPy maps names in TWO passes -- dict(zip(stripped, full)) then .update(zip(full, full)) -- +# so when an archive holds both 'a' and 'a.npy', pass 2 re-points the key 'a' at the entry +# literally NAMED 'a'. A single interleaved loop resolves 'a' to whichever came last instead. +# +# Each entry stores its own index, so the expected value IS the entry a key must resolve to. +def add_npz_names(name, member_names, note=""): + buf = io.BytesIO() + with zipfile.ZipFile(buf, "w") as zf: + for i, n in enumerate(member_names): + m = io.BytesIO() + fmt.write_array(m, np.array([i], dtype=np.int8)) + zf.writestr(n, m.getvalue()) + raw = buf.getvalue() + + with np.load(io.BytesIO(raw)) as z: # real NumPy decides the expectations + files = list(z.files) + entries = {} + for key in dict.fromkeys(list(member_names) + [n.removesuffix(".npy") for n in member_names]): + try: + entries[key] = {"ns_dtype": "SByte", "shape": [1], + "ns_bytes": np.asarray(z[key], dtype=np.int8).tobytes().hex()} + except KeyError: + pass # NumPy has no such key; the absent-key path is covered by npz_missing_key tests + + CASES.append({ + "name": name, "file": f"cases/{name}.npz", "kind": "npz", "bytes": raw, "compressed": False, + "descr": None, "np_dtype": None, "ns_dtype": None, "shape": None, + "fortran_order": None, "version": None, "data_offset": None, "ns_bytes": None, + "write_exact": False, "load_error": None, + "entries": entries, "files": files, "suffix_alias": False, + "note": note, + }) + + +add_npz_names("npz_duplicate_names", ["dup.npy", "dup.npy"], + note="a zip may repeat a name: .files lists both, but the lookup takes the LAST " + "(zipfile's name->info dict), where .NET's GetEntry would take the first") +add_npz_names("npz_stripped_and_full", ["a", "a.npy"], + note="both members strip to the key 'a'. NumPy's SECOND mapping pass (full->entry) then " + "re-points 'a' at the entry literally named 'a' -> entry 0, not the later 'a.npy'. " + "A single interleaved loop gets entry 1 here") +add_npz_names("npz_full_and_stripped", ["a.npy", "a"], + note="the reverse order: 'a' resolves to entry 1 and 'a.npy' to entry 0") +add_npz_names("npz_names_interleaved", ["b.npy", "a", "a.npy", "b"], + note="both collisions at once, proving the passes are over ALL entries, not per entry") +add_npz_names("npz_names_triple", ["x.npy", "x.npy", "x"], + note="a duplicate AND a stripped/full collision together") +add_npz_names("npz_name_empty_and_dotnpy", ["", ".npy"], + note="'' and '.npy' both strip to the empty key") + +# An npz holding a non-.npy member: NumPy returns its raw bytes. +_mixed = io.BytesIO() +with zipfile.ZipFile(_mixed, "w", zipfile.ZIP_DEFLATED) as zf: + b = io.BytesIO() + fmt.write_array(b, np.arange(3, dtype=np.int32)) + zf.writestr("data.npy", b.getvalue()) + zf.writestr("readme.txt", b"not a numpy file") +CASES.append({ + "name": "npz_non_npy_member", "file": "cases/npz_non_npy_member.npz", "kind": "npz", + "bytes": _mixed.getvalue(), "compressed": True, + "descr": None, "np_dtype": None, "ns_dtype": None, "shape": None, + "fortran_order": None, "version": None, "data_offset": None, "ns_bytes": None, + "write_exact": False, "load_error": None, + "entries": {"data": {"ns_dtype": "Int32", "shape": [3], "ns_bytes": np.arange(3, dtype=np.int32).tobytes().hex()}}, + "raw_entries": {"readme.txt": b"not a numpy file".hex()}, + "note": "npz members that are not .npy are returned as raw bytes by NumPy", +}) + +# --------------------------------------------------------------------------------------------- +# 11. Multiple arrays appended to ONE .npy stream (np.save(f, a); np.save(f, b)) +# --------------------------------------------------------------------------------------------- +_multi = io.BytesIO() +_seq = [np.arange(3, dtype=np.int32), _values("float64", 4).reshape(2, 2), np.array(9, dtype=np.int8)] +for a in _seq: + fmt.write_array(_multi, a) +CASES.append({ + "name": "stream_multi_array", "file": "cases/stream_multi_array.npy", "kind": "sequence", + "bytes": _multi.getvalue(), + "descr": None, "np_dtype": None, "ns_dtype": None, "shape": None, + "fortran_order": None, "version": None, "data_offset": None, "ns_bytes": None, + "write_exact": True, "load_error": None, + "sequence": [ + {"ns_dtype": _ns_dtype_of(a), "shape": [int(d) for d in a.shape], "ns_bytes": _ns_bytes(a)} + for a in _seq + ], + "note": "three arrays appended to one stream; each np.load reads exactly one and leaves the " + "position at the next. A 4th load must raise EOFError.", +}) + + +# --------------------------------------------------------------------------------------------- +# Emit +# --------------------------------------------------------------------------------------------- +def main(): + os.makedirs(os.path.dirname(OUT), exist_ok=True) + manifest = [] + # ZIP_DEFLATED + a fixed date_time keeps the committed zip byte-stable across regenerations. + with zipfile.ZipFile(OUT, "w", zipfile.ZIP_DEFLATED, compresslevel=9) as zf: + for case in CASES: + case.setdefault("load_via", "load") + case.setdefault("max_header_size", None) + raw = case.pop("bytes") + info = zipfile.ZipInfo(case["file"], date_time=(1980, 1, 1, 0, 0, 0)) + info.compress_type = zipfile.ZIP_DEFLATED + info.external_attr = 0o644 << 16 + zf.writestr(info, raw) + case["size"] = len(raw) + manifest.append(case) + + info = zipfile.ZipInfo("manifest.json", date_time=(1980, 1, 1, 0, 0, 0)) + info.compress_type = zipfile.ZIP_DEFLATED + info.external_attr = 0o644 << 16 + zf.writestr(info, json.dumps({ + "numpy_version": np.__version__, + "generator": "test/oracle/gen_npy_oracle.py", + "cases": manifest, + }, indent=1, sort_keys=True)) + + kinds = {} + for c in manifest: + kinds[c["kind"]] = kinds.get(c["kind"], 0) + 1 + print(f"wrote {os.path.relpath(OUT)} ({os.path.getsize(OUT):,} bytes)") + print(f" cases : {len(manifest)}") + for k, v in sorted(kinds.items()): + print(f" {k:10s} : {v}") + print(f" write_exact : {sum(1 for c in manifest if c['write_exact'])}") + print(f" load_error : {sum(1 for c in manifest if c['load_error'])}") + print(f" numpy : {np.__version__}") + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/test/oracle/interop_write.cs b/test/oracle/interop_write.cs new file mode 100644 index 000000000..3b2b2d34d --- /dev/null +++ b/test/oracle/interop_write.cs @@ -0,0 +1,97 @@ +#:project ../../src/NumSharp.Core +#:property AssemblyName=NumSharp.DotNetRunScript +#:property PublishAot=false +#:property AllowUnsafeBlocks=true +#:property LangVersion=latest + +// Writes a spread of .npy/.npz files with NumSharp for verify_npy_interop.py to read back with real +// NumPy. This is the reverse of the committed oracle: gen_npy_oracle.py proves NumSharp READS NumPy +// and reproduces its bytes, this proves NumPy READS NumSharp — the direction byte-equality cannot +// cover for .npz, whose zip framing legitimately differs. +// +// Run via: python test/oracle/verify_npy_interop.py + +using System.Globalization; +using System.Numerics; +using System.Text.Json; +using NumSharp; + +string outDir = args.Length > 0 ? args[0] : Path.Combine(Path.GetTempPath(), "numsharp_interop"); +Directory.CreateDirectory(outDir); + +var manifest = new List>(); + +void Npy(string name, NDArray arr, string expect) +{ + np.save(Path.Combine(outDir, name + ".npy"), arr); + manifest.Add(new Dictionary + { + ["file"] = name + ".npy", ["kind"] = "npy", ["expect"] = expect, + ["dtype"] = arr.typecode.ToString(), ["shape"] = arr.shape, + }); +} + +void Npz(string name, Dictionary arrays, bool compressed, string expect) +{ + string path = Path.Combine(outDir, name + ".npz"); + if (compressed) np.savez_compressed(path, arrays); + else np.savez(path, arrays); + + manifest.Add(new Dictionary + { + ["file"] = name + ".npz", ["kind"] = "npz", ["expect"] = expect, + ["entries"] = arrays.ToDictionary(kv => kv.Key, kv => (object)new Dictionary + { + ["dtype"] = kv.Value.typecode.ToString(), ["shape"] = kv.Value.shape, + }), + }); +} + +// --- every dtype NumSharp can write --- +Npy("bool", np.array(new[] { true, false, true }), "[True, False, True]"); +Npy("uint8", np.array(new byte[] { 0, 1, 255 }), "[0, 1, 255]"); +Npy("int8", np.array(new sbyte[] { -128, 0, 127 }), "[-128, 0, 127]"); +Npy("int16", np.array(new short[] { -32768, 0, 32767 }), "[-32768, 0, 32767]"); +Npy("uint16", np.array(new ushort[] { 0, 1, 65535 }), "[0, 1, 65535]"); +Npy("int32", np.array(new[] { -2147483648, 0, 2147483647 }), "[-2147483648, 0, 2147483647]"); +Npy("uint32", np.array(new uint[] { 0, 1, 4294967295 }), "[0, 1, 4294967295]"); +Npy("int64", np.array(new[] { long.MinValue, 0L, long.MaxValue }), "[-9223372036854775808, 0, 9223372036854775807]"); +Npy("uint64", np.array(new[] { 0UL, 1UL, ulong.MaxValue }), "[0, 1, 18446744073709551615]"); +Npy("float16", np.array(new[] { (Half)1.5f, (Half)(-2.25f), Half.PositiveInfinity }), "[1.5, -2.25, inf]"); +Npy("float32", np.array(new[] { 1.5f, float.NaN, float.NegativeInfinity }), "[1.5, nan, -inf]"); +Npy("float64", np.array(new[] { 1.5, -0.0, double.MaxValue }), "[1.5, -0.0, 1.7976931348623157e+308]"); +Npy("complex128", np.array(new[] { new Complex(1, 2), new Complex(-3, double.NaN) }), "[1+2j, -3+nanj]"); +Npy("char", np.array(new[] { 'a', 'Z', 'é' }), "['a', 'Z', 'é'] as ()), "shape (0,)"); +Npy("2d", np.arange(6).reshape(2, 3), "shape (2,3)"); +Npy("3d", np.arange(24).reshape(2, 3, 4), "shape (2,3,4)"); +Npy("empty_2d", np.zeros(new Shape(0, 4), NPTypeCode.Int32), "shape (0,4)"); + +// --- layouts --- +Npy("fortran", np.arange(6).reshape(2, 3).copy('F'), "F-contiguous, fortran_order: True"); +Npy("transposed", np.arange(12).reshape(3, 4).T, "transposed view -> fortran_order: True"); +Npy("strided", np.arange(12)["::2"], "strided view -> C-order copy"); +Npy("sliced_2d", np.arange(12).reshape(3, 4)["1:, ::2"], "offset+strided view -> C-order copy"); + +// --- large (chunked write path) --- +Npy("large", np.arange(100_000).astype(typeof(double)), "100k float64, chunked"); + +// --- npz --- +Npz("npz_single", new() { ["arr_0"] = np.arange(6).reshape(2, 3) }, false, "single member"); +Npz("npz_multi", new() +{ + ["weights"] = np.arange(12).reshape(3, 4).astype(typeof(float)), + ["biases"] = np.array(new[] { 0.5, 1.5 }), + ["flag"] = np.array(true), + ["text"] = np.array(new[] { 'h', 'i' }), +}, false, "mixed dtypes"); +Npz("npz_compressed", new() { ["big"] = np.zeros(new Shape(50_000), NPTypeCode.Double) }, true, "deflated"); +Npz("npz_fortran", new() { ["f"] = np.arange(6).reshape(2, 3).copy('F') }, false, "F-order inside npz"); + +File.WriteAllText(Path.Combine(outDir, "manifest.json"), + JsonSerializer.Serialize(manifest, new JsonSerializerOptions { WriteIndented = true })); + +Console.WriteLine($"{manifest.Count} files -> {outDir}"); diff --git a/test/oracle/verify_npy_interop.py b/test/oracle/verify_npy_interop.py new file mode 100644 index 000000000..dc238c623 --- /dev/null +++ b/test/oracle/verify_npy_interop.py @@ -0,0 +1,163 @@ +""" +verify_npy_interop.py — prove real NumPy can read what NumSharp writes. + +The committed oracle (gen_npy_oracle.py + NpyOracleTests) proves the other direction: NumSharp reads +NumPy's files and reproduces NumPy's bytes exactly. That covers .npy completely — byte-equality is +the strongest claim available — but NOT .npz, whose ZIP framing (timestamps, deflate parameters, +Zip64 records) legitimately differs between Python's zipfile and .NET's ZipArchive. A NumSharp .npz +only has to be a *valid* archive of valid members, so the way to prove it is to have NumPy open it. + +This runs NumSharp (via `dotnet run`), then reads every file it produced with NumPy and checks the +dtype, shape, layout and values. It needs both Python and the .NET SDK, so it is a developer/manual +gate rather than a CI test — the same role the nightly fuzz soak plays. + + python test/oracle/verify_npy_interop.py [--keep] +""" +import json +import os +import subprocess +import sys +import tempfile + +import numpy as np + +HERE = os.path.dirname(os.path.abspath(__file__)) +REPO = os.path.abspath(os.path.join(HERE, "..", "..")) +WRITER = os.path.join(HERE, "interop_write.cs") + +# NPTypeCode -> the dtype NumPy must report. Char rides ' {out}") + r = subprocess.run( + ["dotnet", "run", "-c", "Release", WRITER, "--", out], + cwd=REPO, capture_output=True, text=True, + ) + if r.returncode != 0: + print(r.stdout[-4000:]) + print(r.stderr[-4000:], file=sys.stderr) + raise SystemExit(f"NumSharp writer failed (exit {r.returncode})") + print(" " + r.stdout.strip().splitlines()[-1]) + + manifest = json.load(open(os.path.join(out, "manifest.json"))) + + for case in manifest: + path = os.path.join(out, case["file"]) + name = case["file"] + + if case["kind"] == "npy": + arr = np.load(path) # allow_pickle=False: nothing NumSharp writes may need it + verify_array(name, arr, case) + + # The layout claims must survive: an F-order file is F-contiguous once NumPy reads it. + if name in ("fortran.npy", "transposed.npy"): + check(arr.flags.f_contiguous and not arr.flags.c_contiguous, name, + "expected NumPy to read this back as F-contiguous") + if name in ("2d.npy", "3d.npy", "strided.npy", "sliced_2d.npy"): + check(arr.flags.c_contiguous, name, "expected C-contiguous") + + # Spot-check values NumSharp claimed to write. + if name == "int64.npy": + check(list(arr) == [np.iinfo(np.int64).min, 0, np.iinfo(np.int64).max], name, f"values {arr}") + elif name == "uint64.npy": + check(list(arr) == [0, 1, np.iinfo(np.uint64).max], name, f"values {arr}") + elif name == "float64.npy": + check(arr[0] == 1.5 and np.signbit(arr[1]) and arr[1] == 0.0 and arr[2] == np.finfo(np.float64).max, + name, f"values {arr} (note -0.0 must keep its sign bit)") + elif name == "float32.npy": + check(arr[0] == 1.5 and np.isnan(arr[1]) and arr[2] == -np.inf, name, f"values {arr}") + elif name == "float16.npy": + check(arr[0] == 1.5 and arr[1] == -2.25 and arr[2] == np.inf, name, f"values {arr}") + elif name == "complex128.npy": + check(arr[0] == 1 + 2j and arr[1].real == -3 and np.isnan(arr[1].imag), name, f"values {arr}") + elif name == "char.npy": + check(list(arr) == ["a", "Z", "é"], name, f"values {list(arr)} — NumSharp Char must widen to UCS-4") + elif name == "scalar.npy": + check(arr.shape == () and arr[()] == 42, name, f"0-d scalar, got {arr!r}") + elif name == "fortran.npy": + check(np.array_equal(arr, np.arange(6).reshape(2, 3)), name, f"values {arr}") + elif name == "transposed.npy": + check(np.array_equal(arr, np.arange(12).reshape(3, 4).T), name, f"values {arr}") + elif name == "strided.npy": + check(np.array_equal(arr, np.arange(12)[::2]), name, f"values {arr}") + elif name == "sliced_2d.npy": + check(np.array_equal(arr, np.arange(12).reshape(3, 4)[1:, ::2]), name, f"values {arr}") + elif name == "large.npy": + check(np.array_equal(arr, np.arange(100_000, dtype=np.float64)), name, "100k values differ") + elif name == "2d.npy": + check(np.array_equal(arr, np.arange(6).reshape(2, 3)), name, f"values {arr}") + + # The header must be exactly what NumPy's own writer would emit for this array. + with open(path, "rb") as fh: + raw = fh.read() + import io + mine = io.BytesIO() + np.lib.format.write_array(mine, arr) + check(mine.getvalue() == raw, name, + "file is not byte-identical to NumPy's own np.save output for the same array") + + else: # npz + with np.load(path) as z: + check(sorted(z.files) == sorted(case["entries"]), name, + f".files {sorted(z.files)} != {sorted(case['entries'])}") + for key, meta in case["entries"].items(): + verify_array(f"{name}[{key}]", z[key], meta) + + if name == "npz_multi.npz": + check(np.array_equal(z["weights"], np.arange(12).reshape(3, 4).astype(np.float32)), + name, "weights differ") + check(list(z["biases"]) == [0.5, 1.5], name, "biases differ") + check(bool(z["flag"]) is True, name, "flag differs") + check(list(z["text"]) == ["h", "i"], name, "text differs") + elif name == "npz_fortran.npz": + f = z["f"] + check(f.flags.f_contiguous and not f.flags.c_contiguous, name, + "F-order must survive inside an npz") + check(np.array_equal(f, np.arange(6).reshape(2, 3)), name, "values differ") + elif name == "npz_compressed.npz": + check(np.array_equal(z["big"], np.zeros(50_000)), name, "values differ") + + print(f"\n{len(manifest)} files, {checked} checks") + if failures: + print(f"\n{len(failures)} FAILURE(S):") + for f in failures: + print(" " + f) + raise SystemExit(1) + + print("PASS — NumPy reads every file NumSharp wrote, and every .npy is byte-identical to NumPy's own output.") + if keep: + print(f"files kept at {out}") + else: + import shutil + shutil.rmtree(out, ignore_errors=True) + + +if __name__ == "__main__": + main()