Skip to content

zarr backend: non-ASCII variable names are percent-encoded in store keys #337

Description

@velochy

When sampling into a zarr_store, array keys are written percent-encoded while the
dimension_names metadata is not. Any model with a non-ASCII variable or coordinate name
therefore reads back under a different name than the model declared, and its dimension
coordinate is demoted to a data variable. The arrow backend is unaffected.

Reproduction

import pymc as pm, nutpie, tempfile
from nutpie import zarr_store

with pm.Model(coords={"abieluvõrdsus_outp": ["a", "b", "c"]}) as m:
    z = pm.Normal("abieluvõrdsus", dims="abieluvõrdsus_outp")
    pm.Normal("obs", z.sum(), 1.0, observed=1.0)

c = nutpie.compile_pymc_model(m, backend="numba")
arrow = nutpie.sample(c, draws=20, tune=20, chains=2, progress_bar=False)
zarr = nutpie.sample(c, draws=20, tune=20, chains=2, progress_bar=False,
                     zarr_store=zarr_store.LocalStore(tempfile.mkdtemp()))

print(sorted(map(str, arrow["posterior"].dataset.data_vars)))
print(sorted(map(str, zarr["posterior"].dataset.data_vars)))
['abieluvõrdsus']
['abieluv%C3%B5rdsus', 'abieluv%C3%B5rdsus_outp']

Two things went wrong. The variable is renamed, and the coordinate array — which should be a
dimension coordinate — is now a data variable, because its name no longer equals its dimension.

On disk the store contains /posterior/abieluv%C3%B5rdsus and
/posterior/abieluv%C3%B5rdsus_outp, while the arrays' dimension_names metadata still reads
abieluvõrdsus_outp. So the encoding is applied to the store key but not to the metadata that
has to agree with it.

Scope of the encoding: only non-ASCII bytes are affected — spaces, hyphens and parentheses come
through untouched (Euroopa Liit_outp is fine). A literal % is escaped as %25, so the
transformation is invertible with urllib.parse.unquote.

Impact

Downstream, pm.sample_posterior_predictive transposes every posterior data variable to
(chain, draw, ...), and the demoted coordinate has neither:

ValueError: Dimensions {'draw', 'chain'} do not exist.
Expected one or more of ('abieluvõrdsus_outp',)

The quieter failure is the one that worries me more: a non-ASCII variable name with no
coordinate sharing its name is silently renamed, with nothing raised at all. Every model in a
language with non-ASCII column names is affected — this surfaced across a set of Estonian and
Romanian survey models, where it broke four of six.

Where the encoding comes from

nuts-rs builds keys as plain UTF-8 strings — create_arrays in src/storage/zarr/common.rs
does format!("{}/{}", group_path, name) and hands that to ArrayBuilder::build. The encoding
is applied one layer down, in the object-store adapter:

  • zarrs_object_store 0.6.2, src/lib.rs:50
    fn key_to_path(key: &StoreKey) -> Path { Path::from(key.as_str()) }, called on every store
    operation.
  • object_store 0.13.2, src/path/parts.rs:106PathPart::from(&str) runs
    percent_encode(other, INVALID), which is what escapes the non-ASCII bytes.

dimension_names escapes this because it is JSON content inside the array metadata payload,
never a store key, so it never passes through key_to_path.

That also explains the asymmetry. object_store::path::Path decodes again via parts(), so a
Rust reader going through the same adapter round-trips consistently — but zarr-python reads the
raw keys and sees the encoded form. A zarr key is UTF-8 by spec, so pairing a writer that
applies object-store path semantics with a reader that doesn't is the actual defect; whether
that's best fixed in zarrs_object_store or by nuts-rs avoiding that adapter's encoding for
zarr keys is your call.

One caveat on the above: the zarrs crate itself wasn't available locally to inspect, so I
can't rule out that it normalizes a key before handing it to key_to_path. Everything else
here was read from the vendored sources.

Workaround

For anyone hitting this before it's fixed — decode the keys after reading and re-promote the
arrays whose name matches a dimension:

from urllib.parse import unquote

mangled = {n: unquote(str(n)) for n in ds.variables if "%" in str(n)}
mangled = {k: v for k, v in mangled.items() if v != k}
ds = ds.rename(mangled).set_coords([v for v in mangled.values() if v in ds.dims])

Versions: nutpie 0.16.10, pymc 6.0.1, zarr 3.1.5, xarray 2026.2.0, numba backend, Linux.
Happy to test a fix against the real models that surfaced it.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions