feat: Handle all forms of range requests in fsspec

[gschulze]

Closes #259.

Adds support for all forms of range requests documented by fsspec, without changing obstore's own API.

Implemented:

• cat_file and cat_ranges accept every range form fsspec documents: either bound alone, and either counting back from the end of the object.
• cat_ranges additionally takes a scalar or None broadcast across all paths, and None elements. It also honors max_gap and batch_size, which were previously ignored; fsspec's own _cat_ranges raises NotImplementedError for max_gap. on_error stays ignored, as it is upstream.
• test_cat_ranges_mixed is no longer xfail. New tests cover the range forms cat_file accepts, the request each one turns into, the bounded/open-ended split, and the forwarding of max_gap and batch_size. All but the first would still return the right bytes if they regressed.

Request cost: A plain start/end goes through get_range; start-only and negative-start become get with {"offset": n} and {"suffix": n}, so they stay single requests. cat_ranges splits its input the same way _get_partial_values does in the zarr PR: bounded ranges batch per object through get_ranges so nearby ones are merged, and each open-ended range is a separate get request. The object size is only needed for a negative end, or a negative start paired with an end, since neither has a GetOptions equivalent.

Degenerate ranges: Zero-length, inverted, and start-past-the-end ranges still raise rather than returning empty, as they do today. The first two are rejected by validate_range in obstore/src/get.rs, the third by object_store itself. fsspec's own backends disagree here anyway: memory returns empty for all three, file raises for inverted.

This only touches fsspec.py and its tests. get_range and get_ranges are unchanged.

[kylebarron]

This seems unnecessarily complicated when you could just add a parameter into a dict

[kylebarron]

Resolving a range should never need to know the size of the object (except on Azure, which doesn't support suffix requests).

I'd strongly recommend taking a similar approach to the Zarr-Python obstore adapter. https://github.com/zarr-developers/zarr-python/blob/d44f9f92ab4f12a8008de2553a7c9988669e3910/src/zarr/storage/_obstore.py#L440-L491

(For Azure, you can have a config parameter for the fsspec adapter for whether to avoid suffix requests, which then would make a HEAD request to always know the size)

[kylebarron]

Can you link to source code in fsspec that supports this typing? I.e. are there tests or a code path where we know that starts and ends can take None, either in isolation or as an element in a sequence?

[kylebarron]

Overall this PR still seems to be overly complex for what it does (I'd say it seems to be written too much by Claude), and needs to be simplified a lot before merge

[kylebarron]

Context from a Claude review on my side, if you're interested. From a quick read of this summary, I think all of these are valid points that would need to be addressed before merging.

---

Thanks for the rework — going the adapter-only route is right, and the overall shape is what I had in mind: bounded ranges batched per object through get_ranges_async, open-ended ones through get_async with {"offset"}/{"suffix"}, mirroring _get_partial_values.

I built this branch and ran the suite against minio: 28 passed, 1 xfail, and all the new tests pass. ruff is clean; pyright shows only the pre-existing missing-stub errors for the fsspec imports.

Some follow-ups on my inline comments, including one where I was wrong.

Follow-ups on the inline comments

_CoalesceKwarg (fsspec.py:126)

Stands — and it's a one-liner. Both pyright and mypy accept the plain form:

kw: dict[str, int] = {} if max_gap is None else {"coalesce": max_gap}
store.get_ranges_async(path, starts=..., ends=..., **kw)

I see where the TypedDict came from: ebca428 refactor: Accept None for coalesce so the default lives in one place got reverted wholesale in 275e3c6 along with the get_opts change. My objection was only to get_range routing through get_opts — coalesce: int | None = None on the binding is independently reasonable and I'd take it as a follow-up if you'd rather have the default live in one place.

Needing the object size (fsspec.py:131)

I was too absolute here. A negative end genuinely has no HTTP range equivalent: data[:-10] is bytes 0..size-10, and a suffix request gives the last N bytes, not all but the last N. fsspec's own reference implementation does exactly what this PR does:

# fsspec/spec.py, AbstractFileSystem.cat_file
if end is not None:
    if end < 0:
        end = f.size + end

And test_cat_ranges_mixed — the test this issue exists to un-xfail — passes ends=[None, -10, -10]. The zarr adapter avoids the problem only because zarr's ByteRequest is a closed set (Range/Offset/Suffix) with no negative end, so it doesn't port directly. Fetching the size for a negative end is fine.

There is a real bug in the neighborhood, though. _resolve_inexpressible_bounds applies the fetched size to every row, not just the rows that needed it:

fs.cat_ranges([path, path], starts=[-5, 0], ends=[None, -10])
# range actually sent for row 0: {'offset': 9995}   ← should be {"suffix": 5}

Row 0 is an exactly-expressible suffix, but it gets rewritten into a size-dependent offset because row 1 happened to need a HEAD. So a pure suffix silently becomes size-dependent based on unrelated entries in the same call. Please gate _apply_object_size on _needs_object_size per row so rows that don't need the size are left alone.

Sequence[int | None] typing (fsspec.py:479)

You're right, and here's the source I was asking for — worth putting a pointer to one of these in a comment:

• AbstractFileSystem.cat_ranges (fsspec/spec.py) forwards each element unchanged: out.append(self.cat_file(p, s, e)). cat_file's documented contract is "start, end: int — If negative, backwards from end, like usual python slices. Either can be None for start or end of file, respectively."
• AsyncFileSystem._cat_ranges (fsspec/asyn.py) does the same: self._cat_file(p, start=s, end=e) for p, s, e in zip(paths, starts, ends).
• fsspec.utils.merge_offset_ranges normalizes starts = [s or 0 for s in starts] and guards e is not None — None elements are expected there.
• Scalar None broadcast follows from if not isinstance(starts, Iterable), since None isn't Iterable.

Other things I found

on_error diverges from upstream, and the comment justifying it is wrong. The code says "Like fsspec's own _cat_ranges, on_error is ignored, so failures propagate." Upstream ignores the on_error argument but hardcodes return_exceptions=True, so it always behaves as the default on_error="return":

upstream memory fs: [b'abc', FileNotFoundError('/nope')]
this branch:        raises FileNotFoundError

Not a regression — the old asyncio.gather also raised — but the comment should be corrected either way. If we want parity it's one kwarg plus widening output_buffers to list[bytes | BaseException].

A fully-unbounded element in cat_ranges still sends a range header. _cat_file special-cases start in (0, None), end=None into a plain get, but _cat_ranges turns it into {"offset": 0} → Range: bytes=0-, which 416s on a zero-length object where cat_file returns b"". Cheap to mirror the special case.

Degenerate ranges. Agreed with the choice to keep raising, but note upstream isn't merely inconsistent between backends — the reference cat_file does f.read(end - f.tell()), and AbstractBufferedFile.read treats a negative length as "rest of file", so an inverted range silently returns the tail. Raising is better behavior; it just deserves a line in the cat_file/cat_ranges docstring since it deviates from the documented slice semantics.

fsspec.asyn._run_coros_in_chunks is private API. It's the only way to honor batch_size and the adapter already leans on fsspec internals, so I'm fine with it — just noting the coupling.

Azure. This introduces suffix requests where the adapter previously made none, and Azure doesn't support them. Out of scope for this PR — I'll open a separate issue. (The zarr adapter catches the failure and falls back to HEAD + bounded range.)

[gschulze]

Thanks for your detailed feedback, this is all fair. Everything you raised is implemented locally. Regarding complexity, I'm afraid the PR has not gotten smaller after incorporating your points, but I think the code quality has improved. I still need some time to check whether everything is consistent now, and whether I can simplify it any further. Will ping when ready for another look.