feat(namespace): support Lance Namespace across table operations

[FANNG1]

Closes #50 (motivating use case: Gravitino Lance REST server managed tables + Daft multimodal processing).

Summary

• add a shared Lance namespace resolver for URI-or-namespace validation, namespace client construction, table resolution, and storage option merging
• wire namespace table support through daft_lance.read_lance, the new daft_lance.write_lance, LanceDataSink, merge/index/compaction operations, and worker scan reconstruction
• keep dataset-open state in an explicit LanceDatasetHandle instead of attaching Daft-private attributes to LanceDataset
• explicitly thread resolved URIs, vended storage options, and namespace commit kwargs through scan, merge, and distributed index paths
• reopen datasets on distributed workers from a serializable DatasetOpenContext instead of shipping a live LanceDataset, so maintenance tasks keep their namespace identity
• add dir namespace coverage for reads/writes, transform and slow-path merges, segmented/non-segmented distributed indexes, compaction, credential vending, and worker reconstruction, plus a gated Gravitino Lance REST namespace integration test
• tighten namespace table-not-found handling and request vended credentials explicitly
• recover the overwrite path from a lost declare_table race by re-describing, while leaving create to fail on the conflict

Review fixes

Three blocking issues raised in review, all fixed here:

Namespace locations are now percent-decoded. Namespaces vend URI-encoded
locations: a table under daft lance/ came back from the dir namespace as
file:///.../daft%20lance/t.lance, and _normalize_file_uri took parsed.path
verbatim. Writes landed in a literally-named daft%20lance directory while the
subsequent describe_table resolved to the decoded one, so create appeared to
succeed and reads found nothing. Covered by unit tests (spaces, non-ASCII,
literal percent, object-store passthrough) plus an end-to-end roundtrip through
a namespace root and table id that both encode.

The construct_lance_dataset() compatibility wrapper is removed. It
returned only .dataset, discarding the resolved uri, open_kwargs,
managed_versioning and default_scan_options. open_kwargs is the
serialization boundary to workers (lance_scan.py puts it in the scan task's
func_args, and open_dataset_from_open_kwargs rebuilds the namespace client
from it), so callers going through the wrapper left workers with {}: no vended
credentials, no namespace triple, and no pinned version, meaning partitions of
one scan could read different snapshots. The stripped-nearest regression was a
second symptom — the dataset's own _default_scan_options has already had
nearest removed by then, so it cannot be recovered from the object. Rather
than reconstruct any of this from LanceDataset private attributes, the wrapper
is deleted; it had no callers inside daft_lance and was never exported.

Namespace + use_mem_wal=True is now rejected at construction time. The
namespace path declares the table up front as a metadata-only reservation, then
_ensure_mem_wal_dataset() asked Lance for a namespace-aware
write_dataset(mode="create"), which declares the same table again —
DeclareTable is not idempotent, so this always raised
TableAlreadyExistsError. The surrounding except (ValueError, FileNotFoundError, OSError) does not cover the namespace error types either. Making it work
requires hoisting the native create to the driver's start() so mem-WAL table
creation is not raced by N workers; that is left to a follow-up rather than
claiming support that cannot work.

Distributed worker reopen

LanceDataset.__reduce__ carries only (uri, storage_options, version, manifest, ...). _namespace_client, _table_id and _namespace_client_managed_versioning
are assigned after construction, so pickle drops them. Compaction, scalar index
and merge each shipped the driver's live dataset into their UDFs, which meant
workers silently lost the table's namespace identity and committed as though it
were a plain URI table.

Workers now receive a frozen, serializable DatasetOpenContext — physical uri,
the driver's resolved numeric version, effective initial storage options, the
namespace triple, managed_versioning, and the worker read options — and reopen
from it, rebuilding the namespace client per process through the existing
lru_cache. The driver keeps its live dataset for planning, validation and
commits.

Reopen goes through the low-level LanceDataset(uri, namespace_client=...)
constructor rather than lance.dataset(None, namespace_client=..., table_id=...),
which resolves the location with a describe_table on every call. Measured
with pylance's ops_metrics: the low-level open costs zero namespace calls
(open and scan), the high-level one costs a round-trip per task. Each UDF
instance opens once, lazily — the reopen costs a pinned-manifest read, so it must
not sit on the per-call path.

The serialized manifest is deliberately not carried, keeping the task payload
independent of fragment count at the cost of that manifest read. This matches
the lance-spark and Lance Ray maintenance paths. Version semantics are preserved:
workers pin the snapshot the driver planned against, and only the index
coordinator steps that must observe worker output reopen at latest.

Scan workers are pinned too. construct_lance_dataset_handle stored the
caller's version argument in open_kwargs — defaulting to None — and
open_kwargs is precisely what crosses to scan workers, so each task
independently opened latest. Both failure modes reproduce: a compaction
landing between planning and execution leaves workers hunting fragment ids that
no longer exist (planned [0,1,2], worker sees [3], get_fragment(0) returns
None), and an overwrite silently substitutes different data with no error at
all. open_kwargs now carries the resolved numeric dataset.version, and
asof is dropped since it is an input to that resolution rather than something
workers should re-evaluate.

Since the context supplies uri, storage options and namespace kwargs, the
internal entry points drop those four parameters rather than carry a second copy
of the same state, and FastPathFragmentWriter loses its private
uri/storage_options fields for the same reason. Scan otherwise keeps its
existing open_kwargs path — unifying the two state models is left to a
separate change so this one does not perturb tag/default-scan-option behavior.

Known limitation. Workers start from the storage options the driver resolved.
pylance installs its namespace refresh provider only when those options are
present (python/src/dataset.rs, gated on initial_storage_options.is_some()),
and the accessor refreshes on expires_at_millis; the upstream provider's
DescribeTableRequest does not set vend_credentials=True. So the wiring is
restored and refresh is possible, but reliable refresh depends on the catalog
vending expiry metadata and credentials — a task outliving its credentials can
still fail. Full credential refresh, including expired-credential tests, is
tracked separately.

Follow-ups (not in this PR)

• Worker credential refresh. See the known limitation above: explicit
  vend_credentials=True on the refresh request, expired-credential and
  replacement-token tests, and refresh failure/retry behavior. It should also
  carry a switch for whether workers may rebuild the namespace client at all —
  lance-spark's executorCredentialRefresh defaults to true but is explicitly
  disableable, since not every catalog is reachable from the worker network.
• Recovery from a failed write after declare. A declared-but-empty table is a
  protocol-allowed intermediate state; the namespace spec deliberately keeps
  catalogs out of slow, unreliable data deletion. Cleanup should be
  DeregisterTable scoped to a table this operation declared and confirmed still
  declared-only, not an unconditional drop_table on worker failure — and Daft's
  DataSink has no abort callback to hang it on.

Validation

• .venv/bin/pytest -q (347 passed, 5 skipped, 2 xfailed, 2 xpassed)
• .venv/bin/ruff format --check daft_lance tests
• .venv/bin/ruff check daft_lance tests
• .venv/bin/mypy daft_lance
• git diff --check

Cross-repo plan

daft_lance.write_lance in this PR looks like a copy of DataFrame.write_lance, but the
direction is the opposite: Eventual-Inc/Daft#6957 already migrated all of daft.io.lance to
this package, and DataFrame.write_lance's inline body is the last piece left behind
upstream. Eventual-Inc/Daft#7282 proposes reducing it to a thin delegation to
daft_lance.write_lance, making this module the single canonical implementation.

Until Eventual-Inc/Daft#7282 lands, namespace users call daft_lance.read_lance /
daft_lance.write_lance directly. (An earlier revision of this PR shipped a patch_daft()
monkeypatch as interim glue for Daft's native entry points; it was removed in favor of the
direct entry points since it had never been released.)

[universalmind303]

thanks for the PR @FANNG1, Will take a look asap!

[universalmind303]

I ran this through a pretty deep review & a no-mocks e2e harness (real dir namespace via lance-namespace: create/append/read/pushdowns/merge columns/index/compaction all through the triple, cross-checked with plain pylance). happy path holds up nicely. found a handful of real issues though.

I think the percent-encoding & compat wrapper ones need fixing before merge, the rest can be follow ups.