Skip to content

DFViewer transport abstraction: pick JSON / b64-parquet / binary per embedding method #881

Description

@paddymul

Summary

DFViewer data payloads are serialized with a transport hardcoded per call site, with no way for an
embedding to pick the best one for its channel:

  • Row windows (infinite_resp): raw binary frames in the widget
    (self.send(msg, [parquet_bytes]), buckaroo_widget.py:447) and server
    (write_message(parquet_bytes, binary=True), websocket_handler.py:227), but base64
    parquet in embedded/static HTML (artifact.py, BuckarooStaticTable.tsx).
  • Summary stats (all_stats): parquet_b64 everywhere, including the server, where it
    rides inside the initial_state JSON text frame (build_state_message
    websocket_handler.py:46) even though the WebSocket has a binary channel sitting right there.

So the server pays a flat +33% base64 inflation on stats for no reason other than that
initial_state is assembled as a single JSON message. Different embeddings have different
capabilities — anywidget (binary buffers), Tornado WS (binary frames), static HTML (no channel;
needs inline b64 or a fetchable URL), Tauri IPC — and today the choice is implicit and inconsistent.

Suggested fix

A small transport abstraction in DFViewer: payloads carry a format tag
(json | parquet_b64 | binary-ref), the resolver (resolveDFData.ts) handles all three
uniformly, and each embedding declares which transports it supports / prefers; the Python side
serializes accordingly. base64 only where there is genuinely no binary channel (static HTML).

Relates to #857 (static-embed should export parquetRead so raw parquet can be fetched without
base64) — same direction, broader scope.

Impact

  • Server stats stop paying the +33% b64 tax (3.2 MB → 4.3 MB on a 191-col frame today).
  • One coherent place to reason about transport instead of per-call-site decisions.
  • Lets the initial-load cache (docs: design for cacheable initial load (jupyter + server) #877) persist raw parquet on disk and choose the wire transport at
    serve time per embedding.

Scope

serialization_utils.py, components/DFViewerParts/resolveDFData.ts + DFWhole.ts
(ParquetB64Payload), the send sites (buckaroo_widget.py, server/websocket_handler.py,
polars_buckaroo.py, xorq_buckaroo.py), server/BuckarooView.tsx / BuckarooServerView.tsx.

Context

Identified while designing the initial-load cache (#877) and measuring base64 overhead: the b64
decode itself is sub-millisecond, but the +33% size is real on every server stats payload.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions