You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
DFViewer data payloads are serialized with a transport hardcoded per call site, with no way for an
embedding to pick the best one for its channel:
Row windows (infinite_resp): raw binary frames in the widget
(self.send(msg, [parquet_bytes]), buckaroo_widget.py:447) and server
(write_message(parquet_bytes, binary=True), websocket_handler.py:227), but base64
parquet in embedded/static HTML (artifact.py, BuckarooStaticTable.tsx).
Summary stats (all_stats): parquet_b64everywhere, including the server, where it
rides inside the initial_state JSON text frame (build_state_message → websocket_handler.py:46) even though the WebSocket has a binary channel sitting right there.
So the server pays a flat +33% base64 inflation on stats for no reason other than that initial_state is assembled as a single JSON message. Different embeddings have different
capabilities — anywidget (binary buffers), Tornado WS (binary frames), static HTML (no channel;
needs inline b64 or a fetchable URL), Tauri IPC — and today the choice is implicit and inconsistent.
Suggested fix
A small transport abstraction in DFViewer: payloads carry a format tag
(json | parquet_b64 | binary-ref), the resolver (resolveDFData.ts) handles all three
uniformly, and each embedding declares which transports it supports / prefers; the Python side
serializes accordingly. base64 only where there is genuinely no binary channel (static HTML).
Relates to #857 (static-embed should export parquetRead so raw parquet can be fetched without
base64) — same direction, broader scope.
Impact
Server stats stop paying the +33% b64 tax (3.2 MB → 4.3 MB on a 191-col frame today).
One coherent place to reason about transport instead of per-call-site decisions.
Identified while designing the initial-load cache (#877) and measuring base64 overhead: the b64
decode itself is sub-millisecond, but the +33% size is real on every server stats payload.
Summary
DFViewer data payloads are serialized with a transport hardcoded per call site, with no way for an
embedding to pick the best one for its channel:
infinite_resp): raw binary frames in the widget(
self.send(msg, [parquet_bytes]),buckaroo_widget.py:447) and server(
write_message(parquet_bytes, binary=True),websocket_handler.py:227), but base64parquet in embedded/static HTML (
artifact.py,BuckarooStaticTable.tsx).all_stats):parquet_b64everywhere, including the server, where itrides inside the
initial_stateJSON text frame (build_state_message→websocket_handler.py:46) even though the WebSocket has a binary channel sitting right there.So the server pays a flat +33% base64 inflation on stats for no reason other than that
initial_stateis assembled as a single JSON message. Different embeddings have differentcapabilities — anywidget (binary buffers), Tornado WS (binary frames), static HTML (no channel;
needs inline b64 or a fetchable URL), Tauri IPC — and today the choice is implicit and inconsistent.
Suggested fix
A small transport abstraction in DFViewer: payloads carry a format tag
(
json|parquet_b64|binary-ref), the resolver (resolveDFData.ts) handles all threeuniformly, and each embedding declares which transports it supports / prefers; the Python side
serializes accordingly. base64 only where there is genuinely no binary channel (static HTML).
Relates to #857 (static-embed should export
parquetReadso raw parquet can be fetched withoutbase64) — same direction, broader scope.
Impact
serve time per embedding.
Scope
serialization_utils.py,components/DFViewerParts/resolveDFData.ts+DFWhole.ts(
ParquetB64Payload), the send sites (buckaroo_widget.py,server/websocket_handler.py,polars_buckaroo.py,xorq_buckaroo.py),server/BuckarooView.tsx/BuckarooServerView.tsx.Context
Identified while designing the initial-load cache (#877) and measuring base64 overhead: the b64
decode itself is sub-millisecond, but the +33% size is real on every server stats payload.