Make datui feel native: a home screen, system theming, and a desktop entry by derekwisong · Pull Request #56 · derekwisong/datui

derekwisong · 2026-09-07T17:57:40Z

Eight commits toward datui feeling like part of the system rather than a command
you invoke. Three strands, plus two bugs found on the way that are worth reading
on their own.

Each commit stands alone and they are ordered to be reviewed in sequence.

1. A home screen — datui as a place

datui with no arguments now opens a home screen listing datasets you can open,
with what each one contains shown before you open it.
Ctrl+O returns there from anywhere.

  datui                                                    ~/work/analysis

  › sal▏

  RECENT
  ▸ sales                        hive       2.4M × 18    340 MB    2d
    customers.parquet                         89k × 12     4 MB    1w

  /mnt/data                                                    configured
    events                       hive        1.1M × 9    120 MB    3h

  ~/work/analysis                                       current directory
    raw_export.csv                                       1.2 GB    3h

Why: datui could only answer "show me this file", which required already
knowing where it was. That resolution step is the friction — hive datasets are
directories, so completion walks into year=2024/month=01/ rather than stopping
at the dataset, and ls cannot tell you whether a Parquet file holds three rows
or thirty million.

Roots. Code lives in your working directory; the datasets usually do not.
The listing is built from [data] directories, the working directory, and —
importantly — the directories of datasets you have opened before. That last
one bridges "code here, data there" with no configuration: you establish the
association by using it once.

Datasets, not files. A hive tree collapses to one row; a homogeneous
directory of Parquet becomes one multi-file dataset; a source tree with two
stray CSVs stays a directory. Classification reads at most 8 entries, so a
dataset with five thousand partitions costs what one with three costs.

Scent. Row and column counts come from Parquet footers, so they are free,
summed across at most 64 files — past that the count is blank rather than a
partial total. A CSV's row count cannot be known without scanning it, so it
stays blank rather than guessed. The preview pane shows the full schema, each
type in the colour the table will use.

Not a catalogue. datui persists exactly one thing: a list of recently opened
paths. Everything else is read at draw time and forgotten. Nothing to register,
nothing to curate, no metadata store to go stale.

Design. datui had no visual identity, so this establishes one: no boxes,
structure from alignment and negative space, one accent spent on the wordmark,
section headers, selection marker and prompt, metadata right-aligned into
columns. Every symbol resolves through glyphs, which falls back to ASCII when
the locale is not UTF-8 — datui has to be readable over SSH on a plain server.
No Nerd Font characters anywhere.

Desktop recents, as places only. A fresh install has no recents, so datui
reads freedesktop's recently-used.xbel and offers the directories it
mentions — never the file names. That list routinely holds things nobody wants
on a screen they are sharing, and they are all valid CSVs, so those places are
listed unexpanded and nothing inside is shown until you press Enter.
[data] use_desktop_recents = false turns it off.

2. System theming

A top-level import key naming config files to merge before the importing
file's own values, matching Alacritty's general.import semantics:

import = ["~/.local/state/omarchy/current/theme/datui.toml"]

Precedence, lowest first: defaults → each import in order → the importing file →
CLI flags. Deliberately generic — no desktop detection, no hardcoded state
paths. The one system-specific line lives in the user's own config, which is how
Omarchy themes Alacritty and btop too.

contrib/omarchy/datui.toml.tpl maps all 43 colour slots onto an Omarchy
palette, with a test that fails if ColorConfig and the template drift apart —
that drift is invisible in review, since a missing slot just silently keeps
datui's default.

Known limitation, documented: ColorConfig::merge distinguishes "set" from
"unset" by comparing against the default, so a colour set to the same string as
datui's default cannot override an import. datui's default error is red, so
error = "red" will not override one. Acceptable while import is opt-in; it
would need a presence-based merge before any auto-applied theming.

3. A desktop entry

datui shipped no .desktop file, so it was invisible to every launcher on every
desktop. scripts/packaging/datui.desktop now installs to
/usr/share/applications/ from all three packagers. Terminal=true,
Exec=datui %F so a bare launch opens the home screen and a file manager can
offer "Open with datui".

This is also the whole Omarchy launcher story: its menu lists applications from
the freedesktop database and folds a desktop entry's Keywords into its search
aliases, so data and parquet find datui with no Omarchy-specific
integration at all.

Two bugs found on the way

Light terminals were unreadable

Seven default colour slots resolve to fixed xterm-256 indices — indexed(235)
for table_header_bg, controls_bg, alternate_row_color and
sidebar_border. Unlike the ANSI names used elsewhere, these do not adapt. On a
light terminal that is header text on a near-black bar, and a row stripe that
renders as a solid dark band.

Header text on the header fill, before → after:

terminal background	before	after
pure white	1.39:1	15.02:1
catppuccin-latte	1.90:1	15.02:1
flexoki-light	1.26:1	15.02:1

1.0:1 is invisible. This is independent of any theming — it affects anyone on a
light terminal scheme today. Fixed with theme.mode (auto/dark/light) and
a light palette where every text slot clears WCAG AA. Default still returns
the dark set, so no existing config changes appearance.

The interface froze for the whole of every load

Opening a 312 MB CSV with slow schema inference froze datui for 28 seconds: no
repaint, throbber stuck on one frame, every key ignored. Two causes.

The event loop drained deferred work without drawing. The codebase is full
of a "return a follow-up event so the UI can draw this phase first" idiom —
every Do* event, and the comment above DoLoadSchema says so outright. It
never worked, because the loop drained the whole chain in one pass. Breaking
after a handler returns a follow-up fixes the idiom everywhere at once: phase
labels render, throbbers animate, keys are polled between phases.

Scanning ran on the event thread. build_lazyframe_from_paths only ever
touched self.app_config, so it becomes a static function and both CSV entry
points funnel through one spawn_scan onto the blocking pool, gated on
task_generation like the existing schema and row-count results.

Measured after: throbber animates, Ctrl+O reaches the home screen in about a
second instead of 28. Verified that an abandoned 28-second scan does not clobber
a dataset opened in the meantime.

Abandoning is not cancelling. Polars cannot stop a scan once started, so the
work runs to completion and its result is discarded. Documented plainly, since
"it keeps using CPU until it finishes" is a real consequence.

Also

./scripts/dev/setup-test-data.sh — the statistics, distribution and
pivot/melt tests read generated fixtures needing Polars, NumPy, pyarrow,
fastavro and openpyxl. A fresh checkout failed 14 tests with a bare "Sample
data generation failed" and no indication of the fix. The script does what CI
does; the harness now prefers .venv/bin/python, so no activation is needed.
Removed from tkinter import W from the sample-data generator — an unused
stray import that made it require tk installed.
The home screen puts every plain character into the filter, so q types a
q. The control bar advertised "q Quit" anyway, and key routing meant Ctrl+C
did not work either — the home screen had no exit at all. Ctrl+C now quits and
Esc escalates: clear filter → leave directory → back to your data → quit.

Verification

105 new and updated tests: 65 config, 34 home screen, 6 desktop entry
The template-drift and desktop-entry tests were checked to actually fail when
the thing they guard is broken
CIEDE2000 implementation validated against the Sharma et al. reference pairs;
light palette contrast computed against white, catppuccin-latte and
flexoki-light
Exercised end to end on Omarchy 4.0.2-1 across light and dark themes, and
under LANG=C for the ASCII fallback
Full workspace suite passes, clippy and fmt clean

🤖 Generated with Claude Code

https://claude.ai/code/session_01KYovFFSm6WaquKvGxo5PUd

datui's ~44 colour slots were only reachable from the user's own config.toml. Systems that generate app themes (Omarchy, chezmoi, home-manager, a dotfiles repo) had nowhere to put a generated file, because they render into their own state directory rather than into an app's config directory. Add a top-level `import` key naming files to merge before the importing file's own values, matching Alacritty's `general.import` semantics: import = ["~/.local/state/omarchy/current/theme/datui.toml"] Precedence, lowest first: defaults -> each import in order -> the importing file -> CLI flags. So an imported theme restyles datui while the user's explicit settings still win. Deliberately generic: no desktop detection and no hardcoded state path. The one system-specific line lives in the user's own config, which is also how Omarchy themes Alacritty and btop. Details: - Paths may be absolute, relative to the importing file, or use ~ and $VAR. - Imports nest; chains are capped at 8 files and cycles are an error. - A missing import is skipped with a warning — a generated theme file may not exist yet, and datui must still start. - An import that exists but cannot be read or parsed is fatal, so a broken theme does not just look like one that failed to apply. - `merge` deliberately ignores `import`; it is a load-time directive. `AppConfig::load_from_file` is the new testable seam; `load` delegates to it. Documents the known merge limitation: a colour set to the same string as datui's default is indistinguishable from an unset one, so it cannot override an imported value. Acceptable while `import` is opt-in; it would need a presence-based merge before any auto-applied theming. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KYovFFSm6WaquKvGxo5PUd

Seven of datui's default colour slots resolve to fixed xterm-256 indices: controls_bg, table_header_bg, alternate_row_color, sidebar_border -> indexed(235) #262626, near-black on every terminal text_secondary -> indexed(240), keybind_labels -> indexed(252) secondary_chart_series_color -> indexed(245) Unlike the ANSI names datui uses elsewhere, these do not adapt: indexed(235) is an absolute point in the 256-colour cube. On a light terminal that puts header text on a near-black bar in an otherwise white UI, and turns the alternating row stripe into a solid dark band. Measured contrast of header text on the header fill, before -> after: pure white 1.39:1 -> 15.02:1 catppuccin-latte 1.90:1 -> 15.02:1 flexoki-light 1.26:1 -> 15.02:1 (1.0:1 is invisible.) The row stripe correspondingly goes from ~14:1 — it was shouting — to ~1.2:1, which is what a stripe should be. There is no ANSI colour meaning "slightly off from the background", so these slots cannot be made adaptive; the fix is to pick the right fixed set. Adds: - `ThemeMode` (`auto` | `dark` | `light`) at `theme.mode`, and `ColorConfig::light()` alongside the existing set, now `ColorConfig::dark()`. - `auto` reads COLORFGBG and falls back to dark. Alacritty, Kitty and Ghostty do not set it, so light users there set `mode = "light"` explicitly; this is documented, and an imported theme can declare `mode` itself. - Every light text slot clears WCAG AA (>= 4.5:1) on white and on catppuccin-latte; hues illegible on white (plain cyan, plain yellow) are replaced with darker equivalents. `Default for ColorConfig` still returns the dark set, so no existing config changes appearance. Loading is restructured to collect the import chain into layers before merging, because which base palette the layers merge onto depends on a mode declared within those same layers. `theme.mode` is `Option`, so "absent" and "explicitly auto" stay distinguishable — presence-based merging, unlike the colour slots. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KYovFFSm6WaquKvGxo5PUd

Adds `contrib/omarchy/datui.toml.tpl`, mapping all 43 of datui's colour slots onto an Omarchy palette, plus a user guide page covering the whole workflow. The template lives in the repo rather than only in the docs so it cannot drift: `test_omarchy_template_covers_every_color_slot` fails if a slot is added to `ColorConfig` without being added to the template, and if the template names a slot that does not exist. That drift is invisible in review otherwise — a missing slot just silently keeps datui's default in the generated theme. Derived shades use Omarchy's `{{ mix background foreground N% }}` rather than fixed colours, so they track each theme's own contrast direction and stay correct on light themes as well as dark. The template also sets `theme.mode` from the theme's own polarity, so a light Omarchy theme gets datui's light chrome automatically, including for any slot a future release adds. The docs page covers setup, the two override routes, and two traps found while testing this against all 22 stock themes: - A colour set to the same string as datui's default cannot override an import (the existing merge limitation), so overrides need an explicit form. - A per-theme file named `datui.toml` *replaces* the generated file rather than layering onto it, silently dropping every slot it does not restate. Named `datui.override.toml` and imported after the generated file, it merges per-key instead. It also sets expectations honestly: the chrome follows the theme closely, but column-type colours will look much as they did before, because datui's defaults for those are ANSI names that already resolved through the terminal's themed palette. On near-monochrome themes several column types look alike; that is the theme's palette, and the per-theme override is the way out. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KYovFFSm6WaquKvGxo5PUd

`datui` with no arguments now opens a home screen listing datasets you can open, with what each one contains shown before you open it. Ctrl+O returns there from anywhere, including mid-load. ## Why datui could only answer "show me this file", which required you to have already answered "where is it, exactly". That resolution step is the real friction: hive datasets are directories, so tab-completion walks down into `year=2024/month=01/` rather than stopping at the dataset; you do not remember whether it was `sales_2024.parquet` or `2024_sales.parquet`; and `ls` cannot tell you whether a Parquet file holds three rows or thirty million. ## Roots: how datui learns where the data is Code lives in your working directory; the interesting datasets usually do not. They are on a mount, a NAS, a scratch volume. So the listing is built from three sources: - `[data] directories` from config — PATH-shaped, a list of places - the working directory — for local exports and fixtures - **directories of recently opened datasets** — opening something on a mount once is enough for datui to know about that mount from then on The third is what bridges "code here, data there" with no configuration. ## Not a catalogue datui persists exactly one thing: a list of recently opened paths, in the existing history store. Every listing is computed from the filesystem when drawn and forgotten at exit. Nothing to register, nothing to curate, no metadata store to go stale. ## Datasets, not files A hive tree collapses to one row rather than a directory to walk; a homogeneous directory of Parquet becomes one multi-file dataset; a source tree with two stray CSVs stays a directory. Classification reads at most 8 entries, so a dataset with five thousand partitions costs what one with three costs. Row and column counts come from Parquet footers, so they are free, and are summed across at most 64 files for hive and multi-file datasets — past that the count is left blank rather than reported as a partial total. A CSV's row count cannot be known without scanning it, so it stays blank. The preview pane shows the full schema, each type in the colour the table will use. ## Design datui had no visual identity, so this establishes one: no boxes, structure from alignment and negative space, one accent spent on the wordmark, section headers, the selection marker and the prompt, and metadata right-aligned into columns. The typed schema preview is the signature — no file picker tells you the shape of your data before you open it. ## Plain terminals Every symbol resolves through `glyphs`, which falls back to ASCII when the locale is not UTF-8, overridable with `[display] unicode`. datui's interface uses no Nerd Font characters anywhere, so no patched font is needed. ## Omarchy `contrib/omarchy/omarchy-menu.jsonc` adds a Data entry with two rows: "Explore" focuses a running datui, "New window" always starts a fresh one for comparing two datasets side by side. Rows naming a specific dataset should open a new window for the same reason. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KYovFFSm6WaquKvGxo5PUd

A fresh install has no recents of its own, so the home screen has nowhere to point you. datui now reads `recently-used.xbel` — the freedesktop list that file managers and GTK applications write — and offers the directories it mentions. Only the directories. Never the file names. That distinction is the whole design, and it is not hypothetical. The recently-used list holds whatever you last opened anywhere on the machine, and it is regularly something you would not want on a screen you are demoing — a bank export, a password vault dump. Every one of those is a perfectly valid CSV that passes any mime-type filter cleanly. So these places are listed unexpanded under ELSEWHERE, ranked below every other kind of root, and nothing inside one is shown until you press Enter on it. Descending is the explicit ask; until then datui names the place and stops. `test_desktop_places_are_listed_but_never_expanded` pins that guarantee, and its companion checks contents do appear once you descend. - Ranked last: useful only before datui has recents of its own, never outranking one. - Deduplicated against configured, recent-derived and cwd roots, so a place that is already expanded elsewhere does not also appear as a stub. - Filtered to files datui can open that still exist, so the list is short and live. - `[data] use_desktop_recents = false` turns it off entirely. XBEL is scanned for `href="file://…"` rather than parsed as XML: the attribute is all that is needed and it avoids an XML dependency for one file. Percent escapes are decoded. datui reads this file and nothing else about your desktop, never writes to it, and never sends anything anywhere. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KYovFFSm6WaquKvGxo5PUd

Two problems reported from actually using this. ## `q` did not quit, and nothing else did either The home screen routes every plain character into the filter, which is correct — `q` has to type a `q` or you could never search for "quarterly". But the control bar still advertised "q Quit", and `home_key` intercepted keys before the normal quit handling, so Ctrl+C did not work either. The home screen had no exit at all except killing the terminal. - Ctrl+C (and Ctrl+Q) now quit, checked first so nothing can swallow them. - Esc escalates instead of doing one fixed thing: clear the filter, then leave a directory, then return to your data, and quit once there is nothing left to back out of. That keeps Esc as the "get me out of this" key whatever "this" currently is. - The control bar labels Esc with what it will actually do next, and advertises `^C` rather than `q`. `home_control_keys` is now a pure function so the invariant is testable: four tests cover all sixteen state combinations, asserting no bare `q` is ever advertised and a way out is always offered. Backspace-to-ascend now shares `home_ascend` with Esc, which also fixes it refusing to leave a directory that classified as a dataset. ## Sample data needed a documented ritual nobody would find The statistics, distribution-detection and pivot/melt tests read generated fixtures. The generator needs Polars, NumPy, pyarrow, fastavro and openpyxl, which a system Python does not have — so a fresh checkout failed 14 tests with a bare "Sample data generation failed" and no indication of the fix. I had been writing these off as a pre-existing environment quirk for most of a day; they were a missing setup step. - `scripts/dev/setup-test-data.sh` does what CI does: create `.venv`, install `scripts/requirements.txt`, generate the fixtures. Uses uv when available, falls back to `python -m venv`. `--force` regenerates from scratch. - The test harness now prefers `.venv/bin/python` over a system Python, so the virtualenv does not have to be activated, and its failure message names the script instead of leaving you to guess. - Removed `from tkinter import W` from the generator — an unused stray import that made the whole thing require tk to be installed. The full workspace suite now passes from a clean checkout. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KYovFFSm6WaquKvGxo5PUd

Opening a 312 MB CSV with slow schema inference froze datui for 28 seconds: no repaint, throbber stuck on one frame, every key ignored. A Ctrl+O pressed at t=2s sat in the terminal buffer and fired after the load it was meant to skip. Measured, not inferred — my earlier claim that Ctrl+O escaped mid-load was wrong. Two independent causes. ## The event loop drained deferred work without drawing The codebase is full of a "return a follow-up event so the UI can draw this phase first" idiom — every `Do*` event, and the comment above DoLoadSchema says so outright. It never worked: the main loop drained the whole chain in one pass, so the follow-up ran before any frame was rendered. Breaking the drain after a handler returns a follow-up fixes the idiom everywhere at once, not just for loads: phase labels now render, throbbers animate, and keys are polled between phases. Event order is unchanged, since the follow-up was appended to the queue either way. ## Scanning ran on the event thread `build_lazyframe_from_paths` does CSV schema inference and hive directory walks inline. It only ever touched `self.app_config`, so it becomes a static function taking the cloud config, and both CSV entry points now funnel through one `spawn_scan` helper onto the blocking pool. The result arrives as `BackgroundLazyFrameReady` and is gated on `task_generation`, matching the pattern already used for schema and row-count results. Measured after: throbber animates, and Ctrl+O reaches the home screen in about a second instead of 28. ## Abandoning is not cancelling Polars cannot stop a scan once started, so leaving a load abandons it: the work runs to completion and its result is discarded. Verified that an abandoned 28-second scan does not clobber a dataset opened in the meantime. Documented plainly, since "it keeps using CPU until it finishes" is a real consequence users should not have to discover. ## Also The key drain that runs after a busy operation discarded everything typed during it. Reasonable for a held arrow key; wrong for Ctrl+C, Ctrl+Q or Ctrl+O, which are exactly the keys someone presses when they want out. Those three now survive the drain. The two S3/GCS integration tests asserted the scan returned its outcome synchronously. That behaviour is intentionally gone; they now wait on the channel. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KYovFFSm6WaquKvGxo5PUd

datui shipped no `.desktop` file, so it was invisible to every launcher on every desktop. The Omarchy menu entry added earlier was working around that gap in the wrong place. ## The desktop entry `scripts/packaging/datui.desktop`, installed to `/usr/share/applications/` by all three packagers (deb, rpm, aur). `Terminal=true` because datui needs a real TTY; `Categories=Utility;Development;ConsoleOnly;`. `Exec=datui %F` so a bare launch opens the home screen and a file manager can pass a selection through as "Open with datui". `MimeType` lists the formats datui reads, which makes it *available* as a handler without making it the default — that stays the user's choice in mimeapps.list. Verified by installing it: it registers correctly and resolves as a handler for Parquet, CSV, JSON and Arrow. `desktop-file-validate` passes (one hint, that two main categories may list datui in two menu sections — accurate, and fine for a tool that is both). ## Why the menu entry goes Omarchy's menu reads exactly two files — its own default and one user file at `~/.config/omarchy/extensions/omarchy-menu.jsonc`. There is no drop-in directory, so no package can contribute an entry; it can only be a snippet the user pastes. That alone made it a poor mechanism. Three things finish it: - The menu's Apps provider is backed by the freedesktop database, so the desktop entry above already puts datui there. - `Menu.qml` concatenates a desktop entry's `Keywords` into the searchable aliases, so `data`, `parquet` and `dataframe` already find datui. The snippet's `aliases` were pure duplication. - The root menu is Omarchy's own information architecture — theme, capture, hardware, system. An application claiming a slot beside those is presumptuous, and it does not compose: every app wanting one would edit the same file. What is lost is the focus-an-existing-instance behaviour, which a desktop entry cannot express. That was worth something before the home screen; now that datui is somewhere you stay and Ctrl+O reaches any dataset from inside it, one window you switch to normally covers it. The remaining Omarchy-specific artifact is the theme template, which has no freedesktop equivalent and so genuinely needs one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KYovFFSm6WaquKvGxo5PUd

Four things, found by opening a real dataset directory rather than a synthetic one. ## Recents only ever saw home-screen opens `push_recent` was called from `home_open_path` alone, so a dataset named on the command line — most of them — was never recorded. Opening a hive directory and then finding it absent from RECENT is what surfaced it. Recording moves to the `Open` handler, which every open funnels through, skipping remote URLs since `push_recent` canonicalises. ## Which then made the home screen take 20 seconds to draw `rebuild` enriched every row of every root eagerly. With a real directory in recents — 14 datasets, several hundred Parquet files — that is hundreds of footer reads before the first frame. The screen simply did not appear. The note this feature was designed from says discovery must be lazy and bounded; the code was neither. Now: - Rows are measured only when they are about to be drawn, six per frame, so the first paint is immediate and counts fill in over the next few frames. - Results are cached for the session, so scrolling back costs nothing. - A folded section's rows are not drawn, so they are not read. An idle home screen has nothing else asking it to redraw, so the main loop now draws again while measurements remain — the first paint was otherwise final. Four tests pin this: rebuild reads nothing, a pass spends only its budget and reports more work, a short window measures a short list, and a folded section is not measured. ## The layout was wasting a third of the screen On a 20-row terminal it showed 7 datasets; it now shows 17. Blank spacer rows are gone, the outer margin drops from two columns to one, and the wordmark becomes a filled title bar mirroring the control bar so the list sits between two anchors. Section headers are filled and the selected row reverses across its full width — both borrowed from the table view, so the home screen reads as part of datui rather than a different program. ## Reported, and fixed - Sections stay grouped while filtering. Ranking across sections read better as a hit list but lost the one thing grouping is for: seeing where a dataset lives. A name alone rarely says that. - Sections fold with left/right, showing a count of what they hide, and the state survives a rebuild (keyed by title, since rebuilding renumbers). - Roots on a network filesystem are marked `network`, read from /proc/self/mountinfo. A hint, never a gate. - A dataset too large to count now shows `? x 39` rather than a blank: its width is known even when its length is not. - The preview pane keeps a long path to one line instead of wrapping it over three. - The cursor starts on a dataset rather than a section header, so the preview has something to show without a keypress. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KYovFFSm6WaquKvGxo5PUd

…ction Three faults, all found by opening a dataset on a real NFS share. ## History files were corrupted by concurrent writes The recents file on this machine held lines like .../chart_integration_test/home/derek/.../null_values_per_col_test.csv — two paths concatenated — and one truncated mid-path. `save_history_file` truncated in place and wrote line by line, so a reader could see a half-written file and two writers interleaved into one. Writing to a sibling and renaming over the target fixes corruption: a rename is atomic, so a reader sees the old file or the new one. That is not enough on its own. Both instances still read `[x, y]`, one writes `[a, x, y]` and the other `[b, x, y]`, and whichever lands second wins outright. Opening two datasets at once is ordinary — a launcher, a file manager, two terminals — so `update_history_file` now holds an exclusive lock across the whole read-modify-write, using the fs2 dependency already present. The lock policy took two attempts. Giving up after a few quick tries turned out to drop updates under exactly the contention it was meant to survive: the race test failed one run in five. It now waits against a 250 ms deadline, which is orders of magnitude more than a fifty-line rewrite needs and exists only to bound the wait if a peer wedges. The test was verified to fail without the lock, and passes eight consecutive runs with it. Query and SQL history use the same store, so both are fixed too. ## The test suite wrote to the developer's real cache Recording every open meant `cargo test` pushed its fixtures into ~/.cache/datui/recents_history.txt — and several tests running at once are what corrupted it. `CacheManager::new` now honours `DATUI_CACHE_DIR`, and `common::test_runtime` points it at a scratch directory, so every test that builds an App is isolated without having to remember. Verified by checksumming the real file across a full run. ## Network detection missed the case it was written for `/mnt/gilead/data` is an NFS share automounted at that path, so mountinfo holds two entries for it: autofs, then nfs4. Taking the first match at the deepest mount point reported the automount and missed the network entirely — the marker returned false for the very share that prompted it. Later entries shadow earlier ones at the same mount point, so the last match now wins. autofs joins the network list: an automount that has not triggered blocks on first access, which is what the marker warns about. The mount-table logic is split from reading /proc so the shadowing case can be tested against a fixture. Also: a derived root that cannot be *read* is now shown as unavailable rather than dropped. A share that has stopped answering is precisely what the heading exists to report. Note for later: `TemplateManager::save_template` takes its lock *after* opening with truncate, so a concurrent reader can still see an empty template file. Same class of bug, left alone here. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KYovFFSm6WaquKvGxo5PUd

An unreachable NFS share does not fail, it blocks. On a `soft` mount that is seconds per call; on a `hard` mount, which is the default, it is indefinite and uninterruptible, so datui could not even be killed. The home screen made seventeen filesystem calls on the interface thread, and any one of them on a share that had gone away would freeze it before the first frame. The whole fix rests on one fact: `is_network_path` reads only /proc/self/mountinfo, and a URL is recognised from its scheme. Both answer whether a path is remote *without touching it*. So the listing can be built entirely from local knowledge, and the reading happens somewhere allowed to block. - `roots()` no longer canonicalises or lists a remote path. It is taken at face value and probed later. - `rebuild` never scans a remote root. The section appears immediately, marked `network · checking`, and fills in when its probe returns. - Recents are not stat'ed when remote; `exists()` and `metadata()` both block. - Rows on a remote path are never measured, and the schema preview will not open a remote file. Both read the file. - The probe runs on the blocking pool and is abandoned, never joined: a thread stuck on a `hard` mount is unreclaimable, so a root that fails once is marked unreachable rather than retried into another wedged thread. datui now starts at the same speed whether the network is present or not. ## Object-store URLs belong in recents They were excluded, on the reasoning that canonicalising a URL is meaningless. But `s3://bucket/warehouse/events/year=2024` is the path least worth retyping and most worth remembering. URLs are now recorded verbatim and treated as remote throughout, so they are listed without anything reaching for them. ## A dataset now reads the same wherever it is listed Reported: a hive directory opened from the command line showed as `hive` under its own root but `dir` under Recent. The Recent row was guessed from the name to avoid reading a remote path, and "directory" was the wrong guess. A remote path that cannot be classified from its name is now `EntryKind::Unknown` — offered as openable, and left unlabelled rather than described wrongly. Once the containing root's probe lands, its classification is authoritative and the Recent row adopts it, so both rows agree. Five tests cover the never-touch rule, using an injected network predicate so the property can be checked without a remote. The central one was verified to fail when the guard is removed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KYovFFSm6WaquKvGxo5PUd

Two findings, one of which invalidates the approach taken so far. ## A named pipe froze it, locally, with no network involved ``` scan_dir -> 2 rows in 43µs <- safe enrich(data.parquet) kind=File ... <- never returns ``` A FIFO called `data.parquet` in the working directory hung the home screen permanently: opening a named pipe blocks until a writer appears. A directory listing reports it like any other file. So the previous defence — recognise network paths and avoid them — cannot hold. The ways a filesystem call can block are open-ended: pipes, device nodes, sockets, a FUSE mount nobody classified, a disk that has stopped answering. Enumerating them is a losing game. Two changes, one cheap and one structural. **Read only regular files.** Every open is now gated on `file_type().is_file()`, and a listing no longer offers anything else as a dataset. `stat` does not block the way `open` does, so this costs nothing and removes the whole class. **Nothing reads on the interface thread.** `build_listing` is a free function taking a request, so it runs on a worker; measuring runs on a worker; the schema preview runs on a worker. The interface thread decides *what* is worth reading — it knows what is on screen — and never does the reading. The guarantee is now structural rather than a list of guards that has to stay complete. The previous listing stays up while a new one is built, so a refresh never blanks the screen, and the cursor stays on the same dataset across one. `enrich_visible` is gone, replaced by `unmeasured_visible` plus a worker. `measure_now` remains for tests and library callers, documented as something the application never calls. ## Stale results wedged the pipeline A superseded listing or measurement returned early *while still marked in flight*, so no further request was ever made and the screen stopped updating for good. The flags are now cleared before the generation is checked. ## The preview was eating the scent At 100 columns — an ordinary terminal width — the preview pane left the list 53 columns, below the threshold for the metadata columns, so every row lost its size and shape. The scent is what the list is *for*, so the preview now yields first: it appears only when the list can keep its columns as well. Verified at 80, 100 and 120 columns. Tests cover the FIFO, a symlink cycle, that a listing can be built on another thread entirely, and that a refresh does not move the cursor. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KYovFFSm6WaquKvGxo5PUd

The home screen could tell you what files exist. It could not answer the question a data person actually has: *which of these has a `customer_id`?* It turns out that answer was already being read and thrown away. The Parquet footer datui opens for a row count carries the column names alongside it, so searching by column costs nothing beyond keeping them. Typing into the filter now matches dataset names and column names both. A column match is shown against the row, so a result never looks like the filter having gone wrong, and name matches always outrank column matches so typing a dataset's name still finds the dataset. ``` › ordered_at ▾ RECENT 2 ▸ sales ·ordered_at 2.4M × 5 6.0 MB 16h exports ·ordered_at 8k × 5 37.2 KB 16h ``` ## Remembered between runs Measurements are written to `datasets.json` in the cache directory, so a cold start has row counts, column counts and column names before it has read anything. Measured: a column search returns results one second after launch, against datasets nothing has opened this run. This is a **cache, not a catalogue**, and the distinction is enforced rather than asserted. Every entry carries the size and modification time it was taken from, and is ignored unless both still match — a changed dataset invalidates itself. `scan_dir` already stat'ed every row, so verifying that costs nothing. The index is capped, and a missing or corrupt one reads as no knowledge rather than an error. Nothing in it was curated by a user; deleting it costs speed and nothing else. Writes are locked and atomic, like the history files, so two instances measuring at once neither tear the file nor lose each other's work. ## Also in this pass - `Rows: 0` on the home screen was the table's counter, which on a screen showing no table is at best meaningless. It now reads `76 datasets`. - The preview pane sat empty for any format whose schema needs a scan. It now leads with what is actually known — kind, rows, columns, size, modified — and the schema below it when there is one. - Section headers show their count whether folded or not. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KYovFFSm6WaquKvGxo5PUd

Three faults, all in the part that has to work when the network does not. ## Remote datasets were blank on every run The cache verifies a fingerprint before trusting an entry — size and modification time must still match. Checking that means a `stat`, which is the call that blocks on a share that has gone away, so a remote row could never be verified and its remembered facts were never used. The datasets hardest to reach were the only ones that stayed empty. A remote row now trusts what was recorded. The entry was written from a real read, this is a cache of what a dataset looked like, and a stale row count is a far better answer than none for something on a mount that may not answer today. Local rows are still verified, and re-measured when either half changes. The kind is recorded rather than re-derived, for the same reason: a remote path cannot be classified without reading it, and guessing is how the same dataset came to read `hive` in one section and something else in another. ## Nothing was ever recorded for them either `facts_for` is called from the measurement pass, and remote rows are excluded from that on purpose — they are measured by the probe that lists their root. The probe was not recording. So even a successful probe taught the next run nothing. ## The probe never ran at all `spawn_home_probes` picks its roots out of the sections, and it was called when a listing was *requested* rather than when it arrived — so on a cold start it read an empty listing and found nothing to do. Moved to the completion handler. Measured after all three: a cold start shows `prices 17.4M × 39 1.6 GB` for a dataset on NFS one second in, having touched nothing, and `vwap` finds `returns ·fwd_vwap_ret_1d` and `prices ·vwap` immediately. ## Probes no longer take from the pool that loads data A probe of an unreachable `hard` mount holds its thread until the process exits. It now runs on a detached OS thread rather than the runtime's blocking pool, which is shared with the work that actually opens datasets — a few dead shares must not eat the capacity that loading depends on. Concurrent probes are capped, since the number of threads that never come back has to be bounded. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KYovFFSm6WaquKvGxo5PUd

## Picking a dataset that will not open was a dead end Opening a corrupt file showed "Failed to load", and dismissing that left an empty table screen with no data and no indication of what to do. Ctrl+O recovered, if you knew. For a screen whose whole purpose is choosing between datasets, a bad choice has to be recoverable. Dismissing the error now returns to the list when there is nothing to fall back to, carrying the reason: ``` › corrupt▏ …parquet: 'parquet scan': the file must end with PAR1 ``` The status keeps the tail of the message rather than the head: these read "Failed to load <long path>: <reason>", and the reason is the part worth the space — the name is already on the row. ## Sorting Tab cycles natural, size, modified, rows, with the current mode in the control bar. Natural is recency under Recent and name under a directory, which is what each section is ordered by anyway. Rows with nothing to sort by go last rather than counting as zero, so "largest first" does not open with a page of datasets whose size has not been read yet. Tab because every plain key goes into the filter — the same reason `q` cannot quit here. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KYovFFSm6WaquKvGxo5PUd

derekwisong and others added 3 commits September 7, 2026 10:25

derekwisong changed the title ~~Add config import so external theme systems can style datui~~ Theming: config import, light-terminal chrome fix, and an Omarchy template Sep 7, 2026

derekwisong and others added 5 commits September 7, 2026 17:33

derekwisong changed the title ~~Theming: config import, light-terminal chrome fix, and an Omarchy template~~ Make datui feel native: a home screen, system theming, and a desktop entry Sep 8, 2026

derekwisong and others added 7 commits September 7, 2026 21:41

Navigation Menu

Make datui feel native: a home screen, system theming, and a desktop entry - #56

Make datui feel native: a home screen, system theming, and a desktop entry#56
derekwisong wants to merge 15 commits into
mainfrom
feature/omarchy-theme

derekwisong commented Sep 7, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

derekwisong commented Sep 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

1. A home screen — datui as a place

2. System theming

3. A desktop entry

Two bugs found on the way

Light terminals were unreadable

The interface froze for the whole of every load

Also

Verification

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

derekwisong commented Sep 7, 2026 •

edited

Loading