Linux support: AT-SPI2 engine + shared session/MCP layers - #5
Open
NORTHTEKDevs wants to merge 17 commits into
Open
Linux support: AT-SPI2 engine + shared session/MCP layers#5NORTHTEKDevs wants to merge 17 commits into
NORTHTEKDevs wants to merge 17 commits into
Conversation
Adds `ghost-linux`, the Linux counterpart to `ghost-core`. It mirrors that crate's module tree and signatures so the shared session and MCP layers can switch engines with a single cfg alias. The architecture is inverted relative to Windows. There, background dispatch (driving an app without stealing focus or moving the cursor) is built on posted window messages. Linux has a cleaner analogue in AT-SPI2 actions: DoAction, SetTextContents, SetCurrentValue and GrabFocus make the application act through its own toolkit, with no synthetic input, no pointer movement and no consent prompt - and identically under X11 and Wayland. So accessibility actions are the primary path here and synthetic input is only the fallback for elements that expose no usable action. tree/actions atspi 0.30 over zbus async bridge dedicated thread + current-thread runtime, double-bounded input X11 x11rb XTEST, keysym->keycode from the live server keymap input Wayland RemoteDesktop portal, persisted+rotated restore token input headless uinput via evdev (GHOST_INPUT=uinput) capture X11 GetImage capture Wayland Screenshot portal (deliberately not ScreenCast/PipeWire) Everything is pure Rust, so no -devel packages are needed to build and the crate cross-compiles. Choosing the Screenshot portal over ScreenCast avoids linking libpipewire for a frame stream Ghost never uses. AT-SPI roles are mapped onto the same numeric id space the Windows engine uses, so role_id_to_name, is_editable_role and the load-bearing edit->document alias are identical across platforms and cannot drift. Capabilities that genuinely do not exist are reported as Unsupported rather than faked: per-window capture on native Wayland, root capture under XWayland, local OCR, and absolute pointer coordinates on uinput. Platform-independent modules (roles, handles, keysyms, verification) are not cfg-gated, so their parity tests run on any host - including Windows CI, which is where the role-table drift would otherwise go unnoticed.
Prepares ghost-core to sit behind a shared engine seam. Window handles become `isize` across the public API (WindowInfo.hwnd, find_by_name_in_hwnd, find_by_role_in_hwnd, find_all_in_hwnd, BackgroundClicker::click, ensure_foreground). `0` keeps meaning "none", matching the existing HWND(0) convention. HWND conversion moves inside the functions. This is what lets the Linux engine offer the same signatures, where a handle is an interned AT-SPI (bus name, object path) rather than a window handle. Adds system::window with the three desktop reads the session layer needs and cannot get from an element: foreground_window, cursor_pos, window_rect (plus foreground_window_rect). The session layer used raw Win32 calls inline for these; routing them through the engine is what removes its last Windows dependency. Gates the crate to Windows with #![cfg(windows)] so it stays a plain workspace member: off Windows it compiles to nothing and ghost-linux takes its place. Its deps and the capture latency probe are gated the same way. No behaviour change on Windows; the workspace suite stays green.
ghost-session and ghost-mcp become shared code. The locator tiers, grounding cascade, reflection buffer, act-then-verify loop and the whole 20-verb MCP surface are now written once and run on Windows and Linux; only the engine underneath changes. `ghost_session::engine` is the seam - a cfg alias to ghost-core on Windows and ghost-linux on Linux. Both expose the same module tree (uia, input, capture, system, process, ocr, error) and the same signatures, so call sites are identical. ghost-mcp, ghost-cli and ghost-http get the same alias. Removes the last Windows-specific code from the session layer: foreground window, cursor position and window rect now go through engine::system, and the HWND round trips are gone. Adds cursor_unchanged, which treats an unreadable pointer position as unknown rather than moved. Wayland exposes no pointer position to a client, and reporting "moved" there would be wrong: background dispatch on Linux is an AT-SPI action, which has no pointer involvement at all. Drops dead ghost-core dependencies from ghost-cache and ghost-intent (neither referenced it), and target-gates the rest. All four binaries now build for x86_64-unknown-linux-gnu.
ghost_shell works on Linux. bash is the default (sh and zsh are accepted, pwsh too if installed), and persistent sessions behave exactly as on Windows - variables, cwd and env survive across send calls - using the same base64 framing and per-session nonce sentinel, so command text never has to survive shell quoting and a late sentinel from a timed-out command cannot be mistaken for a later one. ghost doctor gains real Linux checks, each mapping to a specific failure the setup guide describes: session type, AT-SPI bus reachability, whether any application actually exposes a tree, which input backend was selected, and a capture probe. The accessibility check is the important one. A bus that answers while zero applications publish a tree is the signature of accessibility being switched off, which is the most common Linux setup failure by a wide margin, so it gets a specific remedy rather than a generic "no windows found".
Adds docs/linux-fedora.md: install, enable accessibility, build, register with Claude Code, the on-device verification checklist, Wayland consent and unattended uinput setup, known limitations and troubleshooting. capabilities_for(Linux).functional stays false. The engine is implemented and every binary cross-builds and links, but nothing here has run against a live desktop, and the rule is that the flag flips only after on-device verification. The status string says exactly that. Gates the Windows live-app tests (Notepad, Calculator, CV probe) to Windows; they drive Windows applications through the Windows engine and cannot compile off it.
The repo's existing CI is Windows-only, and its release gate notes that GitHub runners have no interactive desktop. That is true on Windows - but not on Linux, where a full desktop can be synthesised in CI: Xvfb -> an X11 display dbus-run-session -> a session bus at-spi-bus-launcher -> the accessibility bus GTK publishes to So the Linux engine gets what the Windows engine cannot have: automated live verification against a real application on every push. Adds crates/ghost-linux/tests/live_atspi.rs - ignored by default, run by CI with --ignored. The tests drive a real GTK dialog (zenity) end to end: the a11y bus is reachable, the window is listed with a real pid, its entry is findable by role, its buttons by name, describe_screen returns elements with real rectangles, and capture produces a valid PNG. The one that matters is sets_and_reads_back_entry_text. It writes through AT-SPI EditableText - no keystrokes, no pointer, no focus change - then re-resolves the element and reads the value back from the application. That is Ghost's wedge, and this proves it on Linux rather than asserting it. The workflow also confirms the release binaries are ELF and smoke-tests the MCP server over stdio (initialize + tools/list) so a binary that cannot even speak its own protocol fails CI rather than the user's first session. Clippy: the new crate is held to --all-targets -D warnings; the rest of the workspace is checked at the same scope the Windows job already uses.
Cross-platform build hygiene turned up by adding a Linux CI job.
Windows-only targets now keep a stub `main` instead of vanishing: gating a bench
or example body with an inner `#![cfg(windows)]` removes its `main` too, and a
bench/example target must always produce an executable. `convert.rs` and
`notepad_hello.rs` gate their bodies and keep a no-op entry point.
Clippy is clean on both targets at the scope CI enforces:
- collect_matching, not find_all_in_hwnd, is the 9-argument function
- WindowState::from_str is inherent by design - the Linux engine mirrors the
name so shared code calls it identically; same for ReflectionBuffer::default
- the downsample averages now use checked_div, which states the empty-cell case
(average 0, never a divide by zero) instead of implying it
- a HashMap value type in the locator cache gets a name
- ghost-platform's test module moves below the item it was splitting
None of this changes behaviour; the workspace suite stays at 452 passing.
…nder bash Both fixes come from the first real Linux CI run. The MCP server refused to start when the accessibility bus was unreachable, because GhostSession::new eagerly opened the connection. On Windows that is harmless - COM and UIA are always there. On Linux an absent bus is an ordinary condition (accessibility off, headless box, service context), and dying at startup means the client shows "MCP server failed" with no explanation and no way to ask why. The bridge now starts its thread immediately and connects on first use. The server boots, answers tools/list, and the first real call returns the actionable message naming the gsettings command. A failed connection is not cached, so enabling accessibility takes effect on the next call rather than after a restart. Jobs carry a fail arm alongside the run arm, both holding a clone of the same reply channel, so a caller always receives a real error instead of a dropped sender surfacing as an opaque "worker disconnected". The live suite failed for a duller reason: dbus-run-session exec's the script directly, so with no shebang the kernel used sh, where `set -o pipefail` is illegal. Added one. CI already proves the first job of the run: build, unit tests and clippy all pass on real Linux.
… Windows Adds scripts/install.sh - the Linux counterpart to kit/install.ps1. It builds, installs to ~/.local/bin (never sudo), enables toolkit accessibility if it is off, registers the MCP server with Claude Code idempotently, and finishes by running `ghost doctor` so a broken setup is reported at install time rather than in the middle of the first session. It warns about missing runtime packages instead of failing, because the engine itself needs none of them to build. Adds .gitattributes pinning *.sh to LF. This is not cosmetic: Ghost is developed on Windows, and a shell script committed with CRLF fails on Linux with `bad interpreter: bash^M` - fatal, mystifying, and very easy to do by accident. The installer is also marked executable in the index, since `git add` from Windows does not set the mode and `./scripts/install.sh` would otherwise be permission denied. README documents Linux: the platform table, install path, requirements, and what `ghost doctor` checks on each OS.
The live CI run found this: 7 of 9 tests passed, and the 2 that failed both failed on role="edit". GTK 3 exposes `zenity --forms`' single-line entry as AT-SPI role `text`, not `entry`. Toolkits disagree here and the answer varies by version, so matching on the role name alone is not sound. The reliable discriminator is the **EditableText interface**: an object that implements it is, by definition, something an agent can type into. A generic `text` object that is editable is now normalised to `edit` -- the same name the Windows engine uses for the same thing -- and `role="edit"` falls back to the interface regardless of what the toolkit called the widget. A read-only label is also `text` but is not editable, so it is still correctly excluded. The normalisation lives in one place (ops::effective_role) and is used by matching, describe_screen, text extraction and Element::role_name, so a field found via role="edit" also reports `edit` instead of leaking the toolkit's name. describe_screen now treats anything editable as actionable, so a text field is visible to an agent whatever role its toolkit reports. Live tests gained a roles_in() diagnostic: a failure now prints the roles that were actually on screen instead of just "not found".
The stub main added for non-Windows targets discarded run()'s Result, which Windows CI turns into an error via RUSTFLAGS=-D warnings. Verified locally under the same flag: 452 passing, zero warnings.
sets_and_reads_back_entry_text covers the write path; this covers the action path. Invoking Cancel through AT-SPI must actually dismiss the dialog, observed by the window leaving the accessibility tree - a test that only asserted do_action returned Ok would pass even if the application ignored it.
…e proves The live AT-SPI suite passes 10/10 on real Linux in CI, against a real GTK application on a synthesised desktop. That is on-device verification, so capabilities_for(Linux).functional becomes true - the first time Ghost's rule has been satisfied for a platform other than Windows. `supported` lists exactly the six features that suite exercises: ElementDiscovery, Act, PerActionVerify, BackgroundDispatch, StructuredSnapshot and Screenshot. KeyInput, EditShortcuts and VisionGrounding are implemented but deliberately NOT claimed, because nothing has verified them end to end on Linux; the Wayland portal paths are likewise unclaimed, since CI runs X11. Tests assert both the presence and the absence, so a future change that quietly widens the claim fails. BackgroundDispatch is the one worth stating plainly. The cross-platform doc has said since the port began that the wedge might degrade off Windows and should be treated as "unknown -> measure". It was measured: AT-SPI actions make the application act through its own toolkit, so nothing is raised and no pointer moves - a cleaner guarantee than posted window messages, not a weaker one. The docs now say that instead of hedging. macOS remains a scaffold and claims nothing.
… runs Two gaps a tester would have hit before any of the engine work mattered. **The agent-facing schema was wrong.** `ghost_shell` advertised shell=powershell|pwsh|cmd and described itself as "Run terminal / PowerShell commands", including a Windows-only recipe for starting a Claude session. An agent reads that schema to decide what to send, so on Linux every ghost_shell call it composed would have been wrong before it was sent. The description and enum are now platform-aware (bash|sh|zsh|pwsh, bash default), and `ghost_window op=state` says plainly that only `close` works on Linux instead of letting an agent burn turns on calls that always report Unsupported. **Nothing a user actually runs had ever been executed on Linux.** The engine had live tests; the product did not. Added: - `crates/ghost-mcp/tests/live_mcp_linux.rs` - spawns the real ghost-mcp binary and speaks JSON-RPC to it over stdio exactly as Claude Code does, driving a real GTK app: tools/list, ghost_window, ghost_see returning real roles, ghost_shell running a real command, and a persistent shell keeping a variable across sends. Layers can each be right while the whole is not - a wrong argument name or an unreadable response envelope only shows up from outside. - CLI smoke in CI: `ghost doctor` (exit status enforced, since it is the first command anyone runs), `ghost list-windows`, `ghost screenshot`. - An installer job: shellcheck plus actually executing `scripts/install.sh` and asserting all three binaries land, so a bad path or a lost +x bit fails here rather than on the tester's machine. - **A Fedora 41 container job** running the whole live suite on the distro the docs target, installing exactly the packages docs/linux-fedora.md lists - so those instructions are themselves under test.
Caught by the new CI jobs on both Ubuntu and Fedora: `ghost doctor` reported FAIL when zero applications exposed an accessible tree, and told the user to enable a setting that was already correct. Zero windows is ambiguous, and the two readings are indistinguishable from inside doctor: accessibility may be off so nothing publishes a tree, or nothing may be open - which is entirely normal on a fresh session. Calling that a failure means the first command anyone runs on a clean desktop declares the install broken. It now WARNs and explains both readings, and only an unreachable bus is fatal. Tests pin the policy: zero warns, one passes, and only a FAIL sets a non-zero exit code. CI now runs doctor with an application open, so the PASS path is what gets exercised rather than the empty-desktop path, and follows it with `ghost list-windows` asserting that app appears.
…ording The doctor fix worked - it now passes with a warning instead of failing - but the CLI smoke step still failed: zenity got a fixed 3-second head start while the Rust live tests poll for twenty. Under Xvfb, GTK startup varies by an order of magnitude between a warm and a cold runner, so the fixed sleep made this job flaky in a way no user would ever experience. Both live jobs now poll `ghost list-windows` for the window, and print the full window list if it never appears so a real failure is diagnosable rather than a bare non-zero exit. Also corrects the accessibility message, which still told users to enable a setting even in the ordinary case of simply having nothing open.
The shell-enum assertion failed because verb descriptions cross-reference each other, so splitting the raw JSON on "ghost_shell" landed in another tool's prose rather than the tool definition. It now finds the tool object by name and reads inputSchema.properties.shell.enum directly, and also asserts the Linux description does not tell an agent to write PowerShell. Everything else in the run was green: ghost doctor reported all-PASS with the app detected, the CLI smoke passed, and the engine live suite was 10/10.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Ghost runs on Linux.
ghost-mcp,ghostandghost-httpall build and runthere, exposing the same 20 MCP verbs as on Windows.
capabilities_for(Linux).functionalis nowtrue— the first time Ghost'son-device rule has been satisfied for a platform other than Windows.
Verification
Both platforms are green in CI:
The live suite is the point. GitHub runners have no desktop — but on Linux one
can be synthesised (
Xvfb+dbus-run-session+at-spi-bus-launcher), so theengine is tested against a real GTK application on every push. It proves both
halves of the wedge:
sets_and_reads_back_entry_text— text written throughEditableTextis readback from the application, with no keystrokes and no pointer movement.
invoking_a_button_through_atspi_has_a_real_effect—Action.DoActionactually dismisses the dialog, observed by the window leaving the tree. A test
that only asserted
Okwould pass even if the app ignored it.Plus window enumeration with real PIDs, role and name lookup,
describe_screenwith real rectangles, PNG capture, and XTEST pointer control.
Shape
One
cfgalias. Both engines expose the same module tree and signatures; windowhandles are
isizeon both (0= none) — anHWNDon Windows, an internedAT-SPI
(bus name, object path)on Linux.The wedge got better, not worse.
docs/cross-platform.mdhad long warnedthat background-without-focus-steal might degrade off Windows and should be
treated as "unknown → measure". Measured: AT-SPI actions make the application act
through its own toolkit, so nothing is raised and no pointer moves — a cleaner
guarantee than posted window messages, and identical under X11 and Wayland.
Synthetic input is only the fallback.
Pure Rust throughout (
atspi/zbus,x11rb,ashpd,evdev): no-develpackages to build, and it cross-compiles. The Wayland capture path uses the
one-shot Screenshot portal rather than ScreenCast + PipeWire, because Ghost
captures stills, not video — PipeWire would add C linkage and DMA-BUF handling
for nothing.
Bugs this caught before release
scores 0.00098 — under the old 0.002 threshold. A one-word text change would
have verified as nothing happened, defeating the reason the grid is fine.
role
text, notentry, sorole="edit"missed it. Matching now keys off theEditableTextinterface, which is the actual discriminator; a read-onlylabel is also
textand is still correctly excluded.On Linux that is an ordinary condition, and dying at startup means the client
shows "MCP server failed" with no diagnosis. The connection is now lazy: the
server starts, answers
tools/list, and the first real call returns themessage naming the fix.
install.shwould have failed on Fedora withbad interpreter: bash^M..gitattributespins*.shto LF and the file is marked executable.Not claimed
KeyInput,EditShortcutsandVisionGroundingare implemented but absent fromsupported, because nothing has verified them end to end on Linux. The Waylandpaths (RemoteDesktop portal input, Screenshot portal capture) are likewise
unverified — CI runs X11. Tests assert both the presence and the absence of these
claims, so a future change that quietly widens them fails.
Reported as
Unsupportedrather than faked: per-window capture on nativeWayland, root capture under XWayland, local OCR, absolute pointer coordinates on
uinput, and window minimize/maximize/restore (
closeworks).docs/linux-fedora.mdcarries setup, the on-device checklist for the Waylandpaths, and troubleshooting.
scripts/install.shbuilds, installs, registers theMCP server and runs
ghost doctor.