Plugin platform: real weather (GFS + HRRR), plugin-owned connections, per-plugin logs, and a full settings/UI overhaul#51
Open
beetlebugorg wants to merge 33 commits into
Open
Plugin platform: real weather (GFS + HRRR), plugin-owned connections, per-plugin logs, and a full settings/UI overhaul#51beetlebugorg wants to merge 33 commits into
beetlebugorg wants to merge 33 commits into
Conversation
…eference plugin Implements the host side of the plugin system (specs/plugin-system.md Phase 1): - internal/engine/plugin: NDJSON JSON-RPC 2.0 protocol + session, manifest + content-hash packaging, plugins.json state store, wazero (Tier A) and os/exec (Tier B) runtimes, the Broker with capability enforcement (vessel/ais/raw publish, status, host-mediated tcp.connect, storage, config), and the Manager (lifecycle, 1s→30s backoff, circuit breaker, ping liveness) mirroring nmea.Manager's shape. - nmea store extensions: Store.PublishDeltas (attributed SignalK path→field writer with provenance), AISStore.Upsert/Feeder, per-source provenance maps. - sdk/: event-driven Go plugin SDK (single-threaded, wasip1-safe — no background goroutines; async host calls resolved by the read loop; per-message batching). - plugins/core.tcp-client: the built-in tcp-client NMEA source reimplemented as the reference Tier-A WASM plugin. The built-in stays; parity is the acceptance test (parity_test.go drives the real wasm module through the broker end-to-end and asserts the host store matches the built-in nmea parse path).
- server: initPlugins wires plugin.Manager with a Host backed by the shared
vessel/AIS/raw stores; /api/plugins (list/install/enable/disable/grants/
config/remove) + /api/plugins/stream SSE; /plugins/<id>/{ui,serve}/* static
serving via serveFile (Range, .wasm/.mjs mime); CSP tightened to
connect-src 'self' so trusted plugin UI can't phone home.
- plugin: ManagerOpts.NoStart (state-only manager for CLI one-shots),
DevRun (run an unpacked dir under the broker with restart), factored
startInstance shared by runners and dev.
- cli: `chartplotter plugin install|list|enable|disable|remove|dev`.
- Makefile build-plugins target + core.tcp-client manifest.
Adds the frontend plugin host (spec §8, §11, Appendix A.3) and refactors the two built-in overlays to run through it, driven ONLY by a pure declarative ctx (no raw map or plotter object): - core/plugin-host.mjs: PluginHost loads controllers (constructor(ctx)/start()/ destroy()) and assembles ctx — vessel, ais (wrapped server feed), layers, markers, camera (follow/anchor/gesture), hud/panels mounts, settings, notify, callout, units. It is the single place that touches map/plotter internals. - plugins/plugin-layers.mjs: declarative GeoJSON layer host with named z-bands, style-reload self-healing and retained data (replaces own-ship's hand-rolled _ensureLayers). - plugins/own-ship.mjs, plugins/ais-overlay.mjs: converted to ctx controllers (default exports). Behaviour preserved — pose tween, GPS watchdog, follow break-out, wheel-zoom anchor, AIS glyphs/danger. own-ship no longer exposes zoomAnchor(); it registers the anchor via ctx.camera.registerFollowAnchor. - chartplotter.mjs: constructs PluginHost and registers core.own-ship / core.ais instead of newing the overlays; _zoomAnchor aggregates registered anchors. Verified: ESNext syntax + import-graph resolution (esbuild), modules serve with js mime, backend feeds vessel + AIS. Live browser screenshot not run (no browser in this environment).
serve() blocks reading the plugin's stdout; a context cancel didn't unblock it, so plugin dev ignored SIGINT/SIGTERM. Kill the session on cancellation so the read returns EOF and DevRun exits.
Adds a Plugins tab to the settings dialog to install and manage plugins over the /api/plugins* endpoints, modeled on the Connections tab: - data/plugins-service.mjs: client (list, install multipart upload, enable, disable, setGrants, remove, status SSE stream). - plugins/plugins-panel.mjs: <plugins-panel> element — installed list with live status badges + tier tag, install-by-upload, enable/disable, a per-plugin capability grant editor (plain-language descriptions), and remove. Status ticks patch badges in place; structural changes re-render. - plugins/plugins-manager.mjs: PluginsController registers the tab via the settings-registry render(host) escape hatch. - chartplotter.mjs: construct PluginsController alongside ConnectionsController.
…API + live status - Revoking net.tcp-client (or serial) now closes the plugin's open host-dialed sockets, so a source plugin actually stops receiving/publishing instead of continuing over a connection the host opened under a now-revoked grant. - Revoking ais.write evicts the targets that plugin contributed (AISStore.EvictSource) so stale AIS doesn't linger until TTL; new publishes are already denied by the capability check. - Manager.SetConfig updates settings while leaving grants intact (the previous config path wiped grants); hot-applied via host.grantsChanged. - Plugin status.update is now reflected in the plugins UI (a per-runner status hook), so the badge shows the plugin's own state/detail. - core.tcp-client manifest gains a ui.settings schema (host/port). - Parity-test poll widened so a loaded machine doesn't flake; EvictSource test.
- plugins-panel: a Configure editor per plugin — a schema-driven form when the manifest declares ui.settings.items (text/number/toggle/select), or a raw-JSON fallback otherwise; saved via PUT /api/plugins/<id>/config. Tighter, roomier row/action spacing and a shared editor style for grants + config. - plugins-service: setConfig(). - Settings dialog widened (set-wide 520→760px) and taller (max-height 66vh/620→ 82vh/860) so the Plugins tab (and others) aren't cramped.
Nine focused, cross-linked Docusaurus pages under docs/docs/plugins/ (overview, getting-started, manifest, capabilities, sdk, protocol, ui, packaging, examples) plus a "Plugins" sidebar category. Grounded in the implemented source, with implemented vs. roadmap features clearly marked (signing, services.*, serial I/O, net.udp/http, http.register, detachable, native sandboxing are noted as roadmap).
…g from UI - Rename core.tcp-client → core.nmea0183 (dir, manifest id, name "NMEA 0183"); the plugin's identity is the NMEA 0183 source it provides, not the TCP transport it happens to use. Description now says what it does (reads GPS/ heading/speed/depth/wind + AIS), not that it's a WASM plugin. - Update the parity test, Makefile build-plugins target, and docs references. - plugins-panel: remove the per-row wasm/native/ui tier tag (implementation detail, not useful in the list).
- serve.set / serve.clear: publish a blob at GET /plugins/<id>/serve/<name>, host-served with Range + caching. The zero-RPC "baker, not live server" path for tile archives and weather grids — a sandboxed WASM plugin (no filesystem) can now produce served artifacts. - http.fetch: host-mediated outbound HTTP, allow-listed by the net.http grant and size-capped, for live data pulls. - SDK: ServeSet/ServeClear and Fetch (event-driven callbacks).
The "grid, not tiles" model end to end: a Tier-A WASM plugin decodes a GRIB2 surface-wind field and publishes it as a compact grid artifact; the frontend animates it as wind streamlines client-side (the plugin is never in the frame path). - plugins/core.weather/grib: a spec-compliant GRIB2 codec for regular lat/lon grids with grid-point simple packing (templates 3.0/4.0/5.0), with a round-trip test. GFS complex packing (5.3) is a documented follow-up. - gen/: synthesizes a plausible mid-Atlantic wind field (background flow + a cyclonic low) and writes the embedded offline sample.grib2. - main.go: decodes the embedded sample (or a live GRIB via net.http) → the standard two-record wind JSON → serve.set. Offline by default. - ui/plugin.mjs: self-contained wind-particle overlay (canvas streamlines over the chart, coloured by speed, with a HUD show/hide toggle) using ctx.map.
- PluginHost.start() discovers installed, enabled plugins that ship a UI (manifest ui.entry), dynamically imports each one's entry module from its archive (/plugins/<id>/ui/…), and keeps the set in sync via the plugins SSE (load on enable, unload on disable). This is how third-party UI plugins — like the weather overlay — actually load; builtins still register directly. - ctx.map: the raw MapLibre instance, the use-at-your-own-risk tier for controllers that need custom rendering (the wind-particle layer). The declarative handles remain the compatibility-promised surface; own-ship/AIS still don't use ctx.map.
A LayerRegistry aggregates every map overlay with a persisted show/hide state, surfaced as a "Layers" settings tab. Distinct from enabling/disabling a plugin: hiding is visual only — the plugin keeps running and is told via an onVisible signal so it can pause expensive work (the wind animation) while staying loaded. - core/layer-registry.mjs: register/setVisible/list + localStorage persistence. - plugins/layers-panel(.mjs/-el.mjs): the Layers tab (grouped switches). - ctx.overlays.register (plugin-host) — plugins register overlays, namespaced. - plugin-layers: setVisible on the declarative handle (survives style rebuilds). - own-ship registers "Course & heading vectors"; AIS registers "AIS targets" (feed keeps running while hidden); weather registers "Wind streamlines" (replacing its one-off HUD toggle).
- The weather plugin now decodes and publishes multiple forecast hours (grouped by GRIB2 forecast time) as a multi-step wind document. - The sample generator produces a 7-step series (0–48h) in which the cyclonic low drifts north-east and intensifies, so scrubbing shows the storm evolve. - The wind overlay adds a time scrubber that linearly interpolates the u/v field between the two bracketing steps and re-labels the forecast hour; hidden with the overlay.
Adds GRIB2 data-representation templates 5.2 (complex packing) and 5.3 (complex packing + spatial differencing) — the packing GFS uses — so the weather plugin can decode real forecasts, not just the simple-packed sample. The decoder follows NCEP g2clib's comunpack: read the spatial-difference extras, the per-group reference/width/length metadata (byte-aligned between subsections — the NCEP encoder aligns them; without it the second-order integration runs away), the packed group data, then undo the differencing and apply the reference/scale. Validated in CI against a real GFS 1° UGRD/VGRD field byte-range-fetched from the NOAA open-data archive (testdata/gfs_1deg_uv.grib2): decodes to plausible wind (−26..33 m/s) at the correct 360×181 grid.
- The weather plugin now pulls live GFS: given a GFS product URL it fetches the wgrib2 .idx, finds the 10 m UGRD/VGRD records, and byte-ranges just those messages (a few MB) instead of the whole multi-hundred-MB file. SDK gains FetchOpts (request headers) for the Range request. - A global 0.25° field is ~2M points (~30 MB JSON) — over the 16 MiB NDJSON line limit and far finer than streamlines need — so the plugin downsamples oversized grids by an integer stride (regional fields pass through). 0.25° GFS lands as a 480×241 field. - The particle layer now spawns in the current viewport, so a global field is visible wherever the chart is, not scattered across the planet.
- Dynamic import() of an installed plugin's UI resolved relative to /src/core/plugin-host.mjs (→ 404 /src/core/plugins/<id>/ui/…), so no plugin UI ever loaded. Resolve the archive URL against document.baseURI instead. This is why the wind overlay showed nothing. - The wind sampler assumed −180..180 longitude; GFS grids are 0–360, so it never found wind over the US. Wrap longitude for global grids (and wrap the bilinear x-neighbour across the antimeridian).
… flicker Round of weather-overlay fixes from live testing: - Binary grid, not JSON: publish the wind field as a compact aligned Float32 blob (~8× smaller), so a full-resolution field fits the 16 MiB wire line without the aggressive downsampling JSON forced. The frontend zero-copies the Float32 arrays. - Real data, not just a visual: a live readout of the actual wind at the vessel (speed in the mariner's units + the compass direction it blows FROM, with an arrow), combined into one on-map control that is also the enable/disable toggle — kept in sync with the Layers control via the shared registry (ctx.overlays now returns a handle). - No more flicker: only clear the trail canvas on ZOOM, not on pan — the follow-camera eases the map every fix, and clearing there blanked the whole field ~1×/s (the "all at once" flicker). Re-projection tracks pans; the fade cleans smear. - Contrast: draw each streamline with a dark casing under a bright core so it reads on both light (day) and dark (night) charts.
…cycle The weather plugin now defaults to "gfs": it lists the NOAA open-data archive, picks the newest date + cycle (so it stays current with no hardcoded date), and byte-range-fetches that cycle's 10 m wind. On any failure it reports degraded and draws nothing — no fabricated/fallback data. "sample" remains available for offline/demo use, and an explicit GFS or GRIB2 URL still works.
Set CHARTPLOTTER_NO_REBAKE=1 to treat baked charts as current regardless of the engine/version stamp, so rebuilding the app during dev doesn't kick off a slow re-bake on every restart. Genuinely-missing charts still bake.
- ctx.hud.mount() returns a plain element (querySelector, not getElementById); the weather control used getElementById and threw, so the layer couldn't be turned on. Fix the call and add a scoped getElementById shim to the mount so the documented examples work and no plugin hits this again. - Wind particles were overwhelming: scale the count to the viewport (~900/Mpx, capped) instead of a fixed 2600, thinner lines, shorter/lighter trails.
…isting) The old discovery listed the S3 bucket with prefix=gfs. and took the max — but S3 returns the first 1000 keys in lexicographic (chronological) order, i.e. the OLDEST dates, so it always resolved to an ancient cycle (~2023-09) regardless of what's actually current. Instead, use the module's wall clock: start ~5h back, floor to a 6h boundary, and walk older cycles until one's .idx is available. Now fetches today's run (verified: GFS 20260715 12z). Also: 0.5° resolution for speed, slider docked under the wind control (off the bottom status chrome), and lighter/slower/thicker particles.
… grid-relative wind rotation HRRR decodes natively: projection forward-mapping, bilinear resample to regular lat/lon windows, and earth-relative wind rotation (~13° in the mid-Atlantic — skipping it visibly skews directions). pipeline_test locks the Go-encode → JS-parse byte layout (v4 plane-mask format).
…nest wind rendering - three published layers: GFS base, 3 km HRRR sailing window (auto-centred from the GPS fix or viewport), ~15 km HRRR CONUS look-around; sampled hi→mid→base everywhere - map-bearing-corrected streamlines and readout arrows (course-up was skewing every direction by the bearing) - per-forecast details (wind/gust/temp/cloud), compact scrubber card, tap-to-probe marker, ship's-clock local times - ↻ refresh via config nonce + 10-min staleness auto-refresh - real id (org.beetlebug.weather) and a single honest setting: the HRRR data-budget toggle (~60 MB/refresh) instead of a fake 'source' field
…as placeholder cruft)
Signal loss keeps the greyed last-known fix (ECDIS convention); deliberately disabling/removing a source now clears every path it wrote (provenance-keyed Store.ClearSource) and own-ship drops off instead of freezing forever.
…eveloper mode - settings: fixed-geometry dialog (no layout shift), single scroll container with bottom fade, no nested frames, 7-tab rail (Chart absorbs Display/Text/ Depths; Units into General; Vessel = calibration); Advanced is now Developer, gated by a General toggle so production builds can ship it off - connections: no standalone tab — a data-source plugin (provides: nmea.source) DEFINES a connection type; its Plugins row drills into the connections view (rows, schema-driven add/edit, sniffer, pause) - tap arbitration: ctx.taps.claim(fn) — overlays own chart taps while active (wind probe), pick report stays the fallback - HTTP/1.1 socket budget: settings panels poll with transient fetches, never pin SSE sockets (6/origin; the core streams hold three) — fixes blank panels from fetch starvation - plugin rows: full-width descriptions; loading states; normalized buttons/ radii/spacing per the new style guide (docs/style-guide)
Plugin stderr (SDK Log) lands in a 400-line ring on the manager, served at /api/plugins/<id>/logs; ctx.plugin.log's UI-half lines are captured too and merged into one timeline. The Plugins row gains Logs — a drill-down view with level/substring filters, follow-tail, and copy (logs are voluminous; an inline box was too cramped). Toggle settings rows let the label breathe.
…tings - layers get their own round button + anchored popover (they're a navigation control, not configuration) — the settings tab is gone - Advanced is now Developer, hidden by a General toggle (registry-level when() predicates) so production builds can ship it off and re-enable for debugging - viewing groups fold into Chart; screen calibration into General; rail is General · Chart · Plugins · Developer (+ per-plugin tabs)
…1.2.6) Without a GPS fix the 3 km window centres on the map view (zoom-gated, rate-limited) — high-res coverage where you're LOOKING, not just where the boat is. Cycle discovery, publishes, and refreshes now log to the plugin log ring.
…d service spec Using chartplotter (simple, plain language: connecting instruments, weather on your chart) vs Developing (architecture, style guide, plugins incl. the weather.grid contract: discovery, weather.json schema, WGRD/4 format, sampling reference, provider internals).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Weather
weather.gridservice: every layer is indexed in a servedweather.json(model, cycle, hours, fields, grid, URL) behindprovides: [{service: "weather.grid"}]; the WGRD/4 plane-mask format grows fields without breaking consumers. Spec in docs.Platform & SDK
ConfigWatcher: config hot-applies to running plugins — no restarts (drives the HRRR window follow and refresh).ctx.taps.claim(): tap arbitration — an active overlay owns chart taps; the ECDIS pick report is the fallback./api/plugins/<id>/logs, merged with UI-halfctx.plugin.loglines in a full-pane Logs viewer (level/text filters, follow, copy).org.beetlebug.*); the reserved-prefix trap that forcedorg.example.*installs is gone.Connections & data sources
provides: nmea.sourcedefines a connection type — its Plugins row drills into the connections view (status, schema-driven add/edit, pause, raw sniffer).Store.ClearSource) — no phantom own-ship; signal loss still keeps the greyed last-known fix.Settings & UI overhaul
docs/style-guide) codifying tokens, scales, layout rules, and a review checklist; all panels normalized to it.Docs
Split into Using chartplotter (plain-language: connecting instruments, weather on your chart) and Developing (architecture, style guide, plugins incl. the
weather.gridcontract).Verified throughout with a headless-chromium screenshot loop (settings tabs, drill-downs, themes) and live end-to-end checks against the running app.