Track live input health and recover GPS failover

[FrogAi]

Summary

• Record semantic event validity and consumer-local receipt time for decoded subscriptions without changing the existing Read() API.
• Continuously poll both GPS providers, prefer a usable external fix, and fall back to a cached usable internal fix when external GPS becomes invalid or stale.
• Scope output validity to the inputs each feature actually depends on: carState health gates the envelope, GPS/map/route health gates only map-derived output, and vision-curve output survives both.

Motivation

At Base 68813e05, a subscriber reports success after structural decode and union selection. It does not inspect Event.valid or retain when the message was received, so invalid or indefinitely retained car, model, and GPS data is indistinguishable from live input.

GPS selection compounds that problem. The first decoded external packet permanently latches the external subscriber; the internal queue is no longer drained, so a later external stall cannot fall back to a healthy internal source. The main loop also republishes retained state as valid before reading new inputs, which allows stale limits, hazards, curves, and paths to remain externally valid.

These are one transition contract rather than independent fixes: a normal 20 Hz poll can legitimately see no new packet between 10 Hz external GPS updates, so failover needs the same persisted validity and receiver-observed freshness state used to determine output health.

Health contract

Input/state	Usable condition	Effect when unusable
carState	Event.valid and local receipt age < 500 ms	Publish invalid/default-zero mapdOut. Every published field derives from car state, so this is the one input that invalidates the whole envelope. On recovery, skip the unknown outage interval instead of feeding it into the car timing average (gap threshold stays 100 ms).
modelV2	Event.valid and local receipt age < 500 ms	Omit the stale vision value and exclude it from speed arbitration. Reset the vision moving average on recovery so the first fresh trajectory is not blended with pre-stall history.
External GPS	Event.valid, a reported fix, and local receipt age < 1 s	Prefer internal GPS if its cached sample remains usable.
Internal GPS	Event.valid, a reported fix, and local receipt age < 10 s	No selected position source; clear map-derived state. Vision-curve and external speed limit output continue.
Offline tile	Loaded and covering the selected position	Clear map-derived state and publish an empty extended path; tileLoaded continues to report the condition.
Current road	Successfully resolved from the selected position	Clear map-derived state and publish an empty extended path.
Lookahead/curvature	Optional derived state	Clear only the affected upcoming or curve state; keep valid current-road output.
mapdIn / mapdCli	Successfully decoded command	Preserve existing behavior regardless of Event.valid; these on-demand command envelopes do not expire.

Freshness uses receiver-local time, not producer logMonoTime or GPS wall time. The boundary is strict: a sample is stale at exactly its timeout.

The carState window is 500 ms rather than a ten-period 100 ms, because the daemon's own loop can exceed 100 ms between reads when a tile-boundary crossing loads and parses a tile; a tighter window converts that routine work into spurious invalid frames. The 100 ms threshold is retained where it belongs, as the inter-message gap that rebases the car timing average.

GPS fix compatibility accepts hasFix or the legacy bit-zero fix flag. Cap'n Proto exposes no presence bit for hasFix, so an odd legacy flag remains authoritative when hasFix is false.

Source selection is debounced: external is adopted immediately when no healthy internal fix exists, but while internal is healthy external must stay healthy for 2 s before it takes over. Failover away from a stale external source is never delayed.

Behavior

Transition	Base	Head
Startup without healthy driving inputs	Publishes a valid envelope containing retained/default state	Publishes an invalid envelope with default-zero fields
Decoded invalid car/model/GPS event	Applies the payload	Records the receipt but does not apply it
Healthy external GPS between normal 10 Hz packets	External remains latched	External remains selected until its 1 s freshness window expires
External stalls while internal remains fresh	Stops receiving positions and never checks internal again	Fails over to internal and continues draining both providers
External recovers	Resumes only because it was permanently selected	Retakes priority after 2 s of continuous health
External degraded to roughly its timeout cadence	Latched; no switching	Debounce holds one source instead of switching per tick
Both GPS providers become unusable	Retains and republishes the last map state as valid	Clears map-derived state; vision-curve and external speed limit output continue
No tiles downloaded, or driving off-map	Publishes empty map fields with tileLoaded false	Unchanged, and vision-curve/external speed limit control keep working
Tile load or current-road resolution fails	Can retain and republish dependent route/control fields	Clears map-derived state until a later position produces loaded map data and a current road
Lookahead or curvature derivation fails	Can retain affected derived state	Clears the affected upcoming/curve state without discarding valid current-road data
Car input recovers after a gap	The outage interval can enter distance timing	Advances timestamps without inserting the unknown interval into the moving average
Model input recovers after a gap	Blends the fresh trajectory with retained pre-stall samples	Reinitializes vision smoothing from the fresh trajectory
Default-invalid mapdIn command	Handled	Unchanged; handled

Validation

• Exact change: Head 669b5e07 changes six production files and adds no test files.
• Exact-head review: Static state-transition oracles covered unseen, valid, invalid, future-dated, fresh, exact-timeout, and recovered subscriber states; external/internal GPS priority and fallback; command compatibility; car/model recovery; tile/current-road failures; short and dead-end roads; optional lookahead/curvature failures; and both output serializers.
• Independent review: An adversarial audit of the preceding head bedfde92 drove the corrections in this head. It reproduced, with a fake-clock harness over real msgq queues, 18 GPS source switches in 10 s when external GPS degraded to roughly a 1 s cadence while internal stayed healthy, and established that an all-or-nothing output gate silenced vision-curve control and external speed limits whenever GPS, tiles, or current-road resolution were unavailable — including permanently on an install with no tiles downloaded. That audit separately verified the failover state machine itself against its specification: cold start, preference, strict timeout boundaries, recovery, invalid-event and no-fix rejection, and legacy flag acceptance.
• Failover harness: An audit-only fake-clock harness over the selection logic confirms the corrected head holds one source across 200 ticks of degraded external cadence (0 switches, against 18 on bedfde92), adopts external immediately when no internal fix exists, defers promotion while internal is healthy, fails over on external staleness, and reports both-dead as no source with Fresh() false.
• Compile evidence: FrogAi Build #35 completed make build successfully for exact Head 669b5e07.
• Evidence limit: No hardware or on-road failover run was executed. The transition matrix above is exact-source reasoning supported by the fake-clock harness, not a runtime end-to-end test.

Compatibility and scope

• No Cap'n Proto schema, published field definition, setting, CLI contract, or subscriber Read() signature changes.
• Event.valid on mapdOut now carries health semantics: it is false when car state is stale, and its payload is default-zero in that case. Consumers that ignore Event.valid see zeroed fields during a car-state outage.
• Map-derived fields (road identity, speed limits, hazards, advisory speeds, map curve speed, extended path) return to their default values while GPS, tiles, or current-road resolution are unavailable, rather than retaining their last values. tileLoaded continues to signal the missing-map case.
• Vision-curve speed and externally supplied speed limits are unaffected by GPS, tile, or current-road availability.
• Command channels retain decoded-message semantics, including current Event.valid=false download/cancel inputs.
• External GPS remains preferred, and internal GPS is now kept current for fallback.
• mapdExtendedOut remains valid for independent settings and download progress; only its GPS/map/current-road path is emptied when unavailable.
• No producer-timestamp ordering, coordinate plausibility/accuracy policy, tile retry/backoff, schema-level per-field validity, or downstream-consumer change is included.

Merge order with #114

#114 rate-limits tile-load retries in the same loop this PR restructures. The two conflict in main.go, but only in the both-added declaration block near the top, where the union is the only resolution that compiles. The semantically important hunk auto-merges silently about 115 lines below, inside the GPS branch, where a maintainer reviewing the conflict would never look — the same class of hazard that made the #112/#113 merge silently corrupt tiles.

A trial merge confirms #114's retry bookkeeping lands above this PR's early exits, so the backoff is not defeated. What remains is two stacked "map unusable" exits with different bookkeeping: this PR's if !state.MapValid { ClearRoute(); continue } and #114's bare tile-loaded gate, which sets neither MapValid nor clears the route. No currently reachable stale-state leak was constructed, so it is redundant rather than broken — but because MapValid would then be assigned only inside the backoff branch, any future widening of the retry delay makes the bare gate the live exit and silently skips this PR's invalidation.

Land this PR first, then fold #114's gate into the single form state.MapValid = state.Data.Loaded; if !state.MapValid { state.ClearRoute(); continue } rather than leaving both. Composition with #118 and #119 is clean and needs no intervention; the four-PR stack is byte-identical to the union of the pairs.

Rebased onto current main (8dcfd0a)

Two things landed in main underneath this PR: shadow-subscription settings, which changed how the GPS subscribers are constructed, and #114's tile-retry rate limiting, which rewrote the same map-load block this PR restructures. The rebase keeps main's SubscriberSettings.ShadowGpsLocation / ShadowGpsLocationExternal wiring alongside this PR's failover state machine, and unions the two declaration blocks.

The map-load composition was resolved deliberately rather than mechanically. #114's retry gate is preserved, and state.MapValid = state.Data.Loaded is now assigned on every GPS tick rather than only inside the retry branch, with an unloaded tile clearing route state before continuing. Leaving both gates in place would have worked today but made MapValid sticky: it would only be reassigned when the backoff allowed a load attempt, so any future widening of the retry delay would silently skip this PR's invalidation. The collapsed form also means a corrupt tile clears the current way rather than freezing the previous cell's speed limit — the regression #114 was itself corrected for.

Re-verified against the rebased base: the failover harness still shows no source flapping across 200 ticks of degraded external cadence, immediate adoption of external when no internal fix exists, deferred promotion while internal is healthy, failover on external staleness, and both-dead reported as no source with Fresh() false. FrogAi Build #59 passed for the exact head.

[FrogAi]

Closing this version because it encodes several undocumented product policies—required input health, GPS fix/failover, stale-output clearing, and recovery—inside one six-file state machine. Those contracts need to be established before implementation, then any proven defect should be split to the smallest owner. Keeping this open would ask the maintainer to review policy guesses rather than a bounded correction.