Skip to content

Skip generated cells with no assigned ways - #112

Closed
FrogAi wants to merge 1 commit into
pfeiferj:mainfrom
FrogAi:codex/skip-empty-generated-cells
Closed

Skip generated cells with no assigned ways#112
FrogAi wants to merge 1 commit into
pfeiferj:mainfrom
FrogAi:codex/skip-empty-generated-cells

Conversation

@FrogAi

@FrogAi FrogAi commented Aug 9, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Determine generated-cell emptiness from the ways actually assigned to that cell.
  • Honor GenerateEmptyFiles=false before allocating or serializing an empty tile.
  • Preserve the existing output for populated cells and for every cell generated with GenerateEmptyFiles=true.

Motivation

The current empty-cell check does not inspect ways. It first requires the requested bounds to contain a cell, then asks whether those same bounds overlap the cell. The overlap is guaranteed by the preceding containment check, so GenerateEmptyFiles=false cannot suppress any selected cell.

The generator only knows whether a cell is empty after applying its existing way-assignment rule. This change moves that unchanged assignment loop before Cap'n Proto allocation and skips the cell when the resulting way list is empty and empty-file generation is disabled.

Behavior

A focused generation fixture selected nine quarter-degree cells containing three populated cells and six cells with no assigned ways:

Setting Base 68813e05 Head c57618e
GenerateEmptyFiles=false: total cells written 9 3
GenerateEmptyFiles=false: zero-way cells written 6 0
GenerateEmptyFiles=true: total cells written 9 9
Populated cells written 3 3

All six retained empty files in the true case decoded as valid zero-way tiles. The three populated files kept the same paths, expected way counts, and byte-identical packed contents across flag modes.

"Empty" remains defined by the existing inclusive way-box overlap check, including the configured overlap margin. This does not change boundary assignment or replace the current AABB model with exact road geometry.

Validation

The focused oracle used a real Delaware PBF normalized through the repository's osmium add-locations-to-ways preprocessing step:

  • Reproduced the defect on base: disabling empty files still wrote all nine selected cells, including six readable zero-way tiles.
  • Confirmed the candidate writes only the three populated cells when empty files are disabled.
  • Confirmed enabling empty files still writes all nine cells.
  • Compared exact populated filenames, way counts, and packed bytes between both flag modes.
  • Ran go test ./..., go vet ./..., and go build ./... with Go 1.25.1 on Linux/amd64.
  • Reproduced the focused oracle in a fresh network-disabled review environment.
  • FrogAi Build #41 passed make build for exact commit c57618ebde71678997eb0fa8933d98a401e2c7cd.

Compatibility

  • No offline schema, filename format, public API, dependency, or populated-tile content changes.
  • GenerateEmptyFiles=true retains the current selected-cell output.
  • GenerateEmptyFiles=false intentionally stops writing newly generated cells with no assigned ways.
  • mapd generate is no longer idempotent for a reused output directory. Skipping the write means any previously generated tile for a now-empty cell survives, so the output becomes the union of every PBF ever generated into that directory — including stale populated tiles, not just the stale empty ones. Reproduced twice independently: regenerating with a source that no longer has ways for a cell leaves the old tile in place, and the daemon then serves deleted roads as authoritative. Base overwrote it with a zero-way tile. Reachable by default, since generate bounds cover the whole world.
  • A missing tile is not an empty tile at runtime. main.go continues when FindWaysAroundPosition errors, skipping the block that is the only place CurrentWay, NextWays, Curvatures, and TargetVelocities are assigned. Base loaded the empty tile, GetCurrentWay returned fail, and the speed limit cleared to 0; head leaves the previous road's wayName and speed limit republishing at 20 Hz for as long as you are in that cell, with speedLimitSuggestedSpeed still clamping cruise — effectively an unconditional hold-last-seen. It also flips tileLoaded from true to false for genuinely road-free cells and logs a warning per GPS fix. Reachability is bounded by scripts/filter_planet.sh, which strips the planet to 12 highway classes before generation, so an "empty" cell has no motorway through residential road; in a Delaware run, zero cells were drivable-yet-empty. Sparse regions are unproven either way. Fixing this means touching main.go (drop the continue, or only continue on a real I/O fault rather than ENOENT), which is out of scope here and interacts with #114.
  • The hosted corpus does not shrink on its own: scripts/upload_offline.sh uses rclone copy rather than sync, and client extraction never deletes, so the size win is not delivered to downloaders until the bucket is purged. Once it is, the download progress counter stalls below 100% on road-free groups, because countFilesForBounds counts group rectangles arithmetically while 404s skip the downloaded-file increment.

Merge order with #113 — read before resolving

Merging #113 with this PR conflicts, and the obvious resolution silently corrupts every generated tile. This PR relocates the way-assignment loop; #113 replaces that loop with a bucketed version. Git reports a conflict at the bucketed block, but this PR's relocated loop auto-merges about 20 lines above the conflict markers, where it is invisible in the conflict view. Taking the incoming block therefore appends to area.Ways twice. Measured on the naive resolution: 33 tiles, 151,994 ways, 75,997 duplicate way IDs, 2x tile size — with go build, go vet, and go test all green, so CI will not catch it.

Correct resolution, built and verified byte-identical to this PR alone: delete this PR's relocated loop and its gate, keep #113's bucketed block after the Contains check, then place if len(area.Ways) == 0 && !s.GenerateEmptyFiles { continue } after it and before the capnp arena allocation. #97 merges cleanly with this PR (output byte-identical); stacking all three adds only a self-diagnosing stdmath "math" import-alias conflict.

  • Spatial bucketing and the remaining O(selected cells x scanned ways) assignment cost are handled separately.
  • The global generation-grid allocation and degenerate origin entries addressed by PR Avoid allocating global offline area grid #97 are unchanged.
  • Runtime missing-tile retry behavior and the ignored generator input-file setting are unchanged.

@FrogAi

FrogAi commented Aug 10, 2026

Copy link
Copy Markdown
Contributor Author

Closing this because omitting empty tiles changes runtime semantics from a loaded, authoritative empty tile to a missing/unloaded tile. On current runtime paths, that changes retry/state behavior and can leave prior road-derived state in use. Empty-tile generation should not change until the runtime contract for intentionally empty coverage and stale-state clearing is defined.

@FrogAi FrogAi closed this Aug 10, 2026
@FrogAi
FrogAi deleted the codex/skip-empty-generated-cells branch August 10, 2026 04:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant