Problem
When files are deleted from disk, Tantivy indexes retain entries forever, causing stale search results and increased index size.
Proposed change
Add index garbage collection during the crawl/run step: collect all current filesystem paths and remove any indexed documents whose stored path no longer exists on disk.
Implementation notes
- After file traversal, build a set of current paths (normalized).
- Use Tantivy's delete_by_term on the path field for documents that are missing on disk. Batch deletes for performance and commit periodically.
- If the number of deletes is extremely large, consider triggering a full reindex as a fallback.
- Provide logging and a --dry-run mode to preview deletions.
- Add tests validating that deleted files are removed from the index after a run.
Acceptance criteria
- Deleting a file on disk results in that file being removed from search results within one subsequent cix run.
- Index size does not accumulate unbounded due to deleted files.
Problem
When files are deleted from disk, Tantivy indexes retain entries forever, causing stale search results and increased index size.
Proposed change
Add index garbage collection during the crawl/run step: collect all current filesystem paths and remove any indexed documents whose stored path no longer exists on disk.
Implementation notes
Acceptance criteria