Fix stale Milvus vectors when documentation files are deleted or renamed

[Prasukjain271]

Summary

Fixes #221

This PR fixes stale/orphaned vectors remaining in Milvus during the full rebuild ingestion pipeline.

Problem

kubeflow-pipeline.py contains the full rebuild pipeline, which refreshes the Milvus collection from the current GitHub documentation snapshot.

Before this change:

• Existing documents were refreshed if they appeared in the latest ingestion run.
• However, deleted or renamed documents were not detected.
• Their vectors remained in Milvus because they were no longer part of the new ingestion data and therefore were never removed.

Example:

Initial ingestion:

docs/
├── installation.md
├── deployment.md

Milvus contains vectors for both files.

Later the repository changes:

docs/
├── setup.md (renamed from installation.md)
├── deployment.md

The pipeline would insert/update setup.md, but the old installation.md vectors would remain in Milvus.

Changes

1. Updated store_milvus() parameters

The store_milvus() function inside kubeflow-pipeline.py was updated to accept two additional parameters:

• repo_name
• directory_path

These parameters are required to identify the exact documentation scope being processed and prevent reconciliation from affecting unrelated documents in the Milvus collection.

2. Added reconciliation during full rebuild ingestion

Added a reconciliation step before inserting new embeddings.

The pipeline now:

1. Fetches existing file_unique_id values from Milvus for the current repository/directory.
2. Collects the file_unique_id values from the current GitHub ingestion run.
3. Compares both sets:

existing Milvus files - current GitHub files = orphan files

4. Removes orphan vectors from Milvus.

This handles cases where files are deleted or renamed in the source repository.

3. Refactored Milvus deletion logic

Moved the existing deletion logic into a reusable _delete_file_unique_ids() helper.

This helper is now used for:

• Removing old chunks before refreshing existing files.
• Removing orphaned vectors during reconciliation.

4. Added scoped Milvus querying

Added helper functions to build scoped queries based on:

• repository name
• directory path

This ensures reconciliation only affects documents belonging to the current ingestion scope.

A fallback query path is included if scoped querying fails.

5. Improved insertion error reporting

Improved batch insertion error handling by reporting:

• failing batch location
• affected file_unique_id values

This makes debugging failed Milvus insertions easier.

Testing

Tested locally using the complete ingestion flow:

download_github_directory
↓
chunk_and_embed
↓
store_milvus

Verified:

✅ New files are inserted into Milvus
✅ Existing files are refreshed correctly
✅ Deleted files are removed from Milvus
✅ Renamed files do not leave stale vectors behind

Tested using a local Milvus instance and a small test documentation repository

[google-oss-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign tarekabouzeid for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[Sharkyii]

Nice fix for the stale-vector problem - the scoping and fallback-query logic looks solid.

One blocking issue though: the reconciliation step isn't gated on records being non-empty. If download_github_directory fails to fetch any files (GitHub rate limit, transient API error, etc.) it silently returns an empty list rather than raising, and store_milvus would then treat every existing document in that repo/directory scope as orphaned and delete it turning a temporary fetch failure into a full collection wipe. Could you add a guard so reconciliation is skipped (or the run aborts) when records is empty?

[Prasukjain271]

@Sharkyii Thanks for catching this! I've added a guard to abort the pipeline when no records are produced, preventing reconciliation from treating an empty ingestion as a full deletion.

One more thought: if download_github_directory() returns a partial directory due to a transient GitHub/API error, reconciliation would still treat that partial snapshot as the source of truth and could incorrectly delete valid vectors (existing - partial_current).
The cleanest fix is to make download_github_directory() fail fast on traversal/API errors instead of returning partial results.

[Prasukjain271]

I addressed this by adding error handling in download_github_directory(). If a directory or file cannot be fetched, the pipeline now raises an error instead of ingesting a partial snapshot, preventing incorrect orphan deletion in Milvus.

I also added bounded retry logic (max 5 retries with backoff) for transient GitHub API failures, including rate limiting, to make large full rebuilds more resilient.