Skip to content

fix: preserve untouched fragments in slow merges - #56

Merged
universalmind303 merged 1 commit into
daft-engine:mainfrom
FANNG1:fix/preserve-untouched-fragments
Jul 22, 2026
Merged

fix: preserve untouched fragments in slow merges#56
universalmind303 merged 1 commit into
daft-engine:mainfrom
FANNG1:fix/preserve-untouched-fragments

Conversation

@FANNG1

@FANNG1 FANNG1 commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

Summary

  • preserve untouched fragment metadata when merge_columns_df uses the slow path with partial-fragment input
  • share the fragment-completion logic with the existing fast path
  • add a four-fragment regression test covering partially touched and completely untouched fragments

Problem

LanceOperation.Merge builds the new manifest from the fragment metadata passed to it. The slow path previously passed only metadata returned by fragment.merge(...), so fragments absent from the input DataFrame disappeared from the committed dataset version.

The fix completes the metadata list from the driver dataset snapshot before committing. The existing read_version check continues to reject concurrent changes rather than mixing snapshots.

Testing

  • .venv/bin/pytest -q tests/io/lancedb/test_lance_merge_evolution.py tests/io/lancedb/test_fast_path_merge.py (44 passed)
  • Ruff check and format check
  • Mypy on daft_lance/lance_merge_column.py
  • git diff --check

Fixes #55

@universalmind303 universalmind303 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks again @FANNG1

@universalmind303
universalmind303 merged commit 1b1fa3c into daft-engine:main Jul 22, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Slow-path partial-fragment merge silently drops untouched fragments

2 participants