Skip to content

fix: prevent document ingestion finalization errors - #304

Merged
suguanYang merged 5 commits into
mainfrom
fix/wangbinqi/document-ingestion-errors
Aug 19, 2026
Merged

fix: prevent document ingestion finalization errors#304
suguanYang merged 5 commits into
mainfrom
fix/wangbinqi/document-ingestion-errors

Conversation

@suguanYang

Copy link
Copy Markdown
Contributor

Summary

  • make S3 upload and confirm-upload handoff idempotent for the same job
  • skip duplicate worker deliveries when the processing lock is held
  • deduplicate sanitized document chunk paths before publication
  • remove PostgreSQL-incompatible NUL characters at the persistence boundary while preserving the original result archive

Why

The staging Logfire alerts showed three correctness failures:

  1. S3 upload and confirm-upload could enqueue two deliveries for one logical job; the losing delivery could eventually fail/refund the active job.
  2. Parser output could contain duplicate non-null source_chunk_path values, violating uq_document_chunks_revision_path.
  3. Parser text or metadata could contain NUL characters, causing PostgreSQL persistence to fail after parsing completed.

Verification

  • API handoff contract tests: 3 passed
  • Worker processing/publication contract tests: 4 passed
  • Ruff: passed
  • git diff --check: passed

Production was not changed by this PR.

Comment thread apps/worker/tests/contract/test_processing_run_contract.py Fixed
Comment thread apps/worker/tests/contract/test_processing_run_contract.py Fixed
@suguanYang
suguanYang merged commit ae5604e into main Aug 19, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants