Avoid duplicate parse dispatch when S3 notification races confirm-upload

[suguanYang]

Problem

A file upload can dispatch two parse_task messages when the S3 ObjectCreated notification races the explicit POST /v1/jobs/{job_id}/confirm-upload fallback.

This was observed during the 2026-08-14 staging Fargate queue-drain drill for job_27204d831b6a:

• confirm-upload read the Job in waiting-file and dispatched task 3c0331ef-f6ef-4c5c-aee7-f321696eceed.
• The S3 SNS event had concurrently loaded the same waiting-file Job. After one CAS retry, its handoff accepted pending → pending and dispatched task 4bb9aa00-037a-4674-b052-ceaaf8d6b562.
• RedisJobLock allowed only one task to process. The duplicate retried after 120 seconds, observed terminal state done, and skipped.
• The Job completed once and billing remained exactly one page (0.015 credits).

The current safety layers prevent duplicate processing and billing, but the redundant dispatch consumes worker/broker capacity and creates avoidable retry noise.

Confirmed cause

DocumentIngestionConfirmationService permits both waiting-file and pending, while the S3 upload handler guards on a previously loaded waiting-file row. Both paths call DocumentIngestionHandoffService.start_uploaded_file_workflow(). Same-state transitions are valid, and a successful pending → pending outcome still calls the worker dispatcher.

This is an existing application-level race, not a Fargate/Redis queue-persistence failure, and is non-blocking for Ontos-AI/knowhere-api-infra#19.

Acceptance criteria

• A concurrent S3 upload notification and confirm-upload request produce one worker parse dispatch for a Job.
• Repeated S3 notifications and repeated confirm-upload requests are idempotent at the dispatch boundary.
• confirm-upload remains a functional fallback when the storage event is absent.
• Add contract tests at the real handoff/dispatch seam, including the concurrent S3-plus-confirm case.
• Preserve one terminal Job result and one billing charge.
• Do not weaken RedisJobLock or the worker terminal-state gate; they remain defense-in-depth for broker redelivery and other at-least-once delivery paths.

Scope

Track and fix this separately from the infrastructure cost-reduction work. Do not expand the Fargate migration to include this application optimization.