Fix permanent Lucene index corruption under concurrent indexing requests#35
Open
vany0114 wants to merge 2 commits into
Open
Fix permanent Lucene index corruption under concurrent indexing requests#35vany0114 wants to merge 2 commits into
vany0114 wants to merge 2 commits into
Conversation
IndexDocuments creates a fresh IndexWriter per request. Lucene allows only one IndexWriter per directory, so two concurrent indexing requests against the same index race: the loser fails with LockObtainFailedException, and under sustained concurrency a torn commit corrupts the segment files permanently (FileNotFoundException: .../_N.si on every subsequent write until the index directory is deleted from disk). Reproduced with 30 concurrent writers x 30 sequential mergeOrUpload requests against a fresh index: 889/900 requests returned 500 and the index remained broken after concurrency stopped. With this change the same load succeeds 900/900 and the index stays healthy. A per-index SemaphoreSlim serializes IndexDocuments per index name; requests against different indexes still run in parallel. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ch bodies, per-item outcomes The emulator logged only [HTTP METHOD] path — no response status, no batch body, no per-item results. A rejected or dropped index merge was therefore invisible: the Azure SDK IndexDocuments does not throw on per-item failures by default, so neither the client nor the emulator left any trace (cost a full diagnosis session on the Curbit devbox: an order READIED merge reached the emulator but the doc never showed READIED, and nothing said why). - request middleware logs the response status after the pipeline ([HTTP POST 207] ...) and any unhandled exception with type + message - search.index logs the full batch body before deserialization (a malformed or wrong-fields payload is otherwise undiagnosable) - search.index logs per-item outcomes keyed ([INDEX <idx>] 5/6 ok — key:200, key:404 FAILED(...)) — the only place a dropped write is guaranteed to surface Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Context
LuceneNetSearchIndexer.IndexDocumentsopens a freshIndexWriterper request. Lucene allows only a singleIndexWriterper directory, so two concurrent indexing requests against the same index race:LockObtainFailedException(HTTP 500), andSystem.IO.FileNotFoundException: Could not find file '.../_N.si'until the index directory is deleted from disk.We hit this in a CI environment where several services index into one shared index: 500 consecutive indexing requests failed after the corruption event, and the index never recovered on its own.
Repro
30 concurrent workers × 30 sequential
mergeOrUploadrequests against a freshly created index:FileNotFoundExceptionon segment files) — the index is permanently broken.Change
A static
ConcurrentDictionary<string, SemaphoreSlim>gatesIndexDocumentsper index name. Writes to the same index are serialized; requests against different indexes still run in parallel. Read paths are untouched.Test plan
dotnet test— unit tests 134/134 pass; integration tests 24/25 (the single failure pre-exists onmainand is unrelated).🤖 Generated with Claude Code