fix(cost): quarantine unreadable session logs instead of failing every poll - #224
Open
rohanpoudel2 wants to merge 4 commits into
Open
fix(cost): quarantine unreadable session logs instead of failing every poll#224rohanpoudel2 wants to merge 4 commits into
rohanpoudel2 wants to merge 4 commits into
Conversation
…y poll `readSessionUsage` treated its two failure modes asymmetrically. An oversized event marked the session `unreadable` before rethrowing, so the next poll skipped that file and the failure surfaced once. A session file that could not be opened at all — a non-ENOENT `open()` error such as EACCES — was rethrown without marking anything, so every subsequent poll reopened the same file and threw again, forever. The permanence compounds in `#readSessions`, which only defers a failure when the session's thread is already known. Because `open()` failed, the thread id was never read, so the error bypasses the `included` thread-tree filter and aborts the whole scan even when the file belongs to an unrelated prior session. A root-owned rollout left behind by a single `sudo codex` run is enough to trigger it: with a cost limit the poll's `onError` aborts the scan almost immediately, and without one `stop()` rejects and reports a scan that completed successfully as failed. Quarantine unopenable sessions through the same path as oversized ones so the failure is reported once and later polls keep tracking the sessions they can read. `isMissingFile` handling is unchanged: a file that vanished between the directory walk and the open is still skipped silently. Also guard the `refresh()` inside `stop()`. `stop()` runs after the turn is over, so it cannot abort anything and cannot under-enforce `--max-cost`; enforcement happens in the polling path, which still reports errors. Letting it reject only discarded the authoritative usage the completed turn handed to it, which is exactly the fallback the existing "falls back to the completed turn when session logs are unavailable" test documents but which was unreachable whenever the sessions directory existed and could not be scanned.
Author
|
Note on overlap: #177 and #198 also change |
Collaborator
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: fdc1be55f4
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
Security review completed. No security issues were found in this pull request. Reviewed commit: Only the user who started this review can view the report in Codex. ℹ️ About Codex security reviews in GitHubThis is an experimental Codex feature. Security reviews are triggered when:
Once complete, Codex will leave suggestions, or a comment if no findings are found. |
Two follow-ups to the quarantine change, both about usage the tracker stops observing. `stop()` swallows a failed final refresh so a completed turn is not reported as a failure, but it then returned `#snapshot` purely because it was non-null. When the last poll succeeded and the final refresh did not - the scan's own rollout gaining an oversized event, or its permissions changing as the turn completes - that snapshot predates the completed turn, so the caller's authoritative `fallbackUsage` was discarded, no fresh `onCost` fired, and `api.ts` recorded the stale below-limit cost for a turn that had already passed `maxCostUsd`. `stop()` now tracks whether this refresh succeeded and, when it did not, takes whichever of the two charges more, so a stale poll can no longer hide spend and the snapshot still wins whenever it counts delegated worker threads the completed turn does not. The open failure exit quarantined unconditionally, which is wrong for a process-wide shortage: an EMFILE from momentary descriptor pressure retires the session permanently, and every later refresh short-circuits at the `session.unreadable` guard, so its usage is never observed again even after descriptors free up. Quarantine is now limited to persistent file-specific codes (EACCES, EPERM, EISDIR, ELOOP, ENAMETOOLONG, ENOTDIR); anything else stays retryable. The root-owned rollout from issue openai#223 is EACCES, so it is still reported once and then skipped. Four tests in tests-ts/cost.test.ts: the stale snapshot losing to a larger completed turn and winning against a smaller one, and an open failure that recovers on a later poll for EMFILE while EACCES is never reopened.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #223
Problem
readSessionUsagehas two failure exits and only one of them records that the session is bad. The oversized-event exit setssession.unreadable = truebefore rethrowing, so theif (session.unreadable) return;guard skips that file on every later poll — one error, then the tracker moves on. Theopen()exit just rethrows, so the next poll reopens the same file and fails identically, for the life of the scan.#readSessionsamplifies it: a failure is only deferred to theincludedthread-tree filter whensession.threadId !== null. An unopenable session never got a thread id, soif (session.threadId === null) throw error;fires and the error escapes the filter that exists precisely to ignore unrelated sessions. A root-owned rollout from onesudo codexrun kills every subsequent scan.The suite already pins the tolerant half — "ignores oversized events from unrelated prior credential sessions". This PR makes the unopenable case behave the same way.
Change
src/cost.ts, +11/−4. The three quarantine lines move into aquarantineSession(session)helper called from both exits, so an open failure is handled exactly like a parse failure.isMissingFilehandling is untouched: a file that vanished between the directory walk and the open is still skipped silently, with no quarantine.The
stop()guard, and why tolerant is rightstop()now wrapsawait this.refresh()intry { … } catch {}. This deserves scrutiny because weakening cost handling could weaken--max-cost, so to be explicit about why it does not:start()→refresh()→onCost→ abort).stop()runs after the turn is over. There is nothing left to abort, and rejecting cannot recover spend that already happened.onError; swallowing it instop()hides no first occurrence.fallbackUsageis the completed turn's own usage — better than the log-scraped snapshot, not worse. Rejecting discarded it and turned a successful scan into a failure.onErrorwould have been self-defeating. Inapi.ts,onErrorabortscostAbortControllerandonFinalizecallsthrowIfAborted(signal, scanDir)immediately afterstop(), so reporting fromstop()would reconstruct the exact failure this fixes.api.tsalready doesawait costTracker?.stop().catch(() => null)on the failure path.Precedence is unchanged — a real snapshot still wins over the fallback, and no max-of-the-two logic was added.
Verification
Two tests in
tests-ts/cost.test.tsusing the repository'stestPosixpattern, each restoring0o600in afinallysoafterEachteardown can still delete the fixture. The first is deliberately order-independent: whichever filereaddiryields first, the firstrefresh()rejects and the second must succeed — that second call is the permanence bug.Before:
After:
15 pass / 0 fail.The two halves are independently load-bearing: reverting only the
stop()guard while keeping the quarantine gives14 pass / 1 fail, with just the fallback test failing. So the quarantine alone fixes the permanence, and thestop()guard alone fixes the discarded fallback.Full suite: 719 pass / 5 skip / 0 fail (717 baseline plus the 2 new tests).
pnpm run typesandpnpm run formatare clean.Deliberately out of scope
sessionFileshas the same shape at directory level — a non-ENOENTreaddirerror has no per-directory state to quarantine and would also repeat every poll. It is outside this trigger and needs new state rather than reuse of the existing flag, so I left it for a separate change. Noted in #223.