Skip to content

[task-service] 🤖 fix: prevent false workspace turn interruptions - #3915

Open
coadler wants to merge 9 commits into
mainfrom
fix/workspace-turn-false-interruptions
Open

[task-service] 🤖 fix: prevent false workspace turn interruptions#3915
coadler wants to merge 9 commits into
mainfrom
fix/workspace-turn-false-interruptions

Conversation

@coadler

@coadler coadler commented Aug 21, 2026

Copy link
Copy Markdown
Collaborator

Summary

Prevent false workspace-turn interruptions during nested child activity.

Background

Workspace 7b72901012 showed several completed child turns as interrupted.

Terminal reports intentionally replaced queued progress reports. The cleanup also invoked the cancellation callback.

Synthetic continuation rows could also hide the active workspace-turn correlation.

Implementation

  • Remove replaced progress reports without canceling the owning workspace turn.
  • Clear queued progress for every finalized best-of task.
  • Preserve correlation across file-change notices and correlated nested reports.
  • Queue a correlated terminal continuation before removing queued progress.
  • Claim terminal-attention notifications during direct and drain delivery.
  • Preserve delivery cleanup when queue reordering removes stale turn correlation.
  • Re-arm durable terminal attention after all accepted startup failure paths.
  • Use generation-specific terminal queue keys for reawakened stable children.
  • Persist report generations through artifacts, history rows, and restart recovery.
  • Select recovery artifacts by the notification outcome before generation matching.
  • Mark queued terminal fallbacks as workspace-turn continuations.
  • Recheck stream-end ordering when the active workspace-turn handle changes.
  • Await reserved continuation outcomes for correlated and uncorrelated stream ends.
  • Recheck continuation reservations under the settlement lock before terminal settlement.

Risks

This change modifies task and workspace lifecycle handling.

A regression could delay a real interruption or duplicate a terminal response. Exact correlation and durable claim checks limit this risk.


Generated with mux • Model: openai:gpt-5.6-sol • Thinking: xhigh • Cost: $1.44

@coadler

coadler commented Aug 21, 2026

Copy link
Copy Markdown
Collaborator Author

@codex review

Please review the workspace-turn lifecycle changes.

@chatgpt-codex-connector

This comment has been minimized.

@coadler

coadler commented Aug 21, 2026

Copy link
Copy Markdown
Collaborator Author

@codex review

Please review the implementation. The prior response only contained the completed security review.

@chatgpt-codex-connector

This comment has been minimized.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8fbe142a01

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/node/services/taskService.ts Outdated
Comment thread src/node/services/taskService.ts Outdated
@coadler

coadler commented Aug 21, 2026

Copy link
Copy Markdown
Collaborator Author

Addressed both Codex findings in f5674e42d.

  • PRRT_kwDOPxxmWM6bQLsI: An uncorrelated stream ending now stays nonterminal only while an exact correlated continuation is pending or streaming. The code no longer relies on terminal resettlement.
  • PRRT_kwDOPxxmWM6bQLsL: The terminal report continuation is accepted before queued progress is removed. An enqueue failure keeps the progress wake available.

Added regression tests for waiter preservation, handoff order, enqueue failure, and explicit cancellation.

@coadler

coadler commented Aug 21, 2026

Copy link
Copy Markdown
Collaborator Author

@codex review

Both P1 findings are addressed in f5674e42d. Please review the new handoff behavior.

@chatgpt-codex-connector

This comment has been minimized.

@coadler

coadler commented Aug 21, 2026

Copy link
Copy Markdown
Collaborator Author

@codex review

The security review completed, but no code-review approval appeared. Please run the code review for f5674e42d.

@chatgpt-codex-connector

This comment has been minimized.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f5674e42de

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/node/services/taskService.ts
@coadler

coadler commented Aug 21, 2026

Copy link
Copy Markdown
Collaborator Author

@codex review

The CI unit test now matches the queued terminal continuation behavior in 07eba9212. Please review the latest commit.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 07eba9212c

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/node/services/taskService.ts Outdated
@chatgpt-codex-connector

This comment has been minimized.

@coadler

coadler commented Aug 21, 2026

Copy link
Copy Markdown
Collaborator Author

Addressed PRRT_kwDOPxxmWM6bQkb8 in 3da714e8c.

The terminal attention record is now persisted before the background continuation starts. The onAccepted callback marks it delivered before stream startup continues. This prevents the fast stream-end from missing the record and causing a later duplicate drain.

Added a regression test that invokes onAccepted during sendMessage and verifies no pending attention remains.

@coadler

coadler commented Aug 21, 2026

Copy link
Copy Markdown
Collaborator Author

@codex review

The final attention-ordering finding is addressed in 3da714e8c. Please review the latest commit.

@chatgpt-codex-connector

This comment has been minimized.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3da714e8cf

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/node/services/taskService.ts Outdated
Comment thread src/node/services/taskService.ts
@coadler

coadler commented Aug 21, 2026

Copy link
Copy Markdown
Collaborator Author

Addressed the current Codex findings in a5f36592d.

  • Clear queued progress for every finalized group member: cleanup now covers every finalization.taskIds entry.
  • Hide the reservation from concurrent attention drains: owner-scoped notification claims now prevent duplicate delivery.
  • Re-arm attention when accepted stream startup fails: all accepted failure paths restore pending attention and schedule retry.

The fix also preserves delivery callbacks after correlation removal.

Stable child generations now use generation-specific terminal queue keys.

Local validation passed:

  • bun test src/node/services/taskService.test.ts
  • bun test src/node/services/workspaceService.test.ts
  • bun test src/node/services/messageQueue.test.ts
  • make static-check

@coadler

coadler commented Aug 21, 2026

Copy link
Copy Markdown
Collaborator Author

@codex review

Please review commit a5f36592d.

@chatgpt-codex-connector

This comment has been minimized.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a5f36592d3

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/node/services/messageQueue.ts Outdated
@coadler

coadler commented Aug 21, 2026

Copy link
Copy Markdown
Collaborator Author

Addressed Return a removal result even without callbacks in a64261fa1.

MessageQueue.removeWorkspaceTurn() now returns an empty callback object after a successful callback-free removal.

AgentSession.removeQueuedWorkspaceTurn() therefore emits the queue change, updates the background queue flag, and returns true.

Added a regression for a workspace-turn entry with no callbacks.

Local validation passed:

  • bun test src/node/services/messageQueue.test.ts
  • bun run tsgo --noEmit -p tsconfig.main.json
  • make static-check

@coadler

coadler commented Aug 21, 2026

Copy link
Copy Markdown
Collaborator Author

@codex review

Please review commit a64261fa1.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Already looking forward to the next diff.

Reviewed commit: a64261fa1f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@chatgpt-codex-connector

This comment has been minimized.

@coadler
coadler added this pull request to the merge queue Aug 21, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Aug 21, 2026
@coadler
coadler added this pull request to the merge queue Aug 21, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Aug 21, 2026
Mux added 6 commits August 21, 2026 17:41
Prevent intentional report coalescing from canceling its owning workspace turn. Preserve correlation across machine-generated continuation rows. Allow exact correlated results to repair inferred uncorrelated interruptions.

---

_Generated with `mux` • Model: `openai:gpt-5.6-sol` • Thinking: `xhigh` • Cost: `$0.13`_

<!-- mux-attribution: model=openai:gpt-5.6-sol thinking=xhigh costs=0.13 -->

Co-authored-by: Mux <noreply@coder.com>
Queue the correlated terminal continuation before removing queued progress. Keep uncorrelated stream endings nonterminal while an exact continuation exists. Retain queued progress if the terminal continuation cannot start.

---

_Generated with `mux` • Model: `openai:gpt-5.6-sol` • Thinking: `xhigh` • Cost: `$0.13`_

<!-- mux-attribution: model=openai:gpt-5.6-sol thinking=xhigh costs=0.13 -->

Co-authored-by: Mux <noreply@coder.com>
Update the nested terminal report test for queued correlated continuation delivery.

---

_Generated with `mux` • Model: `openai:gpt-5.6-sol` • Thinking: `xhigh` • Cost: `$0.26`_

<!-- mux-attribution: model=openai:gpt-5.6-sol thinking=xhigh costs=0.26 -->

Co-authored-by: Mux <noreply@coder.com>
Persist terminal attention before a background terminal continuation can start. Mark it delivered during acceptance to prevent a duplicate idle drain.

---

_Generated with `mux` • Model: `openai:gpt-5.6-sol` • Thinking: `xhigh` • Cost: `$0.26`_

<!-- mux-attribution: model=openai:gpt-5.6-sol thinking=xhigh costs=0.26 -->

Co-authored-by: Mux <noreply@coder.com>
@coadler
coadler force-pushed the fix/workspace-turn-false-interruptions branch from a64261f to b11b7ea Compare August 21, 2026 22:49
@coadler

coadler commented Aug 21, 2026

Copy link
Copy Markdown
Collaborator Author

@codex review

Rebased onto current main. Please review the rebased commit.

@chatgpt-codex-connector

This comment has been minimized.

@coadler

coadler commented Aug 21, 2026

Copy link
Copy Markdown
Collaborator Author

@codex review

Please review the rebased commit. The prior automated response only reported the security review result.

@chatgpt-codex-connector

This comment has been minimized.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b11b7eacea

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/node/services/taskService.ts Outdated
Comment thread src/node/services/taskService.ts
@coadler

coadler commented Aug 21, 2026

Copy link
Copy Markdown
Collaborator Author

@codex review

All required CI checks now pass on b11b7eacea. Please run the code review for this rebased commit.

@chatgpt-codex-connector

This comment has been minimized.

@coadler

coadler commented Aug 21, 2026

Copy link
Copy Markdown
Collaborator Author

Addressed both findings in 0e8fde0ff.

  1. Terminal reservation settlement race: The active workspace turn now publishes an exact-correlation handoff reservation under the settlement lock. An uncorrelated stream-end captures that reservation under the same lock and waits for its result. It returns only after a continuation installs. It retries the original settlement after a failed handoff. The terminal attention drain uses the same reservation contract.
  2. Deferred best-of progress cleanup: The deferred grouped finalizer now removes queued progress for every finalized sibling. It preserves notifyCancellation: false.

Validation:

  • Full src/node/services/taskService.test.ts: 397 passed.
  • Full TypeScript checks passed.
  • ESLint passed.
  • Formatting, eager-import, code-to-docs, and shellcheck checks passed.

@coadler

coadler commented Aug 21, 2026

Copy link
Copy Markdown
Collaborator Author

@codex review

Both new lifecycle findings are addressed in 0e8fde0ff. Please review the latest commit.

@chatgpt-codex-connector

This comment has been minimized.

@coadler

coadler commented Aug 21, 2026

Copy link
Copy Markdown
Collaborator Author

@codex review

The security review for 0e8fde0ff completed without findings. Please run the code review for this commit.

@chatgpt-codex-connector

This comment has been minimized.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 0e8fde0fff

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/node/services/taskService.ts
Comment thread src/node/services/taskService.ts Outdated
Comment thread src/node/services/taskService.ts
@coadler

coadler commented Aug 21, 2026

Copy link
Copy Markdown
Collaborator Author

@codex review

All findings are resolved and local validation passes. Please run the code review for 0e8fde0ff.

@chatgpt-codex-connector

This comment has been minimized.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 0e8fde0fff

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/node/services/taskService.ts Outdated
Mux added 2 commits August 21, 2026 19:11
---

_Generated with `mux` • Model: `openai:gpt-5.6-sol` • Thinking: `xhigh` • Cost: `$1.44`_

<!-- mux-attribution: model=openai:gpt-5.6-sol thinking=xhigh costs=1.44 -->
---

_Generated with `mux` • Model: `openai:gpt-5.6-sol` • Thinking: `xhigh` • Cost: `$1.44`_

<!-- mux-attribution: model=openai:gpt-5.6-sol thinking=xhigh costs=1.44 -->
@coadler

coadler commented Aug 22, 2026

Copy link
Copy Markdown
Collaborator Author

@codex review

The latest commits address all reported continuation and restart-recovery races.

@chatgpt-codex-connector

Copy link
Copy Markdown

Security review completed. No security issues were found in this pull request.

Reviewed commit: f4616ece24

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f4616ece24

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines 6563 to 6567
onCanceled: () => {
this.releaseTerminalAttentionClaims(ownerWorkspaceId, pending);
this.scheduleTerminalAttentionDrainAfterIdle(ownerWorkspaceId);
},
onAcceptedPreStreamFailure: async () => {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Preserve fallback delivery callbacks after correlation stripping

When this idle-only fallback queues behind a busy turn and user input is later prioritized ahead of it, MessageQueue.revalidateWorkspaceTurnCorrelations() strips onCanceled and onAcceptedPreStreamFailure along with the stale workspace-turn metadata. These callbacks also own the terminal-attention claims, so clearing the reordered entry or failing during pre-stream startup leaves the notifications permanently claimed and prevents subsequent drains from retrying them until restart. Pass the claim-release/re-arm work through the delivery callbacks, as the direct terminal-continuation path does.

Useful? React with 👍 / 👎.

}

const correlation = this.buildWorkspaceTurnMuxMetadata(record);
if (this.hasConcreteSameTurnContinuation(event, correlation)) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Settle deferred turns when monitor wakes are canceled

When an uncorrelated queue-cut stream-end reaches this branch while a bash-monitor wake is pending, it returns without settling the handle or recording a deferred event. That wake can subsequently be removed when its process is canceled or its output is shown, but the wake's cancellation callback only retires its own delivery record and never re-enters TaskService; no replacement stream-end then arrives, leaving the workspace-turn handle and its waiters running indefinitely. Re-evaluate the captured stream-end when the cancelable wake is removed, or only defer once the continuation is durably accepted.

AGENTS.md reference: AGENTS.md:L211-L211

Useful? React with 👍 / 👎.

Comment on lines +714 to +715
const WORKSPACE_TURN_UNCORRELATED_STREAM_END_ERROR =
"Workspace turn superseded by an uncorrelated workspace stream-end";

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Allow inferred interruptions to self-heal

Fresh evidence is that this newly introduced marker is documented as correctable by later correlated evidence, but isSelfHealEligibleSettledWorkspaceTurn() does not include it. If an uncorrelated stream-end settles the handle just before a correlated final becomes visible, the later final passes allowTerminalResettle: true but is rejected because the current interrupted record is not eligible; read-time reconciliation skips it for the same reason, leaving the handle and caller permanently stuck with the false interruption. Add this marker to the inferred/self-heal eligibility set while continuing to exclude explicit user cancellations.

AGENTS.md reference: AGENTS.md:L110-L110

Useful? React with 👍 / 👎.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant