fix(agents): isolate agents-shell workspaces - #13441
Conversation
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 88ed6002fc
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
88ed600 to
4af2edf
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4af2edfb73
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
ec18543 to
6517ce0
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6517ce057e
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
6517ce0 to
c560af0
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c560af0ab9
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
c560af0 to
90455a6
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 90455a6917
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
050aaaa to
759460f
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 050aaaaf7b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
759460f to
a12e6cb
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a12e6cb92f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
a12e6cb to
448f1b7
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 448f1b7682
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
448f1b7 to
9d4ac85
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9d4ac8583e
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
9d4ac85 to
6c56aa6
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6c56aa640e
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
6c56aa6 to
d3eeeef
Compare
There was a problem hiding this comment.
💡 Codex Review
lab/services/agents/src/server/agents-shell/cli-policy.ts
Lines 88 to 94 in d3eeeef
Fresh evidence beyond the environment-only kubeconfig fix is that this branch accepts every argument after an allowlisted verb, so ['get', 'pods', '--kubeconfig=/leased/path/config'] overrides KUBECONFIG=/dev/null; the kubectl get parent options describe --kubeconfig as the path used for CLI requests. A lease owner can plant a kubeconfig containing an exec credential plugin, which kubectl then launches under the inspection identity; in the inspected production deployment that process can read the projected service-account credential and use the unrestricted egress plus cluster-admin binding to mutate the cluster through a tool declared read-only. Reject kubeconfig/credential/endpoint overrides or enforce a server-owned loading configuration that caller arguments cannot replace.
AGENTS.md reference: AGENTS.md:L97-L97
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
d3eeeef to
393bdbf
Compare
|
Exact head |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 393bdbff19
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
393bdbf to
3b4312a
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3b4312a292
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
3b4312a to
73ee2e1
Compare
0fe407d to
df3af81
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: df3af813de
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
df3af81 to
d9d3ffe
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d9d3ffe035
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
d9d3ffe to
4934c50
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4934c509b2
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
4934c50 to
865752b
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 865752b024
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
865752b to
ad0ce95
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ad0ce955dd
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
ad0ce95 to
f76b700
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f76b70047c
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
f76b700 to
8bf488c
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8bf488c636
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
8bf488c to
e067803
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e067803fb4
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
e067803 to
a63d396
Compare
There was a problem hiding this comment.
💡 Codex Review
lab/services/agents/src/server/agents-shell/http.ts
Lines 321 to 322 in a63d396
During Kubernetes termination, installing these signal listeners suppresses Bun's default exit behavior, but the callbacks only shut down the runner and never call server.stop() or exit. I reproduced this with the checked Bun runtime: a Bun.serve process with the same process.once('SIGTERM', ...) pattern remained alive and continued serving requests after SIGTERM. Consequently every agents-shell rollout waits for the chart's 30-second termination grace period and ends in SIGKILL; the shutdown handler should also stop the HTTP server and let the process terminate.
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
a63d396 to
732c927
Compare
|
Published on exact-main head |
732c927 to
b57e16d
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b57e16dacc
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
192945a to
4936691
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 493669115f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
4936691 to
99413e1
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 99413e127f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
99413e1 to
16603fb
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 16603fb964
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| job.timeout = setTimeout(() => { | ||
| if (job.status !== 'running') return | ||
| job.timedOut = true | ||
| job.status = 'timed_out' | ||
| job.signal = 'SIGTERM' | ||
| this.killProcessGroup(job, 'SIGTERM') | ||
| }, input.timeoutSeconds * 1000) |
There was a problem hiding this comment.
Sweep the lease UID when shell jobs time out
When a shell_start command ignores SIGTERM or moves a descendant into another process group with setsid, this timeout changes the job to timed_out but kills only the original process group. The surviving process can continue modifying the leased workspace, and because runningJobs() counts only running jobs, repeated timeouts can also bypass maxConcurrentJobs; use the same UID-wide sweep and stream cleanup already used by synchronous mutating-tool timeouts.
Useful? React with 👍 / 👎.
| entry.expiryTimeout = setTimeout( | ||
| () => | ||
| void withSessionDispatch(entry, () => closeSession(entry, 'access_token_expired', entry.issuedAuth)).catch( | ||
| () => undefined, | ||
| ), | ||
| Math.max(1, auth.payload.exp * 1000 - Date.now()), |
There was a problem hiding this comment.
Discard stale token-expiry callbacks after refresh
When one stateful request holds dispatchTail, a same-subject refresh request can authenticate and queue before the old token expires, after which the old timer fires and queues this unconditional close behind the refresh. The refresh then schedules the new deadline, but the already-queued callback still runs next and revokes the valid session and its renewed lease; capture the scheduled expiry or a generation and recheck it when the callback reaches the dispatch queue.
Useful? React with 👍 / 👎.
Summary
refs/remotes/origin/mainbefore resolving new lease bases; reject config includes and neutralize executable helpersgit greppager form before execution/nonexistentHOME plus/dev/nullkubeconfig with in-cluster discovery, and grant explicit read/traverse access to the root-owned seedread_file, reject anonymous stateful allocation, keep lease expiry manager-owned after parent exit, package the native helper in Docker/Nix, and atomically enable the read-only seed mount during normal compatible-image promotionRelated Issues
Resolves PROOMPT-440.
Testing
Exact source head
8bf488c6367f32562936c441f596def0009245b6, one commit across exactly 40 files on baseae4d23650c20cecbde2bac8416bc2b734381cb69.refs/remotes/origin/main--localinspection is explicitly anchored withgit -Cto the validated repository; the production proof passes the acquired workspace path to owned read-only Gitkubectl -n "${namespace}" apply -f -for the production-isolation Pod.gitrepository exists, and performs owned read-only Git through that acquired path rather than an adopted fixture pathconfig/gitconfigwith symlinks to protected targets; descriptor-boundO_NOFOLLOWdirectory/file opens reject both, whilefchown/fchmodleave target bytes, ownership, and mode unchangedCAP_FOWNERis absent, prepares the lease runtime twice, and verifies every runtime directory remains UID/GID 200000 mode 0700 andgitconfigmode 0600; mode normalization occurs before ownership transfer and final metadata is re-read from the same descriptorcore.worktreeexfiltration regression poisons an acquired repository, proves rawgit diffreads the external root, then proves server-owned canonical--work-treebinding prevents the read for privileged and confined Gitcustom-columns-file,go-template-file, andjsonpath-fileacross short, long, equals, and attached output forms before trusted kubectl startscore.fsmonitordemonstrably exfiltratesGIT_TOKENbefore the fix; bootstrap chowns first, atomically replaces local config with non-executable server-owned settings, succeeds withoutCAP_FOWNER, and never executes the markergit diff --name-onlyandgit status --shortto report each through explicit canonical--git-dirplus--work-treebinding/workspace/.agents-shelland its contents as UID/GID 1000, proves bootstrap migrates them to root withoutCAP_FOWNER, rejects root/nested symlinks and hard-linked files, and verifies every server directory is contained, server-owned, mode-normalized, and scratch-cleaned at restartshell_startloop rewritingfilter.racing.clean, prove inspection is rejected while the writer exists and new mutation is blocked during scan-to-spawn, then prove quiescent Git reports both tracked and untracked changes without invoking the helperfeature/selectedbranch, rejects ambiguous non-origin refs, fetches the exact selected origin refspec with server-owned auth, and checks out the expected commit; exact object IDs remain allowed only when presentgit diff, untrackedgit status, private scratch cleanup, legacy control migration, absentCAP_FOWNER, and the packaged bootstrap exploit defenseupdate-index --really-refreshbefore the caller command, enables optional locks only for that private scratch, and records deterministic refresh-before-command audit ordering; the Nix proof requires nonzero captured status bytes and emits the full structured result on failure/dev/nullread-write failure while preserving zero writes outside scratch/etcand traversal-ID regressions prove privileged recovery never runs or mutates the protected targetGIT_DIR=/dev/nullprobe uses the approved URL directly, disables generic and GitHub-scoped proxies, forces TLS verification, carries only the server-owned header, and deletes only when the advertised branch head exactly equals localHEAD; the regression plants hostile proxy/TLS config in the lease and proves it is absent from the probe and token/audit outputO_NOFOLLOWdirectory-FD walk from the validated seed/lease root; the native Landlock helper requires the inherited FD, verifies it is a directory,fchdirs before confinement/UID drop, and never re-resolves the caller path. One regression swaps a parent to a projected-secret symlink before FD acquisition and proves fail-closed; another swaps the final path after acquisition and proves the process remains on the pinned safe inode; the real native test repeats the rename/symlink attack through compiledfchdirO_NOFOLLOW, type/link/UID/GID/mode verified, never lease-controlled, and removed after each inspection. The real Nix proof requires lease-freegit ls-files -- README.mdto returnREADME.mdwhileread_filecannot expose the projected service-account token.gitlinks are deleted before regular working bytes are preserved outside the control tree. The packaged upgrade proof runs both cases and a second bootstrap restart for each, proving no preserved symlink can poison future initializationrenewedAt/expiresAtare persisted and required-audited, the prior expiry timer is replaced, and any audit failure revokes fail-closed. A real job sleeps beyond the original token expiry and completes only after a later-token renewal; durable state and audit evidence remain active at the renewed deadlinesafe.directory, so the real Nix proof reaches and demonstrates the intendedFETCH_HEADsymlink overwrite before the shipped bootstrap rejects symlink and hard-link variants.gitsymlinks or hard-linked files before any root Git command; the packaged proof first demonstrates rawfetchoverwrites a symlink target, then proves the exact chart blocks both symlink and hard-link variants without changing the target before a clean upgrade succeedscore.alternateRefsCommandis neutralized in both static and dynamically discovered privileged/read-only Git configs; a real alternate object database regression proves rawgit log --alternate-refsexecutes a helper that reads a projected token, while the accepted tool path returns no token and never invokes the helpergpg.<format>.programverifier commands are neutralized; a synthetic signed-commit regression proves rawgit log --show-signatureexecutes an OpenPGP helper that reads a projected token, while accepted inspection returns no token and never invokes the helper-n "${namespace}"on namespace create, pod apply/log/get, and namespace cleanupumask 077to022, appliesa+rX, and the in-image proof requires lease-free UID 65534 seed search and read-only Git successrequired=true, external diff/textconv, fsmonitor, pager, and interactive filter; inspection succeeds and no helper marker is createdgit diff --submodule=diffexecutes a child clean filter; the read-only policy rejects all--submodule,--recurse-submodules, and caller--ignore-submodulesforms while forcingsubmodule.recurse=false, short submodule rendering, disabled summaries, and--ignore-submodules=all-s, and impersonation override is rejected before the trusted kubectl process startsHOME=/nonexistent,KUBECONFIG=/dev/null, retain only in-cluster discovery, and never execute the plugin or receive GitHub secretsseedReadOnly, and writes root plus exactly CHOWN/DAC_OVERRIDE/KILL/SETGID/SETUIDBreaking Changes
Agents-shell write tools require an authenticated stateful MCP session and active workspace lease. Stateful and read-only access to a leased workspace remains bound to that lease subject and UID. Stateless reads remain limited to the shared read-only seed.
Checklist