Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 13 additions & 2 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -214,21 +214,32 @@ jobs:

- uses: dtolnay/rust-toolchain@stable

- name: Host lifecycle and closure proofs
- name: Host lifecycle and closure proofs (Linux)
if: runner.os == 'Linux'
run: |
cargo test -p mc-host \
--test broca_protocol \
--test broca_subprocess \
--test harness_closure \
--test protocol_vectors

- name: Native module adapter and CLI lifecycle
- name: Native module adapter and CLI lifecycle (Linux)
if: runner.os == 'Linux'
run: |
cargo test -p mc-module --bin ck-mc-host
cargo test -p mc-module \
--test host_adapter \
--test lifecycle_cli

# The release ships darwin payloads and the GA evidence gate requires
# darwin target proofs, so the native lifecycle binary and its CLI
# contract need macOS proof alongside the Linux integration set.
- name: Native lifecycle binary and CLI contract (macOS)
if: runner.os == 'macOS'
run: |
cargo build -p mc-module --bin ck-mc-host
cargo test -p mc-module --test lifecycle_cli

check-plugin:
name: Check (plugin)
runs-on: ubuntu-latest
Expand Down
26 changes: 25 additions & 1 deletion crates/mc-module/src/bin/ck_mc_host/spawn.rs
Original file line number Diff line number Diff line change
Expand Up @@ -94,18 +94,42 @@ pub fn spawn_detached(
};

let mut pipe_fds = [0 as libc::c_int; 2];
// Linux creates the pipe already close-on-exec. Darwin has no `pipe2`, so
// there the flag is applied in a second step below.
#[cfg(target_os = "linux")]
// SAFETY: pipe2 writes exactly two descriptors into the array.
cvt(
unsafe { libc::pipe2(pipe_fds.as_mut_ptr(), libc::O_CLOEXEC) },
"envelope pipe creation failed",
)?;
// SAFETY: the descriptors were just returned by pipe2 and are owned here.
#[cfg(target_os = "macos")]
// SAFETY: pipe writes exactly two descriptors into the array.
cvt(
unsafe { libc::pipe(pipe_fds.as_mut_ptr()) },
"envelope pipe creation failed",
Comment on lines +105 to +109

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Preserve retained descriptors before remapping standard fds

When ck-mc-host starts on macOS with any standard descriptor closed, resolve_generation_launcher or open_log can reuse fd 0, 1, or 2 before this newly enabled pipe path runs. The child remaps stdin/stdout/stderr before copying the retained executable to fd 3, so that remapping can overwrite the launcher descriptor; a same-fd dup2 also leaves the newly applied FD_CLOEXEC flag intact. Service managers that launch the CLI with closed stdio can therefore make daemon startup time out or lose its log, so duplicate all retained descriptors above the standard-fd range before performing the child remaps.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I implemented this one before I could defend it, then reverted it. The reasoning about the child is correct, but the premise it rests on cannot hold in this binary, so the fix was guarding a state that does not occur.

Everything you say about the child is accurate, and I want to be clear that I checked it rather than waving it off:

  • Nothing in this repository sanitizes stdio. There is no /dev/null guard and no F_GETFD sweep anywhere under crates/mc-module/src/bin/ck_mc_host/.
  • The ordering hazard is real if a retained descriptor is low. dup2(log_fd, 1) would overwrite a launcher at fd 1, and the later dup2(exe_fd, 3) would then pin the log, leaving fexecve(3) pointed at the log file.
  • Your close-on-exec point is the sharper half and it is also correct. A same-fd dup2 succeeds as a no-op and does not clear FD_CLOEXEC, so a read end at fd 0 or a log at fd 1 would be closed at exec rather than inherited. Worth noting this half is not Darwin-specific: Linux sets O_CLOEXEC at creation via pipe2, and open_log passes O_CLOEXEC on both platforms, so a same-fd dup2 would strand the descriptor there too.

Where it breaks down is the first step: ck-mc-host cannot reach resolve_generation_launcher or open_log with fd 0, 1, or 2 free, because Rust's standard library reopens closed standard descriptors before main runs. std::sys::pal::unix::init calls sanitize_standard_fds(), which points any closed standard descriptor at /dev/null, and aborts the process if it cannot.

Measured on this toolchain (1.98.0), launching a Rust binary from a parent that closed fd 0 with exec 0<&-:

open fds at main(): ["0->/dev/null", "1->socket:[...]", "2->socket:[...]", "3->/proc/.../fd"]
first open() got fd 3

The same parent running ls instead shows 0 -> /proc/<pid>/fd, so fd 0 was genuinely free — the difference is the Rust runtime, not the launcher.

Since you scoped this to macOS, that is the case I checked most carefully in the std source rather than assuming it generalizes. The poll fast path is in fact excluded there:

// The poll on Darwin doesn't set POLLNVAL for closed fds.
target_vendor = "apple",

But the fcntl fallback immediately below it is not excluded for Darwin — its list is emscripten, fuchsia, vxworks, l4re, horizon, vita — and it does the same repair:

for fd in 0..3 {
    if libc::fcntl(fd, libc::F_GETFD) == -1 && errno() == libc::EBADF {
        open_devnull();
    }
}

So on darwin-arm64 and darwin-x64, the two targets this PR adds to CI, a closed standard descriptor is reopened before any of this code runs.

I also checked the one route that would bypass startup repair, a descriptor freed later at runtime. The only stdin use in the binary is std::io::stdin().lock() in read_envelope and read_launcher_envelope, which borrows the shared handle and never closes fd 0. Nothing else touches fds 0-2.

For completeness on what I discarded: I had move_above_stdio hoisting the log, launcher, and pipe read end above the standard range with F_DUPFD_CLOEXEC, plus a lifecycle_cli test that stages a real launcher and runs start from a child with fd 0 closed. That test is what settled it — it passes with the hoist removed, because the launcher lands at fd 3 either way. I would rather carry no test than one that cannot fail, and adding unsafe and a new SpawnError path to guard an unreachable state is the kind of defensive code this repository asks me not to accrete.

Happy to reconsider if you can point at a path where a standard descriptor is free after main — that would make this reachable and I would take the hoist back.

)?;
// SAFETY: the descriptors were just returned by pipe2/pipe and are owned here.
let (pipe_r, pipe_w) = unsafe {
(
OwnedFd::from_raw_fd(pipe_fds[0]),
OwnedFd::from_raw_fd(pipe_fds[1]),
)
};
// Both ends are owned before the flag is set, so a failure here closes them
// instead of leaking a descriptor pair. Unlike `pipe2` this is not atomic
// with creation: a concurrent exec in another thread could inherit the ends
// in that window. The child below is the only exec this binary performs, it
// happens after this point, and it keeps the read end deliberately by
// dup2-ing it onto stdin (which clears close-on-exec) while closing every
// descriptor above 3.
#[cfg(target_os = "macos")]
for fd in [pipe_r.as_raw_fd(), pipe_w.as_raw_fd()] {
// SAFETY: fd is owned by pipe_r/pipe_w and open for this call.
cvt(
unsafe { libc::fcntl(fd, libc::F_SETFD, libc::FD_CLOEXEC) },
"envelope pipe cloexec failed",
)?;
}

// Everything the child touches is prepared before fork: with tokio
// worker threads alive, the child may only use async-signal-safe calls
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -1721,6 +1721,102 @@ describe("embedItemsDetailed", () => {
}
});

it("reports a page cancelled when the abort lands inside its re-validation", async () => {
const db = ledgerDb();
try {
const host = new DetailedHost();
const controller = new AbortController();
let demands = 0;
const provider = new SynapseEmbeddingProvider({
connectionFile: "fixture",
projectRoot: "/repo",
session: "ses-1",
model: MODEL,
fingerprint: FP,
tableEpoch: 0,
dims: 3,
recommendedBatch: 2,
batchTimeoutMs: 5_000,
clientFactory: async () => host,
demandStart: async () => {
demands += 1;
// The abort lands while the managed demand is in flight, so
// `initialize` observes it on its own await and reports the
// plain `false` a rejected `raceSignal` is folded into.
controller.abort();
await new Promise((resolve) => setTimeout(resolve, 0));
return {
ok: true,
reason: "started",
storage: "ready",
authenticatedDaemonId: new Uint8Array([7, 7]),
};
},
});
// Certify the lane before installing the managed origin: `initialize`
// is the only writer of the identity, and the pre-loop initialize
// must return from the already-certified state so the first page
// dispatches instead of demanding.
expect(await provider.initialize()).toBe(true);
const internals = provider as unknown as {
connectionOrigin: string;
compatibleDaemonId: Uint8Array | null;
initialized: boolean;
};
internals.connectionOrigin = "managed-default";
internals.compatibleDaemonId = new Uint8Array([7, 7]);

// Reproduce the state a rotation on an earlier page installs: the
// lane is managed and no longer certified, which is precisely the
// precondition the per-page re-validation exists to answer. Doing it
// from the first page's own response keeps the second page's
// `signal.aborted` check ahead of the abort, so the abort can only
// be observed inside the re-validation itself.
host.resultPages = (_jobId, items) => {
if (items.some((item) => item.id === "memory:1")) internals.initialized = false;
return {
result: {
...ENVELOPE,
done: true,
vectors: items.map((item) => ({
id: item.id,
content_sha256: item.content_sha256,
vector: [1, 2, 3],
})),
},
};
};

const result = await provider.embedItemsDetailed(
detailedItems([
{ id: "memory:1", group: "g1" },
{ id: "memory:2", group: "g2" },
]),
detailedContext(db),
controller.signal,
);

// The first page completed before the identity was invalidated.
expect(result.receipts).toHaveLength(1);
expect(result.receipts[0].applicationGroup).toBe("g1");
// Exactly one demand: the second page's re-validation.
expect(demands).toBe(1);
expect(result.failures).toHaveLength(1);
const g2 = result.failures[0];
expect(g2.applicationGroup).toBe("g2");
// This read `transport`/`retryable` before the signal was re-checked
// after initialization, which invites a retry of a request the caller
// withdrew and disagrees with the `cancelled` every later page reports.
expect(g2.code).toBe("cancelled");
expect(g2.message).toBe("Synapse request aborted");
expect(g2.disposition).toBe("retryable");
// The cancelled page must never have reached the wire.
expect(host.batchCalls()).toHaveLength(1);
} finally {
closeQuietly(db);
}
});

it("scopes an exhausted restart budget to its page and leaves sibling pages runnable", async () => {
const db = ledgerDb();
try {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -895,6 +895,11 @@ export class SynapseEmbeddingProvider implements EmbeddingProvider {
}
for (let start = 0; start < items.length; ) {
if (signal?.aborted || this.permanentFailure) break;
// A `module_restarted` failure on an earlier page invalidated the
// compatible daemon identity. Re-run the full initialization so
// the remaining pages only proceed against an incarnation that
// re-passed lifecycle compatibility validation.
if (!this.initialized && !(await this.initialize(signal))) break;
const page = this.nextPage(items, start);
start += page.length;
try {
Expand Down Expand Up @@ -1024,6 +1029,35 @@ export class SynapseEmbeddingProvider implements EmbeddingProvider {
});
continue;
}
// A `module_restarted` failure on an earlier page invalidated
// the compatible daemon identity; re-validate before this page
// so it never rides an unverified incarnation.
if (!this.initialized && !(await this.initialize(signal))) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Preserve cancellation while revalidating a page

When the caller's AbortSignal fires while this newly added revalidation is awaiting initialize(signal), initialize catches the aborted raceSignal and returns false. This branch then records the current page as a retryable transport failure even though the immediately preceding check classified the same signal as cancelled; later pages are also reported as cancelled. Check the signal again after initialization so an aborted multi-page request does not misreport one page as an infrastructure failure and invite an unintended retry.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed and fixed in 348a1f84. Your read of the mechanism is exactly right, including the detail that makes it easy to miss: the abort arrives as a return value, not as an exception.

What I verified before changing anything:

  • initialize swallows the abort. The tail is try { return await raceSignal(initialization, signal); } catch { return false; }, so a raceSignal rejection from the abort listener is folded into a plain false with no way for the caller to tell it apart from a real initialization failure.
  • The managed path folds it the same way one level up: const outcome = signal ? await raceSignal(demand, signal) : await demand; sits inside a try whose catch logs and returns false.
  • The disagreement you describe is real. The branch hard-coded code: "transport" and disposition: "retryable", while the check at the top of the same loop classifies the identical signal as cancelled. So one page of a cancelled multi-page request was recorded as an infrastructure failure and the rest as cancelled.

The signal is now re-read after initialization, with permanentFailure keeping precedence so the branch matches the ordering of the check above it:

const aborted = signal?.aborted === true;

The consequence you flagged is the one I care about most: retryable on a withdrawn request invites a retry of work the caller already abandoned.

On coverage, the test is reports a page cancelled when the abort lands inside its re-validation, and I confirmed it is not vacuous — with the aborted read forced to false it fails with Expected: "cancelled" / Received: "transport", which is the exact symptom you reported.

Getting it to land in that branch took some care, and the reason is worth recording. In the injected-client shape the branch is reachable but not abortable: getSharedClient memoizes per factory, and the constructor pre-seeds metadata from fingerprint/tableEpoch/dims, so a second initialize has no suspension point and returns true before any signal can fire. The managed path does have a real await, but recertifyForRestart re-initializes eagerly, so a page that restarts leaves initialized true again for the next page. The test therefore installs the branch's documented precondition directly — managed origin, identity cleared from the first page's own response — the same way the existing tests in this file install post-rotation state through internals. That keeps the second page's signal.aborted check ahead of the abort, so the abort can only be observed inside the re-validation itself. It asserts one demand, one receipt for the first group, and that the cancelled page never reached the wire.

I left embedItems alone. It has the same !this.initialized && !(await this.initialize(signal)) shape, but it breaks without recording a failure, so there is no classification to get wrong.

// `initialize` reports an abort raised during its own await
// as a plain `false`, so the signal is re-read here. Without
// it a caller-cancelled request records this one page as a
// retryable `transport` failure — inviting a retry of work
// the caller withdrew — while every later page correctly
// reports `cancelled` from the check above.
const aborted = signal?.aborted === true;
result.failures.push({
applicationGroup,
items: manifest,
rowId: null,
code: this.permanentFailure
? "artifact_invalid"
: aborted
? "cancelled"
: "transport",
message: this.permanentFailure
? "Synapse lane disabled after a permanent failure"
: aborted
? "Synapse request aborted"
: "Synapse lane is unavailable",
disposition: this.permanentFailure ? "permanent" : "retryable",
});
continue;
}
try {
result.receipts.push(
await this.runDetailedPage(page, applicationGroup, context, signal),
Expand Down
20 changes: 20 additions & 0 deletions packages/plugin/src/shared/mc-host-lifecycle/compatibility.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -137,4 +137,24 @@ describe("semver parsing", () => {
expect(parseSemverTriple(bad)).toBeNull();
}
});

test("leading zeroes are rejected rather than normalized", () => {
expect(parseSemverTriple("0.1.0")).toEqual([0, 1, 0]);
// Each of these would parse to an in-range triple under `\d+`, so the
// range gate would accept a non-canonical version.
for (const bad of ["00.1.0", "0.01.0", "0.1.00", "00.01.000", "01.2.3"]) {
expect(parseSemverTriple(bad)).toBeNull();
}
});

test("a non-canonical daemon version fails the compatibility gate", () => {
// `00.01.000` normalizes to `[0, 1, 0]`, which is inside the supported
// half-open range, so only canonical-form rejection keeps this closed.
const verdict = evaluateDaemonCompatibility("mc-host/00.01.000");
expect(verdict.ok).toBe(false);
if (!verdict.ok) {
expect(verdict.reason).toBe("incompatible_daemon");
expect(verdict.detail).toBe("daemon version is not a canonical mc-host/X.Y.Z value");
}
});
});
69 changes: 58 additions & 11 deletions packages/plugin/src/shared/mc-host-lifecycle/compatibility.ts
Original file line number Diff line number Diff line change
Expand Up @@ -24,8 +24,17 @@ export type CompatibilityVerdict =

type SemverTriple = [number, number, number];

/**
* Each part is a semver numeric identifier: a single `0`, or a non-zero digit
* followed by any digits. `\d+` would also admit leading zeroes, which
* `Number.parseInt` then silently normalizes — `00.01.000` would parse to
* `[0, 1, 0]` and pass the range gate, so a non-canonical peer version would
* satisfy a check whose verdict promises a canonical `X.Y.Z` value.
*/
const CANONICAL_SEMVER = /^(0|[1-9]\d*)\.(0|[1-9]\d*)\.(0|[1-9]\d*)$/;

export function parseSemverTriple(value: string): SemverTriple | null {
const match = /^(\d+)\.(\d+)\.(\d+)$/.exec(value);
const match = CANONICAL_SEMVER.exec(value);
if (!match) return null;
const triple: SemverTriple = [
Number.parseInt(match[1] as string, 10),
Expand Down Expand Up @@ -182,19 +191,57 @@ export function evaluateEpochCompatibility(observed: ObservedEpochs): Compatibil
return { ok: true };
}

export interface CompatibilityInput {
authenticatedDaemonVer: string;
catalog: CatalogEntry[];
epochs: ObservedEpochs;
}

/**
* The single ordered source of truth for the compatibility gate: stage id,
* the CLI check id it reports under, and its evaluator. `evaluateCompatibility`,
* the managed probe's `evaluatedThrough` labels, and the policy's emitted
* `compatibility.*` checks all derive from this list, so a stage added or
* reordered in one place cannot leave the probe sequence and the reported
* checks disagreeing.
*/
export const COMPATIBILITY_STAGES = [
{
stage: "daemon",
checkId: "compatibility.daemon",
evaluate: (input: CompatibilityInput): CompatibilityVerdict =>
evaluateDaemonCompatibility(input.authenticatedDaemonVer),
},
{
stage: "modules",
checkId: "compatibility.modules",
evaluate: (input: CompatibilityInput): CompatibilityVerdict =>
evaluateModuleCompatibility(input.catalog),
},
{
stage: "epochs",
checkId: "compatibility.epochs",
evaluate: (input: CompatibilityInput): CompatibilityVerdict =>
evaluateEpochCompatibility(input.epochs),
},
] as const;

export type CompatibilityStage = (typeof COMPATIBILITY_STAGES)[number]["stage"];

/** Position of `stage` in the ordered gate; the order is the array order. */
export function compatibilityStageIndex(stage: CompatibilityStage): number {
return COMPATIBILITY_STAGES.findIndex((entry) => entry.stage === stage);
}

/**
* The composed demand/status/doctor gate order: daemon range, then modules,
* then epochs. First failure wins and is reported without any stop, replace,
* or restart side effect (R17).
*/
export function evaluateCompatibility(input: {
authenticatedDaemonVer: string;
catalog: CatalogEntry[];
epochs: ObservedEpochs;
}): CompatibilityVerdict {
const daemon = evaluateDaemonCompatibility(input.authenticatedDaemonVer);
if (!daemon.ok) return daemon;
const modules = evaluateModuleCompatibility(input.catalog);
if (!modules.ok) return modules;
return evaluateEpochCompatibility(input.epochs);
export function evaluateCompatibility(input: CompatibilityInput): CompatibilityVerdict {
for (const stage of COMPATIBILITY_STAGES) {
const verdict = stage.evaluate(input);
if (!verdict.ok) return verdict;
}
return { ok: true };
}
Loading
Loading