Skip to content

Deferred values + @readStdin: the value-returning half of colorless implicit futures - #149

Merged
assapir merged 17 commits into
mainfrom
concurrency-deferred-value-read
Aug 21, 2026
Merged

Deferred values + @readStdin: the value-returning half of colorless implicit futures#149
assapir merged 17 commits into
mainfrom
concurrency-deferred-value-read

Conversation

@assapir

@assapir assapir commented Aug 21, 2026

Copy link
Copy Markdown
Owner

Deferred values + @readStdin — the value-returning half of colorless implicit futures

Fixes #151. Builds on slice-1 (@, @sleep, now()). Adds the first value-returning @ primitive, @readStdin, plus the deferred-value machinery and a generic deferred-value runtime core. The type checker stays byte-identical (no Task/Future type — forcing keys off the operation, never the type; git diff under src/typechecker/ is empty).

What it does

@readStdin() -> Text (in core.io) reads one line from stdin. Calling it launches the read on a background fiber and returns a deferred Text immediately (the caller does not wait). The value threads lazily through bindings / records / calls and is forced on use — the fiber parks until the bytes are ready — only where a strict primitive reads them. At end-of-input it yields "".

<< core.io
<< core.test
^ = () -> Num => <
  line = @readStdin()       ~ launches; returns a deferred Text (no wait)
  assertEq(line, "hello")   ~ the == FORCES it: parks until ready, reads bytes
  0
>
~ echo hello | quilon run examples/readStdin.ql

How it's built

  • Generic runtime core (quilon-rt/src/deferred.rs, quilon-rt: generalize the deferred-value launch/promise core (FnOnce producer), not stdin-specific #151): launch<T>(producer) -> *mut Promise<T> spawns a fiber that runs producer (which parks however it needs) and resolves the cell when it returns; force<T: Copy>(cell) -> T is the memoized park-until-ready wait. Promise<T> is GC-allocated so a GC pointer inside T stays scannable. @readStdin is a thin wrapper: __read_launch = launch(|| read_stdin_text(site)), __force_text = force::<QlSlice>. A future value-returning @ primitive (file/socket/HTTP) reuses launch with a different producer — no new park/promise plumbing. The C-ABI boundary stays per-primitive *_launch + per-representation force_* wrappers; only the Rust core is generic.
  • Single-reader stdin gate: stdin is one serial stream, so reads are serialized (acquire/release via the scheduler's address-wait) — two concurrent @readStdin launches read consecutive lines instead of racing fd 0 (which would double-register the fd → EEXIST → crash) or corrupting the shared line buffer. The scheduler's Park::Waiting/park_on_address/wake_address is the one rendezvous serving both the value force and the gate. (Correctness-review finding.)
  • Hybrid rep, zero-overhead for pure code: a Text is {ptr,i64}; a deferred Text is {promise, -1} (a real byte length is never negative → unambiguous sentinel). Only tainted values carry it.
  • Deferred-taint pass (src/deferral.rs): forward dataflow computing the force-set — the strict slots where codegen must force (operands, ?/match scrutinee, print/write/native args incl. the callee expression, indexing, field/method receiver, array/record construction, interpolation holes, function/method/lambda body result). Lazy carriers = = bindings and ?/ternary/match arms. Reads the type oracle, adds no types.
  • Codegen (exprs.rs force_deferred_text + a one-line generate_expr seam; calls.rs @readStdin lowering; intrinsics.rs prototypes): forces only at taint-marked spans → pure programs byte-identical.
  • Scope join = run-to-completion (allSettled): the scheduler drives every launched read to completion before the program exits.
  • Origin site: each @readStdin carries its path:line:col so a read fault reports where the IO was called.
  • Corelib @-declaration fix: quilon check corelib/io.ql / corelib/time.ql no longer false-errors — the front-end trusts a bundled corelib source (matched by content) to declare @ primitives, while user code declaring one is still rejected.

Tests / proof

  • tests/read_stdin_test.rs — pipes stdin to JIT + AOT: matching input exits 0, non-matching exits 101 (the real value flowed, not a constant); force at a print arg; and two reads serialize into consecutive lines (hello, world).
  • quilon-rt deferred tests: launch → park-on-readiness → produce → force over a pipe (delayed writer); force memoization; the stdin gate serialization (at most one reader at a time); line splitting.
  • src/deferral.rs: 11 taint-pass tests. Driver/modules: corelib @-decl allowed, user @-decl rejected.
  • examples/readStdin.ql — a normal runnable example: with no piped input @readStdin() hits EOF and returns "", so it asserts line == "" and exits 0 in the gate (which feeds it /dev/null stdin so it never blocks). Pipe it a line to watch a value flow.

Deferred to a follow-up (documented)

Cross-source overlap as a showcase (independent reads finishing in max-time) arrives with a networked primitive (@get) — the generic launch is ready for it. Cross-function promise pipelining (a function returning a deferred value) is later; here a promise is forced at any escape (call arg / body result / construction), so it stays within its function body.

Gate: cargo fmt --all --check, cargo clippy --all-targets --all-features -D warnings, and the full suite under RUSTFLAGS=-D warnings (36 test binaries) all green. Checker byte-identical. Rebased on latest main.

Parked — do not merge without explicit approval.

assapir and others added 16 commits August 21, 2026 10:57
Add the runtime half of colorless implicit futures for value-returning @
primitives: a GC-allocated Promise cell, __read_launch (eager background
stdin line read, returns a deferred Text {promise, -1}), and __force_text
(park-until-ready, memoized). Scheduler gains Park::Promise + park_on_promise
+ wake_promise so a forcing fiber waits on a producing fiber. Stdin reader
parks on reactor readiness (SourceFd, os-ext mio feature).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Prove launch/park-on-readiness/produce/force end to end (delayed writer, so
the reader really parks), force memoization, and line splitting. Reader is now
generic over an fd (read_line_from) so a pipe drives it in tests; stdin is the
fd-0 wrapper. 28 rt tests green, clippy clean.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
deferral.rs now computes the force-set: a forward dataflow colors which exprs
may hold a deferred Text (@READ through lazy carriers: = bindings, ?/ternary
arms, block/match results) and marks the strict slots where codegen must force
(operands, call args, print/write, indexing, construction, interp holes, body
result). Only tainted spans get forces -> pure code byte-identical. read_call_sites
maps each @READ to path:line:col (origin, filled by the driver). Adds core.io
@READ decl (() -> Text) and the __read_launch/__force_text LLVM prototypes.
11 taint tests green. Codegen force-on-use lands next.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
generate_at_primitive gains the read arm: builds the path:line:col launch-site
arg (read_launch_site) and calls __read_launch, yielding the deferred Text
{promise, -1} without dereferencing it. Force-on-use wrapping lands next.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
generate_expr now defers to generate_expr_inner then, at a taint-marked force
site, calls force_deferred_text: a runtime sentinel check (len == -1) that parks
via __force_text and reads the bytes, else passes the ready Text through. Pure
programs hit no force site, so their codegen is byte-identical.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The 'user code cannot declare an @ primitive' rule fired when checking a corelib
file itself (quilon check corelib/time.ql), a false error the editor surfaced.
The front-end now trusts a file whose content IS a bundled corelib source
(modules::is_corelib_source, matched by content so it is path-independent and
never mistakes user code) and skips the rejection there; ordinary user code is
still rejected. Tests: corelib @-decls check clean; a user @-decl is rejected.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
read_stdin_test spawns the compiler with a controlled stdin (the examples gate
pipes none): a deferred @READ Text bound then compared in assertEq forces at the
comparison — matching input exits 0, non-matching exits 101 (proving the real
read value flowed through, not a constant). Also proves force at a print arg
(echo), and AOT parity when a linker is present.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A self-asserting @READ example: binds a deferred Text, forces it at the assertEq
comparison. It needs stdin, so the no-input examples gate compiles it but skips
the auto-run (new NEEDS_STDIN list); its runtime is proven by read_stdin_test.
Run as documented: echo hello | quilon run examples/deferred_read.ql.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A pre-existing untracked scratch crate accidentally swept into an early WIP
commit by git add -A; it is not a workspace member and not part of this change.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…nsic-link gate

read_line_from now reads first and only registers fd + parks on WouldBlock, so a
ready or non-pollable source (piped data, a redirected file, /dev/null) never hits
epoll (which rejects such fds) — only a not-ready pollable source is parked on.
The every-intrinsic smoke program now reaches __read_launch/__force_text (via
@READ + .length), and its run paths feed empty stdin so @READ yields "" at EOF.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Document @READ as the value-returning deferred primitive: force-on-use, deferred
Text threads lazily, EOF -> "", type-invisible (checker unchanged). Updates the
core.io table, the concurrency section (@READ runnable; overlap still headed via
@get), the feature matrix, and a CHANGELOG entry (cites #120). Leaves the @sleep
example reference untouched (renamed by a separate PR).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Match the sleep.ql naming (bare verb). Updates the NEEDS_STDIN gate list, the
example header, and the CHANGELOG/LANGUAGE.md references.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…r core)

- Rename the Quilon-facing primitive read -> readStdin (stdin-specific): corelib
  decl @readStdin, codegen interception, taint READ_PRIMITIVE. Runtime intrinsic
  keeps __read_launch/__force_text internally.
- Serialize stdin (correctness review, HIGH): a single-reader gate (acquire/release
  via the scheduler's address-wait) so concurrent @readStdin launches read
  consecutive lines instead of racing fd 0 (two register-fd-0 -> EEXIST -> crash) or
  corrupting the shared line buffer. Generalized Park::Promise -> Park::Waiting /
  park_on_address / wake_address (serves both the value force and the gate).
- Taint: recurse into Call.func too (latent soundness fix), and drop the test-only
   set from DeferInfo (force_sites is the whole codegen surface).
- gate serialization unit test; doc corrected (no nonexistent GC test claim).

Example/tests/docs rename follows in the next commit.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… example

- examples/read.ql: normal runnable example — line = @readStdin(); assertEq(line, "");
  on end-of-input @readStdin returns "". Dropped the NEEDS_STDIN special-case; the
  examples gate now redirects fd 0 to /dev/null (in-process) and pipes null stdin to
  the JIT/AOT subprocesses, so a stdin-reading example gets instant EOF and never hangs.
- read_stdin_test: renamed sources to @readStdin; added a two-read case proving the
  stdin gate serializes concurrent launches into consecutive lines (hello, world).
- Renamed @READ -> @readStdin in intrinsic_link_test, docs/LANGUAGE.md, CHANGELOG,
  and the stale sleep.ql/driver.rs comments.
- Collapsed a let-and-return in the taint visit (clippy).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…le to readStdin.ql

- quilon-rt: factor a GENERIC launch<T>(producer) -> *mut Promise<T> and
  force<T: Copy>(cell) -> T. Promise<T> holds Option<T>, GC-allocated so a GC pointer
  inside T stays scannable. @readStdin is now a thin wrapper: __read_launch calls
  launch(|| read_stdin_text(site)) and __force_text is force::<QlSlice>. The stdin
  producer (gate + line read + fault-with-site) is one FnOnce; a future value-returning
  @ primitive reuses launch with a different producer and its own per-rep force wrapper.
  The C-ABI boundary and language surface are unchanged (behavior-preserving); the launch
  site moved from the cell into the producer closure. Park::Waiting/park_on_address/
  wake_address already generic. Test helper now exercises launch too.
- Rename examples/read.ql -> examples/readStdin.ql (matches the primitive); updated the
  header and the CHANGELOG/LANGUAGE.md references.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@assapir assapir changed the title Deferred values + @read: the value-returning half of colorless implicit futures Deferred values + @readStdin: the value-returning half of colorless implicit futures Aug 21, 2026
…write debug guards

Behavior-preserving internal cleanup of the deferred-value core (quilon-rt):
- Replace the PENDING/READY consts with enum DeferredState<T> { Pending, Ready(T) },
  carrying the resolved value INSIDE Ready so "ready but value absent" is
  unrepresentable. force<T: Copy> matches Ready(value) and copies it out (memoized);
  the value still lives in the GC-scanned cell, so GC-visibility is unchanged.
- Rename the runtime type Promise<T> -> Deferred<T> (and locals/docs) to match our
  colorless vocabulary; JS-colored "Promise" is gone. DEFERRED_SENTINEL stays a const
  (the ABI {ptr,len} tag, not a lifecycle state). C-ABI intrinsic names and the Quilon
  surface are unchanged.
- Add debug_assert guards around each unsafe cell write: the alloc/init path asserts a
  fresh non-null zeroed cell; the resolve path asserts state is still Pending
  ("resolving an already-resolved Deferred"). debug_assert, so release is unaffected;
  seeds the real check for the future M:N runtime.

@readStdin identical (match 0, mismatch 101, EOF "" 0); checker byte-identical; full
suite green under -D warnings.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@assapir
assapir merged commit 5ce6f75 into main Aug 21, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

quilon-rt: generalize the deferred-value launch/promise core (FnOnce producer), not stdin-specific

1 participant