monterey

Thread-per-core async runtime built on Linux io_uring.

No work-stealing. No shared run queues. No cross-core synchronization on the hot path. Each OS thread pins to a CPU and owns its own io_uring instance, executor, buffer pool, and scheduler.

Requirements

• Linux 6.7+ (uses IORING_OP_FIXED_FD_INSTALL, IORING_OP_FUTEX_WAIT, DEFER_TASKRUN, SINGLE_ISSUER)
• Rust 1.75+
• NVMe passthrough requires a character device at /dev/ngXnY

Quick Start

use monterey::{Runtime, RuntimeConfig, spawn};
use std::sync::Arc;

fn main() -> std::io::Result<()> {
    let config = RuntimeConfig {
        core_count: 2,
        init_fn: Some(Arc::new(|| {
            spawn(async {
                println!("hello from core {}", monterey::current_core_id());
            });
        })),
        ..RuntimeConfig::default()
    };
    let (_handle, join) = Runtime::start(config)?;
    join.join()
}

Architecture

monterey (runtime)          ← entry point: Runtime::start(), re-exports, cross-core APIs
├── monterey-executor       ← task storage (slab), priority scheduler, event loop, waker
│   └── monterey-reactor    ← io_uring ring setup, SQE submission, CQE dispatch, fixed fd table
│       └── monterey-buffers ← fixed buffer pool, provided buffer rings, alignment
├── monterey-fs             ← file open/read/write/fsync/splice/pipe/statx (all via ring)
├── monterey-net            ← TCP/UDP, multishot accept/recv, zero-copy send
├── monterey-nvme           ← NVMe passthrough via 128-byte SQEs (URING_CMD)
├── monterey-timer          ← sleep/timeout futures via IORING_OP_TIMEOUT
├── monterey-sync           ← mutex, rwlock, semaphore, notify, barrier + kernel futex variants
└── monterey-channel        ← bounded/unbounded/oneshot channels (intra-core, !Send)

Two-Ring Design

Each core owns up to two io_uring instances:

• Main ring (128-byte SQEs): network, timers, NVMe passthrough, cross-core messaging. Interrupt-driven via DEFER_TASKRUN.
• IOPOLL ring (optional): dedicated to O_DIRECT file I/O. Kernel-polled completions (no interrupts) for lower latency. O_DIRECT reads/writes auto-route here when available.

Key Design Decisions

• All fds are registered as direct file descriptors — no process fd table overhead, no register_files() syscall per operation.
• All I/O goes through the ring — no synchronous libc calls for operations that have io_uring opcodes.
• Everything is !Send at the core level. Cross-core work uses spawn_on() (requires Send + 'static) delivered via IORING_OP_MSG_RING.
• Two buffer tiers: small provided-ring buffers (4 KiB, network recv) and large fixed-pool buffers (128 KiB, file/NVMe I/O), registered in a single IORING_REGISTER_BUFFERS call.
• NUMA-local allocation: memory policy is bound to the local NUMA node during buffer allocation, then reset.

Features

• Priority scheduling — 4 levels (HIGH, NORMAL, LOW, IDLE) with the PriorityScheduler as default
• Direct fd installation — sockets, files, and pipes bypass the process fd table entirely
• NUMA-local buffers — fixed and provided buffers are pre-faulted on the local NUMA node
• NVMe passthrough — submit NVMe commands directly via 128-byte SQEs to /dev/ngXnY character devices
• Zero-copy networking — send_zc, recv with provided buffers, splice for proxying
• Huge page support — optional MAP_HUGETLB for buffer allocations (HugePages::Auto / HugePages::Require)
• Linked SQE chains — IoChain for atomic write→fsync→write sequences in a single submission
• Cross-core messaging — typed and untyped messages via IORING_OP_MSG_RING, no atomics
• Futex-based sync — FutexMutex and FutexCondvar (Send + Sync) using IORING_OP_FUTEX_WAIT/WAKE

Crate Map

Crate	Description
monterey	Runtime entry point, re-exports, cross-core APIs
monterey-reactor	io_uring ring setup, SQE/CQE dispatch, fixed fd table, completion slab
monterey-executor	Task storage, priority scheduler, event loop, waker integration
monterey-buffers	Fixed buffer pool, provided buffer rings, aligned allocation, huge pages
monterey-fs	File open/read/write/fsync, fallocate, splice, pipe, rename, statx
monterey-net	TCP/UDP, multishot accept/recv, zero-copy send, zero-copy RX
monterey-nvme	NVMe passthrough via URING_CMD with 128-byte SQEs
monterey-timer	sleep, sleep_until, timeout via IORING_OP_TIMEOUT
monterey-sync	Mutex, RwLock, Semaphore, Notify, Barrier, FutexMutex, FutexCondvar
monterey-channel	Bounded, unbounded, and oneshot channels (intra-core, !Send)

Examples

cargo run --example echo_server                              # TCP echo server with multishot accept/recv
cargo run --example tcp_proxy -- 127.0.0.1:8080 127.0.0.1:9090  # Zero-copy TCP proxy using splice
cargo run --example hello_nvme -- /dev/ng0n1                 # NVMe read/write/flush
cargo run --example multi_core_nvme -- /dev/ng0n1            # Parallel NVMe I/O across all cores
cargo run --example cross_core_channels                      # Cross-core messaging with flume/tokio channels
cargo run --example priority_tasks                           # Priority scheduling demonstration

Build & Test

cargo build                          # Debug build
cargo build --release                # Release build (LTO, codegen-units=1, opt-level=3)
cargo check                          # Type-check without linking
cargo test                           # Run all tests
cargo bench --bench nvme_iops        # Run a specific benchmark

Documentation

• Getting Started — installation, first server, key concepts
• Architecture — rings, event loop, buffers, cross-core messaging
• I/O Guide — file, network, NVMe, pipes, linked chains
• Concurrency — tasks, channels, sync primitives, timers

License

Licensed under either of Apache License, Version 2.0 or MIT License at your option.