A C++23 order-matching engine with price-time priority, lock-free SPSC pipelines, zero-allocation hot paths, and io_uring-based asynchronous UDP networking.
Nexus implements the path from a framed UDP order through decoding, price-time-priority matching, and fill encoding. The matching path uses preallocated pools, cache-aware containers, and intrusive lists to avoid heap allocation after initialization.
Two dedicated threads communicate through lock-free SPSC queues. The network thread uses kernel-managed UDP sockets through io_uring on Linux, with a non-blocking POSIX fallback; the matching thread spin-polls inbound messages and dispatches them to per-symbol order books. CPU affinity and real-time scheduling hooks are available, but the committed measurements below were collected in an ARM64 Linux VM rather than a tuned bare-metal deployment.
+-----------------+
| UDP Client |
+--------+--------+
| sendto / recvfrom
v
+-----------------------------------------------------------------------+
| Nexus Server |
| |
| +-------------------+ SPSC Queue +-------------------+ |
| | Network Thread | ---(InboundMessage)-->| Matching Thread | |
| | | | | |
| | io_uring / POSIX | | MatchingEngine | |
| | Frame decode | <-(OutboundMessage)---| OrderBook[] | |
| | Frame encode | SPSC Queue | PoolAllocator | |
| +-------------------+ +-------------------+ |
| |
+-----------------------------------------------------------------------+
OrderBook (one per symbol):
+------------------+
| Bids (sorted | FlatPriceMap<greater> -- contiguous sorted array
| desc by price) | Each level: intrusive doubly-linked list of Orders
+------------------+
| Asks (sorted | FlatPriceMap<less> -- binary search O(log n) lookup
| asc by price) | Price-time priority within each level
+------------------+
| FlatHashMap | Robin Hood hashing, O(1) cancel/modify
| PoolAllocator | O(1) alloc/dealloc, zero system calls
+------------------+
These results were recorded on a 12-core ARM64 system reported at 2 GHz in an
OrbStack Linux VM, using a release build with -O3 and LTO. They are local
microbenchmark observations, not production latency claims; raw benchmark
output is not committed. The benchmark programs under benchmarks/ reproduce
the measurement paths.
| Operation | Latency | Throughput |
|---|---|---|
| Insert (no match) | 27 ns | 37M orders/sec |
| Full match (cross) | 43 ns | 23M orders/sec |
| Cancel | 25 ns | 40M orders/sec |
| Modify (qty down) | 1.4 ns | 710M ops/sec |
| Iceberg replenish | 22 ns | 46M ops/sec |
| Stop trigger + fill | 228 ns | 4.4M ops/sec |
| Depth snapshot (10 lvl) | 34 ns | 30M ops/sec |
| Operation | Latency |
|---|---|
| No match | 38 ns |
| Full match | 59 ns |
| Cancel | 27 ns |
| Percentile | ARM64 VM |
|---|---|
| p50 | 17.5 μs |
| p90 | 45.9 μs |
| p99 | 2.2 ms |
The full round trip is client sendto() → kernel UDP stack → io_uring receive
→ decode → SPSC → match → fill → SPSC → encode → io_uring send → kernel UDP
stack → client recvfrom(). The VM result includes scheduler and hypervisor
jitter; no cause is inferred from the numbers alone.
This benchmark removes socket and kernel-networking work and measures the SPSC and matching-thread path directly. It is not a DPDK benchmark; Nexus does not implement DPDK or another kernel-bypass data path.
| Percentile | ARM64 VM |
|---|---|
| p50 | 875 ns |
| p90 | 917 ns |
| p99 | 1.0 μs |
| p99.9 | 14.0 μs |
| Component | ARM64 VM |
|---|---|
| SPSC push+pop (single thread) | 420M ops/sec |
| SPSC throughput (cross-thread) | 370M+ ops/sec |
The following ranges were retained from the original design notes for transparency. They are estimates for hardware/configurations that have not been benchmarked in this repository and must not be read as observed results.
| Metric | x86_64 bare metal (estimated) | x86_64 + isolcpus (estimated) |
|---|---|---|
| UDP round-trip p50 | 8–12 μs | 5–8 μs |
| UDP round-trip p90 | 15–25 μs | 10–15 μs |
| UDP round-trip p99 | 50–100 μs | 20–40 μs |
| Thread-to-thread p50 | 350–500 ns | — |
| Thread-to-thread p90 | 400–550 ns | — |
| Thread-to-thread p99 | 500–700 ns | — |
| Thread-to-thread p99.9 | 2–5 μs | — |
| SPSC push+pop | 700M–1B ops/sec | — |
| SPSC cross-thread throughput | 500–800M ops/sec | — |
- Price-time priority limit order book with intrusive doubly-linked lists
- Order types: Limit, Market, IOC, FOK, Iceberg (hidden quantity), Stop-Limit
- io_uring-based asynchronous UDP networking (Linux) with a non-blocking POSIX fallback (macOS/Linux)
- Lock-free SPSC queues for inter-thread communication
- Custom data structures: FlatPriceMap (sorted contiguous array), Robin Hood hash map
- Pool allocators with O(1) alloc/dealloc and zero hot-path heap allocation
- CPU core pinning and real-time scheduling support
- Market data depth snapshots with visible-only quantities (iceberg-aware)
- Binary wire protocol with 16-byte frame headers and sequence numbering
- Latency histograms with percentile reporting
- Profile-Guided Optimization (PGO) build infrastructure
Floating-point arithmetic is non-deterministic across platforms and compiler settings (-ffast-math, FMA contraction). Real exchanges use fixed-point integer pricing for exact reproducibility. We use int64_t with 6 decimal places ($150.25 = 150'250'000).
std::map is a node-based tree, while FlatPriceMap stores price levels in a
contiguous sorted array. That makes lookup O(log n) and insertion O(n) due
to shifting, in exchange for sequential storage when the matching loop walks
levels from the best price. The included matching benchmark exercises
multi-level walks; it does not claim a committed std::map comparison.
std::unordered_map uses separate chaining — each bucket is a heap-allocated linked list, causing cache misses on every lookup. Our FlatHashMap uses open addressing with Robin Hood probe distance balancing: entries with longer probe chains "steal" slots from entries with shorter chains, reducing worst-case variance. Backward-shift deletion avoids tombstone accumulation, which is critical for cancel-heavy workloads where erasures are as frequent as insertions. Fibonacci hashing distributes sequential order IDs uniformly across the table.
malloc/new are unbounded in latency — a single allocation can trigger mmap, brk, or page faults. Our pool pre-allocates a contiguous block of Order-sized slots and manages a singly-linked free list threaded through unused slots. Allocate pops the head; deallocate pushes to the head. O(1), zero system calls, deterministic.
Non-intrusive containers (like std::list) allocate a separate node for each element. Intrusive lists embed the prev/next pointers directly in the Order struct. Since orders come from the pool, they are already linked-list nodes — zero extra allocation, zero indirection. O(1) insertion (append to tail), O(1) removal (unlink by pointer without search).
io_uring exposes memory-mapped submission and completion rings and can batch
work submitted to the kernel. Nexus uses those rings around ordinary
kernel-managed UDP sockets to reduce control-path overhead and keep multiple
operations in flight. This is asynchronous networking, not kernel bypass;
packets still traverse the Linux networking stack.
The network thread and matching thread have fundamentally different performance profiles. The network thread is I/O-bound; the matching thread is compute-bound. Separating them allows:
- CPU pinning each to an isolated core (no scheduler interference)
- The matching thread never touches a syscall (pure compute)
- Natural backpressure via the SPSC queue (no locks, no mutexes)
More threads would add synchronization complexity without benefit — the order book is inherently sequential (price-time priority requires total ordering).
With exactly one producer and one consumer, SPSC queues need only atomic load/store with acquire/release ordering — no CAS loops, no mutexes, no contention. The head and tail counters are placed on separate cache lines to eliminate false sharing between cores. Power-of-2 capacity enables branchless index wrapping via bitmask. On x86, the TSO memory model means acquire/release compiles to plain loads/stores — zero fence overhead.
- The networking backends are io_uring and non-blocking POSIX UDP. DPDK, AF_XDP, RDMA, and other kernel-bypass transports are not implemented.
- The committed performance table is from one ARM64 virtualized environment; bare-metal x86_64 figures are explicitly projections.
- Raw output for the recorded measurements is not committed, so results should be reproduced with the included benchmark binaries before comparison.
- The server is an exchange-engine prototype, not a production venue: it does not provide persistence, replication, risk checks, authentication, or a production market-data protocol.
# Prerequisites: CMake 3.22+, C++23 compiler (GCC 13+ or Clang 17+)
# Linux: liburing-dev for io_uring support
# Debug build (with sanitizers)
cmake -B build -DCMAKE_BUILD_TYPE=Debug -DENABLE_ASAN=ON -DENABLE_UBSAN=ON
cmake --build build --parallel
# Release build (with LTO)
cmake -B build-release -DCMAKE_BUILD_TYPE=Release
cmake --build build-release --parallel
# Profile-Guided Optimization build
./scripts/pgo_build.sh # Full PGO pipeline
./scripts/pgo_build.sh --compare # With before/after comparison
# Run tests (18 suites, 216 declared test cases)
cd build-release && ctest --output-on-failure
# Run benchmarks
./build-release/benchmarks/bench_matching
./build-release/benchmarks/bench_pipeline
./build-release/benchmarks/bench_wire_to_wire
# Start the server
./build-release/nexus_server --port 9000 --symbols 1,2,3
# Run the latency test client
./build-release/test_client --host 127.0.0.1 --port 9000 --orders 1000018 test suites with 216 declared test cases covering:
- Core: Order book matching (limit, market, IOC, FOK, iceberg, stop-limit), cancel, modify, depth snapshots, market data callbacks
- Data structures: FlatPriceMap (sorted array), FlatHashMap (Robin Hood), PoolAllocator, SPSC queue (including multi-threaded stress tests)
- Protocol: Binary codec round-trips, frame encode/decode, message dispatch pipeline, sequence tracking
- Network: io_uring source, POSIX source, UDP loopback
- Integration: End-to-end UDP round-trip (order → fill), burst tests (100 concurrent pairs), graceful shutdown, invalid frame handling
nexus/
├── src/
│ ├── core/ # Matching engine + order book
│ │ ├── types.h # Order, Fill, Price, CancelRequest, DepthSnapshot
│ │ ├── price_level.h # Intrusive doubly-linked list FIFO queue
│ │ ├── flat_price_map.h # Sorted contiguous array (replaces std::map)
│ │ ├── order_book.h/cpp # Single-symbol LOB with price-time priority
│ │ ├── matching_engine.h/cpp # Multi-symbol order router
│ │ └── matching_thread.h/cpp # Dedicated matching thread with spin-poll loop
│ ├── memory/ # Zero-allocation infrastructure
│ │ ├── pool_allocator.h # Fixed-size object pool (free-list based)
│ │ ├── flat_hash_map.h # Robin Hood hash map with backward-shift deletion
│ │ └── spsc_queue.h # Lock-free Lamport queue, cache-line separated
│ ├── protocol/ # Wire format + framing
│ │ ├── messages.h # Cache-line aligned wire structs (64B orders, 32B cancel)
│ │ ├── messages_internal.h # Compact SPSC transport types (no alignment padding)
│ │ ├── binary_codec.h # Zero-copy encode/decode (memcpy-based)
│ │ ├── framing.h # 16-byte UDP frame header (magic, version, seq)
│ │ ├── message_dispatcher.h # Type-safe wire-to-engine dispatch
│ │ └── sequencer.h # Inbound gap/duplicate detection, outbound numbering
│ ├── network/ # Packet I/O
│ │ ├── io_uring_source.h/cpp # Async UDP via io_uring (Linux)
│ │ ├── posix_source.h/cpp # Non-blocking UDP fallback (macOS/Linux)
│ │ └── network_thread.h/cpp # Dedicated I/O event loop
│ ├── server/ # Orchestration
│ │ ├── nexus_server.h/cpp # Thread lifecycle, ordered startup/shutdown
│ │ └── main.cpp # CLI args, POSIX signal handling
│ ├── platform/
│ │ └── thread_utils.h/cpp # CPU pinning, RT scheduling, thread naming
│ └── telemetry/
│ ├── latency_tracker.h # Fixed-bucket histogram (100ns resolution)
│ └── stats.h # Throughput counters
├── tests/ # 18 Google Test suites (216 declared cases)
├── benchmarks/ # 5 Google Benchmark suites
├── tools/
│ ├── order_generator.cpp # Realistic order flow generator (1M orders)
│ └── test_client.cpp # UDP latency measurement client
├── cmake/
│ ├── CompilerWarnings.cmake # -Wall -Wextra -Wpedantic -Wshadow ...
│ ├── Sanitizers.cmake # ASan, UBSan, TSan
│ ├── IoUring.cmake # Conditional io_uring detection
│ └── PGO.cmake # Profile-Guided Optimization support
└── scripts/
└── pgo_build.sh # Automated PGO build pipeline
| Platform | Status | Notes |
|---|---|---|
| Linux x86_64 | Primary | Full support including io_uring |
| Linux ARM64 | Supported | Full support; 128-byte cache lines |
| macOS ARM64 | Dev only | Matching engine + tests work; no io_uring |
MIT