Context
Split out of the perf program (AISIX-Cloud#1259). A local x86 screening pass (12-core VM; harness reuses the bench measurement core; clean-verified batch: first-to-last baseline drift −0.12%, floor headroom 11.9–16×, every run with full valid windows) measured, on top of the already-shipped allocator + fat-LTO release profile (#939):
| candidate |
Δ rps vs baseline |
note |
-Wl,--emit-relocs control |
+1.9% |
BOLT's entry requirement is essentially free at runtime |
| PGO |
+31.5% |
vs the relocs control: +29.1pt attributable to the profile itself |
| PGO + BOLT |
+36.6% |
BOLT's increment on top of PGO: +3.9% |
Profile-dilution check: training on a three-dialect mix (chat + messages + responses) instead of the single measured endpoint cost only −1.09% — the gains live in the generic hot path (HTTP parsing / serde / dispatch), not in one endpoint's branches, so they are expected to transfer across traffic shapes.
Qualification that must travel with these numbers: measured against a synthetic small-body single-upstream workload. Per the platform decision below (2026-08-13), x86 is the primary verification platform for this work, so these numbers are the current gate-1 evidence; the remaining uncertainty is workload shape, not architecture. arm64 verification is deferred to a final acceptance pass.
Platform decision (2026-08-13)
x86 is the primary battlefield; the current x86 host environment is the verification rig. Most production deployments run on x86, so it is the more representative target. The arm64 (Graviton) benchmark rig is not a precondition for any early-phase work: early phases only need to demonstrate a clearly significant win on x86, and no arm regression runs are required until the end. A final acceptance pass on arm64 happens once the x86 side is complete. This supersedes the original "Graviton rig verification first" hard gate below.
Why this is a release-pipeline feature, not a code change
PGO/BOLT change no source. The deliverable is the build pipeline for shipped artifacts:
- PGO: build instrumented → run a training workload on the build host → rebuild with
-Cprofile-use.
- BOLT (optional second stage): link with
-Wl,--emit-relocs → llvm-bolt post-link layout optimization; needs the llvm-bolt toolchain on the builder.
- Channel analysis: the gains ship in our release tarballs and Docker images. Third parties building from source with a plain
cargo build --release will not get them (committed-profdata schemes are toolchain-version-fragile; BOLT cannot ride a source build at all). This asymmetry is normal in the ecosystem — official binaries of major compilers, browsers and runtimes are PGO'd builds of the same source.
- Host-native constraint: the instrumented binary must execute on the build host, so cross-compiled targets cannot self-train.
aarch64-linux artifacts need native arm64 runners (GitHub provides them). Survey note per the repo research rule: at least one mainstream Rust AI gateway ships mandatory fail-closed PGO for its host-native release targets while its cross-compiled aarch64-linux artifact remains un-PGO'd for exactly this reason; none of the mainstream gateways surveyed ship BOLT. Shipping PGO'd arm64 artifacts is therefore an open differentiation opportunity, and BOLT is one on every platform.
Training-set design (decided 2026-08-13)
Principles:
- A profile is a union of hotness, not a traffic-mix replica. Run each shape separately for a few thousand requests against a local mock, then merge the
.profraw files (llvm-profdata merge). No "realistic concurrent user simulation" — determinism and reproducibility win.
- Coverage matters, proportions don't. Missing an entire hot path is the failure mode; a skewed ratio is not (bounded by the −1.09% dilution measurement above).
v1 shape matrix (~8 shapes):
- dialects:
/v1/chat/completions, /v1/messages, /v1/responses
- streaming AND non-streaming — the throughput bench drives non-streaming only, but production streaming traffic must not be handed a cold layout
- body sizes: minimal (~86 B) and mid (~4 KB)
- config posture: default (observability on) for most shapes, plus one shape with a rate-limit policy configured so the quota path trains hot
- error paths: deliberately untrained — they are cold in production, and PGO treating them as cold is correct
Maintenance rule: the training mix mirrors the bench suites; when the product grows a new hot path, add a shape in the same PR.
Hard gates
- x86 verification first (updated 2026-08-13; supersedes the original Graviton-first wording — see the platform decision above). Same-session anchored A/B on the x86 host per the program methodology (front/back anchors, fail=0, in-batch control leg, port-ownership pid preflight, floor headroom ≥2×). The clean 2026-08-13 screening batch (drift −0.12%, headroom 11.9–16×) already provides this evidence; re-run only if the training set or workload shape changes materially. arm64 (Graviton) verification moves to a final acceptance pass after the x86 work is complete.
- Fail-closed. Once adopted, PGO is mandatory for release artifacts: any phase failing (instrumented build, training producing no
.profraw, merge, optimized build) fails the release — never a silent fallback to a plain build. A positive proof marker (merged profile size + profraw count, written only after a successful -Cprofile-use build) is asserted by the release workflow before shipping.
- Holdout validation. Keep one scenario out of the training set; the PGO'd binary must not regress beyond the noise band vs the non-PGO binary on that holdout. Re-run on every training-set change.
- Training scripts and payloads are version-controlled; profiles are regenerated per release and never committed stale.
Measured costs (local, for planning)
- Build wall-clock: plain release 6m33s → PGO ≈16m (2.5×) → PGO+BOLT ≈23m (3.5×; BOLT itself is ~33 s — the rest is the full recompile that a RUSTFLAGS change triggers).
- The real CI requirement is running a workload mid-pipeline (once for PGO, once more for BOLT), not the toolchain dependencies.
- BOLT pitfall observed during screening: a distro
/usr/bin/llvm-bolt resolves its runtime library by prefix and fails (library not found: /usr/lib/libbolt_rt_instr.a); the LLVM bin dir (e.g. /usr/lib/llvm-18/bin) must precede it on PATH.
Decision points (in order, all user-level)
- x86 verification outcome (gate 1) — the clean screening batch stands as the current evidence.
- Adopt PGO for release artifacts — accepting the CI complexity and ~2.5× release build time?
- Adopt BOLT on top — ~+4% more for 3.5× build time and an extra toolchain, priced separately?
- Interaction with the allocator candidate from the same screening pass (a combined allocator+PGO+BOLT stack cell was still being measured when this issue was filed; its result feeds here).
References
Not scheduled — this issue is the self-contained handoff; implementation will be picked up by a dedicated session.
Context
Split out of the perf program (AISIX-Cloud#1259). A local x86 screening pass (12-core VM; harness reuses the bench measurement core; clean-verified batch: first-to-last baseline drift −0.12%, floor headroom 11.9–16×, every run with full valid windows) measured, on top of the already-shipped allocator + fat-LTO release profile (#939):
-Wl,--emit-relocscontrolProfile-dilution check: training on a three-dialect mix (chat + messages + responses) instead of the single measured endpoint cost only −1.09% — the gains live in the generic hot path (HTTP parsing / serde / dispatch), not in one endpoint's branches, so they are expected to transfer across traffic shapes.
Qualification that must travel with these numbers: measured against a synthetic small-body single-upstream workload. Per the platform decision below (2026-08-13), x86 is the primary verification platform for this work, so these numbers are the current gate-1 evidence; the remaining uncertainty is workload shape, not architecture. arm64 verification is deferred to a final acceptance pass.
Platform decision (2026-08-13)
x86 is the primary battlefield; the current x86 host environment is the verification rig. Most production deployments run on x86, so it is the more representative target. The arm64 (Graviton) benchmark rig is not a precondition for any early-phase work: early phases only need to demonstrate a clearly significant win on x86, and no arm regression runs are required until the end. A final acceptance pass on arm64 happens once the x86 side is complete. This supersedes the original "Graviton rig verification first" hard gate below.
Why this is a release-pipeline feature, not a code change
PGO/BOLT change no source. The deliverable is the build pipeline for shipped artifacts:
-Cprofile-use.-Wl,--emit-relocs→llvm-boltpost-link layout optimization; needs the llvm-bolt toolchain on the builder.cargo build --releasewill not get them (committed-profdata schemes are toolchain-version-fragile; BOLT cannot ride a source build at all). This asymmetry is normal in the ecosystem — official binaries of major compilers, browsers and runtimes are PGO'd builds of the same source.aarch64-linuxartifacts need native arm64 runners (GitHub provides them). Survey note per the repo research rule: at least one mainstream Rust AI gateway ships mandatory fail-closed PGO for its host-native release targets while its cross-compiled aarch64-linux artifact remains un-PGO'd for exactly this reason; none of the mainstream gateways surveyed ship BOLT. Shipping PGO'd arm64 artifacts is therefore an open differentiation opportunity, and BOLT is one on every platform.Training-set design (decided 2026-08-13)
Principles:
.profrawfiles (llvm-profdata merge). No "realistic concurrent user simulation" — determinism and reproducibility win.v1 shape matrix (~8 shapes):
/v1/chat/completions,/v1/messages,/v1/responsesMaintenance rule: the training mix mirrors the bench suites; when the product grows a new hot path, add a shape in the same PR.
Hard gates
.profraw, merge, optimized build) fails the release — never a silent fallback to a plain build. A positive proof marker (merged profile size + profraw count, written only after a successful-Cprofile-usebuild) is asserted by the release workflow before shipping.Measured costs (local, for planning)
/usr/bin/llvm-boltresolves its runtime library by prefix and fails (library not found: /usr/lib/libbolt_rt_instr.a); the LLVM bin dir (e.g./usr/lib/llvm-18/bin) must precede it on PATH.Decision points (in order, all user-level)
References
Not scheduled — this issue is the self-contained handoff; implementation will be picked up by a dedicated session.