Skip to content

perf(purejava): parallelize Q8 FFN preparation#56

Merged
bsbodden merged 2 commits into
mainfrom
perf/q8-parallel-ffn-preparation
Jul 23, 2026
Merged

perf(purejava): parallelize Q8 FFN preparation#56
bsbodden merged 2 commits into
mainfrom
perf/q8-parallel-ffn-preparation

Conversation

@bsbodden

Copy link
Copy Markdown
Member

Summary

  • partition residual addition, RMS normalization, and Q8 activation preparation by active prompt row inside the retained seven-stage layer plan
  • enable the path only for all-Q8 staged gate/up projections with block-major activations and at least two GGUF participants
  • expose the default-off deployment/recommendation setting through deterministic plan schema pure-java-v14
  • document the controlled full-model retention gate

Why

The post-block-major JFR profile showed stage-completion skew while one participant prepared every FFN input row. The new path balances that independent row work across the existing GGUF participants without adding a stage or changing batch-one decode.

Vectors support is merged at 523f3aa95f503a35babc02b76e5537df5f06891c.

Correctness

  • test-first red/green/refactor development
  • exact 40-token forward-pass comparison covering a full batch and partial tail batch
  • bit-identical final logits, complete K/V buffers, and next autoregressive logits
  • deterministic serial fallback for mixed formats, packed activations, and single-participant runtimes

Performance

SmolLM2 360M Q8_0, GraalVM Java 25, fixed 1 GiB heap, eight workers, batch 32, six counterbalanced fresh-process pairs, five warmups plus five measured requests per process:

Metric Serial Batch-row Change
p50 TTFT 913.102 ms 899.064 ms -1.54%
p95 TTFT 930.788 ms 914.340 ms -1.77%
p50 prefill 172.543 tok/s 175.592 tok/s +1.77%
p50 CPU 6,860 ms 6,840 ms -0.29%

All six process-pair medians and 28/30 corresponding trials improved. All 60 trials succeeded with exact corresponding input counts, output counts, and output hashes; median process RSS did not regress.

Validation

  • ./gradlew :models-backend-purejava:check spotlessCheck
  • repository-wide Spotless configuration: all 31 checks pass

@bsbodden
bsbodden merged commit 423621b into main Jul 23, 2026
11 checks passed
@bsbodden
bsbodden deleted the perf/q8-parallel-ffn-preparation branch July 23, 2026 16:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant