Add native Apple Metal/MPS backend - #2077
Conversation
|
I built this branch on an M4 to check the numbers, since the ones in the description come from an M1. They hold, though the ratio is lower here. Setup: Apple M4 (10 CPU / 10 GPU, 16 GB), macOS 26, Release build of The C++ suite gives 370 passed and 2 skipped (
MPS float16 and MPS float32 both produced text identical to CPU float32 over the whole workload. Three things came out of it that may be worth folding into the PR.
>>> ctranslate2.get_supported_compute_types("cpu")
{'float32'}
# ctranslate2 4.8.1 from PyPI
>>> ctranslate2.get_supported_compute_types("cpu")
{'float32', 'int8', 'int8_float32'}I would put
I did not test Marian, batch sizes above 1, multi-GPU, or memory pressure on an 8 GB machine. |
ctranslate2 has no Metal backend in any released version, which is the entire reason a seven-minute recording cost seven minutes of CPU on a machine with an idle GPU. OpenNMT/CTranslate2#2077 adds one. Built from that branch and installed, the reference recording: CPU int8 443.1 s 725 words Metal float16 80.4 s 724 words Same text, five and a half times faster. asr_device decides: auto asks the library whether it has the backend, so a stock wheel keeps behaving exactly as before, and mps insists and explains itself rather than quietly running slowly. int8 becomes float16 on the GPU, because carrying the CPU's default over picks the slowest configuration the build offers, 57.6 s against 24.9 s, while looking like a request for the fast one. The device is in the cache fingerprint: float16 on the GPU is not the same transcript as int8 on the CPU, and a cache that ignored that would hand back the other one and call it a hit. WITH_RUY is not optional when building that branch, even though its own instructions omit it: without it there is no int8 on the CPU at all, so the fallback stops working. That and the CMake 4 flag the vendored dependency needs are in the README. Live text does not go through any of this. It runs on Apple's model on the Neural Engine, which is neither the CPU nor the GPU.
|
@nerln |
|
Thanks a lot for your work My initial feedback:
|
798c17d to
11d9bd4
Compare
|
@jordimas Thanks for the feedback. I have addressed the requested items in the latest update.
Full evaluation results:
CPU used FP32 and MPS used FP16 with identical models and decoding settings. Model loading and warm-up were excluded from inference timing. Exact commands, hardware details, and methodology are documented in the benchmark README. |
|
The Successful run: https://github.com/TBO22/CTranslate2/actions/runs/32951573494/job/98123905546 |
|
@nerln The latest commits now resolve generic MPS int8 to int8_float16 and cache the expanded FP16 weights once, so warm inference uses the optimized FP16 kernels instead of repeatedly quantizing activations and running the slower INT32 path. Based on your measurements, this should reduce the previous 57.61 s result toward 24.91 seconds which is about 2.31× faster than the old MPS INT8 path and roughly 2.03× faster than CPU INT8. Could you retest the latest PR head (74b510a) on your M4? The Ruy and CMake 4 build flags are now documented as well. |
|
Rebuilt at 74b510a on the same M4 as before. The build side is clean; the latency re-test is in a separate comment below. The documented cmake line now works as written. The C++ suite, run from the In August at 2f6a066 the same suite gave 370 passed and 2 skipped. Compute types from that build, which is the check that the earlier documented line was failing: >>> ctranslate2.get_supported_compute_types("cpu")
{'float32', 'int8', 'int8_float32'}
>>> ctranslate2.get_supported_compute_types("mps")
{'bfloat16', 'float16', 'float32', 'int8', 'int8_bfloat16', 'int8_float16', 'int8_float32'}
>>> ctranslate2.get_mps_device_count()
1CPU int8 is there, so the fallback path a Whisper user drops onto is intact. |
|
Retested at 74b510a on the M4. Short answer: the resolution landed, the speed did not, and there is a correctness difference that I think matters more than either.
A generic It is still the slowest path in the build. 2.46x slower than MPS float16, and 0.88x against CPU int8, which is exactly where it sat in August. A second pass gave 16.03, 37.95 and 36.03 for MPS float16, MPS int8 and CPU int8, and a second audio file gave 6.14, 18.66 and 16.96, so the ordering is stable. Switching the weight cache off costs 1.5%, inside the window-to-window spread. If expanding the weights once were what the time was going into, that switch should have hurt far more, so the remaining cost looks like it is per call on the activations rather than in the weights. You can see the profile and I cannot. The part I would chase first is not the timing. On one recording the MPS int8 path produces different words from the other four configurations: two words missing from the start of a window, reproducible across both passes and with the cache off. CPU float32, CPU int8, MPS float16 and MPS float32 agree with each other word for word on that same window. It did not reproduce on a short public clip, so it may depend on content, and the recording is private so I cannot attach it. A public file where MPS int8 and CPU int8 disagree on the text is probably the same bug. One caveat on the table: every row is lower than my August numbers, CPU rows included, and I no longer have that script, so I cannot promise the input was identical. Compare within this table, and across the two only as ratios. |
|
@nerln Thank you for retesting this and sharing the detailed results. Profiling showed that the main overhead comes from quantizing activations on every call and running the native INT8 matrix operations. This path can also introduce enough numerical variation to change token selection during decoding. I have addressed this in commit In my local Whisper beam size 5 test with a 12.5 second audio, the updated path completed in 5.11 seconds, compared with 10.53 seconds for native INT8 and 5.21 seconds for explicit FP16. The generated tokens matched the FP16 result exactly. The complete MPS enabled C++ suite and all Python MPS tests also passed. It would be helpful if you could repeat the M4 test using the latest commit. |
|
Retested at Public audio this time, so the numbers can be checked: first 16 clips of the FLEURS
MPS Text: CPU float32, MPS float16, MPS int8 and MPS int8 with the cache off produced the same 157 words, token for token, in all three passes. The divergence I reported last time is gone. CPU int8 differs by two commas and one dropped article. The CPU rows were taken under memory pressure, hence the 160-second range on float32. Read them as an upper bound; the MPS rows are tight. One trap for anyone else rebuilding this. My first retest reported nothing had changed, from a stale binary: C++ suite from the same |
3 participants
Summary
This PR adds an experimental native Metal/MPS backend for CTranslate2 on Apple Silicon. It is opt-in with
-DWITH_MPS=ONand makes the existing C++ and Python APIs acceptdevice="mps"without changing the default CPU, CUDA, or HIP behavior.I started this port about six months ago because CTranslate2 could use Apple CPUs efficiently but had no path to the Apple GPU. The first versions could run a few operations, but real Marian and Whisper inference exposed missing operators, excessive synchronization, lifetime bugs, and poor batch-size-1 performance. I used those end-to-end workloads—not only large synthetic GEMMs—to guide the implementation in this PR.
What is included
Device::MPSsupport in device parsing, dispatch, storage, synchronization, the CLI, and Python bindingsctranslate2.get_mps_device_count()andget_supported_compute_types("mps")compute_type="auto"selects FP16M == 1GEMV path for autoregressive projections, including a fused bias/residual/activation epiloguek <= 8), with MPSMatrix TopK used for supported larger valuesThe dispatch changes also preserve non-MPS builds. In particular, FP16/BF16 dispatch is restricted to enabled GPU backends so MSVC does not instantiate unsupported CPU half-precision symbols in CUDA wheel builds.
Build and use
cmake -S . -B build-mps \ -DCMAKE_BUILD_TYPE=Release \ -DCMAKE_POLICY_VERSION_MINIMUM=3.5 \ -DWITH_MPS=ON \ -DWITH_ACCELERATE=ON \ -DWITH_MKL=OFF \ -DWITH_RUY=ON \ -DOPENMP_RUNTIME=NONE cmake --build build-mps -jThe minimum deployment target is macOS 11. MPS is not enabled in prebuilt wheels by this PR and currently has to be built from source.
Correctness and CI validation
Local validation and the reportable benchmarks were run on a MacBook Air with an Apple M1 (7-core GPU, 8 GB unified memory), arm64, macOS 26.5.2, using Release builds.
The CI configuration includes a dedicated real Apple Silicon MPS GPU job. It builds with
-DWITH_MPS=ON, runs the MPS runtime suite, and then runs the normal CPU/shared suite from the same MPS-enabled build. It also builds the Python bindings, tests explicitdevice="mps"inference, and verifies thatdevice="auto"remains on CPU so the experimental backend is opt-in.The equivalent workflow passed on the self-hosted Apple M1 runner: 192 MPS C++ tests passed, 198 CPU/shared C++ tests passed with one existing CPU-only skip, and both Python MPS integration tests passed.
https://github.com/TBO22/CTranslate2/actions/runs/32951573494/job/98123905546
The upstream
macos-15-xlargejob could not be scheduled because larger-runner billing is not enabled for the repository. No test process started in that failed job; this was a runner availability/billing issue rather than a test failure. The upstream workflow retainsmacos-15-xlarge, so the merged CI configuration does not depend on contributor-owned hardware.MPS coverage includes FP32/FP16/BF16, INT8, odd sizes, tails, broadcast and nonzero batch strides, transposed GEMM, alpha/beta, interior offsets, dependent asynchronous dispatches, TopK tie behavior, Gather, Conv1D, and translation output comparison.
git diff --checkalso passes.Quality Validation
The reproducible evaluation scripts and methodology are under
tools/benchmark/quality/. CPU used FP32 and MPS used FP16 with the same models, inputs, sample order, and decoding settings. Model loading and one warm-up inference were excluded from inference timing.Performance
Performance varies by model and sequence length, so this is not intended as a universal claim. One representative end-to-end result from the M1 test machine was a 12.52-second, batch-size-1 Whisper transcription/translation workload:
This is a 3.54x end-to-end speedup for that workload. The PR also includes
tools/benchmark_mps_marian.py, which runs CPU and MPS in separate processes with warmups and repeated measurements across source lengths and beam sizes, andtests/benchmark_mps.ccfor decode GEMM, vocabulary projection, prefill GEMM, TopK, and copy-heavy operations.Current limitations
WITH_MPScannot currently be combined with CUDA or HIP in one buildAI-assisted development disclosure
I used Codex with GPT-5.6 as a pair-programming tool for repository navigation, repetitive implementation work, test scaffolding, diff review, and tracing build and lifetime failures. I did not treat generated suggestions as authoritative. I researched the execution model and kernel design myself, including how Apple MLX kernels differ from ggml/PyTorch and CUDA implementations and how Apple unified memory affects synchronization and ownership. I selected the design, ran the benchmarks, reproduced correctness failures, reviewed the changes, and remain responsible for the implementation and results in this PR.
I understand that this is a large, performance-sensitive change and am happy to respond to detailed review or split follow-up work where maintainers prefer a different boundary.