What if the best speech recognition in the world didn’t live in a data center? What if it ran faster than real-time, on your laptop, with no API key, no network request, and no audio leaving your machine — and still beat OpenAI’s flagship model on the standard benchmark?

speech-swift — our open-source Swift library for on-device speech AI — ships models that outperform Whisper Large v3 on LibriSpeech, the industry-standard speech recognition benchmark. Two different architectures, two different reasons: Qwen3-ASR is an audio language model — a new class of automatic speech recognition (ASR) that uses a production LLM as its decoder instead of Whisper’s purpose-built one. Parakeet TDT is a transducer with no autoregressive decoder at all — a fundamentally different design that eliminates Whisper’s specific hallucination failure mode — generating text during silence or after audio ends. Both run entirely on-device, on Apple Silicon.

Press enter or click to view image in full size

The Numbers

LibriSpeech test-clean — 2,620 utterances, the industry benchmark for ASR. All our numbers are fully reproducible; the benchmark script is in the repo and takes 15 minutes to run.

Press enter or click to view image in full size

Three on-device models beat Whisper Large v3 (FP16). Qwen3-ASR 1.7B 8-bit leads at 2.35% WER — 26% smaller than Whisper and 13% more accurate. Qwen3-ASR 1.7B 4-bit matches Whisper quality at 1.2 GB, 61% smaller. Qwen3-ASR 0.6B 8-bit matches Whisper at 40% of the parameter count in 960 MB.

Parakeet TDT does it in 634 MB as a CoreML model, running entirely on the Neural Engine and leaving CPU and GPU completely free. All numbers on M2 Max, fully reproducible.

The Journey: From Transcription to a Full Speech Library

speech-swift didn’t start as a benchmark story. It started as a question: could Apple Silicon’s unified memory and MLX’s Metal acceleration run serious speech models natively — no Python, no server, no tensor copying between CPU and GPU?

The answer turned out to be yes, and then some. Qwen3-ASR 0.6B established the MLX patterns: KV cache, RoPE, 4-bit quantized matmuls running on the GPU. Parakeet TDT added a second inference path — CoreML on the Neural Engine, a dedicated hardware accelerator built into every Apple Silicon chip, from M-series Macs to iPhones. MLX on the GPU is faster; CoreML on the Neural Engine is more power-efficient — the right choice for battery-constrained devices.

Once both paths worked, the right question wasn’t can we match Whisper — it was what does it take to beat it. That meant understanding Whisper’s specific failure modes — hallucination on silence, repetition loops, context lost at 30-second chunk boundaries — and finding architectures that address them structurally, not just at scale.

Why These Models Beat Whisper: Two Different Bets

Whisper Large v3 is a 32-layer encoder-decoder transformer trained on 680,000 hours of weakly-supervised web audio. Its autoregressive decoder is both the source of its fluency and the root of its hallucination problem.

An audio language model: Qwen3-ASR

Audio → Mel (128-dim) → AuT Encoder → Projector → Qwen3 LLM Decoder → Text

Qwen3-ASR belongs to what research is now calling the Large Audio-Language Model (LALM) paradigm. The idea: take a production LLM and give it ears. An audio encoder converts speech into a sequence of representations; a learned projector maps those into the LLM’s embedding space; the LLM generates the transcript from there. The AuT (Audio Transformer) encoder was pretrained on approximately 40 million hours of audio — around 60x Whisper’s training set — before the Qwen3 LLM decoder is even counted.

In Whisper, the decoder is a cross-attention module purpose-built for transcription: compact, fast, trained only on speech data. In Qwen3-ASR, that decoder is Qwen3 — the same LLM architecture as the Qwen3 text model family. That matters most when audio is acoustically ambiguous. When their and there sound identical, a dedicated ASR decoder resolves it from speech statistics. A Qwen3 decoder resolves it from the entire distribution of written language — a far stronger prior on how words fit together in context.

The spectral input is also richer: the AuT encoder uses 128-dim mel features with 8× downsampling to 12.5 Hz tokens — finer temporal resolution than Whisper’s fixed 30-second chunking, which loses context at segment boundaries.

The practical consequence of a stronger decoder: you don’t need beam search. Whisper keeps 5 candidate sequences alive in parallel, scoring each at every decoding step and picking the best at the end — roughly 5x the compute of simply taking the most probable next token at each step (greedy decoding). Whisper needs this because its decoder is uncertain enough that the greedy path often isn’t the best one. Qwen3’s LLM decoder is confident enough that greedy is nearly always right. The result: 5x fewer decoder passes per token, better accuracy, and a large share of the speed advantage explained by a single architectural choice.

The non-autoregressive bet: Parakeet TDT

Audio → Mel (128-dim) → FastConformer Encoder → TDT Joint Network → Text

Parakeet takes a structurally different approach. The Token-and-Duration Transducer doesn’t generate text token-by-token — it maps encoder frames to tokens directly through a joint network, predicting both the token and how many frames to advance. It can’t generate text during silence or after audio ends — the joint network scores each frame independently and outputs either a token or blank based solely on what’s in that frame. There’s no autoregressive loop, no repetition spiral, no beam search.

Hallucination in Whisper isn’t a bug — it’s what autoregressive decoders do when they’re uncertain. Parakeet eliminates the failure mode at the architecture level.

Speed: 40x Real-Time

Whisper.cpp on the same M2 Max runs at RTF ~0.10 for Large v3. Our best model runs at RTF 0.023 — nearly 4x faster, with better accuracy.

Press enter or click to view image in full size

A one-hour podcast transcribed in 84 seconds. Cold start to first transcription in under 3 seconds.

Where the speed comes from

Five things compound:

Unified memory, zero copy. MLX tensors live in Apple Silicon’s shared CPU/GPU memory. Audio arrives as a Float32 buffer and becomes an MLX array without a single copy. On discrete GPUs, the CPU→GPU transfer alone takes longer than our full inference.

4-bit quantization cuts memory bandwidth 4x. Transformer inference is memory-bandwidth-bound — you read every weight once per token. At 4-bit, you read 4x less data per matmul. The M2 Max has 400 GB/s of bandwidth; we use it more efficiently.

Pre-compiled Metal shaders. We compile the MLX Metal shader library at build time. Without this, MLX JIT-compiles kernels on first use — ~5x overhead. With compiled metallib, every matmul and attention kernel dispatches instantly.

Greedy decoding, not beam search. Whisper defaults to beam_size=5 — tunable, but 5 is what most deployments run and what the benchmarks reflect. That's roughly 5x the decoder compute of greedy. The Qwen3 LLM decoder is confident enough that the greedy path is nearly always right, so we don't pay that cost.

KV cache stays on GPU. For autoregressive decoding, the key-value cache from all previous decoder steps stays in GPU memory across the entire sequence. CoreML doesn’t expose persistent memory between calls — the cache has to be passed in and returned each step, crossing the Swift ↔ Neural Engine boundary twice per token. For a 200-token transcription, that’s 400 avoided memory transfers on the MLX path.

Compound these: zero-copy × 4x less bandwidth × no JIT × 5x fewer passes × GPU-resident cache = RTF 0.023.

Beyond English: The Quantization Cliff

Whisper benchmarks are always on English LibriSpeech. We tested Qwen3-ASR 0.6B — both 4-bit and 8-bit — on FLEURS, Google’s multilingual evaluation set (400+ utterances per language), across 10 languages:

Press enter or click to view image in full size

European languages are solid across both variants. The gap between 4-bit and 8-bit is the story for everything else. Korean: 19.95% → 6.89% — a 65% error reduction just from switching quantization. Japanese: 16.11% → 8.64%.

If you serve multilingual users, don’t use 4-bit. The English numbers barely change; the non-English numbers change dramatically.

The Compression Sweet Spot

Press enter or click to view image in full size

The 1.7B at 4-bit is the best quality-to-size trade: Whisper Large v3 quality in 1.2 GB. The 0.6B at 4-bit is the edge deployment option — fits on any Apple device, still matches Whisper Small.

What We’re Building

speech-swift is an open-source Swift library for on-device speech AI on Apple Silicon. ASR brought us here. The library now covers the full speech stack:

• ASR: Qwen3-ASR (52 languages), Parakeet TDT (25 languages)
• TTS: Qwen3-TTS, CosyVoice3, Kokoro — with voice cloning
• Speech-to-speech: PersonaPlex 7B — full-duplex, one model, no pipeline
• VAD: FireRedVAD (99% F1), Silero, Pyannote
• Speaker diarization: who spoke when
• Speech enhancement: DeepFilterNet3 noise suppression

Everything runs locally. MLX for GPU inference, CoreML for Neural Engine. Native Swift async/await.

An Android port is also in progress at soniqo/speech-android.

let model = try await Qwen3ASRModel.fromPretrained()
let text = model.transcribe(audio: samples, sampleRate: 16000)

Try It

git clone https://github.com/soniqo/speech-swift
cd speech-swift && make build

# Transcribe
.build/release/audio transcribe recording.wav# Reproduce all benchmark numbers (15 min)
python scripts/benchmark_asr.py --batch --engine qwen3 --model 0.6B
python scripts/benchmark_asr.py --batch --engine qwen3 --model 1.7B

Full benchmark results and raw data: soniqo.audio/benchmarks

All benchmarks on M2 Max, 64 GB, macOS 14. Whisper reference numbers from original OpenAI papers (FP16). Our numbers are fully reproducible — scripts and dataset config included in the repo. A deep dive into MLX vs CoreML tradeoffs — INT4 vs INT8 performance, KV cache management on the Neural Engine, and concurrent GPU + ANE scheduling — is coming next.