SwiftLM ⚡️ – A Native Swift MLX Inference Server (0 Python, 100% Metal)

SharpAI's SwiftLM is a native Swift inference server optimized for Apple Silicon that supports MLX models with an OpenAI-compatible API, delivering high performance without Python runtime or GIL overhead. It features TurboQuant KV cache compression and SSD streaming for efficiently handling large 100B+ parameter MoE models, significantly improving throughput, memory usage, and time-to-first-token compared to baseline performance. Benchmarking on Apple Silicon, especially with the gemma-4-26b-a4b-it-4bit model, shows up to 2.4× speedup and major memory savings using TurboQuant, enabling fast, scalable LLM inference on macOS and iOS devices.