Unlocking High-Performance AI Inference on 2021 M1 Max with Qwen3.8-27B
Recent advancements in AI model deployment have opened new possibilities for leveraging Apple Silicon hardware, particularly the M1 Max, to run large language models efficiently. A recent project showcases how modifications and custom kernel development can significantly boost performance, enabling the running of a 27-billion-parameter model, Qwen3.8-27B, at impressive speeds on a 2021 M1 Max.
Background: The Rise of Splash for Local Model Inference
Splash, developed by the Inco AI team, is a highly optimized inference engine tailored for Apple Silicon chips. It leverages model-specific Metal kernels and DFlash speculative decoding to deliver accelerated performance for large language models. Initially, Splash demonstrated remarkable speed improvements, especially on newer chips like the M3, with the ability to decode at twice the rate of comparable engines. However, supporting older hardware like the M1 Max posed challenges.
Challenges with Running Splash on the M1 Max
The original Splash engine relies heavily on Apple’s MPP matrix library and bfloat16 arithmetic, neither of which are optimized or even supported on the M1 Max. Specifically:
- Bfloat16 Support: The M1 Max does not natively support bfloat16 hardware operations.
- Library Dependencies: The MPP library, used for matrix computations, is slow and not tailored for the older GPU architecture.
These limitations restrict the speed and efficiency of inference on M1 Max hardware when using unmodified Splash kernels.
Custom Kernel Development for the M1 Max
To overcome these obstacles, the author engineered custom Metal kernels tailored for the M1 Max’s capabilities:
- Exact Arithmetic Handling: The custom kernels convert 4-bit weights into exact half-precision floating points using a clever bit trick (
nibble | 0x6400) to facilitate precise computations. -
Native Format Computations: Activations are promoted to FP32 for multiplication, allowing the calculations to leverage the GPU’s native instruction set. This approach ensures:
-
Exact results for weight-activation multiplications.
-
Summations that preserve numerical accuracy, with results matching the original model’s outputs to over 99.9%.
-
Optimized Parallel Request Handling: Weights are unpacked once and reused across parallel inference requests, improving throughput.
By writing these custom kernels, the inference process benefits from closer-to-peak GPU performance, reaching approximately 8.6 to 9.9 TFLOPS, well within the GPU’s theoretical capabilities.
Performance Benchmarks: Significant Speed Gains
The modified setup achieved a substantial increase in inference speed:
- Average Throughput: 39 tokens per second (tok/s) on a typical five-prompt benchmark, more than doubling the 19 tok/s observed with the stock Splash kernels.
- Real-World Coding Session: During a lengthy code refactoring task involving context sizes from 30K to 90K tokens, throughput ranged from 19 to 36 tok/s, demonstrating practical usability.
These results represent around a 2x to 3x speedup over previous performance levels, enabling faster iteration and making large-scale local inference more feasible on mature hardware.
Methodology: Simplified Deployment
The author provides a straightforward installation script, eliminating the need for Xcode or manual compilation:
bash
curl -fsSL https://github.com/paperniuk/splash/releases/download/1.0.2-m1/install-m1.sh | sh
splash-m1 serve --model incoai/Qwen3.8-27B-Splash
This installs the modified engine alongside the official Splash, sharing model caches seamlessly, and allows for quick deployment and testing.
Broader Implications and Future Considerations
- The impressive speedup on an M1 Max indicates that even five-year-old hardware can handle large language models effectively with tailored optimization.
- The techniques used—custom kernel development and precise arithmetic handling—can inspire similar optimizations for other models and hardware configurations.
- Preliminary tests suggest that M2 chips might benefit further from these modifications, although further verification is needed.
Concluding Remarks
This project demonstrates that through targeted software engineering, existing Apple Silicon hardware remains highly capable of demanding AI inference tasks. As the ecosystem evolves, such optimizations could become standard practice, democratizing access to powerful language models without requiring expensive server infrastructure.
For developers interested in replicating or building upon this work, the code repository with custom kernels is available here. Contributions and testing on newer hardware are encouraged to extend these performance benefits further.
Author: [Your Name], AI & Hardware Optimization Enthusiast
Published: [Date]
