Running a 120B Language Model Across Six Unlikely Devices — A DIY Success Story
In the rapidly evolving world of artificial intelligence, deploying large-scale models often requires high-end hardware and substantial infrastructure. However, what if you could stretch the limitations of your existing hardware and achieve something extraordinary? Recently, I embarked on an experiment to load OpenAI’s GPT-OSS-120B model—a colossal 120-billion parameter language model—across six diverse and underpowered devices. The results were surprising and educational, blending ingenuity with persistence.
The Challenge: Can a Cluster of Inappropriate Devices Handle 120B?
The GPT-OSS-120B model is a massive, quantized version (~60GB) designed to run efficiently with 4-bit quantization. It normally demands high-end GPU capacity, but I set out to test how far I could push a heterogeneous collection of hardware:
- A 12GB Windows laptop (more of a paperweight than a powerhouse)
- A mini PC equipped with an RTX 3060 GPU (12GB VRAM)
- A Mac mini (16GB RAM)
- An M3 MacBook (16GB RAM)
- A 2017 Intel-based MacBook running solely on CPU
- An Android device (smartphone)
Network connectivity was split: some devices connected via Ethernet, others on Wi-Fi. The goal? Load the model in a distributed manner, with no prior expectations of success.
Setting Up the Cluster: Distributed Loading of a Gigantic Model
The core idea was to run the model in a distributed setup. I designated the mini PC with the RTX 3060 as the primary node, responsible for orchestrating the process. It would load as many shards of the model as possible locally, then dynamically distribute the remaining shards to the other devices based on their capabilities.
The loading process was approximate and chaotic—taking about 11 minutes—mainly due to the need to transfer shards over Wi-Fi to slower devices. During this time, the cluster’s components gradually assembled their respective parts.
The Unexpected Outcome: It Worked
Once the shards were loaded, the model seamlessly came online. I tested it by asking questions, and it responded promptly, similar to how a typical large language model would. Several points stood out:
- Distribution Success: Five out of the six devices contributed activity, with the older Intel Mac surprisingly holding a shard despite its limited resources.
- Persistence: Despite the hardware limitations, the system was able to hold and serve the model effectively.
- Resilience: I left the setup running overnight. The cluster adjusted to network fluctuations, shifting shards as the phone went offline and back. It maintained operation without interruption or errors.
Insights and Lessons Learned
One key takeaway was the importance of designing load timeouts proportionally to model size rather than hardcoding a fixed limit. This flexibility could enhance robustness in similar distributed setups.
Additionally, this experiment showcased that even modest hardware combined cleverly can handle tasks traditionally reserved for high-end infrastructure—if approached creatively and with patience.
Final Thoughts: A DIY Demonstration of Distributed AI
This experiment profoundly challenges assumptions about hardware requirements for large AI models. While individual devices might fall short, a carefully coordinated cluster can achieve remarkable results.
For those interested in the full experience, including video demonstrations, check out these links:
This journey reinforces that innovation often lies in repurposing what we already have, pushing the boundaries of feasible AI deployment—even on hardware that traditionally wouldn’t handle such tasks.
