Surprisingly fast Qwen3.6-35B-A3B on a 4 GB VRAM!

Optimizing Large Language Model Performance on Limited GPU Memory: Case Study with Qwen 3.6-35B on a 4 GB VRAM GPU

In the rapidly evolving field of artificial intelligence, deploying large language models (LLMs) on consumer-grade hardware remains a significant challenge. However, recent experimentation demonstrates that with careful configuration and tuning, it’s possible to run sizable models efficiently even on modest hardware setups. This post shares insights from testing the Qwen 3.6-35B-A3B model on a laptop equipped with just 4 GB of VRAM, highlighting the techniques and parameters that enabled surprisingly fast performance.

Hardware Configuration

The test system comprises a Dell laptop featuring an NVIDIA GeForce GTX 1650 with 4 GB VRAM, backed by a 6-core (12-thread) CPU and 32 GB of system RAM. This hardware is typically considered insufficient for deploying large language models, yet with the right approach, it can still deliver meaningful results.

Model Acquisition and Setup

The model used in this experiment was sourced from the Atomic repository:
hf download AtomicChat/Qwen3.6-35B-A3B-UDT-MTP-GGUF Qwen3.6-35B-A3B-UDT-Q4_K_XL_MTP.gguf
This particular version outperformed other repositories in testing speed, highlighting the importance of model source selection.

Configuration Parameters and Optimization Strategies

The following command-line parameters were employed with llama.cpp to optimize performance:

bash
/opt/llama.cpp/build/bin/llama-server \
-m /opt/llama.cpp/models/Qwen3.6-35B-A3B-UDT-Q4_K_XL_MTP.gguf \
--host 0.0.0.0 \
--port 8080 \
-c 64000 \
-ngl 999 \
--override-tensor "blk..*\.ffn_(gate_up|gate|up|down)_exps\.weight=CPU" \
--flash-attn on \
--cache-type-k q8_0 \
--cache-type-v q8_0 \
--mlock \
-b 3072 \
--ubatch-size 3072 \
-ctxcp 128 \
--reasoning off \
--parallel 1 \
-t 6 \
-tb 12 \
--cache-ram 16384 \
--swa-full \
--no-kv-unified \
--spec-type none \
--cache-reuse 256

Key considerations include:

1. Model Source: The choice of the Atomic repository appeared to yield faster inference times, emphasizing the impact of model files and their optimization.

2. Context Window: Setting a 64,000 token context (-c 64000) was facilitated by configuring the Hermes AI agent, enabling larger prompt handling and reduced reprocessing.

3. Tensor Overrides: Custom tensor overrides (--override-tensor) provided granular control over model behavior and resource utilization during inference.

4. Memory Management: Parameters such as mlock and cache-ram optimize memory use, reducing latency.

5. Batching and Parallelism: Increasing -b and --ubatch-size to 3072 significantly improved prompt throughput, reaching speeds of approximately 116 tokens per second.

6. Additional Settings: Disabling reasoning and fine-tuning parallel execution (--parallel 1) contributed to stable performance on limited hardware.

Performance Achievements

With these configurations, the model achieved throughput of roughly 19 tokens per second during initial testing, with further optimization reducing processing times to over 100 tokens per second. Such performance is impressive given the hardware constraints, illustrating that practical deployment of large models on entry-level GPUs is feasible with meticulous tuning.

Conclusion and Recommendations

This experience demonstrates that deploying and running large language models like Qwen 3.6-35B on hardware with only 4 GB VRAM is achievable, provided that careful parameter adjustments are made. For practitioners:

• Experiment with different model repositories for potential speed benefits.
• Leverage large context windows where necessary, using compatible agent setups.
• Fine-tune batching and caching parameters to maximize throughput.
• Be mindful of memory management options to prevent bottlenecks.

Future improvements could involve exploring model quantization, further memory optimizations, and efficient batching strategies. Feedback and shared experiences from the community are encouraged to refine these techniques further.

Feel free to reach out with questions or insights on optimizing large language model deployment on limited hardware.