Flash-MoE: Running a 400 Billion Parameter Model on a MacBook

Author: chase_ai_

Running a 400 billion parameter model typically requires a server rack. Someone just did it on a MacBook Pro with 48 gigabytes of RAM. The project is called Flash-MoE (referred to as "Flashmo" in the video), and it runs the Qwen 3.5-397B-A17B mixture-of-experts model at production-quality output with tool calling support. The entire engine was built in 24 hours using Claude Code running experiments autonomously, based on an Apple research paper.

How It Works

The key insight is that you do not need to load the entire 209 gigabyte model into memory. Flash-MoE streams model weights from SSD to GPU on demand through a custom Metal compute pipeline. This is not Python. There are no frameworks involved. The engine is pure C, Objective-C, and hand-tuned Metal shaders running on Apple Silicon.

The Flash-MoE GitHub repository showing the project README with benchmark results and configuration table

The approach works because modern SSDs are fast enough to feed weights to the GPU in real time, provided the data pipeline is optimized aggressively. The project achieves around 4.4 tokens per second at 4-bit quantization with the FMA kernel optimization, which is the production configuration. A faster 2-bit configuration hits 7 tokens per second but breaks JSON and tool calling, making it impractical for agent use.

The Architecture

The engine uses several techniques to make SSD-to-GPU streaming viable. SSD expert streaming loads 20GB weights in 4-bit from NVMe with GCD dispatch groups. Only the 4 active experts per layer are loaded at roughly 6.75MB each. An FMA-optimized dequant kernel handles the inner loop math. Hand-written Metal compute shaders replace what would normally be framework overhead. The OS page cache is trusted entirely with no custom caching layer, following the "LLM in a Flash" paper from Apple.

Detailed architecture section showing key techniques including SSD streaming, Metal shaders, and deferred GPU compute

The Benchmarks

The results table shows multiple configurations tested across 50+ experiments. The production configuration (4-bit experts with FMA kernel) delivers 4.36 tokens per second at excellent quality with full tool calling. Before the FMA kernel optimization, the same configuration ran at 3.90 tokens per second. The 2-bit configurations are faster (5.74 and 7.05 tok/s) but sacrifice either JSON reliability or tool calling support.

Benchmark results graph and configuration table showing token speed vs quality tradeoffs

Key Takeaways

A 397 billion parameter model can run on consumer hardware (MacBook Pro M3 Max, 48GB RAM) at usable speeds
The trick is streaming weights from SSD to GPU on demand rather than loading everything into memory
The engine was built in 24 hours with Claude Code running autonomous experiments based on Apple's "LLM in a Flash" research
Pure C/Metal implementation with no Python or framework dependencies keeps the overhead minimal
4-bit quantization is the practical sweet spot: 4.4 tok/s with full tool calling and excellent quality