AirLLM Runs Llama 3.3 70B on a 4GB GPU by Streaming Layers from Disk, but Expect Tokens per Minute

The library the video calls "LLM" is AirLLM, an Apache-2.0 Python package by Gavin Li that runs 70B-class models on small GPUs by loading one transformer layer at a time from disk, and the tradeoff it never mentions is speed. The narrator says a 70B model can run on a MacBook or gaming PC with no supercomputer. That is true in the sense that it runs. It is not true in the sense most viewers will assume, which is that it runs at a usable pace.

What the video actually shows

The transcript names the tool as "an open-source Python library called LLM." The frames correct that. Frame 4 shows a GitHub repo with a rocket logo and the partially visible name "AirLLM." Frame 5 shows README badges (Code License Apache 2.0, a Gavin Li blog link) and an Updates section with entries from August 2024: v2.11.0 adding Qwen2.5, v2.10.1 adding CPU inference, and July 2024 support for Llama 3.1 405B plus 8-bit and 4-bit quantization. Those entries match the current README word for word, so the screenshot is real but stale. The live repo lists updates through September 2026, including LoRA training support and a 4.0.0 PyPI release.

Frame 7 shows a four-step diagram: receive prompt, tokenize and embed in CPU RAM, load the next layer from disk, transfer that single layer to the GPU and pass activations through. That is an accurate description of how AirLLM works. Frames 13 and 14 show the Hugging Face page for meta-llama/Llama-3.3-70B-Instruct, which exists and carries the Llama 3.3 Community License.

The transformer animations (the GPT-3 "175,181,291,520 total weights organized into 27,938 matrices" panel, the Attention and Multilayer Perceptron blocks, the 784-input digit classifier) are lifted from 3Blue1Brown-style explainer footage and are decoration, not evidence about AirLLM.

How layer streaming works

A 70B model in 16-bit precision needs roughly 140GB of weights. A single transformer layer of Llama 70B is under 2GB. AirLLM first decomposes the checkpoint into per-layer files on disk (the README warns this step is "very disk-consuming" and needs free space in the Hugging Face cache). At inference, it loads layer 1 to the GPU, runs the activations through it, frees it, loads layer 2, and repeats for all 80 layers, for every single token generated. The README says prefetching overlaps loading with compute for about a 10 percent gain. The bottleneck is disk read bandwidth, not GPU compute.

The Flash Attention claim

The narrator says AirLLM has "a feature called Flash Attention which keeps memory usage almost flat even with long inputs." I could not confirm this as an AirLLM feature. The current README mentions flash-attn only as a pip dependency required by one specific model (Kimi K3) because that model's own code demands it. There is no README section presenting Flash Attention as an AirLLM memory feature. FlashAttention is a real, separate technique that avoids materializing the full attention matrix, so the underlying claim about memory scaling has a basis. Attributing it to AirLLM as a headline feature does not.

What the video leaves out: speed

This is the part that changes the story. The README does not publish tokens-per-second figures, and I found none in the repo. Community reports are consistent, though. A Hacker News thread on AirLLM 70B inference on a 4GB GPU cites roughly 292 seconds per token. Third-party writeups indexed in search report a range of 0.5 to 3 tokens per second on faster hardware with quantization. Neither number is interactive. One HN commenter worked out that an hour of work at a normal 30 tokens per second would take over a year at the slow figure. I could not independently benchmark, so treat the exact numbers as unverified, but every source agrees on the order of magnitude: the disk is the inference engine, and a chat-style workflow is not realistic.

Apple Silicon

The claim that it runs on a MacBook is partially supported. The README says Apple Silicon is supported via MLX and PyTorch, with a note that only Apple Silicon (not Intel Macs) works and that a native Python install is required. The "gaming computer" claim is where the 4GB GPU figure comes from, and that is the configuration the community speed reports describe.

Key Takeaways

The tool is AirLLM (github.com/lyogavin/airllm), not "LLM." The video's transcript garbles the name; the frames show it.
It works by decomposing a model into per-layer files and streaming one layer at a time through the GPU for every token. That is why it fits in 4GB of VRAM.
Speed is the cost. Community reports range from a few tokens per second to minutes per token depending on disk, GPU, and quantization. None describe interactive use.
The "Flash Attention" feature claim is not supported by the AirLLM README. Prefetching is the documented optimization.
The README screenshot in the video dates from August 2024. The project is still maintained, with a 4.0.0 release in September 2026.
Llama 3.3 70B Instruct is real and available on Hugging Face under the Llama 3.3 Community License.

Resources

AirLLM on GitHub -- the repo shown in the video; README covers setup, supported models, quantization, and Apple Silicon notes
airllm on PyPI -- installable package, version 4.0.0 released September 2026
meta-llama/Llama-3.3-70B-Instruct on Hugging Face -- the model page shown in frames 13 and 14
AirLLM 70B inference with single 4GB GPU on Hacker News -- community discussion with real-world speed figures