Late Interaction Models for RAG: ColBERT, ColPali, and ColQwen Explained

If you've built RAG over PDFs, you know the pain: OCR errors, chunking that splits a chart from its caption, a single document embedding that flattens nuance into a 1,024-dim vector. Late interaction models are the current best answer to all three problems — and one of them (ColPali/ColQwen) skips OCR entirely.

The Core Trick: Many Embeddings Instead of One

Traditional dense retrieval compresses a whole document into a single vector. That's fast to search but destroys fine-grained structure. A 10-page financial filing becomes one 1,024-dim point in space — the same size as "hello world."

ColBERT (Contextualized Late Interaction Over BERT) rethought that. Instead of one embedding per document, it stores one embedding per word. A 500-word document becomes 500 embeddings, indexed in advance.

When you query, ColBERT:

1. Embeds each query word
2. For each query word, finds the best-matching document word
3. Sums those match scores

That late-stage token-level matching is where the name comes from — the interaction between query and document happens late, over fine-grained representations, not over pre-collapsed summary vectors.

ColPali: Same Idea, Applied to Pages as Images

ColPali takes ColBERT's multi-vector idea and moves it into the visual domain. The pipeline:

1. Convert each PDF page to an image
2. Feed the image through PaliGemma (a Google vision-language model)
3. Split the page into a grid of image patches
4. Emit multiple embeddings per page — one per patch

The result: no OCR, no chunking. A chart stays next to its caption. A table keeps its column headers. The layout itself is preserved in the embeddings — because the model is looking at the page the way a human does.

ColQwen: Same Pattern, Stronger Backbone

ColQwen is the same architecture as ColPali, but swaps PaliGemma for Qwen2-VL (Alibaba's vision-language model). In practice, ColQwen generally outperforms ColPali on retrieval benchmarks — the backbone VLM matters.

Model	Modality	Backbone	Best For
ColBERT	Text	BERT	Text-only late interaction
ColPali	Visual	PaliGemma	PDF pages with visual structure
ColQwen	Visual	Qwen2-VL	Same as ColPali, stronger results

The Trade-off: Storage

Storing N embeddings per page instead of one costs N× more space. That can get expensive on a large corpus. The mitigation mentioned in the video is MUVERA — a compression technique that converts multi-vector representations into fixed-length vectors you can throw into a standard vector database without losing much accuracy.

Where These Actually Win

The sweet spot is dense documents with visual structure:

• Financial reports (tables, charts, filings)
• Legal contracts (layout-heavy, footnote-heavy)
• Research papers (figures, equations, captions)

These are exactly the document types where classical chunk-and-embed RAG falls apart — where a chart's meaning is tied to its surrounding text and a table's semantics depend on its headers. Late interaction preserves those relationships natively because it's not trying to compress them into a single vector.

The video claims ~89% on document retrieval benchmarks, referring to results on the ViDoRe benchmark where ColPali-class models have been competitive with or ahead of traditional OCR+chunk+embed pipelines.

Key Takeaways

• Late interaction stores multiple embeddings per document (word-level for ColBERT, patch-level for ColPali/ColQwen) and matches them at query time
• ColPali and ColQwen treat PDF pages as images — no OCR, no chunking — preserving visual layout
• ColQwen generally outperforms ColPali by swapping PaliGemma for Qwen2-VL as the VLM backbone
• The main trade-off is storage; MUVERA compresses multi-vector representations to make them practical
• Best for financial, legal, and research documents where layout and visual structure carry meaning

Resources

• ColBERT Paper (Khattab & Zaharia, 2020) — Original late interaction paper
• ColPali Paper — Extending late interaction to visual document retrieval
• ColPali on HuggingFace — Model weights and demos
• Qwen2-VL — The VLM backbone used by ColQwen
• ViDoRe Benchmark — Visual document retrieval leaderboard