LangChain's Document Chunking Strategies for Better RAG

If your RAG system is hallucinating, the problem is usually in how you chunk your documents. LangChain provides specialized splitters for each document type, and using the right one for each format is one of the most effective fixes for retrieval quality. The key insight: every chunk is a document with page content plus metadata, and that metadata is what keeps retrieval grounded.

Overview

LangChain treats chunking as document-type-specific. Rather than applying one generic splitter to everything, you pick the strategy that matches your source format. Each approach preserves different kinds of structural metadata that improve retrieval accuracy downstream.

Plain Text Documents

Start with the default RecursiveCharacterTextSplitter. It tries to keep paragraphs intact, then falls back to sentences, then to words. This is the baseline for unstructured text.

PDFs

If the PDF has selectable text, load the pages and split the documents directly. If it is a scanned PDF (images of text), chunking cannot help until you run OCR first. PyMuPDFLoader can handle OCR for embedded images using RapidOCR or Tesseract. Once the text is extracted, it becomes chunkable.

HTML and Web Content

First load the page as a document. It already includes metadata like source, title, and language. Then chunk structurally using HTMLHeaderTextSplitter, which splits by headings and stores the header trail in metadata. If chunks are still too big, compose the header splitter first, then apply RecursiveCharacterTextSplitter to enforce a maximum chunk size.

Markdown

Same principle as HTML: split by headers so every chunk carries its heading context in metadata.

JSON

Do not flatten JSON into plain text. Two options: use JSONLoader to load specific fields and add custom metadata, or use RecursiveJsonSplitter for big nested structures so it walks the tree and keeps objects together.

Code

Use language-aware splitting so chunks respect function and class boundaries rather than breaking mid-definition.

Key Takeaways

• Use document-type-specific chunking strategies rather than one generic splitter for everything
• Metadata (source, page, headers, paths) is what keeps retrieval grounded and reduces hallucinations
• Scanned PDFs require OCR before chunking can help; PyMuPDFLoader with RapidOCR handles this
• For HTML and Markdown, split by headers first, then enforce size limits as a second pass
• For JSON, preserve structure rather than flattening; for code, respect function and class boundaries

Resources

• LangChain Text Splitters Documentation -- Official docs for all LangChain chunking strategies
• PyMuPDF -- PDF loader with OCR support for scanned documents