How Uber Fixed Their RAG Chatbot After 60% of Answers Were Wrong

Uber's AI chatbot for internal security policies was giving wrong or misleading answers 60% of the time. They identified four root causes and implemented four corresponding fixes. The fixes build on each other, with each layer improving the effectiveness of the next.

Fix 1: Ditch PDFs, Migrate to Google Docs

Uber tried every PDF loader available: PyPDF, PDFPlumber, LlamaParse. None could handle tables spanning five or more pages. Formatting was destroyed and chunks lost all context.

The fix was to abandon PDFs entirely. They migrated all security policy documents to Google Docs and built a custom loader using Google's Python API. This preserved tables, bullets, and document structure.

Fix 2: Enrich Chunks with LLM-Generated Metadata

Raw chunks with just a title and URL were not enough. Uber's security policies have subtle differences across regions and personas, and the retriever had nothing meaningful to filter on.

The fix was to use an LLM to enrich every chunk with generated summaries, FAQ sets, and relevant keywords. They also indexed access control metadata so answers respect document permissions.

This enriched metadata became the backbone of everything that followed.

Fix 3: Add BM25 Keyword Search Alongside Vector Search

Vector search alone missed critical documents. Semantic similarity could not tell apart policies that looked similar but applied to different teams.

The fix was hybrid search: BM25 keyword search running alongside vector search, with results merged. BM25 leverages those enriched summaries, FAQs, and keywords from the previous step, so the metadata enrichment directly powers better retrieval.

Fix 4: Agentic Query Processing with LangGraph

Even with hybrid search, vague queries returned noisy results. Uber built three agents using LangGraph:

1. Query optimizer rewrites vague questions into precise ones
2. Source identifier uses few-shot examples to narrow which documents to search
3. Post-processor deduplicates and reorders chunks by their original document position before the LLM sees them

Key Takeaways

• PDF parsers cannot reliably handle complex document structures like multi-page tables; migrating to a structured format (Google Docs) with a custom loader solved the ingestion problem
• LLM-generated metadata (summaries, FAQs, keywords) on every chunk dramatically improves retrieval quality
• Hybrid search (vector + BM25) catches documents that semantic similarity alone misses
• Agentic query processing (rewriting, source filtering, reordering) cleans up the last mile of noisy retrieval

Resources

• LangGraph -- Framework for building stateful multi-agent applications
• Uber Engineering Blog -- Source for Uber's technical case studies