Building a Smart PDF Parser for Complex Visual Documents with Fine-Tuned Vision Models

A client needed to parse highly complex PDFs containing hand-drawn architecture sketches, science plots, engineering diagrams, and dense tables, then convert all of it into structured markdown. OCR alone was not enough. They needed semantically accurate descriptions of images and diagrams, and everything had to run on limited inference compute using open-source tools only.

Choosing and Fine-Tuning the Model

After testing several options, the choice landed on Qwen 2.5 VL 7B, a vision-language model that gave decent results out of the box but needed fine-tuning for production quality.

The problem was that no labeled training data existed. The solution: use LlamaParse, an advanced agentic parser, configured with Gemini 2.5 Pro for image understanding. This setup extracted detailed page-level text and image pairs from the client's public PDFs, creating a synthetic training dataset.

The Fine-Tuning Pipeline

With synthetic data in hand, a custom chat template was applied and everything was prepped for training. The Unsloth framework handled parameter-efficient fine-tuning across multiple iterations, with hyperparameters optimized to fit GPU memory limits. All training runs were logged to Weights & Biases.

The result was a fine-tuned Qwen 2.5 VL model served through vLLM for fast inference.

The Production API

A FastAPI service wraps the model. For each page in a PDF, the system converts it to an image, sends it to the model through a custom OpenAI-compatible endpoint, and returns the result with page metadata. The API follows production best practices: modular code, async I/O, Pydantic validation, structured logging, and error handling.

A PubSub integration was added so the system can listen for events, fetch PDFs from S3, and trigger parsing automatically instead of requiring manual uploads.

Key Takeaways

• Qwen 2.5 VL 7B was fine-tuned with synthetic training data generated by Gemini 2.5 Pro via LlamaParse
• No labeled training data existed, so the synthetic data pipeline was the critical piece
• Unsloth handled parameter-efficient fine-tuning within GPU memory constraints
• The production API uses FastAPI with async I/O, vLLM for inference, and PubSub for automated triggers
• The full pipeline covers data generation, model training, API serving, and event-driven ingestion

Resources

• Qwen 2.5 VL -- Vision-language model used as the base for fine-tuning
• LlamaParse -- Agentic PDF parser used to generate synthetic training data
• Unsloth -- Framework for parameter-efficient fine-tuning
• vLLM -- High-throughput inference engine for serving the model
• FastAPI -- Python web framework for the API layer