AI Engineer · advanced
Multimodal RAG for Enterprise Knowledge
A Retrieval-Augmented Generation pipeline that handles documents with text AND images — charts, tables, diagrams.
Project overview
Proves you can handle "messy" real-world data and build production-grade RAG systems that provide accurate citations — a critical skill for AI Engineers working with proprietary data.
Real-world use case
Internal company wikis or customer support bots that need to answer based on private, frequently updated technical manuals rather than general internet knowledge.
Tools and technologies
- LlamaIndex
- Pinecone
- Python
- Claude 3.5
Skills you can demonstrate
Multimodal RAG, Vector DBs, Image/Text Embeddings
Build roadmap
- Source messy enterprise-style docs
Grab SEC 10-Ks, research papers, or technical manuals — anything with tables, charts, and text mixed together. - Choose your multimodal embedding model
CLIP for images + OpenAI or Voyage embeddings for text. Or use a unified model like Cohere multimodal. - Build the ingestion pipeline
Parse PDFs with LlamaParse or Unstructured.io. Extract text, tables, and images separately and embed each. - Set up Pinecone with metadata
Store chunks with metadata (page number, source, type=text/image/table) so you can cite accurately. - Build the query layer
Multi-step retrieval: search text, search images, re-rank, then send top-k to Claude 3.5 or GPT-4o with citations. - Run a hallucination evaluation
Create 50 test questions with known answers. Use RAGAs to measure faithfulness. Publish the eval report — this is what separates production RAG from toy RAG.
Ways to extend the project
Implement "Agentic RAG" where the system can decide to search the web or call an external API if the local knowledge base doesn't have the answer.
How to showcase your work
- GitHub: Include a "Hallucination Evaluation" report showing how different chunking strategies affected quality.
- LinkedIn: Post a side-by-side video: the system answering a question about a chart inside a PDF, with the cited page + region highlighted in real time.
- Resume: Highlight production RAG and citation accuracy.