Doing RAG Right: Key Lessons For Production-Ready AI
Jan Vanalphen
Head of AI Strategy
Retrieval-Augmented Generation (RAG) has emerged as a crucial design pattern that addresses the core limitations of Large Language Models (LLMs). At its heart, RAG serves three critical purposes:
- It enables AI models to incorporate specific domain contexts without retraining the model,
- Grounds responses in trustworthy references to minimize hallucinations, and
- Keeps systems current by integrating new information through the retrieval process rather than model retraining.
In the business context, it is the killer application for LLMs and that is why it caught everyone's attention in the last two years. While Generative AI thrives on unstructured data, many companies hit a glass ceiling when moving from simple prototypes to real-world applications in production.
The barriers to breaking the glass ceiling are typically issues like inconsistent accuracy, where responses fail to pull relevant information reliably; difficulty handling nuanced business contexts, causing the AI to produce generic or off-target outputs; inconsistent user experience, where quality varies significantly, confusing users and undermining trust in the system; and missed optimization opportunities, which leave workflows underperforming in both speed and cost. Regular evaluations address these challenges, aligning RAG workflows with your organization’s specific needs and enabling your AI to perform with accuracy and relevance in real-world applications.
That’s why moving beyond basic prompt management and LLM logging to a disciplined, repeatable approach for measuring accuracy and relevance is non-negotiable. Doing RAG right means building a foundation where your AI learns from your data, adapts to your needs, and performs consistently in the real world.
Lessons from continuous evaluation
Over the past two years, we’ve zeroed in on refining RAG workflows to build robust knowledge bases, tackling every challenge standing in the way of reliable, accurate output through deep evaluation.
Through this process, we’ve distilled our key takeaways - shared in the hope they’ll benefit any technical team striving for production-ready LLM systems.
- Establish a Baseline Before RAG Integration
Begin by independently evaluating a generic Large Language Model (LLM) to establish baseline performance metrics. This benchmark is crucial for assessing the effectiveness of your Retrieval-Augmented Generation (RAG) system. Your RAG workflow, enhanced with additional context, should demonstrably outperform this baseline. If it doesn't, the baseline will help you pinpoint areas needing improvement. - Start simple and focus on retrieval evaluation first
Start with basic retrieval approaches before layering on complex tools or algorithms. Focus on making sure retrieval is working as expected first. Evaluate if the relevant documents necessary to answer the question are correct, and only then focus on the answer generation. Fundamental ranking and filtering techniques often reduce noise and boost initial performance. Focusing on simple retrieval methods allows you to identify and resolve fundamental issues before introducing advanced algorithms. - Narrow Search Space by Leveraging Metadata
In databases with millions of document chunks, metadata filtering is essential. By narrowing the search space from millions to thousands of documents, metadata filtering improves both performance and relevance. Well-organized, metadata-driven retrievals ensure that model responses are accurate and deeply grounded in context. - Involve Domain Experts in Data Processing
Collaborate with domain experts to validate chunk parsing and metadata extraction to ensure that the data is structured correctly, which is crucial for reliable results. Don't assume off-the-shelf embedding models will work perfectly for your use case; test models that make sense to your domain and consider fine tuning your own embeddings. At this point, it is crucial to have retrieval evaluation in place to assess the most efficient embeddings models. - Plan & Execute with a reasoning agent
Instead of diving straight into answers, break complex questions into clear tasks and delegate these tasks to Agents who are specialised in relevant fields. This ‘plan-and-execute’ structure creates focused queries for each step, improving the systems flexibility and significantly increasing the precision and reliability of the results. - Optimize with Specialized Agents
Configure specialized retrievers-Agents for specific subject matters, document types or taxonomies. This focus allows the RAG to pull from the most relevant subset of documents, reducing irrelevant retrievals and speeding up responses.
Ideal RAG Architecture with Agents
- Organize Code for Modularity and Debugging
Structure your code modularly to create flexible workflows. Each module should handle a distinct part of the process and be reusable or adjustable as needed. Modular code enables efficient debugging with step-by-step monitoring and simplifies maintenance. - Balance Model Choice with Cost Efficiency
Select models that balance performance with cost efficiency. Use cheaper models for simpler tasks, like query routing, while reserving high-performing models for final responses. This strategy ensures accuracy where it matters most while controlling costs in complex workflows.
Conclusion
In RAG, data structure reigns supreme: well-organized, metadata-driven retrievals ensure that the model’s responses are accurate and deeply grounded in context. From is experience, we have learned that if you’re not evaluating at every stage and optimizing data structure, you’re not truly engineering for production - you’re just hoping for the best.
The path forward is clear: embrace rigorous, ongoing evaluation. Only then can we build AI that performs consistently in the complexities of the real world.
