Hey! Welcome back. Today we look at how we can get our RAG system ready for scale. We discuss common problems and their solutions, when you introduce more users and more requests to your system. For this we are joined by Nirant Kasliwal, the author of fastembed. Nirant shares practical insights on metadata extraction, evaluation strategies, and emerging technologies like Colipali. This episode is a must-listen for anyone looking to level up their RAG implementations. "Naive RAG has a lot of problems on the retrieval end and then there's a lot of problems on how LLMs look at these data points as well." "The first 30 to 50% of gains are relatively quick. The rest 50% takes forever." "You do not want to give the same answer about company's history to the co-founding CEO and the intern who has just joined." "Embedding similarity is the signal on which you want to build your entire search is just not quite complete." Key insights: Naive RAG often fails due to limitations of embeddings and LLMs' sensitivity to input ordering.Query profiling and expansion: Use clustering and tools like latent Scope to identify problematic query typesExpand queries offline and use parallel searches for better resultsMetadata extraction: Extract temporal, entity, and other relevant information from queriesUse LLMs for extraction, with checks against libraries like Stanford NLPUser personalization: Include user role, access privileges, and conversation historyAdapt responses based on user expertise and readability scoresEvaluation and improvement: Create synthetic datasets and use real user feedbackEmploy tools like DSPY for prompt engineeringAdvanced techniques: Query routing based on type and urgencyUse smaller models (1-3B parameters) for easier iteration and error spottingImplement error handling and cross-validation for extracted metadataNirant Kasliwal: X (Twitter)LinkedInSearch in the LLM Era for AI Engineers (course)Nicolay Gerold: ⁠LinkedIn⁠⁠X (Twitter)query understanding, AI-powered search, Lambda Mart, e-commerce ranking, networking, experts, recommendation, search