project · 2025 · Data Scientist · SAP
Auto Prompt Tuning for RAG Retrieval
A prompt optimization system for enterprise RAG pipelines, using grid search, random search, and reinforcement learning to automatically discover retrieval prompts that consistently surface the most relevant documents.
Writing good retrieval prompts by hand is iterative and fragile. A phrasing that works well for one query pattern breaks on another, and there is no principled way to know when you have found the best version. This project built a system to automate that search: given a retrieval task and a set of historical problem-solution pairs as ground truth, it finds the prompt instructions that maximize retrieval accuracy without human trial and error.
The optimization loop
┌─────────────────────────────────────────────────────┐│ Historical Problem-Solution Pairs ││ (ground truth for evaluation) │└───────────────────────┬─────────────────────────────┘ │ ▼┌─────────────────────────────────────────────────────┐│ Prompt Variant Generator ││ ││ Grid Search · Random Search · LLM Mutation │└───────────────────────┬─────────────────────────────┘ │ ▼┌─────────────────────────────────────────────────────┐│ Retrieval Evaluation ││ ││ Query LLM with prompt + input + candidates ││ Compare output to ground truth ││ Compute reward (correct / incorrect) │└───────────────────────┬─────────────────────────────┘ │ ▼┌─────────────────────────────────────────────────────┐│ Ranking and Pruning ││ ││ Score all variants by reward ││ Keep top performers for next generation │└───────────────────────┬─────────────────────────────┘ │ ┌──────┴──────┐ ▼ ▼ Best Prompt Next Generation Selected of VariantsThree search strategies
The system supports three approaches to generating and evaluating prompt variants, each with a different cost-coverage tradeoff.
Grid search enumerates all combinations of a predefined set of prompt parameters: things like instruction phrasing, emphasis style, output format, and level of context provided. Every combination is evaluated against the ground truth dataset and scored. It is thorough when the parameter space is small and well-defined, but the number of combinations grows fast, making it impractical for large or open-ended parameter sets.
Random search samples a random subset of combinations from the same parameter space rather than exhaustively covering all of them. Because most of the signal in a parameter space tends to come from a small number of dimensions, random search often finds near-optimal prompts in a fraction of the evaluations that grid search would require. It scales better and is the practical default when the parameter space is large.
Reinforcement learning with PPO moves beyond fixed templates entirely. The prompt selector is treated as a policy: it observes the input context (query type, metadata), selects a prompt template as its action, receives a binary reward signal (correct document retrieved or not), and updates its weights via Proximal Policy Optimization to increase the probability of selecting high-reward prompts in similar future contexts. PPO’s clipped objective prevents the policy from making large destabilizing updates, which is important when rewards are sparse and the prompt space is noisy. A multi-armed bandit formulation provides a lighter-weight alternative for simpler settings where the context does not vary enough to justify a full policy network.
PPO framing in detail
The RL components map cleanly onto the retrieval problem:
- Agent: the prompt selector, a policy network that learns which template to use
- Environment: the LLM and the ground truth evaluator
- State: the input query and any available metadata about the request
- Action: choosing a specific prompt template or set of instruction parameters
- Reward: +1 if the LLM selects the correct document, 0 otherwise (or a softer ranked reward if partial credit is appropriate)
Each training episode samples a batch of historical query-solution pairs, runs the current policy to select a prompt, queries the LLM, evaluates the result, and collects (state, action, reward) tuples. After each batch, the policy network is updated using advantage estimation: the update increases the probability of actions that performed better than average for that context and decreases those that performed worse, while the PPO clipping constraint keeps updates within a safe range to avoid unstable learning.
Self-evolving prompts
Beyond fixed-template search, the system implements a self-play loop where the LLM itself generates new prompt variants. The top-ranked instructions from each evaluation round are fed into a meta-prompt that asks the model to mutate, extend, or improve them based on patterns in what worked. Each new variant is evaluated against the ground truth, scored, and either promoted to the next generation or discarded.
The key design insight is that the LLM already encodes knowledge about what makes instructions clear and effective. Rather than relying on a human to enumerate variations, you give it a set of high-performing examples as a reference and ask it to reason about what made them work. The result is a search process that can discover phrasings outside any fixed template library, converging on instructions that are specifically calibrated to the characteristics of the target retrieval task.
Why retrieval, not generation
The focus is deliberately narrow: optimizing how the model selects the right document, not how it phrases the final answer. Retrieval accuracy is measurable, ground truth is available from historical data, and improvements here compound across every downstream use of the retrieved content. Getting document selection right is the highest-leverage intervention in a RAG pipeline because no amount of generation quality can compensate for retrieving the wrong source.