project · 2025 · Data Scientist · SAP
Enterprise Retrieval-Augmented Generation (RAG) for Support Intelligence
A production retrieval-augmented generation service that matches enterprise support queries against technical documentation, selects the most relevant documents using LLMs, and generates precise answers, deployed as a containerized microservice with multi-protocol streaming.
Most RAG systems treat retrieval and generation as two independent steps with a simple handoff between them. This system is more deliberate: every decision in the pipeline, from which documents to surface to how the LLM is instructed to reason about them, is explicitly engineered rather than left to defaults. The result is a production service that routes support queries across multiple documentation types, selects documents using the LLM itself as a ranker, and generates grounded answers with citations.
Pipeline architecture
┌─────────────────────────────────────────────────────────┐│ Incoming Query ││ (title / description / doc type) │└───────────────────────┬─────────────────────────────────┘ │ ▼┌─────────────────────────────────────────────────────────┐│ Retrieval Layer ││ ││ External search API → OpenSearch enrichment ││ Score thresholding → candidate document set │└───────────────────────┬─────────────────────────────────┘ │ ▼┌─────────────────────────────────────────────────────────┐│ Input Anonymization ││ (PII scrubbed before any LLM call) │└───────────────────────┬─────────────────────────────────┘ │ ▼┌─────────────────────────────────────────────────────────┐│ Document Selection ││ ││ LLM Ranker (picks most relevant docs by ID) ││ OR Semantic Retriever (embedding similarity) │└───────────────────────┬─────────────────────────────────┘ │ ▼┌─────────────────────────────────────────────────────────┐│ Answer Generation ││ ││ Prompt template (default or custom) ││ Streaming output: SSE / WebSocket / JSON ││ Citations extracted and returned separately │└───────────────────────┬─────────────────────────────────┘ │ ▼┌─────────────────────────────────────────────────────────┐│ Logging and Persistence ││ PostgreSQL audit trail for all request outcomes │└─────────────────────────────────────────────────────────┘Document selection as a first-class problem
The hardest part of a RAG system is picking the right document, not generating fluent prose. The service handles multiple documentation types: technical articles, help documentation, community posts, and operational records. Each type has its own relevance threshold, field schema, and ID format. Document selection adapts to the type in two ways.
For technical articles, picking and generation are combined into a single LLM call. The model is prompted to review the problem, select the most directly relevant documents, summarize them, and return structured references, all in one pass. This reduces latency and cost for the most common query pattern.
For other documentation types, the pipeline splits into two calls: the LLM first returns a list of selected document IDs, then a second call generates the answer from those documents. This separation gives more control when working with custom prompt templates or mixed document types.
Regex-based ID extraction handles the diversity of document identifier formats across types. A fallback to semantic similarity matching provides robustness when the LLM picking call fails.
Prompt engineering
Prompts are the operational core of the system, not an afterthought. The design principles:
- Strictness over creativity: every generation prompt instructs the model to use only the provided documents and not extrapolate. This prevents hallucinated answers and keeps outputs grounded in retrieved content.
- Validity check first: the model is asked to verify whether the problem statement is specific enough to match against before attempting selection. Vague queries get a clear error message rather than a plausible-sounding non-answer.
- Custom templates: prompt templates support both simple strings and role-split (system / user / assistant) formats, so domain-specific integrations can inject their own instructions without touching core logic.
- Citation triggers: specific document IDs can trigger additional instruction injection, enabling use-case-specific guidance for known complex documents.
Model support and benchmarking
The service routes across several LLMs depending on latency, cost, and quality requirements: GPT-based models for high-quality generation, Llama 3.1 variants (70B and 8B) for cost-sensitive or fast-picking paths, and an in-house large model for cases where external API usage is restricted.
Benchmark suites measure picking accuracy (does the selected set include the ground truth solution?) and generation quality (LLM-judged relevance and completeness). Picking accuracy across models was measured on several hundred test records, with results used to inform model routing decisions and prompt iteration.
Privacy and operations
User input is anonymized through an external PII scrubbing service before any LLM call. The anonymized text is what the model sees; the original query is never forwarded to an external endpoint. All request outcomes, including errors and null solutions, are logged to PostgreSQL for auditability and monitoring. The service runs as a containerized FastAPI application on Kubernetes with async I/O throughout, OAuth2 token management, and structured logging with per-function execution timing.