project · 2024 · ML Engineer · SAP
Human-in-the-Loop Feedback Service for AI Response Evaluation
A production service for capturing structured human feedback on AI-generated search responses, supporting binary, categorical, free-text, and preference signals to close the quality loop on enterprise retrieval systems.
An AI search system you cannot measure is one you cannot improve. Without a structured way to capture what the model got right, what it got wrong, or which responses users actually preferred, quality analysis relies on guesswork. This service provides the data layer for closing that loop, collecting multi-signal feedback from domain experts at the point of interaction and making it available for continuous model evaluation.
Architecture
┌──────────────────────────────────────────────────┐│ Frontend / Client ││ (enterprise AI tooling, search UI) │└───────────────────────┬──────────────────────────┘ │ POST /feedback ▼┌──────────────────────────────────────────────────┐│ FastAPI Service ││ ││ Validation → Hash ID generation → Business logic │└──────────────┬────────────────────────────────────┘ │ ▼┌──────────────────────────────────────────────────┐│ PostgreSQL ││ ││ Prompts · Responses · Feedback · Session state ││ Categorical taxonomy · Role registry │└──────────────────────────────────────────────────┘What gets collected
The service accepts four feedback signal types in a single request, letting the caller provide as much or as little signal as the interaction warrants:
- Binary feedback: a simple positive or negative signal on the response
- Categorical feedback: structured labels across predefined quality dimensions (accuracy, relevance, completeness, factual correctness, style) that give the evaluation team actionable signal beyond a binary score
- Free-text feedback: open-ended notes for cases the taxonomy does not cover
- Preference ranking: an explicit ordering when multiple response candidates are shown, capturing which output the user found most useful
Role-aware collection is built in from the start. Different user types submit feedback under their own role, allowing quality analysis to be segmented by domain expertise rather than treating all feedback as equivalent.
Idempotency and state tracking
A key design decision was using SHA256-based hash IDs rather than auto-increment integers. The session request ID is derived from the user session and query, and the feedback ID adds the response index and timestamp. This means the same query from the same session always maps to the same record, enabling deduplication across distributed clients and making re-submissions safe without complex client-side coordination.
When a user revises a query and resubmits, the service archives the current response-to-feedback mapping to a previous-state field, clears the active map, and begins tracking the new round of responses. This preserves the full history of feedback across query iterations without discarding earlier signals.
Transactional integrity
All validation runs inside the database transaction. If any check fails (missing required fields, an unrecognised label code, or an unknown caller context), the transaction rolls back atomically before any partial write reaches the database. This keeps the feedback dataset clean without requiring a separate cleanup pipeline downstream.
Deployment
The service runs as a containerised FastAPI application on Kubernetes with PostgreSQL as the backing store, accessed through a connection pooler for efficiency at scale. Credentials are injected from cluster secrets, the container runs as a non-root user, and log sanitization is applied to all request data. CI/CD runs tests and coverage checks as part of the Docker build, so a failing test blocks the image from being promoted.