project · 2024 · ML Engineer · SAP
Human Feedback Loop for RAG Search
A scalable human-in-the-loop feedback system for SAP's ISM RAG and search pipelines, with binary, categorical, and fine-grained signals aligned with how Perplexity, GPT, and Bing collect feedback.
As an ML engineer at SAP I worked on the quality of ML-based recommendation systems and the ISM / Coveo search pipelines: benchmarking, fluctuation analysis, and root-cause investigations into what actually moves retrieval quality and ranking relevance.
Closing the loop
A search system you can’t measure is a search system you can’t improve. I designed and implemented a scalable human-feedback system (API + PostgreSQL) for the ISM RAG pipeline, supporting binary, categorical, and fine-grained feedback, modelled on the feedback patterns used by Perplexity, GPT, and Bing rather than inventing our own.
Data that teaches the model
I led data analysis, visualization, and dataset creation over large-scale customer-incident data so the feedback could feed back into better recommendations and search accuracy, and researched LLM feedback-collection methodologies to integrate human-in-the-loop signals into SAP’s RAG workflows.