project · 2024 · ML Engineer · SAP
Recommendation Quality Analysis for Enterprise Search
Root-cause investigation into recommendation accuracy degradation across heterogeneous data sources, identifying data pipeline inconsistencies as the primary driver and restoring parity through targeted pre-processing.
Recommendation systems in production degrade in ways that are not always obvious from model metrics alone. This investigation started from an observed discrepancy: the same underlying model was producing noticeably different recommendation outputs depending on which data source was being queried. The task was to find out why.
Tracing the divergence
The analysis identified three distinct root causes, each responsible for a different slice of the accuracy gap.
The first was anonymization inconsistency. Sensitive tokens in one source were being replaced with placeholder tags while the other retained the original text. This meant semantically identical records looked textually different to the retrieval model, producing different ranking results. The effect was small in most periods but spiked significantly in others, affecting a meaningful share of the record set.
The second was incomplete or over-aggressive anonymization in the opposite direction: cases where the anonymization pipeline was stripping meaningful terms that were not personal data, or missing tags entirely. Both failure modes degraded the input quality fed to the recommendation model.
The third was structural noise in one of the text fields: a log field that mixed human-authored text with automated system messages. This produced the largest accuracy gap of the three, because the model was reasoning over heavily polluted input with no signal of which content was relevant.
What the numbers showed
Accuracy was measured across multiple ranking thresholds. After filtering affected records and cleaning input text, recommendation accuracy between the two sources converged closely, confirming the issues were data-side, not model-side. The pre-processing changes required to achieve parity were targeted and low-risk, which meant they could be deployed without model retraining.
Why this kind of work matters
Recommendation quality investigations like this sit at the intersection of data engineering and applied ML: the model is fine, but the pipeline feeding it is not. Identifying that distinction early prevented a misdirected effort to retrain or replace a model that was already performing correctly on clean input.