project · 2025 · Data Scientist · SAP
LLM-Powered Interaction Quality Analytics
A large-scale LLM classification pipeline over enterprise interaction data, categorizing outcomes and surfacing root causes behind unresolved cases to guide operational quality improvements.
Measuring interaction quality at enterprise scale is difficult. Determining whether a case was handled effectively, and understanding why it was not, requires reading unstructured conversation text across a large and continuously growing dataset. This project built a pipeline to do that automatically, using an LLM as the classifier and a distributed compute layer to handle the volume.
Pipeline overview
┌─────────────────────────────────────────────────────────────┐│ Distributed Data Lake ││ (multi-year interaction records, monthly ││ snapshots across full historical range) │└───────────────────────────┬─────────────────────────────────┘ │ ▼┌─────────────────────────────────────────────────────────────┐│ Pre-processing ││ ││ Deduplication → Empty Filter → Text Cleaning → Normalize │└───────────────────────────┬─────────────────────────────────┘ │ ▼┌─────────────────────────────────────────────────────────────┐│ LLM Classification ││ ││ ┌───────────────────────┐ ┌────────────────────────┐ ││ │ Outcome Judgement │ │ Failure Categorization │ ││ │ (resolved / not) │ │ (root cause label) │ ││ └───────────────────────┘ └────────────────────────┘ ││ ││ Quality Tracking Layer ││ (LLM errors · blank outputs · error rate) │└───────────────────────────┬─────────────────────────────────┘ │ ▼┌─────────────────────────────────────────────────────────────┐│ Labelled Dataset │└──────────┬──────────────────┬──────────────────┬───────────┘ ▼ ▼ ▼ ┌────────────────┐ ┌───────────────┐ ┌───────────────┐ │ Outcome Trends │ │ Failure │ │ Domain Area │ │ over Time │ │ Distributions │ │ Breakdowns │ └────────────────┘ └───────────────┘ └───────────────┘ │ ▼┌─────────────────────────────────────────────────────────────┐│ Stakeholder Report (HTML export) │└─────────────────────────────────────────────────────────────┘The data problem
The dataset covered multiple years of enterprise interactions, processed in evenly spaced periodic snapshots. Each record included unstructured conversation text alongside structured metadata. Producing a clean, analysis-ready dataset required deduplication to retain the latest version of each record, filtering out entries with empty content, cleaning conversational text to remove structural noise, and normalizing fields for downstream consumption. The pipeline read from a distributed data lake and produced consistent monthly outputs across the full historical range.
LLM classification at scale
For each interaction the pipeline invoked an LLM to make two judgements: whether the case reached a satisfactory outcome in the initial exchange, and if not, which failure category best described why it did not. Failure types covered a range of root causes including information gaps, incomplete initial responses, and follow-up signals that indicated the original response fell short.
Running this across a large multi-year corpus produced a labelled dataset with measurable error rates per period. A dedicated quality layer tracked LLM failures and blank outputs separately, so the reliability of the classification itself could be monitored and improved independently of the domain results.
Trend and breakdown analysis
With labelled data in hand, the analysis produced three layers of insight:
- Outcome trends over time tracking how resolution rates shifted across a multi-year window, with visible improvement in more recent periods
- Failure category distributions showing which root causes dominated each period and whether the composition of problems was shifting over time
- Domain-area breakdowns identifying which product and service areas drove the highest volume of unresolved interactions, to focus quality improvement efforts where they would have most impact
The visualization layer was built with a consistent design system and the final outputs were packaged as a self-contained report for stakeholder distribution.