Czech savings bank Česká spořitelna, a division of Austria’s Erste Group, recently collaborated with AI solution builder DataSentics to explore the use of GenAI in call centers. Česká wanted to improve quality control and optimize costs in their inbound call center operations, which receive around 2 million calls per year. They chose the Databricks Data Intelligence Platform to experiment with both internal and external AI models to assess the effectiveness of call center agents.
Exploring a Quality Control System for Customer Support
The call center team at Česká spořitelna wanted to test a quality control system powered by GenAI that would ensure that agents adhere to scripted guidelines during customer interactions. A critical challenge for Ceska was ensuring consistent agent communication for routine customer inquiries. When customers call about account balances, agents need to direct them to online banking solutions, a key business requirement that drives digital adoption and operational efficiency. The support team needed a scalable way to verify agent compliance and maintain communication standards across thousands of customer interactions. To achieve this, the team began by using Whisper, a speech-to-text model from OpenAI, to transcribe conversations accurately. The challenge was to produce human-readable text that accurately represented spoken words used by call center agents without distorting their meaning. The transcriptions needed to make logical sense and reflect the intent of the conversation accurately for further analysis.
Following the transcription, the team explored integrating both internal GPT models and open source models such as Mixtral to evaluate their effectiveness. GenAI models were tested in a simulated QA role, where they were tasked with answering specific questions such as “Did the agent redirect the customer to online banking?”. The goal of this exercise was to assess how well these models could mimic human understanding and decision-making when verifying compliance with established guidelines. By comparing the performance of both the internal GPT model and the open source models, the team aimed to find the most effective solution for improving customer service through automated AI-driven quality control.
Benefits of the Databricks Data Intelligence Platform for GenAI
The DataSentics team evaluated several options for this solution, and ultimately chose to deploy the Databricks Data Intelligence Platform and Mosaic AI tools at Česká spořitelna for several reasons:
- Data Management and Governance Benefits: Unity Catalog makes data easily accessible for different models while keeping sensitive data under restricted access.
- Comprehensive Data Processing Capabilities: the Databricks Platform supports the entire workflow of preprocessing of call center data, from transcription to quality control. This enables us to produce intermediate results that can be leveraged for other models and projects, such as marketing, risk assessment, regulatory compliance, and fraud detection.
- Model Training and Support: Databricks provides robust support and expertise for GenAI, including model architecture and training capabilities. This made it an ideal platform for testing and deploying open source models quickly, enabling us to experiment and iterate efficiently.
- Ease of Cluster Creation: With Databricks, it’s straightforward to create clusters and deploy open-source models. This streamlines the experimentation process and allows us to focus more on model performance and less on infrastructure management.
Insights and Results
Throughout the project, we experimented with various segmentation techniques and gathered several valuable insights:
- Quality of Input Data is Crucial: The quality of the audio recordings varied from client to client, with some speaking quietly or from a distance, which can later affect the accuracy of the transcription. Whisper or similar systems can help solve the problem.
- Category Definition is a Must: We learned that if categories cannot be easily defined for humans, it is equally challenging for LLMs to understand them. This reinforced the need for clear and precise category definitions to train the models effectively.
- Open-Source Models Deliver Results: Open-source models demonstrated that they could compete effectively with proprietary models like ChatGPT. This finding is significant for businesses looking to optimize costs while still achieving high-quality results.
What’s Next
With GenAI tools powered by Databricks Mosaic AI, Česká spořitelna employees are now able to gain access to answers found in a range of documents via “smart search” functionality. For example, the purchasing team may need to consult hundreds of pages of process documentation on how to control and approve payments to different countries. Before leveraging Databricks, it would take employees hours to find the correct information they need. Now, RAG-powered search gives employees answers within seconds, including citations and links to the source document.
Looking ahead, there are plenty of opportunities to explore more GenAI workloads at Česká spořitelna. We aim to create a robust integration between Databricks and Česká spořitelna’s internal database call center recordings. This will unlock new use cases such as churn detection, sentiment analysis, and sales signal detection since Databricks is the go-to platform for streaming data. These daily reports will allow Česká spořitelna to react to changes in real time while achieving cost reductions with improved quality assurance in their call centers.
This blog post was jointly authored by Petra Starmanova (Česká spořitelna), Tereza Mokrenova (DataSentics), Dalibor Karásek (DataSentics) and Joannis Paul Schweres (Databricks).
Source link
lol
