How to Measure the Performance of Your Bots

Businesses are all about data and insights. But deriving insights from data is only half the job.

To define the data needed — and how to structure that data for analysis — first define the metrics that are important to your specific business. Here are some commonly tracked metrics:

Intent and entity recognition rates – In the case of a banking bot that recognizes the intent of the customer, collects data and passes the flow to a human agent, intent recognition and entity recognition rates define the bot’s success. They are defined as the percentage of the intents and entities the bot recognizes versus the total number of intent and entity recognition opportunities presented to the bot.

For our data collection exercise, the intent and entity recognition rates were the primary metrics to detect performance.

Task completion rate – If the goal of a banking bot is to fulfil the customer’s request, task completion rates are the most important metric. They’re defined as the percentage of the total conversations the bot handles and successfully fulfils.

Turns – A turn is a single interaction between a customer and an agent. An entire conversation between a customer and an agent can take several turns to complete.

Turns indicate whether the bot understood the user intent without much clarification — and whether entities were captured the first time they were uttered. A lower number of turns means a better user experience.

We used average turns, along with the intent recognition rates. This helped us understand whether the bot recognized a low-performing intent better when clarifying questions were asked.

Containment – Containment metrics show how many automation opportunities the bot tapped from the entire chat corpus. This helps businesses understand ROI metrics and set goals for bot roll-outs.

Feedback scores – Bot automation improves operational efficiencies and reduces operational costs. However, user experience shouldn’t suffer as a result.

Learn whether the user experience has been positive, negative or neutral. This ensures that savings from bot automation don’t result in a lost customer.

In our initiative, we included icons for users to indicate satisfaction levels with the bot interaction.

Structuring the Data Capture

To compute the above metrics, log as much context about the interaction as possible.

Capture ground truth – In a real-world scenario, it can be difficult to capture ground truth. But it’s easily done in a controlled experimental set-up.

Business analysts and consultants who monitor bot roll-out experiments should capture the expected use case – the ground truth for intent and entities. This ensures that intents and entities recognized by the bot are captured in a structured way.

In our data collection exercise, we stored information on the expected intent from the scenario that was presented to the user. This avoided the need for annotation, which can be time consuming and expensive.

Determining one turn and one conversation – You’ll need enough identifiers to distinguish one turn from another as well as the number of turns in a conversation. It’s also important to distinguish between a turn within a conversation and one in a new conversation.

User session – Understand whether the user closed the conversation after fulfilment or whether it was closed part-way through out of frustration. Did the user make another attempt and was it successful? Capture a unique ID for each user and their session to determine this.

Decoding the chaos with a confusion matrix – A confusion matrix helps derive insights on intent performance and confusable intents. It also gives you a picture of how changes in the bot affect precision, recall and f1 scores. The next blog in this series will explore this in detail.

Automation of Metrics

Two steps for automating insights are defining metrics and structuring data to capture it. If the data is structured properly, you can calculate metrics using assumptions and a formula. However, not all metrics can be automated without compromising accuracy.

Error tolerance is an important parameter in determining whether to capture a metric using automation. Manual annotation is an expensive and time-consuming alternative to metrics automation.

So, it’s a business call to balance the requirement of precision of data metrics with the time and effort required to reach an extremely high level of accuracy that would mandate human annotations.

Author: Robyn Coppell