Are you thinking about adding an AI-powered chatbot to your website in order to improve your customer care, extend the availability of online support or get to know your customers better, but you don’t know how to measure its effectiveness?
For more than 15 years, Inbenta has been supporting companies worldwide in the creation of virtual assistants. We have seen the trends and uses evolve and while user expectations in terms of interactions and conversation have changed significantly, performance metrics have remained quite constant. They remain your main source of analysis to evaluate the impact of an AI chatbot on your company’s results. Therefore, we have gathered the top 10 key metrics to monitor when measuring your chatbot’s performance.
Feedback and learning come with interactions
Whether you go through a Proof of Concept stage or directly on a long term license with the technology of your choice, our first advice is to try to keep the testing phase as short as possible and make the chatbot available to the end-users as soon as possible. And this for one simple reason: it is very difficult to put yourself in your users’ shoes and guess what and how they think.
Indeed, your customers won’t talk to a bot like they do to a human. In the same way, your employees won’t tell an HR team member the things they would say to a bot. So you have to accept that this new communication channel (if it didn’t exist before) will bring its share of surprises. It is, of course, tempting and natural to try to answer as many questions as possible before the bot goes live, but it’s unrealistic to predict the needs on a channel that has never existed before!
So make your bot live as soon as possible with a minimum of content. Only real interactions will provide you with valuable knowledge about this channel and how to continuously improve it.
Identify the key metric for your AI chatbot
Once you have defined the objective and scope of your chatbot, it will soon become clear what the main measure of its performance should be.
Increase in conversion, decrease in incoming contacts with low added value, decrease in average processing time… We advise you to set a target figure on one or two indicators closely linked to the original strategic stake of the project (even though many other statistics will be available).
Customer care specific metrics:
- Decrease of first-level call rate
- Duration of calls generated by the chatbot (via web-callback)
- Email reduction rate
- Escalation rate
Marketing specific metrics:
- Conversion rate (for users having interacted with the bot)
- Average duration of sessions (for users having interacted with the bot)
- Number of pages viewed by visitors who have interacted with the bot
- Basket abandonment rate
HR specific metrics:
- Monthly question volume
- Escalation rate
- Frequent themes
Chatbot user experience metrics
Other indicators can be relevant for cross-analysis, but they can be numerous, therefore it’s easy to get lost or not to correlate the learning they provide.
We’ve summarized here the top 10 metrics to follow in order to gain a better knowledge of your users as well as the impact of your AI chatbot.
Self-service Rate
Percentage of user sessions that did not end with a contact action after using the bot.
Performance Rate
Number of correct answers divided by the number of active sessions (a correct answer is an answer suggested by the bot and clicked by the user in case of multiple choices – or opened instantly in case of strong semantic matching).
Usage Rate Per Login
Volume of active user sessions on the chatbot. To balance out with the average number of sessions on your website.
Bounce Rate
Volume of sessions where the chatbot was opened but not used
Satisfaction Rate
Average grade given when evaluating the chatbot’s answers (to balance out with the evaluation rate).
Evaluation Rate
percentage of user sessions that have given an evaluation of the chatbot’s answers at least once.
Average Chat Time
Allows you to evaluate your users’ interest for your chatbot.
Goal Completion Rate
in case your bot contains targeted actions like CTAs, a form or some cross-selling, that is the rate of users who have reached that specific action through the chatbot.
Non-response Rate
The amount of times the chatbot has failed to push some content following a user question (due to lack of content or misunderstanding).
Average Number of Interactions
Used to evaluate the Customer Effort Score on the chatbot and must be correlated to the satisfaction rate. If the latter is very low, the bot may be engaging the users in too many branches and steps to meet their needs. In this case, a resolution can be to correct the decision trees or knowledge base architecture.
Are these chatbot KPIs enough?
These different KPIs are sufficient to evaluate the ROI and the added value of your chatbot according to your initial goal(s). As obvious as it may seems, a regular monitoring will help you improve the effectiveness of the solution. However, these KPIs should not be the only metrics taken into consideration when evaluating the overall impact of the solution. Until very recently, companies did not need Artificial Intelligence to develop excellent customer relationships or optimal customer journeys.
Thus, beyond the KPI’s directly linked to chatbot, our last piece of advice would be to correlate these metrics with your pre-chatbot indicators (volume of phone contacts, volume of incoming emails via contact form, volume of chat with agents, etc). It also cannot be excluded that some chatbot-driven content may generate more phone calls than before if they are elusive or unsatisfactory.
Only cross-studies will really be able to reveal action plans that go beyond the chatbot’s perimetre by contextualizing it in your global economic environment.
To learn more about Inbenta Chat, click here.
