Cognitive and Conversational Intelligence
how smart and context-aware the chatbot is when understanding and responding to user queries.
Natural Language Understanding (NLU) Accuracy
-
What it is: This measures how accurately the chatbot understands what the user is asking.
-
Example: If a user asks, “What’s my account balance?” and the chatbot correctly identifies that the user wants to know their balance, that’s a correct understanding.
F1-Score
-
What it is: This is a combination of two things: Precision (how many of the chatbot’s answers were correct) and Recall (how many of the chatbot’s answers were all the possible correct answers).
-
Example: If the chatbot helps with “transfers” and “loans,” the F1-score will measure how good the chatbot is at both recognizing and answering questions about these topics correctly.
Context Retention
-
What it is: This measures how well the chatbot remembers previous parts of the conversation. Does it keep track of what was said earlier?
-
Example: If a user asks, “Can I transfer 500 without asking the user to repeat themselves.
Conversational Coherence
-
What it is: This measures how well the chatbot keeps the conversation flowing logically.
-
Example: If a user asks a series of questions like, “How much is in my account?” followed by, “Can I withdraw it?” the chatbot should respond in a way that makes sense given the context.
Task Completion Rate (TCR)
-
What it is: This measures how often the chatbot successfully completes the tasks it’s asked to do.
-
Example: If a user asks to transfer money, and the chatbot successfully completes the transaction, that counts as a successful task.
Semantic Similarity Score
-
What it is: This checks how similar the chatbot’s response is to the best possible response.
-
Example: If the chatbot is asked, “How can I invest in stocks?” and it replies with, “You can open a brokerage account,” that’s a semantically correct answer.
Turn-Taking Balance
-
What it is: This measures how much the chatbot dominates the conversation versus how much the user participates.
-
Example: If a chatbot asks a question and then lets the user answer, that’s a good balance. If it keeps talking and doesn’t let the user respond, it might be too dominant.