“An expert must be BOLD if he hopes to alchemize his homespun theory into conventional wisdom.” ― Steven D. Levitt, Freakonomics: A Rogue Economist Explores the Hidden Side of Everything
We were looking for a moonshot. Something we could do that would measurably improve customer satisfaction scores and call containment - all without a major product overhaul.
Flash back a year ago and I was sitting in a meeting with one of our large financial customers. Their head “numbers” guy asked me a challenging question: “Hebner, what kind of numbers are other companies seeing on the percentage of callers that experience at least one misrecognition when they’re in the IVR?”
Misrecognitions happen when Interactive Voice Response (IVR) systems aren’t confident that they heard what the caller said – they are typically handled with a, “I’m sorry, I didn’t understand.” And then are followed by a re-prompting of the question. These cause major frustrations for callers.
I paused for a second and said, “Honestly, I don’t think anyone measures their systems that way. Most people look at the success of their callers and try to minimize misrecognitions, but don’t look at the percentage that experience them. Why?”
He replied, “Well we are seeing x percent (I don’t remember the percentage) that are being misrecognized, and those callers rate our system measurably lower on customer satisfaction.” I made a mental note and moved the meeting along.
It’s rare I’m in a customer meeting where something unexpected comes to light – in the IVR space we have so many customers and we are building so many applications, that we are typically the ones bringing new ideas. I filed it away and didn’t recall it again until a year later.
A year later I’m in Matt Ellis’ office talking about IVR performance, measurements, and what we could be doing to improve key metrics. Matt asked if I remembered previous research showing misrecognitions were misrouted at the same rate as false acceptances.
A false acceptance is when the system thinks it knows what the caller said, but is wrong. The danger of a false acceptance is that the caller will be sent down the wrong path and to the wrong place. This danger is why the conventional wisdom was to always minimize false accepts, even if that means we’d have more misrecognitions.
I suddenly remembered the conversation I had about customers who were misrecognized and had significantly lower satisfaction scores. I then remembered usability tests where callers were sent down the wrong path and had no idea, as opposed to callers that heard, “I’m sorry, I didn’t hear that” and became upset with their service. I had an AH-HA moment.
Research shows us that if a caller experiences a misrecognition, they will be misrouted at the same rate as a false acceptance. We’ve been minimizing false accepts that don’t seem to upset callers and have been replacing them with misrecognition that MEASURABLY upset callers!
Why do we misrecognize anyone?
The reason we do confirmations and reject low confidence utterances is to make sure we get it right, so we don’t falsely accept requests. Even if callers are frustrated by confirmations, routing them based on a falsely accepted request has a significant negative impact on customer routing.
We are lowering satisfaction to improve routing but we don’t actually improve routing!!!
So we did an experiment. We lowered the recognition thresholds by half – we saw no impact on misrouted callers.
Because of the low volume of satisfaction surveys taken, it takes 6 months to a year to know if we are seeing the improvement in satisfaction – so we have the thresholds lowered really low and we are waiting to see the long term impact.
What did we learn from this? Some best practices are NOT based on data – they are based on “conventional wisdom”.
How many other best practices in speech IVR are homespun theory? We’re on it.