Amanda Stevens, Director of Conversation Design at Master of Code Global (MOC), was our special guest on the Conversational Designers and Chatbot Live this month, which was dedicated to the Feedback Loop.
Stevens, a Canadian who started at MOC in 2018 initially as a Key Account Manager, now oversees the conversational design team that supports brands like T-Mobile, Luxury Escapes, Aveda, World Surf League, just to name a few. Among her responsibilities are establishing, executing, and managing MOC’s conversation design process, along with hiring and training new talent. Here are her words of wisdom – and first, a 5-minute snippet of her interview, for whoever prefers the video version. (Full show link will be present down the road).
HOW I STARTED AT CONVERSATION DESIGN
Working in advertising agencies for 7 years, I was always fascinated by messaging. It was so interesting to see how brand’s messages, whether it be in campaigns or marketing materials, could be so powerful and turn heads. At MOC as a Key Account Manager, I started to look at the data of our client bots I managed and found opportunities to further improve the experience. I looked at where users were getting lost in the flow, NLU training opportunities and how design affected conversions and containment. After some time of experimenting and forming a better understanding of how to improve the user experience, I asked to be more involved in the design process which moved me into conversation design full time.
I first started researching how top brands like Google, IDEO, and Microsoft approach conversation design. Then I blended it with our own best practices and developed a complete process for MOC. From there, I started to grow the team.
What’s great about this field is there are so many conversation designers have that have different interests and passions that have led them to conversational AI. Mine came from brands, and how they use their voice in the world to engage consumers.
- CoCo & Co Content reCommendations
- The Feedback Loop: A basic chatbot development process
- How can Conversational AI help in the fight against the Coronavirus?
- How to avoid privacy risks with Conversational AI?
- Who benefits the most from adopting a CoCo? Q&A
HOW I MASTER THE FEEDBACK LOOP?
Pre-launch – we always conduct internal testing as well as user testing outside the organization with users that fit the target audience of the bot. We get great feedback that way and iterate from their comments and results. We also have our Quality Assurance Engineers who conduct internal testing such as regression and load testing, to ensure the bot can handle the traffic we anticipate.
Post-launch, we look at 2 different types of feedback in the Feedback Loop: In-conversation Feedback and Post-conversation Feedback.
In-conversation Feedback is anything the bot can ask the user to get feedback, such as asking for a ”CSAT score” at the end of the conversation. It can be a request as simple as, ‘how did I do today? Please rate me from 1-5.’
Another example is adding a Feedback Intent to the bot. So when the user finishes their conversation, the bot asks if there is anything else they can help them with. If the user says something the bot doesn’t understand, rather than triggering an error message, they can say ‘thanks for the feedback, I’ll pass this back to my team to train me on this’. It’s a lot nicer than seeing an error message of ‘sorry, I don’t understand.’
I believe the foundation of conversation design comes from a good understanding of user experience and excellent copywriting.
– Amanda Stevens
Don’t concentrate on the lone message level
When we look at Post-Conversation Feedback in the Feedback Loop, that’s us looking from the outside in, at analytical data and user input reports. User Input reports allow us to see what the user typed into the bot unprompted. It shows the bot’s response accuracy, what intents fired, and what inputs triggered the fallback response. This gives some insight if there are trends or growing types of queries users are asking.
However, analyzing one message on its own only provides part of the picture, which is why it’s important to look at complete conversations that show how they fell out of the flow and what steps they took that led them there. We also look at goal funnels per use case. It gives insight as to if there is a particular spot in the flow where people are falling out, and if they can easily get back in.
We also encourage our clients to talk to their live agents. If there’s a point of escalation, it often means the bot is not supporting their request. So we need to understand why this was the case. We ask what are the live agents hearing from the users? Are they complaining about the bot? Is the bot doing what they want them to do? Great insights can be found from simply getting feedback from your client-facing agents.
DOES EVERY BOT YOU MAKE HAVE A BOT MASTER?
Every bot we launch has a dedicated Account Manager, Project Manager, a Conversation Designer, Developer, a QA, and Bot Tuner (or Bot Master as others call it). When you’re optimizing a bot over a long period of time, context and knowledge of that brand is so essential. If you have multiple people working on optimizing the experience without that previous context, there will be a lack of information and you’ll run into inconsistencies and potentially recurring problems.
Within the feedback loop, when you’re looking at the data, you should not only be reactive and fix any issues. This data should also prompt you to be proactive. For example, say you have a retail bot that has a large sale or seasonal promotion. Maybe you didn’t build a specific intent to support questions around this particular promotion. So, you quickly create a ‘Promo’ intent, bot response, and launch it so the bot can start supporting those promo-related queries. When the sale is over, you can simply turn it off.
To be proactive, the Bot Tuner should make a note that if there are future sales, they should proactively create and curate these intents, so the bot can be equipped to answer similar queries for future promotions or other future events such as product launches or store updates.
HOW MUCH IS THE BOT’S PERSONALITY IMPORTANT
A conversation designer can spend a lot of time building out a personality for a bot. Training it to tell jokes, or have all these different Small Talk intents built-in. I think it’s important to look at the use case before completely building out a personality. For example, if someone is booking an appointment, they want to get in and out very quickly. If the use case is related to something like education, we need to keep users engaged. So a robust personality is great because you need to keep users engaged all the time.
Regardless of the use case, your bot needs to be conversational; if it’s robotic you won’t want to converse with it again. You always want to make sure your bot has empathy, it’s informative, helpful, and ensure there’s a healthy balance of personality.
COMMON MISTAKES THAT YOU SEE WITH NEW CONVERSATION DESIGNERS
It’s so essential to have a circular approach to design, and not view it in a linear way. It should always have loops within it. If someone falls out of the flow, the bot should allow them to get back in. If they make a mistake, such as misspell their name when completing a sign-up flow, they should be able to go back and edit their input. Always ensure the user has multiple ways to get to the same place. There are so many ways to converse with each other face-to-face. And the conversational solutions we design should allow for the same.
ANY ADVICE FOR BEGINNER CONVERSATION DESIGNERS?
There’s so many wonderful courses and blogs you can immerse yourself in, to understand how to conversation design and its best practices. But I believe the foundation of conversation design comes from a good understanding of user experience and excellent copywriting. Start your learning there, build that foundation, understand fundamentals of good UX and its best practices. Understand how to write copy for different brands, experiment writing for different personas, and understand user journey mapping. With this understanding, you’ll have the tools you need to grow into conversation design.
Hungry for more articles, videos and tutorials? – Subscribe to our YouTube channel | Join our Discord community | Sign up for our newsletter | Follow us on Facebook, LinkedIn, Instagram or Twitter