Harnessing the value of AI on your CX strategy

Education NZ’s Chat-bot Journey: How AI & Chat-bots are transforming the User Experience

Jamie Crump

Education New Zealand

Director of Planning

Chat-bot development

Attracting and retaining international students is an exercise that many countries are engaged in, or at least were engaged in before the pandemic.

Jamie Crump , the Director of Planning at Education New Zealand (ENZ), says that their agency is “responsible for marketing New Zealand as a study destination to international students.”

Before the pandemic, “it was our country’s fourth largest export industry.” To engage with potential or current students, in 2018 ENZ decided to develop a chat-bot, which is a simulation of a human conversation, using artificial intelligence (AI) features.

This was “a relatively new communications channel,” meaning that “there isn’t an established formula or methodology for chat-bot implementations.”

However, at the time the agency was in no rush, and Jamie Crump was studying a “Masters of Technological Futures and developed an approach for the chat-bot project.”

A five stage process

A five stage approach for chat-bot implementation was adopted by ENZ, and included:

Problem and solution – “It’s really important to be clear on defining the problem that your chat-bot could solve.” For instance, identifying why a chat-bot is the best solution, and who it will target. In the case of ENZ, “we used a lot of existing research and surveyed prospective international students.” The problem that they hoped the chat-bot would solve was that international students needed “easy 24/7 access to accurate and reliable information to ensure the best possible study experience.” A secondary aim was to “reduce reliance on education agents.”

Scope – This stage is about the detail of “how the chat-bot will solve the problem,” including how it will be built. This forms the scope and in the case of ENZ, this was achieved by “getting buy-in and support from people across our organisation.” This resulted in a vision of how the “minimum viable product (MVP) would be built.” The options were to do it in-house or on an existing platform. “We decided to go with a New Zealand owned and operated digital services agency call Theta.” They use a Microsoft framework that allowed for customisation and constant re-shaping.

Prototype – Building an MVP, a prototype, is critical to the process because it necessitates thinking about the important things that “make a good chat-bot, like user experience, persona, name and avatar, trust, conversation design and training.” ENZ created a chat-bot called Tohu, a word which means to instruct, advise or point out. It was placed on the website and on Facebook, and has a distinct and friendly look and feel. However, “you shouldn’t immediately expect a user to trust your chat-bot.” Chat-bots need to be trained and they need to develop “how to convey their message and when to say it.” This means setting up datasets and training the AI to convert this into “friendly conversation, including language guidelines.”

Launch MVP – This stage is “all about getting your prototype out there into the world and testing it.” Friendly users, like colleagues or people from other agencies are a good source of testers, but “they’re more likely to be biased and since they are not from the user cohort, their results may not mirror that of actual users.” At ENZ they overcame that by “inviting about 200 prospective international students to chat with a test demo version of Tohu.” This exposed some gaps and provided grounding for a soft launch which came soon afterwards.

Learn and evolve – Eventually Tohu was formally launched, “but of course things have only just begun once you launch.” Tracking the performance of the chat-bot and measuring success is really important to the ongoing evolution and improvement of the chat-bot. In the case of Tohu, “our digital marketing team meets regularly to identify and address continuous improvement,” and there were five KPIs or success measures that determined whether the process had been worthwhile:

• Engagement – “Are the target audience using the chat-bot?”
• Solving enquires – “Do users get their questions successfully answered?”
• Retention – “Do users return because the bot has been helpful?”
• Conversion – “Is the bot helping to capture leads or sign ups?”
• Saving time – “Is there a reduction in the time taken to service enquires?”

The benefits of a chat-bot

Since Tohu was first launched in late 2018, it has become “a really beneficial tool and member of the ENZ team.” This is largely because of a number of significant factors:

• Scale – “This has enabled us to be where the student is, and has given us the ability to respond 24/7.”
• 80/20 – “80% of inquiries tend to ask the same 20 or fewer questions, and chat-bots are a really efficient way of answering these queries. This has saved significant time. Our social media manager now only responds to a couple of more complex queries a week.”
• User behaviour – This is about staying relevant to the core target audience.
• Business intelligence – “The chat-bot has been a rich source of business intelligence. Tohu has helped us better understand students’ pain points and the most important questions in real-time.”
• Navigating COVID-19 – The pandemic largely put a halt to international students, but with extra training and development, the chat-bot has “provided reliable information for current and prospective students, fielding thousands of questions related to COVID-19 and providing them immediate and accurate responses to those queries.”

Lessons learned

Tohu was developed and launched more than a year before the pandemic, meaning it served its purpose very well before the international student market largely dried up. ENZ therefore learned some key lessons about its development and implementation:

• “It’s really important to have a clear purpose and vision, but start with a narrow focus.” Initially Tohu was set up just for prospective students. Further iterations allowed it to answer questions for current students as well.
• “Launch and learn, and set expectations. Don’t wait for things to be perfect. Get the prototype out quickly and evolve it as people starting chatting with your bot.” Expectations also need to be realistic. “Tohu started out on learner plates and then eventually became a success.”
• “Training is critical and it never stops. Chat-bots aren’t set and forget. They need ongoing quality training.” The initial training was more intense but it pays off when the AI interprets and understands how to work with its audience. “We’re still working on getting 1% better every day.”