Bots and the Future of Customer Experience
Enhance engagement • Drive efficiency • Unlock capacity
Featured Speakers:
Block Title
Rhonda Tsingos
Management Consultant,
Technology Services Division
City of Toronto
Marco Palermo
Director, Digital Government
& Modernization
City of Toronto
A solution to a crisis
City councils are often the first ports of call when there is a crisis.
City councils, therefore, need to be resilient and responsive and need to be able to provide answers. In recent times there has been no greater crisis than COVID-19, and the City of Toronto was intent on being proactive in its dealings with its citizens. However, there were simply not enough staff and too many questions coming in, so a digital or artificial intelligence (AI) solution was sought.
Marco Palermo , the Director of Digital Government & Modernization at the City of Toronto, says that very early on in the pandemic, “we wanted to provide an additional channel to help residents and businesses stay informed in what was and still is a rapidly changing situation. People were confused and unsure where to turn. We needed to address those concerns quite quickly.” They needed an opportunity for citizens to interact with the City without further burdening their staff, so they decided to “go down the avenue of a chat-bot,” which is a simulation of a human conversation, using AI features. Having the idea alone though was not enough because there are many types of chat-bots available, each one with different configurations and modalities.
The City of Toronto also had a few expectations that needed to be met. “Accessibility was very important to the City, so that it was equitable to everybody, and any solution also needed to be scalable, stable and obviously sustainable.” To achieve this, they reached out to partners, because especially during a pandemic when the time was of the essence, “building your own AI is just not a feasible option.” They already had relationships with Cisco and Google, so “we decided to leverage those relationships,” but to make it fit-for-purpose, it needed to be imbued with their own touch.
Creating a customized chat-bot
A chatbot is really “that little thing at the bottom of every screen that allows interaction.” The solution provided by Cisco and Google “had certain accessibility limitations.” Google has since enhanced their generic chat-bot, but in March 2020 “this was our first stumbling block.” The only option, therefore, was to take the template from Google and “leverage our internal framework and our development teams in-house to build own widget that would land seamlessly on our website.”
“Our team was able to build something that was not only fully digitally accessible but had additional features that maybe you wouldn’t get in an out-of-the-box solution. And we did so in a week.”
Marco Palermo, Director, Digital Government & Modernization, City of Toronto
It took a few more weeks to fully roll it out, but within the first week, the chatbot was “interacting with Google’s interfaces.” Part of the delay in rolling it out was because “we had to train this bot. We treated it much like a child, teaching it and training it to do things one step at a time. That’s the nature of artificial intelligence. It’s constantly learning based on the interactions it has with people.”
To make it sustainable and effective long-term, “we also thought about how far we can take the interactions with the chat-bot.” For instance, though some people were likely to ask it basic questions, others may ask it questions that require genuine thought, “like should they get tested? Ultimately we wanted the bot to lead them to a map to show them their closest location for testing.” But rather than do this through “guided FAQs, we decided to make it fully conversational,” which was seen as a controversial decision because it required “a higher level of confidence in the AI.” Chat-bots generally produce a confidence score and “we said for anything that has a confidence score of 50% or less, let’s actually take that question and put it through the City’s website search engine.” So if the AI knowledge base couldn’t find an appropriate answer, the question would be “re-routed to see if we could at least get a webpage that can provide an answer.” If that wasn’t satisfactory, the AI was programmed “to ask,
can you rephrase your question?
”
A successful solution for the times
Rhonda Tsingos , a Management Consultant in the Technology Services Division at the City of Toronto, says that once the chat-bot was finally ready, it “was only available on the City of Toronto COVID-19 pages [1] .” This was a deliberate choice “because the scope of this chat-bot is simply around COVID-19 and its impact,” though even limiting it just to the pandemic has been quite a challenge. The chat-bot was “launched softly at the end of April. We didn’t make it a big deal.” By early May it was promoted on social media, but without fanfare. Nonetheless, since then there have been “over 34,000 unique users who have asked over 29,000 questions.” Based on the confidence score, “almost 81% of the time we’re able to provide an answer with a score of at least 50%.”
The most popular questions are about “local Information and restrictions like stay-at-home orders, plus questions around what the City is doing and what changes there may be to City services.” However, people also have high or unrealistic expectations and are “used to talking to Siri or Alexa so they assume that the bot knows everything about all things related to COVID-19, and sometimes they are looking for loopholes or asking permission to do the wrong thing.” At the same time, this shows there is a need and the bot has been providing a much-needed service.
To get to the success currently being experienced, the initial partnerships worked well, and new ones were created too. There are now partnerships with health, public safety, law enforcement, technology and communications divisions, all of which contribute and provide information. There is a “standalone web app for our content management processes,” and all of the data is stored there for the AI to use. However, “could we have possibly chosen anything more complex, that has gone on for far longer than we anticipated with such a large volume of ever-changing information?” COVID-19 is an overwhelming topic for many, and the questions are endless. People also “ramble, have many typos, and for many, English is a second language so we have to deal with that as well.”
[1] https://www.toronto.ca/home/covid-19/
A“It’s been an interesting exercise interpreting what people are asking, and what their expectations are for the answers. Plus, the information is ever-changing, so we need to adjust for that too.
- Rhonda Tsingos, Management Consultant, Technology Services Division, City of Toronto”
It was ever more challenging because finances to develop something new were scarce, and “in a pandemic, people were redeployed to other frontline areas,” so people who were assigned to the chat-bot project weren’t necessarily experts. The project was however completed, though it became very obvious that “bot really does require daily sustenance.” It needs someone to look at the analytics and trends and to see where there are gaps in the AI knowledge base. “We’ve taken a continuous improvement approach,” and the “content team meets bi-weekly.”
The chat-bot was designed for COVID-19 and is still delivering. However, there are now opportunities to “integrate more existing knowledge bases throughout the City and to modernize. We are looking to create that seamless customer experience across all our channels and customer environments.”
Want more insights like this?
Be sure to join us at the next PSN virtual event
