I’m Happiness Engineer at Octolize, where I provide tech and customer support for our Woo plugins. It’s a niche within a niche: shipping solutions for online stores. The inquires are about possible shipping scenarios, issues with carriers’ APIs, plugin configurations, and everything in between. About a year ago, our team decided to implement an AI tool for our email support channel, and here is my summary.

How the AI tool works for us

We use HelpScout and since it’s a popular platform used by many companies, a dedicated tool called SupportAgent.ai has been developed. We jumped on board, tested it internally and quickly realized its huge potential. Like many other tools, it’s based on OpenAI ChatGPT. By utilizing our conversation history, the SupportAgent.ai is generating reply drafts to all incoming emails.

It works perfectly if:

• the whole query is in a text form (no attachments, screenshots or links)
• a user asks about anything that our team covered in the past
• questions are written in a clear concise way

Human in the loop

In most cases, the generated draft will have accurate information, with proper URLs to our documentation and coherent answers to all questions asked by the user. Therefore, my next step is to proofread the draft. For that, I use LanguageTool Premium to polish the tone and correct sporadic AI grammar mistakes.

The most important task is actually in verifying the accuracy. The AI hallucinations show up when it’s a novel case or AI simply doesn’t have access to some data. For example, shipping carrier is changing the API from XML to REST, and our developers are already working on it, but the AI tool doesn’t know that.

However, once our support team covers such case a couple of times, by manually writing the reply, the SupportAgent.ai learns from it and suggests aptly response next time.

My reflections after one year of using the AI tool as a tech support

Years ago I read “Life 3.0” by Max Tegmark and “AI Superpowers” by Kai-Fu Lee, so I’m excited that I can finally use this technology in my daily work. I’m not afraid that, it’ll replace me. Only because I am eager to learn, hone my skills and know how to use the AI tools better.

Actually, I’m fascinated by the fact that the SupportAgent.ai is learning from my past interactions with clients. It speeds up my work a lot, and I’m now thinking about it, as, like any other tool I use. I want to tweak it. Play and experiment. Look from different angles and find ways to improve it.

Other companies, like Automattic, also use AI-generated answers. Especially in clear, defined cases where additional human input is not needed. Here’s the rep once I got when I requested a feature for my favorite podcast app:

Hi Piotrek,

Thanks for sharing your feedback with us regarding the recommendation filter in Pocket Casts.

At the moment, we don’t have an open-source algorithm recommendation filter in place. However, your suggestion for personalized recommendations based on listening history and subscriptions is valuable. I’ll make sure to pass this feedback along to our development team for consideration in future updates.

If you have any more ideas or feedback, feel free to share them with us. We appreciate your input!

I hope that helps! If you have any additional questions or my answer wasn’t helpful, please respond to this email.

Answered by a Pocket Casts AI Support Assistant

Improvements of AI tools for tech support

Ben Thompson from Stratechery often emphasizes the tech evolution from text, to images, to video. I’m very satisfied with the level of sophistication of the AI tool’s text capabilities. AI reading and understanding images is the necessary next step. Very often our customer threads have multiple screenshots, that explain the issue 10x better than words. If the AI could comprehend the pictured issue, it would provide more suitable answers.

The second thing I’m looking for is the ability to feed the AI with the other company assets. Namely, documentation, blog articles, but also our store data, subscription status or bug tracker. This could help address all other customer support queries. Issues related to client’s billing or current bug status could further enhance the reply draft generation.

Lastly, I can’t wait to see a wide implementation of formal verification. As Max Tegran points out in his recent Ted Talk, this is a mathematically proven method for proving or disproving the correctness of data. With this, we can reduce AI hallucinations and ensure that the generated output is aligned with our values.