Testing: AI-Powered ChatBot Trained on Website Content

Ever since ChatGPT rolled out and shocked the world, I’ve been focused on how I can leverage this type of technology to add value. The vision has always been the same: Utilize an AI-powered chatbot that is trained on my website content.

While those visions were a little blurry in the beginning, the focus has sharpened in recent weeks. Even though we are still in the very early days of this technology, I’m downright giddy about the potential.

In this post, we’ll walk through the goals, the apps I’ve tested, and what precisely I’m doing in this current test. At the bottom, you can even join the beta to test it out yourself.

Apps I Tested

This is the most confusing and frustrating part about this very moment in AI. It’s incredibly crowded, and every app is new and untested.

The result is that no app is truly a finished product. Very few even have a reliable and stable reputation to know what we’re actually getting.

I started by testing every app that had a free trial to give me a sense of how they worked. I honestly couldn’t even list all of the apps I tested here.

But the three most recent tests involved SiteGPT, AI Engine, and Chatbase.

I was actually very close to moving forward with SiteGPT. AI Engine is a WordPress plugin, and I was running into too many bugs (could have been plugin conflicts) that created frustration.

Ultimately, I’m going with Chatbase. It seems to be the most polished while also offering everything I need in addition to future options like API integration.

Trained Content

To train the bot, we need to give it content. Chatbase provides the options of files (PDFs or Word docs), text, website URLs, and Q&A (providing example questions and answers). Since my website has more than 1,000 pages, my primary focus is on website URLs.

You have the option of crawling the website, submitting a sitemap, or entering in URLs manually.

Chatbase

Of course, I have to be careful with this. My website was started in 2011. Since the bot (for now, at least) doesn’t understand recency and won’t know how to handle outdated and conflicting information, it’s important that I feed it with the most relevant information possible.

This actually helped motivate me to craft content for this bot. In the meantime, that content can also improve usability of my website.

Here are examples of pieces of content that I created to help improve the knowledge base of the bot:

In addition to the main “Philosophies” post, I created eight individual posts to expand on those philosophies.

This process also provided motivation to update my Ads FAQ and Glossary (or at least start) since those sections were getting outdated. But they could be perfect content for a useful bot.

I selected 141 pages of content in all to train this bot on. While some outdated information within these pages is inevitable, I prioritized content that was still relevant.

I should add that Chatbase doesn’t have a limit on the number of links that you can submit to train the bot (big benefit). It’s all based on characters, and that’s a limit that I obviously haven’t run into yet (the lowest level has a 2 Million character limit that I have not reached).

Settings

I won’t cover every setting here, but I’ll cover those that are most important or impactful.

1. Base Prompt (system message)

Chatbase

This is where you give the bot some basic instructions on how you want it to behave. This includes things like tone, role, and how it should address certain questions.

This is where the learning curve is highest and there’s a lot of trial and error. I’m still sorting this out.

2. Temperature

Chatbase

Do you want to restrict the bot or give it freedom to be more creative? “Creative” could mean more interesting, but more potential for making stuff up. For now, I’m going fully reserved, but I may loosen it up depending on results I see.

3. Rate Limiting

Chatbase

This is necessary due to the data limits applied. You don’t want one user pushing through all of your data limits. I have no context regarding what the rate limit should be or how difficult it will be to run into data limits. So we’ll see how this goes and adjust accordingly.

4. Suggested Messages

Chatbase

This gives people something to select if they are unsure of how to use the bot. You can use the base prompt to establish how these questions should be answered, though it can be addressed in the Q&A section as well.

5. Customize Chat

You can customize the style, colors, and icons. I’ve done very little of this, but here’s what my bot looks like…

Conversation Logs and Refinement

This is one of the most important sections of the admin side of the bot.

I’m able to see a history of all conversations that are ongoing with the bot. That way, I can evaluate whether people are running into frustrations or if the bot is giving out false or inaccurate information.

An example of this I’ve seen with these bots is that they often reference articles and links that don’t exist. So, someone will ask for an article that addresses a specific topic, and the bot will make one up, complete with a fake URL.

That’s not good! While I think I’ve mostly addressed the fake URL issue with the base prompt, I can still address answers that I don’t like by revising the answer.

Here’s an example…

Chatbase

The question is related to Facebook ads strategies. The answer starts out pretty good, but it finishes up with a scaling strategy that isn’t really relevant. It’s not that the answer is so much wrong here, it’s just awkward.

So, I can click that “revise answer” link and remove the part I don’t like.

Chatbase

This may seem tedious, but this is the third phase of how it learns. First, it learns based on my content. Second, it learns based on the instructions in the base prompt and settings. And finally, it learns when I revise its answers. It won’t (or shouldn’t?) reply that way again.

Keep in mind that the question doesn’t need to be identical for the corrected answer to be used next time. That’s what makes these language models so smart.

Watch the Bot in Action

You can watch the bot in action below…

Your Turn

Have you started experimenting with AI chatbots? How are you using them?

Let me know in the comments below!