Estimate project
← All posts

Which type of AI Chatbot is right for you?

Find the AI chatbot solution that best matches the needs and requirements of your organization.

7 min read
Artificial IntelligenceAWSGoogle Cloud PlatformChatbotsLarge Language Model

An AI Chatbot is a Large Language Model (LLM) that has been instruction-tuned. A chatbot like this can give quick, precise, and relevant responses to the questions it receives. It is, however, restricted to the information, tone of text, format, and style present in its training data.

With the Retrieval Augmented Generation (RAG-Chatbot) technique, you can link the AI Chatbot to your organization’s internal, private documentation, so that it answers questions using the supplied knowledge base. Users ask questions and receive answers grounded in the documentation, even though the model never encountered it in training (your internal documentation remains private and secure!). The AI retrieves the right documents based on the user query and attaches them to the query as context before passing it to the AI Chatbot.

To build a RAG-Chatbot, you need to convert all of the organization’s documentation into the proper format and make a handful of design decisions. Figure 1 shows every possible option for designing and deploying a RAG-Chatbot.

public vs self hosted

Figure 1. Options for hosting your RAG-Chatbot.

Which solution fits best depends on the use case. Below, I go through each approach and weigh its advantages and disadvantages.

RAG-Chatbot solutions

Public API (LLM SaaS)

Solutions built on cloud APIs shine when it comes to development speed, Proof-of-Concept validation, and low usage traffic. They also provide regular model updates at no extra cost. They do, however, have a number of limitations:

  • Customization is limited.
  • They are not cost-effective at very high traffic.
  • They may not offer enough safety for highly confidential data.
  • Foundation model

In most cases, this is the best approach. It can be implemented quickly and performs reasonably well. You pay as you go (per request) plus a small fixed fee. It is the cheapest choice when traffic is uneven (high usage peaks with the system idle most of the time) or relatively low (under 10,000 per day). It also ensures automatic upgrades to newer model versions, so answer quality improves over time. Using Vertex AI from Google Cloud or Bedrock solutions from AWS ensures the privacy of your data. It is the best way to build a RAG-Chatbot Proof-of-Concept for your organization and check whether it is the right solution for you – or simply to build a conversational chatbot if your traffic is moderate and you need no customization.

Custom model
This option naturally extends the previous one, though it can also be chosen as the target solution right from the start. It lets you fine-tune the model on customers’ data and so reach even better RAG-Chatbot performance or features that would not otherwise be achievable. Fine-tuning the model makes sense if:

  • You want the model to behave in a particular way.
  • Imitate a specific tone of voice.
  • Produce output in a particular format.
  • You need to optimize model performance (implement RAG without a knowledge base).
  • Make the model do something it wasn’t trained to do.
  • Above all, fine-tune the model when prompting and the knowledge base option don’t deliver.

A custom model brings higher costs. The dataset has to be prepared in the right format for training, validation, and testing. Training costs depend on the model type, dataset size, and infrastructure provider, but a rough estimate is $0.0008 per 1,000 tokens or ~20$/h. Details on LLM fine-tuning costs are available here for AWS and here for GCP.

Serving the model adds further costs. On-demand pricing no longer applies; a custom model carries a base hourly charge of ~$7-20/h. For that price, you get a private model that is entirely at your disposal.

Fine-tuning a model through a cloud provider API shortens development time but limits how far you can optimize. Only certain models can be fine-tuned, and engineers can’t look under the hood to adjust the model further. You’ll find detailed information on fine-tuning models through a cloud API, using AWS Bedrock and the Titan model as an example, here.

Self-hosted

With self-hosted RAG Chatbots, we fully control the model and the inference pipeline. They provide extensive customization options and data privacy, make it easy to fine-tune both the LLM and the embedder models, and allow building advanced retrieval algorithms. Once a custom Rag Chatbot is built, the question becomes: where should it be hosted? There are two choices – in the cloud or on-premise. Their advantages and disadvantages are outlined below.

Cloud self-hosted
This option is simple to maintain, scalable, and dependable, as the cloud provider handles infrastructure maintenance. On the other hand, it costs more than on-premise solutions. A basic setup runs at about $1.2/h and may reach $12/h per instance. A single instance can handle up to 50k requests a day.

On-premise
An on-premise, self-hosted model offers the strongest data privacy. No external APIs are called, and all data remains inside the organization network. A self-hosted model gives you complete control over the RAG Chatbot and permits any customization you like. Running costs are low, since you only pay for electricity and maintenance. The main downsides of this option are maintenance and scalability.

which model to choose

Figure 2. Decision diagram for implementing a RAG-Chatbot.

Do I need a custom model?

Fine-tuning can considerably improve how well a model performs on particular tasks. When the model sees task-specific data during fine-tuning, it learns to produce more accurate predictions for that task. Fine-tuning a pre-trained model on medical texts, for instance, yields a model specialized in understanding medical language. Fine-tuning on domain-specific data lets the model absorb domain knowledge, strengthening its understanding and generation abilities in that field. All in all, fine-tuning lets you build on the strengths of pre-trained models while adapting them to your particular needs, which results in better performance and efficiency for your specific tasks.

content and model optimization

Figure 3. Diagram of model optimization.

Fine-tuning a model should be a conscious choice. Typically, RAG-Chatbot development follows the path shown in Figure 3. We begin with prompt engineering, the lowest-hanging fruit, get as much out of it as we can, and then move on to Retrieval Augmented Generation with the connected knowledge base. At that point, we already get good-quality answers based on our organization’s internal knowledge. Next, we may choose to push model performance further by fine-tuning it on our data, ending up with a best-in-class solution.

Limitations of prompt engineering

Prompt engineering is always the go-to first step in customizing a model, since it is inexpensive and fast to implement. It is frequently enough on its own, with nothing else required. Its drawback is that it lengthens the context, so you might run into the model’s context limit. If model performance is still unsatisfactory after prompt engineering, consider fine-tuning. Fine-tuning is especially effective at reproducing style and tone. It does, however, take much more work, as you need to prepare the dataset in the right format and train the model on hardware that is not cheap either.

Running costs comparison

Table 1 shows estimated costs and performance for every deployment mode.

Table 1. Comparison of running costs and performance across 4 different deployment options.

AWS pricing: https://aws.amazon.com/bedrock/pricing/
GCP pricing: https://cloud.google.com/vertex-ai/pricing#custom-trained_models

Data privacy

How private your data is depends on the services you use and on the cloud provider. That is why you must read the terms of the agreement carefully before using any cloud provider API. Below is the current situation at the three leading Cloud Solutions providers.

Google Cloud

  • Data sent in Gemini API calls may be used for training and may be viewed by human reviewers, so don’t send sensitive, confidential, or personal information to the service. The Gemini API terms of service Outside of VertexAI are available here
  • Within VertexAI, which is built for enterprise-grade support, you use the Vertex AI Gemini API, whose terms of service forbid using your data to train new foundation models.

AWS

  • Bedrock’s terms of service state that the Titan model’s input and output are not used to train foundation models.
  • “AWS Chatbot, by default, stores and processes user data, including AWS Chatbot configurations, notifications, user inputs, responses generated by AWS Chatbot, and interaction data. AWS Chatbot uses this data to keep improving and developing itself as well as Artificial Intelligence (AI) technologies.” (source).

Azure

  • Azure OpenAI neither stores nor uses your data. (source)
  • OpenAI may use your data to train new models. (source)

To sum up, enterprise-grade solutions don’t make use of input/output data, whereas public APIs use input data to train models and improve AI. Note also that all the privacy policies mentioned above are always subject to local law. Companies must hand over their data when the law or a binding order of a governmental body requires it.

Summary

Table 2. Advantages and disadvantages of Cloud API vs self-hosted RAG-Chatbot.

At our company, we build solutions tailored to what our customers need. Whether you want your RAG-Chatbot delivered fast, anticipate low traffic, or need a custom solution that gets the most out of your data, we can build it. We have partnerships in place with leading cloud providers and can offer free discounts and credits to kick off your project. If you have any questions, don’t hesitate to get in touch – we’ll be glad to help.