Estimate project
← All posts

What does finetuning an LLM actually give you?

A comparison of approaches to making medical Q&A chatbots better.

7 min read
Machine LearningLLM

Sooner or later, anyone working in natural language processing, and with chatbots and Large Language models in particular, comes across terms such as prompt engineering and model fine tuning as ways to tailor or enhance their system. Even so, it is frequently difficult to tell what sort of improvement to expect and which traits each improvement step is likely to bring.

This blog post goes beyond theory. We take a hands-on approach and look at the steps for improving chatbots, using the medical domain as our example. First we describe our methodology, and then we move on to real-world examples. They span a range of medical questions, each examined in one of three setups: a plain large language model, RAG, and a fine tuned LLM. By the end, you’ll have a clear picture of how these setups stack up against each other and how fine tuning affects the performance of a chatbot and an llm model.

Typical steps for improving chatbots

model optimization
To improve chatbots, we go through the pipeline shown in the graphic above.

Prompt engineering and Few-shot prompting

The first step is prompt engineering, which means adding guiding sentences or words that steer the LLM and, as a result, improve its performance. With prompt engineering, we can introduce a persona (marked in blue in the example below), and this noticeably raises the quality of the answers. Few shot prompting, in turn, boils down to supplying example input & output pairs (marked in green in the example below).

At the same time, as prompts keep getting longer, we may hit the limits of our LLM. Current models allow a few thousand words, but once RAG-retrieved facts and the chat history are kept in context, that ceiling can be reached fast.

On top of that, research is increasingly showing that a very long context is not always beneficial, because some facts in the middle may get lost (see “Lost in the Middle: How Language Models Use Long Contexts” for more). A long context also increases processing time, which can be undesirable in certain applications.

Improve Knowledge Base

Extending the context is another promising area for improvement. It takes considerable effort, because locating the right data and ensuring that its facts don’t conflict with those already there is a lot of work. At times, it also pays to take a critical look at the existing knowledge database and fix facts that are ambiguous or simply incorrect (this happens more often than people think).

Fine tune a model – improving model performance

This brings us to what many consider the holy grail of problem-solving: fine tuning. LLMs are typically pre trained on huge amounts of data found on the internet, ie Wikipedia or Common Crawl (bearing their licenses in mind when an LLM goes into production). On that data, the model learns without any labels, i.e., through next token prediction (for the words “LLMs are”, predict the following word).

In fine tuning, at least two different kinds of Q&A LLMs can be distinguished:

  1. LLM Fine tuning, in which we take large amounts of available data and apply methods resembling the pretraining step, while also allowing the pre trained model to specialise somewhat in our target domain.
  2. Task-specific fine tuning (also known as supervised fine tuning), in which we supply a Q&A dataset from our domain and try to teach a model to answer questions. During experimentation, we’ll look at a model trained specifically on Medical Q&A datasets (for example, MedQA contains roughly 12k question-answer pairs, while the broader MedMCQA contains roughly 190k labeled example pairs).

Medical domain characteristic

The medical domain tends to make us think of words like “difficult” and “complex” first (unless you’re a medical profesional, in which case the field is far more straightforward for you than for Casual Joe). The main reasons are:

  1. the need for complex, specialised knowledge – there is a vast amount of specialised terminology, intricate biological processes, and complicated links between symptoms, diseases and treatments.
  2. the high cost of mistakes – medicine leaves no room for error, since hallucinations or wrong answers can put lives at risk.
  3. a dynamic, constantly evolving field – new research, treatments and guidelines keep changing this area, which makes it even harder to build a sound knowledge base with current facts.
  4. ethical and legal concerns – in the medical domain, outputs may be discriminatory or biased.
  5. data quality and availability: medical data is hard to obtain because its privacy is a major concern. And, as noted earlier, the field keeps changing, so older datasets may be out of date in some areas.
  6. ambiguity and missing context: in this field, “it depends” is a very common answer, because answering certain questions often requires wider context (i.e. the patient’s medical history), and cases in the knowledge base frequently contradict one another.

Compared setups

In this check, we look at three setups: a bare question to llama2, llama2 with prompt engineering, and Meditron (llama2 fine tuned on medical data). The assumption of this research is that we use 7B models.

Bare LLM model

LLAMA-2 is a fairly popular model, and over the past 8 months many open-source models have been built on top of it (from Alpaca, an instruction fine tuned version, and Vicuna, a conversation fine tuned version, all the way to Meditron and Galen, fine tuned for the medical domain). For this base model, we apply the default prompting scheme without any extra system instructions.

Prompt engineering

Prompt engineering means shaping an instruction so that a Generative AI model can interpret and understand it. You can think of it as adding specialised phrases or words such as “You are an expert in the medical domain.”.

Techniques like Chain-of-Thought or few-shot prompting also fall under prompt engineering.

For our example, we use the system prompt below (that is, the instruction that controls the system):

Model finetuned on Q&A medical data

Finally, there’s the model fine tuned on Q&A medical datasets, Meditron (which currently tops the medical leaderboards). For the questions given, we use the instructions below to get the most out of this model:

Results

This section goes over several kinds of errors in medical Q&A, giving a sense of what prompt engineering and fine-tuning can deliver.

Errors in answers

Three different answers, one from each model, with the models other than Meditron:7b answering incorrectly

The correct answer here is “chromosome replication without cell division.” Of the three, only Meditron gets it right. Giving the model the persona of a medical expert (prompt engineering) wasn’t sufficient in this case. Thanks to prompt engineering, the model picked a different, more descriptive answer, but that answer was still wrong.

Fine-tuning a model mainly aims to expand its knowledge and strengthen its grasp of the domain’s specialized vocabulary, and in this example that goal was met.

Conciseness of answer

Example number 2: wordy answers from plain llama-2 models versus very concise answers from Meditron:7B

Example number 3: wordy answers from plain llama-2 models versus very concise answers from Meditron:7B

These examples reveal the chief weakness of LLMs aimed at a broad audience (such as plain LLAMA-2): their answers try to suit everyone, including people with no real expertise. As a result, they lack the conciseness a specialised audience needs, which makes them of little use.

Fine tuned models can give shorter, more specific answers without explaining everything, because their target audience may be more specialised in the field (medical professionals such as doctors or nurses).

Compared with general-purpose LLMs, fine tuned LLMs would likely do better on criteria such as correctness (how accurate an answer is) and conciseness (avoiding long answers to improve UX). In the medical domain, a few other criteria are also worth attention:

  • Avoiding unverified treatment (general purpose LLMs, trained on web data, may be prone to this type of error)
  • Public Health (requests for advice on behaviour that goes against general public health norms).

Conclusions

This blog post covered possible ways to improve a model’s performance and presented an experimental setup with 3 variants: plain llama2, llama2 with prompt engineering, and Meditron (a model fine-tuned on medical domain data). We saw clearly that fine-tuning can expand a model’s knowledge and change the characteristics of its default answer format (i.e., making answers more concise), ultimately making it more useful to the end user.