Estimate project
← All posts

Language agents explained

Find out how to recognise language agents and take a look at different architectures to better understand how they are built and what they can do.

4 min read
Machine LearningNLPChatbotsLlm AgentsNatural Language Processing

As apps such as chatGPT, Siri, or Alexa become more widespread, we are gradually surrounding ourselves with technology we may not completely understand. This article offers a quick look into the world of language agents.

The post is organised as follows: we start with what characterises language agents and how to recognise them, and then we go through a selection of language agent architectures.

How can you recognise a language agent?

Windows Clippy is an example of a basic chatbot, source: Figma.

One way to answer this is to compare them like this:

  • Basic chatbots (such as Clippy from Windows) are merely Q&A systems (not agents yet): a user can only pose a question, and the system returns an answer (usually a predefined one).
  • OpenAI’s ChatGPT-4 is an obvious example of a system built on a language agent, offering image generation and text2speech features along with multimodal input.

In short, if you can ask it to do more than one narrowly defined task (a chatbot++, so to speak), there is most likely some kind of language agent working behind the scenes.

Well-known examples of language agents

This section looks at several examples of systems that rely on language agents.

ChatGPT

In its most recent version, it can handle far more than text-based tasks. With ChatGPT you can:

  • Upload a dataset and run basic analysis with the Data Analyst product,
  • Create images from text with Dalle.
  • Turn text into audio with OpenAI’s TTS model.
  • Supply audio or an image and ask questions about it with GPT-V or Whisper.
  • Produce code for the functionality you ask for.
  • And likely a lot more.

Smartphone/Home Assistants

You are surely familiar with tools such as Siri, Google Assistant (powered by Gemini), or Alexa, which rely on language agents under the hood. They handle many tasks, such as playing music, checking the weather, booking appointments, or calling various people. Although these systems take speech as input, they convert it to text (with text-to-speech models) and then apply a range of NLP models.

Devin from Cognition

A software development agent that recently became famous is another excellent example of a language agent. Judging by the various demos, the tool can take software tasks from start to finish, imitating the way humans work: writing requirements, designing the architecture, planning the project, writing code, and testing the result.

In reaction, other agent-based solutions appeared in this space, such as OpenDevin, MetaGPT, or Github Workspace.

Vapi - voice AI assistant

VAPI is a company that provides end-to-end solutions for building automated speech agents, such as sales agents, combining text messaging over email or Whatsapp with real-world speech capabilities, all on top of the LLM inference engine you prefer (e.g., Perplexity or Groq). With it, you can build your own automated sales agent or take care of your company’s inbound support without hiring additional staff for these jobs.

Computer Games

Looking at research, the authors of a paper titled Voyager chose agentic workflows to simulate non-playable characters (NPCs) in Minecraft more convincingly and make them behave more like humans. They did this by building human-like mechanisms into the characters’ reasoning, such as motoric and episodic memory, reflection, and even autonomously learning new complex actions.

Agentic architectures

Agentic architectures are meant to address many of today’s AI problems, including multimodal Q&A, ambiguous questions, answers drawing on several documents, and reasoning errors.

Multi-modality architecture

This is the typical structure of a language agent. It begins with the user’s instructions, which the backend then wraps into a more elaborate prompt containing the available tools (a restricted set of actions), examples, and the task. Based on that prompt, the agent picks the tools needed to carry out the user’s task, and the designated tools supply and adjust the output so it stays consistent with the prompt.

Query translation and planning

image4

Besides combining tasks across modalities, we can also tackle ambiguous questions, like the one shown in the graphic below, step by step. Rather than answering a single difficult question, we can break it into simpler, more precise tasks and use their results to answer the message correctly.

Metadata filtering and routing

image1
Agents can also act as metadata filters, narrowing down relevant data or databases according to what they concern. In the example above, the user wants to know the best pizza place in Germany. Once it is clear that the question is about the country of Germany, the data stores covering German cities, such as Berlin or Hamburg, can be selected, while irrelevant ones, such as the New York guide, are left out.

Corrective RAG Agent

image6
Self-correcting RAG is another intriguing agent architecture. As the diagram shows, it consists of several steps:

  1. As in standard RAG, it looks up relevant documents in the data store.
  2. The agent scores how relevant the documents are
  3. The documents are then checked for whether they have nothing to do with the question.
  4. Once enough documents are available, a candidate answer is produced.
  5. The last step is split in two: evaluating the answer by checking whether it is grounded in the provided documents, and deciding whether it truly answers the original question.

Long-term memory of agents

Finally, there is a mechanism for keeping long-term memory about the user, since this can matter for giving answers better context. To achieve it, agents pull key information out of the user’s questions and answers and save it in a knowledge database. Likewise, when circumstances change (e.g., our user says he is no longer vegan), every passage related to that information should be updated to match.

Conclusion

This post explained what language agents are and where you can come across them. Above all, we looked at a selection of their architectures, which can help in building chatbot systems that cope better with the typical problems chatbots run into.

We have solid experience implementing language agent workflows, so if you need one, feel free to get in touch.