Making LLMs run faster in production
This article walked through a range of excellent tools and libraries for deploying, versioning, and monitoring LLMs efficiently.
Putting large language models (LLMs) into production systems comes with a number of difficulties, and finding the bottlenecks is key to tuning their performance. Typical bottlenecks when running LLMs in production are:
- Latency. LLMs generate output one token at a time by design, so predictions take longer for long prompts and long generated responses.
- Throughput. A large number of requests hitting your server every second, combined with high latency, can freeze the server or cause many timeouts. The main way to raise server throughput at only a small extra cost is to aggregate requests, or better still, to use inflight batching.
- Memory requirements. These depend directly on model size and prompt length. A bigger model and a longer prompt both mean more VRAM.
- Energy consumption. Bigger models with longer prompts and outputs need more compute and take longer to run inference, so they use more energy. In large-scale production systems this can be a major cost.
- Scalability. LLMs need expensive high-end machines. Adding more machines to scale the system is very costly, so squeezing the most performance out of a single machine is essential.
- Model updates and versioning.
Tackling these bottlenecks takes a mix of hardware optimisation, tuning of the model architecture, and efficient deployment strategies. It is also essential to take the particular requirements and limitations of the production environment into account when deploying LLMs. Below, I go through the most important aspects of dealing with these challenges.
Finding the bottleneck
Improving a deployed model’s performance should always begin with pinning down the problem. As listed above, we may face many challenges, so first we need to work out which of them matter most to us. If the first hurdle is getting the model to fit on the GPU, memory optimisation is clearly where we start. Options include quantizing the model to float16 or float8, or loading the model’s layers one at a time during inference. Once a layer’s output has been computed, that layer is moved off the GPU and the next one is transferred from the CPU to the GPU. In this way you can run a model of any size on your system, provided the largest layer fits on your GPU. This feature is available in the accelerate library.
Once the model is up and running, we can turn to the other challenges. If the system has just a few users, or many users and high throughput, but responses come back slowly, we may want to add response streaming or cut down inference time. Ways to do that include Flash-Attention layers, setting a max beam width, removing input padding, using root-mean-square-normalization layers (RMS norm) in place of instance normalization, using GEneral Matrix to Matrix Multiplication (GEMM), or fusing GatedMLP layers horizontally. Quantization can make inference faster too, but only when quantized kernels are used. Most quantization methods just shrink the model without speeding up inference. Note also that not every GPU architecture supports every precision type, so choices about quantization should be based on what the target GPU architecture can do. A precision support matrix for the most popular NVIDIA GPUs is available [here](https://github.com/NVIDIA/TensorRT-LLM#:~:text=may, however, apply.-,Precision,-Various numerical precisions).
If latency is already low but throughput, meaning the number of users the system can handle at once, is the issue, we can implement cellular batching, iteration-level scheduling, or paged attention. The first two let newly arrived requests join, and finished requests be returned, at every iteration of the token generation loop. Paged attention is a clever GPU memory management scheme that keeps memory fragmentation to a minimum while also speeding up inference by caching the KV.
Whatever the problem, I like to begin by profiling the code, because other odd things affecting the system may be going on. For instance, while profiling some LLM code a few months back, I discovered it was trying to load the model in bfloat16 precision on a Tesla T4, which has no bfloat16 support, so it fell back to float32 and used x2 as much memory. Even something as simple as upgrading CUDA from 11.7 to 12.3 can make the model 20% faster.
TensorRT-LLM
This is an excellent tool for optimising LLMs, and it implements every inference speed and memory usage optimisation listed above. To get started with TensorRT-LLM, follow the official installation guide in its GitHub repository. The key step in speeding up a model with TRT-LLM is converting it. For TRT-LLM versions below 8.0, the conversion is done with the following command:
Code snippet 1. Building a TensorRT engine file from a given model. Version 1.
In newer versions, generating the engine takes 2 steps:
Code snippet 2. Building a TensorRT engine file from a given model. Version 2, part 1.
Code snippet 3. Building a TensorRT engine file from a given model. Version 2, part 2.
Once the engine files have been built, you can run an inference test with the _python3 /trtllm/examples/run.py script.
Inflight batching
With a naive batching approach, earlier requests either wait for later ones, or incoming requests are held back until earlier ones are done, which causes long queueing delays. TensorRT-LLM uses BatchManager to provide in-flight batching on request. Setting the _-use_inflightbatching flag during model conversion lets the model add new requests to a batch of requests that are already being processed. Rather than waiting for the earlier request to finish, a new request can join the batch at the iteration level, that is, once the new token for the current iteration has been produced. This raises throughput considerably at the cost of only a little extra computation and memory.
Paged Attention
Inflight batching raises the model’s throughput, but the large batch size it brings also pushes memory usage up. Existing systems struggle because the memory reserved for the key-value cache (KV cache) of each request is big and its size changes dynamically. Managing this memory poorly can waste a lot of it through fragmentation and needless duplication, which in the end limits the batch size the system can handle efficiently. This is where Paged Attention comes in, an algorithm modelled on the classic virtual memory and paging techniques used in operating systems. Rather than reserving the KV cache as one contiguous block of memory, it divides the KV cache into smaller blocks that do not need to be contiguous. On top of that, it adds logic for sharing KV cache blocks both across and within requests. The result is an LLM serving system with almost no wasted KV cache memory and flexible KV cache sharing within and between requests. With TensorRT inflight batching, paged attention must be enabled as well. PaggedAttention is switched on with the _–paged_kvcache flag. Note that in the TensorRT implementation the paged KV cache is allocated upfront, rather than on demand as the paper describes. As a result, we need roughly 60% more VRAM memory than the model itself requires. How much memory is reserved for the KV cache exactly depends on the batch size, the max input/output prompt lengths, and, in the end, the max number of generated tokens.
Triton
Triton is a model serving tool. It integrates with TensorRT-LLM through the TensorRT-LLM-backend. When deploying TensorRT-LLM with Triton, the specific versions must match. For instance, the Triton 24.01 server container image ships with TensorRT-LLM 7.1. Compiling models into the TensorRT engine with a different code version will cause errors at inference time. Pre-built docker images that already include the TenroRT-LLM-backend can be very convenient. You will find all the available images on the Triton NGC page.
To serve a model, we have to set up 4 directories that follow the layout shown in Figure 1.
Figure 1. Directory layout for the Triton inference server.
- Preprocessing. This folder holds the scripts and instructions for preparing requests before they reach the model.
- Tensrrt_llm. This folder holds the model engine files, the model’s configuration files, and configuration files describing the input-output data structures and types.
- Postprocessing. This folder holds the scripts and instructions for post-processing what the model outputs.
- Ensemble. This folder holds the scripts and instructions that coordinate preprocessing, model inference, and postprocessing. It can also handle other operations when needed, for example running two models one after another.
Pbtxt files describe the model’s inputs/outputs, their shape and type, and some hyperparameters where relevant. Among the other hyperparameters, batch size and the decoupled boolean are mandatory. When the latter is set to False, the model returns exactly one response per request.
The name of each subfolder begins with a number that marks the model/code version. A model checkpoint or a Python script can be put inside the folder. The Python script is executed according to the ensemble strategy, while the model checkpoint is loaded by the server automatically. You can find more on the Triton model repository here.
Triton also includes a useful model analyzer that can suggest Triton server parameters and a hardware setup for the best throughput and lowest latency. It can, for example, propose the optimal batch size or number of instances.
Once the Triton model repository is ready, you can start the Triton server with:
Code snippet 4. Starting the triton server.
Or, when using a docker container:
Code snippet 5. Starting a triton server with docker.
Wrap-up
This article walked through a range of excellent tools and libraries for deploying, versioning, and monitoring LLMs efficiently. And if making your model efficient is still a struggle, don’t hesitate to get in touch. We will gladly help.