Estimate project
← All posts

Triton Inference Server: First Impressions

Triton is a fast, resource-efficient tool for deploying models, and it pays off to learn how to configure it well.

7 min read
Machine LearningTritonTensorRT

After spending some time experimenting with Triton, I want to share my initial impressions in this post, walk through its key features, and explain when and how it is worth using. The follow-up post will cover tips and tricks, include a cheat sheet, and offer a code sample for benchmarking various model configurations for computer vision models. Let’s dive in!

What is it?

Triton is a fast machine learning inference runtime that sits underneath many ML deployment tools, including Seldon Core, KServe, MLFlow, and BentoML. What makes it strong is the performance it gets from its C++ implementation, together with a broad feature set that makes it flexible enough for nearly any use case. It works with many frameworks – NVIDIA TensorRT, TensorFlow, PyTorch, OpenVINO, and ONNX – and also with custom C++ and Python backends. It integrates with Kubernetes and exposes service-monitoring metrics out of the box, which makes it convenient to run in production.

Source: https://docs.nvidia.com

‍

Triton ships as part of the JetPack SDK, which runs on NVIDIA Jetson computers designed for edge AI. Its C API lets you run real-time inference with no network overhead.

Triton also helps when you need to deploy large language models. More on that topic can be found in this blog.

First steps

The official repository offers a number of tutorials. Be aware that they contain a few small bugs or simply require some tweaking. I opened a PR with fixes. With those corrections in place, the code runs correctly and gives you a simple way to get to know Triton’s core concepts: its architecture, concurrent model execution, the layout of the model repository, the model configuration file, and optimization techniques.

Triton’s own tools, perf_analyzer and model_analyzer, help you automatically find the best parameters for deployment on particular hardware. This comes in very handy when putting models on a production server.

Sadly, Triton is not easy to work with. Model configuration has lots of exceptions, some options cannot be used together, and so on. When you try to start the server with such a configuration, it fails with unclear error messages. In the sections that follow, I describe the exceptions I ran into and the rules I stuck to while experimenting with Triton.

Model configuration

Here is a minimal configuration file for a model that has 2 input tensors and 1 output tensor. Several fields are required: name, which must be identical to the model’s directory name in the model-repository passed to Triton; platform, which specifies the kind of model runtime; and input and output, which describe the data going into the model.

The available platforms are onnxruntime_onnx, tensorrt_plan, pytorch_libtorch, tensorflow_savedmodel, and tensorflow_graphdef. Model files need extensions that correspond to the chosen platform: .onnx, .plan, .pt, etc.

The supported data types are listed in the documentation. The dims field specifies a tensor’s dimensions. For instance, a model handling ImageNet images would use dims: [3, 224, 224] for the input and dims: [1000] for the output.

Keep in mind: batched inference becomes possible once you set a max_batch_size value, but you must make sure the exported model can actually handle such input. A TensorRT model has to be exported with shapes defined for the minimum and maximum batch size it accepts. I’ll go into this issue in more detail in my next Triton post, so stay tuned!

Instance groups

This feature lets you spin up several instances of a model so that inference runs in parallel, which, if all goes well, raises throughput and lowers latency at the cost of using more resources.

Concurrent model execution

As the documentation puts it, Triton’s architecture lets several models, or several instances of one model, run in parallel on a single system. Every incoming request is placed in a queue and scheduled for execution. Depending on the instance gpus setting, which determines how many model instances run in parallel, requests can be processed concurrently on both GPU and CPU.

Dynamic batching

One way to increase a model’s throughput is to group data into larger batches and run inference on the whole batch rather than on many small inputs. Triton’s dynamic batching feature does precisely this. It has a couple of settings: preferred batch size and max queue delay. The former is self-explanatory; the latter sets how long incoming requests are collected to build a dynamic batch.

According to Triton’s documentation, requests can only be dynamically batched by default when every input has an identical shape across those requests; to use dynamic batching when input shapes frequently differ, the client must pad the input tensors in its requests to a common shape. For models whose inputs have dynamic shapes, Triton offers a ragged batching feature. You can find more about it here.

Priorities

Triton lets you set up several queues with different priority levels. Out of the box, there is just one queue. Priorities are configured per deployed model, in its configuration file.

Cache

Turning on the response cache can help you use resources more efficiently. You enable it per model in the configuration file, but Triton must also be launched with the –cache-config local,size=SIZE flags. Further details on this feature are in the documentation.

Model warmup

Once deployed, some models may experience a cold start, meaning the first few requests they handle take noticeably longer to finish. To avoid this, Triton can perform a dry run of model inference before the models are marked as READY.

Model repository

Use the following command to launch the Triton server with a given model repository:

Rather than a path to a local model repository, you can also point to cloud storage such as S3, Google Cloud Storage, and Azure Storage.

The model repository directory has to follow this structure:

Triton can serve several versions of a deployed model.

Metrics

Out of the box, Triton exposes Prometheus metrics with GPU and request statistics.

Business Logic Scripting

To make model inference faster, you can rely on the business logic scripting (BLS) feature, which separates pre- and post-processing steps written in Python/C++/Java from the core model inference. The core model can then be optimized, for example by exporting it to TensorRT with a suitable configuration. The complete pipeline can be built with Triton’s ensemble model, which describes a sequence of operations along with their input and output parameters. In some situations, this improves the utilization of deployed models, since core models can be shared across different ensemble models.

Is it worth using?

Triton belongs in production environments where you can take advantage of all its features. Its real strengths are grouping requests into batches on the fly and scaling models horizontally with multiple instances. These capabilities only pay off under a high enough request load. If you compare a model running in its native framework, such as PyTorch, with the same model in Triton, Triton may turn out slower at low traffic because of the overhead from queuing, batching, and other processing. With heavy request workloads, however, where dynamic batching and scaling the number of models come into play, Triton will be more efficient.

So if Triton is that good, should it be used everywhere? If losing a few percentage points of throughput doesn’t scare you, Triton is a solid choice. Deploying with it lets you move your solution smoothly into production with very few configuration changes.

How do you use it?

Triton includes two powerful tools for tuning model configuration to a particular hardware and software environment: perfanalyzer* and _model_analyzer*. The best configuration may vary from machine to machine, so You should always tune Your configuration on the target production server.

Start by exporting Your models to ONNX and TensorRT formats with various optimizations, such as dynamic batches, half-precision, and quantization. Then run model_analyzer with a defined search space for the parameters. This can take quite a while, so grab a cup of coffee or head out for lunch. When it’s done, You can produce a summary report that compares the configurations. Finally, You pick the best configuration, apply it, and launch the production Triton service.

Summary

Thanks to its high performance and efficient use of resources, Triton is a production-ready tool for deploying models. The tooling around it, such as perf_analyzer and model-analyzer, makes it quick to test various configurations and choose the best one for deployment. It isn’t easy to pick up at first, but once You understand its concepts and typical pitfalls, it becomes a practical tool for machine learning and DevOps engineers. Other deployment tools use Triton as their underlying runtime within larger MLOps pipelines, which simplifies deployment even further. That’s why it pays to know how to deal with Triton’s configuration. In the next post, I’ll share tips&tricks and present code samples, so stay tuned!