Estimate project
← All posts

Adding CPU, GPU and system metrics to the BentoML service metrics enpoint

The piece that was missing for monitoring services deployed with BentoML.

6 min read
Machine LearningPythonOpenTelemetryMLOpsBentoML

Keeping an eye on deployed services is essential if you want to react fast to performance problems or errors as they appear. When a service relies on machine learning models under the hood, it is vital to monitor the resources its operation depends on.

BentoML, a popular choice for serving machine learning models, exposes a REST API out of the box with basic, generic features such as health checks and service metrics monitoring. The metrics are published in an open-telemetry format and can be hooked up to Prometheus and Grafana for aggregation, monitoring and alerting.

In this article, I’ll first show which metrics BentoML tracks by default. Next, I’ll explain which metrics I believe are missing and how to add them. Lastly, I’ll demonstrate how to check the proposed change and look at how it affects the service’s performance. Let’s dive in!

Why BentoML?

As noted in the introduction, BentoML is a framework for building and deploying production-ready ML services. Whereas MLFlow relies on Flask as its HTTP server (WSGI), BentoML uses FastAPI (ASGI), which is better suited to serving models in production. Put differently, BentoML’s strength lies in its support for asynchronous communication. It works with many popular frameworks, e.g. scikit-learn, Transformers, ONNX, PyTorch and TensorFlow, so a single framework can be used for various kinds of ML models. On top of that, the tool offers plenty of handy features out-of-the-box, such as service API documentation in OpenAPI format and basic metrics on request counts and processing times in Open Telemetry format. It makes deployment optimization quick and simple through various Runner classes, such as Triton Inference Server, and adaptive request batching. It lets you build sequences of distributed models and control GPU and CPU resources as well as the number of workers. BentoML ships with a built-in Model Store for versioning models, the bento artifact packaging format, and a Docker container builder tool that makes containerization and deployment seamless.

All in all, BentoML is a general-purpose framework for developing ML services that offers many of the features you need and enables rapid prototyping. At the same time, it is highly configurable, so it can be tuned to your requirements. Most importantly, it supports the most widely used frameworks, which means it fits nearly any ML project.

Default metrics

The list below shows the metrics a BentoML service tracks by default. Clearly, these are generic, request-focused metrics covering the whole API service and the individual runners.

  • bentoml_runner_request_duration_seconds- How many seconds each runner needs to complete the RPC
  • bentoml_runner_request_in_progress- How many runner RPCs are currently in progress in total
  • bentoml_runner_adaptive_batch_size- The runner’s adaptive batch size
  • bentoml_runner_request_total- The total count of runner RPCs
  • bentoml_api_server_request_total- The total count of server HTTP requests
  • bentoml_api_server_request_in_progress- How many server HTTP requests are currently in progress in total
  • bentoml_api_server_request_duration_seconds- How many seconds it takes to complete a server HTTP request

What metrics are missing?

In my view, what’s missing are system metrics showing how resources are consumed while requests are handled. The key ones are RAM and CPU utilization and, for models using hardware acceleration, VRAM and GPU consumption as well. These basic metrics are worth tracking for any AI model, because they let you judge how ready a service model is to cope with a large number of requests, and help you spot potential memory leaks and edge cases in which memory runs low.

Adding them

Out of the box, BentoML’s logs don’t include system metrics. We can, however, extend the BentoML framework’s Runner class and log any custom metrics we like – in our case, system metrics covering CPU, GPU, RAM and VRAM usage. The code below collects these metrics using the psutil and pynvml libraries.

import psutil
import pynvml
import bentoml

pynvml.nvmlInit()
driver_version = pynvml.nvmlSystemGetDriverVersion()
gpu_number = pynvml.nvmlDeviceGetCount()

metrics_system_keys_dict = {
    "node_ram_usage_percentage": "RAM usage in %",
    "node_ram_usage_bytes": "RAM usage in bytes",
    "node_ram_total_bytes": "RAM total in bytes",
    "node_ram_free_bytes": "RAM free in bytes",
    "node_swap_usage_percentage": "swap memory usage in %",
    "node_swap_usage_bytes": "swap memory usage in bytes",
    "node_swap_total_bytes": "swap total memory in bytes",
    "node_swap_free_bytes": "swap memory free in bytes",
    "node_disk_usage_percentage": "disk usage in %",
    "node_disk_usage_bytes": "disk usage in bytes",
    "node_disk_total_bytes": "disk total in bytes",
    "node_disk_free_bytes": "disk free in bytes",
    "node_process_number": "Number of processes runnning in the system",
}

metric_dict = {
    key: bentoml.metrics.Gauge(name=key, documentation=value)
    for key, value in metrics_system_keys_dict.items()
}

metric_gpu_keys_dict = {
    "total_memory": "VRAM total in bytes",
    "free_memory": "VRAM free in bytes",
    "used_memory": "VRAM used in bytes",
    "power_usage": "GPU device power usage in Watts",
    "fan_speed": "GPU fan speed in rpm",
    "utilization_percent": "GPU utilization in %"

}

for gpu_device in range(0, gpu_number):
    for key, value in metric_gpu_keys_dict.items():
        name = f"node_gpu_{gpu_device}_{key}"
        metric_dict[name] = bentoml.metrics.Gauge(name=name, documentation=value)

def update_gpu_metrics():
    for gpu_device in range(0, gpu_number):
        handle = pynvml.nvmlDeviceGetHandleByIndex(gpu_device)
        info = pynvml.nvmlDeviceGetMemoryInfo(handle)

        metric_dict[f"node_gpu_{gpu_device}_total_memory"].set(info.total)
        metric_dict[f"node_gpu_{gpu_device}_free_memory"].set(info.free)
        metric_dict[f"node_gpu_{gpu_device}_used_memory"].set(info.used)
        metric_dict[f"node_gpu_{gpu_device}_power_usage"].set(
            pynvml.nvmlDeviceGetPowerUsage(handle)
        )
        metric_dict[f"node_gpu_{gpu_device}_fan_speed"].set(
            pynvml.nvmlDeviceGetFanSpeed(handle)
        )
        metric_dict[f"node_gpu_{gpu_device}_utilization_percent"].set(
            pynvml.nvmlDeviceGetUtilizationRates(handle).gpu
        )

cpu_list = psutil.cpu_percent(percpu=True)
cpu_no = len(cpu_list)

cpu_usage_metric_list = []
for index in range(0, cpu_no):
    key = f"node_cpu_{index}_usage_percentage"
    metric_dict[key] = bentoml.metrics.Gauge(
        name=key,
        documentation=f"CPU{index} usage in %",
    )

def update_metrics():
    """Update system metrics."""
    for index, cpu_usage in enumerate(psutil.cpu_percent(percpu=True)):
        key = f"node_cpu_{index}_usage_percentage"
        metric_dict[key].set(cpu_usage)

    ram = psutil.virtual_memory()
    metric_dict["node_ram_usage_percentage"].set(ram.percent)
    metric_dict["node_ram_usage_bytes"].set(ram.active)
    metric_dict["node_ram_total_bytes"].set(ram.total)
    metric_dict["node_ram_free_bytes"].set(ram.free)
    swap = psutil.swap_memory()
    metric_dict["node_swap_usage_percentage"].set(swap.percent)
    metric_dict["node_swap_usage_bytes"].set(swap.used)
    metric_dict["node_swap_total_bytes"].set(swap.total)
    metric_dict["node_swap_free_bytes"].set(swap.free)
    disk = psutil.disk_usage("/")
    metric_dict["node_disk_usage_percentage"].set(disk.percent)
    metric_dict["node_disk_usage_bytes"].set(disk.used)
    metric_dict["node_disk_total_bytes"].set(disk.total)
    metric_dict["node_disk_free_bytes"].set(disk.free)
    metric_dict["node_process_number"].set(len(psutil.pids()))

    update_gpu_metrics()

Keep in mind that the pynvml library relies on the NVIDIA NVML library under the hood, so make sure it is installed in your execution environment (e.g. a Docker container). You’ll find more details in the project repository: https://github.com/gpuopenanalytics/pynvml.

We want the Runner class to call the update_metrics function immediately after running inference on the AI model inside the __call__ function.

"""File containing Transformer model handler."""

from typing import Any, Dict, List

import bentoml
from bentoml.io import JSON
from examples.services.metrics import update_metrics

MODEL_NAME = "classification-pipeline"
SERVICE_NAME = "classification-pipeline-service"

model = bentoml.transformers.get(f"{MODEL_NAME}:latest")
_BuiltinRunnable = model.to_runnable()

class CustomRunnable(_BuiltinRunnable):
   @bentoml.Runnable.method(batchable=True, batch_dim=0)
   def __call__(self, input_data: List[str]) -> Dict[str, Any]:
       output = self.model(input_data)
       update_metrics()
       return output

runner = bentoml.Runner(CustomRunnable)
svc = bentoml.Service(
   SERVICE_NAME,
   runners=[runner],
   models=[model],
)

@svc.api(input=JSON(), output=JSON())  # type: ignore
async def classify(input_series: Dict[str, Any]) -> Dict[str, Any]:
   """Perform inference on the deployed model.

   Args:
       input_series: Dictionary with key "data" containing text.

   Returns:
       JSON response with key "prediction" containing list of predicted
           classes.
   """
   response = await runner.async_run(input_series["data"])  # type: ignore
   return {"predictions": response}

Now, once the model is deployed, the default metrics endpoint (/metrics) of the BentoML service will expose additional logs on system resource consumption.

Results

To check that our metrics are being aggregated and exposed in open-telemetry format on the default /metrics endpoint, we can open the address of the service running the ML model in a browser. On the Swagger API documentation page, we can send a test inference request to the model by querying the /classify endpoint with a suitable payload and then calling the /metrics endpoint. Alongside the default logs, the output will include metrics on system resource consumption.

Performance

The extra logic added to the Runner code makes the model respond more slowly. In benchmark tests, collecting the metrics adds about 5 ms of latency on average. This overhead is constant, as it doesn’t depend on which ML model is used, and it can very likely be optimized.

Summary

This article showed what the metrics provided by a BentoML service look like by default and how to extend them as needed. System metrics make it possible to monitor the key resources that machine learning models use heavily, and tracking them makes it possible to detect and fix memory leaks and edge cases.

In my next article, I’ll show how to use a load testing tool together with a Grafana integration to monitor system resource usage while requests are being handled, relying on the custom metrics introduced here. Stay tuned!