Adding CPU, GPU and system metrics to the BentoML service metrics enpoint
The piece that was missing for monitoring services deployed with BentoML.
Keeping an eye on deployed services is essential if you want to react fast to performance problems or errors as they appear. When a service relies on machine learning models under the hood, it is vital to monitor the resources its operation depends on.
BentoML, a popular choice for serving machine learning models, exposes a REST API out of the box with basic, generic features such as health checks and service metrics monitoring. The metrics are published in an open-telemetry format and can be hooked up to Prometheus and Grafana for aggregation, monitoring and alerting.
In this article, I’ll first show which metrics BentoML tracks by default. Next, I’ll explain which metrics I believe are missing and how to add them. Lastly, I’ll demonstrate how to check the proposed change and look at how it affects the service’s performance. Let’s dive in!
Why BentoML?
As noted in the introduction, BentoML is a framework for building and deploying production-ready ML services. Whereas MLFlow relies on Flask as its HTTP server (WSGI), BentoML uses FastAPI (ASGI), which is better suited to serving models in production. Put differently, BentoML’s strength lies in its support for asynchronous communication. It works with many popular frameworks, e.g. scikit-learn, Transformers, ONNX, PyTorch and TensorFlow, so a single framework can be used for various kinds of ML models. On top of that, the tool offers plenty of handy features out-of-the-box, such as service API documentation in OpenAPI format and basic metrics on request counts and processing times in Open Telemetry format. It makes deployment optimization quick and simple through various Runner classes, such as Triton Inference Server, and adaptive request batching. It lets you build sequences of distributed models and control GPU and CPU resources as well as the number of workers. BentoML ships with a built-in Model Store for versioning models, the bento artifact packaging format, and a Docker container builder tool that makes containerization and deployment seamless.
All in all, BentoML is a general-purpose framework for developing ML services that offers many of the features you need and enables rapid prototyping. At the same time, it is highly configurable, so it can be tuned to your requirements. Most importantly, it supports the most widely used frameworks, which means it fits nearly any ML project.
Default metrics
The list below shows the metrics a BentoML service tracks by default. Clearly, these are generic, request-focused metrics covering the whole API service and the individual runners.
bentoml_runner_request_duration_seconds- How many seconds each runner needs to complete the RPCbentoml_runner_request_in_progress- How many runner RPCs are currently in progress in totalbentoml_runner_adaptive_batch_size- The runner’s adaptive batch sizebentoml_runner_request_total- The total count of runner RPCsbentoml_api_server_request_total- The total count of server HTTP requestsbentoml_api_server_request_in_progress- How many server HTTP requests are currently in progress in totalbentoml_api_server_request_duration_seconds- How many seconds it takes to complete a server HTTP request
What metrics are missing?
In my view, what’s missing are system metrics showing how resources are consumed while requests are handled. The key ones are RAM and CPU utilization and, for models using hardware acceleration, VRAM and GPU consumption as well. These basic metrics are worth tracking for any AI model, because they let you judge how ready a service model is to cope with a large number of requests, and help you spot potential memory leaks and edge cases in which memory runs low.
Adding them
Out of the box, BentoML’s logs don’t include system metrics. We can, however, extend the BentoML framework’s Runner class and log any custom metrics we like – in our case, system metrics covering CPU, GPU, RAM and VRAM usage. The code below collects these metrics using the psutil and pynvml libraries.
import psutil
import pynvml
import bentoml
pynvml.nvmlInit()
driver_version = pynvml.nvmlSystemGetDriverVersion()
gpu_number = pynvml.nvmlDeviceGetCount()
metrics_system_keys_dict = {
"node_ram_usage_percentage": "RAM usage in %",
"node_ram_usage_bytes": "RAM usage in bytes",
"node_ram_total_bytes": "RAM total in bytes",
"node_ram_free_bytes": "RAM free in bytes",
"node_swap_usage_percentage": "swap memory usage in %",
"node_swap_usage_bytes": "swap memory usage in bytes",
"node_swap_total_bytes": "swap total memory in bytes",
"node_swap_free_bytes": "swap memory free in bytes",
"node_disk_usage_percentage": "disk usage in %",
"node_disk_usage_bytes": "disk usage in bytes",
"node_disk_total_bytes": "disk total in bytes",
"node_disk_free_bytes": "disk free in bytes",
"node_process_number": "Number of processes runnning in the system",
}
metric_dict = {
key: bentoml.metrics.Gauge(name=key, documentation=value)
for key, value in metrics_system_keys_dict.items()
}
metric_gpu_keys_dict = {
"total_memory": "VRAM total in bytes",
"free_memory": "VRAM free in bytes",
"used_memory": "VRAM used in bytes",
"power_usage": "GPU device power usage in Watts",
"fan_speed": "GPU fan speed in rpm",
"utilization_percent": "GPU utilization in %"
}
for gpu_device in range(0, gpu_number):
for key, value in metric_gpu_keys_dict.items():
name = f"node_gpu_{gpu_device}_{key}"
metric_dict[name] = bentoml.metrics.Gauge(name=name, documentation=value)
def update_gpu_metrics():
for gpu_device in range(0, gpu_number):
handle = pynvml.nvmlDeviceGetHandleByIndex(gpu_device)
info = pynvml.nvmlDeviceGetMemoryInfo(handle)
metric_dict[f"node_gpu_{gpu_device}_total_memory"].set(info.total)
metric_dict[f"node_gpu_{gpu_device}_free_memory"].set(info.free)
metric_dict[f"node_gpu_{gpu_device}_used_memory"].set(info.used)
metric_dict[f"node_gpu_{gpu_device}_power_usage"].set(
pynvml.nvmlDeviceGetPowerUsage(handle)
)
metric_dict[f"node_gpu_{gpu_device}_fan_speed"].set(
pynvml.nvmlDeviceGetFanSpeed(handle)
)
metric_dict[f"node_gpu_{gpu_device}_utilization_percent"].set(
pynvml.nvmlDeviceGetUtilizationRates(handle).gpu
)
cpu_list = psutil.cpu_percent(percpu=True)
cpu_no = len(cpu_list)
cpu_usage_metric_list = []
for index in range(0, cpu_no):
key = f"node_cpu_{index}_usage_percentage"
metric_dict[key] = bentoml.metrics.Gauge(
name=key,
documentation=f"CPU{index} usage in %",
)
def update_metrics():
"""Update system metrics."""
for index, cpu_usage in enumerate(psutil.cpu_percent(percpu=True)):
key = f"node_cpu_{index}_usage_percentage"
metric_dict[key].set(cpu_usage)
ram = psutil.virtual_memory()
metric_dict["node_ram_usage_percentage"].set(ram.percent)
metric_dict["node_ram_usage_bytes"].set(ram.active)
metric_dict["node_ram_total_bytes"].set(ram.total)
metric_dict["node_ram_free_bytes"].set(ram.free)
swap = psutil.swap_memory()
metric_dict["node_swap_usage_percentage"].set(swap.percent)
metric_dict["node_swap_usage_bytes"].set(swap.used)
metric_dict["node_swap_total_bytes"].set(swap.total)
metric_dict["node_swap_free_bytes"].set(swap.free)
disk = psutil.disk_usage("/")
metric_dict["node_disk_usage_percentage"].set(disk.percent)
metric_dict["node_disk_usage_bytes"].set(disk.used)
metric_dict["node_disk_total_bytes"].set(disk.total)
metric_dict["node_disk_free_bytes"].set(disk.free)
metric_dict["node_process_number"].set(len(psutil.pids()))
update_gpu_metrics()
Keep in mind that the pynvml library relies on the NVIDIA NVML library under the hood, so make sure it is installed in your execution environment (e.g. a Docker container). You’ll find more details in the project repository: https://github.com/gpuopenanalytics/pynvml.
We want the Runner class to call the update_metrics function immediately after running inference on the AI model inside the __call__ function.
"""File containing Transformer model handler."""
from typing import Any, Dict, List
import bentoml
from bentoml.io import JSON
from examples.services.metrics import update_metrics
MODEL_NAME = "classification-pipeline"
SERVICE_NAME = "classification-pipeline-service"
model = bentoml.transformers.get(f"{MODEL_NAME}:latest")
_BuiltinRunnable = model.to_runnable()
class CustomRunnable(_BuiltinRunnable):
@bentoml.Runnable.method(batchable=True, batch_dim=0)
def __call__(self, input_data: List[str]) -> Dict[str, Any]:
output = self.model(input_data)
update_metrics()
return output
runner = bentoml.Runner(CustomRunnable)
svc = bentoml.Service(
SERVICE_NAME,
runners=[runner],
models=[model],
)
@svc.api(input=JSON(), output=JSON()) # type: ignore
async def classify(input_series: Dict[str, Any]) -> Dict[str, Any]:
"""Perform inference on the deployed model.
Args:
input_series: Dictionary with key "data" containing text.
Returns:
JSON response with key "prediction" containing list of predicted
classes.
"""
response = await runner.async_run(input_series["data"]) # type: ignore
return {"predictions": response}
Now, once the model is deployed, the default metrics endpoint (/metrics) of the BentoML service will expose additional logs on system resource consumption.
Results
To check that our metrics are being aggregated and exposed in open-telemetry format on the default /metrics endpoint, we can open the address of the service running the ML model in a browser. On the Swagger API documentation page, we can send a test inference request to the model by querying the /classify endpoint with a suitable payload and then calling the /metrics endpoint. Alongside the default logs, the output will include metrics on system resource consumption.
Performance
The extra logic added to the Runner code makes the model respond more slowly. In benchmark tests, collecting the metrics adds about 5 ms of latency on average. This overhead is constant, as it doesn’t depend on which ML model is used, and it can very likely be optimized.
Summary
This article showed what the metrics provided by a BentoML service look like by default and how to extend them as needed. System metrics make it possible to monitor the key resources that machine learning models use heavily, and tracking them makes it possible to detect and fix memory leaks and edge cases.
In my next article, I’ll show how to use a load testing tool together with a Grafana integration to monitor system resource usage while requests are being handled, relying on the custom metrics introduced here. Stay tuned!