Sending internal JVM metrics to GCP Monitoring with OpenTelemetry (with zero code)
A step-by-step guide to shipping internal metrics of a Java application to the GCP Monitoring service with OpenTelemetry (running as the OpenTelemetry operator on GKE).
In this tutorial, I’ll show how to ship internal metrics of a Java application to the GCP Monitoring service.
I’m going to use OpenTelemetry (running as the OpenTelemetry operator on GKE), and I promise I won’t write even one line of Java code ;).
What is OpenTelemetry?
Let me turn the question around – what isn’t OpenTelemetry?
It isn’t yet another monitoring tool or library. Rather, it’s a standardized way of collecting, processing, and exporting telemetry data such as logs, metrics, and traces.
Imagine our application is instrumented with a particular tool, say Kamon. What happens if we later decide we’d rather use Prometheus? We’d have to rewrite a large part of the application code.
OpenTelemetry changes the rules of the game: its shared SDK and API let developers send signals to practically any destination. The only condition is that the backend speaks the otlp protocol, or that a compatible OpenTelemetry exporter exists.
OpenTelemetry alternatives
When existing applications are already instrumented and expose Prometheus-compatible metrics, you can use them in GCP Monitoring through Managed Service for Prometheus. This is a Prometheus-compatible backend that can be added to a GKE cluster automatically by turning it on in the GKE configuration. Once enabled, it sets up metric collectors, the Google Managed Prometheus controller (gmp-controller), and a few CRDs (such as PodMonitorig) for choosing which pods or services get monitored.
Why use OpenTelemetry? Is GCP monitoring not enough?
GCP Monitoring handles metrics emitted by the GKE system itself very well, along with some container metrics. Some of them, not all. You can’t use it to get metrics from within the application, such as JVM internals, metrics specific to libraries, and so on.
Getting those metrics requires something running inside the JVM that publishes them to a collector.
Existing setup
Let’s say we already have a GKE cluster with a few JVM-based microservices running on it. The GKE nodes already have write access to monitoring, so we skipped workload identity federation, which is the recommended way of granting GKE workloads access to GCP (I chose this to keep things simple).
OpenTelemetry installation
As we’re working with a Kubernetes cluster, we’ll install OpenTelemetry as a Kubernetes operator. The OpenTelemetry operator is made up of:
- the OpenTelemetry controller
- a CRD for the OpenTelemetry collector
- a CRD for OpenTelemetry instrumentation.
You can install the operator in a few ways: with the operator manifest, the Helm chart, or the Operator Hub. I’ll go with the Helm chart. Installing it requires adding the Helm repository first:
$ helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
$ helm repo update
and then installing the chart:
$ helm install --namespace otel-system --create-namespace \
opentelemetry-operator open-telemetry/opentelemetry-operator
Here I used an otel-system namespace, but you can pick any name you like.
Important: the OpenTelemetry operator relies on an admission webhook, which uses TLS by default. With the default settings, Helm assumes cert-manager is already installed in the cluster. The readme file describes a few alternative ways of installing the OpenTelemetry operator chart without cert-manager.
Let’s check that the chart has been installed correctly:
$ kubectl -n otel-system get deployments
NAME READY UP-TO-DATE AVAILABLE AGE
opentelemetry-operator 1/1 1 1 4min
$ kubectl get opentelemetrycollectors
No resources found in default namespace.
Creating the collector
Before we can collect metrics from the applications, we need an OpenTelemetry collector. It is responsible for receiving and processing the metrics, and optionally exporting them to an external system – here, GCP Monitoring.
The collector can be run in several ways:
Deployment
Here, the collector runs like an ordinary application. Scaling it out and in, rolling back to an earlier version, and doing everything else you’d normally do with an application is simple. This is likely the most straightforward way to deploy the collector.
Daemonset
When you need the collector to act as an agent on the Kubernetes nodes, you can run it as a daemonset. This comes in handy for collecting host-level metrics from nodes with the Host Metrics Receiver – in that case, the collector must have direct access to the nodes. The downside is that it can’t be scaled on its own.
Statefulset
This is handy when your pods need static host names.
Sidecar
Here, the collector is injected into a pod as an extra container. The point is to hand off telemetry data to the collector as quickly as possible. Every pod gets its own collector.
We went with the deployment, as it’s the simplest option.
Here’s our collector manifest:
apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
name: gcp
namespace: otel-system
spec:
mode: "deployment"
config: |
receivers:
otlp:
protocols:
grpc:
http:
exporters:
googlecloud:
processors:
batch:
# batch metrics before sending to reduce API usage
send_batch_max_size: 200
send_batch_size: 200
timeout: 5s
memory_limiter:
# drop metrics if memory usage gets too high
check_interval: 1s
limit_percentage: 65
spike_limit_percentage: 20
resourcedetection:
# detect cluster name and location
detectors: [gcp]
timeout: 10s
service:
pipelines:
metrics:
receivers: [otlp]
processors: [batch, memory_limiter, resourcedetection]
exporters: [googlecloud]
Let’s break it down piece by piece.
We’re creating a collector called gcp in the otel-system namespace.
Its spec contains 2 fields: the mode, set to deployment as described above, and the config, which is a regular OpenTelemetry configuration file.
The configuration defines the receivers, processors, and exporters. Finally, these three parts are combined into a pipeline.
We accept only data sent over the otlp protocol, via grpc or http. Each of them gets its own endpoint and port.
Once the data arrives, we process it. First, a batch processor groups individual signals into batches. We set both the batch size and the max size to 200 (the number of metrics) and the timeout to 5s – once that time passes, the batch goes to the next pipeline component no matter how big it currently is.
Next in the pipeline comes the memory_limiter. It’s there to avoid running out of memory when a lot of data has to be processed. We use relative values, so the limits are adjusted dynamically according to the available memory.
The final processor is resourcedetection – it detects resource information from the host system and adds it to signals in the OpenTelemetry format. That’s how we know the name of the source pod, its namespace, and so on. We use the gcp detector to pull information from the google metadata server and GCP-specific environment variables.
Once processed, the data is exported to googlecloud, which simply means it gets written as a metric in the GCP Monitoring system.
Let’s apply the manifest:
kubectl apply -f collector.yaml
Once the manifest is applied, listing the collectors should return some output:
kubectl -n otel-system get opentelemetrycollectors
NAME MODE VERSION READY AGE IMAGE MANAGEMENT
gcp deployment 0.85.0 1/1 3d23h otel/opentelemetry-collector-contrib:0.85.0 managed
So what happens when a collector is created?
The OpenTelemetry controller sets up a workload (here, the deployment) along with a few services for collecting data and monitoring the collector.
kubectl -n otel-system get services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
gcp-collector ClusterIP 10.200.179.87 <none> 4317/TCP,4318/TCP 3d23h
gcp-collector-headless ClusterIP None <none> 4317/TCP,4318/TCP 3d23h
gcp-collector-monitoring ClusterIP 10.200.47.54 <none> 8888/TCP 3d23h
Great, the collector is up and running – time to send some data to it.
Creating the instrumentation
Next, we create an OpenTelemetry object named instrumentation. It describes how autoinstrumentation should be enabled in the application.
For JVM applications, we rely on Java Autoinstrumentation, which runs as a Java agent.
Here’s our instrumentation:
apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
name: java-instrumentation
namespace: default
spec:
exporter:
endpoint: http://gcp-collector.otel-system:4317
java:
env:
- name: OTEL_METRICS_EXPORTER
value: otlp
- name: OTEL_TRACES_EXPORTER
value: none
- name: OTEL_LOGS_EXPORTER
value: none
Since we only collect metrics, we set a few environment variables that turn off traces and logs.
We also point it to the address of the collector we created in the previous step.
At this point, we have the collector and the auto instrumentation configuration, yet there’s still no data. Let’s turn on instrumentation for some services.
Enabling the auto instrumentation
Turning on auto instrumentation only requires adding an annotation to the pod.
Be careful: it’s easy to slip up here and annotate the deployment instead. We need to annotate the pod, so we modify the pod template:
spec:
replicas: 2
template:
metadata:
annotations:
instrumentation.opentelemetry.io/inject-java: “true”
The instrumentation.opentelemetry.io/inject-java annotation accepts these values:
- “true” – injects the instrumentation with the default name from the current namespace (for instance, when there’s just one instrumentation)
- “instrumentation-name” – the instrumentation’s name, when the namespace contains several instrumentations
- “namespace-name/instrumentation-name” – the name of an instrumentation that lives in a different namespace
- “false” – excludes this workload from instrumentation.
You can also annotate the namespace, in which case the instrumentation gets injected into every pod in it.
How is the instrumentation injected? An init container is created that downloads the java agent, a few environment variables are injected, and a volume containing the agent is mounted. The JAVA_TOOL_OPTIONS env variable is also injected or modified to include -javaagent:/otel-auto-instrumentation/javaagent.jar, which in effect attaches the Java agent to the JVM.
Important – this only applies to newly created pods. If we modify the deployment first and create the instrumentation afterwards, it won’t work, and we’ll have to restart the pods by hand.
Checking the metrics in GCP
Now let’s look for our metrics in GCP.
Thanks to the gcp resourcedetector, the monitoring system “is aware” that our JVM metrics come from Kubernetes pods, so in the Metrics explorer we’ll find them under Kubernetes container.
The metrics carry helpful labels such as the service name, pod name, namespace, as well as every Kubernetes label from the pod, which makes filtering data in the metric explorer pretty simple.
Troubleshooting
If something isn’t working as expected, start by looking at Metric management in the GCP Monitoring service. There you can see the error rate and browse the logs (audit logs must be enabled).
Looking through the logs of the collector and the operator controller can help, too.
Wrap Up
OpenTelemetry auto instrumentation lets you collect plenty of useful metrics without adding a single line of code to the application.
Auto instrumentation is only a starting point; instrumenting the application with custom and/or business metrics is probably worth doing as well.
This article is merely a tutorial focused on GCP; I suggest getting to know the OpenTelemetry documentation.