Fixing a 20s Kubernetes API Lag in 15 Minutes with OpenTelemetry

Our Kubernetes API was suddenly taking 20 seconds to respond, but traditional pod metrics showed healthy CPU and memory. I needed true observability to find the root cause without rewriting code or paying Datadog’s $0.05 per custom metric surcharge.

I chose SigNoz for a unified metrics, logs, and traces backend, and the OpenTelemetry Operator to auto-instrument our cluster.

The Setup

First, I installed SigNoz directly into our cluster using Helm:

helm repo add signoz https://charts.signoz.io
kubectl create ns platform
helm --namespace platform install my-release signoz/signoz
Next, to get data flowing without touching our Java source code, I installed the OpenTelemetry Operator and simply added this annotation to our application's deployment manifest:
template:
  metadata:
    annotations:
      instrumentation.opentelemetry.io/inject-java: "true"

Enter fullscreen mode Exit fullscreen mode

The Fix
Almost instantly, traces populated the SigNoz dashboard.
The trace proved the database was responding in single-digit milliseconds. The delay was happening before the database call. By clicking from the trace straight into the correlated JVM metrics, I saw application threads were deadlocking waiting for asynchronous external responses. We patched the thread pool logic, and response times dropped back to milliseconds.
Takeaway
Auto-instrumentation via Kubernetes annotations is a superpower. Standardizing on OpenTelemetry saves weeks of engineering time and stops vendor lock-in. Don’t guess why your app is slow—trace it.