Written by Albert Friedman
on 

Observability in 2025: Tracing, Telemetry, and Reliability Metrics that Actually Help

If you’re standing up (or modernizing) observability this year, you’re doing it in a world where OpenTelemetry (OTel) is the lingua franca, cloud providers accept OTLP directly, and AI/LLM features are shipping into production. That’s great news—and a lot to take in. This article gives you a clear, practical path: how to wire tracing and telemetry you can trust, how to connect metrics to traces (so you can jump from a red chart to a single problematic request), and how to define reliability metrics that match user experience—including for AI workloads.

What’s changed recently that makes this worth revisiting now?

Below, we’ll combine these threads into a crisp workflow you can adopt in weeks, not months.

The new baseline: OpenTelemetry + OTLP

OpenTelemetry gives you SDKs, semantic conventions, and a vendor-neutral collector. The crucial piece is OTLP—one protocol for traces, metrics, and logs. The protocol’s 1.7.0 spec documents stable signals and transport over gRPC and HTTP, so you can standardize instrumentation and switch backends without rewriting exporters. (opentelemetry.io)

On managed platforms, you no longer need bespoke agents per signal. For example, Google Cloud’s Ops Agent can receive OTLP and route your traces to Cloud Trace and metrics to Cloud Monitoring or Managed Service for Prometheus, with a single config. That reduces friction and mistakes when you ship. (cloud.google.com)

Tip: name things before you ship. Set OTEL_SERVICE_NAME (or explicit resources) so telemetry is grouped logically; otherwise you’ll end up with “unknown_service” everywhere. (opentelemetry.io)

Tracing that matters: head vs. tail sampling (and why you need both)

A simple collector snippet (YAML) illustrates the idea:

processors:
  tail_sampling:
    decision_wait: 10s
    policies:
      - name: keep_errors
        type: status_code
        status_code:
          status_codes: [ERROR]
      - name: slow_traces
        type: latency
        latency:
          threshold_ms: 5000

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [tail_sampling]
      exporters: [otlp] # to your backend

Start with head-based sampling (e.g., 5–10%) in the SDK to cap volume, then add tail sampling in the Collector to elevate the “must keep” traces. That hybrid pattern balances cost and fidelity. (opentelemetry.netlify.app)

If you’ve ever stared at a latency heatmap and thought, “I just want to see one of those spikes,” exemplars are for you. An exemplar attaches trace context (trace_id, span_id) to a metric data point. With exemplars enabled in your metrics backend, you can click from a bucket to the exact trace that generated it. (opentelemetry.io)

Here’s a minimal Python example that records a histogram with a trace-linked exemplar using Prometheus client and OpenTelemetry:

from time import sleep, perf_counter
from prometheus_client import Histogram, start_http_server
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor, OTLPSpanExporter

# Set up tracing
trace.set_tracer_provider(TracerProvider())
trace.get_tracer_provider().add_span_processor(
    BatchSpanProcessor(OTLPSpanExporter(endpoint="http://localhost:4318/v1/traces"))
)
tracer = trace.get_tracer(__name__)

# Metrics
latency = Histogram("request_latency_seconds", "Request latency (s)")
start_http_server(8000)  # expose /metrics

def trace_id_hex():
    ctx = trace.get_current_span().get_span_context()
    return f"{ctx.trace_id:032x}" if ctx.trace_id else None

while True:
    with tracer.start_as_current_span("handle_request"):
        t0 = perf_counter()
        sleep(0.05)  # do work
        dt = perf_counter() - t0
        tid = trace_id_hex()
        if tid:
            latency.observe(dt, {"trace_id": tid})  # exemplar with trace_id
        else:
            latency.observe(dt)

This lets you click from a latency chart bucket straight into the corresponding trace when your backend supports exemplars. (prometheus.github.io)

Telemetry for LLMs and agents (yes, it’s different)

Traditional web SLIs (latency, error rate) still matter for AI features, but you also care about:

OpenTelemetry’s Generative AI semantic conventions define standard names for these signals. For example:

As of September 2025, these conventions are in “Development” status with an opt-in mechanism while they stabilize, but they already provide a common shape for AI telemetry. (opentelemetry.io)

Ecosystem support is arriving: LangSmith added end-to-end OTel support in March 2025 for LangChain/LangGraph apps, so you can emit standardized traces and correlate them with system telemetry. If you’re instrumenting agents and tool calls, this makes end-to-end trace context realistic instead of wishful. (changelog.langchain.com)

Reliability metrics that align with users

SRE’s reliability model is still the best starting point:

When picking SLIs, start with the Four Golden Signals—latency, traffic, errors, saturation—and tailor them to your service and dependency calls. These are simple to explain, easy to measure, and catch most regressions before customers do. (sre.google)

For AI features, extend SLIs to include:

You can implement these with the GenAI metrics conventions (plus your own evaluations), then define realistic SLO targets and track budgets weekly.

A 30-day rollout plan

Week 1: Name and ship the basics

Week 2: Correlate metrics and traces

Week 3: Control volume without losing the “spicy” traces

Week 4: Define SLOs and protect privacy

Optional (AI features):

Common pitfalls (and how to avoid them)

The bottom line

Modern observability is less about buying another tool and more about getting the workflow right: name your services, standardize on OTLP, sample smartly, connect metrics to traces with exemplars, and hold yourself to SLOs that reflect real user experience. If you’re shipping AI features, add token/cost/throughput metrics and instrument agent steps with the emerging GenAI conventions. The good news: you don’t have to guess anymore—there’s a clear path, and most of the heavy lifting is built into OpenTelemetry and today’s cloud platforms. (opentelemetry.io)

Happy instrumenting—and may your error budget stay unspent this quarter.