Adaptive trace sampling with the OpenTelemetry Collector: taming telemetry volume without losing signal

Observability pipelines can sound like playlists for cloud infrastructure — everything playing at once creates a messy mix. Traces are the lead guitar: they reveal structure and timing, but at scale they can overwhelm storage, dashboards, and budgets. A recent addition to the OpenTelemetry Collector ecosystem gives pipeline builders a new control: a dynamic sampling processor that adapts sample rates in real time and records the effective rate in the trace context so metrics and downstream systems can weight sampled traces correctly. (github.com)

This article explains what the dynamic sampling processor does, how it fits into collector-based pipelines, and the trade-offs to keep in mind — with concrete config and behavioral details so the idea stays practical, not academic.

Why adaptive sampling matters (quick recap)

• High-volume services and chatty libraries can produce many traces per second; naïvely ingesting all spans is often cost-prohibitive.
• Fixed, probabilistic sampling (keep N%) is simple but can miss important or rare behaviors (errors, specific routes).
• Tail-sampling keeps traces based on their final contents (e.g., contains error), but it can require buffering and can be heavy on memory and latency.

Adaptive sampling sits between those approaches: it observes the traffic, applies rules, and assigns sampling rates that can change per-key (for example per service name, route, or HTTP method) so that frequently noisy keys are sampled at a lower rate while important low-volume keys can retain higher fidelity.

What the new dynamic sampling processor does

• It accumulates spans per trace until a trigger (root span arrival or trace timeout) and then evaluates first-match rules against the accumulated trace. The selected rule routes the trace to a sampler which produces a known 1-in-N sample rate. The effective sample rate is encoded in the OpenTelemetry TraceState as an ot=th entry so downstream components can reason about the sampling probability. (pkg.go.dev)
• Decisions are deterministic: a sampler computes a threshold (the rejection threshold encoding) and compares it to trace randomness (from TraceID or an explicit randomness value) so the same trace ID will be consistently sampled or not. For late-arriving spans the processor keeps a per-trace decision cache to ensure consistency. (pkg.go.dev)
• Sampler types include deterministic (fixed percentage) and adaptive samplers such as EMA-based samplers that aim for a target average sampling percentage across keys and adjust rates over time. The processor exposes tuning knobs: buffer size (num_traces), trace_timeout, decision_delay, and decision cache sizes. (pkg.go.dev)

A compact config example Below is an illustrative collector YAML snippet that shows the form of rules and samplers (semantic only — small edits may be needed for a production config):

processors: dynamic_sampling: trace_timeout: 30s decision_delay: 2s num_traces: 50000 decision_cache: sampled_cache_size: 10000 non_sampled_cache_size: 10000 rules: - name: keep-errors conditions: - ‘status.code == 2’ sampler: type: always_sample - name: payment-service conditions: - ‘resource.attributes[“service.name”] == “payment”’ sampler: type: ema_dynamic ema_dynamic: goal_sampling_percentage: 5 key_fields: [“http.method”, “http.route”] adjustment_interval: 15s weight: 0.5 - name: default sampler: type: ema_dynamic ema_dynamic: goal_sampling_percentage: 20 key_fields: [“service.name”, “http.status_code”]

This example demonstrates:

• Rule ordering matters (first-match wins).
• A rule can assert match conditions (status, resource attributes) and choose a sampler.
• EMA dynamic samplers tune sampling per key-fields, useful for keeping a steady budget across many routes or methods. (pkg.go.dev)

Why encoding the sample rate into TraceState matters Recording the effective sampling probability in the W3C TraceState (the ot=th key defined by OpenTelemetry) is the crucial piece that turns sampling from a destructive filter into a statistically-correctable transformation. Systems that convert traces to metrics or compute service-level rates can use the encoded threshold to extrapolate counts and produce accurate, unbiased aggregates despite sampling. The OpenTelemetry spec defines how a rejection threshold is encoded and how to derive adjusted counts from it, enabling consistent sampling and correct downstream weighting. (opentelemetry.io)

How this compares to other sampling processors

• Probabilistic (fixed-rate) samplers: very cheap, simple, but blind to context. They do not adapt per-key and they don’t directly carry the sampling probability in context unless separately configured.
• Tail-sampling processors: make sampling decisions based on the whole trace content for accuracy (for example, keep traces with errors). They often need larger buffers, increased latency, and careful memory sizing.
• The dynamic sampling processor blends the two: it buffers only long enough to evaluate rules and adaptively sets rates per key using samplers that aim for target percentages. It is designed to offer budget control with better signal preservation than a fixed sampler. (pkg.go.dev)

Operational trade-offs and tuning knobs

• Buffering and latency: trace_timeout and decision_delay determine how long traces are held. Longer delays reduce incorrect early decisions but increase per-trace latency and memory.
• Memory footprint: num_traces and decision cache sizes control memory usage. For very high QPS workloads these parameters should be aligned with available RAM and the acceptable tail-latency for trace forwarding.
• Rule design: overly broad catch-all rules can defeat adaptive behavior. Careful key selection (service name, route, status code) helps target the sampler’s adaptivity where it matters.
• Determinism and late spans: the decision cache prevents inconsistent decisions for late spans; cache sizing therefore affects correctness for late-arriving data.
• Stability and maturity: the component is in the Collector Contrib distribution and carries a development/developmental stability status for traces; engineering teams should plan validation and test benches before broad production rollouts. (github.com)

A realistic analogy Think of the pipeline as a live concert sound engineer. The dynamic sampler is not muting the guitar entirely; it’s turning down sections of the mix that are playing at full tilt (very noisy routes) while ensuring the solo still cuts through when the band hits a critical note (an error or a rare route). The ot=th entry is like a label on each recorded take that tells the mixing desk how loud it actually was on stage — so later, when you master the recording into metrics, you can normalize levels properly.

Limitations and sharp edges

• The development status means edge-case bugs, API changes, or behavioral tweaks are still plausible between releases; test harnesses and pilot rollouts mitigate surprises. (github.com)
• If service architecture creates many micro-keys (extreme cardinality on key_fields), adaptive samplers may still struggle to meet per-key goals without aggressive max_keys limits or additional pre-aggregation.
• The processor makes trade-offs: reducing telemetry volume inevitably reduces fidelity. The encoded ot=th makes statistical correction possible, but it doesn’t reconstruct detailed traces that were never sampled.

Why this matters for pipeline design

• Cost-aware ingestion: encoding sampling decisions and using adaptive rules enables predictable telemetry spend while keeping actionable traces.
• Signal preservation: rules can guarantee full sampling for critical conditions (errors, security events) while letting high-volume normal traffic be sampled down.
• Better downstream analytics: with the sampling threshold propagated alongside spans, metrics-generation systems and trace stores can weight and aggregate sampled data correctly.

Concluding note (no call to action) The dynamic sampling processor is an example of pipeline intelligence — moving early decision-making into the collector so telemetry becomes a controlled, instrumented flow rather than an all-or-nothing flood. Its approach — rule-driven routing to samplers, deterministic decisions, and TraceState-based encoding — addresses the practical tension between observability signal and cost in large-scale systems. For teams managing high-volume tracing, this is a valuable capability to understand and evaluate alongside existing sampling strategies. (pkg.go.dev)