Keeping fewer traces can improve observability—if the traces you keep are the ones that explain failure.
Many teams begin with simple probabilistic sampling: retain a fixed percentage of traces and discard the rest. It is predictable and inexpensive, but it makes the decision before the system knows whether a request will fail, become slow, or reveal an unusual dependency path.
Tail sampling reverses that sequence. It waits for spans from a trace, evaluates the trace against explicit policies, and then decides whether to retain it. That makes it possible to keep errors, high-latency journeys, rare attributes, and a representative baseline without storing every routine request.
The benefit is attractive for teams managing constrained compute, storage, or network capacity. The operational cost is equally important: tail sampling is stateful infrastructure, and a careless deployment can quietly discard the evidence it was introduced to protect.
Head and tail sampling answer different questions
Head sampling decides near the beginning of a trace. It has limited information, but it adds little centralized state and can reduce traffic early. Tail sampling decides after spans arrive at a Collector, so policies can consider the trace outcome and duration.
Neither method is automatically superior. Head sampling is a strong default when simplicity, predictable volume, and low Collector overhead matter most. Tail sampling earns its complexity when retaining specific outcomes—such as failed payments or unusually slow requests—is more valuable than retaining a uniform percentage.
The non-negotiable routing rule
The OpenTelemetry Collector tail-sampling processor requires all spans for a trace to reach the same Collector instance. The processor groups spans by trace ID before making its decision. If a load balancer distributes spans from one trace across several replicas, each sampler sees only a fragment and may make an incorrect or incomplete decision.
Scaling therefore requires trace-aware routing before the stateful sampling tier. Do not assume ordinary round-robin load balancing is sufficient. Validate the route with real multi-service traces and during Collector scaling events.
Decision time is an accuracy budget
The processor waits before evaluating a trace. The current default decision_wait is 30 seconds. A shorter wait reduces memory pressure and decision latency, but increases the chance that late spans arrive after the decision. A longer wait improves completeness for long-running traces while holding more data in memory.
Select the value from observed span-arrival behavior—not intuition. Measure how long spans take to reach the sampler for the workflows you care about, including queued operations, intermittent links, cross-region calls, and delayed callbacks.
This connects directly to our article on span links for asynchronous work. A business journey that continues through queues may produce late or independently rooted spans. Sampling design must reflect the actual causal and timing model.
Memory capacity determines whether traces survive
Tail sampling keeps trace data while it waits. The num_traces setting limits how many traces the processor holds in memory. When the buffer cannot accommodate incoming volume, traces may be removed before the decision window expires.
Increasing num_traces may reduce early removal, but uses more memory. Decreasing decision_wait may also reduce pressure, but can make decisions on incomplete traces. This is a capacity trade-off, not a tuning trick.
Estimate the working set from incoming trace rate, average spans per trace, span size, and the decision window. Then test with realistic bursts—not only average traffic.
Design policies around operational value
A useful starting policy set usually includes:
- retain traces containing error outcomes;
- retain traces exceeding a meaningful latency threshold;
- retain critical business workflows or bounded outcome categories;
- keep a modest probabilistic baseline of routine traffic;
- explicitly suppress health checks or other high-volume, low-value activity where appropriate.
Policies should map to decisions. “Keep all errors” is valuable only if error status is recorded consistently. “Keep slow traces” requires a threshold tied to user or service expectations. Business attributes must be bounded, governed, and free of sensitive data.
A sampling policy is not a substitute for sound instrumentation. If an important failure is not expressed in span status, events, or controlled attributes, the sampler cannot infer it reliably.
Late spans need an explicit plan
A span arriving after a decision may inherit a cached decision, or it may be treated as part of a newly observed trace after earlier state is gone. That can create partial traces or inconsistent outcomes.
Decision caches help the processor remember sampled and non-sampled trace IDs after trace data leaves the active buffer. Their size should reflect trace volume and the expected lateness window. Monitor lateness rather than assuming the default behavior is acceptable.
Observe the sampler itself
The tail-sampling processor exposes internal telemetry for operational tuning. At minimum, watch:
- traces dropped before the configured wait time;
- the age of traces removed from the buffer;
- sampling-decision latency;
- late-span age;
- sampled versus non-sampled decisions by policy;
- Collector memory, refused spans, queue pressure, and export failures.
If the sampler drops traces because its buffer is full, the sampling system is making decisions based on capacity rather than policy. Treat that as a reliability signal.
The sampler also belongs inside the resilience design described in our Collector outage-resilience guide. A backend outage, exporter slowdown, or restart can compound the memory already held for sampling decisions.
Test failure modes before production
- Generate complete traces with known error and latency outcomes.
- Confirm every span for each trace reaches the same sampler.
- Delay selected spans beyond the normal arrival window.
- Burst traffic above the expected steady-state rate.
- Restart and scale the sampling tier during load.
- Slow or disconnect the exporter.
- Verify which traces were retained, rejected, or made incomplete.
Document the expected degradation mode. If the sampler becomes overloaded, should the platform preserve critical traces, fall back to a fixed rate, shed routine traffic, or fail open? The correct answer depends on risk and capacity, but it should not be accidental.
A practical adoption sequence
Begin with head sampling and reliable context propagation. Add tail sampling only for a defined use case. Run it in parallel with the existing pipeline, compare retained traces, and measure memory and decision behavior. Introduce a small policy set first, then change one condition at a time.
Tail sampling is successful when teams retain more diagnostic value per unit of telemetry—not merely when ingestion decreases.
What to do this week
- Identify the incident questions your current sample cannot answer.
- Measure span-arrival delay for one critical journey.
- Verify trace-aware routing through the Collector tier.
- Model the in-memory trace working set under burst load.
- Define one error, one latency, and one baseline policy.
- Alert on early drops, late spans, and decision latency.
Organizations interested in supporting vendor-neutral education on OpenTelemetry operations and cost-conscious reliability can explore sponsorship or workshop collaboration with Observability Africa. Sponsors do not control editorial conclusions.